Data synchronization method and device

Abstract

The invention discloses a data synchronization method and device, and relates to the technical field of computers. The specific implementation mode of the method comprises the steps of obtaining a business change log in real time, and determining one or more pieces of first change data corresponding to the business change log; determining a timestamp of each piece of first change data, and determining a first minimum timestamp corresponding to the current preset time period based on each timestamp; in response to the fact that the first minimum timestamp does not reach the preset time node, storing the first change data corresponding to the current preset time period into an intermediate database according to the timestamp; and in response to the first minimum timestamp reaching the preset time node, obtaining second change data corresponding to a historical preset time period before the current preset time period in the intermediate database, and performing zipper operation on the second change data and historical data in the data warehouse. According to the embodiment, streaming processing is realized, and meanwhile, the data integrity in the data zipper process is ensured.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method and device for data synchronization. Background Art

[0002] Data warehouse zipper technology is a method for tracking data changes. It records data by saving data with different timestamps and data status in a table. It can not only retain the historical version of the data but also reduce storage costs.

[0003] However, in the existing zipping process of data synchronization, it is usually necessary to compare and associate the full data tables of different dates, identify the effective period of each data, and write the data back to the data directory, resulting in a long data zipping time. Moreover, since it is impossible to ensure the consistency of data from multiple data sources at the same time point, multiple full data tables of zipper can only correspond to a single data source. For example, only the commodity price tables of different dates can be zipped to obtain the zipped commodity price table, but it is impossible to zip the commodity price and logistics information at the same time. Summary of the invention

[0004] In view of this, the embodiment of the present invention provides a method and device for data synchronization. By acquiring the first change data corresponding to the business change log of the current preset time period, the full data zipping process can be disassembled into the zipping process of the change data in multiple time periods, reducing the peak of resource processing, and achieving the effect of streaming processing with the real-time acquisition of the business change log. In addition, by comparing the first minimum timestamp corresponding to the current preset time period with the preset time node, the data integrity of the second change data in the data zipping process is guaranteed, and the zipping error caused by data delay in abnormal scenarios is prevented.

[0005] To achieve the above objective, according to one aspect of an embodiment of the present invention, a method for data synchronization is provided.

[0006] A data synchronization method according to an embodiment of the present invention includes: receiving a business change log in real time, and determining one or more first change data corresponding to the business change log; determining a timestamp of each of the first change data, and determining a first minimum timestamp corresponding to a current preset time period based on each of the timestamps; in response to the first minimum timestamp not reaching a preset time node, storing the first change data corresponding to the current preset time period in an intermediate database according to the timestamp; in response to the first minimum timestamp reaching a preset time node, obtaining second change data corresponding to a historical preset time period before the current preset time period in the intermediate database, and performing a zipper operation on the second change data and the historical data in the data warehouse, so as to synchronize the historical data using the second change data.

[0007] Optionally, determining one or more first change data corresponding to the business change log includes: determining one or more target business data tables corresponding to the business change log from a business database; wherein the business database stores business data tables corresponding to different data sources respectively; and obtaining the first change data corresponding to the business change log from each of the target business data tables.

[0008] Optionally, after determining the first minimum timestamp corresponding to the current preset time period based on each of the timestamps, the method further includes: writing the first minimum timestamp corresponding to the current preset time period into a metadata file of the data lake; wherein the metadata file stores second minimum timestamps corresponding to multiple historical time periods; monitoring the metadata file in real time, and determining one or more target second minimum timestamps that are earlier than the first minimum timestamp from the second minimum timestamp; and using the historical time periods corresponding to one or more of the target second minimum timestamps as the historical preset time periods.

[0009] Optionally, in response to the number of the historical preset time periods being multiple; the zipper operation on the second change data and the historical data in the data warehouse includes: sorting the multiple historical preset time periods according to time; for each historical preset time period from early to late, executing in sequence: updating the source code field and cursor field corresponding to the historical data according to the second change data corresponding to the historical preset time period; updating the latest status of the historical data according to the source code field, and updating the historical track of the historical data according to the cursor field.

[0010] Optionally, the zipper operation on the second change data and the historical data in the data warehouse includes: locating the modification position of the second change data in the historical data based on an index pre-constructed in the data warehouse; and performing a zipper operation on the historical data according to the locating result.

[0011] Optionally, the index includes two levels, wherein the first level index is the storage bucket index of the data warehouse; the second level index is the step index corresponding to each storage bucket, indicating multiple step encoding intervals in each storage bucket.

[0012] Optionally, the modification position of the second change data in the historical data is located based on the index pre-constructed in the data warehouse, including: determining a target storage bucket corresponding to the second change data from multiple storage buckets according to the first-level index; determining a target step coding interval corresponding to the second change data from multiple step coding intervals corresponding to the target storage bucket according to the second-level index; and determining the modification position from the target step interval according to the file name of the second change data.

[0013] To achieve the above objective, according to another aspect of an embodiment of the present invention, a device for data synchronization is provided.

[0014] A data synchronization device according to an embodiment of the present invention includes: an acquisition module, which is used to receive a business change log in real time and determine one or more first change data corresponding to the business change log; a determination module, which is used to determine the timestamp of each of the first change data and determine the first minimum timestamp corresponding to the current preset time period based on each of the timestamps; an update module, which is used to store the first change data corresponding to the current preset time period in an intermediate database according to the timestamp in response to the first minimum timestamp not reaching the preset time node; in response to the first minimum timestamp reaching the preset time node, obtain the second change data corresponding to the historical preset time period before the current preset time period in the intermediate database, and perform a zipper operation on the second change data and the historical data in the data warehouse, so as to synchronize the historical data with the second change data.

[0015] To achieve the above objective, according to another aspect of an embodiment of the present invention, an electronic device for data synchronization is provided.

[0016] An electronic device for data synchronization according to an embodiment of the present invention comprises: one or more processors; a storage device for storing one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors implement a method for data synchronization according to an embodiment of the present invention.

[0017] To achieve the above objective, according to another aspect of an embodiment of the present invention, a computer-readable storage medium is provided.

[0018] A computer-readable storage medium according to an embodiment of the present invention stores a computer program, and when the program is executed by a processor, a data synchronization method according to an embodiment of the present invention is implemented.

[0019] One embodiment of the above invention has the following advantages or beneficial effects: by obtaining the first change data corresponding to the business change log of the current preset time period, the full data zipping process can be disassembled into the zipping process of the change data in multiple time periods, reducing the peak of resource processing, and achieving the effect of streaming processing with the real-time acquisition of the business change log. In addition, by comparing the first minimum timestamp corresponding to the current preset time period with the preset time node, the data integrity of the second change data in the data zipping process is guaranteed, and the zipping error caused by data delay in abnormal scenarios is prevented.

[0020] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with specific implementation examples. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.

[0022] Figure 1 is a schematic diagram of the main flow of a method for data synchronization according to an embodiment of the present invention;

[0023] Figure 2 is a schematic diagram of a main process of obtaining first change data according to an embodiment of the present invention;

[0024] Figure 3 is a schematic diagram of setting a message queue according to an embodiment of the present invention;

[0025] Figure 4 is a schematic diagram of interactions between multiple systems involved in data synchronization according to an embodiment of the present invention;

[0026] Figure 5 is a schematic diagram of the interaction process between a calculation operator and a submission operator according to an embodiment of the present invention;

[0027] Figure 6 is a schematic diagram of a main process of determining a historical preset time period according to an embodiment of the present invention;

[0028] Figure 7 is a flowchart of a specific embodiment of data synchronization according to an embodiment of the present invention;

[0029] Figure 8 is a schematic diagram of a main process of zipping the second change data corresponding to a plurality of historical preset time periods respectively according to an embodiment of the present invention;

[0030] Fig. 9 is a schematic diagram of a data structure obtained after data zipping according to an embodiment of the present invention;

[0031] Fig.10is a main flow chart of another data synchronization method according to an embodiment of the present invention;

[0032] Fig.11 is a schematic diagram of a main process of performing index-based positioning according to an embodiment of the present invention;

[0033] Fig.12 is a schematic diagram of main modules of a data synchronization device according to an embodiment of the present invention;

[0034] Fig.13 is an exemplary system architecture diagram to which embodiments of the present invention may be applied;

[0035] Fig.14 It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0036] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0037] It should be pointed out that the embodiments of the present invention and the technical features therein may be combined with each other without conflict.

[0038] It should be noted that the collection, collection, updating, analysis, processing, use, transmission, storage and other aspects of user personal information involved in the technical solution of this disclosure are in compliance with the provisions of relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. Necessary measures are taken for user personal information to prevent illegal access to user personal information data and maintain the security of user personal information, network security and national security.

[0039] For ease of understanding, the specific scenario in which the embodiment of the present invention is applied is first described. Under normal circumstances, offline data synchronization is performed in units of "days", that is, the data of the whole day of day T is synchronized to the data warehouse on day T+1 to align the data in the data warehouse with the data on day T. Specifically, there will be many pieces of data generated at different times in day T, so after the end of day T, it takes a lot of time to zip up all the data of day T at one time, and the data synchronization efficiency is very low. Moreover, for the data generated on day T, different data may correspond to different data sources. For example, the name of the product is modified at 0:00 (the corresponding data source is the product name data table in the business system), the price of the product is modified at 12:00 (the corresponding data source is the product price data table in the business system), and as the product is continuously sold, the remaining inventory of the product is finally modified at 18:00 (the corresponding data source is the product inventory data table in the business system). Then in the data synchronization process, the commodity name data table, commodity price data table and commodity inventory data table in the business system and the data warehouse will be compared in turn, and the data table in the data warehouse will be updated by the difference data obtained by comparison, which means that each data source can only be updated separately, that is, the commodity inventory data table in the data warehouse is updated by using the commodity inventory data table in the business system, which further affects the efficiency of data synchronization. Therefore, the embodiment of the present invention provides a new data synchronization method, which can disassemble the full amount of data to be updated on T day before the end of T day, that is, disassemble the whole day of T day into multiple current preset time periods, and perform streaming processing on multiple data in each current preset time period, thereby reducing the peak value of resource processing. However, the one-time zipper synchronization process of the full amount of data is split into multiple current preset time periods. The data zipper process will inevitably have the problem of data synchronization consistency, that is, there is a situation where some data is delayed due to delay problems. Therefore, the embodiment of the present invention compares the first minimum timestamp corresponding to each current preset time period and compares it with the preset time period to ensure the data integrity in the data zipper process.

[0040] Figure 1 is a schematic diagram of main steps of a method for data synchronization according to an embodiment of the present invention.

[0041] like Figure 1 As shown, the data synchronization method of the embodiment of the present invention mainly includes the following steps:

[0042] Step S101: receiving a service change log in real time, and determining one or more first change data corresponding to the service change log;

[0043] Step S102: determining the timestamp of each first change data, and determining the first minimum timestamp corresponding to the current preset time period based on each timestamp;

[0044] Step S103: In response to the first minimum timestamp not reaching the preset time node, the first change data corresponding to the current preset time period is stored in the intermediate database according to the timestamp;

[0045] Step S104: In response to the first minimum timestamp reaching a preset time node, obtain the second change data corresponding to the historical preset time period before the current preset time period in the intermediate database, and perform a zipper operation on the second change data and the historical data in the data warehouse to synchronize the historical data using the second change data.

[0046] Among them, the business change log refers to the log file generated when the business changes, that is, it is generated after each user operation and submission. For example, the user fills in the name, price, quantity and size of the product on the product management page. When the confirmation submission button is clicked, the background will generate a corresponding business change log, and the business change log will record in detail the content modified by this operation, the specific time of the modification, the operation type, etc. For example, the user puts a new product on the shelf, fills in the name of the product as product 1, the selling price is 100 yuan, and the inventory quantity is 100 in the product management page, then after clicking the confirmation submission button, a business change log including all the operation information will be generated, in which each modified content is a first change data. Specifically, in the actual application process, the business data will be stored in the MySQL database, and the generated business change log will also be stored in the Binlog file of the MySQL database to record the events of the changes made to the database. Among them, the Binlog file is the core of data replication and data recovery in the MySQL database, which records all DDL and DML operations (such as INSERT, UPDATE, DELETE).

[0047] The first change data refers to the data of each change in the business change log. For example, the user changes the name of the product from A to B, adjusts the price to 100 yuan, and adjusts the size to 50 cm * 30 cm on the product management page, and clicks the confirmation submit button after setting. Then the generated business change log actually contains three first change data, namely, the name is changed from A to B, the price is adjusted to 100 yuan, and the size is adjusted to 50 cm * 30 cm, and the three first change data will actually be stored as three source data tables during the data storage process, namely, the product name table, the product price table, and the product size table.

[0048] Therefore, for the process of obtaining the first change data in step 101, the following can be done: Figure 2 As shown, including:

[0049] Step S201: Obtaining business change logs in real time;

[0050] Step S202: determining one or more target business data tables corresponding to the business change log from a business database; wherein the business database stores business data tables corresponding to different data sources respectively;

[0051] Step S203: Obtain first change data corresponding to the business change log from each target business data table.

[0052] Through the above process, it can be known that in the business database, business data tables (source data tables) corresponding to different businesses (i.e. different data sources) will be stored. In addition to the above-mentioned product name table, product price table, and product size table, it can also include, for example, a product logistics and transportation table, a product after-sales status table, etc. As the specific operation content included in the business change log is different, it is only necessary to access the corresponding target business data table to obtain the first change data, and there is no need to access the business data table that is not related to the actual change operation.

[0053] For the timestamp in step S102, in an optional embodiment, it can be equal to the operation time or change time of the first change data. Similarly, taking the above-mentioned adjustment operation of the product name, price and size as an example, since the three first change data are actually submitted synchronously by the user clicking the confirmation submit button, the timestamps of the three first change data are the same, which are all the time when the user clicks the confirmation submit button. However, if the user changes the above three first change data in different operations, the timestamps corresponding to each first change data will be different. Exemplarily, after changing the product name, the user clicks submit once at 10:00, then the user changes the product price and clicks submit again at 10:01, then the user changes the product size and clicks submit at 10:02, then the timestamp of the first change data corresponding to the product name is 10:00, the timestamp of the first change data corresponding to the product price is 10:01, and the timestamp of the first change data corresponding to the product size is 10:02.

[0054] In an optional embodiment, the acquired first change data can be put into different message queues according to different services, and the message queues can be used to implement subsequent asynchronous consumption processing of the first change data. Figure 3As shown in the figure, after the business change log indicates that the user created a new product with a size of 20*30, a price of 60, and a name of Product 1 at 23:50, the product basic information table, product attribute information table, and product price information table corresponding to the business change log are called from the business database to obtain the corresponding first change data. Since the data sources of the three types of first change data are different, they can be placed in the corresponding message queues respectively ( Figure 3 From top to bottom, they are message queue 1 corresponding to the basic information table of the product, message queue 2 corresponding to the product attribute information table, and message queue 3 corresponding to the product price information table, where the message queue can be a kafka queue. Subsequently, the user performed another operation at 23:52 to adjust the size of product 1. Similar to the above process, the corresponding first change data will be obtained from the product attribute information table and put into the corresponding message queue 2. It can be understood that data at different time points in the same message queue are usually processed in chronological order according to the first-in-first-out order.

[0055] As for the current preset time period and the first minimum timestamp in step S102, as mentioned above, the preset time period is actually a division of the whole day of day T, so it can be specifically set according to the actual needs of the user, such as 1 minute, 5 minutes, 10 minutes, etc. However, in order to achieve the purpose of the present invention to synchronize streaming data for the real-time acquired business change log as much as possible, it is not recommended to set the preset time period too long, preferably 1 minute to 5 minutes. That is to say, for day T, 0:00~0:01 is a preset time period, 0:01~0:02 is a current preset time period, 0:02~0:03 is a preset time period... and so on, the whole day of day T can be divided into multiple preset time periods, and as the current time changes, different first change data will fall into different current preset time periods, that is, the data at 0:00 belongs to the current preset time period of 0:00~0:01. Exemplarily, the current preset time period mentioned in the present invention uses the left side as a closed interval and the right side as an open interval, that is, 0:00∈[0:00, 0:01). According to different settings, the right side can also be set as a closed interval and the left side as an open interval, as long as the multiple current preset time periods are continuous and can completely cover the whole day of day T, so as to ensure that the first change data that does not belong to the preset time period will not appear. The first minimum timestamp can be understood as the minimum timestamp corresponding to one or more first change data in the preset time period, that is, it represents the timestamp corresponding to the earliest first change data in one or more first change data. For this, it is actually only necessary to compare the timestamps of one or more first change data that fall within the current preset time period, and the present invention will not explain this further.

[0056] In a further optional embodiment, the open source Flink stream processing framework may be used to implement the process of determining the first minimum timestamp and subsequently storing the first timestamp in an intermediate database. Specifically, Figure 4 As shown, the Flink stream processing framework will first obtain one or more first change data from the business database according to the business change log, and put the first change data into the corresponding message queue, and then calculate the first minimum timestamp corresponding to the current preset time period in each current preset time period according to the timestamp corresponding to each first change data in the message queue. Specifically, in the Flink stream processing framework, there are multiple operators with different functions, such as read operators, calculation operators, submission operators, etc. Among them, the read operator is used to read the timestamp of the first change data from the message queue, and the calculation operator calculates the minimum timestamp of different fields in the current preset time period according to the preset calculation rules (such as the calculation rule of taking the minimum time), and finally sends it to the submission operator, which submits and stores the first minimum timestamp.

[0057] For example, the preset time period can be set to a minute level, such as 1 minute, 3 minutes, 5 minutes, or 10 minutes. The interaction process between the calculation operator and the submission operator is as follows: Figure 5 As shown, for each calculation operator, a certain field will be continuously monitored during the current preset time period corresponding to the current time, and the timestamp of the field will be continuously compared according to the business change log to obtain the minimum timestamp of each field. Then, after the current preset time period ends, each calculation operator sends the minimum timestamp of the corresponding field to the only submission operator, and the submission operator calculates the first minimum timestamp corresponding to all fields.

[0058] It should be noted that since the Flink stream processing framework itself does not have a storage function and is essentially a computing engine, after obtaining the first minimum timestamp, in an optional embodiment, it is necessary to store the first minimum timestamp in the metadata file of the database, and determine the historical preset time period that can be used for data zipping by monitoring the metadata file, and schedule subsequent data zipping processes for the historical preset time period and historical data in the database. Specifically, the process of obtaining the historical preset time period is as follows: Figure 6 As shown, including:

[0059] Step S601: Obtaining business change logs in real time;

[0060] Step S602: determining one or more target business data tables corresponding to the business change log from the business database;

[0061] Step S603: acquiring first change data corresponding to the business change log from each target business data table, and putting each of the first change logs into a corresponding message queue;

[0062] Step S604: sequentially obtaining timestamps corresponding to a plurality of first change data in the message queue within the current preset time period, and determining a first minimum timestamp;

[0063] Step S605: writing the first minimum timestamp corresponding to the current preset time period into the metadata file of the data lake; wherein the metadata file stores the second minimum timestamps corresponding to multiple historical time periods respectively;

[0064] Step S606: monitoring the metadata file in real time, and determining one or more target second minimum timestamps earlier than the first minimum timestamp from the second minimum timestamps;

[0065] Step S607: taking the historical time periods corresponding to one or more target second minimum timestamps as historical preset time periods.

[0066] It is understandable that as time goes by, the current time period will continue to change. For example, if the preset time period is set to 10 minutes, when the time is 0:08, the current preset time period is 0:00~0:10, and when the time is 0:12, the current preset time period is 0:10~0:20, and 0:00~0:10 becomes the historical preset time period. Therefore, as the first minimum timestamps corresponding to multiple current preset time periods do not reach the preset node, multiple former current preset time periods will become historical preset time periods, and the corresponding first change data will be stored in the intermediate database.

[0067] Among them, in the embodiment of the present invention, the first minimum timestamp is submitted through the commit operator of Flink, so the metadata file can be a commit file, and the storage format is similar to the json format. The data lake refers to the Hudi data management framework, which is designed for streaming and batch data processing in big data lakes. In a further optional embodiment, the monitoring of metadata can be performed according to pre-configured monitoring rules. For example, the monitored file type, data table mode, data path, and metadata file name, etc. can be set in advance.

[0068] It is understandable that in order to reduce the peak value of data processing, the preset time period can be set to the minute level so as to realize the processing of the business change log obtained by the implementation, but the essence of data zipper technology is to reduce the number of zippers and improve the data processing effect through a one-time zipper process. Therefore, in an embodiment of the present invention, the preset time node can be set according to the date, that is, 00:00 every day. Taking the preset time period of 10 minutes as an example, the submission operator in the Flink streaming processing framework will submit the first minimum timestamp once every 10 minutes, and generate a metadata file accordingly, wherein each metadata file indicates the modification time and modification content of the business change log executed in the current 10 minutes. That is to say, before 00:00 on T+1, there will actually be multiple historical preset time periods of T day, namely 0:00~0:10, 0:10~0:20, 0:20~0:30, ..., 23:40~23:50, 23:50~24:00 of T day, and the data of multiple historical preset time periods will execute the data zipper process after the first current preset time period after 00:00.

[0069] For ease of understanding, we use Figure 7 Take this as an example to explain in detail:

[0070] exist Figure 7 In the example, the user performed three modification operations at 23:50, 23:52 and 00:01 of the next day, respectively, setting the preset time period to 1 minute and the preset time node to 00:00 every day.

[0071] For the current preset time period of 23:50-23:51, based on the business change log of 23:50, it is determined that the first minimum timestamp is 23:50. By comparing with the preset time node, it is found that the preset time node has not been reached. Therefore, the first change data corresponding to 23:50-23:51 (Product 1 Src Map[] Cru Map[20*30,60]) is stored in the intermediate database.

[0072] For the current preset time period of 23:52-23:53, based on the business change log of 23:52, it is determined that the first minimum timestamp is 23:52. By comparing with the preset time node, it is found that the preset time node has not been reached, so the first change data (Product 1 Src Map[] Cru Map[50*30]) corresponding to 23:52-23:53 is stored in the intermediate database. It can be understood that at this time, 23:50-23:51 is already a historical preset time period before the current preset time period.

[0073] For the current preset time period of 00:01-00:02, based on the business change log of 00:01, it is determined that the first minimum timestamp is 00:01. By comparing with the preset time node, it is found that the preset time node has been reached, which means that the data of the previous day has been stored in the business database, and data synchronization between the business database and the data warehouse can be performed. Then at this time, the zipper operation will be performed on the second change data corresponding to the historical preset time period before 00:01-00:02 in the intermediate database, that is, the synchronization process of the business data of the previous day is completed. It can be understood that at this time, 23:50-23:51 and 23:52-23:53 are both historical preset time periods compared to 00:01-00:02.

[0074] In a further optional embodiment, in response to the first minimum timestamp reaching a preset time node, it is also necessary to store the first change data in the intermediate database, and synchronously delete the second change data corresponding to the historical preset time period in the intermediate database, that is, to ensure that the intermediate database always stores the change data that has not been synchronized.

[0075] It should be noted that in the actual business data storage process, data delays may occur due to certain anomalies, that is, after the user performs a related operation, the first change data is not obtained from the business database in time within the current preset time period of the current processing, but the first change data is obtained in the subsequent current preset time period. In view of this situation, the data synchronization method provided by the present invention can also smoothly synchronize data and avoid the problem of data zipping when the data is incomplete. Specifically, the same Figure 7 Taking the implementation example, when the delayed first change data of 23:58 is received within the current preset time period of 00:01-00:02, then when calculating the first minimum timestamp, the first minimum timestamp obtained is 23:58, which has not reached the preset time node. Therefore, even in the current preset time period of 00:01-00:02, the data zipper technology will not be executed, and the previous historical preset time period will be zipped in the next current preset time period, such as 00:02-00:03.

[0076] After the above Figures 1 to 7, describes a process of how to synchronize second change data corresponding to a historical preset time period with first change data corresponding to a current preset time period when the full amount of data is divided by preset time periods in an embodiment of the present invention. However, since there are usually multiple second change data in an actual process, different from the prior art of data zipping by comparing metadata tables, the present invention provides a new data zipping method, which can realize an atomic zipping process according to the time corresponding to the second change data.

[0077] In an optional embodiment, in response to the situation that there are multiple historical preset time periods, the specific process of performing data zipping can be as follows: Figure 8 As shown, including:

[0078] Step S801: sorting multiple historical preset time periods according to time;

[0079] Step S802: for each historical preset time period from early to late, sequentially executing: updating the source code field and the cursor field corresponding to the historical data according to the second change data corresponding to the historical preset time period;

[0080] Step S803: updating the latest state of the historical data according to the source code field, and updating the historical track of the historical data according to the cursor field.

[0081] It is understandable that the data zipper process can reflect the process of data changes, so it is necessary to synchronize data in the chronological order of the historical preset time period. Among them, for the data table that needs to perform data zippering, the secondary data partition dp partition, the start date start_date field and the end date end_date field are usually set. In the dp partition, two enumeration values ​​ACTIVE and EXPIRED are further included, where ACTIVE is used to store the latest status of the data, and EXPIRED is used to store the historical trajectory of the data. The data closed loop is achieved through the secondary partitions of ACTIVE and EXPIRED. For example, Fig. 9 As shown, in Fig. 9 In , sku_id represents the unique code of the product. For example, code 10000 represents a product of brand 01, and code 10001 represents a product of brand 02. Fig. 9It can be seen that the product with code 10000 in brand 01 is named A from January 1, 2022 to December 31, 2022, and is renamed B from January 1, 2023 to December 31, 2023, and is renamed C from January 1, 2024 to the present, where EXPIRED indicates the historical track state and ACTIVE indicates the current state. Therefore, the embodiment of the present invention updates the source code field and the cursor field in chronological order according to each historical preset time period through the above steps S801 to S803, realizes the update of the latest state and historical track, and finally achieves the effect of data zipper.

[0082] It should be noted that when synchronizing data with a data lake, incremental data is usually written to a .log file separately, and stock data is written to a parquet file. The merging of incremental data and stock data is done based on the table service Compaction and read-time merge Mor. In the prior art, the update is done by overwriting the old data with the new data. In the embodiment of the present invention, Figure 7 As shown, by defining two MAPs, Src and Cur, to store the current state and the historical trajectory state respectively, data zipper can be realized by continuously updating Src MAP and Cur MAP. Compared with the existing technology, it does not require the process of data coverage, which is more efficient and quick.

[0083] Since the embodiment of the present invention performs streaming data synchronization based on the business change log received in real time, it is particularly important to quickly and efficiently identify the location of the data to be changed in the historical data during the data synchronization process. In an optional embodiment, a data synchronization method provided by the embodiment of the present invention is as follows Fig.10 As shown, specifically including:

[0084] Step S1001: acquiring a business change log in real time, and determining one or more first change data corresponding to the business change log;

[0085] Step S1002: determining the timestamp of each first change data, and determining the first minimum timestamp corresponding to the current preset time period based on each timestamp;

[0086] Step S1003: in response to the first minimum timestamp reaching a preset time node, obtaining second change data corresponding to a historical preset time period before the current preset time period in the intermediate database;

[0087] Step S1004: locating the modification position of the second change data in the historical data based on the index pre-constructed in the data warehouse;

[0088] Step S1005: According to the positioning result, a zipper operation is performed on the second changed data and the historical data in the data warehouse.

[0089] Among them, in order to quickly locate the modification position, the present invention optimizes the index in the data warehouse. Taking the data lake as an example, the existing data lake framework usually comes with a bucket index, that is, multiple buckets are set, and each bucket stores multiple data with the same hash value. By locating the bucket, the first positioning of the stored data is achieved, and then the data is located from the bucket according to the file name. In the embodiment of the present invention, the index includes two levels, wherein the first-level index is the bucket index that comes with the data warehouse; the second-level index is the step index corresponding to each bucket, indicating multiple step coding intervals in each bucket. Among them, the step index can be understood as setting a continuous code for each data. For example, the step coding interval corresponding to each bucket is ten thousand, then the code stored in the first bucket is 00001-10001, and the code stored in the second bucket is 10001-20001, and so on, multiple data in multiple buckets are encoded in sequence.

[0090] In a further optional embodiment, the process of positioning based on the index is as follows: Fig.11 As shown, including:

[0091] Step S1101: determining a target storage bucket corresponding to the second changed data from multiple storage buckets according to the first-level index;

[0092] Step S1102: According to the second-level index, determine the target step-length coding interval corresponding to the second change data from the multiple step-length coding intervals corresponding to the target storage bucket;

[0093] Step S1103: Determine the modification position from the target step interval according to the file name of the second change data.

[0094] It can be understood that, for step S1103, the modification position is actually determined by comparing the similarity between the file name of the second change data and the file name of the data stored in the database. Through the above process, the target storage bucket corresponding to the second change data can be quickly located through the first-level index, and the target step encoding interval where the second change data is located can be further quickly located in the target storage bucket using the second-level index. Compared with the prior art that only sets the first-level index, the interval required for file name comparison can be further reduced by setting the second-level index. The speed of comparing the second-level index is much faster than the speed of comparing the file name. Therefore, the positioning speed of the second change data is further improved through the above process.

[0095] It should be noted that since the resource consumption of encoding and sorting different data is very large, and the data lake itself does not belong to the engine, it is impossible to set and maintain the index dictionary for encoding and sorting in the memory. Therefore, in the embodiment of the present invention, the business auto-increment ID is used for step encoding to achieve the sorting effect, and the step index code is associated with the data file. In addition, the first-level index needs to be maintained on the Step field, and the second-level index is preferably maintained on the partition field, otherwise a large number of sorting operations are required to realize the indexing function.

[0096] According to the data synchronization method of the embodiment of the present invention, by obtaining the first change data corresponding to the business change log of the current preset time period, the full data zipping process can be disassembled into the zipping process of the change data in multiple time periods, reducing the peak of resource processing, and achieving the effect of streaming processing with the real-time acquisition of the business change log. In addition, by comparing the first minimum timestamp corresponding to the current preset time period with the preset time node, the data integrity of the second change data in the data zipping process is guaranteed, and the zipping error caused by data delay in abnormal scenarios is prevented.

[0097] Fig.12 4 is a schematic diagram of main modules of a data synchronization device according to an embodiment of the present invention.

[0098] like Fig.12 As shown, the data synchronization device 1200 of the embodiment of the present invention includes:

[0099] The acquisition module 1201 is used to receive a business change log in real time and determine one or more first change data corresponding to the business change log;

[0100] A determination module 1202 is used to determine a timestamp of each of the first change data, and determine a first minimum timestamp corresponding to a current preset time period based on each of the timestamps;

[0101] Update module 1203 is used to store the first change data corresponding to the current preset time period in the intermediate database according to the timestamp in response to the first minimum timestamp not reaching the preset time node; in response to the first minimum timestamp reaching the preset time node, obtain the second change data corresponding to the historical preset time period in the intermediate database that is before the current preset time period, and perform a zipper operation on the second change data and the historical data in the data warehouse to synchronize the historical data using the second change data.

[0102] In an optional embodiment of the present invention, the acquisition module 1201 is also used to determine one or more target business data tables corresponding to the business change log from a business database; wherein the business database stores business data tables corresponding to different data sources respectively; and obtain the first change data corresponding to the business change log from each of the target business data tables.

[0103] In an optional embodiment of the present invention, the update module 1203 is also used to, after determining the first minimum timestamp corresponding to the current preset time period based on each of the timestamps, write the first minimum timestamp corresponding to the current preset time period into the metadata file of the data lake; wherein the metadata file stores second minimum timestamps corresponding to multiple historical time periods respectively; monitor the metadata file in real time, and determine one or more target second minimum timestamps that are earlier than the first minimum timestamp from the second minimum timestamp; and use the historical time periods corresponding to one or more of the target second minimum timestamps as the historical preset time periods.

[0104] In an optional embodiment of the present invention, in response to the number of the historical preset time periods being multiple; the update module 1203 is further used to sort the multiple historical preset time periods according to time; for each of the historical preset time periods from early to late, sequentially execute: according to the second change data corresponding to the historical preset time period, update the source code field and cursor field corresponding to the historical data; update the latest status of the historical data according to the source code field, and update the historical trajectory of the historical data according to the cursor field.

[0105] In an optional embodiment of the present invention, the update module 1203 is also used to locate the modification position of the second change data in the historical data based on the index pre-constructed in the data warehouse; and perform a zipper operation on the second change data and the historical data in the data warehouse according to the positioning result.

[0106] In an optional embodiment of the present invention, the index includes two levels, wherein the first level index is the storage bucket index of the data warehouse; the second level index is the step index corresponding to each storage bucket, indicating multiple step encoding intervals in each of the storage buckets.

[0107] In an optional embodiment of the present invention, the update module 1203 is also used to determine, based on the first-level index, a target storage bucket corresponding to the second change data from multiple storage buckets; based on the second-level index, determine a target step coding interval corresponding to the second change data from multiple step coding intervals corresponding to the target storage bucket; and based on the file name of the second change data, determine the modification position from the target step interval.

[0108] According to the data synchronization device of the embodiment of the present invention, by obtaining the first change data corresponding to the business change log of the current preset time period, the full data zipping process can be disassembled into the zipping process of the change data in multiple time periods, reducing the peak of resource processing, and achieving the effect of streaming processing with the real-time acquisition of the business change log. In addition, by comparing the first minimum timestamp corresponding to the current preset time period with the preset time node, the data integrity of the second change data in the data zipping process is guaranteed, and the zipping error caused by data delay in abnormal scenarios is prevented.

[0109] Fig.13 An exemplary system architecture 1300 is shown to which the method or apparatus for data synchronization according to an embodiment of the present invention may be applied.

[0110] like Fig.13 As shown, system architecture 1300 may include terminal devices 1301, 1302, 1303, network 1304 and server 1305. Network 1304 is used to provide a medium for communication links between terminal devices 1301, 1302, 1303 and server 1305. Network 1304 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0111] The user can use the terminal devices 1301, 1302, 1303 to interact with the server 1305 through the network 1304 to receive or send data, etc. Various communication client applications can be installed on the terminal devices 1301, 1302, 1303, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0112] The terminal devices 1301 , 1302 , and 1303 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0113] Server 1305 may be a server that provides various services, such as a background management server that provides support for business data modified by users using terminal devices 1301, 1302, and 1303. The background management server may analyze and process received data such as business change logs.

[0114] It should be noted that the data synchronization method provided in the embodiment of the present invention is generally executed by the server 1305 , and accordingly, the data synchronization device is generally set in the server 1305 .

[0115] It should be understood that Fig.13 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0116] Reference below Fig.14 , which shows a schematic diagram of the structure of a computer system 1400 of a terminal device suitable for implementing an embodiment of the present invention. Fig.14 The terminal device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0117] like Fig.14 As shown, the computer system 1400 includes a central processing unit (CPU) 1401, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1402 or a program loaded from a storage part 1408 into a random access memory (RAM) 1403. In the RAM 1403, various programs and data required for the operation of the system 1400 are also stored. The CPU 1401, the ROM 1402, and the RAM 1403 are connected to each other through a bus 1404. An input / output (I / O) first interface 1405 is also connected to the bus 1404.

[0118] The following components are connected to the I / O first interface 1405: an input section 1406 including a keyboard, a mouse, etc.; an output section 1407 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1408 including a hard disk, etc.; and a communication section 1409 including a network first interface card such as a LAN card, a modem, etc. The communication section 1409 performs communication processing via a network such as the Internet. A drive 1410 is also connected to the I / O first interface 1405 as needed. A removable medium 1411, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1410 as needed, so that a computer program read therefrom is installed into the storage section 1408 as needed.

[0119] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 1409, and / or installed from the removable medium 1411. When the computer program is executed by the central processing unit (CPU) 1401, the above-mentioned functions defined in the system of the present invention are executed.

[0120] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0121] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0122] The modules involved in the embodiments of the present invention may be implemented in software or hardware. The modules described may also be set in a processor. For example, they may be described as: a processor includes an acquisition module, a determination module, and an update module. The names of these modules do not, in some cases, constitute limitations on the modules themselves. For example, the acquisition module may also be described as "a module that receives a business change log in real time and determines one or more first change data corresponding to the business change log."

[0123] As another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiment; or it may exist independently and not be assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device includes: obtaining a business change log in real time, and determining one or more first change data corresponding to the business change log; determining the timestamp of each of the first change data, and determining the first minimum timestamp corresponding to the current preset time period based on each of the timestamps; in response to the first minimum timestamp not reaching the preset time node, storing the first change data corresponding to the current preset time period in the intermediate database according to the timestamp; in response to the first minimum timestamp reaching the preset time node, obtaining the second change data corresponding to the historical preset time period before the current preset time period in the intermediate database, and performing a zipper operation on the second change data and the historical data in the data warehouse, so as to synchronize the historical data with the second change data.

[0124] According to the technical solution of the embodiment of the present invention, by obtaining the first change data corresponding to the business change log of the current preset time period, the full data zipping process can be disassembled into the zipping process of the change data in multiple time periods, reducing the peak of resource processing, and achieving the effect of streaming processing with the real-time acquisition of the business change log. In addition, by comparing the first minimum timestamp corresponding to the current preset time period with the preset time node, the data integrity of the second change data in the data zipping process is guaranteed, and the zipping error caused by data delay in abnormal scenarios is prevented.

[0125] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for data synchronization, characterized in that: include: Acquire a business change log in real time, and determine one or more first change data corresponding to the business change log; Determine a timestamp of each of the first change data, and determine a first minimum timestamp corresponding to a current preset time period based on each of the timestamps; In response to the first minimum timestamp not reaching a preset time node, storing the first change data corresponding to the current preset time period in an intermediate database according to the timestamp; In response to the first minimum timestamp reaching a preset time node, second change data corresponding to a historical preset time period before the current preset time period in the intermediate database is obtained, and a zipper operation is performed on the second change data and the historical data in the data warehouse to synchronize the historical data using the second change data.

2. The method according to claim 1, characterized in that: The determining one or more first change data corresponding to the service change log includes: Determine one or more target business data tables corresponding to the business change log from a business database; wherein the business database stores business data tables corresponding to different data sources respectively; The first change data corresponding to the business change log is respectively obtained from each of the target business data tables.

3. The method according to claim 1, characterized in that After determining the first minimum timestamp corresponding to the current preset time period based on each of the timestamps, the method further includes: Writing the first minimum timestamp corresponding to the current preset time period into the metadata file of the data lake; wherein the metadata file stores the second minimum timestamps corresponding to multiple historical time periods respectively; Monitor the metadata file in real time, and determine one or more target second minimum timestamps that are earlier than the first minimum timestamp from the second minimum timestamp; One or more historical time periods corresponding to the target second minimum timestamps are used as the historical preset time period.

4. The method according to claim 3, characterized in that In response to the number of the historical preset time periods being multiple; the zipping operation on the second change data and the historical data in the data warehouse includes: Sorting the plurality of historical preset time periods according to time; For each of the historical preset time periods from early to late, sequentially executing: updating the source code field and the cursor field corresponding to the historical data according to the second change data corresponding to the historical preset time period; The latest state of the historical data is updated according to the source code field, and the historical track of the historical data is updated according to the cursor field.

5. The method according to claim 1, characterized in that The performing a zipper operation on the second changed data and the historical data in the data warehouse includes: Locating a modification position of the second change data in the historical data based on an index pre-constructed in the data warehouse; According to the positioning result, a zipper operation is performed on the second changed data and the historical data in the data warehouse.

6. The method according to claim 5, characterized in that: The index includes two levels, where The first-level index is the storage bucket index that comes with the data warehouse; The second level index is a step index corresponding to each storage bucket, indicating a plurality of step encoding intervals in each storage bucket.

7. The method according to claim 6, characterized in that The locating the modification position of the second change data in the historical data based on the index pre-built in the data warehouse includes: Determine, according to the first-level index, a target storage bucket corresponding to the second changed data from a plurality of storage buckets; According to the second-level index, determining a target step-length coding interval corresponding to the second change data from a plurality of step-length coding intervals corresponding to the target storage bucket; The modification position is determined from the target step interval according to the file name of the second change data.

8. A data synchronization device, characterized in that: include: An acquisition module, configured to receive a business change log in real time and determine one or more first change data corresponding to the business change log; A determination module, configured to determine a timestamp of each of the first change data, and determine a first minimum timestamp corresponding to a current preset time period based on each of the timestamps; An updating module, configured to store the first change data corresponding to the current preset time period in an intermediate database according to the timestamp in response to the first minimum timestamp not reaching a preset time node; In response to the first minimum timestamp reaching a preset time node, second change data corresponding to a historical preset time period before the current preset time period in the intermediate database is obtained, and a zipper operation is performed on the second change data and the historical data in the data warehouse to synchronize the historical data using the second change data.

9. An electronic device for data synchronization, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

10. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

11. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the method according to any one of claims 1 to 7.

Images