Method and device for data synchronization

Abstract

The present invention discloses a method and device for data synchronization, which relates to the field of computer technology. A specific implementation of the method includes: acquiring a business change log in real time and determining one or more first change data corresponding to the business change log; determining the timestamp of each first change data, and determining the first minimum timestamp corresponding to the current preset time period based on each timestamp; in response to the first minimum timestamp not reaching the preset time node, storing the first change data corresponding to the current preset time period in an intermediate database according to the timestamp; in response to the first minimum timestamp reaching the preset time node, acquiring second change data corresponding to a historical preset time period that is located before the current preset time period in the intermediate database, and performing a zipper operation on the second change data and the historical data in the data warehouse. This implementation implements streaming processing while ensuring data integrity during the data zippering process.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method and device for data synchronization. Background Art

[0002] Data warehouse zipper technology is a method for tracking data changes. It records data by saving data with different timestamps and data status in a table. This not only preserves historical versions of data but also reduces storage costs.

[0003] However, the existing data synchronization zipping process typically requires comparing and correlating full data tables from different dates, identifying the valid period for each data point, and writing the data back to the data directory, resulting in a very long data zipping process. Furthermore, because the consistency of data from multiple data sources cannot be guaranteed at the same point in time, multiple zipped full data tables can only correspond to a single data source. For example, you can only zip product price tables from different dates to obtain the zipped product price table, but you cannot zip product prices and logistics information simultaneously. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a data synchronization method and apparatus. By acquiring the first change data corresponding to the business change log for the current preset time period, the full data zipping process can be broken down into zipping processes for change data within multiple time periods, reducing resource processing peaks and achieving streaming processing effects with the real-time acquisition of business change logs. Furthermore, by comparing the first minimum timestamp corresponding to the current preset time period with the preset time node, the data integrity of the second change data during the data zipping process is ensured, preventing zipping errors caused by data delays in abnormal scenarios.

[0005] To achieve the above objective, according to one aspect of an embodiment of the present invention, a method for data synchronization is provided.

[0006] A data synchronization method according to an embodiment of the present invention includes: receiving a business change log in real time, and determining one or more first change data corresponding to the business change log; determining a timestamp of each first change data, and determining a first minimum timestamp corresponding to a current preset time period based on each timestamp; in response to the first minimum timestamp not reaching a preset time node, storing the first change data corresponding to the current preset time period in an intermediate database according to the timestamp; in response to the first minimum timestamp reaching a preset time node, obtaining second change data corresponding to a historical preset time period before the current preset time period in the intermediate database, and performing a zipper operation on the second change data and the historical data in the data warehouse, so as to synchronize the historical data with the second change data.

[0007] Optionally, determining one or more first change data corresponding to the business change log includes: determining one or more target business data tables corresponding to the business change log from a business database; wherein the business database stores business data tables corresponding to different data sources respectively; and obtaining the first change data corresponding to the business change log from each of the target business data tables.

[0008] Optionally, after determining the first minimum timestamp corresponding to the current preset time period based on each of the timestamps, the method further includes: writing the first minimum timestamp corresponding to the current preset time period into a metadata file of the data lake; wherein the metadata file stores second minimum timestamps corresponding to multiple historical time periods; monitoring the metadata file in real time, and determining one or more target second minimum timestamps that are earlier than the first minimum timestamp from the second minimum timestamps; and using the historical time periods corresponding to one or more target second minimum timestamps as the historical preset time periods.

[0009] Optionally, in response to the fact that there are multiple historical preset time periods; the zipper operation on the second change data and the historical data in the data warehouse includes: sorting the multiple historical preset time periods according to time; for each historical preset time period from early to late, sequentially executing: updating the source code field and cursor field corresponding to the historical data according to the second change data corresponding to the historical preset time period; updating the latest status of the historical data according to the source code field, and updating the historical trajectory of the historical data according to the cursor field.

[0010] Optionally, the zipper operation on the second change data and the historical data in the data warehouse includes: locating the modification position of the second change data in the historical data based on an index pre-constructed in the data warehouse; and performing a zipper operation on the historical data according to the locating result.

[0011] Optionally, the index includes two levels, wherein the first level index is the bucket index of the data warehouse; the second level index is the step index corresponding to each bucket, indicating multiple step encoding intervals in each bucket.

[0012] Optionally, based on the index pre-constructed in the data warehouse, the modification position of the second change data in the historical data is located, including: determining the target storage bucket corresponding to the second change data from multiple storage buckets according to the first-level index; determining the target step coding interval corresponding to the second change data from multiple step coding intervals corresponding to the target storage bucket according to the second-level index; and determining the modification position from the target step interval according to the file name of the second change data.

[0013] To achieve the above objective, according to another aspect of an embodiment of the present invention, a data synchronization device is provided.

[0014] A data synchronization device according to an embodiment of the present invention includes: an acquisition module for receiving a business change log in real time and determining one or more first change data corresponding to the business change log; a determination module for determining the timestamp of each first change data and determining the first minimum timestamp corresponding to the current preset time period based on each timestamp; an update module for storing the first change data corresponding to the current preset time period in an intermediate database according to the timestamp in response to the first minimum timestamp not reaching the preset time node; and acquiring, in response to the first minimum timestamp reaching the preset time node, second change data corresponding to a historical preset time period before the current preset time period in the intermediate database, and performing a zipper operation on the second change data and the historical data in the data warehouse to synchronize the historical data using the second change data.

[0015] To achieve the above objective, according to another aspect of an embodiment of the present invention, an electronic device for data synchronization is provided.

[0016] An electronic device for data synchronization according to an embodiment of the present invention includes: one or more processors; a storage device for storing one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors implement a method for data synchronization according to an embodiment of the present invention.

[0017] To achieve the above objective, according to another aspect of an embodiment of the present invention, a computer-readable storage medium is provided.

[0018] A computer-readable storage medium according to an embodiment of the present invention stores a computer program, which, when executed by a processor, implements a data synchronization method according to an embodiment of the present invention.

[0019] One embodiment of the above invention has the following advantages or beneficial effects: by obtaining the first change data corresponding to the business change log for the current preset time period, the full data zipping process can be broken down into zipping processes for change data within multiple time periods, reducing resource processing peaks and achieving streaming processing effects with the real-time acquisition of business change logs. In addition, by comparing the first minimum timestamp corresponding to the current preset time period with the preset time node, the data integrity of the second change data during the data zipping process is ensured, preventing zipping errors caused by data delays in abnormal scenarios.

[0020] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings are provided for a better understanding of the present invention and are not intended to limit the present invention.

[0022] Figure 1 This is a schematic diagram of the main flow of a method for data synchronization according to an embodiment of the present invention;

[0023] Figure 2 This is a schematic diagram of the main process of obtaining first change data according to an embodiment of the present invention;

[0024] Figure 3 is a schematic diagram of setting a message queue according to an embodiment of the present invention;

[0025] Figure 4 is a schematic diagram of interactions between multiple systems involved in data synchronization according to an embodiment of the present invention;

[0026] Figure 5 Schematic diagram of the interaction process between the calculation operator and the submission operator according to an embodiment of the present invention;

[0027] Figure 6 This is a schematic diagram of the main process of determining a historical preset time period according to an embodiment of the present invention;

[0028] Figure 7 is a flowchart of a specific embodiment of data synchronization according to an embodiment of the present invention;

[0029] Figure 8 This is a schematic diagram of the main process of zipping the second change data corresponding to multiple historical preset time periods according to an embodiment of the present invention;

[0030] Figure 9 is a schematic diagram of the data structure obtained after data zipping according to an embodiment of the present invention;

[0031] Figure 10This is a schematic diagram of the main flow of another data synchronization method according to an embodiment of the present invention;

[0032] Figure 11 This is a schematic diagram of the main process of index-based positioning according to an embodiment of the present invention;

[0033] Figure 12 is a schematic diagram of main modules of a data synchronization apparatus according to an embodiment of the present invention;

[0034] Figure 13 is an exemplary system architecture diagram in which embodiments of the present invention may be applied;

[0035] Figure 14 It is a schematic diagram of the structure of a computer system of a terminal device or server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0036] The following description of exemplary embodiments of the present invention is made in conjunction with the accompanying drawings, in which various details of the embodiments of the present invention are included to facilitate understanding. These details should be considered as merely exemplary. Therefore, it should be appreciated by those skilled in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0037] It should be pointed out that, in the absence of conflict, the embodiments of the present invention and the technical features therein may be combined with each other.

[0038] It should be noted that the collection, collection, updating, analysis, processing, use, transmission, and storage of user personal information involved in the technical solutions disclosed herein all comply with relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. Necessary measures are taken with respect to user personal information to prevent unauthorized access to user personal information data and to safeguard the security of user personal information, network security, and national security.

[0039] To facilitate understanding, we will first describe the specific scenarios in which embodiments of the present invention are applied. Typically, offline data synchronization is performed on a daily basis. That is, on day T+1, all data from day T is synchronized to the data warehouse to align the data in the data warehouse with the data from day T. Specifically, day T may contain many pieces of data generated at different times. Therefore, after day T ends, a significant amount of time is required to perform a one-time zip of all the data from day T, resulting in very low data synchronization efficiency. Furthermore, different data generated on day T may correspond to different data sources. For example, at 0:00, the name of a product is modified (the corresponding data source is the product name data table in the business system), and at 12:00, the price of the product is modified (the corresponding data source is the product price data table in the business system). As the product continues to sell, the remaining inventory of the product is ultimately modified at 6:00 PM (the corresponding data source is the product inventory data table in the business system). During data synchronization, the product name, price, and inventory tables in the business system and the data warehouse are sequentially compared. The data tables in the data warehouse are updated based on the difference data obtained from the comparison. This means that each data source can only be updated individually, i.e., the product inventory table in the data warehouse is updated using the product inventory table in the business system, further impacting the efficiency of data synchronization. Therefore, an embodiment of the present invention provides a new data synchronization method that can, before the end of day T, break down the entire day of day T into multiple current preset time periods and stream-process the multiple data in each current preset time period, thereby reducing resource processing peaks. However, splitting the one-time zipper synchronization process for the entire data into multiple current preset time periods inevitably leads to data synchronization consistency issues, i.e., some data may be written later due to latency issues. Therefore, an embodiment of the present invention ensures data integrity during the data zippering process by comparing the first minimum timestamp corresponding to each current preset time period and comparing it with the preset time periods.

[0040] Figure 1 FIG. 1 is a schematic diagram of main steps of a data synchronization method according to an embodiment of the present invention.

[0041] like Figure 1 As shown, the data synchronization method of the embodiment of the present invention mainly includes the following steps:

[0042] Step S101: receiving a service change log in real time, and determining one or more first change data corresponding to the service change log;

[0043] Step S102: determining the timestamp of each first change data, and determining a first minimum timestamp corresponding to the current preset time period based on each timestamp;

[0044] Step S103: In response to the first minimum timestamp not reaching the preset time node, the first change data corresponding to the current preset time period is stored in the intermediate database according to the timestamp;

[0045] Step S104: In response to the first minimum timestamp reaching the preset time node, obtain the second change data corresponding to the historical preset time period before the current preset time period in the intermediate database, and perform a zipper operation on the second change data and the historical data in the data warehouse to synchronize the historical data using the second change data.

[0046] A business change log refers to a log file generated when a business change occurs, specifically after each user action and submission. For example, a user enters the name, price, quantity, and size of a product on the product management page. After clicking the Submit button, a corresponding business change log is generated in the backend. This log details the modified content, the specific time of the modification, the type of operation, and so on. For example, a user lists a new product and enters the name "Product 1," the selling price "100 yuan," and the total inventory quantity "100" on the product management page. After clicking the Submit button, a business change log is generated containing all the operation information. Each modified item is considered a first change data item. Specifically, in actual applications, business data is stored in a MySQL database, and the generated business change log is also stored in the MySQL database's Binlog file, recording database modification events. Binlog files are the core of data replication and data recovery in MySQL databases, recording all DDL and DML operations (such as INSERT, UPDATE, and DELETE).

[0047] The first change data refers to the data of each change in the business change log. For example, the user changes the name of the product from A to B, adjusts the price to 100 yuan, and adjusts the size to 50 cm * 30 cm on the product management page, and clicks the Confirm Submit button after setting. Then the generated business change log actually contains three first change data, namely, the name is changed from A to B, the price is adjusted to 100 yuan, and the size is adjusted to 50 cm * 30 cm. The three first change data are actually stored as three source data tables during the data storage process, namely the product name table, the product price table, and the product size table.

[0048] Therefore, for the process of obtaining the first change data in step 101, the following can be done: Figure 2 As shown, including:

[0049] Step S201: Obtain business change logs in real time;

[0050] Step S202: determining one or more target business data tables corresponding to the business change log from the business database; wherein the business database stores business data tables corresponding to different data sources respectively;

[0051] Step S203: Obtain first change data corresponding to the business change log from each target business data table.

[0052] Through the above process, it can be seen that in the business database, business data tables (source data tables) corresponding to different businesses (i.e., different data sources) will be stored. In addition to the above-mentioned product name table, product price table, and product size table, it can also include, for example, a product logistics and transportation table, a product after-sales status table, etc. As the specific operation content included in the business change log is different, it is only necessary to access the corresponding target business data table to obtain the first change data, and there is no need to access business data tables that are not related to the actual change operation.

[0053] In an optional embodiment, the timestamp in step S102 can be equivalent to the operation time or change time of the first change data. Taking the above-mentioned adjustment operations of the product name, price, and size as an example, since the three first change data are actually submitted synchronously by the user clicking the confirm submit button, the timestamps of the three first change data are the same, which is the time when the user clicks the confirm submit button. However, if the user changes the above three first change data in different operations, the timestamps corresponding to each first change data will be different. For example, after changing the product name, the user clicks submit once at 10:00, then the user changes the product price and clicks submit again at 10:01, then the user changes the product size and clicks submit at 10:02, then the timestamp of the first change data corresponding to the product name is 10:00, the timestamp of the first change data corresponding to the product price is 10:01, and the timestamp of the first change data corresponding to the product size is 10:02.

[0054] In an optional embodiment, the acquired first change data can be put into different message queues according to different services, and the message queues can be used to implement subsequent asynchronous consumption processing of the first change data. Figure 3As shown in the following example, after the business change log indicates that the user created a new product with a size of 20*30, a price of 60, and a name of Product 1 at 23:50, the product basic information table, product attribute information table, and product price information table corresponding to the business change log are called from the business database to obtain the corresponding first change data. Since the data sources of the three types of first change data are different, they can be placed in the corresponding message queues ( Figure 3 From top to bottom, the following are message queue 1 corresponding to the product basic information table, message queue 2 corresponding to the product attribute information table, and message queue 3 corresponding to the product price information table. The message queues can be Kafka queues. Subsequently, at 23:52, the user performed another operation to adjust the size of product 1. Similar to the previous process, the corresponding first change data is retrieved from the product attribute information table and placed in the corresponding message queue 2. It is understood that data at different time points in the same message queue is generally processed in chronological order, based on first-in, first-out (FIFO) order.

[0055] As for the current preset time period and the first minimum timestamp in step S102, as mentioned above, the preset time period is actually a division of the whole day of day T, so it can be specifically set according to the actual needs of the user, such as 1 minute, 5 minutes, 10 minutes, etc. However, in order to achieve the purpose of the present invention to synchronize streaming data for business change logs obtained in real time as much as possible, it is not recommended to set the preset time period too long, preferably 1 minute to 5 minutes. That is to say, for day T, 0:00~0:01 is a preset time period, 0:01~0:02 is a current preset time period, 0:02~0:03 is a preset time period... and so on, the whole day of day T can be divided into multiple preset time periods, and as the current time changes, different first change data will fall into different current preset time periods, that is, the data at 0:00 belongs to the current preset time period of 0:00~0:01. Exemplarily, the current preset time period mentioned in the present invention uses the left side as a closed interval and the right side as an open interval, that is, 0:00∈[0:00, 0:01). According to different settings, the right side can also be set as a closed interval and the left side as an open interval. It is sufficient to ensure that multiple current preset time periods are coherent and can completely cover the entire day of day T to ensure that the first change data that does not belong to the preset time period will not appear. The first minimum timestamp can be understood as the minimum timestamp corresponding to one or more first change data in the preset time period, that is, it represents the timestamp corresponding to the earliest first change data in one or more first change data. For this, it is actually only necessary to compare the timestamps of one or more first change data that fall within the current preset time period, and the present invention will not explain this further.

[0056] In a further optional embodiment, the open source Flink stream processing framework can be used to implement the process of determining the first minimum timestamp and subsequently storing the first timestamp in an intermediate database. Specifically, Figure 4 As shown, the Flink stream processing framework will first obtain one or more first change data from the business database according to the business change log, and put the first change data into the corresponding message queue, and then calculate the first minimum timestamp corresponding to the current preset time period in each current preset time period based on the timestamp corresponding to each first change data in the message queue. Specifically, in the Flink stream processing framework, there are multiple operators with different functions, such as read operators, calculate operators, submit operators, etc. Among them, the read operator is used to read the timestamp of the first change data from the message queue, and the calculate operator calculates the minimum timestamp of different fields in the current preset time period according to the preset calculation rules (such as the calculation rule of taking the minimum time), and finally sends it to the submit operator, which submits and stores the first minimum timestamp.

[0057] For example, the preset time period can be set to minute level, such as 1 minute, 3 minutes, 5 minutes, or 10 minutes. The interaction process between the calculation operator and the submission operator is as follows: Figure 5 As shown, each calculation operator continuously monitors a field during the preset time period corresponding to the current time and compares the timestamp of that field with the business change log to obtain the minimum timestamp of each field. After the preset time period ends, each calculation operator sends the minimum timestamp of the corresponding field to a unique submission operator, which calculates the first minimum timestamp for all fields.

[0058] It should be noted that since the Flink stream processing framework itself does not have a storage function and is essentially a computing engine, after obtaining the first minimum timestamp, in an optional embodiment, it is necessary to store the first minimum timestamp in the metadata file of the database, and determine the historical preset time period that can be used for data zipping by monitoring the metadata file, and schedule subsequent data zipping processes for the historical preset time period and historical data in the database. Specifically, the process of obtaining the historical preset time period is as follows: Figure 6 As shown, including:

[0059] Step S601: Obtain business change logs in real time;

[0060] Step S602: determining one or more target business data tables corresponding to the business change log from the business database;

[0061] Step S603: obtaining first change data corresponding to the business change log from each target business data table, and placing each first change log into a corresponding message queue;

[0062] Step S604: sequentially obtaining timestamps corresponding to a plurality of first change data in the message queue within the current preset time period, and determining a first minimum timestamp;

[0063] Step S605: Write the first minimum timestamp corresponding to the current preset time period into the metadata file of the data lake; wherein the metadata file stores the second minimum timestamps corresponding to multiple historical time periods respectively;

[0064] Step S606: monitoring the metadata file in real time, and determining one or more target second minimum timestamps that are earlier than the first minimum timestamp from the second minimum timestamps;

[0065] Step S607: Using the historical time periods corresponding to the one or more target second minimum timestamps as historical preset time periods.

[0066] It's understandable that the current time period will change over time. For example, if the preset time period is set to 10 minutes, at 0:08, the current preset time period is 0:00-0:10. However, at 0:12, the current preset time period becomes 0:10-0:20, and 0:00-0:10 becomes the historical preset time period. Therefore, if the first minimum timestamps corresponding to multiple current preset time periods do not reach the preset node, the multiple former current preset time periods will become historical preset time periods, and the corresponding first change data will be stored in the intermediate database.

[0067] In this embodiment of the present invention, the first minimum timestamp is submitted via Flink's commit operator. Therefore, the metadata file can be a commit file, stored in a format similar to JSON. The data lake refers to the Hudi data management framework, designed specifically for streaming and batch data processing in big data lakes. In further optional embodiments, metadata monitoring can be performed according to preconfigured monitoring rules. For example, the monitored file type, data table schema, data path, and metadata file name can be set in advance.

[0068] It is understandable that in order to reduce the peak value of data processing, the preset time period can be set to the minute level so as to realize the processing of the business change logs obtained by implementation, but the essence of data zipper technology is to reduce the number of zippers and improve the data processing effect through a one-time zipper process. Therefore, in an embodiment of the present invention, the preset time node can be set according to the date, that is, 00:00 every day. Taking the preset time period of 10 minutes as an example, the submission operator in the Flink streaming processing framework will submit the first minimum timestamp once every 10 minutes, and generate a metadata file accordingly, where each metadata file indicates the modification time and modification content of the business change log in the current 10 minutes. That is to say, before 00:00 on T+1 day, there will actually be multiple historical preset time periods of T day, namely 0:00~0:10, 0:10~0:20, 0:20~0:30, ..., 23:40~23:50, 23:50~24:00 on T day, and the data of multiple historical preset time periods will execute the data zipper process after the first current preset time period after 00:00.

[0069] For ease of understanding, we use Figure 7 Take this as an example to explain in detail:

[0070] exist Figure 7 In the example, the user made three modification operations at 23:50, 23:52 and 00:01 the next day, respectively, setting the preset time period to 1 minute and the preset time node to 00:00 every day.

[0071] For the current preset time period of 23:50-23:51, based on the business change log of 23:50, it is determined that the first minimum timestamp is 23:50. By comparing with the preset time node, it is found that the preset time node has not been reached. Therefore, the first change data corresponding to 23:50-23:51 (Product 1 Src Map[] Cru Map[20*30,60]) will be stored in the intermediate database.

[0072] For the current preset time period of 23:52-23:53, based on the business change log at 23:52, the first minimum timestamp is determined to be 23:52. By comparing it with the preset time node, it is found that the preset time node has not been reached. Therefore, the first change data corresponding to 23:52-23:53 (Product 1 Src Map[] Cru Map[50*30]) is stored in the intermediate database. It can be understood that at this time, 23:50-23:51 is already a historical preset time period that is earlier than the current preset time period.

[0073] For the current preset time period of 00:01-00:02, based on the business change log of 00:01, it is determined that the first minimum timestamp is 00:01. By comparing with the preset time node, it is found that the preset time node has been reached, which means that the data of the previous day has been fully stored in the business database, and data synchronization between the business database and the data warehouse can be performed. Then, at this time, a zipper operation will be performed on the second change data corresponding to the historical preset time period before 00:01-00:02 in the intermediate database, that is, the synchronization process of the business data of the previous day is completed. It can be understood that at this time, 23:50-23:51 and 23:52-23:53 are both historical preset time periods compared to 00:01-00:02.

[0074] In a further optional embodiment, in response to the first minimum timestamp reaching a preset time node, it is also necessary to store the first change data in the intermediate database, and synchronously delete the second change data corresponding to the historical preset time period in the intermediate database, that is, to ensure that the intermediate database always stores the change data that has not been synchronized.

[0075] It should be noted that in the actual business data storage process, data delays may occur due to certain anomalies, that is, after the user performs the relevant operation, the first change data is not obtained from the business database in time within the current preset time period of the current processing, but the first change data is obtained within the subsequent current preset time period. In view of this situation, the data synchronization method provided by the present invention can also smoothly synchronize data and avoid the problem of data zipping when the data is incomplete. Specifically, the same Figure 7 Taking the embodiment as an example, when the delayed first change data of 23:58 is received in the current preset time period of 00:01-00:02, then when calculating the first minimum timestamp, the first minimum timestamp obtained is 23:58, which does not reach the preset time node. Therefore, even in the current preset time period of 00:01-00:02, the data zipper technology will not be executed, and the previous historical preset time period will be zipped in the next current preset time period, such as 00:02-00:03.

[0076] After the above Figures 1 to 7, describes a process of how to synchronize data with second change data corresponding to a historical preset time period based on first change data corresponding to a current preset time period when the full amount of data is divided by preset time periods in an embodiment of the present invention. However, since there are usually multiple second change data in the actual process, unlike the prior art of data zipping by comparing metadata tables, the present invention provides a new data zipping method, which can realize an atomic zipping process according to the time corresponding to the second change data.

[0077] In an optional embodiment, in response to the situation where there are multiple historical preset time periods, the specific process of performing data zipping can be as follows: Figure 8 As shown, including:

[0078] Step S801: sorting multiple historical preset time periods by time;

[0079] Step S802: for each historical preset time period from earliest to latest, sequentially executing: updating the source code field and cursor field corresponding to the historical data according to the second change data corresponding to the historical preset time period;

[0080] Step S803: updating the latest status of the historical data according to the source code field, and updating the historical track of the historical data according to the cursor field.

[0081] It is understandable that the data zipper process can reflect the process of data changes, so data synchronization needs to be performed in sequence according to the time sequence of the historical preset time period. Among them, for the data table that needs data zippering, the secondary data partition dp partition, the start date start_date field and the end date end_date field are usually set. In the dp partition, two enumeration values ACTIVE and EXPIRED are further included, among which ACTIVE is used to store the latest status of the data, and EXPIRED is used to store the historical trajectory of the data. Through the secondary partitions of ACTIVE and EXPIRED, a data closed loop is achieved. For example, Figure 9 As shown, in Figure 9 In , sku_id represents the unique code of the product, for example, code 10000 represents a product of brand 01, and code 10001 represents a product of brand 02. Figure 9As can be seen, the product with code 10000 in brand 01 was named A from January 1, 2022, to December 31, 2022, changed to B from January 1, 2023, to December 31, 2023, and changed to C from January 1, 2024 to the present. "EXPIRED" indicates the historical status, and "ACTIVE" indicates the current status. Therefore, this embodiment of the present invention, through steps S801 to S803, updates the source code field and cursor field in chronological order according to each historical preset time period, achieving the latest status and historical track update, ultimately achieving the effect of a data zipper.

[0082] It should be noted that when synchronizing data with a data lake, incremental data is usually written separately to a .log file, while stock data is written to a parquet file. The merging of incremental data and stock data is done based on the table service Compaction and the read-time merge Mor. In the prior art, the update is done by overwriting the old data with the new data. In the embodiment of the present invention, Figure 7 As shown in the figure, by defining two MAPs, Src and Cur, to store the current state and the historical trajectory state respectively, data zipper can be realized by continuously updating Src MAP and Cur MAP. Compared with the existing technology, it does not require the process of data coverage and is more efficient and quick.

[0083] Since the embodiment of the present invention performs streaming data synchronization based on the business change log received in real time, it is particularly important to quickly and efficiently identify the location of the data to be changed in the historical data during the data synchronization process. In an optional embodiment, the embodiment of the present invention provides a data synchronization method such as Figure 10 As shown, specifically including:

[0084] Step S1001: obtaining a business change log in real time, and determining one or more first change data corresponding to the business change log;

[0085] Step S1002: determining the timestamp of each first change data, and determining a first minimum timestamp corresponding to the current preset time period based on each timestamp;

[0086] Step S1003: in response to the first minimum timestamp reaching a preset time node, obtaining second change data corresponding to a historical preset time period before the current preset time period in the intermediate database;

[0087] Step S1004: Locating the modification position of the second changed data in the historical data based on the index pre-built in the data warehouse;

[0088] Step S1005: performing a zipper operation on the second changed data and the historical data in the data warehouse according to the positioning result.

[0089] In order to quickly locate the modification location, the present invention optimizes the index in the data warehouse. Taking the data lake as an example, existing data lake frameworks typically come with a built-in bucket index, meaning multiple buckets are set up, each storing multiple data with the same hash value. By locating the buckets, the stored data is first located, and then located from the bucket based on the file name. In an embodiment of the present invention, the index includes two levels: the first-level index is the data warehouse's built-in bucket index; the second-level index is the step index corresponding to each bucket, indicating the multiple step code intervals in each bucket. The step index can be understood as setting a continuous code for each data item. For example, if the step code interval corresponding to each bucket is 10,000, then the codes stored in the first bucket are 00001-10001, the codes stored in the second bucket are 10001-20001, and so on, sequentially encoding multiple data items in multiple buckets.

[0090] In a further optional embodiment, the process of positioning based on the index is as follows Figure 11 As shown, including:

[0091] Step S1101: determining a target bucket corresponding to the second changed data from multiple buckets according to the first-level index;

[0092] Step S1102: Determine, based on the second-level index, a target step-length coding interval corresponding to the second changed data from a plurality of step-length coding intervals corresponding to the target storage bucket;

[0093] Step S1103: Determine the modification position from the target step interval according to the file name of the second changed data.

[0094] It is understandable that, for step S1103, the modification location is actually determined by comparing the similarity between the file name of the second change data and the file name of the data stored in the database. Through the above process, the target storage bucket corresponding to the second change data can be quickly located through the first-level index, and the target step encoding interval where the second change data is located can be further quickly located in the target storage bucket using the second-level index. Compared with the prior art that only sets the first-level index, the setting of the second-level index can further narrow the interval required for file name comparison. The speed of comparing the second-level index is much faster than the speed of comparing the file name. Therefore, the above process further improves the speed of locating the second change data.

[0095] It should be noted that encoding and sorting different data is very resource-intensive, and since the data lake itself is not part of the engine, it is impossible to set up and maintain the index dictionary used for encoding and sorting in memory. Therefore, in this embodiment of the present invention, step encoding is performed using a business auto-increment ID to achieve the sorting effect, and the step index code is associated with the data file. In addition, the first-level index needs to be maintained on the Step field, and the second-level index is preferably maintained on the partition field. Otherwise, a large number of sorting operations are required to achieve the indexing function.

[0096] According to the data synchronization method of an embodiment of the present invention, by obtaining the first change data corresponding to the business change log for the current preset time period, the full data zipping process can be broken down into the zipping processes of the change data within multiple time periods, reducing resource processing peaks and achieving streaming processing effects with the real-time acquisition of business change logs. In addition, by comparing the first minimum timestamp corresponding to the current preset time period with the preset time node, the data integrity of the second change data during the data zipping process is guaranteed, preventing zipping errors caused by data delays in abnormal scenarios.

[0097] Figure 12 FIG. 4 is a schematic diagram of main modules of a data synchronization apparatus according to an embodiment of the present invention.

[0098] like Figure 12 As shown, the data synchronization apparatus 1200 according to an embodiment of the present invention includes:

[0099] The acquisition module 1201 is configured to receive a service change log in real time and determine one or more first change data corresponding to the service change log;

[0100] A determination module 1202 is configured to determine a timestamp of each first change data item, and determine a first minimum timestamp corresponding to a current preset time period based on each of the timestamps;

[0101] Update module 1203 is used to store the first change data corresponding to the current preset time period in the intermediate database according to the timestamp in response to the first minimum timestamp not reaching the preset time node; in response to the first minimum timestamp reaching the preset time node, obtain the second change data corresponding to the historical preset time period before the current preset time period in the intermediate database, and perform a zipper operation on the second change data and the historical data in the data warehouse to synchronize the historical data using the second change data.

[0102] In an optional embodiment of the present invention, the acquisition module 1201 is further used to determine one or more target business data tables corresponding to the business change log from a business database; wherein the business database stores business data tables corresponding to different data sources respectively; and obtain the first change data corresponding to the business change log from each of the target business data tables.

[0103] In an optional embodiment of the present invention, the update module 1203 is further used to, after determining the first minimum timestamp corresponding to the current preset time period based on each of the timestamps, write the first minimum timestamp corresponding to the current preset time period into a metadata file of the data lake; wherein the metadata file stores second minimum timestamps corresponding to multiple historical time periods; monitor the metadata file in real time, and determine one or more target second minimum timestamps that are earlier than the first minimum timestamp from the second minimum timestamps; and use the historical time periods corresponding to one or more target second minimum timestamps as the historical preset time periods.

[0104] In an optional embodiment of the present invention, in response to the number of the historical preset time periods being multiple, the update module 1203 is further configured to sort the multiple historical preset time periods according to time; and for each of the historical preset time periods from earliest to latest, sequentially perform the following: updating the source code field and cursor field corresponding to the historical data according to the second change data corresponding to the historical preset time period; updating the latest status of the historical data according to the source code field, and updating the historical trajectory of the historical data according to the cursor field.

[0105] In an optional embodiment of the present invention, the update module 1203 is also used to locate the modification position of the second change data in the historical data based on the index pre-constructed in the data warehouse; and perform a zipper operation on the second change data and the historical data in the data warehouse according to the positioning result.

[0106] In an optional embodiment of the present invention, the index includes two levels, wherein the first-level index is the bucket index of the data warehouse; the second-level index is the step index corresponding to each bucket, indicating multiple step coding intervals in each bucket.

[0107] In an optional embodiment of the present invention, the update module 1203 is further used to determine, based on the first-level index, a target storage bucket corresponding to the second change data from multiple storage buckets; based on the second-level index, determine a target step coding interval corresponding to the second change data from multiple step coding intervals corresponding to the target storage bucket; and based on the file name of the second change data, determine the modification position from the target step interval.

[0108] According to an embodiment of the present invention, a data synchronization device, by acquiring the first change data corresponding to the business change log for the current preset time period, can break down the full data zipping process into the zipping processes of the change data within multiple time periods, thereby reducing resource processing peaks and achieving streaming processing effects with the real-time acquisition of business change logs. Furthermore, by comparing the first minimum timestamp corresponding to the current preset time period with the preset time node, the data integrity of the second change data during the data zipping process is ensured, preventing zipping errors caused by data delays in abnormal scenarios.

[0109] Figure 13 An exemplary system architecture 1300 is shown to which the data synchronization method or apparatus according to an embodiment of the present invention may be applied.

[0110] like Figure 13 As shown, system architecture 1300 may include terminal devices 1301, 1302, and 1303, a network 1304, and a server 1305. Network 1304 is used to provide a medium for communication links between terminal devices 1301, 1302, and 1303 and server 1305. Network 1304 may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0111] Users can use terminal devices 1301, 1302, and 1303 to interact with server 1305 via network 1304 to receive or send data, etc. Various communication client applications can be installed on terminal devices 1301, 1302, and 1303, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0112] The terminal devices 1301 , 1302 , and 1303 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers.

[0113] Server 1305 may be a server that provides various services, such as a background management server that provides support for business data modified by users using terminal devices 1301, 1302, and 1303. The background management server may analyze and process received data such as business change logs.

[0114] It should be noted that the data synchronization method provided in the embodiment of the present invention is generally executed by the server 1305 , and accordingly, the data synchronization device is generally set in the server 1305 .

[0115] It should be understood that Figure 13 The number of terminal devices, networks and servers in the embodiment is merely illustrative. Any number of terminal devices, networks and servers may be provided as required.

[0116] Reference below Figure 14 , which shows a schematic structural diagram of a computer system 1400 of a terminal device suitable for implementing an embodiment of the present invention. Figure 14 The terminal device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0117] like Figure 14 As shown, computer system 1400 includes a central processing unit (CPU) 1401, which can perform various appropriate actions and processes according to programs stored in read-only memory (ROM) 1402 or programs loaded from storage unit 1408 into random access memory (RAM) 1403. RAM 1403 also stores various programs and data required for the operation of system 1400. CPU 1401, ROM 1402, and RAM 1403 are connected to each other via bus 1404. A first input / output (I / O) interface 1405 is also connected to bus 1404.

[0118] The following components are connected to the first I / O interface 1405: an input section 1406 including a keyboard, mouse, and the like; an output section 1407 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and speakers; a storage section 1408 including a hard disk; and a communication section 1409 including a network interface card such as a LAN card or a modem. The communication section 1409 performs communication processing via a network such as the Internet. A drive 1410 is also connected to the first I / O interface 1405 as needed. Removable media 1411, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 1410 as needed, so that computer programs read from the media can be installed in the storage section 1408 as needed.

[0119] In particular, according to embodiments disclosed herein, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed herein include a computer program product comprising a computer program embodied on a computer-readable medium, the computer program containing program code for executing the methods illustrated in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 1409 and / or installed from removable media 1411. When executed by central processing unit (CPU) 1401, the computer program performs the aforementioned functions defined in the system of the present invention.

[0120] It should be noted that the computer-readable medium described in the present invention may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. Computer-readable storage media may include, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or components, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device, or component. In the present invention, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such a propagated data signal may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wireline, optical fiber cable, RF, or any suitable combination thereof.

[0121] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the above-mentioned module, program segment, or a part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0122] The modules involved in the embodiments of the present invention may be implemented in software or hardware. The modules described may also be provided in a processor. For example, they may be described as follows: a processor includes an acquisition module, a determination module, and an update module. The names of these modules do not, in some cases, constitute limitations on the modules themselves. For example, the acquisition module may also be described as a "module that receives a business change log in real time and determines one or more first change data corresponding to the business change log."

[0123] As another aspect, the present invention further provides a computer-readable medium, which may be included in the device described in the above embodiment; or may exist independently and not be assembled into the device. The computer-readable medium carries one or more programs, and when the one or more programs are executed by a device, the device includes: obtaining a business change log in real time and determining one or more first change data corresponding to the business change log; determining the timestamp of each first change data, and determining a first minimum timestamp corresponding to a current preset time period based on each of the timestamps; in response to the first minimum timestamp not reaching a preset time node, storing the first change data corresponding to the current preset time period in an intermediate database according to the timestamp; in response to the first minimum timestamp reaching a preset time node, obtaining second change data corresponding to a historical preset time period in the intermediate database that is located before the current preset time period, and performing a zipper operation on the second change data and the historical data in the data warehouse to synchronize the historical data using the second change data.

[0124] According to the technical solution of an embodiment of the present invention, by obtaining the first change data corresponding to the business change log for the current preset time period, the full data zipping process can be broken down into the zipping processes of the change data within multiple time periods, reducing resource processing peaks and achieving streaming processing effects with the real-time acquisition of business change logs. In addition, by comparing the first minimum timestamp corresponding to the current preset time period with the preset time node, the data integrity of the second change data during the data zipping process is guaranteed, preventing zipping errors caused by data delays in abnormal scenarios.

[0125] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A data synchronization method, characterized in that: include: Acquire a business change log in real time, and determine one or more first change data corresponding to the business change log; Determine a timestamp of each first change data, and determine a first minimum timestamp corresponding to a current preset time period based on each of the timestamps; In response to the first minimum timestamp not reaching a preset time node, storing the first change data corresponding to the current preset time period in an intermediate database according to the timestamp; In response to the first minimum timestamp reaching a preset time node, second change data corresponding to a historical preset time period before the current preset time period in the intermediate database is obtained, and a zipper operation is performed on the second change data and the historical data in the data warehouse to synchronize the historical data using the second change data.

2. The method according to claim 1, characterized in that The determining one or more first change data corresponding to the service change log includes: Determining one or more target business data tables corresponding to the business change log from a business database; wherein the business database stores business data tables corresponding to different data sources respectively; The first change data corresponding to the business change log is respectively obtained from each of the target business data tables.

3. The method according to claim 1, characterized in that After determining the first minimum timestamp corresponding to the current preset time period based on the timestamps, the method further includes: Writing the first minimum timestamp corresponding to the current preset time period into the metadata file of the data lake; wherein the metadata file stores the second minimum timestamps corresponding to multiple historical time periods respectively; monitoring the metadata file in real time, and determining one or more target second minimum timestamps earlier than the first minimum timestamp from the second minimum timestamp; The historical time periods corresponding to one or more of the target second minimum timestamps are used as the historical preset time periods.

4. The method according to claim 3, characterized in that In response to the number of the historical preset time periods being multiple, the performing of a zipper operation on the second change data and the historical data in the data warehouse includes: Sorting the plurality of historical preset time periods according to time; For each of the preset historical time periods from earliest to latest, sequentially executing: updating the source code field and the cursor field corresponding to the historical data according to the second change data corresponding to the preset historical time period; The latest state of the historical data is updated according to the source code field, and the historical track of the historical data is updated according to the cursor field.

5. The method according to claim 1, wherein The performing a zipper operation on the second changed data and the historical data in the data warehouse includes: Locating a modification position of the second change data in the historical data based on an index pre-constructed in the data warehouse; According to the positioning result, a zipper operation is performed on the second changed data and the historical data in the data warehouse.

6. The method according to claim 5, characterized in that The index consists of two levels, where The first-level index is the bucket index of the data warehouse itself; The second-level index is a step index corresponding to each storage bucket, indicating multiple step encoding intervals in each storage bucket.

7. The method according to claim 6, characterized in that The locating the modification position of the second change data in the historical data based on the index pre-constructed in the data warehouse includes: Determine, according to the first-level index, a target storage bucket corresponding to the second changed data from a plurality of storage buckets; Determine, according to the second-level index, a target step-length coding interval corresponding to the second change data from a plurality of step-length coding intervals corresponding to the target storage bucket; The modification position is determined from the target step interval according to the file name of the second change data.

8. A data synchronization device, characterized in that: include: An acquisition module, configured to receive a business change log in real time and determine one or more first change data corresponding to the business change log; a determining module, configured to determine a timestamp of each of the first change data, and determine a first minimum timestamp corresponding to a current preset time period based on each of the timestamps; an updating module, configured to store the first change data corresponding to the current preset time period in an intermediate database according to the timestamp in response to the first minimum timestamp not reaching a preset time node; In response to the first minimum timestamp reaching a preset time node, second change data corresponding to a historical preset time period before the current preset time period in the intermediate database is obtained, and a zipper operation is performed on the second change data and the historical data in the data warehouse to synchronize the historical data using the second change data.

9. An electronic device for data synchronization, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

10. A computer-readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

11. A computer program product comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the method according to any one of claims 1 to 7.

Images