Data synchronization method, apparatus, device, medium, and product
By using Merkle tree structure and hash value comparison to identify data changes, the problem of low interface data synchronization efficiency is solved, and a high-efficiency, real-time data synchronization process is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CHINA MOBILE (SUZHOU) SOFTWARE TECH CO LTD
- Filing Date
- 2025-12-02
- Publication Date
- 2026-08-04
AI Technical Summary
In existing technologies, the database synchronization efficiency is insufficient due to changes in data retrieved through interfaces, especially when new jobs are submitted or job status changes occur, requiring real-time updates to the status of submitted jobs and information on newly submitted jobs in the database.
The interface data is divided into blocks using a Merkle tree structure to generate first and second Merkle trees. Data changes are identified by comparing the hash values of the Merkle trees. Only the interface data that has changed is synchronized, without the need for a full comparison of all interface data.
It improves the processing efficiency and real-time performance of the data synchronization process, reduces system complexity, avoids the introduction of additional components, and achieves efficient data synchronization.
Smart Images
Figure CN121681675B_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of data processing technology, and more specifically, to a data synchronization method, apparatus, device, medium, and product. Background Technology
[0002] Since the data retrieved through the API changes as jobs are continuously submitted to the big data component, primarily in two ways—newly submitted job data and changes in the status of already submitted jobs—the database needs to be updated with the status of already submitted jobs and information on newly submitted jobs inserted to meet real-time requirements.
[0003] In related technologies, all interface data is usually stored in a temporary database first, and the binary log of the temporary database is parsed. The interface data is synchronized through the binary log. This will increase the number of databases and lead to insufficient processing efficiency in the data synchronization process. Summary of the Invention
[0004] This disclosure is made in view of the insufficient processing efficiency of the aforementioned data synchronization process. This disclosure provides a data synchronization method, apparatus, device, medium, and product.
[0005] In a first aspect, embodiments of this disclosure provide a data synchronization method, comprising: Obtain the first interface data and the second interface data of the target interface; wherein the second interface data is obtained after the first interface data.
[0006] A first Merkle tree is generated based on the target time interval and the first interface data, and a second Merkle tree is generated based on the time range covered by the first interface data and the second interface data; wherein, the target time interval is the time interval for generating the data block corresponding to the first Merkle tree, and the start time for generating the first Merkle tree and the second Merkle tree is the start timestamp of the first interface data.
[0007] Based on the first Merkle tree and the second Merkle tree, identify the target interface data that has changed in the second interface data.
[0008] Synchronize the target interface data to the database.
[0009] In one embodiment of the first aspect, generating a first Merkle tree based on a target time interval and first interface data, and generating a second Merkle tree based on the time range covered by the first interface data and second interface data, includes: Based on the target time interval, the first interface data is divided into multiple first data blocks.
[0010] The first Merkle tree is generated based on multiple first data blocks.
[0011] If the target time interval remains unchanged, the second interface data is divided into multiple second data blocks according to the target time interval, from the start timestamp of the first interface data to the end timestamp of the first interface data.
[0012] A second Merkle tree is generated based on multiple second data blocks.
[0013] In one embodiment of the first aspect, it further includes: If the target time interval changes, a new first Merkle tree is generated based on the changed target time interval and the second interface data.
[0014] In the next data synchronization cycle, a new second Merkle tree is generated based on the start timestamp of the second interface data, the changed target time interval, and the third interface data; wherein, the third interface data is obtained after the second interface data.
[0015] In one embodiment of the first aspect, identifying the target interface data that has changed in the second interface data based on a first Merkle tree and a second Merkle tree includes: Starting from the root node, compare the hash values of each node in the first Merkle tree and the second Merkle tree layer by layer.
[0016] The nodes with different hash values in the first Merkle tree and the second Merkle tree are identified to obtain the identification results; the identification results are used to indicate whether the nodes with different hash values are leaf nodes in the first Merkle tree and the second Merkle tree.
[0017] The target interface data is determined based on the identification results.
[0018] In one embodiment of the first aspect, determining target interface data based on the identification result includes: If nodes with different hash values are identified as leaf nodes based on the identification results, the target interface data is determined based on the data block corresponding to that node.
[0019] If, based on the identification results, it is determined that a node with a different hash value is not a leaf node, the first Merkle tree and the second Merkle tree are traversed downwards along the node path corresponding to that node until a leaf node is located.
[0020] The target interface data is determined based on the data block corresponding to the leaf node.
[0021] In one embodiment of the first aspect, after synchronizing the target interface data to the database, the method further includes: From the start timestamp of the first interface data to the end timestamp of the second interface data, the second interface data is divided into multiple third data blocks according to the target time interval; A third Merkle tree is generated based on multiple third data blocks.
[0022] The information table in the database is updated based on the third Merkle tree and the time information corresponding to the third Merkle tree; the information table stores the first Merkle tree and the time information corresponding to the first Merkle tree.
[0023] In a second aspect, embodiments of this disclosure provide a data synchronization apparatus, comprising: The data acquisition unit is used to acquire the first interface data and the second interface data of the target interface; wherein the second interface data is acquired after the first interface data.
[0024] The generation unit is used to generate a first Merkle tree based on a target time interval and first interface data, and to generate a second Merkle tree based on the time range covered by the first interface data and second interface data; wherein, the target time interval is the time interval for generating the data block corresponding to the first Merkle tree, and the start time for generating the first Merkle tree and the second Merkle tree is the start timestamp of the first interface data.
[0025] The data identification unit is used to identify the target interface data that has changed in the second interface data based on the first Merkle tree and the second Merkle tree.
[0026] The data synchronization unit is used to synchronize target interface data to the database.
[0027] In a third aspect, embodiments of this disclosure provide an electronic device, including: a processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the memory via the bus, and when the machine-readable instructions are executed by the processor, the steps in the above-described data synchronization method are performed.
[0028] In a fourth aspect, embodiments of this disclosure provide a computer-readable storage medium storing a computer program that, when executed by a processor, performs the steps of the data synchronization method described above.
[0029] In a fifth aspect, embodiments of this disclosure provide a computer program product stored in a storage medium, the program product being executed by at least one processor to implement the steps in the data synchronization method described above.
[0030] As will be described in detail below, a data synchronization method, apparatus, device, medium, and product according to embodiments of this disclosure are disclosed. In embodiments of this disclosure, first interface data of a target interface is acquired, and then second interface data of the target interface is acquired at a time interval thereafter. Then, a first Merkle tree is generated based on the target time interval and the first interface data, and a second Merkle tree is generated based on the time range covered by the first interface data and the second interface data. The target time interval is the time interval for generating the corresponding data block of the first Merkle tree, and the start time for generating the first and second Merkle trees is the start timestamp of the first interface data. Next, based on the first and second Merkle trees, changed target interface data in the second interface data is identified. Finally, the target interface data is synchronized to a database. By using the above processing method, the interface data is represented in the form of a Merkle tree. By comparing the Merkle trees corresponding to the first interface data and the second interface data, and comparing the corresponding hash values in the Merkle trees, the changed interface data can be accurately located without having to perform a full comparison of the complete first and second interface data. This can effectively improve the processing efficiency and real-time performance of the data synchronization process, and without introducing additional components, thereby effectively reducing the complexity of the system.
[0031] It should be understood that both the foregoing general description and the following detailed description are exemplary and intended to provide further illustration of the claimed technology. Attached Figure Description
[0032] The above and other objects, features, and advantages of this disclosure will become more apparent from the more detailed description of the embodiments thereof in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this disclosure and form part of the specification. They are used together with the embodiments of this disclosure to explain the disclosure and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.
[0033] Figure 1 This is a flowchart of a data synchronization method according to an embodiment of the present disclosure.
[0034] Figure 2 This is a schematic diagram of a Merkle tree according to an embodiment of this disclosure.
[0035] Figure 3 This is an overall flowchart of a data synchronization process according to an embodiment of the present disclosure.
[0036] Figure 4 This is a schematic diagram of a data synchronization device according to an embodiment of the present disclosure.
[0037] Figure 5 This is a schematic diagram of an electronic device according to an embodiment of the present disclosure.
[0038] Figure 6 This is a schematic diagram of a computer program product according to an embodiment of the present disclosure. Detailed Implementation
[0039] To make the objectives, technical solutions, and advantages of this disclosure more apparent, exemplary embodiments according to this disclosure will now be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this disclosure, and not all embodiments of this disclosure. It should be understood that this disclosure is not limited to the exemplary embodiments described herein.
[0040] To facilitate understanding of this embodiment, a data synchronization method disclosed in this disclosure will first be described in detail, see [link to relevant documentation]. Figure 1 The diagram shows a flowchart of a data synchronization method provided in an embodiment of this disclosure. This method can be applied to a service module of a cloud platform. The method includes steps S101-S104, wherein: Step S101: Obtain the first interface data and the second interface data of the target interface; wherein the second interface data is obtained after the first interface data.
[0041] Here, the target interface needs to have two characteristics: First, the field values representing the state in the target interface should be updated over time, similar to field value updates in a relational database. Second, the target interface needs to have a field representing time (e.g., timestamp or date), and once this field value is generated, it should not change over time.
[0042] The data obtained from the target interface will mainly change in two ways: one is the addition of new records, and the other is the updating of field values in existing records.
[0043] For example, you can understand the interface characteristics by using the API interface for obtaining job information in Yarn's Resource Manager.
[0044] Yarn's Resource Manager interface for retrieving job information typically includes a `startedTime` field, representing the job's submission time. Since the submission time is fixed and unique, this field's value remains unchanged once generated. The fields that do change are primarily `state`, `finalStatus`, and `finishedTime`. Job submissions generally go through states such as NEW, NEW_SAVING, SUBMITTED, ACCEPTED, RUNNING, FINISHED, FAILED, and KILLED, which identify the submitted job's status. Over time, the job status typically changes. Therefore, this interface suffers from the data synchronization issue mentioned in this disclosure.
[0045] In this regard, the present disclosure can first obtain the first interface data from the target interface (e.g., a Restful API interface), and then obtain the second interface data from the target interface again at a specified time interval. At this time, the second interface data will have two changes, namely, the fields of newly added records and synchronized records will change.
[0046] Step S102: Generate a first Merkle tree based on the target time interval and the first interface data, and generate a second Merkle tree based on the time range covered by the first interface data and the second interface data; wherein, the target time interval is the time interval for generating the data block corresponding to the first Merkle tree, and the start time for generating the first Merkle tree and the second Merkle tree is the start timestamp of the first interface data.
[0047] This disclosure uses the data structure used in blockchain, namely Merkle tree. Merkle tree is used to divide interface data into small data blocks, calculate the hash value of each data block, then merge two adjacent hash values into a string, and then perform a hash operation on the string. In this way, we can push upwards to get a root hash, forming an inverted tree. If the total number of hashes at the bottom layer is odd, we can copy a hash to calculate the root hash value.
[0048] This disclosure uses the interface for obtaining job information in Yarn's Resource Manager as an example to exemplify the process of generating Merkle trees.
[0049] Specifically, first determine the time interval T1 (i.e., the target time interval) for generating the corresponding data block of the Merkle tree. This time interval T1 can be selected as 1 hour, and then stored in the code configuration file or the Info table.
[0050] Specifically, the data blocks on the leaf nodes of the Merkle tree require clearly defined start and end times for the interface data. That is, starting from the timestamp of the first record, records within a time range of T1 are counted as a data block. After setting, this can be modified according to the peak and trough values of business needs. The specific size of the T1 time block can be set according to the data synchronization time required by the business scenario and the characteristics of the interface in the business scenario.
[0051] If the business scenario is relatively stable, T1 can be set to a fixed value; if the interface is frequently triggered in a certain period of time, and only a few times triggered in another period of time, the value of T1 can be modified.
[0052] For example, in the job information retrieval interface of Yarn's Resource Manager, you can set T1 to 1 hour, assuming the first job submission time is 12:00, and the data block time range is 12:00-13:00. That is, all records submitted between 12:00 and 13:00 are stored in this data block. If the business scenario does not involve frequent job submissions to Yarn, and the job execution time is long, T1 can be set to 5 hours or longer, depending on the specific business scenario.
[0053] The Info table is a table that stores information about Merkle trees. The table must store at least one Merkle tree and the timestamp t1 of the first record and the timestamp t2 of the last record in the record that generated another Merkle tree.
[0054] Then, the target interface is triggered to obtain the interface data. Assume that the target interface returns 3 records between 9:00-10:00, denoted as R1, R2, and R3, and 2 records between 10:00-11:00, denoted as R4 and R5. Here, the time period returned by the interface is determined by reading T1 from the Info table. If T1 is 1 hour, then it should be the time period of 9:00-10:00, and so on.
[0055] Secondly, the code creates JavaBeans for the interface fields to receive the records returned by the interface, and stores the received JavaBeans in a List. R1, R2, and R3 are put into list1, and R4 and R5 are put into list2.
[0056] Based on this, the hash values of list1 and list2 are calculated, and these two hash values are used as leaf nodes V1 and V2 of the Merkle tree. The hash values of V1 and V2 are added together, and the result is hashed again to obtain V3. Finally, V1 and V2 are used as leaf nodes, and V3 is used as the root node to construct a complete binary tree. At this point, a Merkle tree with interface data is constructed.
[0057] In this embodiment of the disclosure, the above method can be used to generate a first Merkle tree based on the target time interval and the first interface data, and to generate a second Merkle tree based on the time range covered by the first interface data and the second interface data.
[0058] Step S103: Based on the first Merkle tree and the second Merkle tree, identify the target interface data that has changed in the second interface data.
[0059] As mentioned earlier, a Merkle tree contains multiple hash values. Therefore, when data changes, the hash value change propagates from bottom to top, causing the root hash to change. Thus, if a discrepancy is found between the root hashes of the first and second Merkle trees, it indicates that data has changed. In this case, a top-down search of the Merkle tree can be performed to find the data block where the data has changed, thus identifying the changed interface data—that is, the target interface data that has changed within the second interface data.
[0060] In the above implementation, by generating a Merkle tree and comparing the hash values on the Merkle tree nodes, the complexity of querying changed data can be effectively reduced. That is, the process is transformed from a full table scan to locating data through hash values, which facilitates the improvement of the processing efficiency of the data synchronization process.
[0061] Step S104: Synchronize the target interface data to the database.
[0062] Here, the database can be a database such as MySQL or Redis, or the target interface data can be synchronized to other carriers. This disclosure does not limit this.
[0063] In this embodiment, firstly, first interface data of the target interface is acquired, and then second interface data of the target interface is acquired at a time interval. Next, a first Merkle tree is generated based on the target time interval and the first interface data, and a second Merkle tree is generated based on the time range covered by the first interface data and the second interface data. The target time interval is the time interval for generating the corresponding data block of the first Merkle tree, and the start time for generating the first and second Merkle trees is the start timestamp of the first interface data. Then, based on the first and second Merkle trees, the target interface data that has changed in the second interface data is identified. Finally, the target interface data is synchronized to the database.
[0064] In the above implementation, the interface data is represented in the form of a Merkle tree. By comparing the Merkle trees corresponding to the first interface data and the second interface data, the corresponding hash values in the Merkle trees are compared to accurately locate the changed interface data. This eliminates the need to perform a full comparison of the complete first and second interface data, thereby effectively improving the processing efficiency and real-time performance of the data synchronization process. Furthermore, it eliminates the need to introduce additional components, thus effectively reducing the complexity of the system.
[0065] In an optional implementation, the above steps generate a first Merkle tree based on the target time interval and the first interface data, and generate a second Merkle tree based on the time range covered by the first interface data and the second interface data, specifically including the following steps: According to the target time interval, the first interface data is divided into multiple first data blocks; The first Merkle tree is generated based on the plurality of first data blocks; If the target time interval is determined to remain unchanged, the second interface data is divided into multiple second data blocks according to the target time interval, from the start timestamp of the first interface data to the end timestamp of the first interface data. The second Merkle tree is generated based on the plurality of second data blocks.
[0066] After obtaining the first interface data from the target interface, this disclosure allows the first interface data to be directly stored in a database. The first interface data is then divided into multiple first data blocks according to the target time interval using the method mentioned above, and a first Merkle tree is generated based on the multiple first data blocks.
[0067] After this, the first Merkle tree and the corresponding time information can be stored in the Info table. This time information includes the start and end timestamps of the first interface data, that is, the timestamps of the first and last records in the first interface data.
[0068] If the target time interval remains unchanged, the second interface data is divided into multiple second data blocks based on the time information corresponding to the first Merkle tree stored in the Info table and the target time interval, and a second Merkle tree is generated based on the multiple second data blocks.
[0069] In the above implementation, by using the same time interval and start time, the structures of the first Merkle tree and the second Merkle tree can be kept consistent, facilitating the comparison of the hash values of the two Merkle trees. Furthermore, since the time range of the data block pair is much shorter than the time range corresponding to the entire interface data, the amount of data that needs to be processed can be effectively reduced, thus improving data synchronization efficiency.
[0070] In an optional implementation, the following steps are also included: If it is determined that the target time interval has changed, a new first Merkle tree is generated based on the changed target time interval and the second interface data; In the next data synchronization cycle, a new second Merkle tree is generated based on the start timestamp of the second interface data, the changed target time interval, and the third interface data; wherein the third interface data is obtained after the second interface data.
[0071] In this embodiment of the disclosure, if the target time interval changes, a new first Merkle tree needs to be regenerated based on the changed target time interval and the second interface data. Since the target time interval has changed, a new second Merkle tree cannot be generated in the current period.
[0072] In response, this disclosure, in the next data synchronization cycle, after acquiring the second interface data, acquires the third interface data from the target interface, and divides the third interface data into multiple data blocks according to the changed target time interval from the start timestamp of the second interface data to the end timestamp of the second interface data, and generates a new second Merkle tree based on the multiple data blocks.
[0073] In the above implementation, the granularity of data synchronization, i.e. the target time interval, can be dynamically adjusted according to the fluctuation of actual business volume. If the target time interval changes, the start timestamp of the first interface data is updated to the start timestamp of the second interface data, ensuring that the Merkle tree is generated on a completely new benchmark starting from the next synchronization cycle, thereby improving the flexibility of the data synchronization process.
[0074] In an optional implementation, the above steps, based on the first Merkle tree and the second Merkle tree, identify the target interface data that has changed in the second interface data, specifically including the following steps: Starting from the root node, compare the hash values of each node in the first Merkle tree and the second Merkle tree layer by layer; The nodes with different hash values in the first Merkle tree and the second Merkle tree are identified to obtain identification results; wherein, the identification results are used to indicate whether the nodes with different hash values are leaf nodes in the first Merkle tree and the second Merkle tree; The target interface data is determined based on the identification results.
[0075] like Figure 2 As shown, if a Merkle tree has four levels, with the first level being the root node, the second and third levels being intermediate nodes, and the fourth level being leaf nodes, then the hash values of the root nodes in the first and second Merkle trees, i.e., V0, are compared first.
[0076] If V0 is different, then compare the hash values of the second-level intermediate nodes, namely V1 and V2; if V2 is different, then continue to compare the hash values of the third-level intermediate nodes under V2, namely V5 and V6; if V5 is different, then continue to compare the hash values of the fourth-level leaf nodes under V5, namely V11 and V12; if V12 is different, it means that the record information corresponding to the data block of V12 has changed.
[0077] In this embodiment of the disclosure, it is first determined whether the nodes with different hash values in the first Merkle tree and the second Merkle tree are leaf nodes, i.e., the identification results. Based on the identification results, the leaf nodes corresponding to the nodes with different hash values are determined, so as to determine the target interface data that has changed in the second interface data based on the data block corresponding to the leaf node.
[0078] In the above implementation, by comparing hashes layer by layer, a large number of unchanged nodes can be quickly eliminated, and half of the data can be eliminated each time the layer is traversed down. This can effectively reduce the search time for target interface data and improve data processing efficiency.
[0079] In an optional implementation, the above steps, based on the identification result, determine the target interface data, specifically including the following steps: If, based on the identification result, the node with the different hash value is determined to be the leaf node, the target interface data is determined based on the data block corresponding to that node; If, based on the identification result, it is determined that the node with the different hash value is not the leaf node, the first Merkle tree and the second Merkle tree are traversed downwards along the node path corresponding to that node until the leaf node is located; The target interface data is determined based on the data block corresponding to the leaf node.
[0080] As mentioned earlier, when data changes, the change in hash value is propagated from bottom to top. If the data changes without being merged with changes in other branches, nodes with different hash values will directly become leaf nodes. In this case, the target interface data can be determined directly based on the data block corresponding to that node.
[0081] Correspondingly, if a node with a different hash value is not a leaf node, then a downward traversal is automatically triggered along the node path corresponding to that node until the leaf node corresponding to that node is located. For the specific traversal method, please refer to the previous section on... Figure 2 An exemplary description.
[0082] To address this, the target interface data can be updated to the corresponding location in the database based on the start and end times of the data block corresponding to the leaf node, thereby updating the field values corresponding to the target interface data to the field values originally stored in the database for the first interface data.
[0083] In the above implementation, it is ensured that the corresponding leaf node can be found regardless of which node the hash value is different, and the data block where the data has changed can be found with the shortest path. Based on the data block, the changed interface data, i.e. the target interface data, is determined, thereby optimizing the data search process.
[0084] In an optional implementation, after synchronizing the target interface data to the database, the method further includes the following steps: From the start timestamp of the first interface data to the end timestamp of the second interface data, the second interface data is divided into multiple third data blocks according to the target time interval; A third Merkle tree is generated based on the multiple third data blocks; The information table in the database is updated based on the third Merkle tree and the time information corresponding to the third Merkle tree; wherein, the information table stores the first Merkle tree and the time information corresponding to the first Merkle tree.
[0085] In this embodiment of the disclosure, after synchronizing the target interface data to the database, the second interface data can be divided into multiple third data blocks according to the target time interval, from the start timestamp of the first interface data to the end timestamp of the second interface data, and a third Merkle tree can be generated based on the multiple third data blocks.
[0086] The third Merkle tree is generated using the complete second interface data, whereas the second Merkle tree, generated earlier, used only partial second interface data to align with the first Merkle tree. Furthermore, the purpose of generating the third Merkle tree is to use it as a baseline tree, with the third Merkle tree serving as the object of comparison.
[0087] Based on this, the original first Merkle tree and its corresponding time information stored in the Info table (i.e., the information table) can be updated to the third Merkle tree and its corresponding time information. The time information corresponding to the third Merkle tree includes the start and end times of generating the third Merkle tree, i.e., the start and end timestamps of the second interface data.
[0088] In the above implementation, a baseline Merkle tree update mechanism is proposed, which determines the baseline tree based on the latest data state after each data synchronization, ensuring the effectiveness and accuracy of incremental synchronization.
[0089] The following is combined with Figure 3 The above data synchronization process is described in detail below: S301: Obtain the first interface data and the second interface data of the target interface.
[0090] Here, the data from the second interface is obtained after the data from the first interface.
[0091] S302: Divide the first interface data into multiple first data blocks according to the target time interval.
[0092] S303: Generate the first Merkle tree based on multiple first data blocks.
[0093] S304: If the target time interval is determined to remain unchanged, the second interface data is divided into multiple second data blocks according to the target time interval, from the start timestamp of the first interface data to the end timestamp of the first interface data.
[0094] S305: Generate a second Merkle tree based on multiple second data blocks.
[0095] S306: Starting from the root node, compare the hash values of each node in the first Merkle tree and the second Merkle tree layer by layer.
[0096] S307: Identify nodes with different hash values in the first Merkle tree and the second Merkle tree, and obtain the identification results.
[0097] Here, the identification results are used to indicate whether nodes with different hash values are leaf nodes in the first Merkle tree and the second Merkle tree.
[0098] S308: Based on the identification results, determine the target interface data that has changed in the second interface data.
[0099] S309: Synchronize the target interface data to the database.
[0100] As can be seen from the above description, the technical solution disclosed herein has the following advantages: (1) This disclosure generates a Merkle tree and compares the hash values on the Merkle tree nodes, thereby reducing the time complexity of querying changed data from O(N) of a full table scan to O(logN) during the data change process, thus reducing the time complexity of the data query process. Here, N can be understood as the total number of interface data in the database, and logN can be understood as the level or depth of the Merkle tree.
[0101] (2) This disclosure further narrows the time range of data synchronization and only synchronizes data when it is needed, which can perform data synchronization operations more accurately and will not generate additional query and update overhead.
[0102] (3) This disclosure does not introduce any other middleware except for storing an Info table, which reduces the complexity of the system.
[0103] Based on the same inventive concept, this disclosure also provides a data synchronization device corresponding to the data synchronization method. Since the principle of the device in this disclosure for solving the problem is similar to the data synchronization method described above in this disclosure, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.
[0104] Reference Figure 4The diagram shown is a schematic representation of a data synchronization device provided in an embodiment of this disclosure. The device includes: a data acquisition unit 40, a generation unit 41, a data identification unit 42, and a data synchronization unit 43; wherein: A data acquisition unit is used to acquire first interface data and second interface data of a target interface; wherein the second interface data is acquired after the first interface data. The generation unit is configured to generate a first Merkle tree based on a target time interval and the first interface data, and to generate a second Merkle tree based on the time range covered by the first interface data and the second interface data; wherein, the target time interval is the time interval for generating the data block corresponding to the first Merkle tree, and the start time for generating the first Merkle tree and the second Merkle tree is the start timestamp of the first interface data; The data identification unit is used to identify the target interface data that has changed in the second interface data based on the first Merkle tree and the second Merkle tree; The data synchronization unit is used to synchronize the target interface data to the database.
[0105] In one possible implementation, the device is also used for: According to the target time interval, the first interface data is divided into multiple first data blocks; The first Merkle tree is generated based on the plurality of first data blocks; If the target time interval is determined to remain unchanged, the second interface data is divided into multiple second data blocks according to the target time interval, from the start timestamp of the first interface data to the end timestamp of the first interface data. The second Merkle tree is generated based on the plurality of second data blocks.
[0106] In one possible implementation, the device is also used for: If it is determined that the target time interval has changed, a new first Merkle tree is generated based on the changed target time interval and the second interface data; In the next data synchronization cycle, a new second Merkle tree is generated based on the start timestamp of the second interface data, the changed target time interval, and the third interface data; wherein the third interface data is obtained after the second interface data.
[0107] In one possible implementation, the device is also used for: Starting from the root node, compare the hash values of each node in the first Merkle tree and the second Merkle tree layer by layer; The nodes with different hash values in the first Merkle tree and the second Merkle tree are identified to obtain identification results; wherein, the identification results are used to indicate whether the nodes with different hash values are leaf nodes in the first Merkle tree and the second Merkle tree; The target interface data is determined based on the identification results.
[0108] In one possible implementation, the device is also used for: If, based on the identification result, the node with the different hash value is determined to be the leaf node, the target interface data is determined based on the data block corresponding to that node; If, based on the identification result, it is determined that the node with the different hash value is not the leaf node, the first Merkle tree and the second Merkle tree are traversed downwards along the node path corresponding to that node until the leaf node is located; The target interface data is determined based on the data block corresponding to the leaf node.
[0109] In one possible implementation, the device is also used for: From the start timestamp of the first interface data to the end timestamp of the second interface data, the second interface data is divided into multiple third data blocks according to the target time interval; A third Merkle tree is generated based on the multiple third data blocks; The information table in the database is updated based on the third Merkle tree and the time information corresponding to the third Merkle tree; wherein, the information table stores the first Merkle tree and the time information corresponding to the first Merkle tree.
[0110] The processing flow of each module in the device and the interaction flow between each module can be referred to the relevant descriptions in the above method embodiments, and will not be detailed here.
[0111] Corresponding to Figure 1 In addition to the data synchronization method in the present disclosure, embodiments of this electronic device 50 are also provided. Figure 5 The diagram shown is a schematic representation of an electronic device 50 provided in an embodiment of this disclosure, including: The system includes a processor 51, a memory 52, and a bus 53. The memory 52 stores execution instructions and includes main memory 521 and external memory 522. The main memory 521, also called internal memory, temporarily stores the computational data in the processor 51, as well as data exchanged with external memory such as a hard disk. The processor 51 exchanges data with the external memory 522 through the main memory 521. When the electronic device 50 is running, the processor 51 communicates with the memory 52 through the bus 53, causing the processor 51 to execute the following instructions: Obtain the first interface data and the second interface data of the target interface; wherein the second interface data is obtained after the first interface data; A first Merkle tree is generated based on a target time interval and the first interface data, and a second Merkle tree is generated based on the time range covered by the first interface data and the second interface data; wherein, the target time interval is the time interval for generating the data block corresponding to the first Merkle tree, and the start time for generating the first Merkle tree and the second Merkle tree is the start timestamp of the first interface data; Based on the first Merkle tree and the second Merkle tree, identify the target interface data that has changed in the second interface data; Synchronize the target interface data to the database.
[0112] This disclosure also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the data synchronization method described in the above-described method embodiments. The storage medium may be a volatile or non-volatile computer-readable storage medium.
[0113] This disclosure also provides a computer program product 60, such as... Figure 6 The diagram shown is a schematic diagram of the structure of a computer program product 60 provided in an embodiment of this disclosure. The computer program product 60 carries a computer program 61. The program included in the computer program 61 can be used to execute the steps of the data synchronization method described in the above method embodiments. For details, please refer to the above method embodiments, which will not be repeated here.
[0114] The data synchronization method, apparatus, device, medium, and product according to embodiments of the present disclosure have been described above with reference to the accompanying drawings. In the embodiments of the present disclosure, first interface data of a target interface is acquired, and then second interface data of the target interface is acquired at a time interval thereafter. Then, a first Merkle tree is generated based on the target time interval and the first interface data, and a second Merkle tree is generated based on the time range covered by the first interface data and the second interface data. The target time interval is the time interval for generating the corresponding data block of the first Merkle tree, and the start time for generating the first Merkle tree and the second Merkle tree is the start timestamp of the first interface data. Next, based on the first Merkle tree and the second Merkle tree, the target interface data that has changed in the second interface data is identified. Finally, the target interface data is synchronized to the database. By using the above processing method, the interface data is represented in the form of a Merkle tree. By comparing the Merkle trees corresponding to the first interface data and the second interface data, and comparing the corresponding hash values in the Merkle trees, the changed interface data can be accurately located without having to perform a full comparison of the complete first and second interface data. This can effectively improve the processing efficiency and real-time performance of the data synchronization process, and without introducing additional components, thereby effectively reducing the complexity of the system.
[0115] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.
[0116] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.
[0117] Additionally, as used herein, the "or" used in a list of items beginning with "at least one" indicates a separate list, such that a list of, for example, "at least one of A, B, or C" means A or B or C, or AB or AC or BC, or ABC (i.e., A and B and C). Furthermore, the word "exemplary" does not imply that the described example is preferred or better than other examples.
[0118] It should also be noted that in the systems and methods of this disclosure, the components or steps can be decomposed and / or recombined. These decompositions and / or recombinations should be considered as equivalent solutions to this disclosure.
[0119] Various changes, substitutions, and modifications can be made to the technology described herein without departing from the teachings defined by the appended claims. Furthermore, the scope of the claims of this disclosure is not limited to the specific aspects of the processes, machines, manufactures, events, means, methods, and actions described above. Currently existing or later-developed processes, machines, manufactures, events, means, methods, or actions that perform substantially the same function or achieve substantially the same result as the corresponding aspects described herein can be utilized. Therefore, the appended claims include such processes, machines, manufactures, events, means, methods, or actions within their scope.
[0120] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.
[0121] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.
Claims
1. A method of data synchronization, the method comprising: include: Obtain the first interface data and the second interface data of the target interface; wherein the second interface data is obtained after the first interface data; A first Merkle tree is generated based on a target time interval and the first interface data, and a second Merkle tree is generated based on the time range covered by the first interface data and the second interface data; wherein, the target time interval is the time interval for generating the data block corresponding to the first Merkle tree, and the start time for generating the first Merkle tree and the second Merkle tree is the start timestamp of the first interface data; Based on the first Merkle tree and the second Merkle tree, identify the target interface data that has changed in the second interface data; Synchronize the target interface data to the database; The step of generating a first Merkle tree based on a target time interval and the first interface data, and generating a second Merkle tree based on the time range covered by the first interface data and the second interface data, includes: According to the target time interval, the first interface data is divided into multiple first data blocks; The first Merkle tree is generated based on the plurality of first data blocks; If the target time interval is determined to remain unchanged, the second interface data is divided into multiple second data blocks according to the target time interval, from the start timestamp of the first interface data to the end timestamp of the first interface data. The second Merkle tree is generated based on the plurality of second data blocks; The step of identifying the target interface data that has changed in the second interface data based on the first Merkle tree and the second Merkle tree includes: Starting from the root node, compare the hash values of each node in the first Merkle tree and the second Merkle tree layer by layer; The nodes with different hash values in the first Merkle tree and the second Merkle tree are identified to obtain identification results; wherein, the identification results are used to indicate whether the nodes with different hash values are leaf nodes in the first Merkle tree and the second Merkle tree; The target interface data is determined based on the identification results; The step of determining the target interface data based on the identification result includes: If, based on the identification result, the node with the different hash value is determined to be the leaf node, the target interface data is determined based on the data block corresponding to that node; If, based on the identification result, it is determined that the node with the different hash value is not the leaf node, the first Merkle tree and the second Merkle tree are traversed downwards along the node path corresponding to that node until the leaf node is located; The target interface data is determined based on the data block corresponding to the leaf node.
2. The method of claim 1, wherein, The method further includes: If it is determined that the target time interval has changed, a new first Merkle tree is generated based on the changed target time interval and the second interface data; In the next data synchronization cycle, a new second Merkle tree is generated based on the start timestamp of the second interface data, the changed target time interval, and the third interface data; wherein the third interface data is obtained after the second interface data.
3. The method of claim 1, wherein, After synchronizing the target interface data to the database, the method further includes: From the start timestamp of the first interface data to the end timestamp of the second interface data, the second interface data is divided into multiple third data blocks according to the target time interval; A third Merkle tree is generated based on the multiple third data blocks; The information table in the database is updated based on the third Merkle tree and the time information corresponding to the third Merkle tree; wherein, the information table stores the first Merkle tree and the time information corresponding to the first Merkle tree.
4. A data synchronization device, characterized in that, include: A data acquisition unit is used to acquire first interface data and second interface data of a target interface; wherein the second interface data is acquired after the first interface data. The generation unit is configured to generate a first Merkle tree based on a target time interval and the first interface data, and to generate a second Merkle tree based on the time range covered by the first interface data and the second interface data; wherein, the target time interval is the time interval for generating the data block corresponding to the first Merkle tree, and the start time for generating the first Merkle tree and the second Merkle tree is the start timestamp of the first interface data; The data identification unit is used to identify the target interface data that has changed in the second interface data based on the first Merkle tree and the second Merkle tree; A data synchronization unit is used to synchronize the target interface data to the database; The step of generating a first Merkle tree based on a target time interval and the first interface data, and generating a second Merkle tree based on the time range covered by the first interface data and the second interface data, includes: According to the target time interval, the first interface data is divided into multiple first data blocks; The first Merkle tree is generated based on the plurality of first data blocks; If the target time interval is determined to remain unchanged, the second interface data is divided into multiple second data blocks according to the target time interval, from the start timestamp of the first interface data to the end timestamp of the first interface data. The second Merkle tree is generated based on the plurality of second data blocks; The step of identifying the target interface data that has changed in the second interface data based on the first Merkle tree and the second Merkle tree includes: Starting from the root node, compare the hash values of each node in the first Merkle tree and the second Merkle tree layer by layer; The nodes with different hash values in the first Merkle tree and the second Merkle tree are identified to obtain identification results; wherein, the identification results are used to indicate whether the nodes with different hash values are leaf nodes in the first Merkle tree and the second Merkle tree; The target interface data is determined based on the identification results; The step of determining the target interface data based on the identification result includes: If, based on the identification result, the node with the different hash value is determined to be the leaf node, the target interface data is determined based on the data block corresponding to that node; If, based on the identification result, it is determined that the node with the different hash value is not the leaf node, the first Merkle tree and the second Merkle tree are traversed downwards along the node path corresponding to that node until the leaf node is located; The target interface data is determined based on the data block corresponding to the leaf node.
5. An electronic device, characterized in that, include: The device includes a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, the steps of the data synchronization method as described in any one of claims 1 to 3 are performed.
6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the data synchronization method as described in any one of claims 1 to 3.
7. A computer program product, characterized in that, The computer program product is stored in a storage medium, and the program product is executed by at least one processor to implement the data synchronization method as described in any one of claims 1 to 3.