Data processing method and device, equipment, medium and program product

Abstract

The invention provides a data processing method and device, equipment, a medium and a program product, and can be applied to the technical field of distribution and financial science and technology. The method comprises the following steps: determining a plurality of nodes corresponding to a to-be-executed task; performing entry verification on the respective data version identifiers of the plurality of nodes to obtain a target verification result; under the condition that the target verification result shows that the data version identifier of the target node does not meet the preset updating demand condition, obtaining a target Merkel tree topology used for representing multiple pieces of target slice data in the target to-be-processed data and a reference Merkel tree topology used for representing multiple pieces of reference slice data in the reference to-be-processed data; according to a comparison result of the reference Merkel tree topology and the target Merkel tree topology, determining slice data to be updated from the multiple pieces of target slice data; and performing data synchronization on the to-be-updated slice data in the target to-be-processed data according to the reference slice data to obtain target slice data for executing the to-be-executed task.

Description

Technical Field

[0001] This application relates to the fields of distributed technology and financial technology, and more specifically to a data processing method, apparatus, device, medium and program product. Background Technology

[0002] Distributed batch tasks are executed in parallel by multiple nodes. During the task execution, the data processed on each node will change, so it is necessary to synchronize the data on each node.

[0003] During the data synchronization process, the related technologies suffer from low efficiency due to the large amount of data, and are prone to data concurrency conflicts, leading to unstable data synchronization. Summary of the Invention

[0004] In view of the above problems, this application provides a data processing method, apparatus, device, medium and program product.

[0005] According to a first aspect of this application, a data processing method is provided, comprising: determining multiple nodes corresponding to a task to be executed, wherein each of the multiple nodes is associated with data to be processed, and the nodes execute the task to be executed by processing the data to be processed; verifying the data version identifiers of the multiple nodes to obtain a target verification result, wherein the data version identifiers represent the data update time attribute of the data to be processed; if the target verification result indicates that the data version identifier of the target node does not meet a preset update requirement condition, obtaining a target Merkle tree topology for representing multiple target slice data in the target data to be processed, and a benchmark Merkle tree topology for representing multiple benchmark slice data in the benchmark data to be processed, wherein the target data to be processed is associated with a target node, and the data version identifiers associated with the benchmark data to be processed meet the preset update requirement condition; determining slice data to be updated from the multiple target slice data based on the comparison result of the benchmark Merkle tree topology and the target Merkle tree topology; and synchronizing the slice data to be updated in the target data to be processed with the benchmark slice data to obtain target slice data for executing the task to be executed.

[0006] According to an embodiment of this application, the target Merkle tree topology is determined based on the following operations: calculating the leaf hash values ​​of multiple target slice data, and using the leaf hash values ​​of multiple target slice data as leaf data blocks of the target Merkle tree topology; repeating the following operations until there is exactly one nth fusion hash value: when n > 1, merging the (n-1)th fusion hash values ​​of two adjacent (n-1)th intermediate data blocks to obtain multiple nth fusion hash values, using the multiple nth fusion hash values ​​as multiple nth intermediate data blocks of the target Merkle tree topology, with an edge relationship between two adjacent (n-1)th intermediate data blocks and the corresponding nth intermediate data block, and the multiple nth fusion hash values ​​are obtained by merging adjacent leaf hash values; when there is exactly one nth fusion hash value, using the nth fusion hash value as the root hash value, and using the root hash value as the root data block of the target Merkle tree topology, the root hash value representing the hash value of merging all target slice data.

[0007] According to an embodiment of this application, the comparison result between the benchmark Merkle tree topology and the target Merkle tree topology is determined based on the following operations: Repeating the following operations until the target first intermediate data block is obtained: comparing the nth fusion hash values ​​of multiple nth intermediate data blocks of both the benchmark and target Merkle tree topologies to obtain multiple nth comparison results; based on the nth comparison results, determining the target nth intermediate data block of both the benchmark and target Merkle tree topologies, and the target nth fusion hash value of the target nth intermediate data block, wherein the target... The nth fusion hash value is inconsistent; based on the edge relationships between the benchmark Merkle tree topology and the target Merkle tree topology and the target nth intermediate data block, determine the target (n-1)th intermediate data block of each of the benchmark Merkle tree topology and the target Merkle tree topology; based on the edge relationships between the benchmark Merkle tree topology and the target Merkle tree topology and the target 1st intermediate data block, determine the target leaf data block of each of the benchmark Merkle tree topology and the target Merkle tree topology; use the target leaf data blocks of each of the benchmark Merkle tree topology and the target Merkle tree topology as the comparison result between the benchmark Merkle tree topology and the target Merkle tree topology.

[0008] According to an embodiment of this application, data synchronization is performed on the slice data to be updated in the target data to be processed based on the reference slice data to obtain target slice data for executing the task to be executed. This includes: determining target reference slice data based on the target leaf data block of the reference Merkle tree topology; updating the slice data to be updated in the target data to be processed using the target reference slice data to obtain target slice data, and updating the data version identifier of the target slice data. The target slice data and the target reference slice data are consistent, and the data version identifier of the target slice data and the data version identifier of the target reference slice data are consistent.

[0009] According to an embodiment of this application, the above method further includes: for any node, performing conflict detection on the task to be executed of the node to obtain a conflict detection result; if the conflict detection result meets a preset conflict condition, scheduling the task to be executed to obtain processed data to be processed.

[0010] According to an embodiment of this application, conflict detection is performed on the tasks to be executed of a node to obtain conflict detection results, including: classifying multiple subtasks to be executed of the task to be executed to obtain multiple task categories, wherein the multiple subtasks to be executed include multiple write tasks; for any task category, determining the ratio between the number of write tasks and the total number of subtasks to be executed to obtain the conflict subprobability of the task category; and weighted summing of the conflict subprobabilities of the multiple task categories to obtain the conflict detection results.

[0011] According to an embodiment of this application, scheduling a task to be executed to obtain processed data to be processed includes: for any two subtasks to be executed, if both subtasks process the same slice of data, the two subtasks to be executed are treated as a pair of conflicting tasks; according to a specified scheduling mode, the two subtasks to be executed in the pair of conflicting tasks are isolated to obtain two isolated subtasks to be executed; and according to the two isolated subtasks to be executed, the data to be processed is updated to obtain the processed data to be processed.

[0012] According to an embodiment of this application, before verifying the data version identifiers of multiple nodes, the method further includes: for any node, obtaining the physical clock information and logical clock information of multiple slices of data to be processed using a hybrid logical clock; generating timestamp information of multiple slices of data based on the physical clock information and logical clock information of multiple slices of data; generating data version identifiers of multiple slices of data based on the timestamp information of multiple slices of data; determining the data version identifier of the node based on the data version identifiers of multiple slices of data; wherein, when at least one of the physical clock information and logical clock information changes, the timestamp information is updated accordingly to update the data version identifier.

[0013] The second aspect of this application provides a data processing apparatus, comprising: a first determining module, configured to determine multiple nodes corresponding to a task to be executed, wherein each of the multiple nodes is associated with data to be processed, and the nodes execute the task to be executed by processing the data to be processed; a verification module, configured to verify the data version identifiers of the multiple nodes to obtain a target verification result, wherein the data version identifiers represent the data update time attribute of the data to be processed; a first acquiring module, configured to acquire a target Merkle tree topology for characterizing multiple target slice data in the target data to be processed, and a benchmark Merkle tree topology for characterizing multiple benchmark slice data in the benchmark data to be processed, wherein the target data to be processed is associated with a target node, and the data version identifiers associated with the benchmark data to be processed meet the preset update requirement conditions; a second determining module, configured to determine slice data to be updated from multiple target slice data based on the comparison result of the benchmark Merkle tree topology and the target Merkle tree topology; and a synchronization module, configured to synchronize the slice data to be updated in the target data to be processed based on the benchmark slice data to obtain target slice data for executing the task to be executed.

[0014] A third aspect of this application provides an electronic device comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the method described above.

[0015] A fourth aspect of this application also provides a computer-readable storage medium having a computer program or instructions stored thereon, which, when executed by a processor, implement the steps of the above-described method.

[0016] The fifth aspect of this application also provides a computer program product, including a computer program or instructions that, when executed by a processor, implement the steps of the above-described method.

[0017] According to the embodiments of this application, multiple nodes corresponding to the task to be executed are determined; the data version identifiers of each of the multiple nodes are verified to obtain the target verification result; if the target verification result indicates that the data version identifier of the target node does not meet the preset update requirement condition, the target Merkle tree topology for representing multiple target slice data in the target data to be processed and the benchmark Merkle tree topology for representing multiple benchmark slice data in the benchmark data to be processed are obtained; based on the comparison result of the benchmark Merkle tree topology and the target Merkle tree topology, the slice data to be updated is determined from the multiple target slice data; the slice data to be updated in the target data to be processed is synchronized with the benchmark slice data to obtain the target slice data for executing the task to be executed. By comparing the baseline Merkle tree topology with the target Merkle tree topology, it is not necessary to traverse all the data on each node, thus avoiding the problem of low data processing efficiency caused by the large amount of data to be processed, and improving the processing efficiency of the data synchronization process. In addition, since the data to be updated slice data in the target data to be processed is synchronized according to the baseline slice data, and the baseline slice data meets the preset update requirements, when only the slice data to be updated is modified, concurrent conflicts in the data modification process can be avoided, thus improving the stability of the data synchronization process. Attached Figure Description

[0018] The above-mentioned contents, other objects, features and advantages of this application will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0019] Figure 1 The illustrations depict application scenarios of data processing methods, apparatuses, devices, media, and program products according to embodiments of this application.

[0020] Figure 2 A flowchart illustrating a data processing method according to an embodiment of this application is shown schematically.

[0021] Figure 3a A schematic diagram of the data to be processed according to an embodiment of this application is shown;

[0022] Figure 3b A schematic diagram of a difference bitmap according to an embodiment of this application is shown;

[0023] Figure 3c A schematic diagram of target slice data according to an embodiment of this application is shown;

[0024] Figure 4 The diagram illustrates data synchronization according to an embodiment of this application.

[0025] Figure 5 The illustration shows a schematic diagram of the isolation of conflicting tasks according to an embodiment of this application;

[0026] Figure 6 This illustration schematically shows a data version identifier snapshot alignment according to an embodiment of this application;

[0027] Figure 7 A schematic diagram illustrating a data processing method according to an embodiment of this application is shown.

[0028] Figure 8 A schematic block diagram of a data processing apparatus according to an embodiment of this application is shown; and

[0029] Figure 9 A block diagram schematically illustrates an electronic device suitable for implementing a data processing method according to an embodiment of this application. Detailed Implementation

[0030] The embodiments of this application will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of this application. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of this application for ease of explanation. However, it will be apparent that one or more embodiments may be implemented without these specific details. Furthermore, descriptions of well-known structures and technologies are omitted in the following description to avoid unnecessarily obscuring the concepts of this application.

[0031] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0032] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0033] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0034] In the technical solution of this application, the user information (including but not limited to user personal information, user image information, user device information, such as location information) and data (including but not limited to data used for analysis, stored data, and displayed data) involved are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of related data all comply with relevant laws, regulations, and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entry points for users to choose to authorize or refuse.

[0035] In scenarios involving automated decision-making using personal information, the methods, devices, and systems provided in this application all offer users corresponding entry points for choosing to agree to or reject the automated decision-making results. If the user chooses to reject, the process proceeds to the expert decision-making stage. Here, "automated decision-making" refers to the activity of automatically analyzing and evaluating an individual's behavioral habits, interests, or economic, health, and credit status through computer programs, and then making a decision. Here, "expert decision-making" refers to the activity of making decisions by personnel who specialize in a particular field, possess specialized experience, knowledge, and skills, and have reached a certain level of professional expertise.

[0036] This application provides a data processing method, comprising: determining multiple nodes corresponding to a task to be executed, wherein each of the multiple nodes is associated with data to be processed, and the nodes execute the task by processing the data to be processed; verifying the data version identifiers of the multiple nodes to obtain a target verification result, wherein the data version identifiers represent the data update time attribute of the data to be processed; if the target verification result indicates that the data version identifier of the target node does not meet a preset update requirement condition, obtaining a target Merkle tree topology for characterizing multiple target slice data in the target data to be processed, and a benchmark Merkle tree topology for characterizing multiple benchmark slice data in the benchmark data to be processed, wherein the target data to be processed is associated with a target node, and the data version identifiers associated with the benchmark data to be processed meet the preset update requirement condition; determining slice data to be updated from the multiple target slice data based on the comparison result of the benchmark Merkle tree topology and the target Merkle tree topology; and synchronizing the slice data to be updated in the target data to be processed with the benchmark slice data to obtain target slice data for executing the task to be executed.

[0037] Figure 1 The illustrations depict application scenarios of data processing methods, apparatuses, devices, media, and program products according to embodiments of this application.

[0038] like Figure 1As shown, application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, and a server 105. Network 104 serves as a medium for providing a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, network 104, and server 105. Network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0039] Users can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 via the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0040] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0041] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (this is just an example). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0042] It should be noted that the data processing method provided in the embodiments of this application can generally be executed by server 105. Correspondingly, the data processing device provided in the embodiments of this application can generally be located in server 105. The data processing method provided in the embodiments of this application can also be executed by a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105. Correspondingly, the data processing device provided in the embodiments of this application can also be located in a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105.

[0043] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0044] The following will be based on Figure 1 The described scene, through Figures 2-7 The data processing method according to the embodiments of this application will be described in detail.

[0045] Figure 2 A flowchart illustrating a data processing method according to an embodiment of this application is shown.

[0046] like Figure 2 As shown, the data processing method of this embodiment includes operations S210 to S250.

[0047] In operation S210, multiple nodes corresponding to the task to be executed are identified. Each of the multiple nodes is associated with the data to be processed, and the nodes execute the task by processing the data to be processed.

[0048] According to an embodiment of this application, the task to be executed can be a distributed batch task, which is used to decompose the task into multiple sub-tasks and process them in parallel on multiple nodes. The task to be executed may include processing batch accounting data, batch order data, etc.

[0049] Among them, the tasks to be executed can be tasks that have been assigned to nodes but have not yet been executed, or tasks that are being executed but have not yet been completed. During the execution of the tasks, the data to be processed will change as the tasks are executed.

[0050] According to the embodiments of this application, multiple nodes transmit data through a transmission protocol, which effectively reduces network latency, eliminates the need for full node broadcasting, and allows for inconsistent data to be processed during the node confirmation process without locking the data.

[0051] In operation S220, the data version identifiers of multiple nodes are verified to obtain the target verification result. The data version identifier represents the data update time attribute of the data to be processed.

[0052] According to an embodiment of this application, the data version identifier of a node is used to display the version of the data to be processed in the node. When a task to be executed is assigned to a node, an initial version is added to the corresponding data to be processed. As the task is executed or the node is updated, the data to be processed will change, and the version of the data to be processed will also change accordingly.

[0053] The data update time attribute can be the time point when the data to be processed changes. As this time point changes, the data version identifier changes accordingly.

[0054] According to the embodiments of this application, the data version identifiers of multiple nodes are verified to determine whether the data version identifiers of all nodes are consistent, and the result of this determination is used as the target verification result.

[0055] In operation S230, if the target verification result indicates that the data version identifier of the target node does not meet the preset update requirement conditions, the target Merkle tree topology used to characterize multiple target slice data in the target data to be processed, and the benchmark Merkle tree topology used to characterize multiple benchmark slice data in the benchmark data to be processed are obtained. The target data to be processed is related to the target node, and the data version identifier related to the benchmark data to be processed meets the preset update requirement conditions.

[0056] According to an embodiment of this application, the preset update requirement condition may be that the data version identifier reaches a preset version threshold. For example, the preset version threshold is version 2. If the data version identifier of a node indicates version 1, then the node is the target node, and the data to be processed on the target node is the target data to be processed. If the data version identifier of a node indicates version 3, then the data to be processed on the node is the baseline data to be processed.

[0057] According to embodiments of this application, target slice data is obtained by dividing target data to be processed. For example, the target data to be processed is divided into slices of a fixed size (e.g., 64KB) to obtain target slice data. Similarly, the baseline data to be processed is divided to obtain baseline slice data.

[0058] According to embodiments of this application, a Merkle tree is a tree-like data structure. A Merkle tree begins with multiple leaf data blocks and, based on a bottom-up construction method, merges the leaf hash values ​​of adjacent leaf data blocks to obtain the data blocks of the next level, until the last root data block is obtained, thus completing the construction of the Merkle tree.

[0059] In operation S240, based on the comparison results between the baseline Merkle tree topology and the target Merkle tree topology, the slice data to be updated is determined from multiple target slice data.

[0060] According to embodiments of this application, a baseline Merkle tree topology and a target Merkle tree topology are compared to find the target leaf data block of the target Merkle tree topology, thereby enabling the determination of the slice data to be updated from multiple target slice data.

[0061] According to embodiments of this application, Merkle tree topology can quickly verify the consistency between target data to be processed and baseline data to be processed by comparing the path from leaf data block to root data block without traversing the entire Merkle tree topology, effectively reducing the computational load of comparing target data to be processed and baseline data to be processed.

[0062] In operation S250, the slice data to be updated in the target data to be processed is synchronized with the reference slice data to obtain the target slice data for executing the task to be executed.

[0063] According to an embodiment of this application, reference slice data is used to replace the slice data to be updated in the target data to be processed, so that the reference slice data and the slice data to be updated are synchronized to obtain the target slice data.

[0064] According to the embodiments of this application, if it is determined that the data to be processed on all nodes is consistent, the task to be executed continues. Since the data to be processed on multiple nodes is synchronized only before the task to be executed, the system overhead is reduced while the system stability is improved.

[0065] According to the embodiments of this application, multiple nodes corresponding to the task to be executed are determined; the data version identifiers of each of the multiple nodes are verified to obtain the target verification result; if the target verification result indicates that the data version identifier of the target node does not meet the preset update requirement condition, the target Merkle tree topology for representing multiple target slice data in the target data to be processed and the benchmark Merkle tree topology for representing multiple benchmark slice data in the benchmark data to be processed are obtained; based on the comparison result of the benchmark Merkle tree topology and the target Merkle tree topology, the slice data to be updated is determined from the multiple target slice data; the slice data to be updated in the target data to be processed is synchronized with the benchmark slice data to obtain the target slice data for executing the task to be executed. By comparing the baseline Merkle tree topology with the target Merkle tree topology, it is not necessary to traverse all the data on each node, thus avoiding the problem of low data processing efficiency caused by the large amount of data to be processed, and improving the processing efficiency of the data synchronization process. In addition, since the data to be updated slice data in the target data to be processed is synchronized according to the baseline slice data, and the baseline slice data meets the preset update requirements, when only the slice data to be updated is modified, concurrent conflicts in the data modification process can be avoided, thus improving the stability of the data synchronization process.

[0066] According to an embodiment of this application, the target Merkle tree topology is determined based on the following operations: calculating the leaf hash values ​​of multiple target slice data, and using the leaf hash values ​​of multiple target slice data as leaf data blocks of the target Merkle tree topology; repeating the following operations until there is exactly one nth fusion hash value: when n > 1, merging the (n-1)th fusion hash values ​​of two adjacent (n-1)th intermediate data blocks to obtain multiple nth fusion hash values, using the multiple nth fusion hash values ​​as multiple nth intermediate data blocks of the target Merkle tree topology, with an edge relationship between two adjacent (n-1)th intermediate data blocks and the corresponding nth intermediate data block, and the multiple nth fusion hash values ​​are obtained by merging adjacent leaf hash values; when there is exactly one nth fusion hash value, using the nth fusion hash value as the root hash value, and using the root hash value as the root data block of the target Merkle tree topology, the root hash value representing the hash value of merging all target slice data.

[0067] According to embodiments of this application, calculating the leaf hash value of each of multiple target slice data can be achieved by converting each target slice data into a sequence of numbers based on a hash algorithm, with different leaf hash values ​​corresponding to different target slice data.

[0068] According to an embodiment of this application, merging two adjacent leaf hash values ​​can be achieved by concatenating the two leaf hash values ​​and then generating a new hash value based on the concatenated sequence number to obtain the first merged hash value. Similarly, merging two adjacent first merged hash values ​​yields the second merged hash value, and so on, until only one merged hash value is obtained. This hash value is then used as the root hash value of the Merkle tree topology, thereby completing the construction of the Merkle tree topology.

[0069] Among them, there are edge relationships between data blocks. For example, after leaf hash value A and leaf hash value B are merged to obtain the first merged hash value A, the corresponding leaf data block A and leaf data block B are respectively connected to the first intermediate data block A.

[0070] According to the embodiments of this application, since the leaf hash values ​​corresponding to different target slice data are different, as different hash values ​​are merged, when generating the root hash value, if two nodes have at least one different leaf hash value, then the root hash values ​​of the two nodes must be different, thereby enabling a clearer determination of the consistency between different nodes.

[0071] According to an embodiment of this application, the comparison result between the benchmark Merkle tree topology and the target Merkle tree topology is determined based on the following operations: Repeating the following operations until the target first intermediate data block is obtained: comparing the nth fusion hash values ​​of multiple nth intermediate data blocks of both the benchmark and target Merkle tree topologies to obtain multiple nth comparison results; based on the nth comparison results, determining the target nth intermediate data block of both the benchmark and target Merkle tree topologies, and the target nth fusion hash value of the target nth intermediate data block, wherein the target... The nth fusion hash value is inconsistent; based on the edge relationships between the benchmark Merkle tree topology and the target Merkle tree topology and the target nth intermediate data block, determine the target (n-1)th intermediate data block of each of the benchmark Merkle tree topology and the target Merkle tree topology; based on the edge relationships between the benchmark Merkle tree topology and the target Merkle tree topology and the target 1st intermediate data block, determine the target leaf data block of each of the benchmark Merkle tree topology and the target Merkle tree topology; use the target leaf data blocks of each of the benchmark Merkle tree topology and the target Merkle tree topology as the comparison result between the benchmark Merkle tree topology and the target Merkle tree topology.

[0072] According to an embodiment of this application, when it is determined that the root data blocks of the reference Merkle tree topology and the target Merkle tree topology are inconsistent, each intermediate data block is compared layer by layer from top to bottom. The two nth intermediate data blocks of the next layer below the root data block are compared respectively to obtain multiple nth comparison results. The nth comparison result is whether the nth intermediate data blocks of the reference Merkle tree topology and the target Merkle tree topology are consistent.

[0073] If the nth comparison result shows that the nth intermediate data block of the reference Merkle tree topology and the target Merkle tree topology are inconsistent, then the inconsistent nth intermediate data block is taken as the target nth intermediate data block. Furthermore, based on the edge relationship corresponding to the target 1st intermediate data block, the corresponding two target (n-1)th intermediate data blocks are found until the target 1st intermediate data block is obtained.

[0074] For example, if the root data blocks of the baseline Merkle tree topology and the target Merkle tree topology are inconsistent, the nth fusion hash value A of the nth intermediate data block A and the nth fusion hash value B of the nth intermediate data block B of the two Merkle tree topologies are compared. If the nth fusion hash value A of the two Merkle tree topologies is inconsistent, then based on the edge relationship of the nth intermediate data block A, the (n-1)th intermediate data block A and the (n-1)th intermediate data block B are found for a new round of comparison.

[0075] According to the embodiments of this application, after determining the target first intermediate data block of the reference Merkle tree topology and the target Merkle tree topology, the target leaf data block of the reference Merkle tree topology and the target Merkle tree topology can be determined, thereby finding the slice data in the target node that is inconsistent with the reference data to be processed.

[0076] According to the embodiments of this application, comparing the baseline Merkle tree topology and the target Merkle tree topology only requires comparing a portion of the data blocks based on the edge relationships between the data blocks, without traversing all data blocks, thus reducing the amount of data processing and improving processing efficiency.

[0077] According to an embodiment of this application, data synchronization is performed on the slice data to be updated in the target data to be processed based on the reference slice data to obtain target slice data for executing the task to be executed. This includes: determining target reference slice data based on the target leaf data block of the reference Merkle tree topology; updating the slice data to be updated in the target data to be processed using the target reference slice data to obtain target slice data, and updating the data version identifier of the target slice data. The target slice data and the target reference slice data are consistent, and the data version identifier of the target slice data and the data version identifier of the target reference slice data are consistent.

[0078] According to an embodiment of this application, after the target node determines the slice data to be updated, it sends the slice data to be updated in the form of a difference bitmap to the base node where the base slice data is located via a transmission protocol. The base node verifies the target slice data based on the difference bitmap, and after verification, transmits the target base slice data to the target node via a transmission protocol.

[0079] After receiving the target base slice data, the target node replaces the target slice data with the target base slice data to make the target slice data consistent with the target base slice data. In addition, while replacing the target slice data, the data version identifier of the target slice data is also updated accordingly. After both the data and the data version identifier are updated, the target slice data used to execute the task to be executed is obtained.

[0080] Figure 3a A schematic diagram of the data to be processed according to an embodiment of this application is shown; Figure 3b A schematic diagram of a difference bitmap according to an embodiment of this application is shown; Figure 3c A schematic diagram of target slice data according to an embodiment of this application is shown.

[0081] As shown in Figure 3, the data to be processed at the target node is as follows: Figure 3a As shown, after determining the slice data that needs to be updated, we can obtain the following: Figure 3bThe difference bitmap shown is transmitted to the base node. After verification, the base node transmits the target base slice data to the target node via the transmission protocol. The target node updates the slice data to be updated, resulting in the following: Figure 3c The target slice data is shown.

[0082] For example, if the target node contains 100GB of data to be processed, and it is determined that the target slice data only accounts for 10%, then only 10GB of data needs to be synchronized, effectively reducing the amount of data to be synchronized.

[0083] Furthermore, if there are no nodes among the multiple nodes that meet the preset update requirements, or if multiple nodes that meet the preset update requirements need to synchronize data, the Merkle tree topology of each of the two nodes can be obtained based on the above method to determine the inconsistent slice data in the two nodes. Further, the log data of the task to be executed can be obtained, and the latest slice data in the two nodes can be determined based on the log data of the task to be executed in order to replace the other slice data.

[0084] Figure 4 A schematic diagram illustrating data synchronization according to an embodiment of this application is shown.

[0085] like Figure 4 As shown, when operation S410 determines that there is baseline slice data that meets the preset update requirements, operation S420 determines the difference bitmap of the slice data to be updated in the target node based on the comparison results of the baseline Merkle tree topology and the target Merkle tree topology. In operation S430, the slice data to be updated in the target data to be processed is updated using the target baseline slice data to obtain the target slice data. If operation S410 determines that there is no node that meets the preset update requirements, then operation S440 obtains the log data of the task to be executed, and operation S450 determines the latest slice data in the two nodes based on the log data of the task to be executed, so as to replace the other slice data.

[0086] According to the embodiments of this application, the target reference slice data is used to update the slice data to be updated in the target data to be processed, thereby avoiding conflicts caused by multiple nodes updating the target data to be processed and improving the stability of the data synchronization process.

[0087] According to an embodiment of this application, the above method further includes: for any node, performing conflict detection on the task to be executed of the node to obtain a conflict detection result; if the conflict detection result meets a preset conflict condition, scheduling the task to be executed to obtain processed data to be processed.

[0088] According to an embodiment of this application, conflict detection is performed on the tasks to be executed on a node to determine whether the tasks of the node are prone to conflict, the probability of conflict is calculated, and the result is used as the conflict detection result.

[0089] According to an embodiment of this application, the preset conflict condition can be that the probability of conflict reaches a preset probability threshold. If the threshold is met, it means that the node is likely to generate sufficient tasks to be executed. It is necessary to isolate the sub-tasks that may cause conflicts in the tasks to be executed in order to obtain the processed data to be processed.

[0090] According to the embodiments of this application, conflict detection is performed on the tasks to be executed on the node to avoid concurrent conflicts during task execution, thereby ensuring the stability of the tasks to be executed.

[0091] According to an embodiment of this application, conflict detection is performed on the tasks to be executed of a node to obtain conflict detection results, including: classifying multiple subtasks to be executed of the task to be executed to obtain multiple task categories, wherein the multiple subtasks to be executed include multiple write tasks; for any task category, determining the ratio between the number of write tasks and the total number of subtasks to be executed to obtain the conflict subprobability of the task category; and weighted summing of the conflict subprobabilities of the multiple task categories to obtain the conflict detection results.

[0092] According to embodiments of this application, the task to be executed includes multiple subtasks to be executed, which may include different types of tasks, each type of task having different importance.

[0093] According to an embodiment of this application, the subtask to be executed includes a write task for writing data or modifying data. For any task category, the probability of conflict is first determined based on the ratio between the number of write tasks executing the subtask and the total number of subtasks to be executed.

[0094] According to the embodiments of this application, each type of task has different importance, and therefore the task categories have different weights. Based on the weight corresponding to each conflict sub-probability, multiple conflict sub-probabilities are weighted and summed to obtain the conflict probability of the task to be executed, i.e., the conflict detection result.

[0095] According to the embodiments of this application, by weighted summation of the conflict subprobabilities of multiple task categories, it is possible to flexibly adapt to different sub-tasks to be executed, avoid being affected by marginal sub-tasks, and accurately determine the conflict probability of the sub-tasks to be executed, so as to facilitate further isolation processing of the sub-tasks to be executed.

[0096] According to an embodiment of this application, scheduling a task to be executed to obtain processed data to be processed includes: for any two subtasks to be executed, if both subtasks process the same slice of data, the two subtasks to be executed are treated as a pair of conflicting tasks; according to a specified scheduling mode, the two subtasks to be executed in the pair of conflicting tasks are isolated to obtain two isolated subtasks to be executed; and according to the two isolated subtasks to be executed, the data to be processed is updated to obtain the processed data to be processed.

[0097] According to an embodiment of this application, if both subtasks to be executed perform write operations on the same slice data during their execution, or if one subtask to be executed performs a write operation on the slice data during its execution, while the other subtask to be executed performs a read operation on the slice data, then the two subtasks to be executed are considered a pair of conflicting tasks.

[0098] According to embodiments of this application, the specified scheduling modes include hard isolation mode and soft isolation mode.

[0099] If the conflict detection results meet the preset conflict conditions, the two subtasks to be executed in the conflict task pair are isolated in hard isolation mode. In hard isolation mode, it is necessary to determine the reserved nodes associated with the conflict task pair and schedule the two subtasks to be executed in the conflict task pair to different reserved nodes.

[0100] If the conflict detection result does not meet the preset conflict conditions and there are still conflicting task pairs, the two pending subtasks of the conflicting task pair are isolated in soft isolation mode. In soft isolation mode, the execution time of the two pending subtasks of the conflicting task pair needs to be adjusted by inserting a barrier to delay the start of one of the pending subtasks.

[0101] Figure 5 A schematic diagram illustrating the isolation of conflicting tasks according to an embodiment of this application is shown.

[0102] like Figure 5 As shown, after obtaining the conflicting task pair in operation S510, based on the judgment result of the conflict detection result, if operation S520 determines that the conflict detection result meets the preset conflict conditions, operation S530 uses hard isolation mode to isolate the two sub-tasks to be executed in the conflicting task pair and schedules the two sub-tasks to be executed in the conflicting task pair to different reserved nodes. If operation S520 determines that the conflict detection result does not meet the preset conflict conditions and there is still a conflicting task pair, operation S540 uses soft isolation mode to isolate the two sub-tasks to be executed in the conflicting task pair and adjusts the execution time of the two sub-tasks to be executed in the conflicting task pair.

[0103] For example, if both subtask A and subtask B are to be executed from account X, and the conflict detection result is that the conflict probability is >0.7, then subtask A and subtask B are assigned to physically isolated node groups (Group 1 and Group 2), respectively; if the conflict probability is ≤0.7, then the execution order is adjusted on that node (subtask B starts after subtask A is submitted).

[0104] According to the embodiments of this application, by isolating the two subtasks to be executed in a conflicting task pair, the problem of data abnormality caused by the simultaneous processing of a slice of data during the execution of the task can be avoided, thereby ensuring the stability of the task execution process.

[0105] According to an embodiment of this application, before verifying the data version identifiers of multiple nodes, the method further includes: for any node, obtaining the physical clock information and logical clock information of multiple slices of data to be processed using a hybrid logical clock; generating timestamp information of multiple slices of data based on the physical clock information and logical clock information of multiple slices of data; generating data version identifiers of multiple slices of data based on the timestamp information of multiple slices of data; determining the data version identifier of the node based on the data version identifiers of multiple slices of data; wherein, when at least one of the physical clock information and logical clock information changes, the timestamp information is updated accordingly to update the data version identifier.

[0106] According to embodiments of this application, a hybrid logic clock can determine the global time (physical clock information) on a node based on a physical clock, while using a logic clock to record the change time of slice data (logic clock information).

[0107] According to an embodiment of this application, a hybrid logic clock records the physical clock information and logic clock information of multiple slice data respectively, thereby generating a data version identifier for each of the multiple slice data. If at least one of the physical clock information and the logic clock information changes, the data version identifier will change accordingly.

[0108] The data version identifier can be represented as "timestamp_node ID_serial number".

[0109] In addition, the hybrid logic clock takes into account both the logical causal consistency of the sliced ​​data and the physical time of the sliced ​​data, transforming the inevitable physical drift between nodes into logical drift within the nodes. Compared to physical drift, logical drift allows nodes to continue generating monotonically increasing timestamps without any external physical synchronization.

[0110] Figure 6 This illustration schematically shows a data version identifier snapshot alignment according to an embodiment of this application.

[0111] Specifically, nodes need to perform snapshot alignment of data version identifiers, such as... Figure 6 As shown, in operation S610, the node obtains the initial data version identifier based on the hybrid logic clock. If the data to be processed changes, the data version identifier needs to be updated in operation S620. The node will periodically traverse the data version identifiers of multiple slice data. In operation S630, it is determined whether the data version identifier of the slice data is consistent with the initial data version identifier. If they are inconsistent, the node's data version identifier is updated in operation S640 so that the node's data version identifier is the latest version.

[0112] Figure 7 This illustration schematically shows a data processing method according to an embodiment of the present application.

[0113] like Figure 7 As shown, during the initialization phase, the system needs to determine multiple nodes corresponding to the tasks to be executed; further, it updates the versions of the data to be processed on the nodes and performs conflict detection on the tasks to be executed in order to complete the data loading; further, it synchronizes the data to be processed on multiple nodes. When the data to be processed on multiple nodes is consistent, it indicates that the calculation results between the nodes are ready and multiple nodes are ready to execute the tasks to be executed.

[0114] For example, when processing 100,000 interbank transfers in batches, synchronizing all account balance data would cause network congestion. Instead, each node asynchronously loads account data (data to be processed), and then uses an optimistic algorithm to verify the data version identifier of the data to be processed on each node. After verification, based on strong consistency, the data to be processed on each node is synchronized, synchronizing only the slice data involved in the transfers, which reduces network traffic by 90% during the data synchronization process.

[0115] For example, in distributed computation of 1000×1000 matrix multiplication, the intermediate result synchronization delay accounts for 50% of the overall time consumption. By pulling matrix blocks (data to be processed), determining the data version identifier for each block, and after verification, only the modified blocks are transmitted, reducing the amount of synchronized data from 1GB to 100MB. Simultaneously, after modification, conflict detection is performed on the matrix blocks to avoid concurrent conflicts.

[0116] Based on the above data processing method, this application also provides a data processing apparatus. The following will be combined with... Figure 8 The device is described in detail.

[0117] Figure 8 A schematic block diagram of a data processing apparatus according to an embodiment of this application is shown.

[0118] like Figure 8As shown, the data processing device 800 of this embodiment includes a first determining module 810, a verification module 820, a first acquiring module 830, a second determining module 840, and a synchronization module 850.

[0119] The first determining module 810 is used to determine multiple nodes corresponding to the task to be executed, wherein each of the multiple nodes is associated with the data to be processed, and the nodes execute the task by processing the data to be processed. In one embodiment, the first determining module 810 can be used to perform the operation S210 described above, which will not be repeated here.

[0120] The verification module 820 is used to verify the data version identifiers of multiple nodes to obtain the target verification result. The data version identifier represents the data update time attribute of the data to be processed. In one embodiment, the verification module 820 can be used to perform the operation S220 described above, which will not be repeated here.

[0121] The first acquisition module 830 is used to acquire, when the target verification result indicates that the data version identifier of the target node does not meet the preset update requirement condition, a target Merkle tree topology representing multiple target slice data in the target data to be processed, and a baseline Merkle tree topology representing multiple baseline slice data in the baseline data to be processed. The target data to be processed is related to the target node, and the data version identifier related to the baseline data to be processed meets the preset update requirement condition. In one embodiment, the first acquisition module 830 can be used to perform the operation S230 described above, which will not be repeated here.

[0122] The second determining module 840 is used to determine the slice data to be updated from multiple target slice data based on the comparison results between the baseline Merkle tree topology and the target Merkle tree topology. In one embodiment, the second determining module 840 can be used to perform the operation S240 described above, which will not be repeated here.

[0123] The synchronization module 850 is used to synchronize the slice data to be updated in the target data to be processed with the reference slice data, so as to obtain the target slice data for executing the task to be executed. In one embodiment, the synchronization module 850 can be used to perform the operation S250 described above, which will not be repeated here.

[0124] According to the embodiments of this application, multiple nodes corresponding to the task to be executed are determined; the data version identifiers of each of the multiple nodes are verified to obtain the target verification result; if the target verification result indicates that the data version identifier of the target node does not meet the preset update requirement condition, the target Merkle tree topology for representing multiple target slice data in the target data to be processed and the benchmark Merkle tree topology for representing multiple benchmark slice data in the benchmark data to be processed are obtained; based on the comparison result of the benchmark Merkle tree topology and the target Merkle tree topology, the slice data to be updated is determined from the multiple target slice data; the slice data to be updated in the target data to be processed is synchronized with the benchmark slice data to obtain the target slice data for executing the task to be executed. By comparing the baseline Merkle tree topology with the target Merkle tree topology, it is not necessary to traverse all the data on each node, thus avoiding the problem of low data processing efficiency caused by the large amount of data to be processed, and improving the processing efficiency of the data synchronization process. In addition, since the data to be updated slice data in the target data to be processed is synchronized according to the baseline slice data, and the baseline slice data meets the preset update requirements, when only the slice data to be updated is modified, concurrent conflicts in the data modification process can be avoided, thus improving the stability of the data synchronization process.

[0125] According to embodiments of this application, any plurality of modules among the first determining module 810, the verification module 820, the first acquiring module 830, the second determining module 840, and the synchronization module 850 can be combined into one module, or any one of these modules can be split into multiple modules. Alternatively, at least part of the functionality of one or more of these modules can be combined with at least part of the functionality of other modules and implemented in one module. According to embodiments of this application, at least one of the first determining module 810, the verification module 820, the first acquiring module 830, the second determining module 840, and the synchronization module 850 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or implemented in hardware or firmware by any other reasonable means of integrating or packaging the circuitry, or implemented in any one of the three implementation methods of software, hardware, and firmware, or in a suitable combination of any of these. Alternatively, at least one of the first determining module 810, the verification module 820, the first acquiring module 830, the second determining module 840, and the synchronization module 850 may be implemented at least partially as a computer program module, which can perform corresponding functions when the computer program module is run.

[0126] According to an embodiment of this application, the first acquisition module 830 includes a calculation submodule, a fusion submodule, and a first acquisition submodule.

[0127] The calculation submodule is used to calculate the leaf hash value of each of the multiple target slice data, and use the leaf hash value of each of the multiple target slice data as the leaf data block of the target Merkle tree topology.

[0128] The fusion submodule is used to repeatedly perform the following operations until there is exactly one nth fusion hash value: when n>1, the (n-1)th fusion hash values ​​of two adjacent (n-1)th intermediate data blocks are merged to obtain multiple nth fusion hash values. The multiple nth fusion hash values ​​are used as multiple nth intermediate data blocks of the target Merkle tree topology. There is an edge relationship between two adjacent (n-1)th intermediate blocks and the corresponding nth intermediate block. The multiple nth fusion hash values ​​are obtained by merging adjacent leaf hash values.

[0129] The first module is used as a submodule to take the nth fusion hash value as the root hash value when there is exactly one nth fusion hash value, and to take the root hash value as the root data block of the target Merkle tree topology. The root hash value represents the hash value of fusion of all target slice data.

[0130] According to an embodiment of this application, the second determining module 840 includes a comparison submodule, a first determining submodule, a second determining submodule, a third determining submodule, and a second determining submodule.

[0131] The comparison submodule is used to repeatedly perform the following operations until the target first intermediate data block is obtained: compare the nth fused hash values ​​of multiple nth intermediate data blocks of the baseline Merkle tree topology and the target Merkle tree topology respectively, and obtain multiple nth comparison results.

[0132] The first determining submodule is used to determine the target nth intermediate data block of the benchmark Merkle tree topology and the target Merkle tree topology, and the target nth fusion hash value of the target nth intermediate data block, based on the nth comparison result, wherein the target nth fusion hash value of the benchmark Merkle tree topology and the target Merkle tree topology are inconsistent.

[0133] The second determining submodule is used to determine the target (n-1)th intermediate data block of each of the benchmark Merkle tree topology and the target Merkle tree topology based on the edge relationships between the benchmark Merkle tree topology and the target Merkle tree topology and the target nth intermediate data block.

[0134] The third determination submodule is used to determine the target leaf data blocks of the benchmark Merkle tree topology and the target Merkle tree topology based on the edge relationships between the benchmark Merkle tree topology and the target Merkle tree topology and the target first intermediate data block.

[0135] The second module is a submodule used to take the target leaf data blocks of the baseline Merkle tree topology and the target Merkle tree topology as the comparison results between the baseline Merkle tree topology and the target Merkle tree topology.

[0136] According to an embodiment of this application, the synchronization module 850 includes a fourth determining submodule and a first updating submodule.

[0137] The fourth determination submodule is used to determine the target baseline slice data based on the target leaf data block of the baseline Merkle tree topology.

[0138] The first update submodule is used to update the slice data to be updated in the target data to be processed using the target reference slice data to obtain the target slice data and update the data version identifier of the target slice data. The target slice data is consistent with the target reference slice data, and the data version identifier of the target slice data is consistent with the data version identifier of the target reference slice data.

[0139] According to an embodiment of this application, the data processing apparatus 800 further includes a detection module and a scheduling module.

[0140] The detection module is used to perform conflict detection on the tasks to be executed on any node and obtain the conflict detection results.

[0141] The scheduling module is used to schedule the task to be executed when the conflict detection result meets the preset conflict conditions, and obtain the processed data to be processed.

[0142] According to an embodiment of this application, the detection module includes a classification submodule, a fifth determination submodule, and an acquisition submodule.

[0143] The classification submodule is used to classify multiple subtasks to be executed for the task to be executed, resulting in multiple task categories. The multiple subtasks to be executed include multiple write tasks.

[0144] The fifth determination submodule is used to determine the ratio between the number of write tasks and the total number of subtasks to be executed for any task category, thereby obtaining the conflict subprobability of the task category.

[0145] The obtained submodule is used to perform a weighted summation of the conflict subprobabilities of multiple task categories to obtain the conflict detection results.

[0146] According to an embodiment of this application, the scheduling module includes a third submodule, an isolation submodule, and a second update submodule.

[0147] The third module, as a submodule, is used to treat any two subtasks to be executed as a conflicting task pair if both subtasks process the same slice of data during their execution.

[0148] The isolation submodule is used to isolate two pending subtasks of a conflicting task pair according to a specified scheduling mode, resulting in two isolated pending subtasks.

[0149] The second update submodule is used to update the data to be processed based on the two isolated subtasks to be executed, so as to obtain the processed data to be processed.

[0150] According to an embodiment of this application, the data processing apparatus 800 further includes a second acquisition module, a first generation module, a second generation module, and a third determination module.

[0151] The second acquisition module is used to acquire the physical clock information and logical clock information of each of the multiple slices of data to be processed for any given node using a hybrid logical clock.

[0152] The first generation module is used to generate timestamp information for each of the multiple slice data based on their respective physical clock information and logical clock information.

[0153] The second generation module is used to generate data version identifiers for each of the multiple slice data based on their respective timestamp information.

[0154] The third determination module is used to determine the data version identifier of a node based on the data version identifiers of the multiple slice data; wherein, when at least one of the physical clock information and logical clock information changes, the timestamp information is updated accordingly to update the data version identifier.

[0155] Figure 9 A block diagram schematically illustrates an electronic device suitable for implementing a data processing method according to an embodiment of this application.

[0156] like Figure 9 As shown, an electronic device 900 according to an embodiment of this application includes a processor 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage portion 908 into a random access memory (RAM) 903. The processor 901 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 901 may also include onboard memory for caching purposes. The processor 901 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of this application.

[0157] RAM 903 stores various programs and data required for the operation of electronic device 900. Processor 901, ROM 902, and RAM 903 are interconnected via bus 904. Processor 901 executes various operations of the method flow according to embodiments of this application by executing programs in ROM 902 and / or RAM 903. It should be noted that the programs may also be stored in one or more memories other than ROM 902 and RAM 903. Processor 901 may also execute various operations of the method flow according to embodiments of this application by executing programs stored in said one or more memories.

[0158] According to embodiments of this application, the electronic device 900 may further include an input / output (I / O) interface 905, which is also connected to a bus 904. The electronic device 900 may also include one or more of the following components connected to the input / output (I / O) interface 905: an input section 906 including a keyboard, mouse, etc.; an output section 907 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 908 including a hard disk, etc.; and a communication section 909 including a network interface card such as a LAN card, modem, etc. The communication section 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to the input / output (I / O) interface 905 as needed. A removable medium 911, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 910 as needed so that computer programs read from it can be installed into the storage section 908 as needed.

[0159] This application also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of this application.

[0160] According to embodiments of this application, the computer-readable storage medium can be a non-volatile computer-readable storage medium, such as including but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this application, the computer-readable storage medium may include ROM 902 and / or RAM 903 and / or one or more memories other than ROM 902 and RAM 903 described above.

[0161] Embodiments of this application also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code is used to enable the computer system to implement the data processing methods provided in the embodiments of this application.

[0162] When the computer program is executed by the processor 901, it performs the functions defined in the system / apparatus of this application embodiment. According to the embodiments of this application, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0163] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and downloaded and installed via the communication section 909, and / or installed from a removable medium 911. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0164] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 909, and / or installed from the removable medium 911. When the computer program is executed by the processor 901, it performs the functions defined in the system of this application embodiment. According to the embodiments of this application, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0165] According to embodiments of this application, program code for executing the computer programs provided in the embodiments of this application can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0166] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0167] Those skilled in the art will understand that the features described in the various embodiments of this application can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this application. In particular, the features described in the various embodiments of this application can be combined and / or combined in various ways without departing from the spirit and teachings of this application. All such combinations and / or combinations fall within the scope of this application.

Claims

1. A data processing method, characterized in that, The method includes: Multiple nodes are identified that correspond to a task to be executed, wherein each of the multiple nodes is associated with data to be processed, and the nodes execute the task by processing the data to be processed; The data version identifiers of the multiple nodes are verified to obtain the target verification result, wherein the data version identifiers represent the data update time attribute of the data to be processed. When the target verification result indicates that the data version identifier of the target node does not meet the preset update requirement condition, a target Merkle tree topology for characterizing multiple target slice data in the target data to be processed, and a benchmark Merkle tree topology for characterizing multiple benchmark slice data in the benchmark data to be processed are obtained. The target data to be processed is related to the target node, and the data version identifier related to the benchmark data to be processed meets the preset update requirement condition. Based on the comparison results between the baseline Merkle tree topology and the target Merkle tree topology, the slice data to be updated is determined from the multiple target slice data; and Based on the baseline slice data, the slice data to be updated in the target data to be processed is synchronized to obtain the target slice data for executing the task to be executed.

2. The method according to claim 1, characterized in that, The target Merkle tree topology is determined based on the following operations: Calculate the leaf hash value of each of the multiple target slice data, and use the leaf hash value of each of the multiple target slice data as the leaf data block of the target Merkle tree topology; Repeat the following steps until the nth fusion hash value is exactly one: When n > 1, the (n-1)th fusion hash value of two adjacent (n-1)th intermediate data blocks is fused to obtain multiple nth fusion hash values. The multiple nth fusion hash values ​​are respectively used as multiple nth intermediate data blocks of the target Merkle tree topology. There is an edge relationship between two adjacent (n-1)th intermediate blocks and the corresponding nth intermediate block. The multiple nth fusion hash values ​​are obtained by fusing adjacent leaf hash values. When there is exactly one nth fusion hash value, the nth fusion hash value is used as the root hash value, and the root hash value is used as the root data block of the target Merkle tree topology. The root hash value represents the hash value of fusion of all target slice data.

3. The method according to claim 2, characterized in that, The comparison results of the baseline Merkle tree topology and the target Merkle tree topology are determined based on the following operations: Repeat the following steps until the target first intermediate data block is obtained: The nth fusion hash values ​​of multiple nth intermediate data blocks of the baseline Merkle tree topology and the target Merkle tree topology are compared respectively to obtain multiple nth comparison results; Based on the nth comparison result, the target nth intermediate data block of the benchmark Merkle tree topology and the target Merkle tree topology are determined, as well as the target nth fusion hash value of the target nth intermediate data block, wherein the target nth fusion hash value of the benchmark Merkle tree topology and the target Merkle tree topology are inconsistent; Based on the edge relationships between the benchmark Merkle tree topology and the target Merkle tree topology and the target intermediate data block, determine the target (n-1)th intermediate data block of each of the benchmark Merkle tree topology and the target Merkle tree topology; Based on the edge relationships between the reference Merkle tree topology and the target Merkle tree topology and the target intermediate data block, the target leaf data block of each of the reference Merkle tree topology and the target Merkle tree topology is determined; The target leaf data blocks of the baseline Merkle tree topology and the target Merkle tree topology are used as the comparison results of the baseline Merkle tree topology and the target Merkle tree topology.

4. The method according to claim 3, characterized in that, The step of synchronizing the slice data to be updated in the target data to be processed based on the reference slice data to obtain the target slice data for executing the task to be executed includes: Based on the target leaf data block of the baseline Merkle tree topology, determine the target baseline slice data; Using the target reference slice data, the slice data to be updated in the target data to be processed is updated to obtain the target slice data, and the data version identifier of the target slice data is updated. The target slice data is consistent with the target reference slice data, and the data version identifier of the target slice data is consistent with the data version identifier of the target reference slice data.

5. The method according to claim 1, characterized in that, The method further includes: For any of the aforementioned nodes, Perform conflict detection on the tasks to be executed on the node and obtain the conflict detection results; If the conflict detection result meets the preset conflict conditions, the task to be executed is scheduled and processed to obtain the processed data to be processed.

6. The method according to claim 5, characterized in that, The process of performing conflict detection on the tasks to be executed on the node to obtain conflict detection results includes: The multiple subtasks to be executed for the task to be executed are classified to obtain multiple task categories, and the multiple subtasks to be executed include multiple write tasks; For any of the task categories, determine the ratio between the number of write tasks of the subtask to be executed and the total number of subtasks to be executed, and obtain the conflict subprobability of the task category; The conflict detection result is obtained by weighted summation of the conflict subprobabilities of multiple task categories.

7. The method according to claim 6, characterized in that, The process of scheduling the task to be executed to obtain the processed data to be processed includes: For any two subtasks to be executed, if both subtasks process the same slice of data during their execution, the two subtasks are considered a conflicting task pair. According to the specified scheduling mode, the two subtasks to be executed of the conflicting task pair are isolated to obtain two isolated subtasks to be executed. Based on the two isolated subtasks to be executed, the data to be processed is updated to obtain the processed data to be processed.

8. The method according to claim 1, characterized in that, Before verifying the data version identifiers of the respective nodes, the method further includes: For any of the aforementioned nodes, The physical clock information and logical clock information of each of the multiple slices of the data to be processed are obtained using a hybrid logical clock. Based on the physical clock information and logical clock information of each of the multiple slice data, generate the timestamp information of each of the multiple slice data; Based on the timestamp information of each of the multiple slice data, generate data version identifiers for each of the multiple slice data; The data version identifier of the node is determined based on the data version identifiers of each of the multiple slice data. Specifically, when at least one of the physical clock information and the logical clock information changes, the timestamp information is updated accordingly to update the data version identifier.

9. A data processing apparatus, characterized in that, The device includes: The first determining module is used to determine multiple nodes corresponding to the task to be executed, wherein each of the multiple nodes is associated with data to be processed, and the node executes the task to be executed by processing the data to be processed; The verification module is used to verify the data version identifiers of the multiple nodes to obtain the target verification result. The data version identifiers represent the data update time attribute of the data to be processed. The first acquisition module is used to acquire, when the target verification result indicates that the data version identifier of the target node does not meet the preset update requirement condition, a target Merkle tree topology for characterizing multiple target slice data in the target data to be processed, and a benchmark Merkle tree topology for characterizing multiple benchmark slice data in the benchmark data to be processed, wherein the target data to be processed is related to the target node, and the data version identifier related to the benchmark data to be processed meets the preset update requirement condition. The second determining module is used to determine the slice data to be updated from the plurality of target slice data based on the comparison results of the baseline Merkle tree topology and the target Merkle tree topology; and The synchronization module is used to synchronize the slice data to be updated in the target data to be processed with the reference slice data to obtain the target slice data for executing the task to be executed.

10. An electronic device, comprising: One or more processors; Memory, used to store one or more computer programs. The characteristic feature is that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 8.

11. A computer-readable storage medium having a computer program or instructions stored thereon, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 8.

12. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 8.

Images