Data processing method, apparatus, system, device, and readable storage medium

Abstract

The application provides a data processing method, device, system, equipment and readable storage medium, relates to the technical field of block chain, and the method comprises the following steps: when the data stored by a first node is behind the data stored by a second node, obtaining the difference value between the latest batch of the data stored by the first node and the latest batch of the data stored by the second node, obtaining recovery data according to the difference value, and recovering the data stored by the first node according to the recovery data so that the data stored by the first node is the same as the data stored by the second node. In the technical scheme, different recovery data is obtained through the difference value, the data of the first node is recovered, in the scene that there are various different data batch spans between the first node and the second node, the data recovery can be flexibly and effectively realized, the first node can be ensured to be timely and quickly consistent with the data stored by the second node, and the stability of the distributed system is improved.

Description

Technical Field

[0001] This application relates to the field of blockchain technology, and in particular to a data processing method, apparatus, system, device, and readable storage medium. Background Technology

[0002] Distributed systems typically have multiple replica nodes, which communicate with each other to maintain data consistency and improve system availability. In practical applications, some replica nodes may become invalid due to downtime or anomalies, leading to data inconsistencies across replica nodes. In such cases, it is necessary to promptly restore the data of these invalid nodes to prevent the number of invalid nodes from increasing and causing system downtime.

[0003] In existing technologies, when recovering data from a failed node, all data from the non-failed nodes is typically copied to the failed node to achieve data recovery.

[0004] However, existing data recovery methods often contain data that has already been stored on the failed node, which is present in all the data of the unfailed node. When copying this data to the failed node, data overlap occurs, affecting data recovery efficiency and making the data recovery timeliness poor, thus reducing the stability of the distributed system. Summary of the Invention

[0005] This application provides a data processing method, apparatus, system, device, and readable storage medium to address the problem of low data recovery efficiency in existing distributed systems.

[0006] In a first aspect, embodiments of this application provide a data processing method applied to a first node of a distributed system, the distributed system comprising multiple nodes, the method comprising:

[0007] The system detects whether the data stored in the first node lags behind the data stored in the second node, where the second node is the node in the distributed system that stores complete data.

[0008] When the data stored in the first node lags behind the data stored in the second node, obtain the difference between the latest batch of data stored in the first node and the latest batch of data stored in the second node.

[0009] Based on the difference, recovered data is obtained, wherein the recovered data is new data or the recovered data is a combination of new data and lagging data, the lagging data is the data stored by the second node before the current height, the new data is the data stored by the second node at the current height, and the second node can store multiple batches of data at each height;

[0010] Based on the recovered data, the data stored in the first node is recovered so that the data stored in the first node is the same as that stored in the second node.

[0011] In one possible design of the first aspect, detecting whether the data stored by the first node lags behind the data stored by the second node includes:

[0012] Determine whether the latest batch of data stored in the second node is the same as the latest batch of data stored in the first node;

[0013] If the latest batch of data stored by the second node is different from the latest batch of data stored by the first node, then it is determined that the data stored by the first node is behind the data stored by the second node.

[0014] In another possible design of the first aspect, the step of detecting whether the data stored by the first node lags behind the data stored by the second node further includes:

[0015] When the first node starts up, it sends a data request to each node of the distributed system. The data request is used to request the latest batch of data stored by each node.

[0016] Receive the latest batch of data fed back from each of the nodes;

[0017] Based on the latest batch of data fed back by each node, the second node is determined from among the nodes, and the second node is the node with the largest latest batch of data.

[0018] In another possible design of the first aspect, obtaining the recovered data based on the difference includes:

[0019] When the difference is greater than a preset batch span threshold, the new data and the lagging data are acquired.

[0020] When the difference is less than or equal to the batch span threshold, the new data is obtained.

[0021] In another possible design of the first aspect, when the number of second nodes is multiple, the step of acquiring the new data and the lagging data when the difference is greater than a preset batch span threshold includes:

[0022] When the difference is greater than the preset batch span threshold, the target node is selected from each of the second nodes;

[0023] Obtain the lagging data from the target node;

[0024] The new data is obtained from each of the second nodes.

[0025] In another possible design of the first aspect, when the recovered data consists of new data and lagging data, the step of recovering the data stored in the first node based on the recovered data to make the data stored in the first node identical to that stored in the second node includes:

[0026] Based on the lagging data, perform the first data recovery on the data stored in the first node;

[0027] After the first data recovery is completed, a second data recovery is performed on the first node based on the new data, so that the data stored in the first node is the same as that stored in the second node.

[0028] In another possible design of the first aspect, the first data recovery of the data stored in the first node based on the lagging data includes:

[0029] Obtain the batch of each sub-data in the lagging data;

[0030] Based on the batch order of each sub-data in the lagging data, each sub-data is copied to the first node in sequence.

[0031] In another possible design of the first aspect, if the recovered data is new data, then the step of restoring the data stored in the first node based on the recovered data to make the data stored in the first node identical to that stored in the second node includes:

[0032] Obtain the target data from the new data, wherein the batch of the target data is the same as the latest batch of data stored in the first node;

[0033] Insert the data of batches in the new data that are larger than the target data after the latest batch of data in the first node, so that the data stored in the first node and the second node are the same.

[0034] In another possible design of the first aspect, detecting whether the data stored by the first node lags behind the data stored by the second node includes:

[0035] While the first node is running, obtain the batch of data currently received by the first node and the latest batch of stored data;

[0036] Determine whether the batch of data currently received is consecutive to the latest batch of stored data;

[0037] If the batch of data currently received is not consecutive with the latest batch of stored data, then it is determined that the data stored by the first node is behind the data stored by the second node.

[0038] In another possible design of the first aspect, restoring the data stored in the first node based on the restored data to make the data stored in the first node identical to that stored in the second node includes:

[0039] Obtain the batch difference between the batch of data currently received by the first node and the latest batch of stored data;

[0040] When the batch difference exceeds a preset value, the data stored in the first node is restored according to the restored data so that the data stored in the first node is the same as that stored in the second node.

[0041] In yet another possible design of the first aspect, the method further includes:

[0042] After restoring the data stored in the first node, the latest batch of data stored in the first node is updated according to the batch of new data.

[0043] Secondly, embodiments of this application provide a data processing apparatus, including:

[0044] The detection module is used to detect whether the data stored in the first node is behind the data stored in the second node, where the second node is a node in the distributed system that stores complete data.

[0045] The difference acquisition module is used to acquire the difference between the latest batch of data stored in the first node and the latest batch of data stored in the second node when the data stored in the first node lags behind the data stored in the second node.

[0046] The data acquisition module is used to acquire recovered data based on the difference, wherein the recovered data is new data or the recovered data is a combination of new data and lagging data, the lagging data is data stored by the second node before the current height, the new data is data stored by the second node at the current height, and the second node can store multiple batches of data at each height;

[0047] The data recovery module is used to recover the data stored in the first node based on the recovered data, so that the data stored in the first node is the same as that stored in the second node.

[0048] Thirdly, embodiments of this application provide a distributed system including multiple nodes, wherein the nodes communicate with each other;

[0049] When the data stored at the first node lags behind the data stored at the second node, the first node will acquire and restore the data so that the data at the first node is the same as the data at the second node. The restored data is either new data or a combination of new data and lagging data. The lagging data is the data stored by the second node before the current altitude, and the new data is the data stored by the second node at the current altitude. The second node can store multiple batches of data at each altitude.

[0050] Fourthly, an electronic device includes: a processor, and a memory communicatively connected to the processor;

[0051] The memory stores computer-executed instructions;

[0052] The processor executes computer execution instructions stored in the memory to implement the above-described method.

[0053] Fifthly, embodiments of this application provide a readable storage medium storing computer instructions, which, when stored by a processor, are used to implement the above-described method.

[0054] Sixthly, embodiments of this application provide a program product including computer instructions that, when stored by a processor, implement the above-described method.

[0055] The data processing method, apparatus, system, device, and readable storage medium provided in this application divide the data to be recovered by the first node into new data and lagging data. Then, based on the batch difference of the data, new data or a combination of new and lagging data is selected to recover the data of the first node. In scenarios where there are various different data batch spans between the first node and the second node, data recovery can be flexibly and effectively achieved, ensuring that the first node can keep the data stored by the second node consistent with the data stored by the second node in a timely and fast manner, thereby improving the stability of the distributed system. Attached Figure Description

[0056] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application;

[0057] Figure 1 A schematic diagram of a scenario for the data processing method provided in the embodiments of this application;

[0058] Figure 2a A schematic diagram of the replica node state machine cache difference embodiment provided in this application;

[0059] Figure 2b A schematic diagram of the replica node state machine cache difference embodiment two provided in this application;

[0060] Figure 3 A flowchart illustrating an embodiment of the data processing method provided in this application;

[0061] Figure 4 A flowchart illustrating Embodiment 2 of the data processing method provided in this application;

[0062] Figure 5 A flowchart illustrating Embodiment 3 of the data processing method provided in this application;

[0063] Figure 6 This is a schematic diagram of the structure of the data processing apparatus provided in the embodiments of this application;

[0064] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0065] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0066] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0067] First, let me explain the terms used in this application:

[0068] Distributed systems are software systems built on a network. They typically achieve fault tolerance by maintaining multiple replica nodes. At any given time, all replica nodes have the same result for a certain value stored in them, that is, they maintain consistency in their shared storage, thereby improving the availability of the system.

[0069] Internal events:

[0070] An internal event (CheckPoint) can be viewed as the state of the blockchain at a certain height, where the height is determined by a configuration file, for example, N*1000 (N is a positive integer). This state records the execution batch of ledger block hashes and state machine data at the corresponding height.

[0071] State machine cache: In a distributed system, each replica node records all messages of the largest batch of state machines executed since the last checkpoint up to the current moment in its own state machine cache. The purpose is to provide state machine replication for other lagging replica nodes.

[0072] Figure 1 This is a schematic diagram of a scenario for the data processing method provided in the embodiments of this application, such as... Figure 1 As shown, a distributed system typically has multiple replica nodes, such as node 11, node 12, and node 13, which communicate with each other. To ensure the stability and availability of the distributed system, it is necessary to maintain consistency among the replica nodes. In a cluster with consistent nodes, all replica nodes have the same result for a certain value stored therein at any given time, that is, they maintain consistency with their shared storage. The cluster has the property of automatic recovery; when a few replica nodes fail, it will not affect the normal operation of the entire cluster. Therefore, it is important that a small number of replica nodes can recover quickly and correctly after failure or anomaly, so as to prevent the failure of a small number of nodes from evolving into a majority and causing service outage.

[0073] In relevant blockchain technologies, both ledger data synchronization and state machine replication can achieve state recovery of failed replicas. Nodes that have crashed or fallen behind due to anomalies can catch up on their state data through state machine replication. However, when the difference in state machine cache between the lagging replica and the complete replica crosses the checkpoint, the data catch-up process is imperfect. Figure 2a This is a schematic diagram of an embodiment of the replica node state machine cache difference provided in this application. Figure 2a As shown, in a non-checkpoint scenario, the state machine cache of the lagging replica node and the state machine cache of other replica nodes are at the same height. Figure 2b This is a schematic diagram of the second embodiment of the replica node state machine cache difference provided in this application. Figure 2b As shown, in a cross-checkpoint scenario, the state machine cache of the lagging replica node and the state machine cache of other replica nodes are at different heights.

[0074] To address the aforementioned issues, the data processing method, apparatus, system, device, and readable storage medium provided in this application divide the data to be recovered by the first node into new data and lagging data. Then, based on the batch difference of the data, it selects either new data or a combination of new and lagging data to recover the data from the first node. This ensures the correctness of data replication when the lagging node is in a cross-checkpoint scenario during startup. Furthermore, it automatically detects data lag during the operation of the lagging node and triggers the state machine to correctly replicate the data for synchronization, ensuring timely data recovery and avoiding large-scale data recovery at the same time. Finally, during the data recovery process, the distributed system can continue to operate normally and execute business logic, ensuring the stability of the distributed system.

[0075] The technical solution of this application will now be described in detail through specific embodiments. It should be noted that the following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0076] Figure 3 This is a flowchart illustrating an embodiment of the data processing method provided in this application. The method can be applied to a distributed system, which includes at least two nodes. During the operation of the distributed system, one or more nodes may experience a period of downtime followed by a restart, or packet loss due to network issues during operation, causing the node's state machine cache to lag behind other nodes. Such a node will be referred to as the first node in the following text, while the node that did not experience packet loss and whose cached data is complete will be referred to as the second node in the following text. For example... Figure 3 As shown, the method may specifically include the following steps:

[0077] S301. Detect whether the data stored in the first node is behind the data stored in the second node.

[0078] In this system, the second node is the node in the distributed system that stores complete data. For example, the second node can be a node in the distributed system that has not experienced any failures and can normally receive and store data, while the first node is a node that has experienced failures, such as network problems causing packet loss, system crashes and restarts, etc. In this case, the first node cannot normally receive and store data, causing the data stored by the first node to lag behind that of the second node.

[0079] In this embodiment, each node possesses the topology of the entire blockchain consensus network and can communicate with other nodes. The data stored by a node can be simply understood as all messages from the largest batch of state machines executed since the last checkpoint. The batch of data stored by a node refers to the batch of executed state machines.

[0080] In this embodiment, the first node can be in a running state or a started state. In either state, the data stored by the first node may lag behind that of the second node. It is understood that consistency among the nodes of a distributed system is essential to ensure system availability; that is, the data stored between nodes needs to be synchronized. When the first node fails (i.e., network problems causing packet loss or system crashes and restarts), it will be unable to receive data normally, while other non-faulty nodes can still receive data normally. This results in the first node's data lagging behind that of the second node.

[0081] For example, if packet loss occurs for a long period of time due to network problems during the operation of the first node, the first node will not receive data during this period, while the second node will still be able to receive data. This period of time may cause the data stored by the first node to lag behind the data stored by the second node.

[0082] S302. When the data stored in the first node lags behind the data stored in the second node, obtain the difference between the latest batch of data stored in the first node and the latest batch of data stored in the second node.

[0083] For example, the batches of data received by nodes in a distributed system may differ in different time periods. For instance, in the first time period, the first node and the second node in the distributed system receive and store the first batch of data. In the second time period, the second node in the distributed system receives and stores the second batch of data. However, in the second time period, the first node may fail and be unable to receive the second batch of data, resulting in the stored data lagging behind that of the second node.

[0084] In this embodiment, each checkpoint can be viewed as the state of the blockchain at a certain height, where the height is determined by a preset configuration file. For example, the height can be configured to 1000, then the checkpoint can be viewed as the state at a height of N*1000. This state records the execution batch of ledger block hashes and state machine data at the corresponding height.

[0085] In this distributed system, each node stores in its cache all messages from the largest state machine batch executed since the last checkpoint. (See above.) Figure 2aThe first node records in its cache that the largest batch of state machines executed so far is 5, which means the latest batch of stored data is 5. The second node records in its cache that the largest batch of state machines executed so far is 997. At this time, the latest batch of data stored by the first node is behind the latest batch of data stored by the second node, and the difference does not exceed the height of 1000 (that is, the lag does not cross the checkpoint and is still within the same checkpoint).

[0086] Continue to refer to Figure 2b In this context, the first node records the latest batch of the largest state machine data executed so far as 5 in its cache, while the second node records the latest batch of the largest state machine data executed so far as 1997 in its cache. At this point, the second node stores all messages of the largest state machine batch executed since the first node's checkpoint. That is, the difference between the data batch stored by the second node and the data batch stored by the first node has exceeded 1000, meaning that the data lag has crossed a checkpoint.

[0087] S303. Obtain the recovered data based on the difference.

[0088] The recovered data can be either new data or a combination of new and outdated data. Outdated data refers to data stored by the second node before the current altitude, while new data refers to data stored by the second node at the current altitude. The second node can store multiple batches of data at each altitude.

[0089] In this embodiment, the current height can be determined based on the latest batch of data stored at each node. For example, continue to refer to the above. Figure 2a ,exist Figure 2a In the first node, the latest batch of data stored is 5, and the current height of the first node is 1*1000. The latest batch of data stored in the second node is 997, and the current height of the second node is also 1*1000. Figure 2b In the example, the first node is currently at a height of 1*1000, the second node stores data up to batch 1997, and the second node is currently at a height of 2*1000. For example, at each height, each node can store 1000 batches of data.

[0090] As a further example, a one-to-one correspondence can exist, where when the difference is less than 1000, the corresponding recovered data is new data; when the difference is greater than or equal to 1000, the corresponding recovered data includes both new and lagging data. (Continue referring to the above...) Figure 2a ,exist Figure 2aIf the difference is less than 1000, the corresponding recovery data is new data. The new data is the data stored by the second node at the current height, which is 1*1000. This means that the data stored at the current height includes at least all the data from batch 5 to batch 997. All of this data needs to be copied to the first node so that the first node can catch up with the data of the second node, in order to keep the data stored by the first node and the second node consistent and ensure the availability of the entire distributed system.

[0091] Continue to refer to the above Figure 2b ,exist Figure 2b In the middle, if the difference is greater than 1000, the corresponding recovery data consists of new data and lagging data. The second node is at a current height of 2*1000. The data stored before the current height (i.e., lagging data) includes at least all data from batch 5 to batch 1000. The new data includes the data stored at the current height, i.e., all data from batch 1001 to batch 1997.

[0092] S304. Based on the recovered data, restore the data stored in the first node so that the data stored in the first node is the same as that stored in the second node.

[0093] In this embodiment, when restoring data stored in the first node, it can be first determined whether the data to be restored is new data or a mix of new and outdated data. If the data to be restored is new data, it can be directly copied to the end of the data currently stored in the first node. For example, continue referring to the above... Figure 2a The first node currently stores data batch 5, and the new data is the data stored by the second node at the current height (i.e., data from batch 1 to batch 997). The new data can be directly copied to the first node and overwrite the data currently stored by the first node (i.e., data from batch 1 to batch 5). This makes the data on the first node the same as the data stored on the second node.

[0094] For example, if the new data is the data stored by the second node at the current height (i.e., data from batch 1 to batch 997), it is possible to first delete the duplicate data in the new data (i.e., data from batch 1 to batch 997) that is the same as the data currently stored by the first node, based on the batch of the data currently stored by the first node (i.e., data from batch 1 to batch 5). Then, the new data after deleting the duplicate data (i.e., data from batch 6 to batch 997) is inserted after the data currently stored by the first node. In this way, the data of the first node can also be the same as the data stored by the second node.

[0095] This application embodiment divides the data to be recovered by the first node into new data and lagging data. Then, based on the batch difference of the data, it selects new data or new data and lagging data to recover the data of the first node. In scenarios where there are various different data batch spans between the first node and the second node, data recovery can be flexibly and effectively realized, ensuring that the first node can keep the data stored with the second node in a timely and fast manner, thereby improving the stability of the distributed system.

[0096] In some embodiments, step S301 above can be implemented through the following steps:

[0097] Determine whether the latest batch of data stored on the second node is the same as the latest batch of data stored on the first node;

[0098] If the latest batch of data stored on the second node is different from the latest batch of data stored on the first node, then it is determined that the data stored on the first node is behind the data stored on the second node.

[0099] In this embodiment, when the detected data is lagging, two or more nodes can be randomly selected from the distributed system, and then the latest batch of data currently stored at these two nodes can be compared to determine whether there are any nodes with lagging data.

[0100] For example, continue to refer to the above. Figure 2a The latest batch of data stored on the first node is 5, while the latest batch of data stored on the second node is 997. Therefore, it can be determined that the data on the first node is behind the data on the second node.

[0101] This application embodiment compares the latest batch of data stored in the second node with the latest batch of data stored in the first node to detect whether there is a data lag when the first node starts up, and triggers the data recovery action of the first node in a timely manner, so as to avoid a large number of data lag nodes in the distributed system, which would cause the distributed system service to stop.

[0102] Furthermore, based on the above embodiments, in some embodiments, the above data processing method may further include the following steps:

[0103] When the first node starts up, it sends data requests to all nodes in the distributed system.

[0104] Receive the latest batch of data from each node;

[0105] Based on the latest batch of data reported by each node, the second node is determined from among the nodes.

[0106] The data request is used to request the latest batch of data stored on each node, and the second node is the node with the largest latest batch of data.

[0107] In this embodiment, when a faulty node in the distributed system (i.e., the first responder) restarts (e.g., after a system crash), it sends a data request to other nodes in the distributed system. Nodes responding to the data request will provide the latest batch of data from their currently executing state machines. Among the responding nodes, there may also be faulty nodes (i.e., nodes with outdated data). In this case, the first node needs to compare the latest batches provided by these responding nodes to see if they are consistent. If there are inconsistencies, the node with the outdated latest batch is the faulty node, while the others that are consistent are the second nodes (i.e., non-faulty nodes, whose stored data is complete).

[0108] It should be noted that in the actual processing, after the first node sends a data request to other nodes, it needs to determine whether it has received the latest batch from f+1 nodes, where f represents the number of faulty nodes. This ensures that at least one of the responding nodes is a non-faulty node (i.e., the second node), and that the latest batch from the non-faulty node is the largest. Subsequently, the lagging data and new data can be obtained from the non-faulty node.

[0109] This application embodiment identifies a second node with complete data from among the nodes in the distributed system. During subsequent data recovery, interference from other faulty nodes can be avoided, erroneous recovery data can be obtained from other faulty nodes, and the correctness of the data recovery of the first node can be guaranteed.

[0110] In some embodiments, step S203 above can be implemented through the following steps:

[0111] When the difference exceeds the preset batch span threshold, new data and lagging data are acquired.

[0112] New data is retrieved when the difference is less than or equal to the batch span threshold.

[0113] In this embodiment, the batch span threshold can be used to indicate whether the lag between the data stored by the first node and the data stored by the second node has crossed a checkpoint. For example, if a checkpoint refers to the state of the blockchain at height N*1000, then the preset span threshold is 1000. When a checkpoint is crossed, the recovered data consists of both new and lagging data; when a checkpoint is not crossed, the recovered data consists of new data.

[0114] For example, if the current height of the first node is 2*1000 and the current height of the second node is 3*1000, then the data stored in the first node is all the messages of the largest state machine batch executed since the first checkpoint, and the data stored in the second node is all the messages of the largest state machine batch executed since the second checkpoint. That is, the first node and the second node are across checkpoints.

[0115] Specifically, the current height of the first node can be determined based on the latest batch of data stored in the first node, and the current height of the second node can be determined based on the latest batch of data stored in the second node.

[0116] This application provides a data recovery implementation method in both cross-checkpoint and non-cross-checkpoint scenarios by setting a preset span threshold. When the difference span exceeds the preset span threshold, it is determined that the data of the current first node is lagging behind the checkpoint, and both new data and lagging data need to be obtained simultaneously. When the value span does not exceed the preset span threshold, it is determined that the data of the current first node is lagging behind without crossing the checkpoint, and only new data needs to be obtained to restore the data of the first node.

[0117] Furthermore, based on the above embodiments, in some embodiments, when the number of second nodes is multiple, the above step "when the difference is greater than the preset batch span threshold, acquire new data and lagging data" can be specifically implemented through the following steps:

[0118] When the difference is greater than the preset batch span threshold, the target node is selected from each of the second nodes;

[0119] Obtain lagging data from the target node;

[0120] Retrieve new data from each second node.

[0121] In this embodiment, lagging data can be obtained from any of the second nodes. For example, when obtaining new data, a new data request needs to be sent to each of the second nodes. After a second node responds to the new data request sent by the first node, it can respond to the new data request and feed back the data stored at the current height to the first node.

[0122] For example, the first node can first compare whether the data stored at the current height reported by each second node is consistent. If they are consistent, it is determined that the new data reported is correct. If they are inconsistent, it means that there may be a faulty node among the second nodes. At this time, the new data reported by the faulty node can be removed, and the new data reported by other non-faulty second nodes can be retained to ensure that the new data will not be erroneous.

[0123] This application embodiment obtains new data from each second node and compares whether the new data of each second node is consistent, which can ensure the accuracy of the new data and the correctness of the data recovery of the first node.

[0124] In some embodiments, when the recovered data consists of both new and outdated data, step S304 can be implemented through the following steps:

[0125] The first data recovery is performed on the data stored in the first node based on the lagging data;

[0126] After the first data recovery is completed, a second data recovery is performed on the first node based on the new data to ensure that the data stored on the first node is the same as that stored on the second node.

[0127] In this embodiment, if the current height of the second node is 2*1000, the lagging data is the data stored before the current height, which may include data from batch 1 to batch 1000. This data can be directly copied to the first node to complete the first data recovery.

[0128] The new data is the data cached by the second node at the current height. For example, if the latest batch of data currently stored by the second node is 1997, then the new data is the data from batch 1001 to batch 1997. The new data will be inserted into the first node based on the first data recovery, so that the data stored by the first node is batch 1 to batch 1997, which is the same as the data currently stored by the second node.

[0129] This application embodiment uses new data and lagging data as recovery data. When performing data recovery on the first node, the data recovery of the first node is first performed based on the lagging data, and then the data recovery of the first node is performed a second time based on the new data. This ensures the correctness of the data recovery process and prevents data recovery chaos.

[0130] Furthermore, in some embodiments, the above step "performing the first data recovery based on the lagging data" can be specifically implemented through the following steps:

[0131] Retrieve batches of individual sub-data within the lagging data;

[0132] Based on the batch order of each sub-data in the lagging data, each sub-data is copied to the first node in sequence.

[0133] In this embodiment, the lagging data includes various sub-data, and each sub-data also has a different batch. For example, the lagging data is the data stored by the second node before the current height 2*1000, and it includes 1000 sub-data, which are batches 1 to 1000. When copying these data to the first node, they need to be copied sequentially to avoid batch confusion. For example, the data of batch 1000 is copied to the first node first, and the data of batch 1 is copied to the first node last.

[0134] This application embodiment sorts the batches of each sub-data and copies them sequentially to the first node, preventing confusion in the order of the sub-data during the copying process and improving the accuracy of data recovery.

[0135] In some embodiments, if the recovered data is new data, then step S304 can be implemented through the following steps:

[0136] Retrieve the target data from the new data;

[0137] Insert the data of batches in the new data that are larger than the target data after the latest batch of data in the first node, so that the data stored in the first node and the second node are the same.

[0138] The target data batch is the same as the latest batch of data stored in the first node.

[0139] For example, if the new data includes data from batch 1 to batch 997, and the latest batch of data currently stored on the first node is 5, then the target data is the data from batch 5. In this case, the data from batch 6 to batch 997 in the new data will be inserted after the latest batch of data currently stored on the first node (i.e., the data from batch 5), ultimately making the data stored on the first node the same as the data stored on the second node.

[0140] This application embodiment inserts data from batches of new data that are larger than the target data after the latest batch of data in the first node. This eliminates the need to copy the entire new data to the first node, reducing the amount of data recovery, improving data recovery efficiency, and lowering the risk of a large number of lagging data nodes in the distributed system.

[0141] In some embodiments, step S3201 above can also be implemented through the following steps:

[0142] When the first node is running, obtain the batch of data currently received by the first node and the latest batch of stored data;

[0143] Determine whether the batch of data currently received is consecutive to the latest batch of stored data;

[0144] If the batch of data currently received is not consecutive with the latest batch of stored data, then it is determined that the data stored by the first node is behind the data stored by the second node.

[0145] In this embodiment, the first node may also experience data lag during operation. For example, when network problems cause packet loss, even if the first node continues to run, it may not receive a certain batch of data. Thus, after the network is restored, the batch of data currently received by the first node may not be continuous with the latest batch of data stored.

[0146] For example, the first node stores data batch 5 during operation. After storing batch 5, a network problem causes packet loss, and it may not receive data batch 6. When the network is restored, it receives data batch 7. At this point, it can be determined that the data stored by the first node is behind the data stored by the second node.

[0147] This application embodiment determines whether the batch of currently received data is consecutive to the latest batch of stored data. This enables the detection of data lag in the first node during its operation, allowing for timely discovery of lagging nodes and timely data recovery. This avoids a large number of lagging nodes in the distributed system and ensures the availability of the distributed system.

[0148] Furthermore, based on the above embodiments, in some embodiments, step S304 can be implemented through the following steps:

[0149] Get the batch difference between the batch of data currently received by the first node and the latest batch of stored data;

[0150] When the batch difference exceeds the preset value, the data stored in the first node is restored based on the restored data to make the data stored in the first node the same as that stored in the second node.

[0151] In this embodiment, a trigger condition for data recovery can be set. The data recovery action of the first node is triggered only when the batch of data currently received by the first node exceeds a preset difference from the latest batch of stored data.

[0152] Specifically, the preset difference can be 5. For example, after storing data in batch 5, if the first node experiences a network problem that causes packet loss, it may not receive data in batch 6. When the network is restored, it receives data in batch 7. At this time, the batch difference is 7-5=2, which does not exceed the preset difference. Therefore, the first node will not perform data recovery.

[0153] This application embodiment can automatically recover data from lagging nodes by setting data recovery trigger conditions, thereby minimizing the number of lagging nodes, preventing a large number of lagging nodes from appearing in the distributed system, and ensuring the stable and normal operation of the distributed system.

[0154] In some embodiments, the above method may further include the following steps:

[0155] After restoring the data stored in the first node, the batch of data stored in the first node is updated according to the batch of new data.

[0156] In this embodiment, after the first node recovers the data, the latest batch of data stored by the first node should be the same as the latest batch of data stored by the second node, that is, an update is required at this time.

[0157] Figure 4 This is a flowchart illustrating a second embodiment of the data processing method provided in this application. This method can be applied to situations where a node's stored data lags behind during startup and needs to be recovered. Figure 4 As shown, the method may specifically include the following steps:

[0158] S401, the node starts up.

[0159] S402, Consensus module started.

[0160] S403, Status transmission started.

[0161] When a node starts up, it first activates the consensus module within the node to establish a peer-to-peer network for message exchange between consensus nodes during the state machine replication process, thereby enabling data communication with other nodes.

[0162] S404, Query the latest block height from other nodes.

[0163] In this embodiment, the block height refers to the current height of other nodes, which can be N*1000, where N is a positive integer. That is, the block height can be 1*1000, 2*1000...N*1000.

[0164] S405. Determine whether the target number of block height responses have been received.

[0165] In this embodiment, the target number can be f+1, where f represents the number of faulty nodes. The block heights responded by faulty nodes are incorrect; only the block heights responded by non-faulty nodes are correct. When a non-faulty node reports a block height to the first node, if the block heights reported by other nodes are inconsistent with those reported by the non-faulty node, it indicates that the other nodes are faulty. The first node needs to continue collecting block heights reported by other non-faulty nodes to ensure that the first node receives the correct block height.

[0166] S406. Determine if it crosses the block height.

[0167] In this embodiment, the block height reported by other nodes can be 2*1000. If the current block height of the first node is also 2*1000, then the block height is not crossed. If the current block height of the first node is 1*1000, then the block height is crossed.

[0168] S4071, Request new data from a reliable node.

[0169] S4072. Determine whether consistent new data has been received from the target number of nodes.

[0170] S4073, Replay new data.

[0171] S4074, Startup complete.

[0172] In this embodiment, a reliable node is the non-faulty node determined in the above steps. When the block height is not crossed, new data is directly requested from each non-faulty node, where the new data is the data stored by each non-faulty node at the current height. When new data is received from all non-faulty nodes and the new data is consistent, the new data is copied to the first node to complete data replay, and the first node starts up.

[0173] S4075. Determine whether the number of requests for new data exceeds the specified limit.

[0174] S4076, Startup failed.

[0175] In this embodiment, if no new data is received from all non-faulty nodes, the first node will continuously request new data from each non-faulty node until it receives new data from all non-faulty nodes or the number of requests exceeds a predetermined number.

[0176] S4081. Select any reliable node.

[0177] S4082, Request lagging data from any reliable node.

[0178] S4083, Replay of outdated data.

[0179] S4084. Determine whether all lagging data has been replayed.

[0180] S4085, Has the number of requests for lagging data exceeded the specified number?

[0181] In this embodiment, if the data spans a block height, lagging data needs to be retrieved from any reliable node and then restored to the first node. If the lagging data is not replayed during this process, and the first node requests data from any reliable node more than the specified number of times, the first node will fail to start.

[0182] Furthermore, once the lagging data has been recovered to the first node, new data can be requested to further recover the new data to the first node.

[0183] S409. Determine if the number of queries for the latest block height message has exceeded the limit.

[0184] Figure 5 This is a flowchart illustrating a third embodiment of the data processing method provided in this application. This method can be applied to situations where data stored by a node during operation is outdated and data recovery is required. Figure 5 As shown, the method may include the following steps:

[0185] S501: After the node has been running normally for a period of time, it receives the latest batch of consensus messages.

[0186] In this embodiment, the node continuously receives new consensus messages during operation. These consensus messages are stored as data after the node executes them, thus becoming the data stored by the node.

[0187] S502. Determine whether the latest batch and the largest batch of data stored in the node are consecutive.

[0188] S503. Determine whether the latest batch is smaller than the largest batch.

[0189] S5041, Discard the consensus message.

[0190] S5051, Store the latest batch of consensus messages into the cache.

[0191] In this embodiment, after the latest batch of consensus messages is stored in the cache, if the data recovery condition is subsequently triggered, the latest batch of consensus messages will be retrieved after the node completes the data recovery and will continue to be processed by the node.

[0192] S5052. Determine whether the data recovery conditions have been triggered.

[0193] S5053. Based on the difference between the largest batch and the latest batch, obtain the recovered data and perform data recovery.

[0194] In this embodiment, the condition for triggering data recovery can be set to the difference between the latest batch and the largest batch exceeding a preset difference. For example, the preset difference can be set to 5, the latest batch can be 11, and the largest batch of data stored by the node can be 5. In this case, 11-5=6, which will trigger data recovery.

[0195] After the data recovery conditions are triggered, data recovery will be performed according to the method nodes provided in the above embodiments.

[0196] S506, Process the consensus messages for the new batch.

[0197] S507. Update the largest batch to the latest batch.

[0198] Specifically, when the latest batch and the maximum batch are consecutive, the node is still in normal operation and there is no data lag. At this time, the node will process the consensus message normally. After processing, the consensus message will be stored in the node. The maximum batch of the data stored in the node will be updated to the latest batch of the consensus message.

[0199] The following are embodiments of the apparatus of this application, which can be used to execute the embodiments of the method of this application. For details not disclosed in the embodiments of the apparatus of this application, please refer to the embodiments of the method of this application.

[0200] Figure 6 This is a schematic diagram of the structure of the data processing device provided in an embodiment of this application. This device can be integrated onto the first node, or it can be implemented independently of the first node while cooperating with the first node to achieve this solution. Figure 6 As shown, the data processing device 60 includes a detection module 61, a difference acquisition module 62, a data acquisition module 63, and a data recovery module 64.

[0201] The detection module 61 is used to detect whether the data stored in the first node lags behind the data stored in the second node. The difference acquisition module 62 is used to acquire the difference between the latest batch of data stored in the first node and the latest batch of data stored in the second node when the data stored in the first node lags behind the data stored in the second node. The data acquisition module 63 is used to acquire recovered data based on the difference. The data recovery module 64 is used to recover the data stored in the first node based on the recovered data so that the data stored in the first node is the same as that stored in the second node.

[0202] The second node is a node in the distributed system that stores complete data. The recovered data is either new data or a combination of new data and outdated data. Outdated data is the data stored by the second node before the current height, and new data is the data stored by the second node at the current height. The second node can store multiple batches of data at each height.

[0203] In some embodiments, the detection module described above can be specifically used for:

[0204] Determine whether the latest batch of data stored on the second node is the same as the latest batch of data stored on the first node;

[0205] If the latest batch of data stored on the second node is different from the latest batch of data stored on the first node, then it is determined that the data stored on the first node is behind the data stored on the second node.

[0206] In some embodiments, the data processing apparatus may further include a node determination module, configured to:

[0207] When the first node starts up, it sends data requests to each node in the distributed system. The data requests are used to request the latest batch of data stored on each node.

[0208] Receive the latest batch of data from each node;

[0209] Based on the latest batch of data fed back from each node, the second node is determined from among the nodes. The second node is the node with the largest latest batch of data.

[0210] In some embodiments, the data acquisition module described above can be specifically used for:

[0211] When the difference exceeds the preset batch span threshold, new data and lagging data are acquired.

[0212] New data is retrieved when the difference is less than or equal to the batch span threshold.

[0213] In some embodiments, when there are multiple second nodes, the data acquisition module described above can be specifically used for:

[0214] When the difference is greater than the preset batch span threshold, the target node is selected from each of the second nodes;

[0215] Obtain lagging data from the target node;

[0216] Retrieve new data from each second node.

[0217] In some embodiments, when the data to be recovered consists of both new and outdated data, the data recovery module described above can be specifically used for:

[0218] The first data recovery is performed on the data stored in the first node based on the lagging data;

[0219] After the first data recovery is completed, a second data recovery is performed on the first node based on the new data to ensure that the data stored on the first node is the same as that stored on the second node.

[0220] In some embodiments, the data recovery module described above can be specifically used for:

[0221] Retrieve batches of individual sub-data within the lagging data;

[0222] Based on the batch order of each sub-data in the lagging data, each sub-data is copied to the first node in sequence.

[0223] In some embodiments, if the recovered data is new data, the data recovery module described above can be specifically used for:

[0224] Retrieve the target data from the new data, where the batch of the target data is the same as the latest batch of data stored in the first node;

[0225] Insert the data of batches in the new data that are larger than the target data after the latest batch of data in the first node, so that the data stored in the first node and the second node are the same.

[0226] In some embodiments, the detection module described above can also be used for:

[0227] When the first node is running, obtain the batch of data currently received by the first node and the latest batch of stored data;

[0228] Determine whether the batch of data currently received is consecutive to the latest batch of stored data;

[0229] If the batch of data currently received is not consecutive with the latest batch of stored data, then it is determined that the data stored by the first node is behind the data stored by the second node.

[0230] In some embodiments, the data recovery module described above can also be used for:

[0231] Get the batch difference between the batch of data currently received by the first node and the latest batch of stored data;

[0232] When the batch difference exceeds the preset value, the data stored in the first node is restored based on the restored data to make the data stored in the first node the same as that stored in the second node.

[0233] In some embodiments, the data processing apparatus may further include an update module for:

[0234] After restoring the data stored on the first node, the latest batch of data stored on the first node is updated according to the batch of new data.

[0235] The apparatus provided in this application embodiment can be used to execute the methods in the above embodiments, and its implementation principle and technical effect are similar, so they will not be described again here.

[0236] It should be noted that the division of the various modules in the above device is merely a logical functional division. In actual implementation, they can be fully or partially integrated into a single physical entity, or they can be physically separated. Furthermore, these modules can be implemented entirely in software via processing elements; they can be fully implemented in hardware; or some modules can be implemented in software via processing elements, while others are implemented in hardware. For example, the detection module can be a separate processing element, or it can be integrated into a chip within the device. Alternatively, it can be stored as program code in the device's memory, and its functions can be called and executed by a processing element. The implementation of other modules is similar. Moreover, these modules can be fully or partially integrated together, or they can be implemented independently.

[0237] This application also provides a distributed system, which includes multiple nodes, such as a first node and a second node, wherein the data stored in the first node is later than that in the second node, and the second node is the node in the distributed system that stores complete data.

[0238] When the data stored in the first node lags behind the data stored in the second node, the first node will acquire and restore the data to make the data in the first node the same as the data in the second node. The restored data is either new data or a combination of new data and lagging data. The lagging data is the data stored by the second node before the current height, and the new data is the data stored by the second node at the current height. The second node can store multiple batches of data at each height.

[0239] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 7 As shown, the electronic device 70 includes at least one processor 71, a memory 72, a bus 73, and a communication interface 74. For example, the electronic device 70 can be a local computer device or a cloud server.

[0240] The processor 71, communication interface 74, and memory 72 communicate with each other via bus 73.

[0241] A communication interface is used to communicate with other devices. For example, in this embodiment, the communication interface can also be used to communicate data with other nodes. The communication interface includes a communication interface for data transmission.

[0242] Processor 71 is used to execute computer execution instructions stored in memory 72, specifically performing the relevant steps in the methods described in the above embodiments. The processor may be a central processing unit (CPU). The electronic device includes one or more processors, which may be processors of the same type, such as one or more CPUs; or they may be processors of different types, such as one or more CPUs and one or more ASICs.

[0243] Memory 72 is used to store instructions executed by the computer. The memory may include high-speed RAM, and may also include non-volatile memory, such as at least one disk drive. Furthermore, the memory may also be used to cache data executed by the node, such as executed state machines and batches of state machines.

[0244] This embodiment also provides a readable storage medium storing computer instructions. When at least one processor of the electronic device stores the computer instructions, the electronic device executes the data processing methods provided in the various embodiments described above.

[0245] This embodiment also provides a program product including computer instructions stored in a readable storage medium. At least one processor of an electronic device can read the computer instructions from the readable storage medium, and the processor executes the computer instructions to cause the electronic device to perform the data processing methods provided in the various embodiments described above.

[0246] In this application, "at least one" means one or more, and "more than one" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates an "or" relationship between the preceding and following related objects; in formulas, the character " / " indicates a "division" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0247] It is understood that the various numerical designations used in the embodiments of this application are merely for descriptive convenience and are not intended to limit the scope of the embodiments of this application. In the embodiments of this application, the order of the above-mentioned process numbers does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0248] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A data processing method, characterized in that, The method, applied to the first node of a distributed system comprising multiple nodes, includes: The system detects whether the data stored in the first node lags behind the data stored in the second node, where the second node is a node in the distributed system that stores complete data; wherein, the data is a state machine message. When the data stored in the first node lags behind the data stored in the second node, obtain the difference between the latest batch of data stored in the first node and the latest batch of data stored in the second node. Based on the difference, recovered data is obtained, wherein when the difference is greater than a preset batch span threshold, new data and lagging data are obtained; when the difference is less than or equal to the batch span threshold, the new data is obtained. The lagging data is the data stored by the second node before the current height, and the new data is the data stored by the second node at the current height. The second node can store multiple batches of data at each height. Based on the recovered data, the data stored in the first node is recovered so that the data stored in the first node is the same as that stored in the second node.

2. The method according to claim 1, characterized in that, The step of detecting whether the data stored in the first node lags behind the data stored in the second node includes: Determine whether the latest batch of data stored in the second node is the same as the latest batch of data stored in the first node; If the latest batch of data stored by the second node is different from the latest batch of data stored by the first node, then it is determined that the data stored by the first node is behind the data stored by the second node.

3. The method according to claim 2, characterized in that, The step of detecting whether the data stored in the first node lags behind the data stored in the second node further includes: When the first node starts up, it sends a data request to each node of the distributed system. The data request is used to request the latest batch of data stored by each node. Receive the latest batch of data fed back from each of the nodes; Based on the latest batch of data fed back by each node, the second node is determined from among the nodes, and the second node is the node with the largest latest batch of data.

4. The method according to claim 1, characterized in that, When there are multiple second nodes, the step of acquiring the new data and the lagging data when the difference is greater than a preset batch span threshold includes: When the difference is greater than the preset batch span threshold, the target node is selected from each of the second nodes; Obtain the lagging data from the target node; The new data is obtained from each of the second nodes.

5. The method according to claim 1, characterized in that, When the recovered data consists of both new and outdated data, the step of restoring the data stored in the first node based on the recovered data to make the data stored in the first node identical to that stored in the second node includes: Based on the lagging data, perform the first data recovery on the data stored in the first node; After the first data recovery is completed, a second data recovery is performed on the first node based on the new data, so that the data stored in the first node is the same as that stored in the second node.

6. The method according to claim 5, characterized in that, The first data recovery based on the lagging data includes: Obtain the batch of each sub-data in the lagging data; Based on the batch order of each sub-data in the lagging data, each sub-data is copied to the first node in sequence.

7. The method according to claim 1, characterized in that, If the recovered data is new data, then restoring the data stored in the first node based on the recovered data to make the data stored in the first node the same as that stored in the second node includes: Obtain the target data from the new data, wherein the batch of the target data is the same as the latest batch of data stored in the first node; Insert the data of batches in the new data that are larger than the target data after the latest batch of data in the first node, so that the data stored in the first node and the second node are the same.

8. The method according to claim 1, characterized in that, The step of detecting whether the data stored in the first node lags behind the data stored in the second node includes: While the first node is running, obtain the batch of data currently received by the first node and the latest batch of stored data; Determine whether the batch of data currently received is consecutive to the latest batch of stored data; If the batch of data currently received is not consecutive with the latest batch of stored data, then it is determined that the data stored by the first node is behind the data stored by the second node.

9. The method according to claim 8, characterized in that, The step of restoring the data stored in the first node based on the restored data to make the data stored in the first node identical to that stored in the second node includes: Obtain the batch difference between the batch of data currently received by the first node and the latest batch of stored data; When the batch difference exceeds a preset value, the data stored in the first node is restored according to the restored data so that the data stored in the first node is the same as that stored in the second node.

10. The method according to any one of claims 1-9, characterized in that, The method further includes: After restoring the data stored in the first node, the latest batch of data stored in the first node is updated according to the batch of new data.

11. A data processing apparatus, characterized in that, include: The detection module is used to detect whether the data stored in the first node is behind the data stored in the second node, where the second node is a node in the distributed system that stores complete data; wherein, the data is a state machine message; The difference acquisition module is used to acquire the difference between the latest batch of data stored in the first node and the latest batch of data stored in the second node when the data stored in the first node lags behind the data stored in the second node. The data acquisition module is used to acquire recovered data based on the difference, wherein when the difference is greater than a preset batch span threshold, new data and lagging data are acquired; when the difference is less than or equal to the batch span threshold, the new data is acquired; the lagging data is the data stored by the second node before the current height, and the new data is the data stored by the second node at the current height; the second node can store multiple batches of data at each height. The data recovery module is used to recover the data stored in the first node based on the recovered data, so that the data stored in the first node is the same as that stored in the second node.

12. A distributed system, characterized in that, It includes multiple nodes, and the nodes communicate with each other. When the data stored at the first node lags behind the data stored at the second node, the first node obtains the difference between the latest batch of data stored at the first node and the latest batch of data stored at the second node, obtains recovery data based on the difference, and performs data recovery to make the data of the first node the same as the data of the second node. Here, the data is a state machine message. When the difference is greater than a preset batch span threshold, new data and lagging data are obtained. When the difference is less than or equal to the batch span threshold, the new data is obtained. The lagging data is the data stored by the second node before the current height, and the new data is the data stored by the second node at the current height. The second node can store multiple batches of data at each height.

13. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1-10.

14. A readable storage medium, characterized in that, The readable storage medium stores computer instructions that, when executed by a processor, are used to implement the method as described in any one of claims 1-10.

15. A program product comprising computer instructions, characterized in that, When executed by a processor, the computer instructions implement the method described in any one of claims 1-10.

Images