Data reconciliation method and device and electronic equipment

Abstract

The invention provides a data reconciliation method and device and electronic equipment, and the method comprises the steps: responding to the obtained data provided by database cache equipment, constructing a local Merkle tree, and generating a local signature; pulling the target signature from the database cache device based on the trigger condition, and comparing the target signature with a local signature; in response to inconsistency between the target signature and the local signature, performing hash value comparison on the local Merkle tree and the target Merkle tree, and positioning a difference state; and pulling the difference data matched with the difference state from the database cache device, updating the local data and the local Merkle tree based on the difference data, and updating the local signature. According to the method, the data difference can be rapidly detected through signature comparison, a large amount of redundant calculation is avoided, the difference part can be accurately and rapidly positioned through comparison of the Merkle tree structure, data restoration is carried out, and the verification efficiency is ensured.

Description

Technical Field

[0001] The present application relates to the field of communication technology, and in particular to a data reconciliation method, device and electronic equipment. Background Art

[0002] Traditional distributed systems (systems composed of multiple computers that communicate and collaborate over a network) require periodic data verification to ensure data consistency across multiple nodes. However, as the amount of system data grows, full reconciliation becomes very costly and inefficient.

[0003] At present, in the face of the problems existing in full reconciliation, the following two solutions are provided:

[0004] 1. Using digital signature technology, you can quickly verify whether the data on both sides are consistent.

[0005] 2. Use hash pointers to store data in a chain, which can quickly determine whether the data has been tampered with.

[0006] However, traditional digital signature technology has the problem of low verification efficiency; the method of using hash pointer chain to store data still requires traversing the entire data structure when locating modification details.

[0007] In summary, the existing technology has the problems of low efficiency and weak ability to accurately locate and modify data when performing large-scale distributed data consistency verification. Summary of the invention

[0008] In view of the above problems, embodiments of the present application provide a data reconciliation method, device and electronic device that overcome the above problems or at least partially solve the above problems.

[0009] In a first aspect, an embodiment of the present application provides a data reconciliation method, which is applied to a controller, where the controller is a node controller in a distributed system, including:

[0010] In response to obtaining the data provided by the database cache device, construct a local Merkle tree and generate a local signature;

[0011] Pulling a target signature from the database cache device based on a trigger condition, comparing the target signature with the local signature, the target signature being generated based on a target Merkle tree constructed by the database cache device, the target Merkle tree being constructed based on data pulled from the database, and the target Merkle tree and the target signature being updated in conjunction with updates of the database;

[0012] In response to the target signature being inconsistent with the local signature, comparing the hash values ​​of the local Merkle tree and the target Merkle tree to locate the difference state;

[0013] Pulling difference data matching the difference state from the database cache device, updating local data and the local Merkle tree based on the difference data, and updating the local signature;

[0014] Wherein, multiple node controllers perform data reconciliation between the node controllers based on the interaction with the database cache device.

[0015] In a second aspect, an embodiment of the present application provides a data reconciliation method, which is applied to a database cache device, including:

[0016] In response to a signature acquisition request sent by the controller based on a trigger condition, providing a target signature for the controller, wherein the target signature is generated based on a target Merkle tree constructed by the database cache device, the target Merkle tree is constructed based on data pulled from a database, and the target Merkle tree and the target signature are updated in conjunction with updates of the database;

[0017] Receive a second pull request carrying a difference state sent by the controller, the second pull request is sent by the controller after identifying that the target signature and the local signature are inconsistent, and comparing the hash values ​​of the local Merkle tree and the target Merkle tree to locate the difference state, the local Merkle tree is constructed based on the data provided by the database cache device, and the local signature is generated based on the local Merkle tree;

[0018] In response to the second pull request, providing the controller with difference data matching the difference state, wherein the difference data is used to update local data of the controller, the local Merkle tree, and the local signature;

[0019] The controller is a node controller in a distributed system, and multiple node controllers perform data reconciliation between the node controllers based on interaction with the database cache device.

[0020] In a third aspect, an embodiment of the present application provides a data reconciliation device, which is applied to a controller, where the controller is a node controller in a distributed system, including:

[0021] A first processing module, configured to construct a local Merkle tree and generate a local signature in response to acquiring data provided by the database cache device;

[0022] A pull comparison module, configured to pull a target signature from the database cache device based on a trigger condition, and compare the target signature with the local signature, wherein the target signature is generated based on a target Merkle tree constructed by the database cache device, the target Merkle tree is constructed based on data pulled from the database, and the target Merkle tree and the target signature are updated in linkage based on updates of the database;

[0023] A comparison and positioning module, configured to compare the hash values ​​of the local Merkle tree and the target Merkle tree to locate the difference state in response to the target signature being inconsistent with the local signature;

[0024] A pull update module, configured to pull difference data matching the difference state from the database cache device, update local data and the local Merkle tree based on the difference data, and update the local signature;

[0025] Wherein, multiple node controllers perform data reconciliation between the node controllers based on the interaction with the database cache device.

[0026] In a fourth aspect, an embodiment of the present application provides a data reconciliation device, which is applied to a database cache device, including:

[0027] A first providing module, configured to provide a target signature for the controller in response to a signature acquisition request sent by the controller based on a trigger condition, wherein the target signature is generated based on a target Merkle tree constructed by the database cache device, the target Merkle tree is constructed based on data pulled from a database, and the target Merkle tree and the target signature are updated in linkage based on an update of the database;

[0028] A first receiving module is used to receive a second pull request carrying a difference state sent by the controller, wherein the second pull request is sent after the controller recognizes that the target signature and the local signature are inconsistent, compares the hash values ​​of the local Merkle tree and the target Merkle tree to locate the difference state, and the local Merkle tree is constructed based on the data provided by the database cache device, and the local signature is generated based on the local Merkle tree;

[0029] a second providing module, configured to provide the controller with difference data matching the difference state in response to the second pull request, wherein the difference data is used to update local data of the controller, the local Merkle tree, and the local signature;

[0030] The controller is a node controller in a distributed system, and multiple node controllers perform data reconciliation between the node controllers based on interaction with the database cache device.

[0031] In a fifth aspect, an embodiment of the present application provides an electronic device, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the data reconciliation method described in the first aspect or the second aspect.

[0032] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the data reconciliation method described in the first aspect or the second aspect are implemented.

[0033] The technical solution of the embodiment of the present application is to pull the target signature from the database cache device based on the trigger condition, compare the pulled target signature with the local signature, and when the target signature is inconsistent with the local signature, locate the difference state according to the comparison of the hash value in the Merkle tree, pull the difference data matching the difference state from the database cache device, and update the local data, local Merkle tree and local signature based on the difference data. It can perform data reconciliation between controllers based on the interaction between each controller and the database cache device, realize real-time and high-frequency reconciliation of large-scale data in a distributed environment, provide an efficient solution for data consistency of distributed systems, and improve the reliability, availability and security of the system. And by quickly detecting data differences through signature comparison, a large number of redundant calculations can be avoided, reducing network load and computing resource consumption; by comparing the Merkle tree structure, the difference part can be accurately and quickly located to ensure verification efficiency; by transmitting only necessary data, the amount of data transmission can be reduced, resource occupation can be reduced, and the scalability of the distributed system can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 A schematic diagram showing a data reconciliation method applied to a controller provided in an embodiment of the present application;

[0035] Figure 2 A schematic diagram showing the working process of the controller provided in an embodiment of the present application;

[0036] Figure 3 A schematic diagram showing a data reconciliation method applied to a database cache device provided in an embodiment of the present application;

[0037] Figure 4 A flowchart showing the interaction between the server, the controller and the database cache device provided in an embodiment of the present application;

[0038] Figure 5 A schematic diagram showing a data reconciliation device applied to a controller provided in an embodiment of the present application;

[0039] Figure 6 A schematic diagram showing a data reconciliation device applied to a database cache device provided in an embodiment of the present application;

[0040] Figure 7 A structural block diagram of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0041] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0042] It should be understood that the references to "one embodiment" or "an embodiment" throughout the specification mean that the specific features, structures, or characteristics associated with the embodiment are included in at least one embodiment of the present application. Therefore, the references to "in one embodiment" or "in an embodiment" appearing throughout the specification do not necessarily refer to the same embodiment. In addition, these specific features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0043] In the various embodiments of the present application, it should be understood that the size of the serial numbers of the following processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0044] In response to the problems of low efficiency and weak ability to accurately locate modifications in periodic data consistency checks in large-scale distributed systems, the present application provides an efficient reconciliation method for distributed data based on Merkle trees, which can greatly improve the reconciliation efficiency of large amounts of data in a distributed environment while ensuring security and data integrity, and can quickly detect data inconsistencies and accurately locate changes, thereby helping each node to perform necessary repairs and ensure data consistency.

[0045] The embodiment of the present application provides a data reconciliation method applied to a controller, wherein the controller is a node controller in a distributed system, and multiple node controllers interact with a database cache device, and multiple node controllers perform data reconciliation between node controllers based on the interaction with the database cache device. That is, the database cache device acts as an intermediate medium for interacting with multiple node controllers, and each node controller implements data consistency verification between different node controllers through interaction with the database cache device.

[0046] like Figure 1 As shown, the data reconciliation method applied to the controller includes the following steps:

[0047] Step 101: In response to obtaining data provided by a database cache device, a local Merkle tree is constructed and a local signature is generated.

[0048] After obtaining the data from the database provided by the database cache device, the controller builds a local Merkle tree corresponding to the controller based on the obtained data, and generates a local signature based on the local Merkle tree. The Merkle tree is a binary tree structure that aggregates data to the root node through hash operations, and can efficiently verify whether the data is in the tree. The signature generated based on the Merkle tree can be the hash value of the root node, or it can be generated using a specific algorithm based on the hash value of the root node.

[0049] Step 102: Pull the target signature from the database cache device based on the trigger condition, compare the target signature with the local signature, the target signature is generated based on the target Merkle tree constructed by the database cache device, the target Merkle tree is constructed based on the data pulled from the database, and the target Merkle tree and the target signature are updated in conjunction with the update of the database.

[0050] When the controller detects the trigger condition of data verification, it pulls the target signature from the database cache device and compares the pulled target signature with the local signature. Since the signature can reflect the integrity of the data, by comparing the target signature with the local signature, the consistency of the data on both sides can be verified through the signature.

[0051] Among them, the target signature is generated based on the target Merkle tree constructed by the database cache device, and the target Merkle tree is constructed based on the data pulled from the database. When the database is updated, the target Merkle tree and the target signature are updated in conjunction with the database update to ensure that a target signature that is compatible with the database can be provided.

[0052] Step 103: In response to the target signature being inconsistent with the local signature, the local Merkle tree and the target Merkle tree are compared in hash value to locate the difference status.

[0053] After comparing the target signature with the local signature, if the target signature is inconsistent with the local signature, the controller requests to obtain the tree structure of the target Merkle tree, that is, initiates a request to the database cache device, pulls multiple hash values ​​corresponding to the target Merkle tree, and then compares the hash values ​​of the local Merkle tree and the target Merkle tree. By comparing the hash values ​​in different Merkle trees, different hash values ​​are identified, and then the difference state is located. The difference state is a state determined based on the difference hash value.

[0054] The difference state may be caused by a change in the data in the controller or a change in the data in the database cache device. The data in the database cache device is kept consistent with the data in the database, so the data in the database cache device shall prevail.

[0055] Step 104: Pull the difference data matching the difference state from the database cache device, update the local data and the local Merkle tree based on the difference data, and update the local signature.

[0056] After locating the difference state, the difference data matching the difference state is pulled from the database cache device based on the difference state, and the local data is updated based on the difference data to maintain data consistency; and after obtaining the difference data, the local Merkle tree needs to be updated, and the local signature is updated based on the updated local Merkle tree to achieve the update of data, Merkle tree and signature.

[0057] By using the controller to actively pull signatures and data, the database cache device becomes the party that passively provides signatures and data, making it easier for the controller to perform data verification and pull difference data on demand to maintain the consistency of data on both sides. And because multiple controllers interact with the database cache device, data reconciliation between controllers can be achieved based on the interaction between each controller and the database cache device.

[0058] The above implementation scheme of the present application pulls the target signature from the database cache device based on the trigger condition, compares the pulled target signature with the local signature, and when the target signature is inconsistent with the local signature, locates the difference state based on the comparison of the hash value in the Merkle tree, pulls the difference data matching the difference state from the database cache device, and updates the local data, local Merkle tree and local signature based on the difference data. Based on the interaction between each controller and the database cache device, data reconciliation between controllers can be performed to achieve real-time and high-frequency reconciliation of large-scale data in a distributed environment, providing an efficient solution for data consistency of distributed systems and improving the reliability, availability and security of the system. Rapid detection of data differences through signature comparison can avoid a large number of redundant calculations, reduce network load and computing resource consumption; by comparing the Merkle tree structure, the difference part can be accurately and quickly located to ensure verification efficiency; by transmitting only necessary data, the amount of data transmission can be reduced, resource usage can be reduced, and the scalability of the distributed system can be improved.

[0059] In a specific implementation, the data reconciliation method provided in the embodiment of the present application is applied to a cloud computing scenario. In the cloud computing infrastructure, SDN (Software Defined Networking) controllers are distributed across various devices, and the controllers need to perform periodic data consistency reconciliation to ensure the uniformity of the network status. The data reconciliation method provided in the embodiment of the present application can achieve efficient reconciliation between massive controllers in a cloud computing environment, which not only ensures reconciliation efficiency but also ensures status consistency; at the same time, it can accurately locate the difference status and synchronize only the necessary parts, reducing the synchronization cost and adapting to the dynamic needs of cloud computing.

[0060] The following describes the process of the controller constructing a local Merkle tree and generating a local signature. Optionally, in response to obtaining data provided by the database cache device, constructing a local Merkle tree and generating a local signature include:

[0061] In response to a first received notification message indicating a database update, sending a first pull request indicating pulling data to a database cache device;

[0062] Receiving data from the database sent by the database cache device;

[0063] A local Merkle tree is constructed based on the received data, and a local signature is generated based on the local Merkle tree.

[0064] In the embodiment of the present application, when data is updated in the database, the database cache device and the controller will both receive a notification message, and the database cache device pulls data from the database based on the notification message to update the local data, and the controller pulls data from the database cache device based on the notification message to update the local data. Since it is necessary to ensure that the database cache device has obtained the latest data when the controller pulls data from the database cache device, the notification message sender usually first sends a notification message to the database cache device, and after a certain period of time, sends a notification message to the controller to ensure that the controller can pull the latest data.

[0065] For each controller, after it receives the notification message indicating the database update for the first time, it pulls data from the matching database cache device for the first time. At this time, it needs to send a first pull request indicating the pull of data to the database cache device. The first pull request is a full data pull request. The controller receives the full data from the database sent by the database cache device. That is, the controller pulls the full data from the database cache device based on the notification message indicating the database update received for the first time. For the database cache device, after it receives the notification message indicating the database update for the first time, it pulls the full data from the database to store all the data in the current database and provide the controller with the full data.

[0066] After pulling the full data, the controller builds a local Merkle tree based on the full data and generates a local signature based on the built local Merkle tree to obtain the local Merkle tree and local signature in the initial state. For the database cache device, after pulling the full data from the database, it builds a target Merkle tree based on the full data and generates a target signature based on the built target Merkle tree to obtain the target Merkle tree and target signature in the initial state.

[0067] In the above implementation scheme, after the controller receives the notification message indicating the database update for the first time, it pulls the full data from the database cache device, builds a local Merkle tree based on the pulled data and generates a local signature, so as to trigger the construction of the local Merkle tree and the generation of the local signature based on the first notification message.

[0068] The following describes the process of pulling a signature based on a trigger condition. Optionally, when pulling a target signature from a database cache device based on a trigger condition, it includes:

[0069] In response to reaching a verification opportunity for periodic data verification, sending a signature acquisition request to a database cache device to obtain a target signature;

[0070] The local signature is updated based on the restart of the controller, and the comparison between the updated local signature and the target signature is used to verify the integrity of the data saved after the controller is restarted.

[0071] The controller needs to perform periodic data verification. By periodically pulling the target signature from the database cache device, the consistency of the controller's local data and the data in the database cache device can be verified through signature comparison. Therefore, based on the data verification between each controller and the database cache device, periodic data verification between different controllers can be achieved.

[0072] For the controller, restarting the controller may cause data loss. After the restart occurs, the local Merkle tree is rebuilt based on the data saved after the restart, and the local signature is generated to update the local Merkle tree and local signature. When the verification time for periodic data verification arrives, the controller sends a signature acquisition request to the database cache device to obtain the target signature provided by the database cache device, and then compares the updated local signature with the pulled target signature to verify the integrity of the data saved after the controller restarts, thereby avoiding the situation where the controller cannot be identified in time when data loss occurs.

[0073] In the case where the controller is determined to have data loss based on the comparison between the local signature and the target signature, the lost data is pulled from the database cache device in a timely manner. Before pulling the lost data, the hash value of the target Merkle tree is first pulled, and the difference is located based on the comparison between the updated local Merkle tree and the hash value of the target Merkle tree, so as to pull the lost data based on the difference. After pulling the lost data, the local data is updated based on the pulled lost data, and the data is repaired in a timely manner to ensure the consistency of the data in the controller and the data in the database cache device, thereby ensuring the consistency of the data status of the current controller and other controllers.

[0074] The above implementation scheme actively pulls the target signature and compares it with the local signature when the verification time of periodic data verification is reached, so as to verify whether the local data has changed through signature comparison. When it is determined that the local data has changed, the matching data is pulled from the database cache device to update the local data, and the data can be repaired in time to maintain the consistency of the data of both parties; and the periodic verification mechanism can ensure the periodic reconciliation between different controllers, and update the data in time when problems are found to ensure state consistency.

[0075] Optionally, the data provided by the database cache device is pulled from the database cache device after the controller receives a notification message indicating a database update for the first time; when the target signature is pulled from the database cache device based on the trigger condition, it includes:

[0076] In response to a notification message indicating a database update that is not received for the first time, a signature acquisition request is sent to the database cache device to obtain a target signature; wherein the comparison between the target signature and the local signature is used to verify the consistency between the local data of the controller and the data cached by the database cache device.

[0077] After the controller receives the notification message indicating the database update for the first time, it pulls the full amount of data from the database cache device to build a local Merkle tree and generate a local signature; after the database cache device receives the notification message indicating the database update for the first time, it pulls the full amount of data from the database to build a target Merkle tree and generate a target signature. The database cache device receives the notification message before the controller to ensure that when the controller pulls data from the database cache device, the database cache device has already pulled the data from the database.

[0078] For notification messages indicating database updates that are not received for the first time, the database cache device directly pulls the updated data from the database, updates the local data and target Merkle tree based on the updated data, and updates the target signature. The local situation can be updated only by pulling necessary data and updating the database, which reduces the amount of data transmission and reduces resource usage.

[0079] For notification messages indicating database updates that are not received for the first time, the controller directly sends a signature acquisition request to the database cache device, obtains the target signature, compares the target signature with the local signature, and when it is determined that the signatures are inconsistent, pulls the hash value of the target Merkle tree, compares the hash value of the target Merkle tree corresponding to the database cache device with the local Merkle tree, locates the difference state, and requests the difference data, updates the local data based on the difference data, and updates the local data based on the database update, so as to maintain the consistency of the local data with the data in the database and the data in the database cache device. It is also necessary to update the local Merkle tree, and update the local signature based on the updated local Merkle tree to achieve the update of data, Merkle tree and signature.

[0080] In the above implementation scheme, for notification messages indicating database updates that are not received for the first time, the controller pulls the signature from the database cache device. When it is determined that there is a data difference based on the signature comparison, the difference data is pulled to update the local data. The local data can be updated based on the trigger conditions of the database update to maintain data consistency.

[0081] The following is an introduction to the working process of the controller through an overall implementation process. Figure 2 As shown:

[0082] Step 201: The controller obtains the full data, builds a local Merkle tree, and generates a local signature based on the notification message indicating the database update received for the first time.

[0083] Step 202: Pull the target signature from the database cache device based on the triggering condition of the data verification.

[0084] The triggering condition for data verification may be that a time for periodic verification has arrived, or that a notification message indicating a database update is not received for the first time.

[0085] Step 203: Compare the target signature with the local signature to see if they are consistent. If they are consistent, execute step 207; if they are inconsistent, execute step 204.

[0086] Step 204: Compare the hash values ​​of the local Merkle tree and the target Merkle tree corresponding to the database cache device to locate the difference status.

[0087] Step 205: Pull the difference data matching the difference state from the database cache device.

[0088] Step 206: Update local data and local Merkle tree based on the difference data, and update local signature, then return to step 202 to enter the next round of reconciliation.

[0089] Step 207: If the reconciliation is successful, return to step 202 to enter the next round of reconciliation.

[0090] In the above implementation process, redundant calculations can be avoided by detecting data differences through signature comparison; the difference parts can be accurately and quickly located by comparing the Merkle tree structure to ensure data verification efficiency; and only necessary data is transmitted during the data verification phase, which can reduce the amount of data transmission and reduce resource usage.

[0091] The present application also provides a data reconciliation method applied to a database cache device, such as Figure 3 As shown, including:

[0092] Step 301, in response to a signature acquisition request sent by the controller based on a trigger condition, a target signature is provided to the controller, the target signature is generated based on a target Merkle tree constructed by a database cache device, the target Merkle tree is constructed based on data pulled from a database, and the target Merkle tree and the target signature are updated in conjunction with updates of the database.

[0093] When the controller detects the trigger condition of data verification, it sends a signature acquisition request to the database cache device. Based on the signature acquisition request, the database cache device provides the controller with the target signature so that the controller can actively pull the target signature from the database cache device.

[0094] The target signature is generated based on the target Merkle tree constructed by the database cache device. The target Merkle tree is constructed based on the data pulled from the database. When the database is updated, the database cache device updates the local data. The target Merkle tree and the target signature are updated in conjunction with the database update to ensure that a target signature that is compatible with the database can be provided.

[0095] Step 302: Receive a second pull request carrying a difference state sent by the controller. The second pull request is sent after the controller identifies that the target signature and the local signature are inconsistent, compares the hash values ​​of the local Merkle tree and the target Merkle tree to locate the difference state. The local Merkle tree is constructed based on the data provided by the database cache device, and the local signature is generated based on the local Merkle tree.

[0096] After obtaining the target signature, the controller compares the target signature with the local signature, and verifies the consistency of the data of the devices on both sides through the signature. Among them, the local signature of the controller is generated based on the local Merkle tree, and the local Merkle tree is constructed based on the data from the database provided by the database cache device. After comparing the target signature with the local signature, if the target signature is inconsistent with the local signature, the controller requests the hash value of the target Merkle tree, compares the hash values ​​of the local Merkle tree and the target Merkle tree, and identifies different hash values ​​by comparing the hash values ​​in different Merkle trees, and then locates the difference state, which is the state determined based on the difference hash value. After locating the difference state, the controller sends a second pull request carrying the difference state to the database cache device, and the database cache device executes step 303 after receiving the second pull request.

[0097] Step 303: In response to the second pull request, provide the controller with difference data matching the difference state, where the difference data is used to update the local data, local Merkle tree, and local signature of the controller.

[0098] The second pull request is used to request differential data. The database cache device determines that differential data needs to be provided to the sender of the second pull request based on the received second pull request, that is, in response to the received second pull request, the database cache device provides the controller with differential data that matches the differential state, so that after obtaining the differential data, the controller updates the local data, the local Merkle tree, and the local signature of the controller based on the differential data. The differential data provided by the database cache device is necessary data. By transmitting only differential data, the amount of data transmission can be reduced and resources can be used reasonably.

[0099] The controller in this embodiment is a node controller in a distributed system, and multiple node controllers interact with the database cache device, and data reconciliation between the node controllers is performed based on the interaction with the database cache device. That is, the database cache device acts as an intermediary, and each node controller implements data consistency verification between different node controllers through interaction with the database cache device.

[0100] The above implementation scheme of the present application, after providing the target signature for the controller, provides the controller with difference data matching the difference state based on the second pull request sent by the controller indicating to pull data, so that the controller updates the local data, local Merkle tree and local signature according to the difference data, so as to realize real-time and high-frequency reconciliation of large-scale data in a distributed environment based on the interaction between the database cache device and each controller, provide an efficient solution for the data consistency of the distributed system, and improve the reliability, availability and security of the system.

[0101] Moreover, the signature comparison method for detecting data differences can avoid a large amount of redundant calculations, reduce network load and computing resource consumption; comparing the Merkle tree structure can accurately and quickly locate the difference part to ensure verification efficiency; by transmitting only necessary data (such as signatures, hash values, and difference data), the amount of data transmission can be reduced, resource usage can be reduced, and the scalability of distributed systems can be improved.

[0102] In an optional embodiment of the present application, the method further includes:

[0103] Receive a first pull request sent by a controller indicating to pull data, the first pull request being sent by the controller after first receiving a notification message indicating a database update; provide the controller with data from the database; wherein the data matching the first pull request is used to construct a local Merkle tree.

[0104] When the database is updated, the database cache device and the controller will receive a notification message. The database cache device pulls data from the database based on the notification message to update the local data; the controller pulls data from the database cache device based on the notification message to update the local data.

[0105] For the database cache device, in response to the first received notification message indicating database update, it pulls the full amount of data from the database, builds the target Merkle tree, and generates the target signature. That is, after the database cache device receives the notification message indicating database update for the first time, it pulls the full amount of data from the database to store all the data in the current database, builds the target Merkle tree based on the full amount of data, generates the target signature, and obtains the target Merkle tree and target signature in the initial state.

[0106] After receiving the notification message indicating the database update for the first time, the controller sends a first pull request to the database cache device indicating to pull the full data. The database cache device provides the controller with the full data from the database according to the first pull request, so that the controller builds a local Merkle tree based on the full data and generates a local signature based on the built local Merkle tree.

[0107] For the database cache device, in response to a notification message indicating a database update that is not received for the first time, it pulls the update data from the database, updates the target Merkle tree based on the update data, and updates the target signature based on the updated target Merkle tree. That is, for a notification message indicating a database update that is not received for the first time, the database cache device directly pulls the update data from the database, updates the local data and the target Merkle tree based on the update data, and updates the target signature. The local situation can be updated by pulling the necessary data, which reduces the amount of data transmission and reduces resource usage.

[0108] For notification messages indicating database updates that are not received for the first time, the controller directly sends a signature acquisition request to the database cache device. When the target signature is obtained and it is determined that the target signature is inconsistent with the local signature, the controller locates the difference state based on the comparison of the Merkle tree structure, requests the difference data, and updates the local data based on the difference data to maintain data consistency. It is also necessary to update the local Merkle tree and local signature to achieve the update of data, Merkle tree and signature.

[0109] The above is a data reconciliation solution applied to database cache devices. It is based on the data verification between the database cache device and each controller to ensure the reconciliation between different controllers, and can realize real-time and high-frequency reconciliation of large-scale data in a distributed environment, thereby improving the reliability, availability and security of the reconciliation. It also organizes data using the Merkle tree structure and determines the signature based on the tree root hash value, so that batch and efficient data integrity verification can be performed. The difference can be accurately and quickly located based on the comparison of the Merkle tree structure, which can ensure the verification efficiency, so as to meet the needs of large-scale distributed data reconciliation while taking into account security and efficiency.

[0110] It should be noted that in the embodiment of the present application, the server provides notification messages to the controller and the database cache device. The server receives user requests, organizes resources (such as resource creation and resource modification) based on the user requests, and then updates the database. After updating the database, a notification message is sent to the controller and the database cache device, and is sent to the database cache device first, and then to the controller. The following takes the cloud computing scenario as an example to introduce the interaction between the server, the controller (SDN controller), and the database cache device.

[0111] Among them, under the SDN architecture, the server is responsible for implementing specific SDN control applications, such as traffic engineering, load balancing, etc.; responsible for implementing network functions, configuring switches and routers, and storing configuration information in the database. The database cache device is responsible for caching and assembling data obtained from the database; responsible for maintaining the status information of the SDN network, including topology and device status; responsible for managing the network topology; responsible for caching data and building Merkle trees for the reconciliation process. The controller is responsible for executing the control plane logic of the SDN, formulating network policies and decisions; and responsible for reconciling with the database cache device to maintain data consistency with the database cache device.

[0112] The interaction process between the server, controller and database cache device is as follows Figure 4 As shown:

[0113] After updating the database, the server sends a notification message to the database cache device and the controller, with priority given to the database cache device and then to the controller. After receiving the notification message for the first time, the database cache device pulls the full data from the database for cache, builds the target Merkle tree, and generates the target signature. After receiving the notification message for the first time, the controller pulls the full data from the database cache device for cache, builds the local Merkle tree, and generates a local signature.

[0114] For notification messages that are not received for the first time, the database cache device pulls updated data from the database, and updates local data, target Merkle tree, and target signature based on the updated data. For notification messages that are not received for the first time, the controller pulls the target signature from the database cache device. When the target signature is inconsistent with the local signature, the controller locates the difference state based on the comparison of the Merkle tree structure, requests the difference data from the database cache device, and updates local data, local Merkle tree, and local signature based on the difference data.

[0115] The above interaction process can update the database cache device and the controller in a linked manner based on the database update to maintain data consistency.

[0116] The present application embodiment provides a data reconciliation device applied to a controller, where the controller is a node controller in a distributed system, such as Figure 5 As shown, the device comprises:

[0117] The first processing module 501 is used to construct a local Merkle tree and generate a local signature in response to acquiring data provided by the database cache device;

[0118] A pull comparison module 502 is used to pull a target signature from the database cache device based on a trigger condition, and compare the target signature with the local signature, wherein the target signature is generated based on a target Merkle tree constructed by the database cache device, the target Merkle tree is constructed based on data pulled from the database, and the target Merkle tree and the target signature are updated in conjunction with updates of the database;

[0119] A comparison and positioning module 503 is used for comparing the hash values ​​of the local Merkle tree and the target Merkle tree to locate the difference status in response to the target signature being inconsistent with the local signature;

[0120] A pull update module 504, configured to pull difference data matching the difference state from the database cache device, update local data and the local Merkle tree based on the difference data, and update the local signature;

[0121] Wherein, multiple node controllers perform data reconciliation between the node controllers based on the interaction with the database cache device.

[0122] Optionally, the first processing module includes:

[0123] A sending submodule, configured to send a first pull request indicating to pull data to the database cache device in response to a notification message indicating a database update received for the first time;

[0124] A receiving submodule, configured to receive data from the database and sent by the database cache device;

[0125] A generation submodule is constructed, which is used to construct the local Merkle tree based on the received data and generate the local signature based on the local Merkle tree.

[0126] Optionally, the pull comparison module is further used for:

[0127] In response to reaching a verification opportunity for periodic data verification, sending a signature acquisition request to the database cache device to obtain the target signature;

[0128] The local signature is updated based on the restart of the controller, and the comparison between the updated local signature and the target signature is used to verify the integrity of the data saved after the restart of the controller.

[0129] Optionally, the data provided by the database cache device is pulled from the database cache device by the controller after the controller first receives a notification message indicating a database update; the pulling and comparing module is further used to:

[0130] In response to a notification message indicating a database update that is not received for the first time, sending a signature acquisition request to the database cache device to acquire the target signature;

[0131] The comparison between the target signature and the local signature is used to verify the consistency between the local data of the controller and the data cached by the database cache device.

[0132] The present application also provides a data reconciliation device applied to a database cache device, such as Figure 6 As shown, including:

[0133] A first providing module 601 is used to provide a target signature for the controller in response to a signature acquisition request sent by the controller based on a trigger condition, wherein the target signature is generated based on a target Merkle tree constructed by the database cache device, the target Merkle tree is constructed based on data pulled from the database, and the target Merkle tree and the target signature are updated in linkage based on updates of the database;

[0134] A first receiving module 602 is used to receive a second pull request carrying a difference state sent by the controller, wherein the second pull request is sent after the controller recognizes that the target signature and the local signature are inconsistent, compares the hash values ​​of the local Merkle tree and the target Merkle tree to locate the difference state, and the local Merkle tree is constructed based on the data provided by the database cache device, and the local signature is generated based on the local Merkle tree;

[0135] A second providing module 603 is used to provide the controller with difference data matching the difference state in response to the second pull request, wherein the difference data is used to update the local data of the controller, the local Merkle tree and the local signature;

[0136] The controller is a node controller in a distributed system, and multiple node controllers perform data reconciliation between the node controllers based on interaction with the database cache device.

[0137] Optionally, the device further comprises:

[0138] A second receiving module is used to receive a first pull request sent by the controller to instruct to pull data, wherein the first pull request is sent by the controller after receiving a notification message indicating a database update for the first time;

[0139] A third providing module, used for providing the controller with data from the database;

[0140] The data matching the first pull request is used to construct the local Merkle tree.

[0141] Optionally, the device further comprises:

[0142] A second processing module, configured to, in response to a first received notification message indicating a database update, pull the full amount of data from the database, construct the target Merkle tree, and generate the target signature;

[0143] The third processing module is used to pull update data from the database in response to a notification message indicating a database update that is not received for the first time, update the target Merkle tree based on the update data, and update the target signature based on the updated target Merkle tree.

[0144] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0145] On the other hand, an embodiment of the present application also provides an electronic device, including a memory, a processor, a bus, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps in the above-mentioned data reconciliation method when executing the program.

[0146] For example, Figure 7 A schematic diagram of the physical structure of an electronic device is shown.

[0147] like Figure 7 As shown, the electronic device may include: a processor 710, a communication interface 720, a memory 730 and a communication bus 740, wherein the processor 710, the communication interface 720, and the memory 730 communicate with each other through the communication bus 740. The processor 710 can call the logic instructions in the memory 730, and the processor 710 is used to execute the various processes of the data reconciliation method of the embodiment of the present application, which will not be described one by one here.

[0148] In addition, the logic instructions in the above-mentioned memory 730 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art, and the computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk.

[0149] On the other hand, an embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the data reconciliation method provided in the above embodiments.

[0150] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0151] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0152] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A data reconciliation method, applied to a controller, characterized in that: The controller is a node controller in a distributed system, comprising: In response to obtaining the data provided by the database cache device, construct a local Merkle tree and generate a local signature; Pulling a target signature from the database cache device based on a trigger condition, comparing the target signature with the local signature, the target signature being generated based on a target Merkle tree constructed by the database cache device, the target Merkle tree being constructed based on data pulled from the database, and the target Merkle tree and the target signature being updated in conjunction with updates of the database; In response to the target signature being inconsistent with the local signature, comparing the hash values ​​of the local Merkle tree and the target Merkle tree to locate the difference state; Pulling difference data matching the difference state from the database cache device, updating local data and the local Merkle tree based on the difference data, and updating the local signature; Wherein, multiple node controllers perform data reconciliation between the node controllers based on the interaction with the database cache device.

2. The method according to claim 1, characterized in that In response to acquiring the data provided by the database cache device, constructing a local Merkle tree and generating a local signature, including: In response to a first received notification message indicating a database update, sending a first pull request indicating pulling data to the database cache device; Receiving data from the database sent by the database cache device; The local Merkle tree is constructed based on the received data, and the local signature is generated based on the local Merkle tree.

3. The method according to claim 1, characterized in that Pulling the target signature from the database cache device based on the trigger condition includes: In response to reaching a verification opportunity for periodic data verification, sending a signature acquisition request to the database cache device to obtain the target signature; The local signature is updated based on the restart of the controller, and the comparison between the updated local signature and the target signature is used to verify the integrity of the data saved after the restart of the controller.

4. The method according to claim 1 or 2, characterized in that: The data provided by the database cache device is pulled from the database cache device by the controller after the controller first receives a notification message indicating a database update; Pulling the target signature from the database cache device based on the trigger condition includes: In response to a notification message indicating a database update that is not received for the first time, sending a signature acquisition request to the database cache device to acquire the target signature; The comparison between the target signature and the local signature is used to verify the consistency between the local data of the controller and the data cached by the database cache device.

5. A data reconciliation method, applied to a database cache device, characterized in that: include: In response to a signature acquisition request sent by the controller based on a trigger condition, providing a target signature for the controller, wherein the target signature is generated based on a target Merkle tree constructed by the database cache device, the target Merkle tree is constructed based on data pulled from a database, and the target Merkle tree and the target signature are updated in conjunction with updates of the database; Receive a second pull request carrying a difference state sent by the controller, the second pull request is sent by the controller after identifying that the target signature and the local signature are inconsistent, and comparing the hash values ​​of the local Merkle tree and the target Merkle tree to locate the difference state, the local Merkle tree is constructed based on the data provided by the database cache device, and the local signature is generated based on the local Merkle tree; In response to the second pull request, providing the controller with difference data matching the difference state, wherein the difference data is used to update local data of the controller, the local Merkle tree, and the local signature; The controller is a node controller in a distributed system, and multiple node controllers perform data reconciliation between the node controllers based on interaction with the database cache device.

6. The method according to claim 5, characterized in that The method further comprises: receiving a first pull request sent by the controller indicating to pull data, wherein the first pull request is sent by the controller after receiving a notification message indicating a database update for the first time; providing the controller with data from the database; The data matching the first pull request is used to construct the local Merkle tree.

7. The method according to claim 5, characterized in that The method further comprises: In response to the first received notification message indicating a database update, pull the full amount of data from the database, build the target Merkle tree, and generate the target signature; In response to a notification message indicating a database update that is not received for the first time, update data is pulled from the database, the target Merkle tree is updated based on the update data, and the target signature is updated based on the updated target Merkle tree.

8. A data reconciliation device, applied to a controller, characterized in that: The controller is a node controller in a distributed system, comprising: A first processing module, configured to construct a local Merkle tree and generate a local signature in response to acquiring data provided by the database cache device; A pull comparison module, configured to pull a target signature from the database cache device based on a trigger condition, and compare the target signature with the local signature, wherein the target signature is generated based on a target Merkle tree constructed by the database cache device, the target Merkle tree is constructed based on data pulled from the database, and the target Merkle tree and the target signature are updated in linkage based on updates of the database; A comparison and positioning module, configured to compare the hash values ​​of the local Merkle tree and the target Merkle tree to locate the difference state in response to the target signature being inconsistent with the local signature; A pull update module, configured to pull difference data matching the difference state from the database cache device, update local data and the local Merkle tree based on the difference data, and update the local signature; Wherein, multiple node controllers perform data reconciliation between the node controllers based on the interaction with the database cache device.

9. A data reconciliation device, applied to a database cache device, characterized in that: include: A first providing module, configured to provide a target signature for the controller in response to a signature acquisition request sent by the controller based on a trigger condition, wherein the target signature is generated based on a target Merkle tree constructed by the database cache device, the target Merkle tree is constructed based on data pulled from a database, and the target Merkle tree and the target signature are updated in linkage based on an update of the database; A first receiving module is used to receive a second pull request carrying a difference state sent by the controller, wherein the second pull request is sent after the controller recognizes that the target signature and the local signature are inconsistent, compares the hash values ​​of the local Merkle tree and the target Merkle tree to locate the difference state, and the local Merkle tree is constructed based on the data provided by the database cache device, and the local signature is generated based on the local Merkle tree; a second providing module, configured to provide the controller with difference data matching the difference state in response to the second pull request, wherein the difference data is used to update local data of the controller, the local Merkle tree, and the local signature; The controller is a node controller in a distributed system, and multiple node controllers perform data reconciliation between the node controllers based on interaction with the database cache device.

10. An electronic device, characterized in that: The method comprises a processor, a memory and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, the steps of the data reconciliation method as described in any one of claims 1 to 4 or 5 to 7 are implemented.

Images