Consistency verification method, device and equipment

Abstract

The invention provides a consistency verification method, device and equipment, and relates to the technical field of computers or databases. The method comprises the steps of obtaining a submitted transaction event and generating a standardized event tuple, wherein the standardized event tuple comprises a global submission serial number, a time window identifier, a key space fragment identifier, a transaction intention abstract, a result set hash and a database instance identifier; a Merkel directed acyclic graph is constructed based on the standardized event tuple, the Merkel directed acyclic graph takes time window identification and key space fragment identification as dimensions, and a window root and a global root are generated through step-by-step aggregation; window roots are exchanged across database instances and consistency verification is performed. According to the method, the consistency verification efficiency can be improved.

Description

Technical Field

[0001] This application relates to the field of computer technology or database technology, and in particular to a consistency verification method, apparatus and device. Background Technology

[0002] In high-availability business systems such as finance, telecommunications, and cloud computing, multiple computing nodes need to process massive amounts of transactional data in real time. Ensuring global data consistency is a core requirement for guaranteeing business correctness and system reliability. For example, in a distributed database, multiple nodes may simultaneously process transactions such as user transfers, order creation, and inventory updates. If data inconsistencies between nodes are not detected in a timely manner, it may lead to business logic errors, data loss, or system crashes.

[0003] While methods such as full data comparison, log replay verification, and periodic checksums can solve some problems, they have limitations such as high verification costs, poor positioning accuracy, and inability to capture timeline-related defects. Summary of the Invention

[0004] This application provides a consistency verification method, apparatus, and device to improve consistency verification efficiency.

[0005] Firstly, this application provides a consistency verification method, comprising: obtaining committed transaction events and generating standardized event tuples, wherein the standardized event tuples include a global commit sequence number, a time window identifier, a key space shard identifier, a transaction intent summary, a result set hash, and a database instance identifier; constructing a Merkel directed acyclic graph based on the standardized event tuples, wherein the Merkel directed acyclic graph generates a window root and a global root by hierarchical aggregation using the time window identifier and the key space shard identifier as dimensions; and exchanging window roots across database instances and performing consistency verification.

[0006] In one possible embodiment, constructing a Merkel directed acyclic graph based on standardized event tuples includes: dividing transaction events according to time window identifiers and key space shard identifiers to form transaction event sets; arranging the transaction events in each transaction event set in chronological order indicated by the global commit sequence number, and calculating leaf node hashes, whereby the leaf node hashes are generated by aggregating the transaction intent summary, result set hashes, and global commit sequence numbers; and aggregating the leaf node hashes from bottom to top to generate intermediate node hashes until the window root and global root are formed.

[0007] In one possible embodiment, the bottom-up aggregation of leaf node hashes to generate intermediate node hashes until the window root and global root are formed includes: for any leaf node, concatenating the adjacent leaf nodes, the time window identifier of the transaction event, the key space shard identifier, the database instance identifier, and the metadata to generate intermediate node hashes until the window root and global root are generated.

[0008] In one possible embodiment, after exchanging window roots across database instances and performing consistency verification, the consistency verification method further includes: if the window roots are inconsistent, performing a recursive binary search along the time window identifier and the key space shard identifier to locate the differing leaf nodes.

[0009] In one possible embodiment, after locating the differing leaf nodes, the consistency verification method further includes: extracting the redo log fragments and data block snapshots corresponding to the differing leaf nodes; performing replay verification in an isolated environment and generating a minimum number of counterexamples.

[0010] In one possible embodiment, generating a minimum counterexample includes: extracting relevant information of the differing leaf nodes, including the global commit sequence number, key space shard identifier, time window identifier, and redo log offset corresponding to the differing leaf nodes; and generating a minimum counterexample based on the relevant information.

[0011] In one possible implementation, the global commit number is either a system change number or a hybrid logic clock.

[0012] In one possible embodiment, before generating standardized event tuples, the consistency verification method further includes: dynamically adjusting the key space sharding strategy according to real-time load and transaction hotspot areas. The key space sharding strategy specifically includes one of the following: dividing the key space sharding by transaction hotspot areas; dividing the key space sharding by transaction conflict rate; and dividing the key space sharding by transaction frequency.

[0013] Secondly, this application provides a consistency verification apparatus, comprising: an event tuple generation module, used to obtain committed transaction events and generate standardized event tuples, wherein the standardized event tuples include a global commit sequence number, a time window identifier, a key space shard identifier, a transaction intent summary, a result set hash, and a database instance identifier; a directed acyclic graph (DAG) construction module, used to construct a Merkle DAG based on the standardized event tuples, wherein the Merkle DAG generates a window root and a global root through hierarchical aggregation using the time window identifier and the key space shard identifier as dimensions; and a verification module, used to exchange window roots across database instances and perform consistency verification. Thirdly, this application provides an electronic device, comprising: a processor, and a memory communicatively connected to the processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to implement the method as described in any of the first aspects.

[0014] Fourthly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any of the first aspects.

[0015] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the method of any one of the first aspects.

[0016] In this embodiment, by obtaining committed transaction events and generating standardized event tuples, a standardized transaction format for transaction events can be provided, ensuring that the same transaction event can generate identical standardized transaction tuples. Furthermore, the global commit sequence number, time window identifier, key space shard identifier, transaction intent summary, result set hash, and database instance identifier of the standardized event tuple facilitate subsequent traceability.

[0017] A Merkel directed acyclic graph (DAG) is constructed based on standardized event tuples. The DAG uses time window identifiers and key space shard identifiers as dimensions, generating a window root and a global root through hierarchical aggregation. In other words, a unified model is formed from both the time and data shard dimensions to create the Merkel DAG, allowing for consideration of both temporal and data consistency during subsequent consistency verification.

[0018] By exchanging window roots across database instances and performing consistency verification—that is, by exchanging window roots and verifying their consistency—consistency verification of transaction events in a distributed system can be achieved across multiple database instances. Firstly, it eliminates the need for full data verification or replay verification, thus reducing overhead and increasing speed, supporting online consistency verification. Secondly, since window roots are generated using time window identifiers and key space shard identifiers, verifying the consistency of window roots ensures both time consistency and data consistency, thereby guaranteeing the accuracy of consistency verification. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0020] Figure 1 This is a schematic diagram illustrating an application scenario of the consistency verification method according to an embodiment of this application;

[0021] Figure 2 This is a flowchart of the consistency verification method according to an embodiment of this application;

[0022] Figure 3 This is a flowchart of a consistency verification method according to another embodiment of this application;

[0023] Figure 4 This is a schematic diagram of a conformance verification device according to an embodiment of this application;

[0024] Figure 5 This is a schematic diagram of an electronic device according to an embodiment of this application.

[0025] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0026] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0027] Related technology 1 uses full data comparison for consistency verification. Specifically, it verifies data consistency by periodically scanning and comparing the full data of each node in the cluster. Its advantage is that the verification results are accurate and reliable, but the computational complexity is O(Data) (proportional to the amount of data), and it requires transmitting the full data over the network, leading to prolonged verification time and high resource consumption. Furthermore, new write operations need to be isolated during verification, affecting system availability and making it difficult to meet real-time requirements.

[0028] Related technology 2 performs consistency verification by replaying transaction logs, which can capture transaction-level differences. However, its implementation relies on storing the complete log, which is costly to replay and makes it difficult to locate differing transactions. When the data volume is large, the overhead of log storage and replay increases exponentially, and it cannot effectively identify timeline-related defects (such as out-of-order commits).

[0029] Related technology 3 performs consistency verification through periodic checks. Specifically, it compares the checksums of data blocks or tables periodically, which is simple and has low overhead. However, it can only verify the consistency of data block content and cannot capture timeline-related inconsistencies (such as incorrect transaction commit order, ghost commits, etc.). Furthermore, locating discrepancies requires additional methods and lacks automation capabilities.

[0030] In summary, the shortcomings of related technologies include at least one of the following: high cost of consistency verification (high consumption of computational, storage, and network resources), poor accuracy in difference location (difficulty in quickly locating the smallest set of differing transactions), inability to verify timeline consistency (such as defects like out-of-order transactions and ghost commits), and the fact that most verifications are offline and cannot support real-time monitoring. Furthermore, the lack of a unified model for "transaction intent-commit sequence number-result summary" in these technologies makes it impossible to automate the consistency proof and difference location processes, hindering their adaptation to the actual needs of high-concurrency, large-scale distributed systems.

[0031] Based on this, embodiments of this application provide a consistency verification method, apparatus, and device. Based on a two-dimensional modeling (timeline and key space sharding) and hierarchical summary structure of transaction commit events in a distributed system, it achieves efficient consistency proof and accurate difference location through a full-process processing including event standardization, summary aggregation, root reconciliation, difference drill-down, and on-demand backtracking verification. This aims to address at least one deficiency in related technologies.

[0032] Figure 1 This is a schematic diagram illustrating an application scenario of the consistency verification method according to an embodiment of this application.

[0033] like Figure 1 As shown, users 1 through X each initiate service requests through application 2 installed on their respective terminal devices, where X is an integer greater than or equal to 2. The X service requests from users 1 through X are sent to electronic device 3, which can be a server. Electronic device 3 processes the X service requests, and the processing involves a distributed system. For example, the data related to the X service requests processed by electronic device 3 is stored in a distributed database, which can be a shared storage cluster, i.e., the distributed database is deployed on a storage cluster composed of multiple nodes. After receiving a service request event indicating that multiple nodes are concurrently reading and writing the same dataset, electronic device 3 can execute the consistency verification method of this application embodiment.

[0034] Specifically, the embodiments of this application can be applied to real-time transfers in financial scenarios, order processing in telecommunications services, and distributed storage in cloud computing platforms.

[0035] The technical solution of this application and how it solves the above-mentioned technical problems will be described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described below with reference to the accompanying drawings.

[0036] Figure 2 This is a flowchart illustrating a conformance verification method according to an embodiment of this application. The conformance verification method in this embodiment is executed by an electronic device, which may be a server.

[0037] like Figure 2 As shown, the consistency verification method of this application embodiment includes steps S110 to S130.

[0038] S110. Obtain the committed transaction events and generate normalized event tuples.

[0039] A committed transaction event refers to a concurrent event in a distributed database where a transaction is committed, such as a business request event involving concurrent read and write operations on the same dataset across multiple database instances. This business request event includes INSERT, UPDATE, or DELETE operations.

[0040] A normalized event tuple is a tuple of data generated after structured processing of committed transaction events. A normalized event tuple includes the global commit sequence number G, the time window identifier W, the key space shard identifier S, the transaction intent summary I, the result set hash R, and the database instance identifier ID.

[0041] For example, the global commit sequence number G is either a System Change Number (SCN) or a Hybrid Logical Clock (HLC). The System Change Number is a globally monotonic commit number, and the Hybrid Logical Clock is a timestamp. This global commit sequence number ensures the global order and visibility of transaction commits in a distributed system.

[0042] A transaction intent digest is a lightweight summary of the data items, key ranges, and operation types affected by a transaction event. It is used to quickly identify the scope and type of operation impacted by a transaction event. The transaction intent digest can be represented in a structured manner using hash values. A transaction intent digest may include: table name, primary key range, operation type, and number of rows affected.

[0043] Result set hash is a hash digest of the affected subset of data or query results that is visible after a transaction event is committed, used to verify the consistency of the transaction execution results. Result set hash is a hash value based on a fixed query template, primary key ordered output, and NULL normalization. The fixed query template uses a uniform query field order and filtering conditions for the affected subset of data. Primary key ordered output means that if a transaction event affects multiple rows of data (e.g., batch updates), the data is sorted in ascending or descending primary key order before serialization to avoid different hashes due to differences in row order. NULL normalization uses a globally uniform representation of NULL values ​​(e.g., using the string "#NULL#") to avoid differences in the storage / return format of NULL values ​​across different nodes (e.g., empty strings, null keywords). The result set hash is obtained by calculating the normalized data string after the above three steps using a hash algorithm.

[0044] A database instance identifier is used to identify a single database instance in a distributed database. The database instance identifier allows you to locate the database instance where transaction events were executed and committed.

[0045] The time window identifier is used to identify the time window in which a transaction event falls; the time window is a time interval.

[0046] The key-space shard identifier is used to uniquely identify a key-space shard. A key-space shard is obtained by dividing a data field.

[0047] For example, an electronic device can divide the key space into fragments according to dimensions such as table partitions, primary key ranges, and physical block numbers.

[0048] For example, electronic devices can pre-deploy event collectors in a sidecar manner on each database instance of a distributed database to intercept or subscribe to locally committed transaction events and record event tuples. These event tuples can be processed into standardized event tuples through the aforementioned structured processing.

[0049] In one possible embodiment, the electronic device may incorporate two globally unique, dynamically generated random values ​​(i.e., random salts) into the hash calculation process of the standardized event tuple, so that the hash calculation result of the standardized event tuple of each transaction event is unforgeable and unrepeatable, preventing attackers from replaying legitimate transaction events, and avoiding hash collisions in different scenarios that could lead to misjudgments in consistency verification, thereby ensuring the security and accuracy of consistency verification.

[0050] Specifically, electronic devices can dynamically generate anti-replay random salts according to the time windows in which transaction events fall. One time window corresponds to one anti-replay random salt, which can be globally allocated and synchronized to all database instances in the distributed database. The anti-replay random salt automatically expires after the consistency verification of transaction events within the corresponding time window is completed.

[0051] When each database instance starts, the electronic device generates a corresponding instance-level random salt, which remains constant throughout the database instance's lifecycle. The instance-level random salt is regenerated upon database instance restart. This allows for accurate differentiation of transaction events executed by different database instances.

[0052] S120. Construct Merkel directed acyclic graph based on standardized event tuples.

[0053] Merkle Directed Acyclic Graphs (Merkle DAGs) use time window identifiers and key space partition identifiers as dimensions, generating window roots and global roots through hierarchical aggregation. Specifically, Merkle DAGs use time window identifiers as the primary dimension and key space partition identifiers as the secondary dimension.

[0054] Based on Merkel directed acyclic graphs (DAGs) with time window identifiers and key space partition identifiers as dimensions, Merkel DAGs support cross-dimensional binary drill-down.

[0055] The Merkle directed acyclic graph in this application embodiment can be understood as a hierarchical digest structure, which is a tree structure consisting of multiple leaf nodes, at least one level of intermediate nodes, and at least one root node from bottom to top. Each node is represented using a hash digest.

[0056] Based on Merkel's directed acyclic graph, with time window identifier as the primary dimension and key space partition identifier as the secondary dimension, the window root (Root(W)) refers to the root hash value of a specific time window identifier and the full key space partition associated with that specific time window identifier.

[0057] The global root refers to the root hash value obtained by aggregating all window roots, which represents the data consistency state of the entire distributed database.

[0058] S130, Exchange window roots across database instances and perform consistency verification.

[0059] Consistency verification refers to the process of exchanging window roots across database instances and verifying their consistency. For example, database instance A and database instance B exchange window roots with the same time window identifier. If their window roots are consistent, the consistency verification is considered successful.

[0060] For example, if the consistency verification passes, the electronic device can generate a consistency proof report, which includes the same window root, the standardized event tuples corresponding to the transaction events, etc.

[0061] For example, if the consistency verification fails, the electronic device can generate evidence of discrepancies, which includes inconsistent window roots, normalized event tuples corresponding to transaction events, etc.

[0062] In this embodiment, by obtaining committed transaction events and generating standardized event tuples, a standardized transaction format for transaction events can be provided, ensuring that the same transaction event can generate identical standardized transaction tuples. Furthermore, the global commit sequence number, time window identifier, key space shard identifier, transaction intent summary, result set hash, and database instance identifier of the standardized event tuple facilitate subsequent traceability.

[0063] A Merkel directed acyclic graph (DAG) is constructed based on standardized event tuples. The DAG uses time window identifiers and key space shard identifiers as dimensions, generating a window root and a global root through hierarchical aggregation. In other words, a unified model is formed from both the time and data shard dimensions to create the Merkel DAG, allowing for consideration of both temporal and data consistency during subsequent consistency verification.

[0064] By exchanging window roots across database instances and performing consistency verification—that is, by exchanging window roots and verifying their consistency—consistency verification of transaction events in a distributed system can be achieved across multiple database instances. Firstly, it eliminates the need for full data verification or replay verification, thus reducing overhead and increasing speed, supporting online consistency verification. Secondly, since window roots are generated using time window identifiers and key space shard identifiers, verifying the consistency of window roots ensures both time consistency and data consistency, thereby guaranteeing the accuracy of consistency verification.

[0065] In one possible embodiment, step S120, which constructs a Merkel directed acyclic graph based on standardized event tuples, includes: dividing transaction events according to time window identifiers and key space shard identifiers to form transaction event sets; arranging the transaction events in each transaction event set according to the time order indicated by the global commit number, and calculating the leaf node hash; aggregating the leaf node hashes from bottom to top to generate intermediate node hashes, until the window root and global root are formed.

[0066] For example, each set of transaction events corresponds to a time window identifier and a key space shard identifier. This is equivalent to each set of transaction events having two labels: a time window identifier label and a key space shard identifier. This minimizes the granularity of consistency verification, facilitating precise identification of discrepancies later.

[0067] For example, multiple transaction events are first divided using time window identifiers, specifically time window identifier W1 and time window identifier W2. For the same time window identifier W1, further division is performed based on key space shard identifiers. For instance, this can be divided into three transaction event sets: [W1, S1], [W1, S2], and [W1, S3], where S1 to S3 are three different key space shards. For any transaction event among the multiple transaction events, falling into one of the three transaction event sets [W1, S1], [W1, S2], or [W1, S3], based on the time window being a time interval and the key space shard being a data domain interval, each of [W1, S1], [W1, S2], and [W1, S3] includes at least one transaction event.

[0068] For any set of transaction events, the transaction intent digest I, global commit number G, result set hash R, and global commit number G of each transaction event in the set are concatenated. Additionally, when the electronic device generates anti-replay random salts and instance-level random salts, the transaction intent digest I, global commit number G, result set hash R, global commit number G, anti-replay random salt, and instance-level random salt of each transaction event in the set are concatenated. The result of the string concatenation is then hashed to obtain the leaf node hash.

[0069] Using the example above, the three transaction event sets [W1, S1], [W1, S2], and [W1, S3] are used as leaf nodes to calculate leaf node hashes. Similarly, the three transaction event sets [W1, S1], [W1, S2], and [W1, S3] can be aggregated based on the same time window identifier to generate corresponding intermediate node hashes.

[0070] This application embodiment divides transaction events into transaction event sets by time window identifiers and key space sharding identifiers. During subsequent consistency verification, it is not necessary to process all transaction events, but to use the transaction event set as the granularity, thereby reducing the consistency verification overhead.

[0071] By arranging transaction events in each transaction event set according to the time sequence indicated by the global commit number, and calculating leaf node hashes and aggregating leaf node hashes from bottom to top to generate intermediate node hashes, until the window root and global root are formed, the hierarchy and scalability of the Merkel directed acyclic graph can be ensured. Such a Merkel directed acyclic graph can support subsequent binary drill-down to locate differences and can support locating to leaf nodes to improve the accuracy of difference location.

[0072] In one possible embodiment, the bottom-up aggregation of leaf node hashes to generate intermediate node hashes until the window root and global root are formed includes: for any leaf node, concatenating the adjacent leaf nodes, the time window identifier of the transaction event, the key space shard identifier, the database instance identifier, and the metadata to generate intermediate node hashes until the window root and global root are generated.

[0073] Metadata refers to additional information related to transaction events, such as transaction conflict rate and commit timestamp.

[0074] Taking [W1, S1] as a leaf node in the above example, its adjacent leaf nodes are [W1, S2]. That is, the adjacent leaf nodes of a leaf node have the same time window identifier and adjacent key space fragment identifiers. Taking [W1, S2] as a leaf node in the above example, its adjacent leaf nodes are [W1, S1] and [W1, S3].

[0075] In this embodiment, for any leaf node, the intermediate node hash is generated by concatenating the adjacent leaf nodes, the time window identifier of the transaction event, the key space shard identifier, the database instance identifier, and metadata, until the window root and global root are generated. This allows for accurate and efficient construction of the hierarchical summary structure of the Merkel directed acyclic graph. Furthermore, by integrating the time window identifier, key space shard identifier, database instance identifier, and metadata into each intermediate node hash, the representativeness of each intermediate node hash is improved.

[0076] Unlike the above embodiments, in another possible embodiment, for any leaf node, the time window identifier, key space shard identifier, and database instance identifier of the transaction event of the leaf node are concatenated to generate an intermediate node hash, until the window root and global root are generated.

[0077] like Figure 2 As shown, in one possible embodiment, after the cross-database instance window root exchange and consistency verification in step S130, the consistency verification method further includes step S140.

[0078] S140. If the window roots are inconsistent, a recursive binary search is performed along the time window identifier and the key space fragment identifier to locate the differing leaf nodes.

[0079] Recursive binary search refers to the process of repeatedly splitting the range of nodes to be searched into two equal parts by recursively calling and dividing the range into two equal parts. Each time, only the aggregate hash of the two parts is compared. The equal part with no difference is excluded. The comparison operation is repeated for the equal part with difference until the range of nodes to be searched is reduced to a single leaf node, which is the difference leaf node.

[0080] A differential leaf node is a leaf node that causes inconsistency in the window root.

[0081] In this embodiment, after exchanging window roots across database instances and performing consistency verification, if the window roots are inconsistent, the differing leaf nodes are located through a recursive binary search along the time window identifier and the key space shard identifier. This method avoids full scans or log replays by exchanging only the necessary branch hashes on the differing path through recursive binary search, achieving a complexity of O(log|W|×log|S|), where |W| is the number of time window identifiers and |S| is the number of key space shard identifiers. This reduces the difference location complexity from O(Data) to O(HashPath), improving the efficiency and accuracy of difference location.

[0082] like Figure 2 As shown, in one possible embodiment, after locating the differing leaf nodes in step S140, the consistency verification method further includes steps S150 to S160.

[0083] S150. Extract the redo log fragments and data block snapshots corresponding to the differential leaf nodes.

[0084] A redo log fragment refers to a database replay log recorded after a transaction event is committed.

[0085] A data block snapshot is a backup of the affected data blocks after a transaction event is committed.

[0086] S160. Perform replay verification in an isolated environment and generate a minimum number of counterexamples.

[0087] An isolated environment includes an environment or container used for testing that is physically isolated from the actual production environment.

[0088] The minimum counterexample is the finest-grained evidence that causes inconsistency at the window root. For example, the difference data includes this minimum counterexample.

[0089] For example, after an electronic device performs replay verification in an isolated environment, the hash result of the replay verification is compared with the hash of the Merkel directed acyclic graph to generate a minimum number of counterexamples.

[0090] In this embodiment, the electronic device extracts the redo log fragments corresponding to the differing leaf nodes to accurately obtain these fragments as the basis for subsequent restoration of transaction event execution. This eliminates the need to analyze the full redo log, reducing log processing overhead. The data block snapshots corresponding to the differing leaf nodes represent the initial data state before the transaction event execution. Extracting these snapshots facilitates maintaining a consistent data state during subsequent replay verification in an isolated environment.

[0091] By replaying the verification in an isolated environment, the execution flow of transaction events within the differing leaf nodes can be fully reproduced without affecting the business system, thus allowing the determination of the cause of window root inconsistency. Generating minimal counterexamples supports auditing, facilitating accurate identification of the root inconsistency by relevant personnel.

[0092] In one possible embodiment, step S160 of generating the minimum counterexample includes: extracting relevant information of the differential leaf nodes and generating the minimum counterexample based on the relevant information.

[0093] The relevant information includes the global commit sequence number, key space shard identifier, time window identifier, and redo log offset corresponding to the differential leaf node.

[0094] In this embodiment, relevant information from the differing leaf nodes is extracted, and a minimum counterexample is generated based on this information. The global commit sequence number corresponding to the differing leaf node uniquely identifies the global execution order of transaction events, avoiding deviations caused by out-of-order transaction events. The key space shard identifier accurately locates the data shard in which it resides, eliminating interference from other data shards. The time window identifier is used to lock the time batch of the transaction event, avoiding cross-time batch investigation. The redo log offset is used to locate the physical location of the redo log corresponding to this transaction event in the production environment, without needing to traverse the entire redo log. This improves the ease of subsequent investigation.

[0095] Unlike embodiments of electronic devices that partition key space according to dimensions such as table partitioning, primary key range, and physical block number, such as... Figure 2 As shown, in one possible embodiment, before generating the standardized event tuple in step S110, the consistency verification method further includes step S101.

[0096] S101. Dynamically adjust the key space sharding strategy based on real-time load and transaction hotspot areas.

[0097] Real-time load includes the load on computing resources, storage resources, memory resources, network resources, etc.

[0098] A transaction hotspot region refers to a key space region that is modified frequently. For example, a transaction hotspot region includes the range of user IDs used in high-frequency transactions in a financial system.

[0099] Key space sharding strategies specifically include at least one of the following: sharding the key space according to transaction hotspot areas; sharding the key space according to transaction conflict rate; and sharding the key space according to transaction frequency.

[0100] Transaction conflict rate refers to the probability of a conflict occurring during the commit of a transaction event. For example, in the order processing scenario of a telecommunications system, frequently modified order ID ranges may have a high conflict rate. Transaction conflicts typically occur in transaction hotspots. Therefore, by partitioning the key space based on the transaction conflict rate to distribute high-conflict data domains, transaction hotspots can be dispersed. Furthermore, transaction conflicts cause transaction events to queue and increase latency, which in turn attracts more transaction event requests to retry, exacerbating transaction hotspots. Therefore, partitioning the key space based on the transaction conflict rate reduces transaction conflicts and also prevents the aggravation of transaction hotspots.

[0101] Similarly, high-frequency transactions are a significant factor contributing to transaction hotspots. By partitioning the key space according to transaction frequency to distribute high-frequency transactions, transaction hotspots can also be dispersed.

[0102] Real-time load is often correlated with transaction hotspots; for example, transaction hotspots cause high real-time load. In this embodiment, by dynamically adjusting the key space sharding strategy based on real-time load and transaction hotspots, dynamic adaptive key space sharding can be achieved, realizing key space sharding that balances load balancing and hotspot distribution.

[0103] Figure 3 This is a flowchart of a consistency verification method according to an embodiment of this application.

[0104] like Figure 3 As shown, the consistency verification method includes steps S201 to S217.

[0105] S201, The database instance submits transaction events to the event collector.

[0106] S202, The event collector generates standardized event tuples.

[0107] S203, The event collector pushes standardized event tuples to the summary builder.

[0108] The summary builder is used to generate Merkel directed acyclic graphs based on normalized event tuples, specifically including the following steps S204 to S207.

[0109] S204, The summary builder generates a set of event transactions.

[0110] S205, The digest builder calculates the leaf node hash.

[0111] S206, The digest builder calculates the hash of intermediate nodes.

[0112] S207, The summary builder generates the window root and global root.

[0113] S208, The summary builder submits the window root to the root reconciler.

[0114] The root reconciler is used to exchange window roots across database instances and perform consistency verification.

[0115] S209, The root reconciler exchanges window roots across database instances and verifies consistency.

[0116] If the window roots are the same, proceed to step S210. If the window roots are not the same, proceed to step S211.

[0117] S210, Consistency verification passed. At this point, the root reconciler can send a message to the digest builder indicating that the consistency verification has passed.

[0118] S211, The root reconciliation unit triggers the difference drill-down to execute a recursive binary search. The difference drill-down is used to execute the recursive binary search. The recursive binary search includes S212 and S213, which are executed iteratively.

[0119] S212, the difference drill-down requests and returns intermediate node hashes to the root reconciliation unit. Step S212 includes the difference drill-down requesting intermediate node hashes from the digest builder, and the digest builder responding to the request and sending the intermediate node hashes to the difference builder.

[0120] S213, Differential drill bit selection of inconsistent intermediate node hashes.

[0121] S214, Differential Drilling Tool Positioning Differential Leaf Node.

[0122] S215. The difference tracing module extracts replay log fragments and data block snapshots. These replay log fragments and data block snapshots refer to the replay log fragments and data block snapshots of the difference leaf nodes.

[0123] S216. The difference tracing module performs replay verification in an isolated environment.

[0124] S217. The discrepancy tracking module generates a minimum negative example. This minimum negative example can be sent to the root reconciliation unit by the discrepancy tracking module.

[0125] Steps S201 to S203 can be understood as the transaction event collection phase. Steps S204 to S207 can be understood as the Merkel directed acyclic graph construction phase. Steps S208 to S217 can be understood as the consistency verification phase.

[0126] In summary, the consistency verification method of this application embodiment has at least one of the following technical effects:

[0127] 1. The complexity of consistency verification is reduced and the network transmission overhead is reduced (only inconsistent window root and necessary intermediate node hashes are transmitted), and the output / output (I / O) overhead is reduced (full scan is avoided).

[0128] 2. Improved difference positioning accuracy: Difference positioning time has been reduced from hours to minutes, and the average difference drill-down depth is controlled within 3-5 levels. Furthermore, the minimum negative example contains accurate information such as the global commit sequence number of the transaction event and the key space fragmentation identifier.

[0129] 3. Supports time consistency verification: Supports monotonicity verification of global commit sequence numbers, thereby capturing time defects such as out-of-order visibility and ghost commits.

[0130] 4. Supports near-online consistency monitoring: Supports real-time reconciliation at the window root level (second-level response).

[0131] 5. Convenient Deployment: The event collector is deployed as a sidecar, supporting non-intrusive integration. Furthermore, this application, through replay verification in an isolated environment, can be decoupled from log replay mechanisms of related technologies.

[0132] Figure 4 This is a schematic diagram of the conformance verification device according to an embodiment of this application. Figure 4 As shown, the consistency verification device provided in this application embodiment includes: an event tuple generation module 310, a directed acyclic graph construction module 320, and a verification module 330.

[0133] The event tuple generation module 310 is used to obtain committed transaction events and generate standardized event tuples. The standardized event tuples include the global commit sequence number, time window identifier, key space shard identifier, transaction intent summary, result set hash, and database instance identifier.

[0134] The directed acyclic graph construction module 320 is used to construct Merkel directed acyclic graphs based on standardized event tuples. The Merkel directed acyclic graph uses time window identifiers and key space partition identifiers as dimensions, and generates window roots and global roots through hierarchical aggregation.

[0135] Verification module 330 is used to exchange window roots across database instances and perform consistency verification.

[0136] In one possible embodiment, the directed acyclic graph (DAG) construction module includes: a transaction event set determination submodule, used to divide transaction events according to time window identifiers and key space sharding identifiers to form transaction event sets; a leaf node hash generation submodule, used to arrange the transaction events in each transaction event set according to the time order indicated by the global commit sequence number, and calculate the leaf node hash, whereby the leaf node hash is generated by aggregating the transaction intent summary, result set hash, and global commit sequence number; and an intermediate node hash generation submodule, used to aggregate the leaf node hashes from bottom to top to generate intermediate node hashes until the window root and global root are formed.

[0137] In one possible embodiment, the intermediate node hash generation submodule includes: an intermediate node hash generation unit, used to concatenate the adjacent leaf nodes of any leaf node, the time window identifier of the transaction event, the key space shard identifier, the database instance identifier, and the metadata to generate an intermediate node hash, until the window root and the global root are generated.

[0138] In one possible embodiment, the consistency verification device further includes a difference location module, which is used to recursively perform a binary search along the time window identifier and the key space fragment identifier to locate the difference leaf node if the window root is inconsistent.

[0139] In one possible embodiment, the consistency verification device further includes: a post-processing module for extracting redo log fragments and data block snapshots corresponding to the differing leaf nodes; and a minimum counterexample generation module for performing replay verification in an isolated environment and generating a minimum counterexample.

[0140] In one possible embodiment, the minimum counterexample generation module includes: an information extraction submodule for extracting relevant information of the differential leaf nodes, including the global commit sequence number, key space shard identifier, time window identifier, and redo log offset corresponding to the differential leaf nodes; and a minimum counterexample generation submodule for generating a minimum counterexample based on the relevant information.

[0141] In one possible implementation, the global commit number is either a system change number or a hybrid logic clock.

[0142] In one possible embodiment, the consistency verification device further includes: a key space sharding strategy adjustment module, used to dynamically adjust the key space sharding strategy according to real-time load and transaction hotspot areas. The key space sharding strategy specifically includes one of the following: dividing the key space into shards according to transaction hotspot areas; dividing the key space into shards according to transaction conflict rate; and dividing the key space into shards according to transaction frequency.

[0143] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 5 As shown, this application embodiment provides an electronic device including a processor 401 and a memory 402. Optionally, the device further includes a communication component 403. The processor 401, memory 402, and communication component 403 are connected via a bus 404.

[0144] In the specific implementation process, the memory 402 stores code, and the processor 401 runs the code stored in the memory 402 to execute the method of the above method embodiment.

[0145] The specific implementation process of processor 401 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0146] In the above Figure 5In the illustrated embodiments, it should be understood that the processor 401 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0147] The memory 402 may include high-speed RAM memory, and may also include non-volatile memory (NVM), such as at least one disk storage.

[0148] Bus 404 can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Bus 404 can be divided into address bus, data bus, control bus, etc. For ease of illustration, the bus 404 in the accompanying drawings of this application is not limited to only one bus or one type of bus.

[0149] This application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the methods described in the above-described method embodiments.

[0150] The aforementioned computer-readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0151] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0152] This application provides a computer program product, including a computer program that, when executed by a processor, implements the methods provided in any of the embodiments described above.

[0153] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this application.

[0154] It should be further noted that although the steps in the flowchart are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0155] It should be understood that the above-described device embodiments are merely illustrative, and the device of this application can also be implemented in other ways. For example, the division of units / modules in the above embodiments is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple units, modules, or components may be combined, or integrated into another system, or some features may be ignored or not executed.

[0156] Furthermore, unless otherwise specified, the functional units / modules in the various embodiments of this application can be integrated into one unit / module, or each unit / module can exist physically separately, or two or more units / modules can be integrated together. The integrated units / modules described above can be implemented in hardware or as software program modules.

[0157] When integrated units / modules are implemented in hardware, the hardware can be digital circuits, analog circuits, etc. The physical implementation of the hardware structure includes, but is not limited to, transistors, memristors, etc. Unless otherwise specified, the processor can be any suitable hardware processor, such as a CPU, GPU, FPGA, DSP, and ASIC, etc. Unless otherwise specified, the storage unit can be any suitable magnetic or magneto-optical storage medium, such as Resistive Random Access Memory (RRAM), Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), Enhanced Dynamic Random Access Memory (EDRAM), High-Bandwidth Memory (HBM), Hybrid Memory Cube (HMC), etc.

[0158] If the integrated unit / module is implemented as a software program module and sold or used as an independent financial product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software financial product. This computer software financial product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned memory includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0159] In the above embodiments, the descriptions of each embodiment have their own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification.

[0160] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims. It should be understood that this application is not limited to the precise structures described above and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A consistency verification method, characterized in that, include: Obtain committed transaction events and generate standardized event tuples, wherein the standardized event tuples include global commit sequence number, time window identifier, key space shard identifier, transaction intent summary, result set hash, and database instance identifier; A Merkel directed acyclic graph is constructed based on the standardized event tuples. The Merkel directed acyclic graph uses the time window identifier and the key space partition identifier as dimensions, and generates a window root and a global root through hierarchical aggregation. The window root is exchanged across database instances and a consistency verification is performed.

2. The method according to claim 1, characterized in that, The construction of the Merkel directed acyclic graph based on the standardized event tuples includes: The transaction events are divided according to the time window identifier and the key space fragmentation identifier to form a transaction event set; For each set of transaction events, the transaction events are arranged in chronological order as indicated by the global commit sequence number, and a leaf node hash is calculated. The leaf node hash is generated by aggregating the transaction intent summary, the result set hash, and the global commit sequence number. The leaf node hashes are aggregated from bottom to top to generate intermediate node hashes until the window root and the global root are formed.

3. The method according to claim 2, characterized in that, The bottom-up, step-by-step aggregation of the leaf node hashes to generate intermediate node hashes, until the window root and the global root are formed, includes: For any leaf node, the adjacent leaf nodes of the leaf node, the time window identifier of the transaction event, the key space shard identifier, the database instance identifier, and the metadata are concatenated to generate an intermediate node hash, until the window root and the global root are generated.

4. The method according to any one of claims 1-3, characterized in that, After exchanging the window root across database instances and performing consistency verification, the method further includes: If the window roots are inconsistent, a recursive binary search is performed along the time window identifier and the key space segment identifier to locate the differing leaf nodes.

5. The method according to claim 4, characterized in that, After locating the differential leaf node, the method further includes: Extract the redo log fragments and data block snapshots corresponding to the differing leaf nodes; Perform replay verification in an isolated environment and generate a minimal counterexample.

6. The method according to claim 5, characterized in that, The generation of the minimum counterexample includes: Extract the relevant information of the differential leaf node, including the global commit sequence number, key space shard identifier, time window identifier, and redo log offset corresponding to the differential leaf node; A minimum counterexample is generated based on the aforementioned relevant information.

7. The method according to any one of claims 1-3, characterized in that, The global commit sequence number is either the system change number or a hybrid logic clock.

8. The method according to any one of claims 1-3, characterized in that, Before generating the normalized event tuple, the method further includes: The key space sharding strategy is dynamically adjusted based on real-time load and transaction hotspot areas. The key space sharding strategy specifically includes one of the following: dividing the key space into shards according to transaction hotspot areas; dividing the key space into shards according to transaction conflict rate; and dividing the key space into shards according to transaction frequency.

9. A consistency verification device, characterized in that, include: The event tuple generation module is used to obtain committed transaction events and generate standardized event tuples. The standardized event tuples include a global commit sequence number, a time window identifier, a key space shard identifier, a transaction intent summary, a result set hash, and a database instance identifier. A directed acyclic graph (DAG) construction module is used to construct a Merkel directed acyclic graph based on the standardized event tuples. The Merkel directed acyclic graph uses the time window identifier and the key space partition identifier as dimensions, and generates a window root and a global root through hierarchical aggregation. The verification module is used to exchange the window root across database instances and perform consistency verification.

10. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1 to 8.

Images