A method for maintaining consistency of distributed data
By managing data shard versions and transaction confirmations under the repeatable read isolation level, the performance degradation of consistent reads in distributed databases is solved, and efficient consistent reads of multi-shard data are achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- DUXIAOMAN TECH (BEIJING) CO LTD
- Filing Date
- 2022-09-15
- Publication Date
- 2026-07-21
AI Technical Summary
Existing distributed databases suffer from performance degradation when ensuring consistent reads, especially at the serialization isolation level where SELECT operations are blocked, making efficient distributed consistent reads impossible.
Under the repeatable read isolation level, data consistency is ensured through steps such as initializing data shard versions, updating data shards, generating baselines, and reading data. This includes version identification management of data shards and transaction processing confirmation, ensuring that data processing results are only visible to other transactions after certain conditions are met.
It achieves distributed consistent reads of multi-sharded data under the repeatable read isolation level, avoiding blocking of SELECT operations and improving database performance.
Smart Images
Figure CN115470296B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of database and data processing technology, and in particular to a method for maintaining distributed data consistency. Background Technology
[0002] Databases, as enterprise data storage platforms, are becoming increasingly important in enterprise IT systems. With the continuous increase in business volume, the data volume inevitably increases as well. A single-machine MySQL database can no longer meet the storage needs of business operations. Distributed architecture MySQL databases based on the XA protocol solve the problem of insufficient storage capacity for single-machine databases. However, most distributed databases on the market currently do not support distributed consistent reads. Distributed consistent reads refer to the requirement that the aggregated data returned by a transaction across different shards must be consistent. To address this, the XA protocol requires distributed transactions to execute at the serializable isolation level on each object to ensure consistency in distributed reads. However, the serializable isolation level leads to performance degradation because even ordinary SELECT operations require an S lock. If other transactions have already acquired an X lock (exclusive lock, write lock) on the data, the SELECT operation will be blocked, preventing the non-blocking backtracking read of older versions via MVCC. The performance degradation caused by the serializable isolation level is unacceptable in production environments; in fact, native MySQL cannot achieve efficient distributed consistent reads. Summary of the Invention
[0003] Therefore, this invention provides a method for maintaining distributed data consistency, which can effectively solve the above problems. Through the technical solution of this invention, distributed consistent reads of multi-sharded data can be achieved under the repeatable read isolation level.
[0004] To achieve the objectives of this invention, the following technical solution is adopted:
[0005] A method for maintaining data consistency includes the following steps:
[0006] Step 1: Initialize the data shard version;
[0007] Step 2: Data sharding update;
[0008] Step 3: Generate a baseline;
[0009] Step 4: Read the data;
[0010] Step 1 includes:
[0011] (1) When all data shards in a data shard group are processed for the first time, each data shard is assigned a version identifier.
[0012] (2) Except for the above case (1), when processing the data shard group, the version identifier of all shards is obtained.
[0013] The method for maintaining data consistency includes step 2, which comprises: (1) data processing submission; and (2) transaction processing confirmation.
[0014] The method for maintaining data consistency, wherein the data processing includes adding, deleting, and modifying records in the data shards, but does not include read operations.
[0015] The method for maintaining data consistency, wherein step 2 of data sharding update is processed in the following manner:
[0016] Suppose that at the same time or within a certain time period, two transactions simultaneously access a data shard group and perform data processing on that data shard group.
[0017] Determine whether there are duplicate data shards in the target data shards to be processed by the two transactions. If there are no duplicate data shards in the target data shards to be processed by the two transactions, then determine whether at least one sub-transaction of a certain transaction has completed the data processing commit operation. The data processing commit operation is used to update the records of the target data shard and update the version identifier of the target data shard.
[0018] The method for maintaining data consistency includes: when there are duplicate data shards between the target data shards of two transactions, it is determined whether a transaction has performed a data processing commit operation on all its target data shards.
[0019] The method for maintaining data consistency, wherein: step 3 is executed only after the data shards in step 2 are updated, to generate a baseline, the baseline being a set of numbers including version identifiers of the data shard groups.
[0020] The method for maintaining data consistency includes step 3, which involves: determining whether a shard in the data shard group has undergone a version identifier update; if so, replacing the version identifier of the corresponding shard in the current baseline with the version identifier of the shard that underwent the version update, generating a new baseline and recording it.
[0021] The method for maintaining data consistency further includes step 4, reading data: when reading data, first determine the target data shard, then find the version identifier of the target data shard in the current baseline, and read the shard record of the target data shard through the version identifier to obtain the current data of the target data shard.
[0022] The method for maintaining data consistency further includes a cleanup operation.
[0023] A computer-readable storage device stores processor-executable instructions that, when executed by a processor, cause the processor to perform the method described above. Attached Figure Description
[0024] Figure 1 This is a schematic diagram of the method for maintaining distributed data consistency according to the present invention. Detailed Implementation
[0025] The following is in conjunction with the appendix Figure 1 The specific embodiments of the present invention will be described in detail below. Obviously, the described embodiments are merely a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. The embodiments described are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention. Obviously, the embodiments described in the present invention are merely a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0026] The terms "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of the invention include the specific features, structures, or characteristics described in connection with that embodiment. Therefore, the terms "comprising," "including," "having," and variations thereof in this specification mean "including but not limited to," unless otherwise specifically emphasized.
[0027] like Figure 1 As shown, the method for maintaining data consistency according to the present invention includes: 1. initializing data shard versions; 2. updating data shards; 3. generating a baseline; 4. reading data.
[0028] Data sharding can be one of multiple data blocks divided into a set of data according to certain rules. It can be multiple data tables in a database or multiple different databases. Multiple data shards are called a data shard group. The data sharding addressed in this invention can be, for example, a distributed architecture MySQL database; however, other distributed databases can also be applied to this invention.
[0029] In step 1 of this invention, initializing the data shard version includes two cases:
[0030] (1) When all data shards in a data shard group are processed for the first time, each data shard is assigned a version identifier (id), and the initial value of the version identifier of all data shards is set to the same value, such as 0.
[0031] (2) In addition to the above situation (1), when processing the data shard group, the latest baseline is read first to obtain the version identifier (id) of all shards.
[0032] The version identifier (id) is stored in the database.
[0033] Step 2: In order to achieve the goal of maintaining distributed data consistency, the present invention makes special provisions for the data sharding update in step 2, that is, the data sharding update is divided into two sub-steps: (1) data processing submission; (2) transaction processing confirmation.
[0034] Transaction confirmation can only occur after the data processing on a data shard has been committed. This data processing includes modification operations such as insert, delete, and update on the data shard, but excludes read operations. Specifically, it includes writing new data / records to the data shard, modifying existing data / records in the data shard, and deleting existing data / records in the data shard. The aforementioned data processing commit refers to actually writing the modifications to the data shard into the data shard.
[0035] The processing of data shards by a transaction is only visible to other transactions after the transaction is confirmed. Before the transaction is confirmed, the processing of data shards by a transaction is not visible to other transactions.
[0036] Step 2, data sharding update, is processed as follows:
[0037] Suppose that at the same time or within a certain time period, two transactions simultaneously access a data shard group and perform data processing on that data shard group.
[0038] To determine if there are duplicate data shards in the target data shards for processing by the two transactions, for example, if the data shard group includes 10 shards (1-10), and if the first transaction processes shards 1, 3, and 5, while the second transaction processes shards 2, 4, and 6, then there are no duplicate data shards between the two transactions. However, if the first transaction processes shards 1, 2, 3, and 5, while the second transaction processes shards 2, 4, and 6, then there is a duplicate data shard between the two transactions, namely shard 2. Each transaction's processing of a single shard is called a sub-transaction of that transaction. It can be understood that each transaction includes at least one sub-transaction.
[0039] (1) When there are no duplicate data shards between the target data shards to be processed by two transactions, it is determined whether at least one sub-transaction of a certain transaction has completed the data processing commit operation. If so, it is determined whether all sub-transactions have completed the data processing commit operation. If so, the transaction processing confirmation operation is performed. After the transaction processing confirmation operation is completed, the data processing results of all target data shards of the transaction and the version identifier of each shard are visible to other transactions. The version identifier can be used to generate the baseline in step 3.
[0040] The data processing and commit operation of the above sub-transaction not only updates the data / records of the target data shard, but also updates the version identifier of the target data shard, that is, increments the version identifier id of the target data shard by 1.
[0041] If not, wait for other sub-transactions to perform data processing and commit operations. During this period, the updated version identifier of the target data shard and the modified data / records of the sub-transactions that have already been executed are not visible to other transactions, but are only visible to other sub-transactions of this transaction.
[0042] Perform a similar operation on another transaction.
[0043] (2) When there are duplicate data shards between the target data shards to be processed by two transactions, it is determined whether a transaction has performed data processing commit operation and transaction processing confirmation operation for all the target data shards it is processing. If so, it is further determined whether the processing of all the target data shards to be processed by other transactions has not started or whether they have all performed data processing commit operation. If so, the data processing results of all the target data shards of the transaction and the version identifier of each shard are visible to other transactions, and the version identifier can be used to generate the baseline in step 3.
[0044] Otherwise, wait for other transactions to process until all other transactions have executed data processing commit operations and transaction processing confirmation operations for all target data shards they are processing. Only then will the data processing results of all target data shards of all transactions and the version identifiers of each shard be visible to other transactions and can be used to generate a baseline.
[0045] When a data shard version identifier is updated, step 3, generating a baseline, needs to be performed. A baseline is a set of numbers including the version identifier IDs of the data shard group. For example, if the data shard group includes 8 shards, the initial baseline (first baseline) is a combination of the original version identifier IDs of the 8 shards, such as [0,0,0,0,0,0,0,0]. When a shard, such as shard 1 or shard 5, undergoes a version update, its version identifier ID becomes 1. At this time, a second baseline is generated, that is, the version identifier ID of the corresponding shard in the initial baseline is replaced with the version identifier ID of the currently updated shard. The second baseline is represented as [1,0,0,0,1,0,0,0]. After the second baseline is generated, if shard 1 and shard 6 undergo a version update, their version identifier IDs become 2 and 1 respectively. At this time, a third baseline is generated, that is, the version identifier ID of the corresponding shard in the current baseline (second baseline) is replaced with the version identifier ID of the currently updated shard. The third baseline is represented as [2,0,0,0,1,1,0,0].
[0046] Baseline generation involves: determining whether a shard in the data shard group has undergone a version identifier update; if so, replacing the version identifier ID of the corresponding shard in the current baseline with the version identifier ID of the updated shard, generating a new baseline, and recording it, for example, by storing it in a database such as Redis. This new baseline then becomes the current baseline for the next shard processing operation or read operation.
[0047] When reading data in step 4, first determine the target data shard for the read operation, then find the version identifier of the target data shard in the current baseline, and read the shard record of the target data shard through the version identifier to obtain the current data of the target data shard.
[0048] In this invention, the method of data sharding update is strictly defined. That is, the data processing result can only be visible to other transactions after the transaction processing confirmation operation conditions specified in step 3 are met. Therefore, it can be guaranteed that the consistency of distributed data can be maintained when other transactions perform operations such as reading data.
[0049] In addition to the steps mentioned above, the method for maintaining distributed data consistency of the present invention also includes step 5, exception handling.
[0050] Here, "anomaly" refers to a scenario where the distributed transaction state is in an unexpected state due to network anomalies, program anomalies, or other abnormal conditions. When a transaction needs to build a new baseline, it needs to retrieve information about committed transactions from all databases (e.g., Redis). If there is transaction data in the database that has been in a non-final state for a long time, it will affect the progress of baseline generation. In this case, it is considered an anomaly, and step 5, anomaly handling, needs to be executed. The specific steps are as follows:
[0051] a. Lock preemption to prevent multiple exception handling processes from executing concurrently.
[0052] b. Traverse the transaction information stored in the Redis database. If a transaction x is found to be in a non-final state (start state or prepare state) for a long time (more than 1 second), then proceed to step c.
[0053] c. Inspect each (MySQL) data shard to confirm whether there are any lingering sub-transactions for that shard. If so, determine that the transaction x to which the sub-transaction belongs is a lingering transaction (a lingering transaction is one where there are lingering sub-transactions and other non-lingering sub-transactions are in the final state; a lingering sub-transaction is one where the sub-transaction is in the prepare (awaiting data processing and commit) state, and the client connection executing this sub-transaction has been disconnected from MySQL). If transaction x is a lingering transaction, proceed to step d. If transaction x is not a lingering transaction, proceed to step b to continue traversing transactions.
[0054] d. Handling pending matters.
[0055] Case 1: If transaction x involves sub-transactions on data shards, all of which are hanging sub-transactions, then rollback operations will be performed on the hanging sub-transactions;
[0056] Case 2: If transaction x involves sub-transactions on a data shard, including suspended sub-transactions, and other non-suspended sub-transactions are in the commit (data processing commit) final state, then perform a data processing commit operation on the suspended sub-transactions.
[0057] Case 3: If transaction x involves sub-transactions on a data shard, including suspended sub-transactions, and other non-suspended sub-transactions are in the rollback final state, then the suspended sub-transactions will be rolled back.
[0058] e. Change the transaction data in the (Redis) database to the final state. Execute step b to continue traversing the transactions until the traversal is complete.
[0059] The above are the execution steps for one exception handling. In a real production environment, these steps will be executed repeatedly, with a certain time interval between the two executions.
[0060] The method for maintaining distributed data consistency according to the present invention further includes step 6, a cleanup operation, the specific steps of which are as follows:
[0061] (1) Each time baseline a is bound, the reference count count_a corresponding to baseline a is incremented by 1;
[0062] (2) After transaction x, which is bound to baseline a, commits, the reference count count_a is decremented by 1;
[0063] (3) Scan from old to new according to the construction order of the baselines. If the baseline is not the latest baseline and the reference count is 0, then the baseline can be cleaned up.
[0064] (4) Traverse each data shard and execute the cleanup command according to the version ID corresponding to each data shard in the baseline;
[0065] (5) After each data shard receives the cleanup command, it cleans up the expired version ID and expired data records.
[0066] According to one embodiment of the present invention, a computer-readable storage medium is provided. Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor. Therefore, embodiments of this application provide a storage medium storing a plurality of instructions that can be loaded by a processor to execute the steps of any of the methods for maintaining distributed data consistency provided in the embodiments of this application.
[0067] The storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc. More specifically, it may include specific categories such as static random-access memory (SRAM) and dynamic random-access memory (DRAM).
[0068] Since the instructions stored in the storage medium can execute the method of maintaining distributed data consistency of the present invention, the beneficial effects that can be achieved by the embodiments of the present invention can be realized, as detailed in the preceding embodiments, and will not be repeated here.
[0069] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.
Claims
1. A method for maintaining data consistency, characterized in that... Includes the following steps: Step 1: Initialize the data shard version; Step 2: Data sharding update; Step 3: Generate a baseline; Step 4: Read the data; Step 1 includes: (1) When all data shards in a data shard group are processed for the first time, each data shard is assigned a version identifier, and the initial value of the version identifier of all data shards is set to the same value. (2) Except for the above case (1), when processing the data shard group, obtain the version identifier of all shards; Step 2, data sharding update, is processed as follows: Suppose that at the same time or within a certain time period, two transactions simultaneously access a data shard group and process data on that data shard group. Determine whether there are duplicate data shards in the target data shards to be processed by the two transactions. If there are no duplicate data shards in the target data shards to be processed by the two transactions, then determine whether at least one sub-transaction of a certain transaction has completed the data processing commit operation. The data processing commit operation is used to update the records of the target data shard and update the version identifier of the target data shard.
2. The method for maintaining data consistency according to claim 1, characterized in that... Step 2 includes: (1) data processing submission; (2) transaction processing confirmation.
3. The method for maintaining data consistency according to claim 2, characterized in that: The data processing includes adding, deleting, and modifying records in the data slices, but does not include read operations.
4. The method for maintaining data consistency according to claim 1, characterized in that: If there are duplicate data shards between the target data shards of two transactions, then it is determined whether a transaction has performed data processing commit operations on all of its target data shards.
5. The method for maintaining data consistency according to claim 1, characterized in that: Step 3 is executed only after the data shards in step 2 are updated, in order to generate a baseline, which is a set of numbers including the version identifiers of the data shard groups.
6. The method for maintaining data consistency according to claim 1, characterized in that... Step 3 includes: determining whether the version identifier of a shard in the data shard group has been updated; if so, replacing the version identifier of the corresponding shard in the current baseline with the version identifier of the shard that has been updated, generating a new baseline and recording it.
7. The method for maintaining data consistency according to claim 1, characterized in that... It also includes step 4, reading data: When reading data, first determine the target data shard, then find the version identifier of the target data shard in the current baseline, and read the shard record of the target data shard through the version identifier to obtain the current data of the target data shard.
8. The method for maintaining data consistency according to claim 1, characterized in that... Also includes: Cleanup operation.
9. A computer-readable storage device, characterized in that, The device stores processor-executable instructions that, when executed by the processor, cause the processor to perform the method according to any one of claims 1 to 8.