Data sharding processing method and apparatus, device, storage medium, and program product

Abstract

The present application relates to the technical field of distributed storage, and in particular to a data sharding processing method and apparatus, a device, a storage medium, and a program product. The method comprises: in response to a split instruction for a first data shard in a database, creating a second data shard; controlling transactions having not been committed in the first data shard to be suspended from entering a commit phase; in response to the completion of commit of transactions having entered the commit phase in the first data shard, controlling transactions having not been read and written in the first data shard to be suspended from entering a read-write phase; transferring first transaction data from the first data shard to the second data shard, wherein the first transaction data is data, among the data of a first transaction, corresponding to the second data shard obtained by splitting; and in response to the completion of transfer of the first transaction data, controlling the transactions in the first data shard to resume entering the read-write phase and / or the committee phase. The present application can reduce the transaction blocking time while ensuring the execution efficiency of distributed transactions.

Description

Data fragmentation processing method, device, equipment, storage medium and program product

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on November 6, 2023, with application number 2023114701631 and application name “Data sharding processing method, device, equipment, storage medium and program product”, all contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of distributed storage technology, and in particular to data sharding processing technology. Background Art

[0003] In a system that supports distributed transactions, timely scheduling of data distribution can disperse sudden hotspots and achieve load balancing.

[0004] In related technologies, administrators can manually modify data distribution strategies; or they can automatically achieve load balancing by splitting overloaded data shards during blocking transactions.

[0005] However, the manual method in the related art involves a large amount of data migration, which is prone to errors and affects transaction processing; the automatic method in the related art splits overloaded data shards, which will prolong the blocking transaction time and easily affect the read, write and commit performance of the transaction.

[0006] Summary of the Invention

[0007] The present application provides a data sharding processing method, apparatus, device, storage medium, and program product, which can shorten the time of blocking transactions while ensuring the execution efficiency of distributed transactions; the technical solution is as follows.

[0008] According to one aspect of the present application, a data sharding processing method is provided, which is executed by a computer device, and the method includes:

[0009] In response to a split instruction for a first data shard in a database, creating a second data shard; the split instruction is used to instruct to split data in the first data shard to the second data shard;

[0010] Controlling uncommitted transactions in the first data shard to suspend and enter the commit phase;

[0011] In response to the completion of the transaction in the first data shard that has entered the commit phase, controlling the unread and unwritten transactions in the first data shard to pause and enter the read and write phase;

[0012] Transferring first transaction data from the first data shard to the second data shard; the first transaction data is the data of the first transaction corresponding to the second data shard after the split; the first transaction is an active transaction on the first data shard that needs to transfer data to the second data shard, and the active transaction is a transaction in the read / write phase;

[0013] In response to completion of the first transaction data transfer, the transaction recovery in the first data shard is controlled to enter the read-write phase and / or the commit phase.

[0014] According to one aspect of the present application, a data sharding processing device is provided, the device comprising:

[0015] a creation module, configured to create a second data shard in response to a split instruction for a first data shard in a database; the split instruction is configured to instruct data in the first data shard to be split into the second data shard;

[0016] A first suspension module, configured to control uncommitted transactions in the first data shard to suspend and enter a commit phase;

[0017] a second pausing module, configured to control unread and unwritten transactions in the first data shard to pause and enter the read and write phase in response to the completion of the transaction in the first data shard that has entered the commit phase;

[0018] a transfer module, configured to transfer first transaction data from the first data shard to the second data shard; the first transaction data is data in the first transaction corresponding to the second data shard after the split; the first transaction is an active transaction in the first data shard whose data needs to be transferred to the second data shard, and the active transaction is a transaction in the read / write phase;

[0019] A recovery module is used to control the transaction recovery in the first data shard to enter the read-write phase and / or the commit phase in response to the completion of the first transaction data transfer.

[0020] According to another aspect of the present application, a computer device is provided, comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the data sharding processing method described above.

[0021] According to another aspect of the present application, a computer-readable storage medium is provided, in which at least one instruction, at least one program, a code set or an instruction set is stored. The at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement the data sharding processing method described above.

[0022] According to another aspect of the present application, a computer program product is provided, which includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. A processor reads and executes the computer instructions from the computer-readable storage medium to implement the data sharding processing method described above.

[0023] The technical solutions provided by the embodiments of the present application may have the following beneficial effects:

[0024] The solution shown in the embodiment of the present application is to create a second data shard in response to a split instruction for the first data shard in the database; control the uncommitted transactions in the first data shard to suspend and enter the commit phase; in response to the completion of the transaction in the first data shard that has entered the commit phase, control the unread and written transactions in the first data shard to suspend and enter the read and write phase; transfer the first transaction data from the first data shard to the second data shard; in response to the completion of the first transaction data transfer, control the transaction in the first data shard to resume and enter the read and write phase and / or the commit phase. Before blocking the transaction, this solution first constructs the second data shard; at the same time, the blocking transaction is divided into two phases: controlling the uncommitted transactions to suspend and enter the commit phase, and controlling the unread and written transactions to suspend and enter the read and write phase. When the blocking transaction is committed, it does not affect the reading and writing of the transaction and the transaction that is about to be submitted. The present application can shorten the time of blocking transactions while ensuring the execution efficiency of distributed transactions. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0026] FIG1 is a diagram of a distributed database architecture of a traditional sharding architecture involving a database and a table;

[0027] FIG2 is a diagram of a distributed database architecture involved in this application;

[0028] FIG3 is a flow chart of a data sharding processing method provided by an exemplary embodiment of the present application;

[0029] FIG4 is a flow chart of a data sharding processing method provided by another exemplary embodiment of the present application;

[0030] FIG5 is a flowchart of a data sharding processing method provided by another exemplary embodiment of the present application;

[0031] FIG6 is a flowchart of a data sharding processing method provided by another exemplary embodiment of the present application;

[0032] FIG7 is a flowchart of a data shard processing method according to an exemplary embodiment of the present application that suspends a new transaction corresponding to a first data shard to enter a read / write phase;

[0033] FIG8 is a flowchart of a data sharding processing method provided by another exemplary embodiment of the present application;

[0034] FIG9 is a diagram of another distributed database architecture involved in this application;

[0035] FIG10 is a flowchart of an implementation method of a data sharding processing method provided by an exemplary embodiment of the present application;

[0036] FIG11 is a splitting flow chart provided by an exemplary embodiment of the present application;

[0037] FIG12 is a flow chart of a 2PC protocol involved in this application;

[0038] FIG13 is a schematic diagram of a 2PC case provided by an exemplary embodiment of the present application;

[0039] FIG14 is a schematic diagram of the submission of the 2PC case in FIG13 ;

[0040] FIG15 is a 2PC flow chart provided by an exemplary embodiment of the present application;

[0041] FIG16 is a block diagram of a data sharding processing apparatus according to an exemplary embodiment of the present application;

[0042] FIG17 is a structural block diagram of a computer device provided by an exemplary embodiment of the present application.

[0043] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application. DETAILED DESCRIPTION

[0044] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0045] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0046] The terms used in this disclosure are for the purpose of describing specific embodiments only and are not intended to limit the disclosure. As used in this disclosure and the appended claims, the singular forms "a," "an," "the," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0047] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, storage, and display, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use, and processing of relevant data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the attack operations and other object behaviors involved in this application were obtained with full authorization.

[0048] It should be understood that although the terms first, second, etc. may be used in this disclosure to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, a first parameter may also be referred to as a second parameter, and similarly, a second parameter may also be referred to as a first parameter without departing from the scope of this disclosure. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0049] The following are some definitions of terms involved in this application:

[0050] A database (DB) is a large, sharable collection of data organized into a structure and stored permanently on a computer. Each database has one or more application programming interfaces (APIs) for creating, accessing, managing, searching, and replicating the data stored within it.

[0051] Data shard: It is the smallest management unit of data and can be used for flexible data scheduling. For example, when the load on a node is particularly high, part of the data shard can be migrated to other nodes.

[0052] A write batch is a data structure used to store active transaction data. Each transaction has its own write batch, and the data stored in a write batch belongs to only one transaction. For distributed transactions, each transaction has its own write batch within the transaction context of each region.

[0053] Transaction data: It is part of the transaction context. When splitting a transaction, after sending the transaction data corresponding to the Old Region to the New Region, a transaction context is constructed on the New Region to ensure that both the Old Region and the New Region have corresponding transaction contexts.

[0054] Raft Group: The basic unit of a data synchronization group, consisting of multiple replicas, one of which is the leader and one or more of which are followers. The leader is responsible for synchronizing data with followers by sending a Raft log (also known as a proposal) to all followers. Once the Raft log is persisted to local disk by a majority of followers, the replicas can execute (apply) the Raft log, performing operations based on the specific contents of the Raft log.

[0055] Two-Phase Commit (2PC) is a common practice for ensuring the atomicity of distributed transaction commits. 2PC divides transaction commit into two phases: the Prepare Phase (P) and the Commit Phase (C). The initiator of a transaction, known as the Coordinator, typically has only one, and is responsible for deciding whether to commit or rollback the entire distributed transaction. The executors of a transaction, known as Participants, typically have multiple participants, each responsible for committing and rolling back their own local transactions.

[0056] Remote Procedure Call (RPC): A protocol for requesting services from a remote computer program over a network without requiring knowledge of the underlying network technology. RPC is a method of communication between modules, requesting services from a remote node over a network.

[0057] Query Per Second (QPS): Also known as the query rate per second, this is the number of queries a server can respond to per second, or the number of requests a server can process per second. QPS is a measure of the amount of traffic a specific query server can handle within a specified timeframe, and is also the maximum throughput capacity.

[0058] Please refer to FIG1 , which shows a distributed database architecture diagram of a traditional sharding database and sharding table architecture involved in this application.

[0059] As shown in Figure 1, a traditional distributed database with a sharded database and table architecture is a simple data distribution solution. Each underlying storage node is essentially a standalone database. For a large data table, the underlying layer will split the original table into multiple tables based on user-specified sharding keys or table keys, distributing them across different storage nodes. A proxy node sits above all storage nodes to perform routing and data aggregation, allowing users to query data without having to worry about the actual distribution of the underlying data.

[0060] Please refer to FIG2 , which shows a distributed database architecture diagram involved in this application.

[0061] As shown in Figure 2, the distributed database architecture divides the entire distributed database into the following three modules:

[0062] Storage cluster: stores KV data and divides the data into regions based on key values ​​to disperse data hotspots. Each data shard corresponds to a Raft Group to achieve master-slave data synchronization.

[0063] Computing cluster: Converts SQL statements for user data into KV requests and sends them to the storage cluster.

[0064] Centralized control cluster: Detects the data distribution of the storage cluster, directs the storage cluster to split, merge, and migrate data shards to ensure load balancing, and also provides global timestamp timing functions.

[0065] Each node in the storage cluster will regularly report the information of local data shards to the nodes in the central control cluster through heartbeat. If a node in the central control cluster finds that the size of a data shard exceeds the set threshold, it will send a task to split the data shard to the leader of the data shard. The task contains the range of the split data shard (including the range of the original data shard and the range of the newly split data shard). After receiving the task, the leader will submit a proposal in the Raft Group. When this proposal is synchronized to the majority of replicas, when each replica applies this proposal, it will each create a new data shard (New Region). When most replicas have created a New Region, the Raft Group corresponding to the underlying New Region has met the majority requirement of Raft. At this time, the leader can be elected and services can be provided.

[0066] The storage cluster uses "keywords" to distinguish each piece of data. For example, in a row-based database, each record stored in the storage cluster corresponds to a row of data in the table. For distributed databases with traditional sharded database and table architectures, the data's "keyword" can be the value of the primary key or sharded table key; for distributed databases based on the KV storage engine, the data's "keyword" can be the encoded key value. The data's "keyword" determines the interval into which the data is partitioned. Currently, common data partitioning schemes can be roughly divided into "partitioning based on keyword hash values" and "partitioning based on keyword intervals."

[0067] Keyword hash value partitioning uses a keyword hash function whose input is the keyword value and whose output is an unsigned integer. This allows the database system to establish a mapping from key values ​​to hash values. Each partition corresponds to a range of unsigned integers, and the ranges of the partitions do not overlap. If the hash value calculated by the hash function for a piece of data's keyword falls within the integer range of a partition, the data is stored in that partition.

[0068] The advantage of this data partitioning method is that, as long as the hash function is calculated reasonably enough, the storage system can evenly divide a large amount of data into multiple partitions, ensuring that the amount of data in each partition is almost the same. Therefore, this data partitioning method has a good effect of dispersing hot spots.

[0069] Key interval-based partitioning maps each partition directly to a continuous key value range, with no overlap between partition ranges. This partitioning method doesn't rely on a hash function and is relatively simple.

[0070] The advantage of "keyword-based partitioning" is that when performing range queries, you only need to find the data partitions within the query range and then perform data queries on only these partitions. This query method narrows the scope of the data query, improves query efficiency, and avoids unnecessary resource consumption.

[0071] The life cycle of a distributed transaction can be roughly divided into two phases: the read-write phase and the commit phase.

[0072] 1. Reading and Writing Stage

[0073] During this phase, users read and write data by issuing SQL requests to the database system. Data written during the read and write phase is considered uncommitted data before the transaction is committed. Due to transaction isolation requirements, this data cannot be accessed by other transactions.

[0074] In some distributed database systems, uncommitted transaction data is cached at the compute layer. Compute nodes only send transaction data to the corresponding data shards in the storage cluster before the transaction enters the commit phase. Because the storage cluster doesn't cache uncommitted transaction data, scheduling operations such as splitting and merging data shards within the storage cluster won't affect ongoing transactions, ensuring, to a certain extent, that data scheduling is imperceptible to users.

[0075] 2. Submission Phase

[0076] When a user issues a COMMIT SQL statement to the database system, the transaction enters the commit process. The database system then performs some checks on the transaction to determine whether committing the transaction will disrupt data consistency (for example, for transactions based on optimistic concurrency control, checking whether there are data conflicts with other transactions). The system also persists the data written by the transaction through logs (such as writing WAL logs to disk or submitting proposals to the Raft Group). If the transaction passes these checks, it is successfully committed, and the data written by the transaction becomes visible to other transactions.

[0077] In some distributed database systems, for simplicity, the atomicity of distributed transactions is guaranteed by the compute layer (e.g., using the Percolator commit protocol). In this architecture, the storage engine doesn't need to have a transactional concept; that is, it doesn't guarantee the atomicity of writing multiple KV data. This makes it easier for the storage engine to handle concurrent KV data writing and data sharding scheduling. If a KV write fails due to data sharding scheduling, a retry is required only for that single KV, without having to retry all data in the transaction.

[0078] Based on the analysis of the data partitioning strategies above, "partitioning based on keyword hash values" effectively disperses hotspots. However, its disadvantage is that range queries require access to all data partitions, which is inefficient. The disadvantage of "partitioning based on keyword intervals" is that data hotspots may still exist. If business access to data is concentrated within a small range, frequent data shard splits must be triggered to resolve the hotspot.

[0079] Considering that data query efficiency should be the top priority, the technical solution of the embodiment of the present application preferably adopts the "keyword interval division based" method.

[0080] Regarding the data hotspot problem, first, the business side can perform some random processing on the data. For example, for businesses that are prone to data hotspots, the business side can perform hash function calculations on the primary key column when writing data to ensure that the data can be distributed to different data shards when inserting data. Second, as long as the impact of data shard splitting on transactions is optimized to be low enough at the storage layer, the impact of data hotspots can be reduced by speeding up the frequency of data shard splitting. The data shard splitting idea provided by the technical solution of the embodiment of the present application achieves data splitting without business perception, which can address this problem.

[0081] While relatively simple to implement, distributed databases with traditional sharded database and table architectures lack effective data distribution detection methods, resulting in inflexible control over data partitioning and the inability to quickly manage sudden hotspot data. For example, when creating a table, the database administrator (DBA) must manually specify the sharding key. For sudden hotspots, the DBA must also manually modify the data distribution strategy to distribute data across adjacent nodes. This data dispersion approach not only consumes human resources but also involves extensive data migration, which can easily impact business operations.

[0082] Therefore, an optional data sharding strategy is: based on the new distributed database architecture, the central management cluster is used to control the splitting, merging, and migration of data shards, and load balancing can be automatically achieved without relying on the support of the DBA.

[0083] However, the existing data sharding solution still impacts business. As mentioned above, by synchronizing proposals within a Raft Group, a new Raft Group cannot be established until the proposal has been synchronized to the majority. After another round of elections, the new data shard can provide data write services. This period spans the Raft Group election timeout and typically takes several seconds. After synchronizing the proposal, the old data shard will no longer support writing data to the range corresponding to the new data shard. This means that the data range corresponding to the new data shard will be unavailable for these seconds, which can easily lead to a decrease in QPS.

[0084] For distributed databases based on the Percolator transaction protocol, since the storage cluster does not need to guarantee the atomicity of multiple data writes, it is simpler to handle the concurrency of data writes and data shard splits. However, the cost of this implementation is that the transaction must ultimately be written to the disk for each KV data. During the submission process, two additional KVs must be written to ensure the atomicity of the transaction (according to the requirements of the Percolator protocol, these three KV data are written to the data column family, the lock column family, and the write column family, respectively). This means that if Percolator is used to ensure the atomicity of transactions, the data communication volume between the computing cluster and the storage cluster, the number of proposals synchronized by the storage cluster's Raft Log, and the input / output (I / O) overhead of data writing to the disk will all increase by 2 to 3 times, resulting in poor submission performance.

[0085] In order to ensure the submission efficiency of distributed transactions, the technical solutions provided in the subsequent embodiments of this application completely sink distributed transactions to the storage cluster. For transactions in the read and write phases, their uncommitted data is maintained in the transaction context and managed in memory by the shard where the data is located. Under this design, this solution can still ensure that when the split of the data shard is triggered, the transaction can continue to execute, will not be rolled back, and the QPS will not fluctuate. Under the premise of ensuring the execution efficiency of distributed transactions, the data shard split is unaware of the business.

[0086] Please refer to Figure 3, which shows a flow chart of a data sharding processing method provided by an exemplary embodiment of the present application. The method is executed by a computer device. As shown in Figure 3, the method may include the following steps 310, 320, 330, 340, and 350.

[0087] Step 310: In response to a split instruction for a first data shard in a database, create a second data shard; the split instruction is used to instruct to split the data in the first data shard into the second data shard.

[0088] In an embodiment of the present application, when a split instruction for a first data shard in a database is received, the computer device may create a new second data shard at the storage cluster node of the first data shard.

[0089] The split instruction is used to instruct to split the data in the first data slice into a new second data slice, that is, to transfer part of the data in the first data slice to the second data slice.

[0090] The first data shard may be a master replica in a Raft Group, and the computer device may synchronize the split instruction to each slave replica corresponding to the first data shard.

[0091] The above-mentioned split instruction is used to indicate the splitting of data shards. For example, if a data shard on a node contains a particularly large amount of data, the computer device can use the split instruction to indicate that part of the data is to be migrated to other nodes. In this case, the data shard needs to be split on the current node.

[0092] On the one hand, the above-mentioned splitting instruction can be automatically sent to the storage cluster node where the data shard (Region) corresponding to the hot data is located after the central control cluster detects the hot data that needs to be migrated.

[0093] On the other hand, the above-mentioned split instruction can also be sent by the developer user to the central control cluster via a Hypertext Transfer Protocol (HTTP) request. Based on the HTTP request, the central control cluster sends the split instruction to the storage cluster node where the corresponding data shard (Region) is located.

[0094] In addition, developers can also manually enable or disable the automatic split function by sending HTTP requests to the central control cluster. After disabling it, the central control cluster will not automatically issue split commands.

[0095] In some embodiments, the above-mentioned split instruction can indicate the ID of the first data shard, the range of the first data shard, and the range of the new second data shard, including the split point (split key), the starting point (start key), the end point (end key), the data shard identification number (Region ID), etc.

[0096] For example, the range of the first data shard is [A, D), and a new second data shard is split with the split key B. Then, the range of the second data shard can be [B, D), that is, the start key is B and the end key is D; correspondingly, the range of the first data shard after the split can be [A, B).

[0097] In some embodiments, the above step 310 includes a pre-step of calculating the split key, specifically including:

[0098] S11: The central control cluster issues a split key calculation task to the first data shard to be split in the storage cluster. After the first data shard calculates the split key, it reports the split key to the central control cluster.

[0099] S12: After the central control cluster obtains the split key calculated for the first data shard, it generates a corresponding split instruction according to the split key, and then issues the split instruction to the first data shard and all slave copies of the first data shard.

[0100] Step 320: Control the uncommitted transactions in the first data shard to suspend and enter the commit phase.

[0101] In an embodiment of the present application, after creating a second data shard in response to a split instruction for the first data shard in the database through step 310, the computer device can control uncommitted transactions in the first data shard to suspend and enter the commit phase.

[0102] In the embodiment of the present application, after the second data shard is constructed, uncommitted transactions are controlled to pause and enter the commit phase, while transactions in the read / write phase and newly started transactions are not affected. This solution can shorten the time it takes for transactions executed based on the first data shard to enter the commit phase, reducing the impact of data shard splitting on transactions.

[0103] In some embodiments, step 320 may be preceded by the following steps:

[0104] S21: Construct a split task context for the first data shard; wherein the split task context includes some task information, such as the first data shard ID, the second data shard ID, the split key, etc.;

[0105] S22: Set the split task status to REGION_SPLIT_INIT;

[0106] S23: Send RPC to confirm that all slave copies of the first data shard have entered the REGION_SPLIT_INIT state.

[0107] Step 330: In response to the completion of the commit of the transaction in the first data shard that has entered the commit phase, control the unread / written transactions in the first data shard to pause and enter the read / write phase.

[0108] In an embodiment of the present application, after the transaction that has entered the commit phase in the first data shard is committed, the computer device can control the new transaction corresponding to the first data shard to suspend entering the read-write phase, that is, suspend the opening of the new transaction. The new transaction is the transaction in the first data shard that has not yet entered the read-write phase, but the above operation has no effect on the transaction that is already in the read-write phase.

[0109] In this embodiment of the present application, before performing the data migration in step 340, the transaction blocking process is divided into two phases: step 320, which blocks uncommitted transactions in the first data shard from entering the commit phase, and step 330, which, after transactions that have already entered the commit phase have completed their commit, suspends new transactions corresponding to the first data shard from entering the read / write phase. This solution can reduce the impact of data shard splitting on transactions.

[0110] Step 340: Transfer the first transaction data from the first data shard to the second data shard; the first transaction data is the data of the first transaction corresponding to the second data shard after the split; the first transaction is an active transaction in the first data shard that needs to transfer data to the second data shard, and the active transaction is a transaction in the read and write phase.

[0111] In an embodiment of the present application, the computer device can traverse the active transactions (i.e., transactions in the read and write phases) corresponding to the first data shard, and transfer the first transaction data from the first data shard to the second data shard through RPC.

[0112] In an embodiment of the present application, among the active transactions of the first data shard, only the transaction data of some of the active transactions may need to be transferred in whole or in part to the second data shard. Therefore, it is necessary to make a judgment based on the split instruction, that is, to traverse the data of each active transaction in the first data shard in order to filter out the transaction data that needs to be transferred to the second data shard.

[0113] For example, the active transactions on the first data shard include transactions 1, 2, and 3. The split instruction is used to instruct the data after the split key to be transferred to the second data shard. If the data of transaction 1 is before the split key, then the data of transaction 1 does not need to be transferred; if the data of transaction 2 is after the split key, then all the data of transaction 2 needs to be transferred to the second data shard; if the data of transaction 3 is partially before the split key and partially after the split key, then the part of the data in transaction 3 after the split key needs to be transferred to the second data shard.

[0114] Step 350: In response to the completion of the first transaction data transfer, control the transaction recovery in the first data shard to enter the read-write phase and / or the commit phase.

[0115] In an embodiment of the present application, after the data transfer is successfully completed through step 340, the data hotspots on the first data shard are dispersed, and the computer device can restore the new transaction corresponding to the first data shard to enter the read and write phase, and restore the transaction in the read and write phase to enter the commit phase.

[0116] In summary, the solution shown in the embodiment of the present application is to create a second data shard in response to a split instruction for the first data shard in the database; control the uncommitted transactions in the first data shard to suspend and enter the commit phase; in response to the completion of the transaction in the first data shard that has entered the commit phase, control the unread and written transactions in the first data shard to suspend and enter the read and write phase; transfer the first transaction data from the first data shard to the second data shard; in response to the completion of the first transaction data transfer, control the transaction in the first data shard to resume and enter the read and write phase and / or the commit phase. This solution constructs the second data shard before blocking the transaction; at the same time, the blocking transaction is divided into two phases: controlling the uncommitted transactions to suspend and enter the commit phase, and controlling the unread and written transactions to suspend and enter the read and write phase. When the blocking transaction is committed, it does not affect the reading and writing of the transaction and the transaction that is about to be submitted. This application can shorten the time of blocking transactions while ensuring the execution efficiency of distributed transactions.

[0117] Please refer to Figure 4, which shows a flow chart of a data sharding processing method provided by another exemplary embodiment of the present application. The method is executed by a computer device. As shown in Figure 4, step 340 in the embodiment shown in Figure 3 above can be implemented as steps 340-10, 340-20, 340-30, 340-40, and 340-50.

[0118] Step 340 - 10: Suspend the write operation of the active transaction.

[0119] In an embodiment of the present application, after controlling the unread and unwritten transactions in the first data shard to suspend and enter the read and write phase through step 330, the computer device may suspend the write operation of the active transaction corresponding to the first data shard; so as to facilitate subsequent data splitting.

[0120] During the write-ban period, if the storage cluster receives a write request from the computing cluster, a retry error code is returned.

[0121] Step 340-20: Create a first container for the first transaction and a second container for the first transaction in the first data shard.

[0122] In the embodiment of the present application, after step 340-10 suspends the write operation of the active transaction, the computer device can establish a first container for the first transaction and a second container for the first transaction in the first data shard to write the transaction data into the corresponding containers; that is, the above-mentioned first container and second container are used to store the data before the split key and after the split key in the first transaction, respectively.

[0123] In other embodiments, the computer device may also split the original write container of the first transaction on the first data shard into a first container and a second container, which are used to store the data before and after the split key in the first transaction, respectively.

[0124] Step 340-30: Write the second transaction data into the first container, and write the first transaction data into the second container; the second transaction data is the data of the first transaction corresponding to the first data shard after the split.

[0125] In the embodiment of the present application, after step 340-20 establishes the first container of the first transaction and the second container of the first transaction, the computer device traverses the transaction data in the read and write phases, writes the second transaction data into the first container, and writes the first transaction data into the second container.

[0126] That is, the transaction data after the split key is the first transaction data, and the transaction data before the split key is the second transaction data. If the data of transaction 1 is before the split key, then the data of transaction 1 belongs to the second transaction data and needs to be written to the first container; if the data of transaction 2 is after the split key, then the data of transaction 2 belongs to the first transaction data and needs to be written to the second container; if the data of transaction 3 is partially before the split key and partially after the split key, then the data after the split key in transaction 3 belongs to the first transaction data and needs to be written to the second container, and the data before the split key belongs to the second transaction data and needs to be written to the first container.

[0127] Step 340-40: Copy the second container to the second data shard, and build the transaction context of the first transaction on the second data shard.

[0128] In an embodiment of the present application, after step 340-30 writes the second transaction data into the first container and the first transaction data into the second container, the computer device may copy the second container to the second data shard and, on the second data shard, construct a transaction context for the first transaction based on the first transaction data in the second container; so that the second data shard can provide corresponding external services based on the transaction context of the first transaction.

[0129] In some embodiments, the computer device may send the data of the second container to the second data shard via RPC, and build a transaction context on the second data shard.

[0130] Step 340-50: In response to the second data being successfully copied to the second data shard and the transaction context of the first transaction being successfully constructed on the second data shard, the write operation of the active transaction is resumed.

[0131] In this embodiment of the present application, steps 340-40 successfully copy the second container to the second data shard and successfully build the transaction context of the first transaction on the second data shard. This means that the split is successful, and the computer device can resume the write operation of the active transaction corresponding to the first data shard to continue providing external services.

[0132] In an embodiment of the present application, an exemplary scheme for transferring first transaction data is provided, including establishing a first container for the first transaction and a second container for the first transaction in a first data shard during a write-disabled phase of an active transaction; then copying the second container to a second data shard, and building a transaction context for the first transaction on the second data shard. This scheme can transfer the first transaction data that needs to be transferred from the first data shard to the second data shard during the write-disabled phase of an active transaction, thereby avoiding conflicts between transaction data transfer and transaction write operations.

[0133] Please refer to Figure 5, which shows a flow chart of a data sharding processing method provided by another exemplary embodiment of the present application. The method is executed by a computer device. As shown in Figure 5, before step 340-50 in the embodiment shown in Figure 4 above, steps 340-42 and 340-44 are also included.

[0134] Step 340-42: In response to the second data being successfully copied to the second data shard and the transaction context of the first transaction being successfully constructed on the second data shard, the first container in the first data shard is updated to the write container of the first transaction in the first data shard.

[0135] In this embodiment of the present application, if the second container is successfully copied to the second data shard through steps 340-40 and the transaction context of the first transaction is successfully constructed on the second data shard, the split can be considered successful, and the computer device can update the first container in the first data shard to the write container of the first transaction in the first data shard; so as to provide services for the first transaction through the first container.

[0136] In some embodiments, after the split is successful, the computer device can modify the end key information of the first data shard according to the split instruction such as the split key; and move the second data shard to the data shard list for external services.

[0137] Step 340-44: Release the second container in the first data shard and the original write container of the first transaction in the first data shard.

[0138] In an embodiment of the present application, after the first container in the first data shard is updated to the write container of the first transaction in the first data shard through steps 340-42, the computer device can release the second container in the first data shard and the original write container of the first transaction in the first data shard; resources can be released in a timely manner to avoid excessive memory consumption.

[0139] In this embodiment of the present application, for the first transaction on the first data shard, the first container is used as the new write container for the first transaction. Since the second container has been copied to the second data shard through steps 340-40 above, there is no need to retain the second container on the first data shard and the second container can be released directly.

[0140] In an embodiment of the present application, a supplementary solution before the above steps 340-50 is provided, including: before resuming the write operation of the active transaction: in response to the successful copying of the second data to the second data shard and the successful establishment of the transaction context of the first transaction on the second data shard, updating the first container in the first data shard to the write container of the first transaction in the first data shard; releasing the second container in the first data shard and the original write container of the first transaction in the first data shard; this solution can promptly update the write container of the first transaction and release redundant containers after the split is successful, thereby saving memory space in the first data shard.

[0141] In some embodiments, the method further comprises:

[0142] In response to the second data being successfully copied to the second data shard and the transaction context of the first transaction being successfully constructed on the second data shard, modifying information of the first data shard;

[0143] Move the second data shard to the data shard list for external services.

[0144] In an embodiment of the present application, after the split is successful, the computer device can modify the information of the first data shard, for example, modify the end key of the first data shard according to the split key; and move the second data shard to the list of data shards for external services, so that the newly created second data shard can provide external services.

[0145] In an embodiment of the present application, a supplementary solution is provided after a successful split, including modifying the information of the first data shard and moving the second data shard to a list of data shards for external services; so that the second data shard can provide services externally and ensure the execution efficiency of distributed transactions.

[0146] In some embodiments, the method further comprises:

[0147] In response to a failure in replicating the second container and / or a failure in establishing a transaction context for the first transaction on the second data shard, deleting the first container and the second container;

[0148] Resuming write operations of active transactions in the first data shard;

[0149] Clean up the second data shard.

[0150] In an embodiment of the present application, if the process of writing the second transaction data to the first container and writing the first transaction data to the second container in the above step 340-30 fails, or the process of copying the second container to the second data shard in the above step 340-40 fails, or the process of building the transaction context of the first transaction on the second data shard fails; the computer device may consider that the split has failed. At this time, it is necessary to delete the first container and the second container in time, restore the write operation of the active transaction in the first data shard, and clean up the second data shard.

[0151] In an embodiment of the present application, a supplementary solution is provided after a split failure, including deleting the first container and the second container, restoring the write operation of the active transaction in the first data shard, and cleaning up the second data shard; this solution can promptly clean up the relevant data of the failed split and avoid useless data occupying database resources.

[0152] Based on the scheme shown in Figure 3 or Figure 4 above, please refer to Figure 6, which shows a flow chart of a data sharding processing method provided by another exemplary embodiment of the present application. The method is executed by a computer device. As shown in Figure 6, step 330 in the embodiment shown in Figure 3 or Figure 4 above can be implemented as steps 330a, 330b, 330c, and 330d.

[0153] Step 330a: In response to the completion of the transaction that has entered the commit phase in the first data shard, a split log for the first data shard is created.

[0154] In an embodiment of the present application, after the transaction that has entered the commit stage in the first data shard is committed, the computer device can update the status of the first data shard from L_WORKING to L_SPLIT; wait for the split log of the first data shard from the replica; and establish the split log of the first data shard.

[0155] In an embodiment of the present application, the split log of the slave replica of the first data shard is synchronized to the split log of the first data shard, with the purpose of ensuring that all replicas are replayed to the same split index before the second data shard provides services.

[0156] Submitted transaction data, including inactive transaction data, will be written to the corresponding node's storage instance. Since all regions on the same node share the same storage instance, the newly created second data shard can also access the inactive transaction data of the first data shard, and this transaction data does not need to be transferred.

[0157] The data of active transactions is not written to the storage instance and can only be accessed by the current Region. Therefore, it needs to be transferred from the first data shard to the newly created second data shard so that the corresponding transactions can continue to execute.

[0158] The storage instance (RocksDB) is a module within the storage cluster that stores data. It includes the Memtable and SSTable data structures. The data in the Memtable is stored in memory and is not persisted.

[0159] Step 330b: Persistently store the cached data in the first data shard on disk.

[0160] In the embodiment of the present application, after step 330a creates the split log of the first data shard, the computer device persistently stores the cached data in the first data shard on the disk.

[0161] In an embodiment of the present application, the data cached in the first data shard includes data of active transactions and data of inactive transactions that has not been flushed to the SSTable.

[0162] Optionally, the data on the disk does not need to be split because the first data shard and the newly created second data shard are on the same storage instance physical node. All regions on the same storage instance physical node will share the data on the disk, so there is no need to split the data on the disk.

[0163] Step 330c: Save a snapshot of the first data shard.

[0164] In the embodiment of the present application, after step 330b persistently stores the cached data in the first data shard to the disk, the computer device saves a snapshot of the first data shard to facilitate subsequent playback processing.

[0165] The first data shard saves a snapshot with a split index so that the first data shard no longer replays the log before the split index, thereby preventing the first data shard from replaying old data from the log and overwriting the new data written by the second data shard.

[0166] Taking Region 1 with the range [A,H) as an example, before the split, the log synchronized when the transaction commits on Region 1 may involve any data in [A,H).

[0167] Suppose that transaction T1 commits a change to E = 1. Then Region 1 splits into Region 2, and the range of Region 1 becomes [A, C) and the range of Region 2 becomes [C, H). At this time, another transaction T2 commits a change to E = 2 in Region 2.

[0168] From the above description, E=2 written by transaction T2 should be the latest version.

[0169] However, if Region 1 is allowed to replay logs before the split index, this is equivalent to allowing Region 1 to replay logs written by transaction T1. Because Region 1 and Region 2 logs are replayed concurrently during node startup, if the log of transaction T1 in Region 1 is replayed after the log of transaction T2 in Region 2, transaction T1's E=1 will be written to RocksDB after transaction T2's E=2. As a result, E=1 will become the latest version, leading to data errors.

[0170] Therefore, it is necessary to prevent Region 1 from replaying logs before the split index. This prevents the logs for transaction T1 from being replayed, ensuring that E=2 is the latest version. This is achieved by saving a snapshot of Region 1 as a split index.

[0171] In some embodiments, after saving the snapshot of the first data shard, the computer device may set the split task state to REGION_SPLIT_CREATE_NEW_REGION; and wait for the slave replica to enter the REGION_SPLIT_CREATE_NEW_REGION state.

[0172] Step 330d: Control the unread and unwritten transactions in the first data shard to suspend and enter the read and write phase.

[0173] In an embodiment of the present application, the computer device suspends the new transaction corresponding to the first data shard from entering the read and write phase, that is, prevents the start of a new transaction executed based on the first data shard.

[0174] In an embodiment of the present application, a supplementary solution to the above-mentioned step 330 is provided, including establishing a split log of the first data shard after the transaction that has entered the commit stage in the first data shard is committed, persistently storing the cached data in the first data shard to the disk, and saving a snapshot of the first data shard; this can facilitate the synchronization of all replica logs of the first data shard and the playback of the first data shard.

[0175] In some embodiments, the method further comprises:

[0176] In response to the split instruction, a fourth data shard corresponding to the third data shard is created; the third data shard is one of the slave copies of the first data shard;

[0177] Get the split log of the first data shard;

[0178] A snapshot is saved for the third data shard.

[0179] In an embodiment of the present application, in response to a split instruction for a first data shard in a database, a computer device creates a fourth data shard corresponding to a third data shard; obtains a split log of the first data shard, and saves a snapshot of the third data shard.

[0180] The third data shard is any one of the slave copies of the first data shard, and the third data shard and the first data shard synchronously respond to the split instruction to create a fourth data shard corresponding to the third data shard.

[0181] In an embodiment of the present application, a task execution scheme for the slave replica during the splitting process is provided, including obtaining the split log of the first data shard and saving a snapshot of the third data shard; so that the slave replica can timely back up the relevant data of the master replica.

[0182] Based on the above embodiment, please refer to Figure 7, which shows a flowchart of pausing new transactions corresponding to the first data shard to enter the read and write phase in a data shard processing method provided by an exemplary embodiment of the present application. The method is executed by a computer device, as shown in Figure 7, and the above step 330 can be implemented as step 3301:

[0183] Step 3301: In response to the completion of the transaction that has entered the commit phase and is executed based on the first data shard, and snapshots are saved for each slave copy of the first data shard, the non-read and non-written transactions in the first data shard are controlled to suspend entering the read and write phase.

[0184] In an embodiment of the present application, in the above step 330, the transaction executed based on the first data shard and entering the commit phase is committed, and snapshots are saved for each slave copy of the first data shard (such as the third data shard). Only then can the computer device suspend the new transaction executed based on the first data shard to enter the read and write phase.

[0185] In an embodiment of the present application, a scheme for executing tasks of slave replicas during the splitting process is provided, including pausing new transactions executed based on the first data shard to enter the read-write phase in response to the completion of the transaction that has entered the commit phase based on the first data shard and the respective slave replicas of the first data shard have saved snapshots; so that the slave replicas can timely back up the relevant data of the master replica and synchronously update the status of each slave replica.

[0186] In some embodiments, the method further comprises:

[0187] In response to the first data shard and the second data shard being successfully split, modifying information of the third data shard;

[0188] Move the fourth data shard to the data shard list for external services.

[0189] In an embodiment of the present application, based on the above-mentioned embodiment of creating a fourth data shard corresponding to the third data shard, if the first data shard and the second data shard are split successfully, that is, the first transaction data is successfully transferred from the first data shard to the second data shard, the computer device can modify the information of the third data shard, such as modifying the end key of the third data shard according to the split key; and move the fourth data shard to the data shard list for external services, so that the newly created fourth data shard can provide external services.

[0190] In an embodiment of the present application, a plan for executing tasks from the replica after a successful split is provided, including modifying the information of the third data shard and moving the fourth data shard to the list of data shards for external services; so that the fourth data shard can provide services externally and ensure the execution efficiency of distributed transactions.

[0191] In some embodiments, the method further comprises:

[0192] In response to a failure in splitting the first data shard and the second data shard, clearing the fourth data shard.

[0193] In an embodiment of the present application, based on the above-mentioned embodiment of creating a fourth data shard corresponding to the third data shard, if the splitting of the first data shard and the second data shard fails, that is, the transfer of the first transaction data from the first data shard to the second data shard fails, the computer device can clean up the fourth data shard.

[0194] In an embodiment of the present application, a solution for executing tasks from a replica after a split failure is provided, including cleaning up the fourth data shard; this solution can clean up the relevant data of the split failure in a timely manner to avoid useless data occupying database resources.

[0195] Based on the solution shown in FIG4 above, please refer to FIG8, which shows a flow chart of a data sharding processing method provided by another exemplary embodiment of the present application. The method is executed by a computer device, as shown in FIG8. Before steps 340-50 in the embodiment shown in FIG4 above, steps 340-46 are also included.

[0196] Step 340-46: Mark the first transaction as a split transaction, and in the transaction context of the first data shard, add the identifier of the second data shard to the data shard identifier corresponding to the first transaction.

[0197] In an embodiment of the present application, before resuming the write operation of the active transaction in step 340-50, the computer device may mark the first transaction as a split transaction and record the ID of the second data shard in the context of the first transaction corresponding to the first data shard; thereby avoiding omitting the new second data shard when the transaction corresponding to the second data shard is committed, and ensuring the atomicity of the transaction submission.

[0198] The method further includes:

[0199] In response to the first data shard receiving a preparation request for a third transaction, where the third transaction is marked as a split transaction, querying a data shard identifier corresponding to the third transaction in a transaction context of the first data shard;

[0200] In response to querying that the data shard identifier corresponding to the third transaction includes the identifier of the second data shard, a preliminary response is returned to the coordinator data shard of the third transaction, and the preliminary response includes the identifier of the second data shard, so that the coordinator data shard adds the identifier of the second data shard as the participant data shard identifier of the third transaction.

[0201] In an embodiment of the present application, when a first data shard receives a preparation request for a third transaction, and the third transaction is a transaction marked as split: the computer device may query the data shard ID corresponding to the third transaction in the transaction context of the first data shard;

[0202] When it is found that the data shard ID corresponding to the third transaction includes the ID of the second data shard: the computer device may return a preliminary response to the coordinator data shard of the third transaction.

[0203] The preparation response includes the ID of the second data shard, so that the coordinator data shard adds the ID of the second data shard as the ID of the participant data shard of the third transaction.

[0204] In an embodiment of the present application, a submission scheme for transactions marked as split is provided, including marking a first transaction as a split transaction before resuming the write operation of the active transaction, and adding the ID of the second data shard to the data shard ID corresponding to the first transaction in the transaction context of the first data shard; when the first data shard receives a preparation request for a split third transaction, the data shard ID corresponding to the third transaction is queried in the transaction context of the first data shard; when it is found that the data shard ID corresponding to the third transaction includes the ID of the second data shard, a preparation response is returned to the coordinator data shard of the third transaction, and the preparation response includes the ID of the second data shard, so that the coordinator data shard adds the ID of the second data shard as the participant data shard ID of the third transaction; this scheme can find all data shards involved in the current transaction to be submitted in the submission process of the split transaction, avoid omission of the split data shard, and ensure the integrity of the transaction submission.

[0205] The above technical solutions of the embodiments of the present application can be applied to another distributed database. Please refer to Figure 9, which shows another distributed database architecture diagram involved in this application. As shown in Figure 9, the distributed database includes a computing cluster (SQLEngine), a storage cluster, and a central control cluster (MetaCluster, MC).

[0206] SQLEngine: Computing engine node, mainly performs Structured Query Language (SQL) layer computing functions.

[0207] Storage cluster: Storage engine nodes, mainly completing transactions and key-value (KV) storage functions.

[0208] MC: Cluster management node, manages data scheduling for the entire cluster and is also responsible for assigning globally unique timestamps.

[0209] The data sharding process based on the database shown in FIG9 can be as follows:

[0210] MC automatically sends data shard splits

[0211] When MC detects hotspot data and needs to migrate the hotspot data:

[0212] First, a splitting instruction (split region job) is issued to the storage cluster node where the data shard (Region) is located. The job contains a split point (split key);

[0213] After receiving the job, the storage cluster will move the right boundary of the current data shard (Old Region) to the left to the split key, and create a new data shard (New Region) at the same time. The left boundary of the New Region is the split key, and the right boundary is the original right boundary of the Old Region.

[0214] In this way, the hot data is split into two data shards.

[0215] The MC then issues a migration task (migrate region job) to migrate one of the regions to another relatively idle storage cluster node, thus completing load balancing.

[0216] Users can also manually enable or disable the automatic region splitting function by sending an HTTP request to MC. If disabled, MC will not automatically issue split region jobs.

[0217] The HTTP request is as follows:

[0218] curl-XPUT

[0219] http: / / {mc_ip}:{mc_port} / meta-cluster / api / cluster / enable-split-region / {cluster_id} / {enable}

[0220] The parameters are described as follows:

[0221] mc_ip represents the IP address of the node where the MC is located, mc_port represents the port number where the MC receives HTTP requests, and enable represents whether to enable the function of automatically detecting the region status and issuing split tasks. 0 represents disable, and other positive integer values ​​represent enable.

[0222] The status of the automatic split function can be queried through the following HTTP request:

[0223] curl-XGET

[0224] http: / / {mc_ip}:{mc_port} / meta-cluster / api / cluster / is-enable-split-region / {cluster_id}

[0225] Manually trigger the splitting of data shards

[0226] By sending an HTTP request to MC, users can also manually request MC to issue a split region job to control the splitting of a region according to a specific split key.

[0227] The HTTP request is as follows:

[0228] curl-XPUT

[0229] http: / / {mc_ip}:{mc_port} / meta-cluster / api / region / split-region / {cluster_id} / {region_id} / {is_force}? key="{key}"

[0230] The API is used to manually trigger Region splitting and is therefore not affected by the split-region-enabled configuration parameter.

[0231] The key parameter can be used to specify a split point. If not specified, the storage cluster will calculate a split point. The key is a hexadecimal string.

[0232] Splitting the main process of the task (how to ensure that the transaction is not killed)

[0233] 1. Preliminary Steps

[0234] 1. MC sends a split key calculation task to the leader replica of the region to be split in the storage cluster. After the leader replica calculates the split key, it reports it to MC (if the storage cluster finds that it cannot find a suitable split point, it returns failure).

[0235] 2. After MC obtains the split key calculated by the storage cluster, it sends a split region job to all replicas of the region in the storage cluster.

[0236] The format of the split region job is, for example:

[0237] Among them, job_id uniquely identifies a data sharding task; RegionMeta contains information such as the region_id, start_key, and end_key of the Region; old_region_mata contains information about the Old Region before the split (start_key and end_key both correspond to the values ​​before the split); new_region_meta contains information about the New Region (start_key is the split_key, and end_key should be the same as the end_key of old_region_meta).

[0238] 2. The Leader replica receives the split task process issued by the MC

[0239] Please refer to Figure 10, which shows a flowchart of a data sharding method according to an exemplary embodiment of the present application. Please refer to Figure 11, which shows a splitting flowchart according to an exemplary embodiment of the present application. As shown in Figures 10 and 11, the splitting task process is as follows.

[0240] S1001: Create and start a new region based on the split key and other metadata. This metadata can be information contained in the split task context or information contained in the split task. The split task context contains some task information, such as the Old Region ID, New Region ID, and split key. See "S1101: Create a New Data Shard" in Figure 11.

[0241] S1002: Build the split task context, set the task status to REGION_SPLIT_INIT, and send an RPC to confirm that other Old Region copies have entered this status.

[0242] S1003: Notify the transaction module to prevent all active transactions that have not entered 2PC from entering the commit phase (ie, participants are not allowed to prepare).

[0243] S1004: Waiting for the transaction entering 2PC to end. Transactions in the read / write phase and newly opened transactions are not affected.

[0244] S1005: Update the current Old Region status from L_WORKING to L_SPLIT.

[0245] S1006: The Old Region synchronizes the region split raft log to ensure that all replicas are replayed to the same split index before the New Region provides services.

[0246] S1007: The data remaining in the memory of the Old Region is persistently stored on the disk, and a snapshot of the Raft Group of the Old Region is saved using the split index.

[0247] S1008: Set the split task status to REGION_SPLIT_CREATE_NEW_REGION.

[0248] S1009: Send RPC to confirm that other copies have entered the above state of step S1008.

[0249] S1010: Prevent new transactions from being opened, and traverse all active transactions, performing the following operations on each transaction:

[0250] a) Set a flag to disallow write operations such as put / delete. If a subsequent write request is received from SQLEngine, a retry error code will be returned.

[0251] b) Create old_region_writebatch and new_region_writebatch, traverse the uncommitted data in the transaction writebatch, and write it to the two newly created writebatches according to the split points;

[0252] c) The data in new_region_writebatch is sent to the leader of the new region via RPC, and a transaction context is established on the leader of the new region. Referring to step "S1102: Transferring Partial Transaction Data to the New Data Shard Based on the Split Point" in Figure 11, partial data of transaction A (Trans A) is transferred to the new data shard via RPC.

[0253] S1011: Reporting to the MC that the node split task is complete. After integrating the task status reported by all nodes, the MC responds to the task status of each node to decide whether to continue or roll back the cleanup process.

[0254] S1012-1: If MC decides to continue, the data shard split takes effect, and all transactions are traversed to take effect and continue to provide services:

[0255] a) After receiving the information, the node modifies the end key information of the Old Region based on the split key and other metadata, and moves the New Region to the data shard list for external services;

[0256] b) Iterate over all active transactions again and perform the following operations on each transaction:

[0257] i. Update old_region_writebatch to writebatch and make it effective, and release new_region_writebatch and old writebatch. Refer to step "S1103: If the split is successful, delete the data after the split point on the new data shard" in Figure 11;

[0258] ii. Set this transaction as a split transaction and record the split identifier (New Region ID). This step is to ensure the atomicity of transaction submission.

[0259] iii. Allow the transaction to perform write operations such as put / delete and allow the transaction to enter the committed state.

[0260] S1012-2: After traversing all transactions, new transactions are allowed to start, and the Region status is changed to L_WORKING, the split is completed, and the process ends.

[0261] S1013-1: If the MC decides to roll back, it cleans up the split transaction and traverses all transactions to make the split invalid and continue to provide external services:

[0262] After receiving the information, this node clears the New Region, traverses all active transactions, and performs the following operations on each transaction:

[0263] i. Release the newly created old_region_writebatch and new_region_writebatch, referring to the step "S1104: If the split fails, directly delete all data in the new data shard" in Figure 11 below;

[0264] ii. Allow the transaction to perform write operations such as put / delete and enter the committed state.

[0265] S1013-2: After traversing all transactions, allow new transactions to be started, change the region status to L_WORKING, complete the split rollback, and end the process.

[0266] In the above steps S1001, S1006, and S1007, if an exception occurs, the split task status is set to REGION_SPLIT_FAILED, and the MC exception reporting process is entered.

[0267] If other replicas are found to be abnormal in step S1010, the process of reporting the abnormality to the MC is directly entered. Since the operation cost of each transaction traversal by the Leader replica is relatively high, invalid execution should be avoided as much as possible.

[0268] 3. Follow receives the split task process issued by MC

[0269] As shown in Figure 10, the splitting task process is as follows:

[0270] S101: Create and start a New Region based on the split key and other metadata.

[0271] S102: Build the split task context and set the task status to REGION_SPLIT_INIT, confirming that all other Old Region replicas have entered this status.

[0272] S103: Update the current Old Region status from F_WORKING to F_SPLIT;

[0273] S104: Waiting for the split log from Apply Leader;

[0274] S105: Save a snapshot of the current Old Region;

[0275] S106: Set the split task status to REGION_SPLIT_NEW_REGION;

[0276] S107: Report the completion of the node split task to the MC. After integrating the task status reported by all nodes, the MC responds to the task status of each node to decide whether to continue or roll back the cleanup environment.

[0277] S108: If the MC decides to continue, after receiving the information, the node modifies the end key information of the Old Region according to the split key and other metadata, moves the New Region to the data shard list for external services, and changes the Region status to F_WORKING, completing the split and ending the process;

[0278] S109: If the MC decides to roll back, after receiving the information, this node clears the New Region and changes the Region status to F_WORKING, completing the split rollback and ending the process.

[0279] In summary, from a region's perspective, the transaction lifecycle can span splits, but the region split and transaction commit are mutually exclusive in time. Therefore, the region must prevent read-write transactions from entering the committed state and wait for transactions already in the committed state to complete their commit before initiating the split. Only after the split is the read-write transaction allowed to continue entering the committed state.

[0280] This limitation exists because, currently, when a storage cluster is restarted (recovered), multiple regions are replayed concurrently. The data accuracy here depends on the raft snapshot meta saved during the split. The snapshot index corresponding to this snapshot meta is the split index. After a region is started, log replay begins at the snapshot index. If a transaction log spans the split index (for example, if the prepare raft log is synchronized before the split index and the commit raft log is synchronized after the split index), then after the region is started, part of the transaction log will be missed and cannot be replayed, resulting in transaction data loss. Therefore, when synchronizing the split region log, it is necessary to ensure that all transactions in the commit process have completed.

[0281] As for the problem of QPS decline when traditional distributed databases split data into shards mentioned in the above "Distributed Database Architecture Selection", the corresponding solution of the technical solution of the embodiment of the present application is to create all replicas of the New Region before prohibiting transactions from writing in the Old Region. After ensuring that the New Region elects a Leader replica, the transaction data of the Old Region is split into the New Region. This can minimize the prohibition time and make it imperceptible to the business.

[0282] Transaction commit integrity is achieved as follows:

[0283] A distributed transaction has a transaction context across multiple regions in the storage cluster. To ensure the atomicity of transactions across multiple regions (either all commit or all rollback), the SQLEngine sends a commit request to one of the regions when committing a transaction. The request includes a list of regions that the SQLEngine has visited. The region that receives the commit request acts as the coordinator, ensuring the atomicity of the transaction commit through 2PC.

[0284] Please refer to Figure 12, which shows a flow chart of a 2PC protocol involved in this application. As shown in Figure 12, the 2PC protocol is completed by the coordinator and participants.

[0285] The coordinator enters the prepare phase and sends a prepare request to each participant;

[0286] After receiving the prepare request, the participant executes the submit response or does not submit the response. Submitting the response includes rewriting and entering the prepare phase after writing the prepare.

[0287] The coordinator collects the responses from each participant and determines whether perform fails:

[0288] If it fails, abort perpare; after sending abort feedback to all participants, all participants write abort;

[0289] If successful, it enters the commit phase and sends a commit request to each participant;

[0290] After receiving the commit request, each participant executes the response, including writing the commit and entering the commit phase;

[0291] The coordinator collects submission feedback from each participant and sends a confirmation request to each participant;

[0292] After receiving the confirmation request, the participant executes the response, including entering the confirmation phase after writing the confirmation and clearing the transaction context;

[0293] After the coordinator collects confirmation feedback from each participant, it clears the transaction context.

[0294] If Region A splits into Region B before a transaction is committed, but SQLEngine is unaware of the existence of Region B, the commit request will not include Region B, and the data in Region B will not be committed in the 2PC process, thus violating the transaction atomicity requirement.

[0295] In the aforementioned "Leader Replica Receives Split Task Process from MC," S1012-1 requires that all active transaction contexts in the Old Region be set to "set this transaction as a split transaction and record the ID of the New Region." The 2PC Prepare request issued by the coordinator includes the full list of participants. If the Old Region discovers that the 2PC Prepare request does not include the ID of the newly split New Region, it returns an error code, requiring the coordinator to supplement the participant list and reissue the 2PC Prepare request.

[0296] Please refer to Figure 13, which shows a schematic diagram of a 2PC case provided by an exemplary embodiment of the present application. As shown in Figure 13, a case is given in which transaction T writes four keys A, B, D, and G during the read and write phases, where A and B are on Region 1, and D and G are on Region 2. In this way, Region 1 and Region 2 will be included in the Region Cache of transaction T. Assume that transaction T stops reading and writing for a while at this time, and then directly executes commit. Coincidentally, in the gap where transaction T stops reading and writing, Region 2 splits, and the split creates a new Region 3. Note that the sub-transaction space of transaction T on Region 2 is marked with is_split=true, new_region=3. After the split, transaction T initiates a commit request and designates Region 1 as the coordinator.

[0297] Please refer to Figure 14, which shows a schematic diagram of the submission of the 2PC case in Figure 13. As shown in Figure 14, when Region 1 receives the transaction submission request from SQLEngine (where region_list is 1 and 2), the processing flow is as follows.

[0298] S1: The coordinator receives a transaction commit request and knows that the transaction participants are Region 1 and Region 2. It then sends a prepare request to Region 1 and Region 2. The prepare request includes a list of all participants known to the coordinator, Region 1 and Region 2.

[0299] S2: After receiving the prepare request, Region 1 checks the is_split variable in the subtransaction space of transaction T in this Region. The value of this variable is false, indicating that the Region has not been split during the life cycle of the subtransaction. Therefore, it directly executes prepare (writes logs and synchronizes them to other replicas). After success, it responds to the coordinator that preparation is complete (prepare ok).

[0300] S3: After receiving the prepare request, Region 2 checks the is_split variable in the subtransaction space of transaction T within this Region. If the value is true, this indicates that the Region split during the subtransaction's lifetime. It then checks to see if the new Region after the split is in the member list of the prepare request. The value of new_region is 3, which is not the case. Therefore, it returns a "prepare not ok" message to the coordinator and informs the coordinator of the value of the new Region. A corresponding error code must be set so that the coordinator can detect and handle this error. For example, the error code EC_TDS_TRANS_PREPARE_NEED_MORE_PART indicates that transaction prepare needs more participants, meaning that new participants need to be added to the participant list. When the coordinator receives this error code, it reads the more_participants field from the response, which contains information indicating that the ID of the new Region is 3.

[0301] S4: After receiving the response from Region 2, the coordinator finds the error code indicating a region split. It then retrieves the ID of the new region, 3, from the request, adds Region 3 to the participant list, and restarts a new prepare phase: it sends prepare requests to Regions 1, 2, and 3 simultaneously. The prepare request includes a list of all participants known to the coordinator: Regions 1, 2, and 3.

[0302] S5: After receiving the prepare request again, Region 1 finds that the status of transaction T in the sub-transaction space of this Region is already in the prepare state. After comparing the number of participants in the request with the number of participants it has recorded, it finds that there is a new one. It then records the newly added Region 3 and submits the log. After success, it responds to the coordinator with prepare ok again.

[0303] S6: After Region 2 receives the prepare request again, it checks the is_split variable of transaction T in the subtransaction space of this Region. The value of this variable is true, indicating that the Region has split during the life cycle of the subtransaction. It further checks whether the new Region after the split is in the member list of the prepare request. The value of new_region is 3. If it is in the member list, it executes prepare (writes logs and synchronizes them to other replicas) and responds to the coordinator with prepare ok after success.

[0304] S7: After receiving the prepare request, Region 3 checks the is_split variable in the subtransaction space of transaction T in this Region. The value of this variable is false, indicating that the Region has not been split during the life cycle of the subtransaction. Therefore, it directly executes prepare (writes logs and synchronizes them to other replicas) and responds to the coordinator with prepare ok after success.

[0305] S8: After receiving the prepare ok replies from Region 1, Region 2, and Region 3, the coordinator continues to push the 2PC transaction toward the commit state.

[0306] From the above process, we can see that when a Region in the coordinator's participant list is split and the split New Region is not in the participant list, an additional round of requests to fill in the participant list will be required.

[0307] If the New Region splits again after the coordinator completes the participant list, and a new Region appears, is another round of requests to complete the participant list needed? From a process design perspective, this is necessary, so the possibility of this actually happening can be limited here.

[0308] A feasible solution is to strictly limit the minimum protection time between the generation and splitting of the New Region, because transactions usually end soon after entering 2PC. In this way, when the minimum protection time is much longer than the commit time of general transactions after entering 2PC, there will almost never be a situation where more than two rounds of participants are required to make up the difference.

[0309] The minimum protection time mentioned above means that after the new region A is created, it must wait at least this minimum protection time before the new region A is allowed to process the split task and create the new region B. This time range can be in the second level, for example, 3 seconds, as long as it is significantly longer than the 2PC commit time (under low load conditions, the 2PC commit time is generally within 1 second).

[0310] The purpose of setting the above minimum protection time is to prevent the coordinator from frequently receiving the EC_TDS_TRANS_PREPARE_NEED_MORE_PART error code. For example:

[0311] After Region A splits off Region B, Region A receives a prepare request and sets this error code. It also places Region B in the more_participants field, informing the coordinator that Region B also needs to be included in the participant list and submitted. However, before the coordinator sends the prepare request to Region B, Region B splits off Region C, causing Region B to also set this error code and place Region C in the more_participants field. This results in excessive rounds of communication spent on expanding the participant list.

[0312] The minimum protection time allows Region B to wait for a while before processing the split request. Therefore, when Region B splits from Region C, the transaction's 2PC process has already completed, reducing the impact on transaction submission and allowing the transaction to complete the 2PC process quickly.

[0313] In addition, Region 1 in the above example will not split during this process because the transaction has successfully entered the prepare state on this Region, which will prevent Region 1 from splitting.

[0314] Please refer to FIG15 , which shows a 2PC flow chart provided by an exemplary embodiment of the present application.

[0315] As shown in Figure 15, the transaction participant list accurately requested by 2PC only includes data shard 1, but data shard 1 splits into data shard 2. In this case, an error is returned to inform the coordinator that the missing participant is missing, but it does not mean that the transaction submission has failed.

[0316] The coordinator updates the participant list and resends the prepare request. The transaction participant list includes data shard 1 and data shard 2.

[0317] In summary, the embodiments of the present application provide a transaction-aware method for splitting data shards. This method can quickly split data shards and disperse hotspot data in a system that supports distributed transactions. This data splitting method ensures the integrity of transaction submissions while maintaining a high splitting speed, ensuring no impact on business writes, no performance fluctuations, and no write failures. This method can help distributed storage systems achieve smooth load balancing without user awareness.

[0318] An embodiment of the present application provides a data shard splitting method that is unaware of data writing. This method is applied to a storage system that supports transaction functions or batch atomic write functions, and can ensure the correctness of data writing and data shard splitting concurrency, as well as the integrity of transaction submission.

[0319] This method shortens the time it takes to block transactions, does not cause a drop in QPS, and does not cause transaction rollbacks. It ensures that data shards are split and users are unaware of writes.

[0320] This method does not require operation and maintenance personnel to manually adjust the partition size. The control node will automatically combine the sizes of each partition to evaluate whether to issue a split task and automatically achieve load balancing.

[0321] Furthermore, the data sharding strategies described in the embodiments of this application are not limited to distributed database systems, and the underlying data units are not limited to KV data. For example, file storage and B-Tree-based table data storage can all use similar methods to implement data sharding, as long as the system can uniquely map data to the corresponding data shard.

[0322] The "synchronized raft log" mentioned in this application's examples is merely a means of synchronizing master and backup data within a Raft group. For storage systems that implement master-backup synchronization using the distributed consensus (Paxos) protocol, this can be replaced by "synchronizing Paxos resolutions." For standalone databases, this can be replaced by "storing a write-ahead log (WAL) to disk."

[0323] This embodiment of the application uses a "keyword interval-based partitioning" data partitioning method, specifying a split key to split a single region into two. However, for storage systems that use "keyword hash value-based partitioning," a split hash value can also be specified to split the data shards. Regardless of the data partitioning method, as long as the data can be split into two, this solution can be used.

[0324] FIG16 is a block diagram of a data sharding processing apparatus according to an exemplary embodiment of the present application. The apparatus may be used to execute all or part of the steps executed by a computer device in the method shown in FIG3 , FIG4 , FIG5 , FIG6 , FIG7 , or FIG8 . The apparatus includes:

[0325] A creation module 1601 is configured to create a second data shard in response to a split instruction for a first data shard in a database; the split instruction is configured to instruct data in the first data shard to be split into the second data shard;

[0326] A first pause module 1602 is used to control uncommitted transactions in the first data shard to pause and enter the commit phase;

[0327] The second suspension module 1603 is configured to control the unread and unwritten transactions in the first data shard to suspend and enter the read and write phase in response to the completion of the transaction in the first data shard that has entered the commit phase.

[0328] Transfer module 1604 is configured to transfer first transaction data from the first data shard to the second data shard; the first transaction data is data in the first transaction corresponding to the second data shard after the split; the first transaction is an active transaction in the first data shard whose data needs to be transferred to the second data shard, and an active transaction is a transaction in the read / write phase;

[0329] The recovery module 1605 is configured to control the transaction recovery in the first data shard to enter the read-write phase and / or the commit phase in response to completion of the first transaction data transfer.

[0330] In some embodiments, the transfer module 1604 is configured to:

[0331] Suspend write operations of active transactions;

[0332] Creating a first container for the first transaction and a second container for the first transaction in the first data shard;

[0333] Writing the second transaction data into the first container and writing the first transaction data into the second container; the second transaction data is the data of the first transaction corresponding to the first data shard after the split;

[0334] Copy the second container to the second data shard, and build a transaction context for the first transaction on the second data shard;

[0335] In response to the second data being successfully copied to the second data shard and the transaction context of the first transaction being successfully constructed on the second data shard, the write operation of the active transaction is resumed.

[0336] In some embodiments, the apparatus further comprises: an updating module configured to:

[0337] In response to the second data being successfully copied to the second data shard and the transaction context of the first transaction being successfully constructed on the second data shard, updating the first container in the first data shard to be the write container of the first transaction in the first data shard;

[0338] Release the second container in the first data shard and the original write container of the first transaction in the first data shard.

[0339] In some embodiments, the apparatus further comprises: a modification module configured to:

[0340] In response to the second data being successfully copied to the second data shard and the transaction context of the first transaction being successfully constructed on the second data shard, modifying information of the first data shard;

[0341] Move the second data shard to the data shard list for external services.

[0342] In some embodiments, the apparatus further comprises: a deletion module configured to:

[0343] In response to a failure in replicating the second container and / or a failure in establishing a transaction context for the first transaction on the second data shard, deleting the first container and the second container;

[0344] Resume write operations of active transactions;

[0345] Clean up the second data shard.

[0346] In some embodiments, the apparatus further comprises: an establishment module for,

[0347] In response to the completion of the transaction in the first data shard that has entered the commit phase, creating a split log for the first data shard;

[0348] Persistently store the cached data in the first data shard to disk;

[0349] A snapshot is saved for the first data shard.

[0350] In some embodiments, the apparatus further comprises: an acquisition module configured to:

[0351] In response to the split instruction, a fourth data shard corresponding to the third data shard is created; the third data shard is one of the slave copies of the first data shard;

[0352] Get the split log of the first data shard;

[0353] A snapshot is saved for the third data shard.

[0354] In some embodiments, the second pause module 1603 is used to:

[0355] In response to the completion of the transaction that has entered the commit phase and is executed based on the first data shard, and snapshots are saved for each slave copy of the first data shard, the unread and unwritten transactions in the first data shard are controlled to suspend entering the read and write phase.

[0356] In some embodiments, the apparatus further comprises: a moving module for:

[0357] In response to the first data shard and the second data shard being successfully split, modifying information of the third data shard;

[0358] Move the fourth data shard to the data shard list for external services.

[0359] In some embodiments, the apparatus further comprises: a cleaning module for:

[0360] In response to a failure in splitting the first data shard and the second data shard, clearing the fourth data shard.

[0361] In some embodiments, the apparatus further comprises: an adding module for:

[0362] Mark the first transaction as a split transaction, and in the transaction context of the first data shard, add the identifier of the second data shard to the data shard identifier corresponding to the first transaction;

[0363] The device also includes: a query module, configured to:

[0364] In response to the first data shard receiving a preparation request for a third transaction, where the third transaction is marked as a split transaction, querying a data shard identifier corresponding to the third transaction in a transaction context of the first data shard;

[0365] In response to the query that the data shard identifier corresponding to the third transaction includes the identifier of the second data shard, a preliminary response is returned to the coordinator data shard of the third transaction, and the preliminary response includes the identifier of the second data shard, so that the coordinator data shard adds the identifier of the second data shard as the participant data shard identifier of the third transaction.

[0366] FIG17 shows a block diagram of a computer device 1700 according to an exemplary embodiment of the present application. The computer device can be implemented as the server in the above-mentioned solution of the present application. The computer device 1700 includes a central processing unit (CPU) 1701, a system memory 1704 including a random access memory (RAM) 1702 and a read-only memory (ROM) 1703, and a system bus 1705 connecting the system memory 1704 and the central processing unit 1701. The computer device 1700 also includes a mass storage device 1706 for storing an operating system 1709, application programs 1710, and other program modules 1711.

[0367] The mass storage device 1706 is connected to the central processing unit 1701 via a mass storage controller (not shown) connected to the system bus 1705. The mass storage device 1706 and its associated computer-readable media provide non-volatile storage for the computer device 1700. In other words, the mass storage device 1706 may include a computer-readable medium (not shown) such as a hard disk or a Compact Disc Read-Only Memory (CD-ROM) drive.

[0368] Without loss of generality, the computer-readable medium may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer-readable instructions, data structures, program modules or other data. Computer storage media include RAM, ROM, Erasable Programmable Read Only Memory (EPROM), Electronically Erasable Programmable Read-Only Memory (EEPROM), flash memory or other solid-state storage technologies, CD-ROM, Digital Versatile Disc (DVD) or other optical storage, tape cassettes, magnetic tape, disk storage or other magnetic storage devices. Of course, those skilled in the art will appreciate that the computer storage medium is not limited to the above-mentioned ones. The above-mentioned system memory 1704 and mass storage device 1706 can be collectively referred to as memory.

[0369] According to various embodiments of the present disclosure, the computer device 1700 can also be connected to a remote computer on a network such as the Internet for operation. That is, the computer device 1700 can be connected to the network 1708 via the network interface unit 1707 connected to the system bus 1705, or the network interface unit 1707 can be used to connect to other types of networks or remote computer systems (not shown).

[0370] The memory further includes at least one computer program, which is stored in the memory. The central processing unit 1701 implements all or part of the steps in the methods shown in the above embodiments by executing the at least one computer program.

[0371] In an exemplary embodiment, a chip is also provided. The chip includes a programmable logic circuit and / or program instructions. When the chip runs on a computer device, it is used to implement the data sharding processing method in the above aspect.

[0372] In an exemplary embodiment, a computer program product is also provided. The computer program product includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions to implement the data sharding processing methods provided in the above-described method embodiments.

[0373] In an exemplary embodiment, a computer-readable storage medium is further provided, in which a computer program is stored. The computer program is loaded and executed by a processor to implement the data sharding processing method provided by the above-mentioned method embodiments.

[0374] Those skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware, or by a program to instruct the relevant hardware, and the program may be stored in a computer-readable storage medium, which may be a read-only memory, a disk, or an optical disk, etc.

[0375] Those skilled in the art will appreciate that in one or more of the above examples, the functions described in the embodiments of the present application can be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein communication media include any media that facilitates the transmission of computer programs from one place to another. The storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0376] The above description is merely an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A data sharding processing method, executed by a computer device, comprising: In response to a split instruction for a first data shard in the database, creating a second data shard; The split instruction is used to instruct to split the data in the first data slice into the second data slice; Controlling uncommitted transactions in the first data shard to suspend and enter the commit phase; In response to the transaction in the first data shard that has entered the commit phase being committed, controlling the unread and unwritten transactions in the first data shard to pause and enter the read and write phase; Transferring first transaction data from the first data shard to the second data shard; The first transaction data is data in the first transaction corresponding to the second data shard after the split; The first transaction is a transaction in an active transaction of the first data shard that needs to transfer data to the second data shard, and the active transaction is a transaction in the read-write phase; In response to completion of the first transaction data transfer, the transaction recovery in the first data shard is controlled to enter the read-write phase and / or the commit phase.

2. The method according to claim 1, wherein transferring the first transaction data from the first data shard to the second data shard comprises: Suspending the write operation of the active transaction; Establishing a first container for the first transaction and a second container for the first transaction in the first data shard; Writing the second transaction data into the first container, and writing the first transaction data into the second container; The second transaction data is data in the first transaction corresponding to the first data shard after the split; Copying the second container to the second data shard, and building a transaction context of the first transaction on the second data shard; In response to the second data being successfully copied to the second data shard and the transaction context of the first transaction on the second data shard being successfully constructed, the write operation of the active transaction is resumed.

3. The method according to claim 2, characterized in that Before resuming the write operation of the active transaction, the method further includes: In response to the second data being successfully copied to the second data shard and the transaction context of the first transaction being successfully constructed on the second data shard, updating the first container in the first data shard to be a write container of the first transaction in the first data shard; The second container in the first data shard and the original write container of the first transaction in the first data shard are released.

4. The method according to claim 2 or 3, further comprising: In response to the second data being successfully copied to the second data shard and the transaction context of the first transaction being successfully constructed on the second data shard, modifying information of the first data shard; Move the second data shard to the data shard list for external services.

5. The method according to any one of claims 2 to 4, further comprising: In response to a failure in replicating the second container and / or a failure in building a transaction context of the first transaction on the second data shard, deleting the first container and the second container; Resuming the write operation of the active transaction; Clean up the second data shard.

6. The method according to any one of claims 1 to 5, wherein in response to the completion of the transaction in the first data shard that has entered the commit phase, controlling the unread and unwritten transactions in the first data shard to pause and enter the read and write phase comprises: In response to the transaction in the first data shard that has entered the commit phase being committed, creating a split log for the first data shard; Persistently storing the data cached in the first data shard to disk; Saving a snapshot of the first data shard; Control the unread and unwritten transactions in the first data shard to pause and enter the read and write phase.

7. The method according to any one of claims 1 to 6, further comprising: In response to the split instruction, create a fourth data slice corresponding to the third data slice; The third data shard is one of the slave copies of the first data shard; Obtaining a split log of the first data shard; A snapshot is saved for the third data shard.

8. The method according to any one of claims 1 to 7, wherein in response to the completion of the transaction in the first data shard that has entered the commit phase, controlling the unread and unwritten transactions in the first data shard to pause and enter the read and write phase comprises: In response to the transaction executed based on the first data shard and entering the commit phase being committed, and snapshots being saved for each slave copy of the first data shard, the unread and unwritten transactions in the first data shard are controlled to suspend entering the read and write phase.

9. The method according to claim 7 or 8, further comprising: In response to the first data shard and the second data shard being successfully split, modifying information of the third data shard; The fourth data slice is moved to the data slice list for external services.

10. The method according to any one of claims 7 to 9, further comprising: In response to a failure in splitting the first data shard and the second data shard, clearing the fourth data shard.

11. The method according to any one of claims 2 to 10, before resuming the write operation of the active transaction in response to the successful construction of the transaction context of the first transaction on the second data shard, further comprising: Marking the first transaction as a split transaction, and adding the identifier of the second data shard to the data shard identifier corresponding to the first transaction in the transaction context of the first data shard; The method further comprises: In response to the first data shard receiving a preparation request for a third transaction, and the third transaction being a transaction marked as split, querying a data shard identifier corresponding to the third transaction in a transaction context of the first data shard; In response to finding that the data shard identifier corresponding to the third transaction includes the identifier of the second data shard, a preliminary response is returned to the coordinator data shard of the third transaction, wherein the preliminary response includes the identifier of the second data shard, so that the coordinator data shard adds the identifier of the second data shard as a participant data of the third transaction. According to the fragment identifier.

12. A data sharding processing device, the device comprising: A creation module, configured to create a second data shard in response to a split instruction for a first data shard in the database; The split instruction is used to instruct to split the data in the first data slice into the second data slice; A first suspension module, used to control uncommitted transactions in the first data shard to suspend and enter a commit phase; A second suspension module is used to control the unread and unwritten transactions in the first data shard to suspend entering the read and write phase in response to the completion of the transaction in the first data shard that has entered the commit phase; A transfer module, configured to transfer first transaction data from the first data shard to the second data shard; The first transaction data is data in the first transaction corresponding to the second data shard after the split; The first transaction is a transaction in an active transaction of the first data shard that needs to transfer data to the second data shard, and the active transaction is a transaction in the read-write phase; A recovery module is used to control the transaction recovery in the first data shard to enter the read-write phase and / or the commit phase in response to the completion of the first transaction data transfer.

13. A computer device, comprising a processor and a memory, wherein the memory stores at least one computer instruction, and the at least one computer instruction is loaded and executed by the processor to implement the data sharding processing method as described in any one of claims 1 to 11.

14. A computer-readable storage medium, wherein at least one computer instruction is stored in the computer-readable storage medium, and the computer instruction is loaded and executed by a processor to implement the data sharding processing method as described in any one of claims 1 to 11.

15. A computer program product, comprising computer instructions stored in a computer-readable storage medium; the computer instructions are read and executed by a processor of a computer device to implement a method for data sharding processing as described in any one of claims 1 to 11.

Images