Data fragmentation processing method, device, equipment, storage medium and program product

Abstract

The present application is about a data sharding processing method, device, equipment, storage medium and program product, and relates to the field of distributed storage technology. The method includes: in response to a split instruction for a first data shard in a database, creating a second data shard; suspending transactions executed based on the first data shard to enter the commit phase; in response to the completion of the transaction in the first data shard that has entered the commit phase, suspending the new transaction corresponding to the first data shard to enter the read-write phase; transferring the first transaction data from the first data shard to the second data shard; the first transaction data is the data of the first transaction that corresponds to the data of the second data shard after the split; in response to the completion of the transfer of the first transaction data, resuming the transaction corresponding to the first data shard to enter the read-write phase and the commit phase. The present application can reduce the time of blocking transactions while ensuring the execution efficiency of distributed transactions.

Description

Technical Field

[0001] The present application relates to the field of distributed storage technology, and in particular to a data sharding processing method, device, equipment, storage medium and program product. Background Art

[0002] In a system that supports distributed transactions, timely scheduling of data distribution can disperse sudden hot spots and achieve load balancing.

[0003] In the related art, the administrator can manually modify the data distribution strategy; or load balancing can be automatically achieved by splitting overloaded data shards during blocking transactions.

[0004] However, the manual method in the related art involves a large amount of data migration, which is prone to errors and affects transactions; the automatic method in the related art blocks transactions for too long during the split period, which is likely to affect the read, write and commit performance of the transaction. Summary of the invention

[0005] The present application provides a data sharding processing method, apparatus, device, storage medium and program product, which can reduce the time of blocking transactions while ensuring the execution efficiency of distributed transactions; the technical solution is as follows.

[0006] According to one aspect of the present application, a data sharding processing method is provided, the method comprising:

[0007] In response to a split instruction for a first data shard in a database, creating a second data shard; the split instruction is used to instruct to split the target data in the first data shard to a new second data shard;

[0008] Suspend the transaction executed based on the first data shard and enter the commit phase;

[0009] In response to the transaction in the first data shard that has entered the commit phase being committed, pausing new transactions corresponding to the first data shard from entering the read-write phase;

[0010] Transferring first transaction data from the first data shard to the second data shard; the first transaction data is the data of the first transaction corresponding to the data of the second data shard after the split; the first transaction is an active transaction of the first data shard, all or part of the data of which corresponds to the transaction of the second data shard;

[0011] In response to the completion of the first transaction data transfer, the transaction corresponding to the first data shard is restored to enter the read-write phase and the commit phase.

[0012] According to one aspect of the present application, a data sharding processing device is provided, the device comprising:

[0013] A creation module, configured to create a second data shard in response to a split instruction for a first data shard in the database; the split instruction is configured to instruct to split the target data in the first data shard into a new second data shard;

[0014] A first pause module, used to pause the transaction executed based on the first data shard to enter the commit phase;

[0015] A second suspension module is used to suspend the new transaction corresponding to the first data shard from entering the read-write phase in response to the transaction in the first data shard that has entered the commit phase being committed;

[0016] A transfer module, configured to transfer first transaction data from the first data shard to the second data shard; the first transaction data is data of the first transaction corresponding to data of the second data shard after the split; the first transaction is an active transaction of the first data shard, all or part of the data of which corresponds to a transaction of the second data shard;

[0017] A recovery module is used to restore the transaction corresponding to the first data shard to enter the read-write phase and the commit phase in response to completion of the first transaction data transfer.

[0018] In some embodiments, the transfer module is used to:

[0019] Suspending the write operation of the active transaction;

[0020] Establishing a first container for the first transaction and a second container for the first transaction in the first data shard;

[0021] Writing the second transaction data into the first container, and writing the first transaction data into the second container; the second transaction data is the data of the first transaction corresponding to the first data shard after the split;

[0022] Copying the second container to the second data shard, and building a transaction context of the first transaction on the second data shard;

[0023] In response to the split being successful, the write operation of the active transaction is resumed.

[0024] In some embodiments, the apparatus further comprises: an updating module, configured to:

[0025] In response to successful splitting, updating the first container in the first data shard to a write container of the first transaction in the first data shard;

[0026] The second container in the first data shard and the original write container of the first transaction in the first data shard are released.

[0027] In some embodiments, the apparatus further comprises: a modification module, configured to:

[0028] In response to successful splitting, modifying information of the first data shard;

[0029] The second data shard is moved to the list of data shards for external services.

[0030] In some embodiments, the device further comprises: a deletion module, configured to:

[0031] In response to a split failure, deleting the first container and the second container;

[0032] Resuming write operations of active transactions executed based on the first data shard;

[0033] Clean up the second data shard.

[0034] In some embodiments, the apparatus further comprises: an establishing module, configured to:

[0035] Creating a split log for the first data shard;

[0036] Persisting the data cached by the first data shard to disk;

[0037] A snapshot is saved for the first data shard.

[0038] In some embodiments, the device further comprises: an acquisition module, configured to:

[0039] In response to the split instruction, a fourth data shard is created corresponding to the third data shard; the third data shard is one of the slave copies of the first data shard;

[0040] Obtaining a split log of the first data shard;

[0041] A snapshot is saved for the third data shard.

[0042] In some embodiments, the second pause module is used to:

[0043] In response to the transaction executed based on the first data shard and entering the commit phase being committed, and each slave copy of the first data shard saving a snapshot, new transactions executed based on the first data shard are suspended from entering the read-write phase.

[0044] In some embodiments, the device further comprises: a moving module, configured to:

[0045] In response to successful splitting, modifying information of the third data shard;

[0046] The fourth data slice is moved to the list of data slices for external services.

[0047] In some embodiments, the device further comprises: a cleaning module, for:

[0048] In response to the split failure, the fourth data shard is cleaned up.

[0049] In some embodiments, the device further comprises: an adding module, configured to:

[0050] Mark the first transaction as a split transaction, and add the ID of the second data shard to the ID of the data shard corresponding to the first transaction in the transaction context of the first data shard;

[0051] The device further comprises: a query module, configured to:

[0052] In response to the first data shard receiving a preparation request for a third transaction, and the third transaction being a transaction marked as split, querying an ID of a data shard corresponding to the third transaction in a transaction context of the first data shard;

[0053] In response to querying that the ID of the data shard corresponding to the third transaction includes the ID of the second data shard, a preliminary response is returned to the coordinator data shard of the third transaction, wherein the preliminary response includes the ID of the second data shard, so that the coordinator data shard adds the ID of the second data shard as the ID of the participant data shard of the third transaction.

[0054] According to another aspect of the present application, a computer device is provided, comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the data sharding processing method as described above.

[0055] According to another aspect of the present application, a computer-readable storage medium is provided, in which at least one instruction, at least one program, a code set or an instruction set is stored, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement the data segmentation processing method described above.

[0056] According to another aspect of the present application, a computer program product is provided, which includes computer instructions stored in a computer-readable storage medium, and a processor reads and executes the computer instructions from the computer-readable storage medium to implement the data segmentation processing method described above.

[0057] The technical solution provided by the embodiments of the present application may have the following beneficial effects:

[0058] The scheme shown in the embodiment of the present application is to create a second data shard in response to a split instruction for the first data shard in the database; suspend the transaction executed based on the first data shard to enter the commit phase; in response to the completion of the transaction in the first data shard that has entered the commit phase, suspend the new transaction corresponding to the first data shard to enter the read-write phase; transfer the first transaction data from the first data shard to the second data shard; in response to the completion of the first transaction data transfer, restore the transaction corresponding to the first data shard to enter the read-write phase and the commit phase. This scheme constructs the second data shard before blocking the transaction; at the same time, the blocking transaction is divided into two phases, and when the blocking transaction is committed, it does not affect the reading and writing of the transaction and the transaction that is about to be submitted. The present application can reduce the time of blocking transactions while ensuring the execution efficiency of distributed transactions. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0060] Figure 1 This is a distributed database architecture diagram of a traditional sub-library and sub-table architecture involved in this application;

[0061] Figure 2 This is a distributed database architecture diagram involved in this application;

[0062] Figure 3 is a flow chart of a data sharding processing method provided by an exemplary embodiment of the present application;

[0063] Figure 4 is a flow chart of a data sharding processing method provided by another exemplary embodiment of the present application;

[0064] Figure 5 is a flow chart of a data sharding processing method provided by another exemplary embodiment of the present application;

[0065] Figure 6is a flow chart of a data sharding processing method provided by another exemplary embodiment of the present application;

[0066] Figure 7 This is a flowchart of pausing a new transaction corresponding to a first data shard to enter a read-write phase in a data shard processing method provided by an exemplary embodiment of the present application;

[0067] Figure 8 is a flow chart of a data sharding processing method provided by another exemplary embodiment of the present application;

[0068] Fig. 9 This is another distributed database architecture diagram involved in this application;

[0069] Fig.10 is an implementation flow chart of a data sharding processing method provided by an exemplary embodiment of the present application;

[0070] Fig.11 is a splitting flow chart provided by an exemplary embodiment of the present application;

[0071] Fig.12 is a flow chart of a 2PC protocol involved in this application;

[0072] Fig.13 It is a schematic diagram of a 2PC case provided by an exemplary embodiment of the present application;

[0073] Fig.14 yes Fig.13 Submission diagram of the 2PC case;

[0074] Fig.15 is a 2PC flow chart provided by an exemplary embodiment of the present application;

[0075] Fig.16 is a block diagram of a data sharding processing device shown in an exemplary embodiment of the present application;

[0076] Fig.17 It is a structural block diagram of a computer device provided by an exemplary embodiment of the present application.

[0077] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application. DETAILED DESCRIPTION

[0078] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0079] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0080] The terms used in this disclosure are for the purpose of describing specific embodiments only and are not intended to limit the disclosure. The singular forms of "a", "said" and "the" used in this disclosure and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0081] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions. For example, the object behaviors such as attack operations involved in this application are all obtained with full authorization.

[0082] It should be understood that although the terms first, second, etc. may be used in the present disclosure to describe various information, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present disclosure, the first parameter may also be referred to as the second parameter, and similarly, the second parameter may also be referred to as the first parameter. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0083] The following are some definitions of terms involved in this application:

[0084] Database (DB): is an organic collection of large amounts of data that are organized in a certain structure and stored in a computer for a long time and can be shared. Each database has one or more different application programming interfaces (APIs) for creating, accessing, managing, searching, and copying the stored data.

[0085] Data sharding: It is the smallest management unit of data and can be used to schedule data more flexibly. For example, when the load on a certain node is particularly high, part of the data shards can be migrated to other physical nodes.

[0086] Write batch: A data structure used to store active transaction data. Each transaction has its own write batch, and the data stored in a write batch belongs to only one transaction. For a distributed transaction, each transaction has its own write batch in the transaction context of each Region.

[0087] Transaction data: is part of the transaction context. When splitting a transaction, after sending the transaction data corresponding to the Old Region to the New Region, a transaction context will be built on the New Region to ensure that both the Old Region and the New Region have corresponding transaction contexts.

[0088] Raft Group: The basic unit of a data synchronization group, which contains multiple replicas, one of which is the primary replica (Leader), and the other one or more replicas are slave replicas (Followers). The Leader is responsible for synchronizing data to the Followers by sending a Raft Log (also called a Proposal) to all Followers. When this Raft Log is persisted on the local disk by most replicas, the replicas can execute (Apply) this Raft Log, that is, perform corresponding operations according to the specific content contained in the Raft Log.

[0089] Two-Phase Commit (2PC): is a common practice to ensure the atomicity of distributed transaction submission. 2PC divides transaction submission into two phases: Prepare Phase (P) and Commit Phase (C). The initiator of a transaction is called a coordinator, usually only one, who is responsible for deciding the submission and rollback of the entire distributed transaction; the executors of a transaction are called participants, usually multiple, who are responsible for the submission and rollback of their own local transactions.

[0090] Remote Procedure Call (RPC): A protocol for requesting services from a remote computer program over a network without having to understand the underlying network technology. RPC is a way of communication between modules, requesting services from a remote node over a network.

[0091] Query Per Second (QPS): Also known as the query rate per second, it is the number of queries a server can respond to per second, or the number of requests a server can handle per second. QPS is a measure of the amount of traffic a specific query server handles within a specified time, and is also the maximum throughput capacity.

[0092] Please refer to Figure 1 , which shows a distributed database architecture diagram of a traditional sub-library and sub-table architecture involved in this application.

[0093] like Figure 1 As shown in the figure, the distributed database with traditional sharding architecture is a simple data distribution solution. Each storage node at the bottom is essentially a stand-alone database. For a large data volume table of a business, the bottom layer will split the original table into multiple tables according to the sharding key or sharding key specified by the user, and distribute them on different nodes. There is a proxy node at the upper layer of all storage nodes for routing and data aggregation, so that users do not need to care about the actual distribution of the underlying data and can query data.

[0094] Please refer to Figure 2 , which shows a distributed database architecture diagram involved in this application.

[0095] like Figure 2 As shown in the figure, the distributed database architecture divides the entire distributed database into the following three modules:

[0096] Storage cluster: stores KV data and divides the data into regions according to the key value to disperse data hotspots. Each data shard corresponds to a Raft Group to achieve master-slave data synchronization.

[0097] Computing cluster: converts SQL statements of user data into KV requests and sends them to the storage cluster.

[0098] Central control cluster: detects the data distribution of the storage cluster, directs the storage cluster to split, merge, and migrate data shards to ensure load balancing, and also provides global timestamp timing function.

[0099] Each node in the storage cluster will regularly report the information of local data shards to the central control node through heartbeat. If the central control finds that the size of a data shard exceeds the set threshold, it will send a task to split the data shard to the leader of the data shard. The task contains the range of the split data shard (including the range of the original data shard and the range of the newly split data shard). After receiving the task, the Leader will submit a Proposal in the Raft Group. When this Proposal is synchronized to the majority of replicas and each replica applies this Proposal, it will create a new data shard (New Region). When most replicas have created a New Region, the Raft Group corresponding to the underlying New Region has met the majority requirement of Raft. At this time, the Leader can be elected and provide services.

[0100] The storage system uses "keywords" to distinguish each piece of data. For example, for a row-based database, each record stored by the storage engine corresponds to a row of data in the table. For a distributed database with a traditional sharding architecture, the "keyword" of the data can be the value of the primary key or the sharding key; while for a distributed database based on a KV storage engine, the "keyword" of the data can be the Key value obtained after encoding. The "keyword" of the data determines the interval into which the data is divided. Currently, common data partitioning schemes can be roughly divided into "partitioning based on keyword hash values" and "partitioning based on keyword intervals."

[0101] The partitioning method of "partitioning based on keyword hash value" is based on a keyword hash function, the input of which is the value of the "keyword" and the output is an unsigned integer. In this way, the database system can build a mapping relationship from "key value" to "hash value". When partitioning, each partition corresponds to a range of unsigned integers, and there is no intersection between the ranges of each partition. If the hash value calculated by the hash function of the keyword value of a piece of data is within the integer range of a partition, then the data will be stored in the partition.

[0102] The advantage of this data division method is that as long as the hash function calculation is reasonable enough, the storage system can evenly divide a large amount of data into multiple partitions to ensure that the amount of data in each partition is almost the same. Therefore, this data division method has a good effect of dispersing hot spots.

[0103] The partitioning method based on keyword interval allows each partition to be directly mapped to a continuous key value interval, and the ranges of each partition do not have any intersection. This partitioning method does not rely on hash functions and is relatively simple.

[0104] The advantage of "keyword-based interval partitioning" is that when doing a range query, you only need to find several data partitions included in the query range, and then only perform data queries on these partitions. This query method narrows the scope of data query, improves query efficiency, and avoids unnecessary resource consumption.

[0105] The life cycle of a distributed transaction can be roughly divided into two phases: the read-write phase and the commit phase.

[0106] 1. Reading and Writing Stage

[0107] In this phase, users will read and write data by sending SQL requests to the database system. The data written in the read and write phase is considered uncommitted data before the transaction is committed. According to the isolation requirements of the transaction, this part of the data is not allowed to be accessed by other transactions.

[0108] In some distributed database systems, uncommitted transaction data is cached in the computing layer. The computing nodes will not send the transaction data to the corresponding data shards of the storage engine until the transaction enters the commit phase. Since the storage cluster does not cache uncommitted data, the scheduling of data shard splitting and merging in the storage cluster will not affect the transactions being executed, which ensures that the data scheduling is imperceptible to users to a certain extent.

[0109] 2. Submission Phase

[0110] When the user sends a COMMIT SQL statement to the database system, the transaction enters the commit process. At this time, the database system will perform some checks on the transaction to determine whether the transaction commit will destroy the consistency of the data (for example, for transactions based on optimistic concurrency control, it is necessary to detect whether there are data conflicts with other transactions), and persist the data written by the transaction through logs (such as WAL logs written to disk, or Proposals submitted to the Raft Group). If the transaction passes the test, the transaction can be successfully committed, and the data written by the transaction will be visible to other transactions.

[0111] In some distributed database systems, for simplicity, the atomicity of distributed transactions is guaranteed by the computing layer (such as using the commit protocol Percolator). In this architecture, the storage engine does not need to have the concept of transactions, that is, the storage engine does not guarantee the atomicity of writing multiple KV data. In this way, the storage engine will be more convenient for concurrent processing of KV data writing and data sharding scheduling. If a KV fails to be written due to data sharding scheduling, it only needs to retry the single KV, without retrying all the data of the transaction.

[0112] From the analysis of the above data partitioning strategies, the partitioning method based on keyword hash value has a good effect of dispersing hot spots, but the disadvantage is that range query requires access to all data partitions, which is inefficient. The disadvantage of "partitioning based on keyword interval" is that data hot spots may still exist - if the business access to data is concentrated in a small range, it is necessary to frequently trigger data shard splitting to split the hot spots.

[0113] Considering that data query efficiency should be the top priority, the technical solution of the embodiment of the present application preferably adopts the method of "keyword interval division".

[0114] Regarding the data hotspot problem, first, the business side can perform some random processing on the data. For example, for businesses that are prone to data hotspots, the business side can perform hash function calculations on the primary key column when writing data to ensure that the data can be dispersed to different data shards when inserting data. Second, as long as the impact of data shard splitting on transactions is optimized to be low enough at the storage layer, the impact of data hotspots can be reduced by speeding up the frequency of data shard splitting. The data shard splitting idea provided by the technical solution of the embodiment of the present application achieves data splitting without business perception, which can deal with this problem.

[0115] Although the distributed database with traditional sharding architecture is relatively simple to implement, it lacks good data distribution detection methods, is not flexible enough in data partitioning control, and cannot quickly dispatch sudden hot data. For example, when building a table, the database administrator (DBA) needs to specify the sharding key. For sudden hot spots, the DBA also needs to manually modify the data distribution strategy to distribute the data to adjacent nodes. This data distribution method not only consumes human resources, but also involves a large amount of data migration, which can easily affect the business.

[0116] Therefore, an optional data sharding strategy is: based on the new distributed database architecture, the splitting, merging and migration of data shards are controlled by the central management cluster, and load balancing can be automatically achieved without relying on the support of DBA.

[0117] However, the existing data sharding solution will still have an impact on the business. As mentioned above, by synchronizing the Proposal in the RaftGroup, you need to wait for the Proposal to be synchronized to the majority, build a new Raft Group, and then after a round of elections, the new data shard can provide data writing services to the outside world. This cycle spans the Raft Group election timeout and usually takes several seconds. After synchronizing the Proposal, the old data shard will no longer support writing data to the range corresponding to the new data shard, which means that the data range corresponding to the new data shard will be unavailable within these few seconds, which can easily cause a decline in QPS.

[0118] For distributed databases based on the Percolator transaction protocol, since the storage cluster does not need to ensure the atomicity of multiple data writes, it is easier to handle the concurrency of data writing and data sharding. However, the cost of this implementation is that for each KV data that the transaction ultimately writes to disk, two additional KVs must be written during the submission process to ensure the atomicity of the transaction (according to the requirements of the Percolator protocol, these three KV data are written to the data column family, the lock column family, and the write column family, respectively). This means that if Percolator is used to ensure the atomicity of transactions, the data communication volume between the computing cluster and the storage cluster, the number of proposals synchronized by the storage cluster Raft Log, and the input / output (I / O) overhead of data writing to disk will all increase by 2 to 3 times, resulting in poor submission performance.

[0119] In order to ensure the submission efficiency of distributed transactions, the technical solution provided in the subsequent embodiments of this application completely sinks the distributed transactions to the storage cluster. For transactions in the read and write phases, their uncommitted data is maintained in the transaction context and managed in memory by the shard where the data is located. Under this design, this solution can still ensure that when the split of the data shard is triggered, the transaction can continue to execute, will not be rolled back, and the QPS will not fluctuate. Under the premise of ensuring the execution efficiency of distributed transactions, the data shard split is unaware of the business.

[0120] Please refer to Figure 3 , which shows a flow chart of a data slicing processing method provided by an exemplary embodiment of the present application. The method is executed by a computer device. Figure 3 As shown, the method may include the following steps 310 , 320 , 330 , 340 , and 350 .

[0121] Step 310: In response to a split instruction for a first data shard in a database, create a second data shard; the split instruction is used to instruct to split the target data in the first data shard to a new second data shard.

[0122] In an embodiment of the present application, when receiving a split instruction for a first data shard in a database, the computer device may create a new second data shard at the storage cluster node of the first data shard.

[0123] The split instruction is used to instruct to split the target data in the first data slice into a new second data slice.

[0124] The first data shard may be a master copy in a Raft Group, and the computer device may synchronize the split instruction to each slave copy corresponding to the first data shard.

[0125] The above split instruction is used to indicate the splitting of data shards. For example, if a data shard on a physical node contains a particularly large amount of data, the computer device can migrate part of it to other nodes. At this time, the data shard needs to be split on the current node.

[0126] On the one hand, the above splitting instruction can be automatically sent to the storage cluster node where the data shard (Region) corresponding to the hot data is located after the central control cluster detects that the hot data needs to be migrated.

[0127] On the other hand, the above-mentioned split instruction can also be sent by the developer user to the central control cluster via a Hypertext Transfer Protocol (HTTP) request. Based on the HTTP request, the central control cluster sends the split instruction to the storage cluster node where the corresponding data shard (Region) is located.

[0128] In addition, developers can also manually turn on or off the automatic split function by sending HTTP requests to the central control cluster. After turning it off, the central control cluster will not issue split instructions automatically.

[0129] In some embodiments, the above-mentioned split instruction can indicate the ID of the first data shard, the range of the first data shard, and the range of the new second data shard, including the split point (split key), the starting point (start key), the end point (endkey), the data shard identification number (Region ID), etc.

[0130] For example, the range of the first data shard is [A, D), and a new second data shard is split with the split key B. Then, the range of the second data shard can be [B, D), that is, the start_key is B and the end key is D; correspondingly, the range of the first data shard after the split can be [A, B).

[0131] In some embodiments, the above step 310 includes a pre-step of calculating the split key, specifically including:

[0132] S11: The central control cluster sends a new splitkey calculation task to the first data shard to be split in the storage cluster. After the first data shard calculates the split key, it reports the split key to the central control cluster.

[0133] S12: After the central control cluster obtains the split key calculated by the storage cluster, it issues a split instruction to the first data shard and all the slave copies of the first data shard.

[0134] Step 320: Pause the transaction executed based on the first data shard to enter the commit phase.

[0135] In an embodiment of the present application, after step 310 creates a second data shard in response to a split instruction for a first data shard in a database, the computer device may suspend the transaction executed based on the first data shard to enter a commit phase.

[0136] In the embodiment of the present application, after the new second data shard is constructed, the transaction is suspended from entering the commit phase, and the transactions in the read-write phase and the newly opened transactions are not affected. The above scheme can shorten the time of blocking the transaction executed based on the first data shard from entering the commit phase, and reduce the impact of the split of the data shard on the transaction.

[0137] In some embodiments, step 320 may include the following steps:

[0138] S21: construct a split task context of the first data shard; wherein the split task context includes some task information, such as the first data shard ID, the second data shard ID, the split key, etc.;

[0139] S22: Set the split task status to REGION_SPLIT_INIT;

[0140] S23: Send RPC to confirm that all slave copies of the first data shard have entered the REGION_SPLIT_INIT state.

[0141] Step 330: In response to the completion of the transaction in the first data shard that has entered the commit phase, suspending the new transaction corresponding to the first data shard from entering the read-write phase.

[0142] In an embodiment of the present application, after step 320 suspends the transaction executed based on the first data shard to enter the commit phase, the transaction in the first data shard that has entered the commit phase is committed, and the computer device can suspend the new transaction corresponding to the first data shard to enter the read-write phase, that is, suspend the opening of the new transaction, but the transaction in the read-write phase is not affected.

[0143] In the embodiment of the present application, before the data transfer in step 340 is performed, the process of blocking transactions is divided into two stages, including blocking the transaction executed based on the first data shard to enter the commit stage in step 320, and pausing the new transaction corresponding to the first data shard to enter the read-write stage after the transaction that has entered the commit stage is committed in step 330. The above scheme can reduce the impact of the split of the data shard on the transaction.

[0144] Step 340: Transfer the first transaction data from the first data shard to the second data shard; the first transaction data is the data of the first transaction corresponding to the data of the second data shard after the split; the first transaction is an active transaction of the first data shard, and all or part of the data corresponds to the transaction of the second data shard.

[0145] In an embodiment of the present application, after step 330 suspends the new transaction corresponding to the first data shard to enter the read-write phase, the computer device can traverse the active transactions corresponding to the first data shard (i.e., transactions in the read-write phase) and transfer the first transaction data from the first data shard to the second data shard through RPC.

[0146] In an embodiment of the present application, among the active transactions of the first data shard, only the transaction data of some of the active transactions may need to be transferred in whole or in part to the second data shard. Therefore, it is necessary to make a judgment based on the split instruction, that is, to traverse the data of each active transaction of the first data shard in order to filter out the transaction data that needs to be transferred to the second data shard.

[0147] For example, the active transactions of the first data shard include transaction 1, transaction 2, and transaction 3, and the split instruction is used to instruct the data after the split key to be transferred to the second data shard. If the data of transaction 1 is before the split key, the data of transaction 1 does not need to be transferred; if the data of transaction 2 is after the split key, the data of transaction 2 needs to be transferred to the second data shard; if part of the data of transaction 3 is before the split key and part is after the split key, the part of the data of transaction 3 after the split key needs to be transferred to the second data shard.

[0148] Step 350: In response to the completion of the first transaction data transfer, the transaction corresponding to the first data shard is restored to enter the read-write phase and the commit phase.

[0149] In an embodiment of the present application, after the data transmission of step 340 is successful, the data hotspots on the first data shard are dispersed, and the computer device can restore the new transaction corresponding to the first data shard to enter the read and write phase, and the transaction in the read and write phase to enter the commit phase.

[0150] In summary, the scheme shown in the embodiment of the present application is to create a second data shard in response to a split instruction for the first data shard in the database; suspend the transaction executed based on the first data shard to enter the commit phase; in response to the completion of the transaction in the first data shard that has entered the commit phase, suspend the new transaction corresponding to the first data shard to enter the read-write phase; transfer the first transaction data from the first data shard to the second data shard; in response to the completion of the first transaction data transfer, restore the transaction corresponding to the first data shard to enter the read-write phase and the commit phase. This scheme constructs the second data shard before blocking the transaction; at the same time, the blocking transaction is divided into two phases, and when the blocking transaction is committed, it does not affect the reading and writing of the transaction and the transaction that is about to be submitted. The present application can reduce the time of blocking transactions while ensuring the execution efficiency of distributed transactions.

[0151] Please refer to Figure 4 , which shows a flow chart of a data slicing processing method provided by another exemplary embodiment of the present application. The method is executed by a computer device. Figure 4 As shown above Figure 3 Step 340 in the illustrated embodiment may be implemented as step 340 - 10 , step 340 - 20 , step 340 - 30 , step 340 - 40 , and step 340 - 50 .

[0152] Step 340 - 10: Suspend the write operation of the active transaction.

[0153] In an embodiment of the present application, after step 330 suspends the new transaction corresponding to the first data shard to enter the read-write phase, the computer device may suspend the write operation of the active transaction corresponding to the first data shard, so as to facilitate subsequent data splitting.

[0154] During the write-ban period, if the storage cluster receives a write request from the computing cluster, a retry error code is returned.

[0155] Step 340-20: Create a first container for the first transaction and a second container for the first transaction in the first data shard.

[0156] In the embodiment of the present application, after step 340-10 suspends the write operation of the active transaction, the computer device can establish a first container of the first transaction and a second container of the first transaction in the first data shard so as to write the transaction data to the corresponding containers; that is, the first container and the second container are respectively used to store the transaction data before and after the split key of the first transaction.

[0157] In other embodiments, the computer device may also split the original write container of the first transaction on the first data shard into a first container and a second container, which are used to store transaction data before and after the split key of the first transaction, respectively.

[0158] Step 340-30: write the second transaction data into the first container, and write the first transaction data into the second container; the second transaction data is the data of the first transaction corresponding to the data of the first data shard after the split.

[0159] In the embodiment of the present application, after step 340-20 establishes the first container of the first transaction and the second container of the first transaction, the computer device traverses the transaction data in the read and write phases, writes the second transaction data into the first container, and writes the first transaction data into the second container.

[0160] That is to say, the transaction data after the split key is the first transaction data, and the transaction data before the split key is the second transaction data. If the data of transaction 1 is before the split key, the data of transaction 1 belongs to the second transaction data and needs to be written to the first container; if the data of transaction 2 is after the split key, the data of transaction 2 belongs to the first transaction data and needs to be written to the second container; if part of the data of transaction 3 is before the split key and part of it is after the split key, the part of the data of transaction 3 after the split key belongs to the first transaction data and needs to be written to the second container, and the part of the data of transaction 3 before the split key belongs to the second transaction data and needs to be written to the first container.

[0161] Step 340-40: Copy the second container to the second data shard, and build a transaction context of the first transaction on the second data shard.

[0162] In an embodiment of the present application, after step 340-30 writes the second transaction data into the first container and the first transaction data into the second container, the computer device can copy the second container to the second data shard, and build a transaction context of the first transaction on the second data shard based on the first transaction data of the second container; so that the second data shard can provide external services based on the transaction context of the first transaction.

[0163] In some embodiments, the computer device may send the data of the second container to the second data shard through RPC, and build a transaction context on the second data shard.

[0164] Step 340 - 50 : In response to the split being successful, resuming the write operation of the active transaction.

[0165] In the embodiment of the present application, after step 340-40 copies the second container to the second data shard and builds the transaction context of the first transaction on the second data shard, the split is successful and the computer device can resume the write operation of the active transaction corresponding to the first data shard to continue external services.

[0166] In an embodiment of the present application, an exemplary scheme for transferring first transaction data is provided, including establishing a first container of the first transaction and a second container of the first transaction in a first data shard during a write-disabled phase of an active transaction; then copying the second container to a second data shard, and building a transaction context of the first transaction on the second data shard; this scheme can transfer the first transaction data that needs to be transferred from the first data shard to the second data shard during the write-disabled phase of an active transaction, thereby avoiding conflicts between transaction data transfer and transaction write operations.

[0167] Please refer to Figure 5 , which shows a flowchart of a data sharding processing method provided by another exemplary embodiment of the present application. The method is executed by a computer device. Figure 5 As shown above Figure 4 Before step 340 - 50 in the illustrated embodiment, steps 340 - 42 and 340 - 44 are also included.

[0168] Step 340-42: In response to the split being successful, updating the first container in the first data shard to be the write container of the first transaction in the first data shard.

[0169] In the embodiment of the present application, after step 340-40 copies the second container to the second data shard and builds the transaction context of the first transaction on the second data shard, the split is successful, and the computer device can update the first container in the first data shard to the write container of the first transaction in the first data shard; so that the first container provides services for the first transaction.

[0170] In some embodiments, after the split is successful, the computer device can modify the end key information of the first data shard according to the split instruction such as the split key; and move the second data shard to the list of external services.

[0171] Step 340-44: Release the second container in the first data shard and the original write container of the first transaction in the first data shard.

[0172] In an embodiment of the present application, after step 340-42 updates the first container in the first data shard to the write container of the first transaction in the first data shard, the computer device can release the second container in the first data shard and the original write container of the first transaction in the first data shard; resources can be released in time to avoid excessive memory consumption.

[0173] In the embodiment of the present application, for the first transaction on the first data shard, the first container is used as the new write container of the first transaction. In the above steps 340-40, the data in the second container has been sent to the second data shard, so there is no need to retain the second container on the first data shard, and the second container is directly released.

[0174] In an embodiment of the present application, a supplementary solution before the above steps 340-50 is provided, including: in response to a successful split, before resuming the write operation of the active transaction: in response to a successful split, updating the first container in the first data shard to the write container of the first transaction in the first data shard; releasing the second container in the first data shard and the original write container of the first transaction in the first data shard; this solution can timely update the write container of the first transaction and release redundant containers after the split is successful, thereby saving memory space of the first data shard.

[0175] In some embodiments, the method further comprises:

[0176] In response to successful splitting, modifying information of the first data shard;

[0177] The second data shard is moved to the list of data shards for external services.

[0178] In an embodiment of the present application, after the split is successful, the computer device can modify the information of the first data shard, for example, modify the end key of the first data shard according to the split key; and move the second data shard to the list of data shards for external services, so that the newly created second data shard can provide external services.

[0179] In an embodiment of the present application, a supplementary solution is provided after the split is successful, including modifying the information of the first data shard and moving the second data shard to the list of data shards for external services; so that the second data shard can provide services to the outside world and ensure the execution efficiency of distributed transactions.

[0180] In some embodiments, the method further comprises:

[0181] In response to the split failure, deleting the first container and the second container;

[0182] Resuming write operations of active transactions executed based on the first data shard;

[0183] Clean up the second data shard.

[0184] In an embodiment of the present application, if the above-mentioned split fails, that is, the process of writing the second transaction data into the first container and the first transaction data into the second container in the above-mentioned step 340-30 fails, or the process of copying the second container to the second data shard and building the transaction context of the first transaction on the second data shard in the above-mentioned step 340-40 fails; the computer device can delete the first container and the second container in time, and at the same time restore the write operation of the active transaction executed based on the first data shard, and clean up the second data shard.

[0185] In an embodiment of the present application, a supplementary solution after a split failure is provided, including deleting the first container and the second container, restoring the write operation of the active transaction executed based on the first data shard, and clearing the second data shard; this solution can promptly clear the relevant data of the split failure to avoid useless data occupying database resources.

[0186] Based on the above Figure 3 or Figure 4 For the scheme shown, please refer to Figure 6 , which shows a flowchart of a data slicing processing method provided by another exemplary embodiment of the present application. The method is executed by a computer device, such as Figure 6 As shown above Figure 3 or Figure 4 Step 330 in the illustrated embodiment may be implemented as step 330a, step 330b, step 330c, and step 330d.

[0187] Step 330a: In response to the completion of the transaction that has entered the commit phase in the first data shard, a split log of the first data shard is created.

[0188] In an embodiment of the present application, after the transaction in the first data shard that has entered the commit phase is committed, the computer device may update the state of the first data shard from L_WORKING to L_SPLIT; wait for the split log of the first data shard from the replica; and establish the split log of the first data shard.

[0189] In an embodiment of the present application, the split log of the first data shard is synchronized from the replica to the first data shard, with the purpose of ensuring that all replicas are played back to the same split index before the second data shard provides services.

[0190] Submitted transaction data, which belongs to inactive transaction data, will be written to the storage instance of the corresponding node. Since all regions of the same node share the storage instance, the newly created second data shard can also access the inactive transaction data of the first data shard, so there is no need to transfer it.

[0191] The data of active transactions is not written to the storage instance and can only be accessed by the current Region. Therefore, it needs to be transferred from the first data shard to the newly created second data shard so that the corresponding transactions can continue to execute.

[0192] The storage instance (RocksDB) is a module used to store data within the storage cluster, including the data structure Memtable and the data structure SSTable. The data in the Memtable is in memory and is not persisted.

[0193] Step 330b: Persist the data cached in the first data shard to the disk.

[0194] In the embodiment of the present application, after step 330a creates the split log of the first data shard, the computer device persists the data cached by the first data shard to the disk.

[0195] In an embodiment of the present application, the data cached by the first data shard includes data of active transactions and data of inactive transactions that have not been flushed to the SSTable.

[0196] Optionally, the data in the disk does not need to be split, because the first data shard and the newly created second data shard are on the same storage instance physical node, and all regions on the same storage instance physical node will share the data in the disk, so there is no need to split.

[0197] Step 330c: Save a snapshot of the first data shard.

[0198] In the embodiment of the present application, after step 330b persists the data cached by the first data shard to the disk, the computer device saves a snapshot of the first data shard to facilitate subsequent playback processing.

[0199] The first data shard saves the snapshot with the split index so that the first data shard will not replay the log before the split index later. This is to prevent the old data replayed from the log by the first data shard from overwriting the new data written by the second data shard.

[0200] Taking Region 1 with a range of [A,H) as an example, before the split, the log synchronized when a transaction is committed on Region 1 may involve any data in [A,H).

[0201] Assume that a transaction T1 commits a change to E=1, and then Region 1 is split into Region 2. The range of Region 1 becomes [A, C), and the range of Region 2 is [C, H). At this time, another transaction T2 commits a change to E=2 on Region 2.

[0202] From the above description, E=2 written by transaction T2 should be the latest version.

[0203] However, if Region 1 is allowed to replay the logs before the split index, it is equivalent to allowing Region 1 to replay the logs written by transaction T1. Since the logs of Region 1 and Region 2 are replayed concurrently when the node is started, once the log of transaction T1 on Region 1 is replayed later than the log of transaction T2 on Region 2, E=1 of transaction T1 will be written to RocksDB later than E=2 of transaction T2, which will cause E=1 to become the latest version, resulting in data errors.

[0204] Therefore, it is necessary to prevent Region 1 from replaying the logs before the split index, so that the logs of transaction T1 will not be replayed, which ensures that E=2 is the latest version. The way to achieve this goal is to save a snapshot of Region 1 as a split index.

[0205] In some embodiments, after saving the snapshot of the first data shard, the computer device may set the split task state to REGION_SPLIT_CREATE_NEW_REGION; and wait for the slave replica to enter the REGION_SPLIT_CREATE_NEW_REGION state.

[0206] Step 330d: Pause the new transaction corresponding to the first data shard and enter the read-write phase.

[0207] In an embodiment of the present application, the computer device suspends a new transaction corresponding to the first data shard from entering the read and write phase, that is, prevents a new transaction executed based on the first data shard from starting.

[0208] In an embodiment of the present application, a supplementary solution to the above-mentioned step 330 is provided, including establishing a split log of the first data shard after the transaction that has entered the commit stage in the first data shard is committed, persisting the data cached by the first data shard to the disk, and saving a snapshot of the first data shard; it can facilitate the synchronization of all replica logs of the first data shard and the playback of the first data shard.

[0209] In some embodiments, the method further comprises:

[0210] In response to the split instruction, a fourth data shard is created corresponding to the third data shard; the third data shard is one of the slave copies of the first data shard;

[0211] Obtain the split log of the first data shard;

[0212] A snapshot is saved for the third data shard.

[0213] In an embodiment of the present application, in response to a split instruction for a first data shard in a database, a computer device creates a fourth data shard corresponding to a third data shard; obtains a split log of the first data shard, and saves a snapshot of the third data shard.

[0214] The third data shard is any one of the slave copies of the first data shard, and the third data shard and the first data shard synchronously respond to the split instruction to create a fourth data shard corresponding to the third data shard.

[0215] In an embodiment of the present application, a task execution scheme for a slave copy during the splitting process is provided, including obtaining a split log of a first data shard and saving a snapshot of a third data shard, so that the slave copy can timely back up the relevant data of the master copy.

[0216] Based on the above examples, please refer to Figure 7 , which shows a flowchart of pausing a new transaction corresponding to a first data shard to enter a read-write phase in a data shard processing method provided by an exemplary embodiment of the present application. The method is executed by a computer device, such as Figure 7 As shown, the above step 330 can be implemented as step 3301:

[0217] Step 3301: In response to the transaction executed based on the first data shard and entering the commit phase being committed, and each slave copy of the first data shard saving a snapshot, suspending new transactions executed based on the first data shard from entering the read-write phase.

[0218] In an embodiment of the present application, in the above step 330, the transaction executed based on the first data shard and entering the commit phase is committed, and each slave copy of the first data shard (such as the third data shard) has saved a snapshot respectively. Only then can the computer device suspend the new transaction executed based on the first data shard to enter the read and write phase.

[0219] In an embodiment of the present application, a scheme for executing tasks of slave copies during the splitting process is provided, including responding to the completion of the transaction that has entered the commit phase and is executed based on the first data shard, and each slave copy of the first data shard has saved a snapshot, pausing the new transaction that is executed based on the first data shard to enter the read-write phase; so that the slave copy can timely back up the relevant data of the master copy and synchronously update the status of each slave copy.

[0220] In some embodiments, the method further comprises:

[0221] In response to the split being successful, modifying information of the third data shard;

[0222] The fourth data shard is moved to the list of data shards for external services.

[0223] In an embodiment of the present application, based on the above-mentioned embodiment of creating a fourth data shard corresponding to the third data shard, if the split is successful, that is, the first transaction data transfer is completed, the computer device can modify the information of the third data shard; and move the fourth data shard to the list of data shards for external services, so that the newly created fourth data shard can provide external services.

[0224] In an embodiment of the present application, a plan for executing tasks from the replica after a successful split is provided, including modifying the information of the third data shard and moving the fourth data shard to the list of data shards for external services; so that the fourth data shard can provide services to the outside world and ensure the execution efficiency of distributed transactions.

[0225] In some embodiments, the method further comprises:

[0226] In response to the split failure, clean up the fourth data shard.

[0227] In an embodiment of the present application, based on the above-mentioned embodiment of creating a fourth data shard corresponding to the third data shard, if the split fails, that is, the first transaction data transfer fails, the computer device can clean up the fourth data shard.

[0228] In an embodiment of the present application, a scheme for executing tasks from a replica after a split failure is provided, including cleaning up the fourth data shard; this scheme can clean up relevant data that failed to split in a timely manner to avoid useless data occupying database resources.

[0229] Based on the above Figure 4 For the scheme shown, please refer to Figure 8 , which shows a flow chart of a data slicing processing method provided by another exemplary embodiment of the present application. The method is executed by a computer device, such as Figure 8 As shown above Figure 4 In the illustrated embodiment, step 340 - 46 is included before step 340 - 50 .

[0230] Step 340-46: Mark the first transaction as a split transaction, and in the transaction context of the first data shard, add the ID of the second data shard to the ID of the data shard corresponding to the first transaction.

[0231] In an embodiment of the present application, before restoring the write operation of the active transaction in step 340-50, the computer device may mark the first transaction as a split transaction and record the ID of the second data shard in the context of the first transaction corresponding to the first data shard; thereby avoiding omitting the new second data shard when the transaction corresponding to the second data shard is submitted, and ensuring the atomicity of the transaction submission.

[0232] The method further includes:

[0233] In response to the first data shard receiving a preparation request for a third transaction, and the third transaction being a transaction marked as split, querying an ID of a data shard corresponding to the third transaction in a transaction context of the first data shard;

[0234] In response to querying that the ID of the data shard corresponding to the third transaction includes the ID of the second data shard, a preliminary response is returned to the coordinator data shard of the third transaction, and the preliminary response includes the ID of the second data shard, so that the coordinator data shard can add the ID of the second data shard as the ID of the participant data shard of the third transaction.

[0235] In an embodiment of the present application, when the first data shard receives a preparation request for a third transaction, and the third transaction is a transaction marked as split: the computer device may query the ID of the data shard corresponding to the third transaction in the transaction context of the first data shard;

[0236] When it is found that the ID of the data shard corresponding to the third transaction includes the ID of the second data shard: the computer device may return a preliminary response to the coordinator data shard of the third transaction.

[0237] The preparation response includes the ID of the second data shard, so that the coordinator data shard adds the ID of the second data shard as the ID of the participant data shard of the third transaction.

[0238] In an embodiment of the present application, a submission scheme for transactions marked as split is provided, including marking a first transaction as a split transaction before resuming the write operation of an active transaction, and adding the ID of a second data shard to the ID of a data shard corresponding to the first transaction in the transaction context of the first data shard; when the first data shard receives a preparation request for a split third transaction, the ID of the data shard corresponding to the third transaction is queried in the transaction context of the first data shard; when it is found that the ID of the data shard corresponding to the third transaction includes the ID of the second data shard, a preparation response is returned to the coordinator data shard of the third transaction, and the preparation response includes the ID of the second data shard, so that the coordinator data shard adds the ID of the second data shard as the ID of a participant data shard of the third transaction; this scheme can find all data shards involved in the current transaction to be submitted in the submission process of the split transaction, avoid the omission of the split data shard, and ensure the integrity of the transaction submission.

[0239] The above technical solution of the embodiment of the present application can be applied to another distributed database. Fig. 9 , which shows another distributed database architecture diagram involved in this application. Fig. 9 As shown, the distributed database includes a computing cluster (SQLEngine), a storage cluster and a central management cluster (Meta Cluster, MC).

[0240] SQLEngine: Computing engine node, mainly performs structured query language (SQL) layer computing functions.

[0241] Storage cluster: storage engine node, mainly completes transactions and key-value (KV) storage functions.

[0242] MC: Cluster management node, which manages the data scheduling of the entire cluster and is also responsible for allocating globally unique timestamps.

[0243] Among them, based on the above Fig. 9 The process of splitting the database into data shards can be as follows:

[0244] MC automatically sends data shard splits

[0245] When MC detects a data hotspot and needs to migrate the hotspot data:

[0246] First, send a split region job to the storage cluster node where the data shard (Region) is located. The job contains a split point (split key).

[0247] After receiving the job, the storage cluster will move the right boundary of the current data shard (Old Region) to the left to the split key, and create a new data shard (New Region) with the left boundary of the New Region as the split key and the right boundary as the original right boundary of the Old Region.

[0248] In this way, the hotspot is split into two data shards.

[0249] Then MC sends a migration task (migrate region job) to migrate one of the regions to other relatively idle storage cluster nodes, thus completing load balancing.

[0250] Users can also manually enable or disable the automatic split function by sending HTTP requests to MC. After it is disabled, MC will not automatically issue split region jobs.

[0251] The HTTP request is as follows:

[0252] curl-XPUT

[0253] http: / / {mc_ip}:{mc_port} / meta-cluster / api / cluster / enable-split-region / {cluster_id} / {enable}

[0254] The parameters are described as follows:

[0255] mc_ip represents the IP address of the node where MC is located, mc_port represents the port number where MC receives HTTP requests, and enable represents whether to enable the function of automatically detecting region status and issuing split tasks. 0 represents disabled, and other positive integer values ​​represent enabled.

[0256] The switch status of the automatic split function can be queried through the following HTTP request:

[0257] curl-XGET

[0258] http: / / {mc_ip}:{mc_port} / meta-cluster / api / cluster / is-enable-split-region / {cluster_id}

[0259] Manually trigger the splitting of data shards

[0260] By sending an HTTP request to MC, users can also manually request MC to issue a split region job to control the splitting of a region according to a specific split key.

[0261] The HTTP request is as follows:

[0262] curl-XPUT

[0263] http: / / {mc_ip}:{mc_port} / meta-cluster / api / region / split-region / {cluster_id} / {region_id} / {is_force}? key="{key}"

[0264] The API is used to manually trigger Region splitting, so it is not affected by the split-region-enabled configuration parameter.

[0265] The key parameter can be used to pass in a specified split point. If not specified, a split point will be calculated by the storage cluster. The key is a string represented by hexadecimal encoding.

[0266] Split the main process of the task (how to ensure that the transaction is not killed)

[0267] 1. Preliminary Steps

[0268] 1. MC sends a new split key calculation task to the Leader copy of the Region to be split in the storage cluster. After the Leader calculates the split key, it reports it to MC (if the storage cluster finds that a suitable split point cannot be found, it returns failure).

[0269] 2. After MC obtains the split key calculated by the storage cluster, it sends the split region job to all copies of the region on the storage cluster.

[0270] The format of the split region job is:

[0271] message SplitRegionJob{

[0272] uint64 job_id = 1;

[0273] RegionMeta old_region_meta=2;

[0274] RegionMeta new_region_meta=3;

[0275] }

[0276] Among them, job_id uniquely identifies a data sharding task; RegionMeta contains information such as region_id, start_key, end_key, etc. of the Region; old_region_mata contains information before the split of the Old Region (start_key and end_key both correspond to the values ​​before the split); new_region_meta contains information about the New Region (start_key is split_key, and end_key should be the same as end_key of old_region_meta).

[0277] 2. Leader receives the split task process issued by MC

[0278] Please refer to Fig.10 , which shows an implementation flow chart of a data sharding processing method provided by an exemplary embodiment of the present application. Please refer to Fig.11 , which shows a splitting flow chart provided by an exemplary embodiment of the present application. Fig.10 , Fig.11 As shown, the splitting task process is as follows.

[0279] S1001: Create and start a New Region based on the split key and other meta-information; the meta-information may be information contained in the split task context, or information contained in the split task; the split task context includes some task information, such as Old Region ID, New Region ID, split key, etc. Fig.11 "S1101: Create a new data shard" step.

[0280] S1002: Build the split task context, set the task status to REGION_SPLIT_INIT, and send an RPC to confirm that other Old Region copies have entered this status.

[0281] S1003: Notify the transaction module to prevent all active transactions that have not entered 2PC from entering the commit phase (ie, participants are not allowed to Prepare).

[0282] S1004: Waiting for the transaction entering 2PC to end. Transactions in the read / write phase and newly opened transactions are not affected.

[0283] S1005: Update the current Old Region status from L_WORKING to L_SPLIT.

[0284] S1006: The Old Region synchronizes the split log (region split raft log) to ensure that all replicas are played back to the same split index (split index) before the New Region provides services.

[0285] S1007: The data remaining in the memory of the Old Region is persisted to the disk, and a snapshot is saved for the Raft Group of the Old Region using the split index.

[0286] S1008: Set the split task status to REGION_SPLIT_CREATE_NEW_REGION.

[0287] S1009: Send RPC to confirm that other copies have entered the above state of step S1008.

[0288] S1010: Prevent new transactions from being opened, and traverse all active transactions, performing the following operations on each transaction:

[0289] a) Set a flag to disallow write operations such as put / delete. If a write request is received from SQLEngine later, a retry error code is returned;

[0290] b) Create old_region_writebatch and new_region_writebatch, traverse the uncommitted data in the transaction writebatch, and write them into the two newly created writebatches according to the split points;

[0291] c) Send the data of new_region_writebatch to the leader of the new region through RPC, and build a transaction context on the leader of the new region; refer to Fig.11 In the step “S1102: transferring part of the transaction data to the new data shard according to the split point”, part of the data of transaction A (Trans A) is transferred to the new data shard through RPC.

[0292] S1011: Report to MC that the split task of this node is completed. MC comprehensively reports the task status reported by all nodes and replies to the status of each node task to decide whether to continue or roll back to clean up the environment.

[0293] S1012-1: If MC decides to continue, the split data shard takes effect, and all transactions are traversed to take effect and continue to provide services:

[0294] a) After receiving the information, this node modifies the end key information of the Old Region according to the split key and other meta-information and moves the New Region to the list of external services;

[0295] b) Traverse all active transactions again and perform the following operations on each transaction:

[0296] i. Update old_region_writebatch to writebatch and make it effective, and release new_region_writebatch and old writebatch. Fig.11 "S1103: If the split is successful, delete the data after the split point on the new data shard";

[0297] ii. Set this transaction as a split transaction and record the split identifier (New Region ID). This step is to ensure the atomicity of transaction submission;

[0298] iii. Allow transactions to perform write operations such as put / delete and allow transactions to enter the committed state.

[0299] S1012-2: After traversing all transactions, new transactions are allowed to be started, and the Region status is changed to L_WORKING, the split is completed, and the process ends.

[0300] S1013-1: If mc decides to roll back, clean up the split transaction, and traverse all transactions to make the split invalid and continue to provide external services:

[0301] After receiving the information, this node clears the New Region, traverses all active transactions, and performs the following operations on each transaction:

[0302] i. Release the newly created old_region_writebatch and new_region_writebatch, refer to the following Fig.11 "S1104: If the split fails, directly delete all data in the new data shard" step;

[0303] ii. Allow transactions to perform write operations such as put / delete and allow transactions to enter the committed state.

[0304] S1013-2: After traversing all transactions, new transactions are allowed to be started, and the region status is changed to L_WORKING, the split rollback is completed, and the process ends.

[0305] In the above steps S1001, S1006, and S1007, if an exception occurs, the split task status is set to REGION_SPLIT_FAILED, and the MC exception reporting process is entered.

[0306] If other replicas are found to be abnormal in the above step S1010, the process of reporting MC abnormality is directly entered. Since the operation cost of each transaction traversal by the Leader is relatively high, invalid execution should be avoided as much as possible.

[0307] 3. Follow receives the split task process issued by MC

[0308] like Fig.10 As shown, the splitting task process is as follows:

[0309] S101: Create and start a New Region based on the split key and other meta information.

[0310] S102: Build the split task context, set the task state to REGION_SPLIT_INIT, and confirm that other OldRegion copies enter this state;

[0311] S103: Update the current Old Region status from F_WORKING to F_SPLIT;

[0312] S104: Waiting for the split log of Apply Leader;

[0313] S105: Save a snapshot of the current Old Region;

[0314] S106: Set the split task status to REGION_SPLIT_NEW_REGION;

[0315] S107: Report to MC that the node split task is completed. MC will respond to the task status of each node after integrating the task status reported by all nodes to decide whether to continue or roll back to clean up the environment;

[0316] S108: If the MC decides to continue the execution, after receiving the information, the node modifies the end key information of the Old Region according to the split key and other meta-information, moves the New Region to the external service list, and modifies the Region status to F_WORKING, completing the split and ending the process;

[0317] S109: If the MC decides to roll back, after receiving the information, this node clears the New Region and changes the Region status to F_WORKING, completing the split rollback and ending the process.

[0318] In summary, from the perspective of the Region, the life cycle of a transaction can span splits, but the Region split action and transaction commit are mutually exclusive in time. Therefore, the Region needs to prevent read-write transactions from entering the commit state and wait for transactions that have entered the commit state to complete the commit before entering the split action; and only after the split action is completed, the read-write transaction is allowed to continue to enter the commit state.

[0319] The reason for this limitation is that when the storage cluster is restarted (recovered), multiple regions will be replayed concurrently. The correctness of the data here depends on the raft snapshot meta saved during the split. The snapshot index corresponding to this snapshot meta is the split index. After the Region is started, the log will be replayed from the snapshot index. If a transaction log spans the split index (for example, the prepare raft log is synchronized before the split index, and the commit raft log is synchronized after the split index), then after the Region is started, part of the transaction log will be missed and cannot be replayed, resulting in transaction data loss. Therefore, when synchronizing the split region log, it is necessary to ensure that all transactions in the commit process have ended.

[0320] As for the problem of QPS decline when splitting data into shards in a traditional distributed database mentioned in the above "Selection of Distributed Database Architecture", the solution corresponding to the technical solution of the embodiment of the present application is to create all copies of the New Region before prohibiting writing transactions on the Old Region, and then split the transaction data of the Old Region to the New Region after the Leader is elected in the New Region. This can reduce the prohibition time as much as possible and make it imperceptible to the business.

[0321] Transaction commit integrity is achieved as follows:

[0322] For a distributed transaction, there will be transaction contexts on multiple regions of the storage cluster. In order to ensure the atomicity of transactions submitted on multiple regions (either all are committed or all are rolled back), when submitting a transaction, SQLEngine will send a commit request to one of the regions. The request contains a list of regions that SQLEngine has visited. The region that receives the commit request acts as a coordinator and ensures the atomicity of transaction submission through 2PC.

[0323] Please refer to Fig.12 , which shows a flow chart of a 2PC protocol involved in this application. Fig.12 As shown in Figure 2, the 2PC protocol is completed by the coordinator and participants.

[0324] The coordinator enters the prepare phase and sends a prepare request to each participant.

[0325] After receiving the prepare request, the participant executes the submit response or does not submit the response. Submitting the response includes rewriting and entering the prepare phase after writing the prepare.

[0326] The coordinator collects responses from each participant and determines whether perform fails:

[0327] If it fails, abort perpare; after sending the abort feedback to all participants, all participants write abort;

[0328] If successful, it enters the commit phase and sends a commit request to each participant;

[0329] After receiving the commit request, each participant executes the response, including entering the commit phase after writing the commit;

[0330] The coordinator collects submission feedback from each participant and sends a confirmation request to each participant;

[0331] After receiving the confirmation request, the participant executes the response, including entering the confirmation phase after writing the confirmation, and clearing the transaction context;

[0332] After the coordinator collects confirmation feedback from each participant, it clears the transaction context.

[0333] If Region A splits into Region B before the transaction is committed, but SQLEngine is not aware of the existence of Region B, the commit request will not contain Region B, and the data on Region B will not be committed in the 2PC process, which destroys the atomicity requirement of the transaction.

[0334] In the above "Leader receives the split task process sent by MC", S1012-1 requires "setting this transaction as a split transaction and recording the split New Region ID" in all active transaction contexts on the Old Region. The 2PC Prepare request sent by the coordinator will carry a full list of participants. If the Old Region finds that the 2PC request does not contain the newly split New Region ID, it will return an error code and require the coordinator to supplement the participant list and then re-send the 2PC Prepare request.

[0335] Please refer to Fig.13 , which shows a 2PC case schematic diagram provided by an exemplary embodiment of the present application. Fig.13 As shown in the figure, a case is given. Transaction T writes four keys A, B, D, and G during the read and write phase. A and B are on Region 1, and D and G are on Region 2. In this way, Region 1 and Region 2 will be included in the Region Cache of transaction T. Assume that transaction T stops reading and writing for a while, and then directly executes commit. Coincidentally, during the gap when transaction T stops reading and writing, Region 2 splits, and the split generates a new Region 3. Note that the subtransaction space of transaction T on Region 2 is marked with is_split=true, new_region=3. After the split, transaction T initiates a commit request and designates Region 1 as the coordinator.

[0336] Please refer to Fig.14 , which shows Fig.13 Schematic diagram of the submission of the 2PC case. Fig.14 As shown in the figure, when Region 1 receives a transaction commit request from SQLEngine (where region_list is 1 and 2), the processing flow is as follows.

[0337] S1: The coordinator receives the transaction commit request and knows that the transaction participants are Region 1 and Region 2, so it sends a prepare request to Region 1 and Region 2. The prepare request carries a list of all participants known to the coordinator, Region 1 and Region 2.

[0338] S2: After receiving the prepare request, Region 1 checks the is_split variable of transaction T in the subtransaction space of this Region. The value of this variable is false, indicating that the Region has not been split during the life cycle of the subtransaction. Therefore, prepare is executed directly (writing logs to other replicas for synchronization), and after success, the coordinator is notified that preparation is complete (prepare ok).

[0339] S3: After receiving the prepare request, Region 2 checks the is_split variable of transaction T in the subtransaction space of this Region. The value of this variable is true, indicating that the Region has split during the life cycle of the subtransaction. It further checks whether the New Region after the split is in the member list of the prepare request. The value of new_region is 3, which is not in the member list. Therefore, it returns to the coordinator that it is not ready (prepare not ok) and tells the coordinator the value of New Region. Among them, the corresponding error code needs to be set so that the coordinator can perceive and handle this error specially. For example, the error code is EC_TDS_TRANS_PREPARE_NEED_MORE_PART, that is, transaction prepare need moreparticipants, which means that new participants need to be added to the participant list. When the coordinator receives this error code, it will read the more_participants field from the response, which contains the information that the ID of NewRegion is 3.

[0340] S4: After receiving the reply from Region 2, the coordinator finds that the corresponding error code is that the Region has split. It then obtains the ID of the New Region 3 from the request, adds Region 3 to the participant list, and restarts a new prepare phase: it sends prepare requests to Region 1, Region 2, and Region 3 at the same time. The prepare request carries a list of all participants known to the coordinator, Region 1, Region 2, and Region 3.

[0341] S5: After Region 1 receives the prepare request again, it finds that the status of transaction T in the subtransaction space of this Region is already in the prepare status. It compares the number of participants in the request with the number of participants it has recorded and finds that there are new participants. It then records the newly added Region 3 and submits the log. After success, it replies to the coordinator again with prepare ok.

[0342] S6: After Region 2 receives the prepare request again, it checks the is_split variable of transaction T in the subtransaction space of this Region. The value of this variable is true, indicating that the Region has split during the life cycle of the subtransaction. It further checks whether the New Region after the split is in the member list of the prepare request. The value of new_region is 3. In the member list, it executes prepare (writes logs and synchronizes them to other replicas) and responds to the coordinator with prepare ok after success.

[0343] S7: After receiving the prepare request, Region 3 checks the is_split variable of transaction T in the subtransaction space of this Region. The value of this variable is false, indicating that the Region has not been split during the life cycle of the subtransaction. Therefore, prepare is executed directly (writing logs to other replicas for synchronization), and responds to the coordinator with prepare ok after success.

[0344] S8: After receiving the prepare ok replies from Region 1, Region 2, and Region 3, the coordinator continues to push the 2PC transaction to the commit state.

[0345] From the above process, it can be seen that when the Region in the coordinator's participant list is split and the split New Region is not in the participant list, there will be an extra round of requests to fill in the participants.

[0346] If the New Region splits again after the coordinator completes the participant list, and a new Region appears, does it need another round of requests to complete the participant list? From the perspective of process design, it is necessary, so the possibility of this actually happening can be limited here.

[0347] A feasible solution is to strictly limit the minimum protection time from the generation to the splitting of the New Region, because the transaction usually ends soon after entering 2PC. In this way, when the minimum protection time is much longer than the commit time of the general transaction after entering 2PC, there will almost never be a situation where more than two rounds of participants are required to make up the difference.

[0348] The above minimum protection time means that after the New Region A generated by the split is created, at least after this minimum protection time has passed, the New Region A is allowed to process the split task and split into the New Region B. Among them, this time range can be in the second level, for example, 3s, and it only needs to be much longer than the time required for 2PC submission (under low load pressure, the general 2PC submission time is within 1s).

[0349] The purpose of setting the above minimum protection time is to prevent the coordinator from frequently receiving the EC_TDS_TRANS_PREPARE_NEED_MORE_PART error code. For example:

[0350] After Region A splits off Region B, Region A receives a prepare request, in which case this error code is set and Region B is placed in the more_participants field, informing the coordinator that Region B also needs to be a participant and submit. As a result, before the coordinator sends a prepare request to Region B, Region B splits off Region C, causing Region B to also set this error code and place Region C in the more_participants field, which results in too many rounds of communication spent on expanding the participant list.

[0351] The minimum protection time allows Region B to wait for a while before processing the split request. Therefore, when Region B splits from Region C, the 2PC process of the transaction has ended, which can reduce the impact on transaction submission and allow the transaction to complete the 2PC process quickly.

[0352] In addition, Region 1 in the above example will not be split during this process because the transaction has successfully entered the prepare state on this Region, which will prevent Region 1 from splitting.

[0353] Please refer to Fig.15 , which shows a 2PC flow chart provided by an exemplary embodiment of the present application.

[0354] like Fig.15 As shown in the figure, the transaction participant list accurately requested by 2PC only includes data shard 1, but data shard 1 splits into data shard 2, then an error is returned to inform the coordinator that the missing participant is missing, but it does not mean that the transaction submission failed;

[0355] The coordinator updates the participant list and resends the prepare request. The transaction participant list includes data shard 1 and data shard 2.

[0356] In summary, the embodiments of the present application provide a method for splitting data shards without transaction awareness, which can quickly split data shards and disperse hot spots in a system that supports distributed transactions. This data splitting method has a faster splitting speed while ensuring the integrity of transaction submission, which can ensure that there is no impact on the writing of the business, will not cause performance fluctuations and write failures, and can help the distributed system achieve smooth load balancing without user awareness.

[0357] An embodiment of the present application provides a data shard splitting method that is unaware of data writing. This method is applied to a storage system that supports transaction functions or batch atomic write functions, and can ensure the correctness of data writing and data shard splitting concurrency, and the integrity of transaction submission can also be guaranteed.

[0358] This method shortens the time it takes to block transactions, does not cause a drop in QPS, and does not cause transaction rollbacks. It ensures that data shards are split and that users are unaware of the writes.

[0359] This method does not require operation and maintenance personnel to manually adjust the partition size. The control node will automatically combine the sizes of each partition to evaluate whether the split task should be issued and automatically achieve load balancing.

[0360] In addition, the data sharding strategy mentioned in the embodiment of the present application is not limited to the distributed database system, and the basic data unit corresponding to the underlying storage is not limited to KV data. For example, file storage, B-Tree-based table data storage, etc. can use similar methods to achieve data sharding, as long as the system has a way to uniquely map the data to the corresponding data shard.

[0361] The "synchronous raft log" mentioned in the embodiments of this application is only a means of synchronizing the master and standby data based on the Raft Group. For storage systems that implement master-standby synchronization using the distributed consensus (Paxos) protocol, it can be replaced with "synchronous Paxos resolution"; for stand-alone databases, it can be replaced with "a write-ahead log (WAL) log on disk".

[0362] The embodiment of the present application uses the data partitioning method of "partitioning based on keyword intervals" to split a single region into two by specifying a split key. However, for a storage system using the "partitioning based on keyword hash value" technology, a split hash value can also be specified to split the data shards. Regardless of the data partitioning method, as long as it can be divided into two, the approach of this solution can be used.

[0363] Fig.16 It shows a block diagram of a data slicing processing device shown in an exemplary embodiment of the present application, which can be used to perform the following steps: Figure 3 , Figure 4 , Figure 5 , Figure 6 , Figure 7 or Figure 8 In the method shown, all or part of the steps performed by the computer device are as follows Fig.16 As shown, the device comprises:

[0364] A creation module 1601 is used to create a second data shard in response to a split instruction for a first data shard in the database; the split instruction is used to instruct to split the target data in the first data shard to a new second data shard;

[0365] A first pause module 1602, used to pause the transaction executed based on the first data shard to enter the commit phase;

[0366] The second suspension module 1603 is used to suspend the new transaction corresponding to the first data shard from entering the read-write phase in response to the transaction in the first data shard that has entered the commit phase being committed;

[0367] The transfer module 1604 is used to transfer the first transaction data from the first data shard to the second data shard; the first transaction data is the data of the first transaction corresponding to the data of the second data shard after the split; the first transaction is an active transaction of the first data shard, all or part of the data of which corresponds to the transaction of the second data shard;

[0368] The recovery module 1605 is used to respond to the completion of the first transaction data transfer and to recover the transaction corresponding to the first data shard to enter the read-write phase and the commit phase.

[0369] In some embodiments, the transfer module 1604 is used to:

[0370] Suspend write operations of active transactions;

[0371] Establishing a first container for the first transaction and a second container for the first transaction in the first data shard;

[0372] Writing the second transaction data into the first container, and writing the first transaction data into the second container; the second transaction data is the data of the first transaction corresponding to the first data shard after the split;

[0373] Copy the second container to the second data shard, and build a transaction context of the first transaction on the second data shard;

[0374] In response to the split being successful, write operations of active transactions are resumed.

[0375] In some embodiments, the apparatus further comprises: an updating module, configured to:

[0376] In response to the split being successful, updating the first container in the first data shard to be a write container of the first transaction in the first data shard;

[0377] The second container in the first data shard and the original write container of the first transaction in the first data shard are released.

[0378] In some embodiments, the apparatus further comprises: a modification module, configured to:

[0379] In response to successful splitting, modifying information of the first data shard;

[0380] The second data shard is moved to the list of data shards for external services.

[0381] In some embodiments, the apparatus further comprises: a deletion module, configured to:

[0382] In response to the split failure, deleting the first container and the second container;

[0383] Resuming write operations of active transactions executed based on the first data shard;

[0384] Clean up the second data shard.

[0385] In some embodiments, the apparatus further comprises: an establishing module for:

[0386] Create a split log for the first data shard;

[0387] Persist the data cached in the first data shard to disk;

[0388] A snapshot is saved for the first data shard.

[0389] In some embodiments, the apparatus further comprises: an acquisition module, configured to:

[0390] In response to the split instruction, a fourth data shard is created corresponding to the third data shard; the third data shard is one of the slave copies of the first data shard;

[0391] Obtain the split log of the first data shard;

[0392] A snapshot is saved for the third data shard.

[0393] In some embodiments, the second pause module 1603 is used to:

[0394] In response to the transaction executed based on the first data shard and entering the commit phase being committed, and each slave copy of the first data shard saving a snapshot, new transactions executed based on the first data shard are suspended from entering the read-write phase.

[0395] In some embodiments, the device further comprises: a moving module, configured to:

[0396] In response to the split being successful, modifying information of the third data shard;

[0397] The fourth data shard is moved to the list of data shards for external services.

[0398] In some embodiments, the device further comprises: a cleaning module, for:

[0399] In response to the split failure, clean up the fourth data shard.

[0400] In some embodiments, the apparatus further comprises: an adding module, configured to:

[0401] Mark the first transaction as a split transaction, and in the transaction context of the first data shard, add the ID of the second data shard to the ID of the data shard corresponding to the first transaction;

[0402] The device also includes: a query module, used to:

[0403] In response to the first data shard receiving a preparation request for a third transaction, and the third transaction being a transaction marked as split, querying an ID of a data shard corresponding to the third transaction in a transaction context of the first data shard;

[0404] In response to querying that the ID of the data shard corresponding to the third transaction includes the ID of the second data shard, a preliminary response is returned to the coordinator data shard of the third transaction, and the preliminary response includes the ID of the second data shard, so that the coordinator data shard can add the ID of the second data shard as the ID of the participant data shard of the third transaction.

[0405] Fig.17The structural block diagram of a computer device 1700 shown in an exemplary embodiment of the present application is shown. The computer device can be implemented as a server in the above-mentioned solution of the present application. The computer device 1700 includes a central processing unit (CPU) 1701, a system memory 1704 including a random access memory (RAM) 1702 and a read-only memory (ROM) 1703, and a system bus 1705 connecting the system memory 1704 and the central processing unit 1701. The computer device 1700 also includes a mass storage device 1706 for storing an operating system 1709, an application program 1710 and other program modules 1711.

[0406] The mass storage device 1706 is connected to the central processing unit 1701 through a mass storage controller (not shown) connected to the system bus 1705. The mass storage device 1706 and its associated computer readable medium provide non-volatile storage for the computer device 1700. That is, the mass storage device 1706 may include a computer readable medium (not shown) such as a hard disk or a Compact Disc Read-Only Memory (CD-ROM) drive.

[0407] Without loss of generality, the computer-readable medium may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer-readable instructions, data structures, program modules or other data. Computer storage media include RAM, ROM, Erasable Programmable Read Only Memory (EPROM), Electronically-Erasable Programmable Read-Only Memory (EEPROM) flash memory or other solid-state storage technology, CD-ROM, Digital Versatile Disc (DVD) or other optical storage, cassettes, tapes, disk storage or other magnetic storage devices. Of course, those skilled in the art will know that the computer storage medium is not limited to the above. The above-mentioned system memory 1704 and mass storage device 1706 can be collectively referred to as memory.

[0408] According to various embodiments of the present disclosure, the computer device 1700 can also be connected to a remote computer on the network through a network such as the Internet. That is, the computer device 1700 can be connected to the network 1708 through the network interface unit 1707 connected to the system bus 1705, or the network interface unit 1707 can be used to connect to other types of networks or remote computer systems (not shown).

[0409] The memory also includes at least one computer program, which is stored in the memory. The central processing unit 1701 implements all or part of the steps in the methods shown in the above embodiments by executing the at least one computer program.

[0410] In an exemplary embodiment, a chip is also provided. The chip includes a programmable logic circuit and / or program instructions. When the chip runs on a computer device, it is used to implement the data slicing processing method in the above aspect.

[0411] In an exemplary embodiment, a computer program product is also provided, the computer program product includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor reads and executes the computer instructions from the computer-readable storage medium to implement the data slicing processing method provided by the above-mentioned method embodiments.

[0412] In an exemplary embodiment, a computer-readable storage medium is also provided, in which a computer program is stored. The computer program is loaded and executed by a processor to implement the data sharding processing method provided by the above-mentioned method embodiments.

[0413] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0414] Those skilled in the art should be aware that in one or more of the above examples, the functions described in the embodiments of the present application can be implemented with hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein the communication media include any media that facilitates the transmission of a computer program from one place to another. The storage medium can be any available medium that a general or special-purpose computer can access.

[0415] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A data sharding processing method, characterized in that: The method comprises: In response to a split instruction for a first data shard in a database, creating a second data shard; the split instruction is used to instruct to split the target data in the first data shard to a new second data shard; Suspend the transaction executed based on the first data shard and enter the commit phase; In response to the transaction in the first data shard that has entered the commit phase being committed, pausing new transactions corresponding to the first data shard from entering the read-write phase; Transferring first transaction data from the first data shard to the second data shard; the first transaction data is the data of the first transaction corresponding to the data of the second data shard after the split; the first transaction is an active transaction of the first data shard, all or part of the data of which corresponds to the transaction of the second data shard; In response to the completion of the first transaction data transfer, the transaction corresponding to the first data shard is restored to enter the read-write phase and the commit phase.

2. The method according to claim 1, characterized in that The transferring the first transaction data from the first data shard to the second data shard comprises: Suspending the write operation of the active transaction; Establishing a first container for the first transaction and a second container for the first transaction in the first data shard; Writing the second transaction data into the first container, and writing the first transaction data into the second container; the second transaction data is the data of the first transaction corresponding to the first data shard after the split; Copying the second container to the second data shard, and building a transaction context of the first transaction on the second data shard; In response to the split being successful, the write operation of the active transaction is resumed.

3. The method according to claim 2, characterized in that Before the write operation of the active transaction is resumed in response to the split being successful, the method further includes: In response to successful splitting, updating the first container in the first data shard to a write container of the first transaction in the first data shard; The second container in the first data shard and the original write container of the first transaction in the first data shard are released.

4. The method according to claim 3, characterized in that The method further comprises: In response to successful splitting, modifying information of the first data shard; The second data shard is moved to the list of data shards for external services.

5. The method according to claim 2, characterized in that: The method further comprises: In response to a split failure, deleting the first container and the second container; Resuming write operations of active transactions executed based on the first data shard; Clean up the second data shard.

6. The method according to any one of claims 1 to 5, characterized in that: In response to the transaction in the first data shard that has entered the commit phase being committed, pausing a new transaction corresponding to the first data shard from entering the read-write phase includes: In response to the transaction in the first data shard that has entered the commit phase being committed, creating a split log for the first data shard; Persisting the data cached by the first data shard to disk; Saving a snapshot of the first data shard; New transactions corresponding to the first data shard are suspended to enter the read-write phase.

7. The method according to claim 6, characterized in that The method further comprises: In response to the split instruction, a fourth data shard is created corresponding to the third data shard; the third data shard is one of the slave copies of the first data shard; Obtaining a split log of the first data shard; A snapshot is saved for the third data shard.

8. The method according to claim 7, characterized in that In response to the transaction in the first data shard that has entered the commit phase being committed, pausing a new transaction corresponding to the first data shard from entering the read-write phase includes: In response to the transaction executed based on the first data shard and entering the commit phase being committed, and each slave copy of the first data shard saving a snapshot, new transactions executed based on the first data shard are suspended from entering the read-write phase.

9. The method according to claim 7, characterized in that: The method further comprises: In response to successful splitting, modifying information of the third data shard; The fourth data slice is moved to the list of data slices for external services.

10. The method according to claim 7, characterized in that The method further comprises: In response to the split failure, the fourth data shard is cleaned up.

11. The method according to claim 2, characterized in that Before the write operation of the active transaction is resumed in response to the split being successful, the method further includes: Mark the first transaction as a split transaction, and add the ID of the second data shard to the ID of the data shard corresponding to the first transaction in the transaction context of the first data shard; The method further comprises: In response to the first data shard receiving a preparation request for a third transaction, and the third transaction being a transaction marked as split, querying an ID of a data shard corresponding to the third transaction in a transaction context of the first data shard; In response to querying that the ID of the data shard corresponding to the third transaction includes the ID of the second data shard, a preliminary response is returned to the coordinator data shard of the third transaction, wherein the preliminary response includes the ID of the second data shard, so that the coordinator data shard adds the ID of the second data shard as the ID of the participant data shard of the third transaction.

12. A data sharding processing device, characterized in that: The device comprises: A creation module, configured to create a second data shard in response to a split instruction for a first data shard in the database; the split instruction is configured to instruct to split the target data in the first data shard into a new second data shard; A first pause module, used to pause the transaction executed based on the first data shard to enter the commit phase; A second suspension module is used to suspend the new transaction corresponding to the first data shard from entering the read-write phase in response to the transaction in the first data shard that has entered the commit phase being committed; A transfer module, configured to transfer first transaction data from the first data shard to the second data shard; the first transaction data is data of the first transaction corresponding to data of the second data shard after the split; the first transaction is an active transaction of the first data shard, all or part of the data of which corresponds to a transaction of the second data shard; A recovery module is used to restore the transaction corresponding to the first data shard to enter the read-write phase and the commit phase in response to completion of the first transaction data transfer.

13. A computer device, characterized in that: The computer device includes a processor and a memory, the memory stores at least one computer instruction, and the at least one computer instruction is loaded and executed by the processor to implement the data sharding processing method as described in any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one computer instruction, and the computer instruction is loaded and executed by the processor to implement the data slicing processing method as described in any one of claims 1 to 11.

15. A computer program product, characterized in that The computer program product includes computer instructions, which are stored in a computer-readable storage medium; the computer instructions are read and executed by a processor of a computer device to implement the method for data sharding processing as described in any one of claims 1 to 11.

Images