Database system, transaction cache-based data operation method, medium and product

Abstract

One or more embodiments of the invention provide a database system, a transaction cache-based data operation method, a medium and a product, each database node in the database system maintains at least one data fragment and a transaction cache set in one-to-one correspondence with each data fragment, the transaction cache set is used for recording transaction caches corresponding to transactions related to the corresponding data fragments; the method is applied to a first database node in the database system. The method comprises the following steps: for a to-be-processed target transaction, determining a target data fragment needing to be operated by the target transaction; and initiating a data operation request corresponding to the target transaction for the target data fragment, so as to execute the target transaction based on a target transaction cache recorded in a transaction cache set corresponding to the target data fragment, the target transaction cache being a transaction cache corresponding to the target transaction.

Description

Technical Field

[0001] One or more embodiments of the present specification relate to the field of database technology, and in particular, to a database system and a data operation method, medium, and product based on transaction cache. Background Art

[0002] Transaction cache is a technology commonly used in database systems that can be used to improve the performance of database operations. For example, transaction cache can be used to store the results of recently executed transactions so that the data can be quickly retrieved during the processing of subsequent transactions without having to access the data on the disk again.

[0003] In the related art, a buffer manager corresponding to a transaction coordinator (also known as TxnCoordinator) is usually used to manage transaction cache. However, this transaction cache is strongly dependent on the transaction coordinator, which results in the need to concentrate the data in the data shards that need to be operated on in the subsequent transaction processing process on the transaction coordinator, which increases additional resource overhead and reduces transaction processing efficiency. Summary of the invention

[0004] In view of this, one or more embodiments of this specification provide the following technical solutions:

[0005] According to a first aspect of one or more embodiments of the present specification, a data operation method based on transaction cache in a database system is proposed, wherein each database node in the database system maintains at least one data shard and a transaction cache set corresponding to each data shard, and the transaction cache set is used to record the transaction caches corresponding to each transaction related to the corresponding data shard; the method is applied to a first database node in the database system; the method comprises:

[0006] Determine the target data shard that the target transaction needs to operate on;

[0007] A data operation request corresponding to the target transaction is initiated for the target data shard, so as to execute the target transaction based on the transaction cache set corresponding to the target data shard.

[0008] According to a second aspect of one or more embodiments of the present specification, another data operation method based on transaction cache in a database system is proposed, wherein each database node in the database system maintains at least one data shard and a transaction cache set corresponding to each data shard, and the transaction cache set is used to record the transaction caches corresponding to each transaction related to the corresponding data shard; the method is applied to a first database node in the database system; the method includes:

[0009] Determine the target data shard that the target transaction needs to operate on;

[0010] A data operation request corresponding to the target transaction is initiated for the target data shard, so as to execute the target transaction based on the transaction cache set corresponding to the target data shard.

[0011] A third aspect of one or more embodiments of the present specification is proposed, a data operating system for a database, wherein each database node in the database system maintains at least one data shard and a transaction cache set corresponding to each data shard, wherein the transaction cache set is used to record the transaction caches corresponding to each transaction related to the corresponding data shard; wherein

[0012] A first database node, used to determine a target data shard that needs to be operated on by a target transaction, and initiate a data operation request corresponding to the target transaction for the target data shard in the first database node and / or the second database node;

[0013] The second database node is used to execute the target transaction based on the transaction cache set corresponding to the target data shard.

[0014] A fourth aspect of one or more embodiments of the present specification is proposed, a computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implements the steps of the method described in the first aspect or the second aspect.

[0015] A fifth aspect of one or more embodiments of the present specification proposes a computer program product, comprising a computer program / instructions, which, when executed by a processor, implement the steps of the method described in the first aspect or the second aspect.

[0016] It can be seen from the above embodiments that in this specification, by maintaining at least one data shard and a transaction cache set corresponding to each data shard on a database node, for example, when the first database node needs to process a target transaction, it can determine the target data shard that the target transaction needs to operate on, and initiate a data operation request corresponding to the target transaction for the target data shard, so as to execute the target transaction based on the transaction cache set corresponding to the target data shard, without the need to concentrate the data of the target data shard at the transaction coordinator for processing, thereby effectively reducing resource overhead and improving transaction processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1a It is a schematic diagram of the architecture of a database system based on transaction cache provided by an exemplary embodiment;

[0018] Figure 1b It is a schematic diagram of the architecture of another database system based on transaction cache provided by an exemplary embodiment;

[0019] Figure 2 It is a flowchart of a data operation method based on transaction cache in a database system provided by an exemplary embodiment;

[0020] Figure 3 It is a schematic diagram of executing a target transaction based on multiple database nodes in a distributed database provided by an exemplary embodiment;

[0021] Figure 4 It is a flowchart of executing a data write statement in a target transaction provided by an exemplary embodiment;

[0022] Figure 5 It is a flowchart of executing a data query statement in a target transaction provided by an exemplary embodiment;

[0023] Figure 6 It is a flowchart of another data operation method based on transaction cache in a database system provided by an exemplary embodiment;

[0024] Figure 7 It is a flowchart of a method for executing transaction operations on a database provided by an exemplary embodiment;

[0025] Figure 8 is a schematic structural diagram of an electronic device provided by an exemplary embodiment;

[0026] Fig. 9 It is a structural schematic diagram of a data operation device based on transaction cache in a database system shown in an exemplary embodiment;

[0027] Fig.10 It is a structural schematic diagram of another data operation device based on transaction cache in a database system shown in an exemplary embodiment;

[0028] Fig.11 The diagram is a schematic structural diagram of a device for executing transaction operations on a database, shown as an exemplary embodiment. DETAILED DESCRIPTION

[0029] When the database system processes transactions, it will generate a corresponding transaction cache for the transaction. The transaction cache is used to store the data modified by the transaction before it is submitted, so that the transaction can query or continue to modify the modified data by reading its own corresponding transaction cache before submission. Of course, due to the need for isolation, different transaction caches are isolated from each other, that is, each transaction can only query the data in its own corresponding transaction cache.

[0030] When a database node in a database system processes a transaction, it will manage the transaction cache corresponding to the transaction through the buffer manager running on the node, for example, the transaction caches of different transactions are uniformly maintained in a transaction cache set, and the buffer manager can further cooperate with the transaction coordinator service running on the node, so that the transaction coordinator service reads the above transaction cache through the buffer manager to meet the query or write requirements of the transaction. Of course, the data that needs to be queried or modified by the above transaction often does not directly hit the above transaction cache. In this case, the coordinator service also needs to continue to determine the data shard containing the above target data from the data storage layer of the local or other database nodes, and obtain the corresponding data from the data shard to carry out further processing in combination with the above transaction cache. Among them, each data shard can store a part of the data in the database in the form of a data set.

[0031] by Figure 1a For example, assume that the database system includes a first database node and a second database node. Each node is deployed with a transaction coordinator service, a transaction cache set, and a data storage layer that maintains multiple data shards. The client can initiate a data operation request to the first database node, and the transaction corresponding to the data operation request is responded to by the transaction coordinator service running in the first database node (corresponding to Figure 1a ). When the data operation request corresponds to a data query operation, the transaction coordinator service may first determine one or more data shards that the transaction needs to operate, and aggregate the data sets in the corresponding data shards to the transaction coordinator service of the first database node, and then process the transaction according to the aggregated data set and the transaction cache maintained by the cache manager of the first database node.

[0032] For example, assuming that a transaction corresponding to the transaction coordinator service of the first database node needs to query data shard 1 of the first database node, then the transaction coordinator service needs to read the data set in the data shard 1 from the local. Assuming that the transaction needs to query data shard 4 of the second database node, then the transaction coordinator service of the first database node needs to interact with the second database node across nodes to read the data set in data shard 4 from the second database node (corresponding to Figure 1aAfter obtaining the data set from the first database node or the second database node, the transaction coordinator service of the first database node needs to merge it with the transaction cache corresponding to the above transaction in the transaction cache set of the first database node before the actual data query operation can be performed. Of course, in cases such as complex queries, the same transaction may also involve multiple data shards of different database nodes at the same time.

[0033] However, the defect of the above transaction processing method is that no matter which node the data shard is in, the data query operation for its data set is completely dependent on the transaction cache in the node where the transaction coordinator is located. If the transaction cache is ignored and only the data to be queried is calculated based on the data set, then the query result may deviate from the actual situation because the modification operations executed on the data to be queried in the transaction cache are not taken into account. In other words, the data query process of the above scheme mainly occurs in the database node where the transaction coordinator service is located after all relevant data sets are concentrated, and then the database node alone participates in the calculation of the query results, and the remaining database nodes (such as Figure 1a The second database node) does not actually participate in any calculation of the query results, but only realizes the transmission of the data set, resulting in low resource utilization of some database nodes and high consumption of network resources, which in turn causes additional performance waste and network overhead, and also reduces the efficiency of transaction processing.

[0034] For example, suppose that a user information table containing 10,000 pieces of user information is maintained in the database, of which the first 5,000 pieces of information are recorded in the data shard 1 of the first database node, and the last 5,000 pieces of information are recorded in the data shard 4 of the second database node. In the same transaction, if the annual income of some user information in the data shard 4 is first modified, and then the sum of the annual incomes of 10,000 users is queried, since the modified income information is recorded in the transaction cache of the first database node, all 10,000 pieces of data in the data shard 1 and the data shard 4 must be concentrated in the first database node, and then merged with the transaction cache of the first database node to obtain accurate user information, and the sum of the annual income is calculated based on this. It can be understood that the second database node needs to consume a lot of network resources to transmit the 5,000 pieces of user information in the data shard 4 to the first database node, and the second database node itself does not participate in any calculation of the above-mentioned sum of annual income, which seriously wastes performance; for the first database node, it is also necessary to complete the merging process and all calculation operations at the same time, resulting in low transaction processing efficiency.

[0035] Based on this, this specification proposes the following technical solutions to solve the above problems.

[0036] Figure 1b FIG. 1 is a schematic diagram of another database system based on transaction cache provided by an exemplary embodiment. Figure 1b As shown, in a distributed database, a client can initiate a data operation request to a first database node, and the transaction corresponding to the data operation request is responded to by a transaction coordinator service running in the first database node (corresponding to Figure 1b The solid line part). After receiving the data operation request, the transaction coordinator service can determine the target data shard that the transaction corresponding to the data operation request needs to operate on, and then perform data operations on the determined target data shard. Among them, the transaction cache is maintained in the transaction cache set of the corresponding data shard, that is, for each transaction, its transaction cache is divided into one or more data shards that need to be operated. Therefore, when performing data operations on the data set in any data shard, the execution of the transaction can be completed directly based on the transaction cache set in the data shard. In this way, the transaction coordinator service only needs to obtain the necessary data related to the above data operation in the above data set (corresponding to Figure 1b The dotted part of the data shard does not need to be returned, which effectively reduces resource overhead and improves transaction processing efficiency.

[0037] Still taking the above user information table as an example, in the scheme of this specification, the modified income information can be recorded in the transaction cache of data shard 4 of the second database node, which means that the second database node can independently calculate the sum of the annual income of the last 5,000 pieces of user information based on the corresponding transaction cache, and then send the corresponding calculation result as an intermediate result to the first database node. Compared with the original transmission of 5,000 pieces of information, this process is optimized to only need to transmit one intermediate result. The first database node only needs to calculate the sum of the annual income of 5,000 pieces of user information of data shard 1, and then calculate it again with the above intermediate result to get the sum of the annual income of 10,000 pieces of user information. Obviously, this method avoids the resource overhead required to transmit 5,000 pieces of information in the form of intermediate results, and each database node participates in part of the calculation process, avoiding performance waste and improving transaction processing efficiency.

[0038] Figure 2 FIG. 1 is a flow chart of a data operation method based on transaction cache in a database system provided by an exemplary embodiment of this specification. Figure 2 As shown, each database node in the database system maintains at least one data shard and a transaction cache set corresponding to each data shard, and the transaction cache set is used to record the transaction caches corresponding to each transaction related to the corresponding data shard; the method is applied to the first database node in the database system; the method includes:

[0039] Step S202: for the target transaction to be processed, determine the target data shard that needs to be operated on by the target transaction.

[0040] When a user needs to operate the data in the database, the corresponding database statement can be input into the client, and then the client constructs a corresponding request to be sent to the first database node in the database system (corresponding to Figure 3 The solid line part). The above database statement may adopt structured query language (SQL) or graph query language (GQL) according to the actual database type and application scenario, and the above request may include all information required to perform the above data operation, such as SQL statement / GQL statement, parameters and transaction control instructions. In addition, the above first database node may be any database node randomly determined in the database system, or may be a database node specified by the node connection information preset by the client or the node connection rule preset by the above database system, which is not limited in this specification.

[0041] For the above-mentioned first database node, after receiving the above-mentioned request from the client, the target transaction to be processed corresponding to the request can be determined, so as to execute the database statement in the above-mentioned request in the target transaction. Among them, the above-mentioned target transaction may have the above two situations: 1. The target transaction already exists, that is, the first database node has executed other database statements for the target transaction, and the above-mentioned target transaction has not been submitted, so the database statement in the above-mentioned request can continue to be executed in the target transaction; 2. The target transaction does not exist, then the above-mentioned first database node can create a new transaction as the above-mentioned target transaction according to the transaction control instruction in the above-mentioned request. Of course, if the transaction control instruction does not exist in the above-mentioned request, then when executing each database statement, it can be regarded as a separate target transaction and automatically submitted after execution. Among them, the above-mentioned database statements executed can be specifically data manipulation language (Data Manipulation Language, DML) type, data query language (Data Query Language, DQL) type or other statement types that depend on transaction execution. For the convenience of discussion, this specification will unify the above-mentioned database statements as DML statements or DQL statements in subsequent descriptions.

[0042] When the target transaction is determined and the first database node is about to execute the corresponding database statement in the target transaction, the transaction coordinator service of the first database node can determine the target data shard that the target transaction needs to operate, that is, the data shard in the database system where the data involved in each DML statement or DQL statement in the target transaction is located. Among them, since the transaction coordinator service is an important component for managing and controlling distributed transactions in a distributed database, it can guarantee the atomicity, consistency, isolation and persistence of transactions across multiple service boundaries or database boundaries as a centralized coordinator. Therefore, even if the target data shard includes the first data shard maintained in the first database node, and / or the second data shard in the second database node different from the first database node in the database system, the transaction coordinator service can still determine the first data shard and / or the second data shard contained in the target data shard.

[0043] The data shard where each data is located is determined by the database sharding strategy, which can include range sharding, hash sharding, random sharding, etc. Of course, no matter which sharding strategy is adopted, the first database node can choose to obtain the data shard to be operated by each database statement based on actual needs, such as range matching, hash mapping, metadata table or directory service, and then determine all the target data shards that the target transaction needs to operate. Taking range sharding as an example, suppose Figure 3 A user information table is maintained in the corresponding database system, which stores the identity information of 1 million users, and each user is assigned a user ID with a corresponding serial number. The range sharding strategy can stipulate that: the identity information of user IDs ranging from 0 to 300,000 is stored in data shard 2 of the first database node; the identity information of user IDs ranging from 300,001 to 600,000 is stored in data shard 3 of the first database node; the identity information of user IDs ranging from 600,001 to 900,000 is stored in data shard 4 of the second database node; the identity information of user IDs ranging from 900,001 to 1,200,000 is stored in data shard 5 of the second database node... and so on. Then, when the above database statement is used as a DML statement and is used to modify the identity information of user IDs 50000, 500000 and 800000 in data table A, the above transaction coordinator service can determine that the three pieces of identity information to be modified are located in data shard 2 and data shard 3 of the first database node, and data shard 4 of the second database node according to the size relationships of 0<50000<300000, 300001<500000<600000, and 600001<800000<900000.

[0044] Step S204, initiating a data operation request corresponding to the target transaction for the target data shard, so as to execute the target transaction based on the target transaction cache recorded in the transaction cache set corresponding to the target data shard, wherein the target transaction cache is the transaction cache corresponding to the target transaction.

[0045] After determining the target data shards that the target transaction needs to operate, the first database node can initiate data operation requests corresponding to the target transaction for each determined target data shard, wherein the target data shards can be located in other nodes such as the first database node and / or the second database node. At the same time, for the database node where the target data shard that receives the data operation request is located, the received data operation request can be processed through the shard management service corresponding to the target data shard.

[0046] Specifically, the data operation request may be initiated by the target transaction coordinator service corresponding to the target transaction running on the first database node, and the sending target may also be determined by the target data shard determined by the target transaction coordinator service in step S202. Figure 3 For example: the target data shard includes the first data shard in the first database node (for example Figure 3 In the case of data shards 1 to 3), the target transaction coordinator service can initiate the above data operation request to the shard management service corresponding to the first data shard in the same first database node (corresponding to Figure 3 The target data shard includes the second data shard in the second database node (eg Figure 3 In the case of data shards 4 to 6), the target transaction coordinator service can interact with the second database node across nodes (corresponding to Figure 3 The right dotted part of ) is used to initiate the above data operation request to the shard management service corresponding to the second data shard.

[0047] It can be understood that in the data query process of this specification, the shard management service only needs to query the transaction cache set corresponding to the local target data shard to return the data related to the target transaction. Compared with related technologies, there is no need to remotely transmit all relevant data to the first database node and then perform data merging and query operations, which effectively reduces resource overhead. Among them, it should be specially noted that, for the case where the target data shard includes the first data shard, although the process of transmitting all data to the cache manager corresponding to the transaction coordinator belongs to local transmission, the resource overhead is smaller than the remote transmission across nodes, but because the transaction cache is configured in the data shard of the database node in the present application, that is, it is in the same data storage layer as the persistent data of the database, the query operation of the persistent data of the data shard and the query operation of the cache data can be implemented by calling the same interface, compared with the traditional technology of first calling the first interface for the query operation of the persistent data to obtain the corresponding persistent data from the data shard, and then after transmitting the persistent data to the cache manager corresponding to the transaction coordinator, calling the second interface for the query operation of the cache data to obtain the corresponding cache data and the persistent data for merging, the above design of the transaction cache in the present application can optimize the calling logic of the corresponding interface, reduce the total number of calls, and make it have the advantages of high development and maintenance efficiency and high execution efficiency.

[0048] In addition, for any data shard, the above-mentioned shard management service can actually be regarded as a shard management process, which is used to perform data operations such as writing and querying on the persisted data set in any data shard. At the same time, each data shard can be uniformly processed by the same shard management process for data operations, or independently processed by a one-to-one corresponding shard management process. This manual does not limit this.

[0049] When the shard management service successfully receives the data operation request, it can execute the target transaction based on the transaction cache set corresponding to the target data shard. Specifically, it executes the database statement corresponding to the target data shard in the target transaction based on the target transaction cache corresponding to the target transaction in the transaction cache set. In this process, the shard management service needs to determine the target transaction cache corresponding to the request. This specification adds additional transaction information to the data operation request.

[0050] In one embodiment, the first database node can record the transaction information of the target transaction, which includes the transaction identifier (TXID or TID) of the target transaction, and then add the transaction information to the data operation request, so as to use the transaction cache matching the transaction identifier in the transaction cache set corresponding to the target data shard as the target transaction cache, and execute the target transaction based on the target transaction cache. In this embodiment, the transaction identifier is an identifier used to uniquely identify a transaction in the DBMS, which can be used to track and manage the life cycle of the target transaction. Therefore, the shard management service can accurately locate the target transaction cache from the transaction cache set corresponding to the target data shard through the transaction identifier. In addition, the transaction isolation level of the target transaction can also be added to the transaction information so that the shard management service can understand the degree of mutual influence between different transactions, such as the amount of data visible to other transactions during the execution of a transaction, and the type of interaction that may occur between transactions.

[0051] The shard management service of this specification can also implement anomaly detection of the transaction cache collection based on transaction information.

[0052] In one embodiment, the transaction information may further include a historical data shard set, which indicates all data shards that the target transaction has ever involved. After the first database node initiates a data operation request corresponding to the target transaction for the target data shard, the target data shard may be updated to the historical data shard set, and the data operation statement corresponding to the data operation request is a data write statement. Wherein, if the transaction cache set corresponding to the target data shard does not contain the target transaction cache, and the historical data shard set contained in the transaction information includes the target data shard, the shard management service may determine that the transaction cache set corresponding to the target data shard is confirmed to be abnormal. In this embodiment, the historical data shard set will change with the execution of the DML statement or DQL statement in the target transaction. For example, the target transaction includes the following two DML statements 1 and 3 and DQL statement 2:

[0053] 1. INSERT INTO employees(employee name, position, joining time) VALUES('User A', 'Software Engineer', '2024-01-01'); assume that it corresponds to data shard 1.

[0054] 2. SELECT employee name, position FROM employees; assume that they correspond to data shards 1, 2, and 3.

[0055] 3. UPDATE employees SET position = 'Senior Software Engineer' WHERE employee name = 'User B'; assume that it corresponds to data shard 2.

[0056] Among them, when the data operation request corresponding to the first DML statement is sent by the first database node, since the DML statement belongs to a data write statement, it will affect the transaction cache of the corresponding data shard. Therefore, the transaction coordinator service of the first database node can record data shard 1 to the historical data shard set. At this time, the historical data shard set includes: data shard 1. Then, when the data operation request corresponding to the second DQL statement is sent by the first database node, since the DQL statement belongs to a data query statement, it will definitely not affect the transaction cache of the corresponding data shard. Therefore, the transaction coordinator service of the first database node does not need to record data shards 1, 2, and 3 to the historical data shard set. At this time, the historical data shard set still only includes: data shard 1. Finally, when the data operation request corresponding to the third DML statement is sent by the first database node, since the DML statement belongs to a data write statement, the transaction coordinator service of the first database node can record data shard 2 to the historical data shard set. At this time, the historical data shard set includes: data shards 1 and 2.

[0057] The above historical data shard set is maintained by the first database node, and the target data shard is recorded in the historical data shard only after the target transaction coordinator service initiates a data operation request corresponding to a data write statement to the target data shard. In other words, for the shard management service that receives the data operation request for the first time, its corresponding transaction cache set must not have a transaction cache matching the above transaction information, that is, the target transaction cache, and the above historical data shard set contained in the above transaction information must not include the above target data shard; if an exception occurs in the shard management service or the transaction cache set, resulting in the loss of the transaction cache, then the above historical data shard set contained in the above transaction information plays a secondary confirmation effect, that is, when the transaction cache set corresponding to the above target data shard does not have the above target transaction cache, but the historical data shard set contained in the above transaction information includes the target data shard, the shard management service can determine the following situation: the first database node has initiated at least one data operation request to the shard management service, and the data operation statement corresponding to the data operation request is a data write statement, but for some reason, the transaction cache corresponding to the target transaction is not generated in the transaction cache set. Then, it is finally determined that the transaction cache set corresponding to the target data shard is confirmed to be abnormal. In addition, the abnormality can be notified to the target transaction coordinator service of the first database node through the shard management service, and then sent to the client as prompt information.

[0058] According to different types of data operation statements corresponding to the target transaction, the process of executing the target transaction based on the transaction cache set corresponding to the target data shard is also divided into a variety of different situations.

[0059] In one embodiment, the data operation statements corresponding to the target transaction may include data write statements; when the target transaction cache does not exist in the transaction cache set corresponding to the target data shard, and the historical data shard set contained in the transaction information does not include the target data shard, it means that no data write statement has ever been executed for the data of the target data shard in the target transaction, and therefore the corresponding shard management service may create a target transaction cache for the target transaction, and execute the data write statement based on the target transaction cache; correspondingly, when the target transaction cache exists in the transaction cache set corresponding to the target data shard, it means that at least one data write statement has been executed for the data of the target data shard in the target transaction, and therefore the data write statement may be directly executed based on the target transaction cache.

[0060] Specifically, the above data write statement may include a data insert statement, a data update statement or a data delete statement, and the corresponding execution conditions of the above shard management service are as follows;

[0061] 1. In the case where the data writing statement is the data inserting statement, the to-be-inserted data corresponding to the data inserting statement may be directly written into the target transaction cache.

[0062] 2. In the case where the data write statement is the data update statement, it is firstly determined whether the target transaction cache contains the data to be updated corresponding to the data update statement. If so, the content of the data to be updated can be updated. If not, the data to be updated can be obtained from the persistent data corresponding to the target data shard, and the obtained data to be updated can be updated and written into the target transaction cache. Of course, if the data to be updated still does not exist in the persistent data, no additional operation can be performed on the target transaction cache, or a prompt message indicating the update failure can be fed back to the target transaction coordinator service.

[0063] 3. In the case where the data write statement is the data delete statement, it can be first determined whether the target transaction cache has the data to be deleted corresponding to the data delete statement. If so, the data to be deleted is marked as deleted; if not, the data to be deleted is obtained from the persistent data corresponding to the target data shard, and the obtained data to be deleted is marked as deleted and written into the target transaction cache. It can be understood by those skilled in the art that the data deletion operation of a database is often implemented by setting specific fields of the corresponding records. Therefore, the operation of marking the data to be deleted as deleted can be regarded as, for example, setting the value of the field representing the mark of the data to be deleted to 1 / 0, which is not limited in this specification.

[0064] It should be noted that the above-mentioned persistent data refers to data that has been securely stored in the database and can be regarded as Figure 2 The data set in the database will not be lost due to system failure. Among them, the database system can access and manage persistent data through the storage engine (StorageEngine). The above storage engine, as a component of the DBMS, can be used to store, index and retrieve data. Different database systems may support multiple storage engines, each of which has its own specific characteristics and optimization points, which are not limited in this specification.

[0065] The following takes the data update statement in the target transaction as an example. Figure 3 and Figure 4 The data writing process of the client is introduced, including: Figure 3 The distributed database system shown in the figure includes two database nodes, namely the first database node and the second database node. Each database node is configured with a unique transaction coordinator service, and the data storage layer of each node maintains three data shards (i.e. Figure 3 data shards 1-3, 4-6) and a transaction cache set corresponding to each data shard, the transaction cache set can be used to record the transaction caches corresponding to each transaction related to the corresponding data shard, Figure 4 The corresponding steps are as follows:

[0066] Step S402: The client selects a first database node to establish a connection.

[0067] In one embodiment, the client may randomly determine a first database node from the database system to create a database connection to facilitate execution of a subsequent target transaction.

[0068] Step S404: The target transaction coordinator service of the first database node registers the target transaction and records the corresponding transaction information.

[0069] In one embodiment, it is assumed that the client sends a transaction control instruction for starting a transaction to the first database node. At this time, the transaction coordinator service in the first database node responds to the transaction coordinator service to register the corresponding target transaction and records the corresponding transaction information. The transaction information may at least include a transaction identifier "1" (assuming that the transaction identifier is determined based on an incremental number), a corresponding transaction isolation level, and a temporarily empty set of historical data shards modified by the transaction.

[0070] Step S406, the target transaction coordinator determines the target data shard that the target transaction needs to operate on and sends a data operation request with transaction information added.

[0071] In one embodiment, it is assumed that the client executes a data update statement "UPDATE employees" for the target transaction.

[0072] SET salary=salary*1.1WHERE department='Sales';", which is used to increase the salaries of employees in the sales department. The target transaction coordinator determines that the data to be updated is located at data node 1 of the first database node and data node 4 of the second database node respectively through parsing and calculating the data update statement. Then the target transaction coordinator will initiate a data operation request with the above transaction information added to the local data node 1 and data node 4 of the second database node. At the same time, after confirming that all data operation requests are successfully sent, the node information corresponding to data node 1 and data node 4 can also be updated to the historical data shard set.

[0073] Step S408: The shard management service receives a data operation request.

[0074] Step S410: The shard management service determines whether there is a target transaction cache in the transaction cache set.

[0075] In one embodiment, the shard management services of the first database node and the second database node respectively receive corresponding data operation requests. Taking the shard management service of the second database node as an example, the transaction cache set corresponding to data shard 4 can be first queried for a transaction cache matching the above transaction identifier "1". If so, step S412 is executed; if not, step S414 is executed.

[0076] Step S412: The shard management service retrieves the target transaction cache from the transaction cache set.

[0077] In one embodiment, the shard management service may directly retrieve the target transaction cache corresponding to the target transaction from the transaction cache set.

[0078] Step S414: The shard management service determines whether the historical data shard set contained in the transaction information includes the target data shard.

[0079] In one embodiment, still taking the shard management service of the second database node as an example, it is known that there is no transaction cache corresponding to the above-mentioned target transaction in the transaction cache set corresponding to the data shard 4. Then, it can be further determined whether the historical data shard set contained in the transaction information includes the above-mentioned data shard 4. If it does, execute step S416; if not, execute step S418.

[0080] Step S416, the shard management service notifies abnormal information.

[0081] In one embodiment, if the target transaction cache does not exist in the transaction cache set corresponding to the target data shard, and the historical data shard set contained in the transaction information includes the target data shard, then the transaction cache set corresponding to the above target data shard can be confirmed to be abnormal, and the target transaction coordinator is notified of the abnormal information.

[0082] Step S418: The shard management service creates a target transaction cache.

[0083] In one embodiment, still taking the shard management service of the second database node as an example, the shard management service can create a target transaction cache for the above target transaction in the transaction cache set corresponding to data shard 4.

[0084] Step S420: The shard management service determines whether there is data to be updated in the target transaction cache.

[0085] In one embodiment, after obtaining the target transaction cache, the shard management service may determine whether there is data to be updated in the target transaction cache. If so, step S422 is executed; if not, step S424 is executed.

[0086] Step S422: The shard management service updates the content of the data to be updated.

[0087] Step S424: The shard management service obtains the data to be updated from the persistent data corresponding to the target data shard.

[0088] Step S426: The shard management service determines whether the data to be updated is obtained.

[0089] In one embodiment, since the target transaction cache does not contain the data to be updated, the shard management service can obtain the data to be updated, that is, the original salary data of the employees in the sales department, from the persistent data corresponding to the target data shard. At the same time, if the data to be updated is successfully obtained in the persistent data, step S428 is executed; if the acquisition fails, step S430 is executed.

[0090] Step S428, the shard management service writes the updated data into the target transaction cache.

[0091] Step S430: The shard management service prompts that the data update fails.

[0092] In one embodiment, the shard management service does not successfully obtain the data to be updated in the persistent data, that is, the original salary data of the employees in the sales department does not exist in the persistent data, so there is no way to update the data.

[0093] Step S432: The shard management service returns the statement execution result to the target transaction coordinator service.

[0094] In one embodiment, the shard management service can return the write result to the target transaction coordinator service. If a write error occurs, it means that the target transaction execution has failed and a rollback operation can be triggered. If the write is successful, the target transaction coordinator service can continue to wait for the next data operation statement.

[0095] In another embodiment, the data operation statement corresponding to the target transaction may include a data query statement; when the target transaction cache does not exist in the transaction cache set corresponding to the target data shard, and the historical data shard set contained in the transaction information does not include the target data shard, the data query statement may be executed for the persistent data corresponding to the target data shard; correspondingly, when the target transaction cache exists in the transaction cache set corresponding to the target data shard, the data query statement may be executed based on the target transaction cache.

[0096] Specifically, the execution process of the above shard management service for data query statements is as follows:

[0097] Determine whether the target transaction cache contains the data to be queried corresponding to the data query statement. If so, obtain the data to be queried from the target transaction cache. If not, obtain the data to be queried from the persistent data corresponding to the target data shard.

[0098] The following takes the execution of the data query statement in the target transaction as an example. Figure 3 and Figure 5 This chapter introduces the data query process of the client, including: Figure 5The corresponding steps are as follows:

[0099] Step S502: The client selects a first database node to establish a connection.

[0100] Step S504: The target transaction coordinator service of the first database node registers the target transaction and records the corresponding transaction information.

[0101] The above steps are consistent with steps S402 to S404, so they are not described in detail in this specification.

[0102] Step S506: The target transaction coordinator determines the target data shard that the target transaction needs to operate on and sends a data operation request with transaction information added.

[0103] In one embodiment, it is assumed that the client executes a data query statement "Select salary from employees WHERE employeesid>5000and employeesid<20000;" for the target transaction, that is, it is used to query the salaries of employees whose employee numbers are between 5000 and 20000 (excluding 5000 and 20000), wherein the target transaction coordinator determines that the data to be updated are located in the data node 2 of the first database node and the data node 4 of the second database node respectively through parsing and calculating the data update statement, then the target transaction coordinator will initiate a data operation request with the above transaction information added to the local data node 2 and the data node 4 of the second database node.

[0104] Step S508: The shard management service receives a data operation request.

[0105] Step S510: The shard management service determines whether there is a target transaction cache in the transaction cache set.

[0106] In one embodiment, the shard management services of the first database node and the second database node respectively receive corresponding data operation requests. Taking the shard management service of the second database node as an example, the transaction cache matching the above transaction identifier can be first queried in the transaction cache set corresponding to data shard 4. If there is one, execute step S512; if not, execute step S516.

[0107] Step S512: The shard management service retrieves the target transaction cache from the transaction cache set.

[0108] Step S514: The shard management service determines whether there is any data to be queried in the target transaction cache.

[0109] In one embodiment, the shard management service may query the target transaction cache for the data to be queried. If the query fails, step S520 is executed. If the query succeeds, step S522 is executed.

[0110] Step S516: The shard management service determines whether the historical data shard set contained in the transaction information includes the target data shard.

[0111] Step S518: The shard management service notifies abnormal information.

[0112] The above steps are consistent with steps S414 to S416, so they are not described in detail in this specification.

[0113] Step S520: query the data to be queried from the persistent data corresponding to the target data shard.

[0114] In one embodiment, still taking the shard management service of the second database node as an example, the shard management service can query the data to be queried from the corresponding persistent data through the storage engine corresponding to the data shard 4.

[0115] Step S522: The shard management service returns the statement execution result to the target transaction coordinator service.

[0116] In one embodiment, the shard management service can return the query results to the target transaction coordinator service. If the query is wrong, it means that the target transaction execution has failed and a rollback operation can be triggered. If the query is successful, the target transaction coordinator service can summarize the results and display the query results.

[0117] In addition, the transaction commit or rollback process of this specification can also be implemented based on the above historical data shard collection.

[0118] In one embodiment, the target transaction coordinator service of the first database node can record the transaction information of the target transaction, which also includes the historical data shard set involved in the target transaction; at the same time, when a commit or rollback instruction for the target transaction is received, a commit request or rollback request for the target transaction can be initiated for each data shard in the historical data shard set, wherein, since the historical data shard set records the target data shard only after the data operation request corresponding to each data write statement in the target transaction is successfully sent, the data shards in the historical data shard set are equivalent to all data shards involved in the data write statements in the target transaction. In other words, the commit request or rollback request initiated based on the historical data shard set can fully cover the data modified in the target transaction, thereby ensuring the atomicity and consistency of the target transaction.

[0119] Figure 6FIG. 1 is a flow chart of another data operation method based on transaction cache in a database system provided by an exemplary embodiment of this specification. Figure 6 As shown, each database node in the database system maintains at least one data shard and a transaction cache set corresponding to each data shard, and the transaction cache set is used to record the transaction caches corresponding to each transaction related to the corresponding data shard; the method is applied to the second database node in the database system; the method includes:

[0120] Step S602: obtaining a data operation request initiated by a first database node different from the second database node in the database system for a target data shard in the second database node, wherein the target data shard is required to be operated for a target transaction to be processed.

[0121] Step S604: Execute the target transaction based on the target transaction cache recorded in the transaction cache set corresponding to the target data shard, where the target transaction cache is the transaction cache corresponding to the target transaction.

[0122] As described above, the obtaining of a data operation request initiated by a first database node different from the second database node in the database system for a target data shard in the second database node includes:

[0123] The data operation request initiated by the shard management service corresponding to the target data shard is obtained, where the data operation request is generated by a target transaction coordinator service corresponding to the target transaction running on the first database node.

[0124] As mentioned above, the method further comprises:

[0125] Acquire transaction information in the data operation request, where the transaction information includes a transaction identifier of the target transaction;

[0126] The transaction cache that matches the transaction identifier in the transaction cache set corresponding to the target data shard is used as the target transaction cache, and the target transaction is executed based on the target transaction cache.

[0127] As mentioned above, the transaction information also includes a historical data shard set; the method also includes: when the target transaction cache does not exist in the transaction cache set corresponding to the target data shard, and the historical data shard set contained in the transaction information includes the target data shard, confirming that there is an abnormality in the transaction cache set corresponding to the target data shard.

[0128] As mentioned above, the data operation statements corresponding to the target transaction include data write statements;

[0129] The executing the target transaction based on the transaction cache set corresponding to the target data shard includes:

[0130] If the target transaction cache does not exist in the transaction cache set corresponding to the target data shard, and the historical data shard set contained in the transaction information does not include the target data shard, create the target transaction cache, and execute the data write statement based on the target transaction cache;

[0131] In a case where the target transaction cache exists in the transaction cache set corresponding to the target data shard, the data write statement is executed based on the target transaction cache.

[0132] As mentioned above, the data write statement includes a data insert statement, a data update statement or a data delete statement; and executing the data write statement based on the target transaction cache includes:

[0133] In the case where the data writing statement is the data insert statement, writing the to-be-inserted data corresponding to the data insert statement into the target transaction cache;

[0134] In the case where the data write statement is the data update statement, determining whether the target transaction cache has the data to be updated corresponding to the data update statement, and if so, updating the content of the data to be updated; if not, obtaining the data to be updated from the persistent data corresponding to the target data shard, updating the obtained data to be updated and writing it into the target transaction cache;

[0135] In the case where the data write statement is the data delete statement, determine whether there is data to be deleted corresponding to the data delete statement in the target transaction cache. If so, mark the data to be deleted as deleted; if not, obtain the data to be deleted from the persistent data corresponding to the target data shard, mark the obtained data to be deleted as deleted and write it into the target transaction cache.

[0136] As mentioned above, the data operation statements corresponding to the target transaction include data query statements;

[0137] The executing the target transaction based on the transaction cache set corresponding to the target data shard includes:

[0138] If the target transaction cache does not exist in the transaction cache set corresponding to the target data shard, and the historical data shard set contained in the transaction information does not include the target data shard, executing the data query statement for the persistent data corresponding to the target data shard;

[0139] In a case where the target transaction cache exists in the transaction cache set corresponding to the target data shard, the data query statement is executed based on the target transaction cache.

[0140] As mentioned above, executing the data query statement based on the target transaction cache includes:

[0141] Determine whether the target transaction cache contains the to-be-queried data corresponding to the data query statement. If so, obtain the to-be-queried data from the target transaction cache. If not, obtain the to-be-queried data from the persistent data corresponding to the target data shard.

[0142] As mentioned above, the method further comprises:

[0143] Acquire transaction information in the data operation request, where the transaction information includes a set of historical data shards of the target transaction;

[0144] When a commit or rollback instruction for the target transaction is received, a commit request or a rollback request for the target transaction is initiated for each data shard in the historical data shard set.

[0145] It should be noted that, compared to the first database node that establishes a connection with the client, the transaction coordinator service of the second database node does not and does not need to participate in the execution of the target transaction, but only provides the target data shard and the corresponding transaction cache set. However, in the event that the target transaction coordinator service of the first database node goes down, this specification can promptly clear the invalid transaction cache of the target transaction in the target data shard of the second database node.

[0146] In one embodiment, the second database node can, when determining that the first database node is down, clean up invalid transaction caches in the transaction cache set corresponding to the above-mentioned target data shard based on a failover mechanism (Failover mechanism). Among them, the failover mechanism involves real-time monitoring capabilities for the first database node, and specifically requires the second database node or a third-party monitoring platform to monitor the target transaction coordinator service status of the first database node in real time. This is usually achieved through heartbeat detection (periodically sending small data packets to confirm the availability of components) or other health check mechanisms, and this statement does not limit this.

[0147] Figure 7 The present invention is a flowchart of a method for executing transaction operations on a database provided by an exemplary embodiment.

[0148] like Figure 7 As shown, the database includes several nodes, each node stores data fragments, and the method includes:

[0149] Step S702: for a first operation included in the target transaction to be processed, determine a target data shard corresponding to the first operation.

[0150] When a user needs to perform a first operation on data in a database, the user may input a corresponding database statement into a client, and then the client constructs a corresponding request to send to a first database node in the database system, wherein the database statement corresponding to the first operation may be a DML statement.

[0151] For the first database node, after receiving the request from the client, it can determine the target transaction to be processed corresponding to the request, so as to execute the database statement in the request in the target transaction.

[0152] When the target transaction is determined and the first database node is about to execute the corresponding database statement in the target transaction, the transaction coordinator service of the first database node can determine the target data shard corresponding to the target transaction when executing the first operation, that is, the data shard in the database system where the data involved in each DML statement in the target transaction is located. Even if the target data shard includes the first data shard maintained in the first database node and / or the second data shard in the second database node different from the first database node in the database system, the transaction coordinator service can still determine the first data shard and / or the second data shard included in the target data shard.

[0153] The data shard where each data is located is determined by the sharding strategy of the database, which may include range sharding, hash sharding, random sharding, etc. Of course, no matter which sharding strategy is adopted, the first database node can choose to obtain the data shard to be operated by each database statement based on, for example, range matching, hash mapping, metadata table or directory service according to actual needs, and then determine all target data shards that the target transaction as a whole needs to operate.

[0154] Step S704: perform a first operation on the target data shard, and an operation result of the first operation is stored in a cache corresponding to the target data shard, and the operation result is marked with target transaction information.

[0155] After determining the target data shards that the target transaction needs to operate, the first database node can initiate data operation requests corresponding to the target transaction for each determined target data shard, wherein the target data shards can be located in other nodes such as the first database node and / or the second database node. At the same time, for the database node where the target data shard that receives the data operation request is located, the received data operation request can be processed through the shard management service corresponding to the target data shard.

[0156] Specifically, the data operation request may be initiated by the target transaction coordinator service corresponding to the target transaction running on the first database node, and the sending target may also be determined by the target data shard determined by the target transaction coordinator service in step S701.

[0157] When the shard management service successfully receives the data operation request, the first operation may be performed on the target data shard, and the operation result may be saved in the cache corresponding to the target data shard, and the operation result may be marked with the target transaction information of the target transaction. For ease of description, the determined cache is collectively referred to as the target transaction cache below. The target transaction information may include the transaction identifier of the target transaction, and the target transaction information may be added to the data operation request by the first database node.

[0158] The shard management service of this specification can also implement anomaly detection of the transaction cache set based on transaction information. The transaction cache set is maintained on each database node in the database system and corresponds one-to-one to each data shard. The above transaction cache set can record the transaction cache corresponding to each transaction related to the corresponding data shard.

[0159] In one embodiment, the above-mentioned transaction information may also include a historical data shard set, which means all data shards involved in the target transaction; after the first database node initiates a data operation request corresponding to the target transaction for the target data shard, the target data shard may be updated to the historical data shard set, and the shard management service may be triggered to perform the above-mentioned first operation. The data operation statement corresponding to the above-mentioned data operation request is a data write statement. In other words, the data operation statement corresponding to the first operation triggered by the above-mentioned data operation request is also a data write statement; wherein, when the transaction cache set corresponding to the above-mentioned target data shard does not contain the above-mentioned target transaction cache, and the historical data shard set contained in the above-mentioned transaction information includes the above-mentioned target data shard, the above-mentioned shard management service may determine that there is an abnormality in the transaction cache set corresponding to the target data shard.

[0160] The above historical data shard set is maintained by the first database node, and the target data shard is recorded in the historical data shard only after the target transaction coordinator service initiates a data operation request corresponding to the data write statement to the target data shard. In other words, for the shard management service that receives the data operation request for the first time, its corresponding transaction cache set must not have a transaction cache matching the above transaction information, that is, the target transaction cache, and the above historical data shard set contained in the above transaction information must not include the above target data shard; if an exception occurs in the shard management service or the transaction cache set, resulting in the loss of the transaction cache, then the above historical data shard set contained in the above transaction information plays a secondary confirmation effect, that is, when the transaction cache set corresponding to the above target data shard does not have the above target transaction cache, but the historical data shard set contained in the above transaction information includes the target data shard, the shard management service can determine the following situation: the first database node has initiated at least one data operation request to the shard management service to trigger the execution of the above first operation, but for some reason did not generate the above target transaction cache in the transaction cache set. Then, it is finally determined that the transaction cache set corresponding to the target data shard has an abnormality. In addition, the abnormality can be notified to the target transaction coordinator service of the first database node through the shard management service, and then sent to the client as prompt information.

[0161] In one embodiment, the data operation statement corresponding to the above-mentioned first operation may include a data write statement; first, it can be determined whether there is a cache corresponding to the above-mentioned target transaction in the cache corresponding to the above-mentioned target data shard, that is, the target transaction cache; if the judgment result is no, the above-mentioned cache corresponding to the above-mentioned target transaction can be created, and the above-mentioned data write statement can be executed based on the above-mentioned cache corresponding to the above-mentioned target transaction; if the judgment result is yes, the above-mentioned data write statement can be executed based on the above-mentioned cache corresponding to the above-mentioned target transaction.

[0162] In another embodiment, the data operation statement corresponding to the first operation may include a data write statement; when the target transaction cache does not exist in the transaction cache set corresponding to the target data shard, and the historical data shard set contained in the transaction information does not include the target data shard, it means that any data write statement has never been executed for the data of the target data shard in the target transaction, and therefore the corresponding shard management service may create a target transaction cache for the target transaction, and execute the data write statement based on the target transaction cache; correspondingly, when the target transaction cache exists in the transaction cache set corresponding to the target data shard, it means that at least one data write statement has been executed for the data of the target data shard in the target transaction, and therefore the data write statement may be executed directly based on the target transaction cache.

[0163] Specifically, the data write statement may include a data insert statement, a data update statement or a data delete statement. When the shard management service performs the first operation, the corresponding situations are as follows:

[0164] 1. In the case where the data writing statement corresponding to the first operation is the data insert statement, the to-be-inserted data corresponding to the data insert statement may be directly written into the target transaction cache.

[0165] 2. In the case where the data write statement corresponding to the first operation is the data update statement, it is firstly determined whether the target transaction cache contains the data to be updated corresponding to the data update statement. If so, the content of the data to be updated can be updated. If not, the data to be updated can be obtained from the persistent data corresponding to the target data shard, and the obtained data to be updated can be updated and written into the target transaction cache. Of course, if the data to be updated still does not exist in the persistent data, no additional operation can be performed on the target transaction cache, or a prompt message indicating the update failure can be fed back to the target transaction coordinator service.

[0166] 3. In the case where the data write statement corresponding to the above-mentioned first operation is the above-mentioned data deletion statement, it can be first determined whether there is data to be deleted corresponding to the above-mentioned data deletion statement in the above-mentioned target transaction cache. If so, the above-mentioned data to be deleted is marked as deleted; if not, the above-mentioned data to be deleted is obtained from the persistent data corresponding to the above-mentioned target data shard, and the obtained data to be deleted is marked as deleted and written into the above-mentioned target transaction cache. It can be understood by those skilled in the art that the data deletion operation of the database is often implemented by setting specific fields of the corresponding records. Therefore, the operation of marking the above-mentioned data to be deleted as deleted can be regarded as, for example, setting the value of the field representing the above-mentioned data to be deleted mark to 1 / 0, and this is not limited in this specification.

[0167] Step S706: For a second operation included in the target transaction, based on the operation result of the first operation marked with the target transaction information and stored in the cache, execute the second operation.

[0168] When the user needs to perform the second operation in the above-mentioned target transaction on the data in the database, the above-mentioned first database node can determine the data shard corresponding to the second operation, and based on the operation result of the above-mentioned first operation marked with the target transaction information, determine the above-mentioned target transaction cache from the transaction cache set of the determined data shard, and execute the second operation through the shard management service corresponding to the data shard, wherein the database statement corresponding to the above-mentioned second operation can be a DQL statement.

[0169] In one embodiment, the data operation statement corresponding to the above-mentioned second operation may include a data query statement; first, it can be determined whether there is a cache corresponding to the above-mentioned target transaction in the cache corresponding to the above-mentioned target data shard; if the judgment result is no, the above-mentioned data query statement is executed for the persistent data corresponding to the above-mentioned target data shard; if the judgment result is yes, the above-mentioned data query statement is executed based on the above-mentioned cache corresponding to the above-mentioned target transaction.

[0170] In another embodiment, the data operation statement corresponding to the second operation may include a data query statement; when the target transaction cache does not exist in the transaction cache set corresponding to the target data shard, and the historical data shard set contained in the transaction information does not include the target data shard, the data query statement may be executed for the persistent data corresponding to the target data shard; correspondingly, when the target transaction cache exists in the transaction cache set corresponding to the target data shard, the data query statement may be executed based on the target transaction cache.

[0171] Those skilled in the art can understand that the data operation statement corresponding to the second operation can include a data write statement; the second operation in this case can be regarded as the first operation, which will not be described in detail in this specification.

[0172] Figure 8 is a schematic structural diagram of a device provided by an exemplary embodiment. Figure 8 At the hardware level, the device includes a processor 802, an internal bus 804, a network interface 806, a memory 808, and a non-volatile memory 810, and may also include hardware required for other functions. One or more embodiments of this specification may be implemented based on software, such as the processor 802 reading the corresponding computer program from the non-volatile memory 810 into the memory 808 and then running it. Of course, in addition to the software implementation, one or more embodiments of this specification do not exclude other implementations, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0173] Please refer to Fig. 9 A data operation device based on transaction cache in a database system can be applied to Fig. 9 In the device shown, the technical solution of this specification is implemented. Each database node in the above database system maintains at least one data shard and a transaction cache set corresponding to each data shard, and the transaction cache set is used to record the transaction cache corresponding to each transaction related to the corresponding data shard; the above device is applied to the first database node in the above database system; the device may include:

[0174] A data shard determination unit 92 is used to determine, for a target transaction to be processed, a target data shard that needs to be operated on by the target transaction;

[0175] The data operation request initiating unit 94 is used to initiate a data operation request corresponding to the above-mentioned target transaction for the above-mentioned target data shard, so as to execute the above-mentioned target transaction based on the target transaction cache recorded in the transaction cache set corresponding to the above-mentioned target data shard, and the above-mentioned target transaction cache is a transaction cache corresponding to the above-mentioned target transaction.

[0176] Optionally, the target data shards include:

[0177] a first data shard maintained in the first database node; and / or,

[0178] A second data shard is maintained in a second database node in the database system that is different from the first database node.

[0179] Optionally, the data operation request initiating unit 94 is specifically configured to:

[0180] Initiate the data operation request to the shard management service corresponding to the first data shard through the target transaction coordinator service corresponding to the target transaction running on the first database node; and / or,

[0181] The data operation request is initiated to the shard management service corresponding to the second data shard through the target transaction coordinator service corresponding to the target transaction running on the first database node.

[0182] Optionally, the above device further includes:

[0183] A transaction information recording unit, used to record the transaction information of the target transaction, wherein the transaction information includes the transaction identifier of the target transaction;

[0184] A transaction information adding unit is used to add the above-mentioned transaction information to the above-mentioned data operation request, so as to use the transaction cache matching the above-mentioned transaction identifier in the transaction cache set corresponding to the above-mentioned target data shard as the above-mentioned target transaction cache, and execute the above-mentioned target transaction based on the above-mentioned target transaction cache.

[0185] Optionally, the transaction information further includes a historical data shard set; and the device further includes:

[0186] A historical data shard set updating unit, after initiating a data operation request corresponding to the target transaction for the target data shard, updates the target data shard to the historical data shard set, wherein the data operation statement corresponding to the data operation request is a data write statement;

[0187] Among them, when the transaction cache set corresponding to the target data shard does not have the target transaction cache, and the historical data shard set contained in the transaction information includes the target data shard, the transaction cache set corresponding to the target data shard is confirmed to be abnormal.

[0188] Optionally, the data operation statement corresponding to the target transaction includes a data write statement;

[0189] The data operation request initiating unit 94 is specifically used for:

[0190] If the target transaction cache does not exist in the transaction cache set corresponding to the target data shard, and the historical data shard set contained in the transaction information does not include the target data shard, create the target transaction cache, and execute the data write statement based on the target transaction cache;

[0191] In the case that there is a target transaction cache matching the transaction information in the transaction cache set corresponding to the target data shard, the data write statement is executed based on the target transaction cache.

[0192] Optionally, the data write statement includes a data insert statement, a data update statement or a data delete statement; the data operation request initiating unit 94 is specifically used to:

[0193] In the case where the data writing statement is the data inserting statement, writing the to-be-inserted data corresponding to the data inserting statement into the target transaction cache;

[0194] In the case where the data write statement is the data update statement, determine whether the target transaction cache has the data to be updated corresponding to the data update statement, and if so, update the content of the data to be updated; if not, obtain the data to be updated from the persistent data corresponding to the target data shard, update the obtained data to be updated, and write it into the target transaction cache;

[0195] In the case where the above-mentioned data write statement is the above-mentioned data delete statement, determine whether there is to-be-deleted data corresponding to the above-mentioned data delete statement in the above-mentioned target transaction cache. If so, mark the above-mentioned data to be deleted as deleted; if not, obtain the above-mentioned data to be deleted from the persistent data corresponding to the above-mentioned target data shard, mark the obtained data to be deleted as deleted and write it into the above-mentioned target transaction cache.

[0196] Optionally, the data operation statement corresponding to the target transaction includes a data query statement;

[0197] The data operation request initiating unit 94 is specifically used for:

[0198] If the target transaction cache does not exist in the transaction cache set corresponding to the target data shard, and the historical data shard set contained in the transaction information does not include the target data shard, execute the data query statement for the persistent data corresponding to the target data shard;

[0199] In the case where there is a target transaction cache matching the transaction information in the transaction cache set corresponding to the target data shard, the data query statement is executed based on the target transaction cache.

[0200] Optionally, the data operation request initiating unit 94 is specifically configured to:

[0201] Determine whether the target transaction cache contains the data to be queried corresponding to the data query statement. If so, obtain the data to be queried from the target transaction cache. If not, obtain the data to be queried from the persistent data corresponding to the target data shard.

[0202] Optionally, the above device further comprises:

[0203] A second transaction information recording unit is used to record the transaction information of the target transaction, wherein the transaction information also includes a set of historical data shards involved in the target transaction;

[0204] The commit request or rollback request unit is used to initiate a commit request or rollback request for the target transaction for each data shard in the historical data shard set when receiving a commit or rollback instruction for the target transaction.

[0205] Please refer to Fig.10 Another data operation device based on transaction cache in a database system can be applied to Figure 8 In the device shown, the technical solution of this specification is implemented. Each database node in the above database system maintains at least one data shard and a transaction cache set corresponding to each data shard, and the transaction cache set is used to record the transaction cache corresponding to each transaction related to the corresponding data shard; the above method is applied to the second database node in the above database system; the data operation device may include:

[0206] A data operation request acquisition unit 102 is used to acquire a data operation request initiated by a first database node different from the second database node in the database system for a target data shard in the second database node, where the target data shard is required to be operated by a target transaction;

[0207] The first target transaction execution unit 104 is configured to execute the target transaction based on the transaction cache set corresponding to the target data shard.

[0208] Optionally, the data operation request obtaining unit 102 is specifically configured to:

[0209] The data operation request initiated by the shard management service corresponding to the target data shard is obtained, and the data operation request is generated by the target transaction coordinator service corresponding to the target transaction running on the first database node.

[0210] Optionally, the above device is also used for:

[0211] The transaction cache cleaning unit is used to clean up invalid transaction caches in the transaction cache set corresponding to the target data shard based on the failover mechanism when it is determined that the first database node is down.

[0212] Please refer to Fig.11 The device for performing transaction operations on a database can be applied to Figure 8 The device shown in the figure is used to implement the technical solution of this specification. The database includes several nodes, each of which stores data slices. The device includes:

[0213] The data shard determination unit 112 is configured to determine, for a first operation included in a target transaction to be processed, a target data shard corresponding to the first operation.

[0214] The first operation execution unit 114 is used to execute a first operation on the target data shard, and the operation result of the first operation is stored in the cache corresponding to the target data shard, and the operation result is marked with target transaction information.

[0215] The second operation execution unit 116 is configured to execute the second operation included in the target transaction based on the operation result of the first operation marked with the target transaction information and stored in the cache.

[0216] Optionally, the device further comprises:

[0217] A first cache determination unit, configured to determine whether there is a cache corresponding to the target transaction in the cache corresponding to the target data shard;

[0218] If the judgment result is no, creating the cache corresponding to the target transaction, and executing the data write statement based on the cache corresponding to the target transaction;

[0219] If the judgment result is yes, the data write statement is executed based on the cache corresponding to the target transaction.

[0220] Optionally, the data write statement includes a data insert statement, a data update statement or a data delete statement; the first cache judgment unit is specifically used to:

[0221] In a case where the data write statement is the data insert statement, writing the to-be-inserted data corresponding to the data insert statement into the cache corresponding to the target transaction;

[0222] In the case where the data write statement is the data update statement, determine whether the data to be updated corresponding to the data update statement exists in the cache corresponding to the target transaction, and if so, update the content of the data to be updated; if not, obtain the data to be updated from the persistent data corresponding to the target data shard, update the obtained data to be updated, and write it into the cache corresponding to the target transaction;

[0223] In the case where the data write statement is the data delete statement, determine whether the data to be deleted corresponding to the data delete statement exists in the cache corresponding to the target transaction. If so, mark the data to be deleted as deleted; if not, obtain the data to be deleted from the persistent data corresponding to the target data shard, mark the obtained data to be deleted as deleted and write it into the cache corresponding to the target transaction.

[0224] Optionally, the data operation statement corresponding to the second operation is a data query statement; and the device further includes:

[0225] A second cache determination unit, configured to determine whether there is a cache corresponding to the target transaction in the cache corresponding to the target data shard;

[0226] If the judgment result is no, executing the data query statement for the persistent data corresponding to the target data shard;

[0227] If the judgment result is yes, the data query statement is executed based on the cache corresponding to the target transaction.

[0228] Optionally, the second cache determination unit is specifically configured to:

[0229] Determine whether the data to be queried corresponding to the data query statement exists in the cache corresponding to the target transaction. If so, obtain the data to be queried from the target transaction cache. If not, obtain the data to be queried from the persistent data corresponding to the target data shard.

[0230] Based on the same concept as the above method, this specification also provides an electronic device, including: a processor; a memory for storing processor executable instructions; wherein the processor implements the steps of the method described in any of the above embodiments by running the executable instructions.

[0231] Based on the same concept as the above method, this specification also provides a computer-readable storage medium on which computer instructions are stored. When the instructions are executed by a processor, the steps of the method described in any of the above embodiments are implemented.

[0232] Based on the same concept as the above method, this specification also provides a computer program product, including a computer program / instruction, which implements the steps of the method described in any of the above embodiments when executed by a processor.

[0233] The above description is only a preferred embodiment of this specification and is not intended to limit this specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of this specification should be included in the scope of protection of this specification.

Claims

1. A data operation method based on transaction cache in a database system, wherein each database node in the database system maintains at least one data shard and a transaction cache set corresponding to each data shard, wherein the transaction cache set is used to record the transaction caches corresponding to each transaction related to the corresponding data shard; The method is applied to a first database node in the database system; the method comprises: For the target transaction to be processed, determine the target data shard that needs to be operated by the target transaction; A data operation request corresponding to the target transaction is initiated for the target data shard, so as to execute the target transaction based on a target transaction cache recorded in a transaction cache set corresponding to the target data shard, wherein the target transaction cache is a transaction cache corresponding to the target transaction.

2. According to the method of claim 1, the target data sharding comprises: A first data shard maintained in the first database node; and / or, A second data shard is maintained in a second database node in the database system that is different from the first database node.

3. According to the method of claim 2, initiating a data operation request corresponding to the target transaction for the target data shard comprises: Initiate the data operation request to the shard management service corresponding to the first data shard through the target transaction coordinator service corresponding to the target transaction running on the first database node; and / or, The data operation request is initiated to the shard management service corresponding to the second data shard through the target transaction coordinator service corresponding to the target transaction running on the first database node.

4. The method according to claim 1, further comprising: Recording transaction information of the target transaction, wherein the transaction information includes a transaction identifier of the target transaction; The transaction information is added to the data operation request, so as to use the transaction cache that matches the transaction identifier in the transaction cache set corresponding to the target data shard as the target transaction cache, and execute the target transaction based on the target transaction cache.

5. The method according to claim 4, The transaction information also includes a historical data shard set; the method further includes: After initiating a data operation request corresponding to the target transaction for the target data shard, updating the target data shard to the historical data shard set, wherein the data operation statement corresponding to the data operation request is a data write statement; Among them, when the target transaction cache does not exist in the transaction cache set corresponding to the target data shard, and the historical data shard set contained in the transaction information includes the target data shard, the transaction cache set corresponding to the target data shard is confirmed to be abnormal.

6. The method according to claim 5, wherein the data operation statement corresponding to the target transaction comprises a data write statement; The executing the target transaction based on the target transaction cache recorded in the transaction cache set corresponding to the target data shard includes: If the target transaction cache does not exist in the transaction cache set corresponding to the target data shard, and the historical data shard set contained in the transaction information does not include the target data shard, create the target transaction cache, and execute the data write statement based on the target transaction cache; In a case where the target transaction cache exists in the transaction cache set corresponding to the target data shard, the data write statement is executed based on the target transaction cache.

7. The method according to claim 6, wherein the data write statement comprises a data insert statement, a data update statement or a data delete statement; and executing the data write statement based on the target transaction cache comprises: In the case where the data writing statement is the data insert statement, writing the to-be-inserted data corresponding to the data insert statement into the target transaction cache; In the case where the data write statement is the data update statement, determining whether the target transaction cache has the data to be updated corresponding to the data update statement, and if so, updating the content of the data to be updated; if not, obtaining the data to be updated from the persistent data corresponding to the target data shard, updating the obtained data to be updated and writing it into the target transaction cache; In the case where the data write statement is the data delete statement, determine whether there is data to be deleted corresponding to the data delete statement in the target transaction cache. If so, mark the data to be deleted as deleted; if not, obtain the data to be deleted from the persistent data corresponding to the target data shard, mark the obtained data to be deleted as deleted and write it into the target transaction cache.

8. According to the method of claim 5, the data operation statement corresponding to the target transaction includes a data query statement; The executing the target transaction based on the target transaction cache recorded in the transaction cache set corresponding to the target data shard includes: If the target transaction cache does not exist in the transaction cache set corresponding to the target data shard, and the historical data shard set contained in the transaction information does not include the target data shard, executing the data query statement for the persistent data corresponding to the target data shard; In a case where the target transaction cache exists in the transaction cache set corresponding to the target data shard, the data query statement is executed based on the target transaction cache.

9. The method according to claim 8, wherein executing the data query statement based on the target transaction cache comprises: Determine whether the target transaction cache contains the to-be-queried data corresponding to the data query statement. If so, obtain the to-be-queried data from the target transaction cache. If not, obtain the to-be-queried data from the persistent data corresponding to the target data shard.

10. The method according to claim 1, further comprising: Recording transaction information of the target transaction, wherein the transaction information also includes a set of historical data shards involved in the target transaction; When a commit or rollback instruction for the target transaction is received, a commit request or a rollback request for the target transaction is initiated for each data shard in the historical data shard set.

11. A data operation method based on transaction cache in a database system, wherein each database node in the database system maintains at least one data shard and a transaction cache set corresponding to each data shard, and the transaction cache set is used to record the transaction cache corresponding to each transaction related to the corresponding data shard; The method is applied to a second database node in the database system; the method comprises: Acquire a data operation request initiated by a first database node in the database system that is different from the second database node and targets a target data shard in the second database node, where the target data shard is required to be operated by a target transaction to be processed; The target transaction is executed based on the target transaction cache recorded in the transaction cache set corresponding to the target data shard, where the target transaction cache is the transaction cache corresponding to the target transaction.

12. The method according to claim 11, wherein obtaining a data operation request initiated by a first database node different from the second database node in the database system for a target data shard in the second database node comprises: The data operation request initiated by the shard management service corresponding to the target data shard is obtained, where the data operation request is generated by a target transaction coordinator service corresponding to the target transaction running on the first database node.

13. The method according to claim 11, further comprising: When it is determined that the first database node is down, invalid transaction caches in the transaction cache set corresponding to the target data shard are cleared based on a failover mechanism.

14. A method for performing transaction operations on a database, the database comprising a plurality of nodes, each node storing a data shard, the method comprising: For a first operation included in the target transaction to be processed, determining a target data shard corresponding to the first operation; Performing a first operation on the target data shard, where an operation result of the first operation is stored in a cache corresponding to the target data shard, and the operation result is marked with target transaction information; For a second operation included in the target transaction, the second operation is executed based on an operation result of the first operation marked with the target transaction information and stored in the cache.

15. The method according to claim 14, wherein the data operation statement corresponding to the first operation comprises a data write statement, and the method further comprises: Determine whether there is a cache corresponding to the target transaction in the cache corresponding to the target data shard; If the judgment result is no, creating the cache corresponding to the target transaction, and executing the data write statement based on the cache corresponding to the target transaction; If the judgment result is yes, the data write statement is executed based on the cache corresponding to the target transaction.

16. The method according to claim 15, wherein the data write statement comprises a data insert statement, a data update statement or a data delete statement; and executing the data write statement based on the cache corresponding to the target transaction comprises: In a case where the data write statement is the data insert statement, writing the to-be-inserted data corresponding to the data insert statement into the cache corresponding to the target transaction; In the case where the data write statement is the data update statement, determine whether the data to be updated corresponding to the data update statement exists in the cache corresponding to the target transaction, and if so, update the content of the data to be updated; if not, obtain the data to be updated from the persistent data corresponding to the target data shard, update the obtained data to be updated, and write it into the cache corresponding to the target transaction; In the case where the data write statement is the data delete statement, determine whether the data to be deleted corresponding to the data delete statement exists in the cache corresponding to the target transaction. If so, mark the data to be deleted as deleted; if not, obtain the data to be deleted from the persistent data corresponding to the target data shard, mark the obtained data to be deleted as deleted and write it into the cache corresponding to the target transaction.

17. The method according to claim 14, wherein the data operation statement corresponding to the second operation is a data query statement; the method further comprises: Determine whether there is a cache corresponding to the target transaction in the cache corresponding to the target data shard; If the judgment result is no, executing the data query statement for the persistent data corresponding to the target data shard; If the judgment result is yes, the data query statement is executed based on the cache corresponding to the target transaction.

18. The method according to claim 17, wherein executing the data query statement based on the cache corresponding to the target transaction comprises: Determine whether the data to be queried corresponding to the data query statement exists in the cache corresponding to the target transaction. If so, obtain the data to be queried from the target transaction cache. If not, obtain the data to be queried from the persistent data corresponding to the target data shard.

19. A database system, wherein each database node in the database system maintains at least one data shard and a transaction cache set corresponding to each data shard, wherein the transaction cache set is used to record the transaction cache corresponding to each transaction related to the corresponding data shard; wherein: The first database node is used to determine, for a target transaction to be processed, a target data shard that needs to be operated on by the target transaction, and initiate a data operation request corresponding to the target transaction for the target data shard in the first database node and / or the second database node; The second database node is used to execute the target transaction based on the target transaction cache recorded in the transaction cache set corresponding to the target data shard, and the target transaction cache is the transaction cache corresponding to the target transaction.

20. A computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the steps of the method according to any one of claims 1 to 18.

21. A computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 18.

Images