Transaction processing method and database based on distributed embedded storage engine

Abstract

The present application provides a transaction processing method, database, server and computer-readable storage medium based on a distributed embedded storage engine. In the method, a distributed transaction processing method is implemented based on a distributed embedded storage engine. Through the engine, when a transaction is executed, in most cases, the required data can be directly hit locally, which is much higher than the performance of traditional distributed database remote access; and the transaction can be completed independently without a transaction coordination node, which greatly reduces the cost of transaction execution compared to traditional distributed transactions.

Description

Technical Field

[0001] The present application relates to the technical field of distributed systems, and in particular to a transaction processing method, database, system and computer-readable storage medium based on a distributed embedded storage engine. Background Art

[0002] In order to achieve high concurrency, high availability and high reliability of data interaction, the Internet generally adopts stateless server design, which is of great reference significance for the design of game server architecture. However, the business logic of games usually has the characteristics of high frequency and strong coupling. If the traditional Internet method is adopted to achieve high concurrency, high availability and persistence, the stateless server architecture design is adopted. Whether it is MySQL cluster, TiDB distributed database, or Redis cache, it cannot meet the high-frequency transaction business processing at a low cost. Summary of the invention

[0003] In view of this, the embodiments of the present application provide a transaction processing method, database, system and computer-readable storage medium based on a distributed embedded storage engine to solve the technical defects existing in the prior art.

[0004] According to a first aspect of an embodiment of the present application, a transaction processing method based on a distributed embedded storage engine is provided, comprising:

[0005] A distributed transaction processing unit of a first service node starts a transaction according to the service logic, wherein the first service node is any node in the distributed service node;

[0006] The distributed transaction processing unit starts executing the transaction and obtains target data required for the transaction according to the storage engine;

[0007] When the operation sequence of the transaction first accesses a data item in the consistent storage client node on the first service node, a corresponding locking operation is performed on the data item;

[0008] Execute the operation sequence, commit the changes and release the lock after the transaction is completed;

[0009] Destroy the transaction.

[0010] According to a second aspect of an embodiment of the present application, a database based on a distributed embedded storage engine is provided, comprising:

[0011] Distributed transaction processing unit;

[0012] Consistent storage client nodes;

[0013] The distributed transaction processing unit and the consistent storage client node are used to execute a transaction processing method based on a distributed embedded storage engine.

[0014] According to a third aspect of an embodiment of the present application, a distributed transaction processing system based on a distributed embedded storage engine is provided, including:

[0015] Multiple distributed business nodes and multiple consistent storage servers;

[0016] The multiple distributed service nodes and the consistent storage server are used to execute a transaction processing method based on a distributed embedded storage engine.

[0017] According to a fourth aspect of an embodiment of the present application, a distributed service node is provided, including:

[0018] Business logic unit, used to execute business functions;

[0019] A data status recording unit, used to store the occupancy status of the business data on the current business node;

[0020] Consistent storage client node, used to store business data;

[0021] A distributed transaction processing unit, which is used to execute transactions according to business logic and modify target data;

[0022] A data request unit, configured to initiate a target data occupation request to a consistent storage server when the target data does not exist in the consistent storage client node or the occupation state of the target data is invalid;

[0023] The data receiving unit is used to receive the target data and the occupancy status information of the target data returned by the consistency storage server; the data receiving unit also sends the target data to the consistency storage client node, and sends the occupancy status information of the received target data to the data status recording unit.

[0024] According to a fifth aspect of an embodiment of the present application, a computer-readable storage medium is provided, wherein the storage medium stores a computer program, and the computer program is executed by a processor to implement a transaction processing method based on a distributed embedded storage engine.

[0025] In the embodiment of the present application, in the distributed system, each business node is embedded with an embedded database. When processing business logic, the business node can directly obtain data from the embedded database except for the first access that requires obtaining data from the consistency storage server, thereby avoiding IO overhead and data delay between servers and greatly improving performance. At the same time, under the data state management of the consistency storage server, when data competition and modification occur between multiple business nodes, there will be no data consistency issues between business nodes. Through the above technical means, a distributed transaction processing method based on a distributed embedded storage engine is realized, which maintains the execution efficiency of traditional game servers while achieving the characteristics of high concurrency, high availability, and reliable data persistence similar to stateless servers of the traditional Internet. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is a structural block diagram of a traditional distributed system in the prior art;

[0027] Figure 2 It is a schematic diagram of the architecture of a distributed embedded storage engine that achieves data consistency provided by an embodiment of the present application;

[0028] Figure 3 It is a schematic diagram of the structure of a database based on a distributed embedded storage engine provided in an embodiment of the present application;

[0029] Figure 4 It is a schematic diagram of the structure of a service node based on a distributed embedded storage engine provided in an embodiment of the present application; DETAILED DESCRIPTION

[0030] Many specific details are described in the following description to facilitate a full understanding of the present application. However, the present application can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of the present application, so the present application is not limited by the specific implementation disclosed below.

[0031] The terms used in one or more embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of the present application. The singular forms of "a", "said" and "the" used in one or more embodiments of the present application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of the present application refers to and includes any or all possible combinations of one or more associated listed items.

[0032] It should be understood that, although the terms first, second, etc. may be used to describe various information in one or more embodiments of the present application, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of the present application, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "in response to determination".

[0033] In the prior art, in order to realize high concurrency and high availability systems, the Internet generally adopts stateless server design, such as Figure 1 As shown in the figure, in this traditional distributed system, there are multiple stateless business servers and distributed databases. Although the distributed database supports distributed transactions, it only supports remote access. For business scenarios that require distributed transactions but do not have much data competition, the traditional distributed database has too high IO overhead and latency as well as transaction development costs. In another way, if an embedded traditional database is used inside a business node, it does not support distributed storage or distributed transactions.

[0034] In the present application, in order to solve the above problems, a transaction processing method, database, system and computer-readable storage medium based on a distributed embedded storage engine are provided, which will be described in detail one by one in the following embodiments.

[0035] Figure 2 The structure of the distributed embedded storage engine that realizes data consistency in this application is shown. Figure 2 As shown, the distributed embedded storage engine is implemented based on multiple business nodes and multiple consistent storage servers.

[0036] When the business logic of the business node needs to access data, a data access request is initiated to the embedded database of the business node. Specifically, the embedded database also includes a consistent storage client node.

[0037] In a specific implementation, if the target data exists in the embedded database, the target data is directly returned to the business logic.

[0038] In this implementation, the database is embedded in the business node, and the business logic can directly obtain data from the embedded database without accessing other remote data servers, thus avoiding IO overhead, greatly improving performance, and eliminating additional delays in data queries.

[0039] In another specific implementation, if the target data does not exist in the embedded database, the first service node initiates a target data occupation request to the consistency storage server, and the first service node is any service node in the distributed system. For example, when the service logic is first run, the embedded database of the first service node may not yet have the target data.

[0040] Furthermore, the data occupation request includes a data sharing right request or a data exclusivity right request.

[0041] The consistent storage server stores a data status record S, which stores the occupation status information of the data. The occupation status information includes the occupation right type of the data and which service nodes currently occupy the data.

[0042] Furthermore, when the target data is not occupied by any business node, the consistency storage server updates the data status record S: the target data is occupied by the first business node; and returns the target data and the data occupancy right information obtained according to the target data occupancy request - data sharing rights or data exclusivity rights, to the first business node.

[0043] In this implementation, the target data may exist on the consistent storage server, or may be obtained by the consistent storage server by requesting other servers, which is not particularly limited here.

[0044] After the first business node receives the target data and the status information of the current target data returned by the consistency storage server, it stores the target data in the embedded database and updates its own data status record C. That is, the business node also has a data status record. The data status record C of the business node saves the data status of the target data obtained by the current business node.

[0045] In another specific implementation, when the target data occupation request initiated by the first business node is a data sharing rights request, and the target data is currently occupied by other business nodes, the consistency storage server updates the data status record S after receiving the request: adds the first business node to the sharing rights list of the target data; and returns the target data and the occupation rights information of the target data - data sharing rights to the first business node.

[0046] After receiving the target data and the status information of the current target data returned by the consistency storage server, the first service node stores the target data in the embedded database, updates its own data status record C, and writes the occupation right information of the target data, namely, the data sharing right, therein.

[0047] In another specific implementation, when the target data occupation request initiated by the first business node is a data sharing right request, and the target data is currently occupied by other business nodes with data exclusive rights, the consistency storage server clears the data exclusive rights record in the data status record S and sends a notification to other business nodes to reduce the possession of the target data to data sharing rights, and then adds the first business node and other business nodes to the sharing rights list of the target data.

[0048] After receiving the notification, other business nodes update their own data status record C and reduce the occupation right information of the target data from exclusive right to shared right.

[0049] After receiving the target data and the status information of the current target data returned by the consistency storage server, the first service node stores the target data in the embedded database, updates its own data status record C, and writes the occupation right information of the target data, namely, the data sharing right, therein.

[0050] In another specific implementation, when the target data occupation request initiated by the first business node is a data exclusivity request, at this time, regardless of whether the target data is occupied by other business nodes with data sharing rights or data exclusivity, the consistency storage server will send a notification to other business nodes to mark the target data as invalid, and update the data status record S: the target data is only occupied by the first business node with data exclusivity.

[0051] After receiving the notification, other service nodes update their own data status record C and set the status of the target data to invalid.

[0052] After receiving the target data and the status information of the current target data returned by the consistency storage server, the first service node stores the target data in the embedded database, updates its own data status record C, and writes the occupancy right information of the target data, namely, data exclusivity, therein.

[0053] Furthermore, when the first service node modifies the target data, it requests the consistency storage server to obtain the exclusive right of the target data.

[0054] In a possible implementation, when the first service node requests the target data of the embedded database, it will find that the target data exists, but the data has been marked as invalid. This happens because other service nodes in the distributed system need to modify the target data and have requested the consistency storage server to obtain data exclusivity.

[0055] In this case, the first business node sends a target data occupation request to the consistency storage server. After receiving the request, the consistency storage server will query the data status record S to find out that the target data is in a state of being exclusively occupied by a certain business node. The consistency storage server then pulls the valid target data from the business node with exclusive rights to the target data and returns it to the first business node. The target data occupation request sent by the first business node can be either a data sharing right request or an exclusive right request. For the specific process, please refer to the aforementioned embodiment, which will not be repeated here.

[0056] Those skilled in the art should know that when a consistent storage server pulls target data from a business node that has exclusive rights to the target data, if the target data is in a write lock state, it will wait for the write lock to be released before pulling the data, which will not be elaborated here.

[0057] In another specific implementation, when the first service node executes a distributed transaction and needs to modify the target data, it is executed based on the working mode of the above-mentioned distributed embedded storage engine.

[0058] The first service node includes a distributed transaction processing unit. When the first service node executes a distributed transaction, the distributed transaction processing unit first creates a transaction and then uses the distributed embedded storage engine to request the consistent storage server to obtain the sharing right and / or exclusive right of the target data.

[0059] For example, when executing transaction A, it is necessary to access target data C and modify target data D. If target data C and D exist in the embedded database, the transaction is directly executed, and the data exclusive right is obtained for the target data D that needs to be modified.

[0060] If the target data C and D do not exist in the embedded database or exist in the embedded database but are in an invalid state, the target data is requested to the consistency storage server based on the working mode of the above-mentioned distributed embedded storage engine, which will not be repeated here; wherein the first business node obtains the sharing right of the target data C and the exclusive right of the target data D.

[0061] After the transaction starts executing, when the transaction's operation sequence first accesses a data item in the database table, the corresponding locking operation is performed on the data item. For example, in transaction A, a read lock is added to the target data C, and a write lock is added to the target data D. Then, an object that supports the transaction operation is returned. For example, when the transaction first accesses the target data C and D, the table name of the table where the target data C and D are located is returned.

[0062] During the transaction execution process, the operations on the transaction objects are recorded and a transaction log is generated. For example, when the first service node executes transaction A, the modification process of the target data D is recorded and written into the WAL log, and the log is written into the log queue for persistence.

[0063] After the transaction is executed, the changes are committed to the database, the lock on the data item is released, and the transaction is then destroyed.

[0064] In the above embodiment, the data occupancy status of each business node is recorded and managed through a distributed embedded storage engine. For data that is frequently accessed and has no competitive modification requirements in most cases, the data access performance is greatly improved while also meeting the data consistency requirements. Moreover, through this engine, when a transaction is executed, in most cases, the required data can be directly hit locally, which is much higher than the performance of remote access of traditional distributed databases; because the ownership of the data accessed by the transaction layer is local, distributed transactions can be completed without coordination with other nodes, that is, transactions can be completed independently without transaction coordination nodes, which greatly reduces the cost of transaction execution compared to traditional distributed transactions; in a few cases, when there is no hit data locally, distributed transaction processing can also be achieved through the data storage layer of this engine.

[0065] For data changes caused by transaction execution, since there is no need for traditional transaction coordination points, the transaction execution node itself can submit modification logs, thus achieving real-time persistence of data at a low cost.

[0066] Through the above-mentioned technical means, the distributed transaction processing method based on this engine maintains the execution efficiency of traditional game servers while achieving the characteristics of high concurrency, high availability, and reliable data persistence similar to the stateless servers of the traditional Internet.

[0067] According to the above description of the embodiment based on the distributed embedded storage engine, the present application also provides a database based on the distributed embedded storage engine, such as Figure 3 As shown, the database includes:

[0068] Distributed transaction processing unit;

[0069] Consistent storage client nodes;

[0070] The distributed transaction processing unit and the consistent storage client node are used to execute the above distributed transaction execution method. Those skilled in the art should know that the database also includes other conventional functions of the database in the prior art, which will not be repeated here.

[0071] According to the above description of the embodiment based on the distributed embedded storage engine, the present application also provides a distributed transaction processing system based on the distributed embedded storage engine, such as Figure 2 As shown, the system includes:

[0072] Multiple distributed business nodes and multiple consistent storage servers;

[0073] The multiple distributed service nodes and the consistency storage server are used to execute the above-mentioned distributed transaction execution method.

[0074] According to the above description based on the distributed embedded storage engine embodiment, the present application also provides a distributed service node, such as Figure 4 As shown, each business node includes:

[0075] Business logic unit, used to execute business functions;

[0076] A data status recording unit, used to store the occupancy status of the business data on the current business node;

[0077] Consistent storage client node, used to store business data;

[0078] A distributed transaction processing unit, which is used to execute transactions according to business logic and modify target data;

[0079] A logging unit is used to incrementally record operations that modify data;

[0080] A data request unit, configured to initiate a target data occupation request to a consistent storage server when the target data does not exist in the consistent storage client node or the occupation state of the target data is invalid;

[0081] The data receiving unit is used to receive the target data and the occupancy status information of the target data returned by the consistency storage server; the data receiving unit also sends the target data to the consistency storage client node, and sends the occupancy status information of the received target data to the data status recording unit.

[0082] According to the above description of the embodiment based on the distributed embedded storage engine, the present application also provides a computer-readable storage medium, which stores computer instructions, which, when executed by a processor, implement the method steps for executing transactions based on the distributed embedded storage engine as described above.

[0083] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the above-mentioned technical scheme based on the distributed embedded storage engine belong to the same concept, and the details not described in detail in the technical scheme of the storage medium can be referred to the description of the above-mentioned technical scheme based on the distributed embedded storage engine.

[0084] The above describes specific embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0085] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0086] It should be noted that, for the convenience of description, the aforementioned method embodiments are all described as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described action sequence, because according to the present application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0087] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0088] The preferred embodiments of the present application disclosed above are only used to help explain the present application. The optional embodiments do not describe all the details in detail, nor do they limit the invention to the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the present application. The present application selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present application, so that those skilled in the art can understand and use the present application well. The present application is only limited by the claims and their full scope and equivalents.

Claims

1. Transaction processing method based on distributed embedded storage engine, It is characterized in that include: A distributed transaction processing unit of a first service node starts a transaction according to the service logic, wherein the first service node is any node in the distributed service node; The distributed transaction processing unit starts to execute the transaction, and obtains the target data required by the transaction according to the storage engine, including: if the target data exists in the consistency storage client node of the first business node, directly returning the data; if the target data does not exist in the consistency storage client node of the first business node or the target data is in an invalid state, requesting the consistency storage server to obtain the target data; When the operation sequence of the transaction first accesses a data item in the consistent storage client node on the first service node, a corresponding locking operation is performed on the data item; Execute the operation sequence, commit the changes and release the lock when the transaction is completed; Destroy the transaction.

2. The method according to claim 1, wherein the target data required for the transaction is obtained according to the storage engine. include: For target data that needs to be modified in a transaction, the distributed transaction processing unit requests the consistency storage server to obtain exclusive rights to the target data; The consistency storage server manages the occupation status of business data.

3. The method according to claim 1, in, If the target data is in an invalid state, requesting the consistency storage server to obtain the target data further includes: According to the request, the consistency storage server obtains the target data from the service node having the exclusive right to the target data and returns the target data to the first service node.

4. The method according to claim 1, further comprising: include: During the transaction execution, an object supporting the transaction operation is returned, and a transaction log is generated according to the object; Write transaction logs to the log queue to achieve data persistence.

5. Database system based on distributed embedded storage engine, It is characterized in that The database system includes: Distributed transaction processing unit; Consistent storage client nodes; The distributed transaction processing unit and the consistent storage client node are used to execute the method according to any one of claims 1-4.

6. Distributed transaction processing system based on distributed embedded storage engine, It is characterized in that include: Multiple distributed business nodes and multiple consistent storage servers; The multiple distributed service nodes and the consistency storage server are used to execute the method described in any one of claims 1-4.

7. A distributed service node device, It is characterized in that Each distributed service node device includes: Business logic unit, used to execute business functions; A data status recording unit, used to store the occupancy status of the business data on the current business node; Consistent storage client node, used to store business data; A distributed transaction processing unit, which is used to execute transactions according to business logic and modify target data; A logging unit is used to incrementally record operations that modify data; A data request unit, configured to initiate a target data occupation request to a consistent storage server when the target data does not exist in the consistent storage client node or the occupation state of the target data is invalid; The data receiving unit is used to receive the target data and the occupancy status information of the target data returned by the consistency storage server; the data receiving unit also sends the target data to the consistency storage client node, and sends the occupancy status information of the received target data to the data status recording unit.

8. A distributed service node device according to claim 7, It is characterized in that The distributed service node device also includes: An embedded database, the embedded database comprising the distributed transaction processing unit and the consistent storage client node.

9. A computer-readable storage medium, wherein the storage medium stores a computer program, It is characterized in that When the computer program is executed by a processor, the steps of the method described in any one of claims 1 to 4 are implemented.

Images