Data processing method and apparatus

Abstract

Embodiments of the present application provide a data processing method and device. In the method, a plurality of database operations of a transaction are grouped, and the plurality of database operations in the group are executed concurrently, so as to improve the overall throughput and processing efficiency of the system. Furthermore, the present application performs conflict detection in advance, so as to divide the database operations that are likely to conflict into different groups, thereby reducing the conflict probability when executed concurrently, and further improving the processing efficiency of the system.

Description

Technical Field

[0001] This application relates to the field of data processing, and more particularly to a data processing method and apparatus. Background Technology

[0002] Database performance is a key indicator of a database product's competitiveness. In the strategic transformation towards a "data-centric" data center, databases store the state data of the data center, and their performance directly impacts the performance of data center equipment. In products such as data communication and industrial data services, due to resource and power consumption limitations, current software optimizations based on the von Neumann processor architecture can no longer meet the ever-increasing performance requirements of data services in resource-constrained environments. To overcome this bottleneck, the industry has proposed various hardware acceleration and heterogeneous computing methods. However, in databases, the isolation and atomicity requirements of the transaction model, which require data operations to be locked or conflict-checked at each stage or data set, limit the hardware's concurrency capabilities, making it impossible to simultaneously achieve high throughput performance during database processing. Summary of the Invention

[0003] This application provides a data processing method and apparatus. In this method, the apparatus reduces the probability of transaction failure and improves system throughput performance by pre-performing conflict filtering and executing database operations in concurrent mode.

[0004] Firstly, embodiments of this application provide a data processing method. The method includes: an apparatus acquiring a transaction, the transaction comprising N database operations, where N is a positive integer greater than 1. Next, the apparatus performs conflict detection on the N database operations to obtain a conflict detection result. The conflict detection result includes an indication that a first database operation and a second database operation are conflicting operations. The first and second database operations belong to the N database operations. Subsequently, based on the conflict detection result, the apparatus divides the N database operations into M groups, where each of the M groups includes multiple database operations, and the first and second database operations belong to different groups, and M is a positive integer greater than 1. The apparatus concurrently executes multiple database operations in each of the M groups. Thus, the apparatus (which can also be understood as a database) in this application embodiment, by pre-performing conflict filtering, can construct batch operations based on non-conflicting operations, reducing the probability of errors and failures, and benefiting the concurrency of the accelerator.

[0005] In one possible implementation, concurrent execution of multiple database operations within each of M groups includes: the device invokes multiple hardware accelerators to concurrently execute multiple database operations within a single group of the M groups; wherein a single hardware accelerator executes a single database operation. Thus, this embodiment of the application improves database processing efficiency and throughput by executing multiple hardware operations in parallel, while reducing CPU usage and achieving hardware-software collaborative work.

[0006] In one possible implementation, multiple hardware accelerators are invoked to concurrently execute multiple database operations within a single group of M groups. This includes: the device triggering X hardware accelerators based on X database operations included in a first group; the first group belongs to the M groups. The device instructs the X hardware accelerators to perform the X database operations, causing the X hardware accelerators to execute the X database operations in parallel. In this way, multiple accelerators can execute in parallel, and software can invoke the hardware accelerators, thereby achieving hardware-software collaboration.

[0007] In one possible implementation, instructing X database operations to X hardware accelerators includes: the device instructing the X hardware accelerators to provide index information corresponding to each of the X database operations, such that the X hardware accelerators execute the X database operations based on the index information corresponding to each of the X database operations. In this way, multiple accelerators can perform parallel operations based on the index information to perform database operations on the data indicated by the index information.

[0008] In one possible implementation, multiple hardware accelerators are invoked to concurrently execute multiple database operations within a single group of M groups. This includes reading the execution results of X database operations from shared memory; wherein the execution results of the X database operations are written after the multiple hardware accelerators have concurrently executed the X database operations. In this way, the CPU and hardware accelerators can transfer data through shared memory. The CPU can then read the corresponding execution results.

[0009] In one possible implementation, multiple hardware accelerators are invoked to concurrently execute multiple database operations within a single group of M groups. This includes: reading the execution results of X database operations from shared memory; triggering Y hardware accelerators based on Y database operations included in a second group; the second group belongs to the M groups; and instructing the Y hardware accelerators to perform the Y database operations in parallel. Thus, by cyclically executing multiple groups and having each group execute in parallel, this embodiment of the application can effectively improve the system's processing efficiency and throughput.

[0010] In one possible implementation, the results of X database operations are placed in a circular queue in shared memory. Thus, this embodiment of the application uses a circular queue so that both the hardware and the CPU can read the information in the circular queue. Furthermore, the information in the circular queue is contiguous and can be reused.

[0011] In one possible implementation, invoking multiple hardware accelerators to concurrently execute multiple database operations within a single group of M groups includes: sending a first indication signal to the memory processor via a first interface, the indication signal instructing the memory processor to trigger X hardware accelerators. Thus, this embodiment of the application, through a pre-defined interface, facilitates the rapid triggering and deployment of hardware accelerators.

[0012] In one possible implementation, before concurrently executing multiple database operations in each of the M groups, the process includes: placing the index information of the delete operations among the N database operations in a first queue; and placing the index information of the reverse operations of the insert operations among the N database operations in a second queue. Thus, in this embodiment, there is no need to mark deletions; delete operations can be maintained solely through queues, reducing the query time for deletion markers and further accelerating processing efficiency. Maintaining the reverse operations of insert operations through queues allows for transaction rollback based on the reverse operations in the event of a transaction rollback.

[0013] In one possible implementation, multiple database operations in each of the M groups are executed concurrently, including: before committing the transaction, executing the delete operations in the first queue in parallel according to the M groups. Thus, in this embodiment, there is no need to mark deletions; the delete operations can be maintained solely through the queue, reducing the query time for deletion marks and further accelerating processing efficiency.

[0014] In one possible implementation, the method further includes: if an operation conflict is detected during the concurrent execution of multiple database operations in each of the M groups, the reverse operation of the increment operation is executed based on the index information of the reverse operation of the increment operation in the second queue, thus completing the transaction rollback. In this way, the embodiments of this application decouple the version information of the transaction, separating the transaction information and better realizing the parallel execution of the hardware.

[0015] In one possible implementation, the method further includes: after a transaction rollback, executing N database operations serially. Thus, this embodiment combines optimistic and pessimistic transaction modes, allowing switching to serial mode to continue execution in the event of a conflict, thereby improving system processing efficiency.

[0016] In one possible implementation, conflict detection is performed on N database operations to obtain the conflict detection results. This includes: obtaining the hash value corresponding to each of the N database operations; detecting whether there are any equal hash values ​​among the N database operations; and determining that the hash values ​​of the first database operation and the second database operation are conflicting operations if the hash values ​​are found to be equal. In this way, by judging the hash values, potentially conflicting database operations can be initially filtered out, ensuring that conflicting operations are not executed together during parallel execution.

[0017] In one possible implementation, based on the conflict detection results, N database operations are divided into M groups, including: dividing the N database operations into M groups based on the conflict detection results and preset grouping rules; the preset grouping rules include at least one of the following: the hash values ​​of multiple database operations in a single group are not equal; the number of database operations in a single group is less than or equal to the number of hardware accelerators; the execution time of the database operations in a single group is within a specified range; the type of database operations in a single group is the same. In this way, operations that may conflict can be pre-determined using hash values, allowing conflicting operations to be grouped into different groups. The amount of parallelism depends on the number of accelerators, ensuring that the number of hardware devices is sufficient to support the amount of parallelism, thus achieving hardware-software collaboration.

[0018] Secondly, embodiments of this application provide a data processing apparatus. The apparatus includes: an acquisition module for acquiring transactions, each transaction comprising N database operations; N being a positive integer greater than 1; a conflict detection module for performing conflict detection on the N database operations to obtain a conflict detection result; wherein the conflict detection result indicates that a first database operation and a second database operation are conflicting operations; the first database operation and the second database operation belong to the N database operations; a grouping module for dividing the N database operations into M groups based on the conflict detection result; each of the M groups includes multiple database operations; wherein the first database operation and the second database operation belong to different groups; M being a positive integer greater than 1; and an execution module for concurrently executing multiple database operations in each of the M groups.

[0019] In one possible implementation, the execution module is specifically used to: invoke multiple hardware accelerators to concurrently execute multiple database operations in a single group of M groups; wherein a single hardware accelerator executes a single database operation.

[0020] In one possible implementation, the execution module is specifically used to: trigger X hardware accelerators based on X database operations included in the first group; the first group belongs to M groups; and instruct the X hardware accelerators to perform X database operations in parallel.

[0021] In one possible implementation, the execution module is specifically used to: read the execution results of X database operations from shared memory; wherein the execution results of the X database operations are written after multiple hardware accelerators concurrently execute X database operations.

[0022] In one possible implementation, the execution module is specifically configured to: after reading the execution results of X database operations from shared memory, trigger Y hardware accelerators based on the Y database operations included in the second group; the second group belongs to M groups; and instruct the Y hardware accelerators to perform the Y database operations in parallel.

[0023] In one possible implementation, the results of X database operations are placed in a circular queue in shared memory.

[0024] In one possible implementation, the execution module is specifically configured to: send a first indication signal to the memory processor via a first interface, the indication signal being used to instruct the memory processor to trigger X hardware accelerators.

[0025] In one possible implementation, the execution module is further configured to: place the index information of the delete operation among the N database operations into a first queue; and place the index information of the reverse operation of the add operation among the N database operations into a second queue.

[0026] In one possible implementation, the execution module is also used to: before committing the transaction, execute the deletion operations in the first queue in parallel according to the M groupings.

[0027] In one possible implementation, the execution module is further configured to: if an operation conflict is detected during the concurrent execution of multiple database operations in each of the M groups, execute the reverse operation of the insert operation based on the index information of the reverse operation of the insert operation in the second queue, and complete the transaction rollback.

[0028] In one possible implementation, the execution module is also used to: execute N database operations serially after a transaction rollback.

[0029] In one possible implementation, the conflict detection module is specifically used to: obtain the hash value corresponding to each of the N database operations; detect whether there are any equal values ​​among the hash values ​​corresponding to the N database operations; and determine that the first database operation and the second database operation are conflicting operations if the hash value of the first database operation is equal to the hash value of the second database operation.

[0030] In one possible implementation, the grouping module is specifically used to: divide N database operations into M groups based on the collision detection results and preset grouping rules; the preset grouping rules include at least one of the following: the hash values ​​of multiple database operations in a single group are not equal; the number of database operations contained in a single group is less than or equal to the number of hardware accelerators; the execution time of the database operations contained in a single group is within a specified range; the type of database operations contained in a single group is the same.

[0031] Thirdly, embodiments of this application provide an electronic device. The electronic device includes: one or more processors; a memory; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and when executed by the one or more processors, cause the electronic device to perform instructions of the method in the first aspect or any possible implementation thereof.

[0032] Fourthly, embodiments of this application provide a computer-readable medium for storing a computer program, the computer program including instructions for performing the method in the first aspect or any possible implementation of the first aspect.

[0033] Fifthly, embodiments of this application provide a computer program including instructions for performing the method in the first aspect or any possible implementation thereof.

[0034] In a sixth aspect, embodiments of this application provide a chip including one or more interface circuits and one or more processors; the interface circuits are used to receive signals from the memory of an electronic device and send signals to the processors, the signals including computer instructions stored in the memory; when the processor executes the computer instructions, it causes the electronic device to execute the instructions of the method in the first aspect or any possible implementation of the first aspect. Attached Figure Description

[0035] Figure 1 This is a schematic diagram of a database structure as an example.

[0036] Figure 2 A schematic diagram of a database structure provided in an embodiment of this application;

[0037] Figure 3This is an example of a collision detection schematic diagram;

[0038] Figure 4 This is an example of a grouping diagram;

[0039] Figure 5 The process of creating a database table is shown as an example.

[0040] Figure 6 This is a schematic diagram illustrating a data processing flow as an example.

[0041] Figure 7 This is a schematic diagram of a queue as an example.

[0042] Figure 8 This is an example of a hardware accelerator call diagram;

[0043] Figure 9 This is a schematic diagram illustrating the data reading process as an example.

[0044] Figure 10 This is a schematic diagram of the structure of a data processing device as an example.

[0045] Figure 11 This is a schematic diagram of the device structure as an example. Detailed Implementation

[0046] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0047] In this article, the term "and / or" is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone.

[0048] The terms "first" and "second," etc., used in the specification and claims of this application are used to distinguish different objects, not to describe a specific order of objects. For example, "first target object" and "second target object," etc., are used to distinguish different target objects, not to describe a specific order of target objects.

[0049] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0050] In the description of the embodiments in this application, unless otherwise stated, "multiple" means two or more. For example, multiple processing units means two or more processing units; multiple systems means two or more systems.

[0051] Before describing the solutions of the embodiments of this application, we will first briefly introduce some technical terms involved in this application:

[0052] Optimistic Transaction Pattern: The optimistic transaction pattern can also be called the optimistic concurrency control pattern or the optimistic transaction processing pattern, etc., and this application does not limit it. In the optimistic transaction pattern, for concurrently executed operations that may conflict, it is assumed that they will not actually conflict, and concurrent execution is allowed until a conflict actually occurs, at which point the conflict is resolved, such as by rolling back the transaction. It can be understood that transactions are executed in an optimistic manner, that is, concurrent execution is adopted first, and transaction rollback is performed only when a conflict is encountered.

[0053] Pessimistic Transaction Mode: The mode corresponding to the optimistic transaction mode is the pessimistic transaction mode. It can also be called the pessimistic transaction processing mode or the pessimistic concurrency control mode, etc., which are not limited in this application. In this mode, for concurrently executing potentially conflicting operations, it is assumed that a conflict will definitely occur. Parallel operations are executed serially by making transactions wait (using locks) or abort (using timestamp sorting). In other words, transaction execution uses locks to protect accessed resources, achieving mutual exclusion between reads and writes, and ensuring that a transaction can complete its process as quickly as possible after acquiring the lock.

[0054] Transaction: Also known as a database transaction, a transaction is a collection of operations that constitute a single logical unit of work. For example, a transaction can include multiple database operations.

[0055] Database operations include, but are not limited to, DDL (Data Definition Language) operations and DML (Data Manipulation Language) operations. DDL operations are used to modify the database structure, such as defining or changing the structure of tables. DML commands are used to manage data in the database, including operations such as INSERT, DELETE, UPDATE, and SELECT.

[0056] Figure 1 This is a schematic diagram illustrating an exemplary database structure. Please refer to... Figure 1 A database includes both client and server components. The client includes, but is not limited to, high-performance applications. The client provides a database operation interface where users can input database commands, such as SQL (Structured Query Language) statements, to trigger the database to execute corresponding transactions, i.e., instruct the database to perform specific database operations. As mentioned above, a transaction can correspond to a database set, and a database set can include multiple database operations.

[0057] Still refer to Figure 1 For example, the server side includes both software and hardware. The software side includes, but is not limited to, a query engine, an execution engine, and a storage engine. The hardware side includes, but is not limited to, devices (or modules, etc.) such as CPU and memory (not shown in the figure).

[0058] In existing technologies, databases typically employ an optimistic transaction model, where, for example, the CPU can be a multi-core CPU. The CPU can execute multiple database operations across multiple cores. As mentioned above, the optimistic transaction model rolls back after a conflict occurs. However, due to the out-of-order nature of database operations within a transaction, operation conflicts may frequently occur. This leads to frequent transaction rollbacks under the optimistic transaction model, impacting database processing efficiency.

[0059] This application provides a data processing method that effectively improves the concurrency of transactions through a hardware-software co-operational execution model. Furthermore, by pre-filtering conflicts, it effectively reduces conflict issues in optimistic transaction modes. Moreover, by combining optimistic and pessimistic modes, it enables switching between them, allowing transactions to continue execution even after conflicts occur, thus meeting the serialization and order preservation requirements without manual intervention. This application can provide a throughput of up to Xmrwops with limited memory resources and minimal CPU resources and power consumption, meeting the low memory capacity and high throughput requirements of database products.

[0060] Figure 2 This is a schematic diagram of a database structure provided for an embodiment of this application. Please refer to... Figure 2 The database includes both client and server components. A description of the client can be found in [reference needed]. Figure 1 The relevant content will not be repeated here. For example, the server side includes a software side and a hardware side. The software side includes, but is not limited to, a query engine, an execution engine, and a storage engine. The hardware side includes, but is not limited to, a memory processor (MP), a CPU (central processing unit), a hardware accelerator, and memory. It should be noted that the memory processor, CPU, and hardware accelerator can be on the same chip or can be independent devices connected via a bus; this application does not impose limitations. Furthermore, the hardware accelerator and memory processor can be logic circuits. Optionally, the software side may include a driver (not shown in the figure) corresponding to the hardware accelerator, such as a hardware acceleration driver. The hardware acceleration driver on the software side can be understood as the logic program of the hardware accelerator. That is, the steps performed by the hardware accelerator in this embodiment can be executed after the hardware accelerator runs its logic program.

[0061] For example, the query engine, execution engine, and storage engine on the software side run on the CPU, that is, the CPU supports each engine in executing the corresponding steps.

[0062] In this embodiment, the query engine includes, but is not limited to, a conflict filtering module (also known as a conflict filter), which can be used to detect conflicts in received database operations (also understood as database commands, such as DML commands). For example, the query engine may also include other modules, such as a parsing module for parsing database speech, etc., which is not limited in this application.

[0063] In the embodiments of this application, Figure 3 For an illustrative example of a collision detection diagram, please refer to... Figure 3Specifically, the conflict detection module can obtain the corresponding hash index operation interface for each primary key database operation (i.e., DML operation, including but not limited to: UPDATE, INSERT, DELETE, etc.), and further obtain the hash value (HashCode) corresponding to each operation. For example, ... Figure 3 As shown, the UPDATE operation can be mapped to lookup, insert, and delete operations. It should be noted that, to distinguish it from DML operations, the English characters for DML operations are represented in uppercase letters in this embodiment, while the operation interfaces of the mapped hash index are represented in lowercase letters. Optionally, the INSERT operation can be mapped to an insert operation. The DELETE operation can be mapped to lookup and delete operations. Next, the conflict detection module obtains the hash value of the operation interface of each hash index. The method of obtaining the hash value can refer to existing technologies, and this application does not limit it.

[0064] For example, the conflict detection module can detect whether there are conflicting values ​​among the obtained hash values ​​to obtain a conflict detection result. Optionally, the conflicting values ​​are the hash values ​​with the same value, and correspondingly, the DML operations corresponding to the hash values ​​are the conflicting operations. For example, after obtaining all hash values, the conflict detection module can compare each hash value with other hash values ​​one by one to detect whether there is a hash value that is the same as the hash value to be compared. In one example, if there are two or more hash values ​​with the same value, the conflict detection module determines that the DML operations corresponding to the hash values ​​with the same value are conflicting. For example, UPDATE operation 1 is mapped to lookup operation 1, insert operation 1, and detele operation 1. DELETE operation 1 is mapped to lookup operation 2 and detele operation 2. The conflict detection module obtains the hash values ​​corresponding to each operation as follows: lookup operation 1 has hash value A, insert operation 1 has hash value B, detele operation 1 has hash value C, lookup operation 2 has hash value D, and detele operation 2 has hash value E. The conflict detection module compares hash value A with hash values ​​B through E (or possibly hash values ​​D and E) one by one to determine if there is a hash value identical to hash value A. If hash value A is not identical to any other hash value, the module continues to compare hash value B with other hash values, and so on, until all hash value comparisons have been completed. For example, assuming hash value C is identical to hash value E, the conflict detection module can determine that the DML operation corresponding to hash value C (i.e., UPDATE operation 1) and the DML operation corresponding to hash value E (i.e., DELATE operation 1) are conflicting operations.

[0065] In this embodiment, the query engine is also used to maintain a transaction mode state machine, which can be used to manage the concurrent processing mode of the database. As mentioned above, the concurrent control modes of the database include, but are not limited to, optimistic transaction mode and pessimistic transaction mode. The query engine can switch the state of the transaction mode state machine (including optimistic transaction mode state and pessimistic transaction mode state) based on the detection mechanism of the conflict filtering module, thereby realizing the switching of the database concurrent mode. In this embodiment, in the initial state, the query engine defaults to setting the transaction mode state machine to optimistic transaction mode. That is, the database defaults to executing database operations concurrently in optimistic transaction mode. When the database (specifically the hardware accelerator) performs a specific operation, if an operation conflict occurs, the underlying hardware (i.e., the hardware accelerator, or possibly other hardware reported, which is not limited in this application) reports an error to the query engine to indicate that a conflict has occurred. Based on the error reported by the underlying hardware, the query engine can switch the transaction mode state machine to pessimistic transaction mode. Then, each module in the database processes transactions serially according to the pessimistic transaction mode, thereby realizing the switching between optimistic mode and pessimistic mode to supplement the optimistic transaction mode and meet the requirements of serial execution.

[0066] Still refer to Figure 2 For example, the execution engine includes, but is not limited to, a hybrid transaction processing module. This module schedules the storage engine based on the transaction mode indicated by the transaction mode state machine of the query module. In one example, if the transaction mode state machine maintained by the query module is set to optimistic transaction mode, the execution engine can group database operations (e.g., DML operations) based on the conflict detection results of the query engine and call the storage engine to execute the corresponding database operations in parallel based on the grouping results. In another example, if the transaction mode state machine maintained by the query module is set to pessimistic transaction mode, the execution engine rolls back the transaction and calls the storage engine to execute the corresponding database operations serially in pessimistic transaction mode.

[0067] In this embodiment, the hybrid transaction processing module is pre-configured with grouping rules. Optionally, the grouping rules include grouping rule 1, which includes, but is not limited to, that any two database operations within the same group do not conflict. In this embodiment, the grouping rules may also include grouping rule 2, which includes, but is not limited to, grouping database operations of the same type together. For example, operations in the same group may all be "insert" operations, or operations in the same group may all be "delete" operations. Optionally, the grouping rules may also include grouping rule 3, which includes, but is not limited to, grouping operations with similar execution costs together. For example, "execution cost" may optionally be the resources consumed by the database to perform data operations. Resources may include, but are not limited to, time, etc., and are not limited in this application. For example, database operations with shorter execution times (e.g., less than a preset threshold, which can be set according to actual needs, and are not limited in this application) are grouped together, while database operations with longer execution times (e.g., greater than or equal to a preset threshold) are grouped together.

[0068] It should be noted that the grouping rules in the embodiments of this application may include grouping rule 1, or grouping rule 1 may be a combination of other grouping rules (such as grouping rule 2 and / or grouping rule 3, or other grouping rules, which can be set according to actual needs, and this application does not limit them).

[0069] In this embodiment, the number of data operations included in a single group depends on the number of execution units. In this embodiment, the execution unit may optionally be the number of devices available for processing database operations in the optimistic transaction mode of the storage engine. In this embodiment, the execution unit is a hardware accelerator in the optimistic transaction mode of the storage engine. Accordingly, the number of database operations included in a single group can be less than or equal to the number of hardware accelerator devices (or, more accurately, the number of hardware acceleration units within the hardware accelerator). For example, if the number of database hardware accelerators is 32, the conflict filtering module can divide the received N operations into multiple (e.g., M) groups based on the grouping rules described above. Each of the M groups contains less than or equal to 32 database operations.

[0070] Figure 4 This is an exemplary grouping diagram. Please refer to... Figure 4The hybrid transaction processing module groups data based on conflict detection results and grouping rules (including grouping rule 1, grouping rule 2, and grouping rule 3) to obtain three groups: Group 1, Group 2, and Group 3. Group 1 contains 10 DELETE operations (the number shown in the diagram is for illustrative purposes only), Group 2 contains 20 INSERT operations, and Group 3 contains 32 INSERT operations. All database operations in Groups 1, 2, and 3 belong to the same transaction; that is, these multiple database operations belong to the same set of operations corresponding to a transaction.

[0071] Still refer to Figure 4 For example, the hash values ​​corresponding to the operations in the first group are all different, the hash values ​​corresponding to the operations in the second group are all different, and the hash values ​​corresponding to the operations in the third group are all different, thus satisfying the requirements of grouping rule 1. For example, all database operations in the first group are DELETE operations, and all operations in the second and third groups are INSERT operations, thus satisfying the requirements of grouping rule 2. For example, the execution cost of each operation in the second group is greater than a preset threshold, and the execution cost of each operation in the third group is less than or equal to a preset threshold, thus satisfying the requirements of grouping rule 3. For example, the number of database operations included in each of the first, second, and third groups is less than or equal to 32. Here, 32 is the number of hardware acceleration units (i.e., 32 units) contained in the hardware accelerator in the electronic device (i.e., the device where the database is located). It should be noted that the types and numbers of database operations included in each group in the grouping results of this application embodiment are only illustrative examples and are not limited in this application.

[0072] It should be noted that, in the embodiments of this application, as Figure 3 As shown, the UPDATE operation corresponds to the lookup, insert, and delete operations. That is, to complete an UPDATE operation, you need to first view the data, then add the data, and finally delete the original data. Specific operations can be found in existing technologies, and will not be elaborated upon here. Correspondingly, during the grouping process, the execution engine can divide the UPDATE operation into DELETE and INSERT operations, and then group the DELETE and INSERT operations corresponding to the UPDATE operation. That is, if a group includes a DELETE operation, this operation may be a database operation indicated by the original statement (i.e., the database statement corresponding to the user-input delete operation), or it may be an operation corresponding to the UPDATE operation; this application does not impose any limitations.

[0073] In this embodiment, the execution engine can determine the execution task (also referred to as an execution plan or execution strategy, etc., which is not limited in this application) based on the grouping results, and call the storage engine to execute it based on the execution task. For example, the execution task includes, but is not limited to, parallel tasks or serial tasks. Parallel tasks are those determined under optimistic transaction mode. This task includes, but is not limited to, the index information corresponding to each database operation in the transaction. For example, for... Figure 4 The second group in the database requires new data. Accordingly, the execution engine (specifically, a hybrid transaction processing module) can generate corresponding index information based on each database operation. This index information indicates the space address where the new data is written. The specific method for generating the index information can be found in existing technical embodiments, and this application does not limit it. To illustrate further, for... Figure 4 In the first group, i.e., the data that needs to be deleted, the execution engine can determine the index information corresponding to the data that needs to be deleted. The specific determination method can be found in existing technical embodiments, and this application does not limit it.

[0074] In this embodiment, a parallel task includes index information corresponding to multiple database operations, instructing the storage engine to call the hardware accelerator to execute multiple database operations in parallel. A serial task includes index information corresponding to one or more serial database operations, instructing the storage engine to call the hardware accelerator to execute database operations serially.

[0075] In this embodiment, the index information can be a hash index or an ART (Adaptive Radix Tree) index. Hash indexes feature balanced read / write operations and a scalable hash table structure, making them suitable for primary key and point queries. ART indexes, with their radix tree structure, support sorting, LPM (Library of Parameterized Modules), adaptive node scaling, and cache friendliness, making them suitable for secondary indexes and range queries.

[0076] For example, after the execution engine determines the task to be executed, it can call the storage engine to perform the corresponding operation. For instance, taking the optimistic transaction pattern as an example, refer to... Figure 4 After the execution engine determines the parallel tasks (e.g., index information) corresponding to each database operation (or transaction) in the first group, it can instruct the storage engine on the corresponding parallel tasks based on the index interface with the storage engine. These parallel tasks include, but are not limited to, the index information of each database operation in the first group, to indicate to the storage engine which transactions are processed as a group in parallel. The storage engine can then invoke the hardware accelerator to execute the corresponding operations in parallel based on the execution engine's instructions.

[0077] In one possible implementation, the grouping step can also be performed by the query engine. The query engine can group based on the conflict detection results and instruct the execution engine on the grouping results. The execution engine can then invoke the storage engine to perform corresponding operations based on the grouping results. In other words, the steps performed by each engine (or module) in this application can also be performed by other engines (or modules). This application does not limit the execution entity, and this will not be repeated below.

[0078] The storage engine can determine the grouping corresponding to each index based on the instructions of the execution engine (i.e., the execution task). This can also be understood as determining which database operations corresponding to which index information need to be processed in parallel.

[0079] Please continue to refer to Figure 2 For example, the storage engine includes a transaction decoupling module. This module has a delayed operation queue, which can place multi-version information of transactions in the delayed queue, thereby decoupling the multi-version information of transactions (which can also be understood as database operations) from the index. In other words, compared to the existing technology of recording multi-version information of transactions through the index, this embodiment records the multi-version information in the queue, thus decoupling it from the index. In this way, when the storage engine calls the hardware accelerator to execute transactions based on the index information, it does not carry multi-version information, facilitating subsequent batch processing. In existing technologies, the storage engine needs to mark the index information of the data corresponding to the deletion operation to mark that version of the data as the version to be deleted. In actual processing, the storage engine needs to mark the index according to the operation status and perform operations such as deletion on the data indicated by the index based on the mark. During this process, the storage engine needs to continuously search for the mark and perform corresponding processing based on the mark. In this embodiment, the transaction decoupling module sets up a delayed operation queue, allowing version information to be stored in the queue without requiring deletion marking. During execution, the storage engine can directly invoke the hardware accelerator to perform corresponding operations on the data indicated by the index (which may be the original version or a copy) based on the index's status in the queue. This reduces the marking and deletion process, simplifying the overall process and improving data processing efficiency. For example, if the database receives multiple operations, including changing data A to data B and changing data B to data C, then data A, data B, and data C are multiple versions of the same data. This can also be understood as the state of data at different times. Exemplarily, data A, data B, and data C each have their own index. The delayed queue maintained by the storage engine can find the corresponding version of the data based on the index and invoke the hardware accelerator to perform the corresponding operation.

[0080] It should be noted that the delayed operation queue in this embodiment follows the first-in, first-out principle.

[0081] In this embodiment, the delayed queue may include a first queue and a second queue. The first queue may be called a commit queue, and the second queue may be a fail queue. The commit queue stores information such as the index corresponding to the delete operation. The fail queue stores information such as the index corresponding to the reverse operation of the insert operation.

[0082] In this embodiment, delete operations are placed in a commit queue. This allows the database to process all delete operations in the commit queue before committing transactions, eliminating the need to mark delete operations as deleted, thus speeding up the transaction processing of delete operations. In other words, in this embodiment, for delete operations, a delayed processing approach is used. After the execution engine completes grouping, it instructs the storage engine to place the delete operations in a delayed queue and generate parallel tasks before committing, instructing the storage engine to call the hardware acceleration module to execute the delete operations in parallel. For insert operations, corresponding parallel tasks are generated immediately after grouping, instructing the storage engine to call the hardware acceleration module to execute the parallel operation. Simultaneously, the execution engine instructs the storage engine to place the reverse operation of the insert operation in a failure queue.

[0083] In this embodiment, the reverse operation of the insert operation is placed in a failure queue. This allows the database to roll back the transaction based on the reverse operation in the failure queue during transaction rollback. For example, the reverse operation of the insert operation is the delete operation. To illustrate, if insert operation A adds data A to a specified address space, then the reverse operation of insert operation A is deleting data A. It can be understood that during transaction rollback, all completed operations need to be restored to their state before the transaction execution; correspondingly, the results of the executed insert operations need to be deleted.

[0084] In one possible implementation, to meet different transaction isolation requirements, the queue can also record the time information corresponding to each database operation. Optionally, database index information and time information can be recorded in the queue. The CPU (which can be an execution engine or a storage engine) can be equipped with a recycling module (also called a cleanup module, etc., which is not limited in this application). The recycling module can delete objects that meet the conditions based on the time information corresponding to each object (i.e., each database operation) in the submission queue. This condition can optionally be the database operation with the oldest time information (or can be understood as the database operation with the smallest time). That is, when the CPU detects that the time information of any operation in the submission queue is the smallest, the CPU or the CPU can call the hardware accelerator (the specific calling method can be referred to the embodiment below) to execute the deletion operation, and after the execution is completed, delete the information (including index information and time information) corresponding to the deletion operation in the queue. It can be understood that the data corresponding to the deletion operation with the smallest time will not be accessed by other operations, and thus will not cause operation conflicts.

[0085] In one possible implementation, the delay queue can also be maintained by the execution engine, which is not limited in this application.

[0086] For example, the storage engine can invoke hardware accelerators to perform corresponding database operations in parallel or serially, based on the instructions of the execution engine (i.e., the task to be executed).

[0087] Please continue to refer to Figure 2 For example, the hardware side includes, but is not limited to: memory processors, CPUs, hardware accelerators, and memory.

[0088] In this embodiment, the memory may include shared memory, the storage space of which is less than or equal to the storage space of the main memory. For example, the shared memory is pre-configured. In this embodiment, both the hardware accelerator and the memory processor can directly read the information and data in the shared memory.

[0089] In this embodiment, the query engine, execution engine, and storage engine on the software side all run on the CPU. Optionally, the CPU can be single-core or multi-core; this application does not impose any limitation.

[0090] In the embodiments of this application, the hardware accelerator may include at least one type of accelerator. For example, it may include a first hardware accelerator and a second hardware accelerator. The first hardware accelerator may include multiple (e.g., 32, depending on the pre-shipment hardware settings, which are not limited in this application) hardware acceleration units, and the second hardware accelerator may include multiple hardware acceleration units. Optionally, the first hardware accelerator may be a hash accelerator (HashAcc), and the second hardware accelerator may be an ART accelerator (ARTAcc).

[0091] For example, a hash accelerator can be used to process hash indexes (see above for the concept), and an ART accelerator can be used to process ART indexes (see above for the concept). In other words, a storage engine can invoke different types of hardware accelerators to process the corresponding types of indexes.

[0092] For example, the memory processor may optionally be a programmable acceleration unit; in this embodiment, the memory processor can also be understood as a coprocessor for scheduling. The memory processor can invoke hardware accelerators based on instructions from the storage engine.

[0093] In this embodiment, the memory processor can provide different types of interfaces to the storage engine (i.e., the CPU) to implement different calls. For example, the memory processor can provide a first interface, a second interface, and a third interface. In one example, the storage engine can call the hardware accelerator through the first interface. For example, the storage engine can send an indication signal to the memory processor through the first interface to indicate the start of the hardware accelerator and the number of hardware acceleration units in the started hardware accelerator. As described above, the hardware accelerator may include multiple hardware acceleration units, and the execution engine can group database operations based on the number of hardware acceleration units. During execution, the storage engine can call hardware acceleration units equal to the number of database operations in the group, so that multiple hardware acceleration units can execute multiple database operations within the group in parallel. For example, in response to the received indication information, the memory processor triggers the hardware accelerator to start the corresponding number of hardware acceleration units, which can also be understood as starting the corresponding number of threads, or triggering the hardware accelerator to execute the corresponding program logic to perform parallel tasks.

[0094] In another example, the memory processor can release the CPU via a second interface. After instructing the hardware accelerator to perform a database operation, the memory processor can send an indication signal to the CPU via the second interface to indicate that the hardware accelerator has started executing the corresponding operation. To avoid CPU waiting and wasting resources, the memory processor sends an indication signal to the CPU via the second interface to indicate that the CPU should be released. In response to the received indication, the CPU can perform other operations to avoid long waiting times, thereby improving the utilization of hardware resources. Optionally, after the hardware accelerator completes a parallel task, the memory processor can send an indication signal to the CPU to instruct the CPU to read the execution results from shared memory. Specific details will be explained in detail below.

[0095] In yet another example, the storage engine can also read data from shared memory via a third interface.

[0096] This can be understood as follows: the first and second interfaces are both control channels used to transmit control signals. The third interface can be a data channel used to transmit data.

[0097] For example, the communication protocols supported by each interface may be the same or different. The format of the information transmitted in the interface conforms to the interface's protocol.

[0098] In this embodiment of the application, the first interface can be the spawn (call) interface, the second interface can be the suspend (stop) interface, and the third interface can be the AccMsgQueque (accelerated information queue) interface.

[0099] For example, as described above, the execution engine can determine the corresponding parallel tasks based on the grouping results. Parallel tasks include, but are not limited to, index information. The execution engine can indicate the parallel tasks to the storage engine. The storage engine can offload (i.e., transfer) the index information corresponding to the parallel tasks to the hardware accelerator, so that the hardware accelerator can be invoked to execute the corresponding database operations in parallel based on the index information.

[0100] It should be noted that, in this embodiment of the application, the storage engine can input the index information corresponding to the parallel task into the shared memory, and the hardware accelerator can read the corresponding index information from the shared memory and perform corresponding operations, thereby realizing data transmission between the storage engine and the hardware accelerator.

[0101] Combination Figure 2The data processing method in this application will be described in detail below with specific embodiments. As mentioned above, database operations include DDL operations and DML operations. Before executing DML operations, the database needs to create database tables through DDL operations. In this application embodiment, only the form of a database table is used as an example for illustration. Other forms may be used in other embodiments, and this application does not limit them.

[0102] Figure 5 For an illustrative example of the database table creation process, please refer to [link / reference]. Figure 5 Specifically, including but not limited to:

[0103] S501, the server retrieves the creation data request.

[0104] For example, a user can create metadata by inputting corresponding database statements through the human-computer interaction interface provided by the database client. In response to the received user command (i.e., the database statement), the client sends a data creation request to the database, instructing the server to create metadata. For example, metadata refers to data that defines the data structure. For instance, the header (i.e., columns) and name of a database table are both metadata corresponding to that table. The data in the database table is the data content. The index information described in this embodiment is used to indicate the spatial address where the data content is stored.

[0105] S502, the server creates metadata based on the data creation request.

[0106] For example, after receiving a data creation request, the server parses the request to obtain the information needed to create the metadata, which can be called schema information. This information includes, but is not limited to, table, column, data type, primary key, and foreign key information.

[0107] The server can create metadata based on the obtained schema information. Specific details can be found in existing technical embodiments, and this application does not limit the scope of the application.

[0108] S503, the server stores the metadata of the hardware accelerator.

[0109] For example, in this embodiment of the application, if the electronic device where the database is located has Figure 2 If the hardware structure in the code includes a hardware accelerator that can accelerate database operations, then the server records relevant information about the hardware accelerator. This information is used to indicate the control where the hardware accelerator is located and which hardware accelerator (e.g., the first or second hardware accelerator) is used to perform subsequent acceleration operations on the metadata created by S302.

[0110] Figure 6This is a schematic diagram illustrating an exemplary data processing flow. Please refer to [the diagram]. Figure 6 Specifically, including but not limited to the following steps:

[0111] S601, the query engine performs conflict filtering on operations.

[0112] In this embodiment of the application, after the metadata (e.g., database tables) is created, the user enters database statements through the interface provided by the client to perform corresponding database operations on the database tables, i.e., DML operations, to perform database operations such as adding, deleting, modifying, and querying on the database tables.

[0113] It should be noted that, Figure 4 The flowchart in the document corresponds to the processing of add, delete, and update operations. For the query operation, which can be understood as reading data from a database table, please refer to [the relevant documentation / reference]. Figure 9 The relevant description in the document.

[0114] The query engine performs conflict detection on all retrieved database operations. For details on the conflict detection methods, please refer to [link / reference needed]. Figure 3 This will not be elaborated upon here. After obtaining the conflict detection results, the query engine can indicate the conflict detection results to the execution engine.

[0115] In this embodiment of the application, after the query engine starts, it sets the transaction mode state machine to optimistic transaction mode. Optionally, the query engine can indicate the current transaction mode, i.e., optimistic transaction mode, to the execution engine, so that the execution engine can determine parallel tasks based on the optimistic transaction mode.

[0116] S602, the execution engine is grouped.

[0117] For example, the execution engine determines that the current mode is an optimistic transaction mode based on the state of the transaction mode state machine maintained by the query engine. Then, the execution engine can further group transactions based on the conflict detection results indicated by the query engine.

[0118] In this embodiment of the application, the database operations obtained by the query engine may include two or more database operations that are dependent on each other. Of course, all the database operations obtained may also be out of order (i.e., they are not dependent on each other).

[0119] In this embodiment, database operations with dependencies can be grouped into the same group. For out-of-order database operations, the execution engine can group the database operations based on conflict detection results to ensure that database operations within the same group do not conflict with each other. Specific grouping methods can be found in the embodiments described above. Figure 4The relevant descriptions will not be repeated here. In other words, in this embodiment of the application, the database can effectively reduce the conflict problems caused by out-of-order operations by performing conflict filtering on database operations in advance.

[0120] Next, the execution engine can determine parallel tasks based on the grouping results. For example, for an insert operation, the execution engine can generate index information and other relevant information. For a delete operation, the execution engine can determine the corresponding index information and other relevant information. To illustrate, let's continue with the example... Figure 4 The execution engine can determine the corresponding index information and other details for each delete database operation in the first group based on the grouping results, and determine that each delete database operation in the first group is a parallel task. Correspondingly, the execution engine generates index information and other relevant information for each insert database operation in the second group based on the grouping results. The execution engine can determine that each insert database operation in the second group is a parallel task.

[0121] It should be noted that, as mentioned above, the delete operation is placed in the commit queue and processed before the transaction is committed. The execution engine may also determine the parallel task corresponding to the delete operation before the transaction is committed. This application does not impose any limitations on this.

[0122] In this embodiment of the application, by pre-filtering multiple database operations in each parallel task (which can also be understood as grouping), 99% (this value is an experimental value and is not limited in this application) of conflicting operations can be filtered out, so that the parallel operations in the optimistic transaction mode can be executed normally, effectively improving the efficiency of data processing.

[0123] In one possible implementation, the parallel tasks and grouping results may not correspond. For example, the execution engine can group non-conflicting operations into one group based on conflict detection results, while conflicting operations are grouped into different groups. For instance, the execution engine detects that operations A, B, and C conflict with each other, and operations D and E also conflict. Accordingly, the execution engine can place operations A, B, and C into groups A, B, and C respectively. Operation D is placed in group B, and operation E is placed in group C. That is, the execution engine can group operations not based on the number of accelerators, but only according to grouping rule 1. In this example, when the execution engine formulates a parallel plan, it can do so based on grouping rules 2 and 3 mentioned above. For example, the execution engine can select 32 database operations (i.e., those meeting the number of accelerators) from group A, where these 32 database operations satisfy grouping rules 2 and 3, thus ensuring that the database operations in the parallel plan do not conflict, and that the execution costs and types of the database operations are similar. The execution engine can also select another 32 database operations from other database operations in group A to formulate a parallel plan. Similarly, the execution engine can select 32 database operations from group B and formulate a parallel plan.

[0124] In this embodiment of the application, the parallel tasks of the database are constructed based on the conflict detection results, which can filter out most of the problems that may cause operational conflicts. In other words, by placing operations that may cause operational conflicts into different parallel tasks (i.e., groups), conflicts in the execution of parallel tasks can be effectively avoided, reducing the probability of conflicts occurring during parallel execution, thereby improving the processing efficiency of the optimistic transaction mode. It can also be understood as an optimization of the optimistic transaction mode to avoid frequent transaction rollbacks caused by operational conflicts.

[0125] In S603, the storage engine uses a delayed queue to manage index information.

[0126] For example, the execution engine can instruct the storage engine to perform at least one parallel task through an index interface with the storage engine. The storage engine can then perform corresponding operations based on the parallel tasks.

[0127] Specifically, the storage engine can obtain the corresponding index information based on parallel tasks. The database operations corresponding to the index information are the operations that need to be processed in parallel.

[0128] In this embodiment of the application, as described above, the storage engine may maintain a first queue and a second queue. Figure 7 For an illustrative diagram of the queue, please refer to... Figure 7 ,like Figure 7As shown, the first queue (i.e., the submission queue) includes multiple index information, such as indexes 11 to 15, and the time information corresponding to the database operation for each index, such as timestamps 11 to 15. For example, index 11 could be... Figure 4 The index information corresponding to the delete operation in the first group is used to point to the data to be deleted (e.g., data 11). (Refer to...) Figure 7 The second queue (i.e., the failure queue) contains multiple index information, such as indices 21 to 25, and the time information corresponding to the database operation for each index, such as timestamps 21 to 25. For example, index 21 could be... Figure 4 The index information corresponding to the reverse operation of the increment operation in the second group (the concept can be referred to above) is used to point to the space address where the data to be written in the increment operation, so that the data content in that space address can be deleted during the reverse operation.

[0129] In this embodiment, as described above, when the execution conditions are met, the storage engine calls the hardware accelerator to execute the corresponding operations for each operation in the commit queue (i.e., the operations corresponding to the index information). In this embodiment, the execution condition for the commit queue is execution upon commit. That is, the delete operations corresponding to each index in the commit queue are executed before being committed. This can also be understood as the storage engine calling the hardware accelerator to execute the corresponding delete operations before the transaction is ready to commit, i.e., when the corresponding data content needs to be deleted. In other words, in this embodiment, by maintaining the commit queue, the database can effectively avoid the time consumption of marking and querying steps for deletion, effectively improving the execution efficiency of delete operations and thus improving the overall processing efficiency of the database. Furthermore, by maintaining the failure queue, the database can obtain the corresponding index information when transaction rollback is needed, and perform rollback based on the index information. This method, on the one hand, allows multi-version information to be stored in the queue without loss. On the other hand, storing multi-version information in the queue eliminates the need for additional marking of the index information, allowing the accelerator to obtain only the index information, enabling the accelerator to complete parallel tasks faster and better.

[0130] In this embodiment of the application, as described above, each operation in the failure queue can be executed immediately, that is, it can be executed without waiting for submission.

[0131] S604, the storage engine calls the hardware accelerator to complete batch operations.

[0132] For example, the storage engine can invoke the hardware accelerator to perform corresponding operations according to the instructions of the execution engine and in accordance with parallel tasks.

[0133] Figure 8This is an illustrative diagram illustrating a hardware accelerator call. Please refer to... Figure 8 For example, the CPU (i.e., the storage engine) sends an instruction signal to the memory accelerator through the first interface to trigger the hardware accelerator and offload the task to be executed (i.e., index information) to the hardware accelerator.

[0134] In response to an indication signal received from the first interface, the memory accelerator can determine whether a hardware accelerator needs to be activated, the type of hardware accelerator to be activated, and the number of hardware acceleration units to be activated. For example, the memory accelerator triggers a second hardware accelerator, which activates 32 acceleration units, or 32 acceleration threads. Each thread (or acceleration unit) processes one index piece of information.

[0135] As mentioned above, the type of hardware accelerator to be activated can be determined based on the type of index information. For example, if the index information is of type ART index, then the second hardware accelerator (i.e., the ART accelerator) will be invoked. As another example, if the index information is of type hash index, then the first hardware accelerator (i.e., the hash accelerator) will be invoked.

[0136] In this example, the hardware accelerator that needs to be started is taken as the second hardware accelerator, and the index information corresponding to 32 database operations in the parallel task (that is, the group includes 32 database operations, this value is only an illustrative example and is not limited in this application) is taken as an example.

[0137] For example, as described above, the storage engine can transmit parallel tasks to the hardware accelerator. For instance, the storage engine can write parallel tasks to shared memory through a third interface. These parallel tasks include, but are not limited to, the index information and corresponding operations (i.e., insert operations, or data to be written) corresponding to the 32 database operations that need to be executed in parallel, to instruct the hardware accelerator to write the corresponding data to the space indicated by the 32 index information.

[0138] Optionally, the storage engine can input information from parallel tasks in the form of a queue in shared memory, which is not limited in this application.

[0139] For example, the second hardware accelerator responds to the trigger of the memory processor by starting 32 hardware acceleration units, which can also be understood as starting 32 acceleration threads, each thread is used to process a database operation, thereby realizing the parallel processing of 32 database operations.

[0140] For example, each of the 32 hardware acceleration units sequentially reads parallel tasks from shared memory. For instance, as shown... Figure 8As shown, hardware acceleration unit 1 (not shown in the figure) reads index 11 and operation information (e.g., increment operation and corresponding data, not shown in the figure) from shared memory; hardware acceleration unit 2 reads index 12 and operation information from shared memory; hardware acceleration unit 3 reads index 13 and operation information from shared memory; and hardware acceleration unit 4 reads index 14 and operation information from shared memory. The figure only shows the operations of four indices (i.e., four acceleration units), but in reality, more may be included; these will not be described in detail here.

[0141] Still refer to Figure 8 For example, each hardware acceleration unit (e.g., hardware acceleration unit 1 to hardware acceleration unit 32) writes corresponding data into the space address indicated by the index information based on the obtained index information and operation information to complete the increment operation.

[0142] For example, after each database operation is completed, the hardware accelerator writes the result to a circular queue in shared memory. For example, the result may be either success or failure.

[0143] For example, in a circular queue, each execution result can correspond to an execution ID, which can be agreed upon with the CPU. For instance, when the CPU writes parallel tasks to shared memory, each index piece of information can also correspond to an execution ID, which can be used to identify the database operation in this parallel task.

[0144] For example, the hardware accelerator can write the execution result and its corresponding execution ID to a circular queue. It should be noted that the data format in the circular queue conforms to the communication protocol requirements of the third interface. That is, both the CPU and the hardware accelerator can read or write data to the circular queue.

[0145] S605, the storage engine detects whether a conflict has occurred.

[0146] For example, the storage engine (i.e., the CPU, or it could be executed by the execution engine, which is not limited in this application) reads the circular queue in shared memory through a third interface. The storage engine can determine the database operation corresponding to each result based on the execution ID corresponding to each result.

[0147] For example, the index 11 corresponding to insert operation 1 has an execution ID of ID1. The hardware accelerator (e.g., hardware acceleration unit 1) writes data to the corresponding space based on index 21 and then writes result 1 to the circular queue. Result 1 includes ID1 and success indication information to indicate the success of the database operation corresponding to ID1 (i.e., database operation 1), meaning that data has been successfully written to the space pointed to by the index corresponding to the database operation.

[0148] To illustrate further, insert operation 2 corresponds to index 22, and its corresponding execution ID is ID2. Hardware acceleration unit 2 writes data to the corresponding space based on index 22 and detects a data write failure. Hardware acceleration unit 2 writes result 2 to the circular queue. Result 2 includes ID2 and failure indication information to indicate that the execution result of the database operation corresponding to ID2 (i.e., database operation 2) has failed, meaning that data could not be successfully written to the space pointed to by the index corresponding to the database operation.

[0149] The storage engine can read results from the circular queue via a third interface. In one example, if any result read fails, an operation conflict is determined, and S607 is executed.

[0150] In another example, if the storage engine reads all results from the circular queue and all results are successful, then S604 to S605 are executed repeatedly. That is, the parallel tasks corresponding to other groups continue to be executed until all insert operations are completed.

[0151] S606, Transaction Commit.

[0152] For example, as described above, the delete operation in the commit queue is executed before the transaction is committed. Before the transaction is committed, which can also be understood as after all operations in the transaction except for the delete operation have been processed, the storage engine, according to the parallel tasks of the execution engine, retrieves the corresponding index information from the commit queue and calls the hardware accelerator to complete the delete operation in parallel. The specific implementation is the same as the insert operation, and will not be repeated here.

[0153] For example, after reading all results from the circular queue, the storage engine deletes the information of the database operations corresponding to all results in the circular queue from the commit queue. For instance, the parallel task includes index information 11 and operation information (e.g., the delete operation) for delete operation 1. The storage engine calls the hardware accelerator to execute the delete operation, deleting the data content in the space address pointed to by index information 11. After the hardware accelerator executes the corresponding delete operation, it writes the execution result to the circular queue. The storage engine reads the result from the circular queue. For example, after reading the results corresponding to all database operations in this parallel task, the storage engine can delete relevant information from the delay queue. For example, the storage engine deletes the information corresponding to delete operation 1 from the commit queue, including index information 11 and time information.

[0154] For example, the storage engine determines that all database operations in this transaction have been completed before committing the transaction and sending the transaction execution result back to the client. Optionally, the transaction execution result includes, but is not limited to, success or failure. The failure result is sent if the transaction fails after switching to pessimistic transaction mode.

[0155] For example, the storage engine clears information from the failure queue.

[0156] S607, the query engine switches to pessimistic transaction mode.

[0157] For example, if the storage engine detects a conflict, it can report it to the query engine. The query engine then switches its transaction mode state machine from optimistic transaction mode to pessimistic transaction mode.

[0158] S608, the storage engine calls the hardware accelerator to perform serial data operations.

[0159] For example, if the execution engine detects that the query engine's state machine has switched to pessimistic transaction mode, it can generate corresponding serial tasks based on the current mode. In other words, in pessimistic transaction mode, data operations will be executed serially to prevent operation conflicts.

[0160] For example, after the execution engine detects a switch to pessimistic transaction mode, it instructs the storage engine to roll back the transaction. As mentioned above, delete operations in the commit queue are processed before the transaction is committed, meaning that delete operations in the commit queue generally do not conflict. The failure queue stores information (including but not limited to index information) corresponding to the reverse operations of insert operations. The storage engine (i.e., the CPU) or the storage engine calls the hardware accelerator to execute the reverse operations corresponding to all completed insert operations in the failure queue to delete the added data. That is, the storage engine or the hardware accelerator can delete the data in the space pointed to by the index information based on the index information corresponding to the reverse operation.

[0161] It should be noted that the information in the submission queue and failure queue is still saved.

[0162] For example, after a transaction rollback, the execution engine generates serial tasks. The execution engine can invoke the hardware accelerator via the memory processor, as described above. Unlike the previous embodiment, this example only requires invoking a hardware acceleration unit within a single hardware accelerator to serially complete multiple database operations. The specific implementation of the pessimistic transaction mode can be found in the descriptions of existing technical embodiments, and will not be repeated here.

[0163] Similar to the embodiments above, the hardware accelerator can write the execution result to a circular queue after each database operation. The CPU (e.g., the execution engine) can read the corresponding execution result, and the implementation is similar to that described above, so it will not be repeated here.

[0164] For example, the storage engine commits the transaction once all database operations for the transaction have been executed.

[0165] In one possible implementation, under pessimistic transaction mode, if the CPU is a multi-core processor, the storage engine may not call the hardware accelerator, but the CPU may execute serial tasks instead, which is not limited in this application.

[0166] In another possible implementation, the query engine may perform operations such as conflict detection, which is not limited in this application.

[0167] Figure 9 For an illustrative diagram illustrating the data reading process, please refer to... Figure 9 Specifically, this includes: the client receiving database statements input by the user to read data from the database (e.g., database tables). The client sends a query request to the server. The execution engine on the server receives the query request, parses it to obtain the corresponding index information and determine the corresponding database operation (i.e., query operation). The execution engine can generate a query task and instruct the storage engine on the query task. The query task includes, but is not limited to, index information and query operations.

[0168] For example, the storage engine can read the corresponding data content from shared memory based on a query task and then return the read data content to the client.

[0169] It should be noted that, Figure 9 This illustrates a direct connection method, meaning that users can input commands through the client, and the client can interact locally with the server. It can also be understood as... Figure 9 All interactions within the system are local interactions (or can be understood as local operations). In another possible implementation, the user can also establish a communication connection with the database device (e.g., a server) through other terminals (such as computers or tablets, which are not limited in this application). This communication connection can be socket communication. Correspondingly, a database client will also be deployed on the terminal. The user enters a query command on the client, and the client communicates with the database server over the network to send the command to the server via the communication connection.

[0170] It should be further noted that the technical solution of this invention can be applied not only to the database processing of data communication platforms, but also to other computing platforms with data sharing and access. For example, data storage platforms, graph computing platforms, and streaming computing platforms. Taking a data storage platform as an example, applying this technical solution can achieve fast data reading and writing, thereby enabling fast writing and querying of data on the data storage platform. On the other hand, the efficient conflict checking mechanism can serve as a basic module for general-purpose or embedded databases, reducing transaction conflicts and improving database throughput.

[0171] This application embodiment can divide the device into functional modules according to the above method example. For example, each function can be divided into its own functional module, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware or as a software functional module. It should be noted that the module division in this application embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.

[0172] Figure 10 This is a schematic diagram of the structure of a data processing apparatus 1000 as an example. Please refer to... Figure 10 The data processing device 1000 includes, but is not limited to, an acquisition module 1001, a conflict detection module 1002, a grouping module 1003, and an execution module 1004. The acquisition module 1001 acquires transactions, each comprising N database operations; N is a positive integer greater than 1. The conflict detection module 1002 performs conflict detection on the N database operations, obtaining a conflict detection result. The conflict detection result indicates that a first database operation and a second database operation are conflicting operations. The first and second database operations belong to the N database operations. The grouping module 1003 divides the N database operations into M groups based on the conflict detection result. Each of the M groups includes multiple database operations; the first and second database operations belong to different groups; M is a positive integer greater than 1. The execution module 1004 concurrently executes the multiple database operations in each of the M groups.

[0173] In one example Figure 10 A schematic block diagram of an apparatus 1000 according to an embodiment of this application is shown. The apparatus 1000 may include a processor 1001 and a transceiver / transceiver pin 1002, and optionally, a memory 1003. The processor 1001 can be used to execute the steps performed by the database in the methods of the foregoing embodiments, and to control the receive pin to receive signals and control the transmit pin to transmit signals.

[0174] The various components of device 1000 are coupled together via bus 1004, which includes a data bus, a power bus, a control bus, and a status signal bus. However, for clarity, all buses are labeled as bus system 1004 in the figure.

[0175] Optionally, the memory 1003 can be used for storage instructions in the foregoing method embodiments.

[0176] All relevant content of each step involved in the above method embodiments can be referenced from the functional description of the corresponding functional module, and will not be repeated here.

[0177] Based on the same technical concept, embodiments of this application also provide a computer-readable storage medium storing a computer program containing at least one piece of code that can be executed by a computer to control the computer to implement the above-described method embodiments.

[0178] Based on the same technical concept, this application also provides a computer program, which, when executed by a computer, is used to implement the above-described method embodiments.

[0179] The program may be stored, in whole or in part, on a storage medium packaged with the processor, or in part or in whole on a memory not packaged with the processor.

[0180] Based on the same technical concept, this application also provides a processor for implementing the above-described method embodiments. The processor can be a chip.

[0181] The steps of the methods or algorithms described in conjunction with the embodiments of this application can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, portable hard disks, CD-ROMs, or any other form of storage medium well known in the art. An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can reside in an ASIC.

[0182] Those skilled in the art will recognize that the functions described in the embodiments of this application in one or more of the above examples can be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein communication media include any medium that facilitates the transfer of a computer program from one place to another. Storage media can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0183] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A data processing method, characterized in that, include: Acquire a transaction, which includes N database operations; N is a positive integer greater than 1; Conflict detection is performed on the N database operations to obtain conflict detection results; wherein, the conflict detection results indicate that the first database operation and the second database operation are conflicting operations; the first database operation and the second database operation belong to the N database operations; Based on the conflict detection results, the N database operations are divided into M groups; each of the M groups includes multiple database operations; wherein the first database operation and the second database operation belong to different groups; M is a positive integer greater than 1; Concurrently execute multiple database operations in each of the M groups; The concurrent execution of multiple database operations in each of the M groups includes: Multiple hardware accelerators are invoked to concurrently execute multiple database operations in a single group from the M groups; wherein a single hardware accelerator executes a single database operation.

2. The method according to claim 1, characterized in that, The method of calling multiple hardware accelerators to concurrently execute multiple database operations in a single group of the M groups includes: Based on the X database operations included in the first group, X hardware accelerators are triggered; the first group belongs to the M groups. Instruct the X database operations to the X hardware accelerators, so that the X hardware accelerators execute the X database operations in parallel.

3. The method according to claim 2, characterized in that, The method of calling multiple hardware accelerators to concurrently execute multiple database operations in a single group of the M groups includes: The execution results of the X database operations are read from shared memory; wherein the execution results of the X database operations are written after the multiple hardware accelerators execute the X database operations concurrently.

4. The method according to claim 3, characterized in that, The method of calling multiple hardware accelerators to concurrently execute multiple database operations in a single group of the M groups includes: After reading the execution results of the X database operations from shared memory, Y hardware accelerators are triggered based on the Y database operations included in the second group; the second group belongs to the M groups. Instruct the Y database operations to the Y hardware accelerators, so that the Y hardware accelerators execute the Y database operations in parallel.

5. The method according to claim 3, characterized in that, The execution results of the X database operations are placed in the circular queue of the shared memory.

6. The method according to claim 2, characterized in that, The method of calling multiple hardware accelerators to concurrently execute multiple database operations in a single group of the M groups includes: A first indication signal is sent to the memory processor via a first interface, the indication signal being used to instruct the memory processor to trigger the X hardware accelerators.

7. The method according to any one of claims 1 to 6, characterized in that, Before concurrently executing multiple database operations in each of the M groups, the process includes: Place the index information of the delete operation among the N database operations into the first queue; The index information of the reverse operation of the insert operation among the N database operations is placed in the second queue.

8. The method according to claim 7, characterized in that, The concurrent execution of multiple database operations in each of the M groups includes: Before submitting the transaction, delete operations in the first queue are performed in parallel according to the M groupings.

9. The method according to claim 7, characterized in that, The method further includes: If an operation conflict is detected during the concurrent execution of multiple database operations in each of the M groups, the reverse operation of the insert operation is executed based on the index information of the reverse operation of the insert operation in the second queue, thereby completing the transaction rollback.

10. The method according to claim 9, characterized in that, The method further includes: After the transaction is rolled back, the N database operations are executed sequentially.

11. The method according to claim 1, characterized in that, The conflict detection process for the N database operations, to obtain conflict detection results, includes: Obtain the hash value corresponding to each of the N database operations; Detect whether there are any hash values ​​with the same value corresponding to the N data operations; If the hash value of the first database operation is found to be equal to the hash value of the second database operation, it is determined that the first database operation and the second database operation are conflicting operations.

12. The method according to claim 11, characterized in that, Based on the conflict detection results, the N database operations are divided into M groups, including: Based on the conflict detection results and the preset grouping rules, the N database operations are divided into M groups; The preset grouping rules include at least one of the following: The hash values ​​of multiple database operations within a single group are not equal; The number of database operations contained in a single group is less than or equal to the number of hardware accelerators; The execution time of database operations contained in a single group is within a specified range; All database operations contained in a single group are of the same type.

13. A data processing apparatus, characterized in that, include: The acquisition module is used to acquire transactions, which include N database operations; N is a positive integer greater than 1; A conflict detection module is used to perform conflict detection on the N database operations and obtain a conflict detection result; wherein, the conflict detection result indicates that the first database operation and the second database operation are conflicting operations; the first database operation and the second database operation belong to the N database operations; The grouping module is used to divide the N database operations into M groups based on the conflict detection results; each of the M groups includes multiple database operations; wherein the first database operation and the second database operation belong to different groups; M is a positive integer greater than 1; The execution module is used to concurrently execute multiple database operations in each of the M groups; The execution module is specifically used for: Multiple hardware accelerators are invoked to concurrently execute multiple database operations in a single group from the M groups; wherein a single hardware accelerator executes a single database operation.

14. The apparatus according to claim 13, characterized in that, The execution module is specifically used for: Based on the X database operations included in the first group, X hardware accelerators are triggered; the first group belongs to the M groups. Instruct the X database operations to the X hardware accelerators, so that the X hardware accelerators execute the X database operations in parallel.

15. The apparatus according to claim 14, characterized in that, The execution module is specifically used for: The execution results of the X database operations are read from shared memory; wherein the execution results of the X database operations are written after the multiple hardware accelerators execute the X database operations concurrently.

16. The apparatus according to claim 15, characterized in that, The execution module is specifically used for: After reading the execution results of the X database operations from shared memory, Y hardware accelerators are triggered based on the Y database operations included in the second group; the second group belongs to the M groups. Instruct the Y database operations to the Y hardware accelerators, so that the Y hardware accelerators execute the Y database operations in parallel.

17. The apparatus according to claim 16, characterized in that, The execution results of the X database operations are placed in the circular queue of the shared memory.

18. The apparatus according to claim 14, characterized in that, The execution module is specifically used for: A first indication signal is sent to the memory processor via a first interface, the indication signal being used to instruct the memory processor to trigger the X hardware accelerators.

19. The apparatus according to any one of claims 13 to 18, characterized in that, The execution module is further configured to: Place the index information of the delete operation among the N database operations into the first queue; The index information of the reverse operation of the insert operation among the N database operations is placed in the second queue.

20. The apparatus according to claim 19, characterized in that, The execution module is also used for: Before submitting the transaction, delete operations in the first queue are performed in parallel according to the M groupings.

21. The apparatus according to claim 19, characterized in that, The execution module is also used for: If an operation conflict is detected during the concurrent execution of multiple database operations in each of the M groups, the reverse operation of the insert operation is executed based on the index information of the reverse operation of the insert operation in the second queue, thereby completing the transaction rollback.

22. The apparatus according to claim 21, characterized in that, The execution module is also used for: After the transaction is rolled back, the N database operations are executed sequentially.

23. The apparatus according to claim 13, characterized in that, The collision detection module is specifically used for: Obtain the hash value corresponding to each of the N database operations; Detect whether there are any hash values ​​with the same value corresponding to the N data operations; If the hash value of the first database operation is found to be equal to the hash value of the second database operation, it is determined that the first database operation and the second database operation are conflicting operations.

24. The apparatus according to claim 23, characterized in that, The grouping module is specifically used for: Based on the conflict detection results and the preset grouping rules, the N database operations are divided into M groups; The preset grouping rules include at least one of the following: The hash values ​​of multiple database operations within a single group are not equal; The number of database operations contained in a single group is less than or equal to the number of hardware accelerators; The execution time of database operations contained in a single group is within a specified range; All database operations contained in a single group are of the same type.

25. An electronic device, characterized in that, include: One or more processors; Memory; And one or more computer programs, wherein the one or more computer programs are stored on the memory, and when the computer programs are executed by the one or more processors, cause the electronic device to perform the method of any one of claims 1 to 12.

26. A computer storage medium, characterized in that, Includes computer instructions that, when executed on an electronic device, cause the electronic device to perform the method as described in any one of claims 1 to 12.

27. A computer program product, characterized in that, When the computer program product is run on a computer, it causes the computer to perform the method as described in any one of claims 1 to 12.

Images