Data processing system, data processing method and related equipment

Abstract

The invention provides a data processing system, the data processing system comprises a master node and at least one slave node, the master node comprises a host, a storage device and a heterogeneous processor; wherein the heterogeneous processor is used for acquiring a transaction log in a host and sending the transaction log to the storage device and the at least one slave node in parallel for persistent storage, and the heterogeneous processor sends the transaction log to the at least one slave node based on a remote direct memory access (RDMA) technology. Due to the fact that the master node and the slave nodes can utilize the heterogeneous processor and the RDMA technology to execute the persistent storage process of the transaction logs in parallel, the overall time delay of completing persistent storage of the transaction logs can be effectively shortened, the data processing system can execute more transactions in unit time length, and the performance of the data processing system is improved. In addition, the invention also provides a corresponding data processing method and related equipment.

Description

Technical Field

[0001] The present application relates to the field of database technology, and in particular to a data processing system, a data processing method and related equipment. Background Art

[0002] In data processing systems, write-ahead logging (WAL) is usually used to ensure the atomicity and persistence of data operations. Specifically, when data in the database needs to be modified (including adding, deleting, and replacing data, etc.), the data will not be modified directly in the database, but the data modification operation will be recorded in the WAL log and stored persistently before the data in the database is modified. In this way, during the process of modifying data, if an error occurs and the data modification fails, the data processing system can restore the data modification by replaying the persistently stored WAL log.

[0003] At present, the data processing system usually includes a master node and a slave node, and the master node may include multiple threads for executing transactions (such as transactions for modifying data), hereinafter referred to as transaction threads, and the transaction thread generates a corresponding transaction log (Xlog) in the process of executing the transaction. Usually, the master node will first write the transaction log generated by the transaction thread to the local storage device, and then send the transaction log remotely to the slave node for persistent storage. After the transaction log is persistently stored in both the master node and the slave node, the transaction thread continues to execute the next transaction.

[0004] However, the latency caused by the master node storing the transaction log locally and persistently in the slave node is high, which will increase the overall latency of the transaction thread executing the transaction, thereby affecting the performance of the data processing system. Summary of the invention

[0005] The present application provides a data processing system for improving the performance of the data processing system. In addition, the present application also provides a data processing method, a computer-readable storage medium, and a computer program product.

[0006] In a first aspect, the present application provides a data processing system, which includes a master node and at least one slave node, the master node includes a host, a storage device and a heterogeneous processor; wherein the heterogeneous processor is used to obtain transaction logs in the host and send the transaction logs in parallel to the storage device and at least one slave node for persistent storage, wherein the heterogeneous processor sends the transaction logs to at least one slave node based on remote direct memory access (RDMA) technology.

[0007] Since the heterogeneous processor sends the transaction log to the local storage device and the remote slave node in parallel, the master node and the slave node can execute the process of persistent storage of the transaction log in parallel, which can effectively shorten the overall delay of the transaction log in the master node and the slave node to complete the persistent storage, thereby shortening the delay of executing the transaction. In this way, the data processing system can execute more transactions per unit time, thereby improving the performance of the data processing system. In addition, the heterogeneous processor sends the transaction log to the slave node based on the RDMA technology, which can effectively shorten the transmission delay of the transaction log between the master node and the slave node compared to the method of sending the transaction log based on the TCP protocol, that is, it can shorten the overall delay of the transaction log in the slave node to complete the persistent storage, thereby further improving the performance of the data processing system. In addition, the operations of storing transaction logs locally and sending transaction logs to remote slave nodes are performed by heterogeneous processors, which makes it unnecessary for the host in the master node to perform the process of persistently storing transaction logs or sending transaction logs, thereby effectively reducing the storage resources (specifically the cache resources required) and computing resources consumed by the host in the process of executing transactions. That is, the host can execute more transactions simultaneously based on limited resource support, thereby further improving the performance of the data processing system.

[0008] In a possible implementation, the host includes multiple transaction threads and multiple private buffer areas, each private buffer area is used to store a transaction log generated by a transaction thread, and the multiple transaction threads are in a dormant state after writing the transaction log to the private buffer area; when the heterogeneous processor obtains the transaction log in the host, it is specifically used to: write the transaction logs in the multiple private buffer areas into a target buffer area in the heterogeneous processor; the heterogeneous processor is also used to wake up the multiple transaction threads after completing the persistent storage of the transaction log.

[0009] In one possible implementation, the heterogeneous processor is also used to: record the log sequence numbers LSN corresponding to multiple transaction threads before sending the transaction log in parallel to a storage device and at least one slave node for persistent storage; when waking up multiple transaction threads, the heterogeneous processor is specifically used to: determine the target LSN, which is the maximum LSN of the transaction log that completes the persistent storage; compare the size between the LSN corresponding to the first transaction thread and the target LSN, the first transaction thread being one of the multiple transaction threads; when the LSN corresponding to the first transaction thread is not greater than the target LSN, wake up the first transaction thread.

[0010] In one possible implementation, when a heterogeneous processor writes transaction logs in multiple private buffer areas into a target buffer area in the heterogeneous processor, it is specifically used to: batch write multiple transaction logs in each private buffer area into the target buffer area in the heterogeneous processor to obtain multiple aggregate logs, each aggregate log including multiple transaction logs in a private buffer area.

[0011] In a possible implementation, when the data processing system meets the log playback condition, the master node or at least one slave node decomposes the aggregate log into multiple transaction logs, and plays back the multiple transaction logs decomposed from the aggregate log.

[0012] In one possible implementation, the heterogeneous processor is further used to: obtain a configuration result for the size of the aggregate log; then, when the heterogeneous processor writes multiple transaction logs in each private buffer area into a target buffer area in the heterogeneous processor in batches, it is specifically used to: according to the configuration result of the size of the aggregate log, batch write the multiple transaction logs in each private buffer area into the target buffer area in the heterogeneous processor to obtain multiple aggregate logs whose sizes meet the configuration result.

[0013] In a possible implementation, the heterogeneous processor includes a smart network card or a data processing unit DPU.

[0014] In a second aspect, the present application provides a data processing method, which is applied to a data processing system, the data processing system includes a master node and at least one slave node, the master node includes a host, a storage device and a heterogeneous processor, the method includes: the heterogeneous processor obtains the transaction log in the host; the heterogeneous processor sends the transaction log in parallel to the storage device and at least one slave node for persistent storage, wherein the heterogeneous processor sends the transaction log to at least one slave node based on remote direct memory access RDMA technology.

[0015] In a possible implementation, a host includes multiple transaction threads and multiple private buffer areas, each private buffer area is used to store a transaction log generated by a transaction thread, and the multiple transaction threads are in a dormant state after writing the transaction log to the private buffer area; a heterogeneous processor obtains the transaction log in the host, including: the heterogeneous processor writes the transaction logs in the multiple private buffer areas into a target buffer area in the heterogeneous processor; the method also includes: after completing the persistent storage of the transaction log, the heterogeneous processor wakes up the multiple transaction threads.

[0016] In a possible implementation, the method also includes: before sending the transaction log in parallel to a storage device and at least one slave node for persistent storage, the heterogeneous processor records the log sequence numbers LSN corresponding to multiple transaction threads respectively; the heterogeneous processor wakes up the multiple transaction threads, including: the heterogeneous processor determines the target LSN, the target LSN is the maximum LSN of the transaction log that completes the persistent storage; the heterogeneous processor compares the size between the LSN corresponding to the first transaction thread and the target LSN, the first transaction thread is one of the multiple transaction threads; when the LSN corresponding to the first transaction thread is not greater than the target LSN, the heterogeneous processor wakes up the first transaction thread.

[0017] In one possible implementation, a heterogeneous processor writes transaction logs in multiple private buffer areas into a target buffer area in the heterogeneous processor, including: the heterogeneous processor writes multiple transaction logs in each private buffer area in batches to the target buffer area in the heterogeneous processor to obtain multiple aggregate logs, each aggregate log including multiple transaction logs in a private buffer area.

[0018] In a possible implementation, when the method meets the log playback condition, the master node or at least one slave node decomposes the aggregate log into multiple transaction logs, and plays back the multiple transaction logs decomposed from the aggregate log.

[0019] In a possible implementation, the method further includes: the heterogeneous processor obtains a configuration result for the size of the aggregate log; then, the heterogeneous processor writes multiple transaction logs in each private buffer area in batches into a target buffer area in the heterogeneous processor, including: the heterogeneous processor writes multiple transaction logs in each private buffer area in batches into the target buffer area in the heterogeneous processor according to the configuration result of the size of the aggregate log, to obtain multiple aggregate logs whose sizes meet the configuration result.

[0020] In a possible implementation, the heterogeneous processor includes a smart network card or a data processing unit DPU.

[0021] The data processing method provided in the second aspect corresponds to the data processing system provided in the first aspect. Therefore, the technical effects of the data processing method in the second aspect and any possible implementation method of the second aspect can refer to the technical effects of the first aspect and the corresponding implementation method of the first aspect, and will not be repeated here.

[0022] In a third aspect, the present application provides a computing device, the computing device comprising a processor and a memory; the memory is used to store instructions, and when the computing device is running, the processor executes the instructions stored in the memory, so that the computing device performs the data processing method in the above-mentioned second aspect or any possible implementation of the second aspect. It should be noted that the memory can be integrated into the processor or can be independent of the processor. The computing device may also include a bus. The processor is connected to the memory via the bus. The memory may include a readable memory and a random access memory.

[0023] In a fourth aspect, the present application provides a computer-readable storage medium, wherein instructions are stored in the computer-readable storage medium, which, when executed on a computing device, enables the computing device to execute the method described in the second aspect or any one of the implementations of the second aspect.

[0024] In a fifth aspect, the present application provides a computer program product comprising instructions, which, when executed on a computing device, enables the computing device to execute the method described in the second aspect or any one of the implementations of the second aspect.

[0025] Based on the implementations provided in the above aspects, this application can also be further combined to provide more implementations. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0027] Figure 1 A schematic diagram of the structure of an exemplary data processing system provided by the present application;

[0028] Figure 2 A flowchart of a data processing method provided in this application;

[0029] Figure 3 A schematic diagram of playing back logs according to log types provided for this application;

[0030] Figure 4 A schematic diagram of the structure of a computing device provided in this application. DETAILED DESCRIPTION

[0031] The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances. This is just a way of distinguishing objects with the same attributes when describing the embodiments of this application.

[0032] See also Figure 1 , which shows a schematic diagram of the structure of a data processing system. Figure 1 As shown, the data processing system 10 includes a master node and at least one slave node, and the master node and the slave node are connected via a network. For ease of understanding and description, Figure 1 The master node 100 and the slave node 200 are taken as an example for explanation. In actual application, the number of slave nodes may also be multiple.

[0033] The master node 100 includes a host 101, a storage device 102, and a heterogeneous processor 103, and the host 101, the storage device 102, and the heterogeneous processor 103 can be connected via a high-speed Internet network.

[0034] The host 101 may be a common computing device, such as a server, etc. The host 101 may include multiple transaction threads, such as Figure 1 Transaction thread A and transaction thread B in the data processing system 10. A transaction is a program execution unit that accesses and possibly updates data items in the data processing system 10. A transaction thread is a thread used to execute a transaction. During the execution of a transaction, the transaction thread generates a corresponding transaction log to record the modifications made to the persistently stored data in the data processing system 10. The host 101 also includes a buffer area, which can be divided into multiple private buffer areas, such as Figure 1 The private buffer area 1011 and the private buffer area 1012 in each transaction thread correspond to a private buffer area for storing the transaction log generated by the transaction thread. Specifically, the private buffer area 1011 can be used to store the transaction log generated by the transaction thread A, and the private buffer area 1012 can be used to store the transaction log generated by the transaction thread B.

[0035] The storage device 102 may be any device that provides persistent storage capabilities, such as a solid state disk (SSD), a shingled magnetic recording (SMR) hard disk, an Intel Optane persistent memory (PMEM), a remote shared storage device, a disk array, a cloud disk, and other devices.

[0036] The heterogeneous processor 103 may be a device with data processing capability, such as a smart network interface card (SmartNIC), a data processing unit (DPU), or other types of devices. The heterogeneous processor 103 may be configured with a buffer area 1031 for caching data, such as caching transaction logs.

[0037] The slave node 200 serves as a disaster recovery node for the master node 100 and may have the same or similar hardware configuration as the master node 100 . Figure 1 In the example, a slave node 200 including a host 201, a storage device 202 and a heterogeneous processor 203 is taken as an example for exemplary description.

[0038] The data processing system 10 can provide a client to the outside. The client, for example, can be a web browser, or can be an application (application) running on a user terminal, for interacting with the user, such as receiving a statement input by the user for modifying data, which statement can be, for example, a structured query language (SQL) statement. The host 101 can create one or more transactions according to the statement input by the user, and assign a corresponding transaction thread to the transaction for execution. Assume that the host 101 creates transaction 1 and transaction 2 according to the statement input by the user, and assigns transaction thread A to execute transaction 1, and assigns transaction thread B to execute transaction 2.

[0039] Transaction thread A and transaction thread B will generate one or more transaction logs during the execution of transactions. For example, when transaction thread A accesses an index (such as a Btree index), a transaction log will be generated. The transaction log is used to record operations such as adding, deleting, and replacing the index. Similarly, when transaction thread A accesses a database (DB) or a heap (heap), a corresponding transaction log will also be generated. The transaction log generated by each transaction thread will be written into the private buffer area corresponding to the transaction thread for cache.

[0040] Generally, the transaction log in the private buffer area needs to be persistently stored to ensure the reliability of storage in the data processing system 10. At this time, if the host 101 first writes the transaction log in the private buffer area to the storage device 102, and then transmits the transaction log to the slave node 200 through the Transmission Control Protocol (TCP) for persistent storage, it will take a long time to persistently store the transaction log, which will cause the overall latency of the transaction thread to execute the transaction to be higher, thereby affecting the performance of the data processing system 10, such as the number of transactions that the data processing system 10 can execute in a unit time is small.

[0041] Based on this, in the data processing system 10 provided by the present application, a heterogeneous processor 103 is used to reduce the latency of persistent storage of transaction logs in the master node 100 and the slave node 200, thereby improving the performance of the data processing system 10. Specifically, the heterogeneous processor 103 obtains the transaction log in the host 101, such as accessing the private buffer area 1011 and the private buffer area 1012 in the host 101 to obtain the transaction log, etc. Then, the heterogeneous processor 103 sends the obtained transaction log in parallel to the local storage device 102 and the remote slave node 200 for persistent storage, wherein the heterogeneous processor 103 can specifically write the transaction log remotely to the memory in the host 201 based on the remote direct memory access (RDMA) technology, so that the node 200 can persistently store the transaction log in the memory to the storage device 202, etc. RDMA technology is a technology that was created to solve the data processing delay between the sender and the receiver during network transmission. It mainly transmits the data in the sender directly to the storage area of ​​the receiver through the network, eliminating multiple data copy operations during the transmission process. It does not require the intervention of the operating systems of both parties, and reduces the host load at the sender and the receiver.

[0042] Since the heterogeneous processor 103 sends the transaction log to the local storage device 102 and the remote slave node 200 in parallel, the master node 100 and the slave node 200 can execute the process of persistently storing the transaction log in parallel, which can effectively shorten the overall delay of completing the persistent storage of the transaction log in the master node 100 and the slave node 200, thereby shortening the delay of executing the transaction. In this way, the data processing system 10 can execute more transactions in a unit time, thereby improving the performance of the data processing system 10.

[0043] In addition, the heterogeneous processor 103 sends the transaction log to the slave node 200 based on the RDMA technology. Compared with the method of sending the transaction log based on the TCP protocol, this can effectively shorten the transmission delay of the transaction log between the master node 100 and the slave node 200, that is, it can shorten the overall delay of completing the persistent storage of the transaction log in the slave node 200, thereby further improving the performance of the data processing system 10.

[0044] In addition, the heterogeneous processor 103 performs the operations of locally storing transaction logs and sending transaction logs to the remote slave node 200, which makes it unnecessary for the host 101 in the master node 100 to perform the process of persistently storing transaction logs or sending transaction logs, thereby effectively reducing the storage resources (specifically the cache resources required to be consumed) and computing resources consumed by the host 101 in the process of executing transactions, that is, the host 101 can execute more transactions simultaneously based on limited resource support, thereby further improving the performance of the data processing system 10.

[0045] It is worth noting that the above Figure 1 The data processing system 10 shown is only used as an exemplary illustration and is not intended to limit the implementation of the data processing system 10 to Figure 1 Example shown. For example, in a Quorum scenario or a distributed consensus framework (DCF) scenario, the number of slave nodes in the data processing system 10 can be 2 or more. Alternatively, the data processing system 10 can also include other types of devices to support the data processing system 10 with more functions. In addition, the host and storage devices included in each node in the data processing system 10 can adopt an integrated storage and computing architecture, or a storage and computing separation architecture. The specific architecture of the data processing system is not limited in this application.

[0046] For ease of understanding, the process of data processing system 10 processing transaction logs is described in detail below with reference to the accompanying drawings.

[0047] See also Figure 2 , shows a flow chart of a data processing method provided by an embodiment of the present application, which can be applied to Figure 1 The data processing system 10 shown in FIG. 1 may be used in other applicable data processing systems. Figure 1 The data processing system 10 shown in FIG. Figure 2 As shown, the method may specifically include:

[0048] S201 : the heterogeneous processor 103 obtains a transaction log in the host 101 included in the master node 100 .

[0049] In this embodiment, the transaction log in the host 101 may be generated by the transaction thread A and the transaction thread B in the host 101 during the execution of the transaction.

[0050] In specific implementation, the host 101 can first create a transaction thread A and a transaction thread B according to the statements provided by the user. For example, the user can input an SQL statement (or other types of statements) through the client provided by the data processing system 10, and the client sends the SQL statement to the host 101. The host 101 can generate a corresponding execution plan tree based on the received SQL statement, and the execution plan tree is used to describe the operations that the host 101 needs to perform and the relationship between different operations. Then, the host 101 can generate corresponding multiple transactions based on the execution plan tree, and create a transaction thread A and a transaction thread B for the multiple transactions, wherein the transaction thread A and the transaction thread B are used to execute the multiple transactions. For ease of understanding and description, this embodiment takes the example of transaction thread A executing transaction 1 in the multiple transactions and transaction thread B executing transaction 2 in the multiple transactions as an example for illustrative explanation.

[0051] The host 101 also allocates the following to the transaction thread A and the transaction thread B: Figure 1 Thus, during the execution of transaction 1, transaction thread A will generate one or more transaction logs, which can be used to record the operations performed by transaction thread A when accessing the index, database or heap in the data processing system 10, and write the generated transaction logs into the private buffer area 1011 for storage. Similarly, transaction thread B will write the transaction logs generated during the execution of transaction 2 into the private buffer area 1012 for storage.

[0052] It should be noted that the transaction thread described in this embodiment executes the transaction, which can be specifically executed by the computing resources (such as processor cores, etc.) allocated to the transaction thread in the host 101 according to the program code corresponding to the transaction thread. To simplify the description, in this embodiment, various operations performed by the computing resources according to the program code corresponding to the thread are simply described as the thread executing the operation.

[0053] Normally, after transaction thread A and transaction thread B use the insert thread to write all transaction logs in the private buffer area into the separately configured buffer area, transaction thread A and transaction thread B will start to commit the transactions they have executed and enter a dormant state (such as a semdown state) to wait for the transaction to be committed. Transaction thread A and transaction thread B will be awakened after all transaction logs in the buffer area have been persistently stored.

[0054] Then, the heterogeneous processor 103 may actively write the transaction logs in the private buffer area 1011 and the private buffer area 1012 into the buffer area 1031 in the heterogeneous processor 103 .

[0055] In specific implementation, the heterogeneous processor 103 may periodically scan each private buffer area to detect whether there is a transaction log stored in each private buffer area. If not, the heterogeneous processor 103 may not process it; if so, the heterogeneous processor 103 may execute the process of writing the transaction log in the private buffer area into the buffer area 1031.

[0056] As a first implementation example, when it is detected that transaction logs are stored in both the private buffer area 1011 and the private buffer area 1012 , the heterogeneous processor 103 may write multiple transaction logs in each private buffer area into the buffer area 1031 one by one.

[0057] As a second implementation example, when it is detected that both the private buffer area 1011 and the private buffer area 1012 store transaction logs, for each private buffer area, the heterogeneous processor 103 may first aggregate multiple transaction logs in the private buffer area, and write the aggregated multiple transaction logs in batches into the buffer area 1031 to form an aggregate log (or a transaction block log), which includes the contents of multiple transaction logs in the same private buffer area and may record information such as the size of each transaction log. In this way, the buffer area 1031 may store multiple aggregate logs corresponding to multiple private buffer areas.

[0058] For example, assuming that transaction logs 1 to 4 are stored in the private buffer area 1011 and transaction logs 5 to 7 are stored in the private buffer area 1012, the heterogeneous processor 103 may first aggregate the four transaction logs in the private buffer area 1011 into an aggregate log 1, and write the aggregate log 1 into the buffer area 1031. Then, the heterogeneous processor 103 aggregates the three transaction logs in the private buffer area 1012 into an aggregate log 2, and writes the aggregate log 2 into the buffer area 1031.

[0059] In this way, by batch processing the multiple transaction logs in the private buffer area, the efficiency of the heterogeneous processor 103 writing the multiple transaction logs into the buffer area 1031 can be improved.

[0060] In actual application, the number of transaction logs stored in some private buffer areas is large. At this time, the heterogeneous processor 103 can aggregate multiple transaction logs in the private buffer area into multiple aggregate logs, such as aggregating 10 transaction logs into 2 aggregate logs, each aggregate log includes the contents of 5 transaction logs, etc.

[0061] The size of the aggregate log can be pre-configured by the administrator, such as the administrator can pre-configure the size of the aggregate log to be no more than 2KB (kilobytes). Alternatively, the size of the aggregate log can be automatically configured by the heterogeneous processor 103. For example, the heterogeneous processor 103 can perform automatic configuration according to the number of data write channels supported by the storage device 102. In this way, the heterogeneous processor 103 can batch write multiple transaction logs in each private buffer area into the buffer area 1031 according to the configuration result for the size of the aggregate log, and obtain multiple aggregate logs whose sizes meet the configuration result.

[0062] At this time, the transaction log is still recorded in the buffer area and has not yet completed persistent storage. Therefore, the transaction thread that generates the transaction log will still be in a dormant state.

[0063] S202: The heterogeneous processor 103 sends the acquired transaction logs in parallel to the storage device 102 and the slave node 200 for persistent storage, wherein the heterogeneous processor 103 sends the transaction logs to the slave node 200 based on the RDMA technology.

[0064] In this embodiment, the transaction log in the host 101 is persistently stored in the master node 100 and the slave node 200. In order to reduce the latency of persistently storing the transaction log, the heterogeneous processor 103 can send the transaction log to the slave node 200 for persistent storage while storing the transaction log locally, that is, sending the transaction log to the storage device 102 and the slave node 200 in parallel.

[0065] When the heterogeneous processor 102 stores transaction logs locally, it may specifically create one or more write processes, and use the one or more write threads to send the transaction logs in the buffer area 1031 to the storage device 102 for persistent storage.

[0066] In addition, the heterogeneous processor 103 will also send the transaction log remotely to the memory in the host 201 based on the RDMA technology (specifically, it can be the RDMA unilateral technology) to speed up the transmission of the transaction log from the master node 100 to the slave node 200.

[0067] In this way, the master node 100 and the slave node 200 can execute the process of persistent storage of transaction logs in parallel, so as to effectively shorten the overall delay of completing persistent storage of transaction logs in the master node 100 and the slave node 200, thereby shortening the delay of executing transactions, and further improving the performance of the data processing system 10. In addition, the heterogeneous processor 103 performs the operations of storing transaction logs locally and sending transaction logs to the remote slave node 200, so that the host 101 in the master node 100 does not need to execute the process of persistent storage of transaction logs, nor does it need to execute the process of sending transaction logs, thereby effectively reducing the storage resources (specifically the cache resources required to be consumed) and computing resources consumed by the host 101 in the process of executing transactions, that is, the host 101 can execute more transactions simultaneously based on limited resource support, so as to further improve the performance of the data processing system 10.

[0068] It can be understood that after the transaction logs generated by the transaction thread A and the transaction thread B executing the transaction are persistently stored in the storage device 102 and the slave node 200, the transaction is committed and the transaction ends. Therefore, the heterogeneous processor 103 can wake up the transaction thread A and the transaction thread B in the dormant state, so that the transaction thread A and the transaction thread B continue to execute the next transaction.

[0069] Illustratively, this embodiment provides the following implementation examples of waking up transaction threads A and B.

[0070] In the first implementation example, the heterogeneous processor 103 may record the LSN corresponding to each transaction thread before persistently storing the transaction log, where the LSN corresponding to each transaction thread is the maximum value of the LSNs corresponding to multiple transaction logs generated by the transaction thread.

[0071] In specific implementation, the heterogeneous processor 103 can determine the maximum value among multiple log sequence numbers (LSN) corresponding to multiple transaction logs stored in the private buffer area 1011, and use the maximum value as the LSN corresponding to the transaction thread A and record it. In a similar manner, the heterogeneous processor 103 can record the LSN corresponding to the transaction thread B.

[0072] When the heterogeneous processor 103 sends multiple transaction logs in the buffer area 1031 to the storage device 102 and the slave node 200 in parallel, the heterogeneous processor 103 can record the target LSN, which is the maximum LSN of the transaction logs that have completed persistent storage. A dedicated thread can be created in the heterogeneous processor 103, and the dedicated thread can continuously monitor the target LSN of the transaction logs that have completed persistent storage, and compare the target LSN with the LSNs corresponding to each transaction thread recorded in advance. When the target LSN of the written transaction log is greater than the LSN corresponding to transaction thread A, the heterogeneous processor 103 wakes up transaction thread A, otherwise, transaction thread A continues to be in a dormant state. When the target LSN of the written transaction log is greater than the LSN corresponding to transaction thread B, the heterogeneous processor 103 wakes up transaction thread B, otherwise, transaction thread B continues to be in a dormant state.

[0073] In this way, by comparing the LSN corresponding to the transaction thread with the target LSN of the transaction log that has completed persistent storage, the timing of waking up the transaction thread can be accurately determined, so that there is no need to wait until all transaction logs have completed persistent storage before waking up each transaction thread. In this way, the sleep time of some transaction threads can be effectively shortened, and some transaction threads can be prevented from being in a long-term sleep state and wasting the computing resources of the host 101, thereby improving the overall efficiency of the transaction thread in executing multiple transactions, thereby improving the overall performance of the data processing system 10. In addition, the heterogeneous processor 103 actively wakes up the transaction thread A and the transaction thread B in the host 101, and the host 101 does not need to consume computing resources to determine whether to change the status of each transaction thread. In this way, the resource consumption on the host side can be further saved, thereby helping to further improve the overall performance of the data processing system 10.

[0074] Actual tests based on the database performance test standard TPC-C can determine that the host 101 can save at least 20% of computing resources, and the throughput of the data processing system 10 can be improved by more than 30%.

[0075] In actual application, the dedicated thread can also use the value of the target LSN to update the global variable LSN in the host 101, so that the host 101 can determine the LSN corresponding to the transaction log that has completed persistent storage in the storage device 102 and the slave node 200 according to the value of the global variable LSN.

[0076] In addition to the above implementation examples, the heterogeneous processor 103 may also wake up each transaction thread based on other implementations. For example, after determining that all transaction logs in the buffer area 1031 have completed persistent storage, the heterogeneous processor 103 wakes up all transaction threads uniformly, etc. This embodiment does not limit this.

[0077] In a further possible implementation, when the data processing system 10 meets the log playback conditions, such as when the master node 100 restarts after a failure, the host 101 can access the persistently stored transaction log in the storage device 102. For example, the host 101 can determine the storage address of the transaction log in the storage device 102 based on the metadata recorded in the multiple second storage areas in the storage device 102, so that the host 101 can read the transaction logs in the multiple first storage areas according to the storage address. Since the amount of metadata recorded in the second storage area is small, the delay caused by the host 101 accessing the metadata to determine the storage location of the transaction log is very small and can be basically ignored (relative to the delay caused by the host 101 accessing the transaction log).

[0078] Then, the host 101 can implement data recovery by replaying the transaction log in the storage device 102 .

[0079] When the heterogeneous processor 103 writes a transaction log to the storage device 102 in the form of an aggregate log, the host 101 can first parse the type of the log after accessing the log in the storage device 102, such as determining the type of the log by parsing the log header. When it is determined that the log is an aggregate log, the host 101 can first decompose the aggregate log into multiple transaction logs and play back each transaction log one by one.

[0080] Specifically, Figure 3 As shown, when the operation content recorded in the transaction log is a page-related operation content, the host 101 can assign the operation content to the queue corresponding to the playback thread, and use the playback thread to sequentially execute the operations for one or more pages in the queue. When the operation content recorded in the transaction log is a table-related operation content, the host 101 can assign the operation content to the queue corresponding to the playback thread according to the table granularity, and use the playback thread to sequentially execute the operations for the pages corresponding to each data in the table in the queue.

[0081] When the transaction log is a log of modifying a table, such as a data definition language (DDL) log, the host 101 can distribute the operation content in the transaction log to the queues corresponding to all playback threads, so that each playback thread can replay the transaction log based on the operations of modifying the table recorded in the transaction log.

[0082] When the operation content recorded in the transaction log indicates updating the transaction status, the host 101 may allocate the operation content to a transaction-related queue, so as to update the transaction status by replaying the transaction log.

[0083] In this embodiment, since the specific implementation process of the host 101 playing back the transaction log has relevant applications in actual application scenarios, it will not be elaborated here.

[0084] like Figure 3 As shown, when it is determined through analysis that the type of the log is a single transaction log, the host 101 can allocate the operation content recorded in the transaction log to a corresponding queue to play back the operation content recorded in the transaction log.

[0085] The process of replaying the transaction log is performed by the host 101. In other possible implementations, the data processing system 10 may include multiple nodes, some of which serve as disaster recovery nodes of the master node 100. At this time, when the data processing system 10 meets the log replay condition, the disaster recovery node in the data processing system 10 may also realize data recovery by replaying the transaction log.

[0086] Taking the data processing system 10 also including the node 200 as an example, when the heterogeneous processor 103 persistently stores the transaction log in the buffer area 1031 to the storage device 102, it will also remotely send the transaction log to the node 200 for persistent storage. For example, the heterogeneous processor 103 can remotely send the transaction log to the memory in the node 200 based on the RDMA technology, and then the node 200 will persistently store the transaction log in the memory. In this way, when the data processing system 10 meets the log playback conditions, such as the failure of the master node 100, the node 200 can perform log playback based on the transaction log persistently stored by itself, so that the data stored in the node 200 is restored to the data stored before the failure of the master node 100, thereby realizing data recovery of the data processing system 10.

[0087] It should be noted that the above Figure 2 The data processing method shown is only an exemplary description. In actual application, the specific implementation of the data processing system 10 for persistent storage of transaction logs is not limited to the above example, and can also be implemented in other ways. As some examples, the following three points are listed in this embodiment to illustrate other possible implementations of persistent storage of transaction logs.

[0088] 1. In other possible embodiments, multiple different users can input SQL statements (or other types of statements) to the data processing system 10 and generate multiple different transactions in the data processing system 10. In this case, the transaction thread A and the transaction thread B can be used to process transactions corresponding to different SQL statements and generate transaction logs. In addition, the number of transaction threads included in the data processing system 10 can be any number, not limited to the transaction thread A and the transaction thread B mentioned above.

[0089] 2. In other possible embodiments, in the process of a transaction thread writing multiple transaction logs to a private buffer area, the heterogeneous processor 103 can continuously and actively pull the transaction logs in the private buffer area to the buffer area 1031. In specific implementation, the heterogeneous processor 103 can continuously monitor whether the private buffer area corresponding to each transaction thread stores a transaction log. When a transaction log is stored, the heterogeneous processor 103 directly writes all the transaction logs currently stored in the private buffer area to the buffer area 1031. At this time, the transaction thread may still be in the stage of executing transactions and generating transaction logs, so the transaction thread will continue to write new transaction logs (still belonging to the currently executed transaction) to its corresponding private buffer area. Accordingly, the heterogeneous processor 103 can detect that a new transaction log is stored in the private buffer area in the next monitoring cycle, and continue to actively write the new transaction log to the buffer area 1031.

[0090] 3. In other possible embodiments, after the heterogeneous processor 103 writes all the transaction logs in the private buffer area to the buffer area 1031, it is not necessary to record the LSN corresponding to each transaction thread. In addition, after the heterogeneous processor 103 sends all the transaction logs in the buffer area 1031 to the storage device 102 for persistent storage, since all the transaction logs generated by the transaction thread A and the transaction thread B have completed persistent storage, the heterogeneous processor 103 can directly wake up the transaction thread A and the transaction thread B, and it is not necessary to determine whether to wake up the transaction thread by comparing the LSN corresponding to the transaction thread with the LSN of the transaction log that has completed persistent storage.

[0091] Combined with the above Figures 1 to 3 , describes in detail the data processing method provided by this application, and will be combined with Figure 4 , describing a computing device provided according to the present application, which can be used to indicate the method executed by the above-mentioned heterogeneous processor 103.

[0092] Figure 4 This is a schematic diagram of the structure of a computing device provided in this application. Figure 4 As shown, computing device 400 includes processor 401, memory 402, communication interface 403 and bus 404. Processor 401, memory 402 and communication interface 403 communicate through bus 404. Bus 404 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4The communication interface 403 is used for external communication, for example, receiving jobs (and target data) submitted by users through terminals or clients.

[0093] It should be understood that in the embodiment of the present application, the processor 401 may be a CPU, or may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete device components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0094] The memory 402 may include a read-only memory and a random access memory, and provide instructions and data to the processor 401. The memory 402 may also include a non-volatile random access memory. For example, the memory 402 may also store information about the device type.

[0095] The memory 402 may be a volatile memory or a nonvolatile memory, or may include both volatile and nonvolatile memories. Among them, the nonvolatile memory may be a read-only memory (ROM), a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0096] The memory 402 stores executable codes, and the processor 401 executes the executable codes to execute the method executed by the aforementioned heterogeneous processor 103 .

[0097] Specifically, in implementing Figure 2 In the illustrated embodiment, the software or program code required to execute the functions of the heterogeneous processor 103 in the illustrated embodiment 2 is stored in the memory 402, and the processor 401 is used to execute the instructions in the memory 402 to implement the method executed by the heterogeneous processor 103.

[0098] It should be understood that the computing device 400 according to the embodiment of the present application may correspond to Figure 2 The heterogeneous processor 103 in the method shown, and the above and other operations and / or functions implemented by the computing device 400 are respectively to implement Figure 2 For the sake of brevity, the corresponding processes of each method in are not repeated here.

[0099] In addition, an embodiment of the present application further provides a computer-readable storage medium, which stores instructions. When the computer-readable storage medium is executed on a computing device, the computing device executes the method executed by the heterogeneous processor 103 in the above embodiment.

[0100] In addition, the embodiment of the present application further provides a computer program product, when the computer program product is executed by a computing device, the one or more computing devices execute any of the aforementioned data processing methods. The computer program product may be a software installation package, and when any of the aforementioned data processing methods is required, the computer program product may be downloaded and executed on a computer.

[0101] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0102] The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances. This is just a way of distinguishing objects with the same attributes when describing the embodiments of this application.

[0103] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0104] The above embodiments can be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded or executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium. The semiconductor medium can be a solid state drive (SSD).

[0105] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A data processing system, characterized in that: The data processing system includes a master node and at least one slave node, wherein the master node includes a host, a storage device, and a heterogeneous processor; The heterogeneous processor is used to obtain the transaction log in the host and send the transaction log in parallel to the storage device and the at least one slave node for persistent storage, wherein the heterogeneous processor sends the transaction log to the at least one slave node based on remote direct memory access (RDMA) technology.

2. The data processing system according to claim 1, characterized in that The host comprises a plurality of transaction threads and a plurality of private buffer areas, each private buffer area is used to store a transaction log generated by a transaction thread, and the plurality of transaction threads are in a dormant state after writing the transaction log into the private buffer area; When the heterogeneous processor obtains the transaction log in the host, it is specifically used to: Writing the transaction logs in the plurality of private buffer areas into a target buffer area in the heterogeneous processor; The heterogeneous processor is further used to wake up the multiple transaction threads after completing the persistent storage of the transaction log.

3. The data processing system according to claim 2, characterized in that: The heterogeneous processor is further configured to: Before sending the transaction logs in parallel to the storage device and the at least one slave node for persistent storage, recording the log sequence numbers LSNs corresponding to the multiple transaction threads respectively; When waking up the multiple transaction threads, the heterogeneous processor is specifically used to: Determine a target LSN, where the target LSN is the maximum LSN of the transaction log that completes persistent storage; Comparing the size of an LSN corresponding to a first transaction thread with the target LSN, where the first transaction thread is one of the multiple transaction threads; When the LSN corresponding to the first transaction thread is not greater than the target LSN, the first transaction thread is awakened.

4. The data processing system according to claim 2 or 3, characterized in that: When the heterogeneous processor writes the transaction logs in the multiple private buffer areas into the target buffer area in the heterogeneous processor, the heterogeneous processor is specifically configured to: The multiple transaction logs in each private buffer area are written in batches into a target buffer area in the heterogeneous processor to obtain multiple aggregate logs, each of which includes multiple transaction logs in one private buffer area.

5. The data processing system according to claim 4, characterized in that: When the data processing system meets the log playback condition, the master node or the at least one slave node decomposes the aggregate log into multiple transaction logs, and plays back the multiple transaction logs decomposed from the aggregate log.

6. The data processing system according to claim 4 or 5, characterized in that: The heterogeneous processor is further used for: Obtaining a configuration result for the size of the aggregate log; Then, when the heterogeneous processor writes the multiple transaction logs in each private buffer area into the target buffer area in the heterogeneous processor in batches, it is specifically used to: According to the configuration result of the size of the aggregate log, multiple transaction logs in each private buffer area are batch-written into the target buffer area in the heterogeneous processor to obtain multiple aggregate logs whose sizes meet the configuration result.

7. The data processing system according to any one of claims 1 to 6, characterized in that: The heterogeneous processor includes an intelligent network card or a data processing unit DPU.

8. A data processing method, characterized in that: The method is applied to a data processing system, the data processing system includes a master node and at least one slave node, the master node includes a host, a storage device and a heterogeneous processor, and the method includes: The heterogeneous processor obtains the transaction log in the host; The heterogeneous processor sends the transaction log in parallel to the storage device and the at least one slave node for persistent storage, wherein the heterogeneous processor sends the transaction log to the at least one slave node based on remote direct memory access (RDMA) technology.

9. The method according to claim 8, characterized in that The host comprises a plurality of transaction threads and a plurality of private buffer areas, each private buffer area is used to store a transaction log generated by a transaction thread, and the plurality of transaction threads are in a dormant state after writing the transaction log into the private buffer area; The heterogeneous processor obtains the transaction log in the host, including: The heterogeneous processor writes the transaction logs in the plurality of private buffer areas into a target buffer area in the heterogeneous processor; The method further includes: after completing persistent storage of the transaction log, the heterogeneous processor wakes up the multiple transaction threads.

10. The method according to claim 9, characterized in that The method further comprises: Before sending the transaction log in parallel to the storage device and the at least one slave node for persistent storage, the heterogeneous processor records the log sequence numbers LSN corresponding to the multiple transaction threads respectively; The heterogeneous processor wakes up the multiple transaction threads, including: The heterogeneous processor determines a target LSN, where the target LSN is the maximum LSN of the transaction log that completes persistent storage; The heterogeneous processor compares the size between the LSN corresponding to the first transaction thread and the target LSN, where the first transaction thread is one of the multiple transaction threads; When the LSN corresponding to the first transaction thread is not greater than the target LSN, the heterogeneous processor wakes up the first transaction thread.

11. The method according to claim 9 or 10, characterized in that: The heterogeneous processor writes the transaction logs in the plurality of private buffer areas into a target buffer area in the heterogeneous processor, comprising: The heterogeneous processor writes the multiple transaction logs in each private buffer area in batches into a target buffer area in the heterogeneous processor to obtain multiple aggregate logs, each of which includes multiple transaction logs in one private buffer area.

12. The method according to claim 11, characterized in that When the method satisfies the log playback condition, the master node or the at least one slave node decomposes the aggregate log into multiple transaction logs, and plays back the multiple transaction logs decomposed from the aggregate log.

13. The method according to claim 11 or 12, characterized in that: The method further comprises: The heterogeneous processor obtains a configuration result for the size of the aggregate log; Then, the heterogeneous processor writes the multiple transaction logs in each private buffer area into the target buffer area in the heterogeneous processor in batches, including: The heterogeneous processor writes multiple transaction logs in each private buffer area in batches into a target buffer area in the heterogeneous processor according to the configuration result of the size of the aggregate log, so as to obtain multiple aggregate logs whose sizes meet the configuration result.

14. The method according to any one of claims 8 to 13, characterized in that The heterogeneous processor includes an intelligent network card or a data processing unit DPU.

15. A computing device, characterized in that: The computing device comprises a processor and a memory, wherein the memory stores program instructions, and the processor runs the program instructions to execute the method according to any one of claims 8 to 14.

16. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, which, when executed on a computing device, enable the computing device to perform the method according to any one of claims 8 to 14.

Images