Data processing system, data processing method and related device

Abstract

Provided in the present application is a data processing system, comprising a first processor, a storage device, and a second processor, wherein the first processor comprises a plurality of threads, each thread corresponds to one data cache area, and the first processor is used for executing the plurality of threads, so as to respectively write a plurality of transaction logs into a plurality of data cache areas that respectively correspond to the plurality of threads; and the second processor comprises a cache, and is used for respectively acquiring the plurality of transaction logs from the plurality of data cache areas, storing the plurality of transaction logs in the cache, and persistently storing in the storage device the plurality of transaction logs from the cache. In this way, during the process of executing the plurality of threads by the first processor, it is unnecessary to mutually compete for calling an insert thread, and it is also unnecessary to execute the process of copying the transaction logs. Therefore, the first processor can finish processing transactions within a shorter time period, and thus the performance of the data processing system can be effectively improved. In addition, further provided in the present application are a corresponding data processing method and a related device.

Description

A data processing system, data processing method and related equipment

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on July 17, 2023, with application number 202310884165.9 and application name “A data processing system, data processing method and related equipment”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of database technology, and in particular to a data processing system, a data processing method, and related equipment. Background Art

[0003] In data processing systems, write-ahead logging (WAL) is often used to ensure the atomicity and durability of data operations. Specifically, when modifying data in a database (including adding, deleting, and replacing data), the data is not modified directly in the database. Instead, the data modification operation is first recorded in the WAL log and stored persistently before the data in the database is modified. This way, if an error occurs during the data modification process and the data modification fails, the data processing system can recover the data modification by replaying the persistently stored WAL log.

[0004] Currently, processors in data processing systems typically include multiple threads for executing transactions (such as transactions that modify data). Each thread generates a corresponding transaction log (Xlog) during the transaction execution process. Typically, each thread writes the generated transaction log to a data buffer allocated for that thread and competes with other threads to call an insert thread in the processor to write the transaction log in its own buffer area to the buffer area corresponding to the WAL log. This allows the thread to continue executing the next transaction after the transaction log in the buffer area is permanently stored.

[0005] However, the competition among multiple threads to call the inserted thread in the processor will increase the latency of the thread execution transaction and the occupation of the processor's computing resources, thereby affecting the performance of the data processing system.

[0006] Summary of the Invention

[0007] The present application provides a data processing system for improving the performance of the data processing system. In addition, the present application also provides a data processing method, a computing device, a computer-readable storage medium, and a computer program product.

[0008] In a first aspect, the present application provides a data processing system comprising a first processor, a storage device, and a second processor, wherein the first processor comprises multiple threads, and each thread corresponds to a data cache area, which can be used to store transaction logs, and the first processor is used to execute the multiple threads to write multiple transaction logs into multiple data cache areas corresponding to the multiple threads respectively, and the multiple transaction logs can be generated when the first processor executes multiple threads to process transactions, and the second processor comprises a cache, and the second processor is used to obtain multiple transaction logs from the multiple data cache areas respectively, and store the multiple transaction logs in the cache, and persistently store the multiple transaction logs from the cache to the storage device.

[0009] Because the second processor can proactively write transaction logs from multiple data caches into the cache used to generate WALs, the first processor eliminates the need for competing calls to insert threads within the first processor (and the first processor may also eliminate the need to create insert threads) when executing multiple threads. This avoids the long delays caused by competing calls to insert threads when the first processor executes multiple threads to process transactions. Accordingly, when executing multiple threads, the first processor can complete transactions in a shorter timeframe, effectively reducing the computing resources consumed by these threads within the first processor. This effectively improves the performance of the data processing system, such as increasing the efficiency of transaction execution and enabling the simultaneous execution of a greater number of transactions. Furthermore, the copying and persistent storage of transaction logs is performed by the second processor, eliminating the need for the first processor to copy transaction logs between different buffer areas or perform the process of persistently storing transaction logs. This effectively reduces the storage resources (specifically, cache resources) and computing resources consumed by the first processor during transaction execution, further improving the performance of the data processing system.

[0010] In one possible implementation, the second processor includes a Smart NIC or a data processing unit (DPU). Thus, the Smart NIC or DPU can offload the transaction log copying process from the first processor, thereby improving the performance of the data processing system.

[0011] In one possible implementation, when the second processor persistently stores multiple transaction logs from the cache to the storage device, it can specifically divide the multiple transaction logs into multiple log segments, each log segment including at least one transaction log, and determine multiple first storage areas in the storage device, and write the multiple log segments in parallel to the multiple first storage areas, each first storage area being used to store one log segment. In this way, by writing the multiple transaction logs in the cache to the storage device in parallel, the second processor can accelerate the persistent storage speed of the transaction logs, thereby reducing transaction latency and improving the efficiency of executing transactions.

[0012] In one possible embodiment, the second processor is further used to determine multiple second storage areas in the storage device, and write metadata corresponding to multiple log segments in parallel into the multiple second storage areas, each second storage area is used to store metadata corresponding to a log segment, and the metadata corresponding to each log segment is used to describe the storage location of the log segment in the storage device. In this way, when data recovery is required based on the transaction log persistently stored in the storage device, the data processing system can verify whether there are holes in the transaction log through the metadata recorded in the second storage area, that is, determine whether part of the transaction log is missing, so that when it is determined that the transaction log is not missing, the second processor can achieve data recovery by replaying the transaction logs recorded in the multiple first storage areas.

[0013] In one possible embodiment, after a first processor executes multiple threads to write multiple transaction logs to multiple data caches, the multiple threads are in a dormant state. The second processor then records the log sequence numbers (LSNs) corresponding to the multiple threads before writing the multiple log segments in parallel to the multiple first storage areas, and determines a target LSN, which is the maximum LSN of transaction logs that have been continuously persistently stored. The second processor can then compare the LSN corresponding to the first thread among the multiple threads with the target LSN, and wake up the first thread when the LSN corresponding to the first thread is not greater than the target LSN. In this way, the second processor can accurately determine the timing of waking up the thread by comparing the LSN corresponding to the thread with the LSN of the transaction logs that have been persistently stored, eliminating the need to wait until all transaction logs have been persistently stored before waking up each thread. This effectively shortens the dormancy time of some threads, avoids wasting the computing resources of the first processor by causing some threads to remain in a dormant state for a long time, and improves the overall efficiency of the first processor in executing multiple transactions using threads, thereby improving the overall performance of the data processing system. Furthermore, the second processor actively wakes up the first thread in the first processor, eliminating the need for the first processor to consume computing resources to determine whether to change the state of each thread. This can further save resource consumption of the first processor, thereby helping to further improve the overall performance of the data processing system.

[0014] In one possible implementation, the number of log segments matches the number of data write channels supported by the storage device. This allows for flexible configuration of the aggregate log size, and therefore the number of log segments, based on the underlying I / O performance of the storage device. This optimizes the performance of the second processor writing transaction logs to the storage device in parallel, thereby improving the overall performance of the data processing system.

[0015] In one possible implementation, when the second processor stores multiple transaction logs in the data cache area into the cache, it may specifically write the multiple transaction logs in each data cache area into the cache simultaneously, thereby obtaining multiple aggregated logs, each of which includes multiple transaction logs in a data cache area; wherein the multiple transaction logs in each data cache area are of the same type, or the multiple transaction logs in each data cache area are of different types. In this manner, by batch processing the multiple transaction logs in the data cache area, the efficiency of the second processor in writing the multiple transaction logs into the cache can be improved.

[0016] In one possible implementation, the second processor can further decompose the aggregate log into multiple transaction logs when log playback conditions are met, and replay the multiple transaction logs decomposed from the aggregate log. In this way, the second processor can perform data recovery on the data processing system by decomposing and replaying the transaction logs in the aggregate log.

[0017] In one possible implementation, the second processor can also obtain a configuration result for the size of the aggregate log. The size of the aggregate log can be configured by an administrator or automatically by the second processor. Then, when the second processor writes multiple transaction logs in each data cache area to the cache simultaneously, it can specifically write the multiple transaction logs in each data cache area to the cache simultaneously based on the configuration result of the size of the aggregate log, thereby obtaining multiple aggregate logs whose sizes meet the configuration result. In this way, the second processor can write multiple transaction logs in each data cache area to the cache simultaneously based on the size of the aggregate log, which not only improves the efficiency of writing transaction logs to the cache, but also enables flexible configuration of the size of the aggregate log.

[0018] In one possible implementation, the second processor can also send cached transaction logs to a disaster recovery node for backup based on Remote Direct Memory Access (RDMA) technology. The first processor, storage device, and second processor can constitute a first node in a data processing system. The data processing system also includes a disaster recovery node, which serves as a disaster recovery for the first node. In this way, if the first node fails, the data processing system can recover data using the transaction logs backed up on the disaster recovery node, thereby ensuring the reliability of the data processing system.

[0019] In a second aspect, the present application provides a data processing method, which is applied to a second processor in a data processing system. The data processing system also includes a first processor and a storage device. The first processor includes multiple threads, each thread corresponds to a data cache area, and the first processor is used to execute multiple threads to write multiple transaction logs into multiple data cache areas corresponding to the multiple threads respectively. The second processor includes a cache; the method includes: the second processor obtains multiple transaction logs from the multiple data cache areas respectively; the second processor stores the multiple transaction logs to the cache; the second processor persistently stores the multiple transaction logs from the cache to the storage device.

[0020] In a possible implementation, the second processor includes a smart network card or a data processing unit DPU.

[0021] In one possible implementation, the second processor persistently stores multiple transaction logs from a cache to a storage device, including: the second processor divides the multiple transaction logs to obtain multiple log segments, each log segment includes at least one transaction log; the second processor determines multiple first storage areas in the storage device; the second processor writes the multiple log segments into the multiple first storage areas in parallel, each first storage area is used to store one log segment.

[0022] In one possible embodiment, the method also includes: a second processor determines multiple second storage areas in the storage device; the second processor writes metadata corresponding to multiple log segments in parallel into the multiple second storage areas, each second storage area is used to store metadata corresponding to a log segment, and the metadata corresponding to each log segment is used to describe the storage location of the log segment in the storage device.

[0023] In one possible implementation, after a first processor executes multiple threads to write multiple transaction logs into multiple data cache areas, the multiple threads are in a dormant state. The method further includes: before a second processor writes the multiple log segments into multiple first storage areas in parallel, the second processor records the log sequence numbers LSN corresponding to the multiple threads respectively; the second processor determines a target LSN, which is the maximum LSN of transaction logs that have been continuously completed for persistent storage; the second processor compares the LSN corresponding to a first thread among the multiple threads with the target LSN; and when the LSN corresponding to the first thread is not greater than the target LSN, the second processor wakes up the first thread.

[0024] In a possible implementation, the number of the multiple log segments is consistent with the number of data writing channels supported by the storage device.

[0025] In one possible implementation, the second processor obtains multiple transaction logs from multiple data cache areas respectively, and the second processor stores the multiple transaction logs in the cache, including: the second processor writes the multiple transaction logs in each data cache area into the cache at the same time to obtain multiple aggregate logs, each aggregate log includes multiple transaction logs in one data cache area; wherein the multiple transaction logs in each data cache area are transaction logs of the same type, or the multiple transaction logs in each data cache area are transaction logs of different types.

[0026] In a possible implementation, the method further includes: when a log playback condition is met, the second processor decomposing the aggregate log into multiple transaction logs; and the second processor replaying the multiple transaction logs decomposed from the aggregate log.

[0027] In one possible implementation, the method further includes: the second processor obtains a configuration result for the size of the aggregate log; then, the second processor writes multiple transaction logs in each data cache area into the cache simultaneously, including: the second processor writes multiple transaction logs in each data cache area into the cache simultaneously according to the configuration result for the size of the aggregate log, to obtain multiple aggregate logs whose sizes meet the configuration result.

[0028] In a possible implementation, the method further includes: the second processor sending the transaction log in the cache to the disaster recovery node for backup based on remote direct memory access (RDMA) technology.

[0029] The data processing method provided in the second aspect corresponds to the data processing system provided in the first aspect. Therefore, the technical effects of the data processing method in the second aspect and any possible implementation of the second aspect can refer to the technical effects of the first aspect and the corresponding implementation of the first aspect, and will not be repeated here.

[0030] In a third aspect, the present application provides a computing device comprising a processor and a memory; the memory is used to store instructions, and when the computing device is running, the processor executes the instructions stored in the memory to cause the computing device to perform the method described in the second aspect or any possible implementation of the second aspect. It should be noted that the memory can be integrated into the processor or can be independent of the processor. The computing device may also include a bus. The processor is connected to the memory via the bus. The memory may include a readable memory and a random access memory.

[0031] In a fourth aspect, the present application provides a computer-readable storage medium, which stores instructions. When the computer-readable storage medium is run on a computing device, the computing device executes the method described in the second aspect or any implementation of the second aspect.

[0032] In a fifth aspect, the present application provides a computer program product comprising instructions, which, when executed on a computing device, enables the computing device to execute the method described in the second aspect or any one of the implementations of the second aspect.

[0033] Based on the implementation methods provided in the above aspects, this application can also be further combined to provide more implementation methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0035] FIG1 is a schematic diagram of the structure of an exemplary data processing system provided by the present application;

[0036] FIG2 is a flow chart of a data processing method provided by the present application;

[0037] FIG3 is a schematic diagram of the processor 103 writing transaction logs and metadata to the storage device 102 in parallel;

[0038] FIG4 is a schematic diagram of metadata and transaction logs recorded in the storage device 102;

[0039] FIG5 is a schematic diagram of log playback according to log type provided by the present application;

[0040] FIG6 is a schematic diagram of the structure of a computing device provided in this application. DETAILED DESCRIPTION

[0041] The terms "first," "second," and so on, in the specification and claims of this application and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate and are merely used to describe the manner in which objects with the same attributes are distinguished in the embodiments of this application.

[0042] 1 , which shows a schematic diagram of a data processing system. As shown in FIG1 , the data processing system 10 includes a processor 101, a storage device 102, and a processor 103. The processor 101, the storage device 102, and the processor 103 can be connected via a high-speed Internet network.

[0043] The processor 101 may include multiple threads, such as thread A and thread B in Figure 1, for executing one or more transactions. A transaction refers to a program execution unit that accesses and may update a data item in the data processing system 10. In the process of executing the multiple threads to process transactions, the processor 101 will generate a corresponding transaction log to record the modifications made to the persistently stored data in the data processing system 10. The processor 101 also includes a cache, and the storage area in the cache can be divided into multiple data cache areas, such as data cache area 1011 and data cache area 1012 in Figure 1. Each thread corresponds to a data cache area, which is used to store the transaction log generated when the processor 101 executes the thread. Specifically, the data cache area 1011 can be used to store the transaction log generated when the processor 101 executes thread A, and the data cache area 1012 can be used to store the transaction log generated when the processor 101 executes thread B.

[0044] The storage device 102 can be any device that provides persistent storage capabilities, such as a solid state disk (SSD), a shingled magnetic recording (SMR) hard disk, Intel Optane Persistent Memory (PMEM), a remote shared storage device, a disk array, a cloud disk, etc.

[0045] The processor 103 may be a device with data processing capabilities, such as a smart network interface card (SmartNIC), a data processing unit (DPU), or other types of devices.

[0046] The data processing system 10 can provide external clients. The client, for example, can be a web browser or an application running on a user terminal, used to interact with the user, such as receiving statements entered by the user for modifying data. Such statements can be, for example, Structured Query Language (SQL) statements. The processor 101 can create one or more transactions based on the statements entered by the user and assign corresponding threads to process the transactions. Assume that the processor 101 creates transactions 1 and 2 based on the statements entered by the user, and assigns thread A to process transaction 1 and thread B to process transaction 2.

[0047] When the processor 101 executes threads A and B to process transactions, one or more transaction logs are generated. For example, when executing thread A, the processor 101 may access an index (such as a Btree index), thereby generating a transaction log for recording operations such as adding, deleting, and replacing the index. Similarly, when executing thread A, the processor 101 may access a database (DB) or a heap, thereby also generating corresponding transaction logs. The transaction logs generated by the processor 101 when executing each thread to process transactions are written to the data buffer corresponding to the thread for cache.

[0048] Typically, the transaction logs in the data cache need to be persistently stored to ensure the reliability of storage in the data processing system 10. At this time, if the processor 101 persistently stores the transaction logs in each data cache, an insertion thread needs to be set in the processor 101, and after the processor 101 executes thread A and thread B to write the transaction logs to the data cache, they will compete with each other to call the insertion thread, so that the transaction logs written in the data cache are inserted into a buffer area (such as WAL buffer) separately configured in the processor 101 by executing the insertion thread, so that the processor 101 can subsequently write the transaction logs in the buffer area to the storage device 102 in the form of WAL. Typically, after the processor 101 executes the insertion thread to write all the transaction logs in the data cache area to the separately configured buffer area, thread A and thread B will enter a dormant state (such as a semdown state), and thread A and thread B will be awakened by the processor 101 after all the transaction logs in the buffer area have completed persistent storage.

[0049] However, the competition among multiple transaction processing threads to call the insert thread in processor 101 increases the latency of processor 101 executing threads to process transactions and increases the utilization of processor 101's computing resources, thereby affecting the transaction processing performance of data processing system 10. Furthermore, the presence of a large number of transaction log copies between different buffer areas of processor 101 limits the buffer and computing resources of processor 101. Consequently, resource limitations of processor 101 further affect the performance of data processing system 10.

[0050] Based on this, the data processing system 10 provided by the present application utilizes the processor 103 to reduce the resource usage of the processor 101 during the execution of transactions, thereby improving the performance of the data processing system 10. Specifically, the processor 103 is configured with a cache, which may be the cache 1031 in FIG1 , so that the processor 103 actively obtains multiple transaction logs from various data cache areas and writes the multiple transaction logs to the cache 1031. Furthermore, the processor 103 is responsible for persistently storing the multiple transaction logs stored in the cache 1031 in the storage device 102.

[0051] Since the processor 103 can actively write the transaction logs in multiple data cache areas into the cache 1031 used to generate WAL, the processor 101 does not need to compete with each other to call the insertion thread in the processor 101 during the execution of thread A and thread B (the processor 101 also does not need to create an insertion thread), thereby avoiding the long delay caused by the processor 101 executing thread A and thread B to process transactions due to competition. Accordingly, the processor 101 can complete the processing of transactions in a shorter period of time when executing thread A and thread B, which can effectively reduce the occupation of computing resources in the processor 101 by these threads. In this way, the performance of the data processing system 10 can be effectively improved, such as the efficiency of the data processing system 10 in executing transactions can be improved, and it can support the execution of a larger number of transactions at the same time.

[0052] In addition, the copying and persistent storage of the transaction log are completed by the processor 103, which means that the processor 101 does not need to copy the transaction log between different buffer areas, nor does it need to perform the process of persistently storing the transaction log. This can effectively reduce the storage resources (specifically the cache resources required to be consumed) and computing resources required by the processor 101 in the process of executing transactions, thereby further improving the performance of the data processing system 10.

[0053] In actual application, after completing the persistent storage of the transaction log, the processor 103 can also wake up the dormant thread A and thread B so that thread A and thread B can continue to execute the next transaction without the need for the processor 101 to wake up thread A and thread B. This can further reduce the resource usage of the processor 101 in the process of persistent storage of the transaction log, thereby further improving the performance of the data processing system 10.

[0054] Furthermore, the data processing system 10 may include multiple nodes. Among them, the processor 101, the storage device 102 and the processor 103 may constitute the first node in the data processing system 10, and the data processing system 10 may also include a second node, which may serve as a disaster recovery node for the first node, that is, the first node may serve as a master node and the second node may serve as a slave node. The first node and the second node are interconnected via a network. At this time, when the processor 103 persistently stores the transaction log, it will not only write the transaction log in the cache 1031 to the storage device 102, but will also send the transaction log to the second node for persistent storage. Exemplarily, the processor 103 may remotely write the transaction log to the memory in the second node based on remote direct memory access (RDMA) technology, so that the second node can persistently store the transaction log in the memory to the storage device included in the second node. Among them, RDMA technology is a technology that was created to solve the data processing delay between the sender and the receiver during network transmission. It mainly transmits data from the sender directly to the storage area of ​​the receiver through the network, eliminating multiple data copy operations during the transmission process. It does not require the intervention of the operating systems of both parties and reduces the processor load at the sender and the receiver.

[0055] It is worth noting that the data processing system 10 shown in FIG1 is only used as an exemplary illustration and is not intended to limit the implementation of the data processing system 10 to the example shown in FIG1 . For example, in a Quorum scenario or a distributed consensus framework (DCF) scenario, the data processing system 10 may include 3 or more nodes, one of which serves as a master node and the remaining nodes serve as slave nodes, and the master node includes a processor 101, a storage device 102, and a processor 103. Alternatively, the data processing system 10 may also include other types of devices to support the data processing system 10 with more functions. In addition, the processor 101 and the storage device 102 in the data processing system 10 may adopt an integrated storage and computing architecture, or a separate storage and computing architecture. The specific architecture of the data processing system is not limited in this application.

[0056] For ease of understanding, the process of processing transaction logs by the data processing system 10 is described in detail below with reference to the accompanying drawings.

[0057] Referring to FIG2 , a flow chart of a data processing method provided in an embodiment of the present application is shown. This method can be applied to the data processing system 10 shown in FIG1 , or can be applied to other applicable data processing systems. For ease of understanding and explanation, the following exemplary description is given using the data processing system 10 shown in FIG1 as an example. As shown in FIG2 , the method may specifically include:

[0058] S201 : When executing thread A and thread B, the processor 101 writes transaction logs generated during the transaction processing into their corresponding data buffers.

[0059] Thread A and thread B may be created by the processor 101 according to statements provided by the user.

[0060] As an implementation example, a user can input an SQL statement (or other types of statements) through a client provided by the data processing system 10, and the client sends the SQL statement to the processor 101. The processor 101 can generate a corresponding execution plan tree based on the received SQL statement, and the execution plan tree is used to describe the operations that the processor 101 needs to perform and the relationship between different operations. Then, the processor 101 can generate a corresponding plurality of transactions based on the execution plan tree, and create a thread A and a thread B for the plurality of transactions, wherein thread A and thread B are used to process the plurality of transactions. For ease of understanding and description, this embodiment is exemplified by taking thread A being used to process transaction 1 among the plurality of transactions and thread B being used to process transaction 2 among the plurality of transactions as an example.

[0061] In addition, the processor 101 will also allocate data cache areas as shown in Figure 1 to thread A and thread B. In this way, when the processor 101 executes thread A to process transaction 1, it will generate one or more transaction logs. The transaction logs can be used to record the operations performed by the processor 101 when accessing the index, database, or heap in the data processing system 10 when executing thread A, and the generated transaction logs will be written to the data cache area 1011 for storage. Similarly, when the processor 101 executes thread B, the transaction log generated during the processing of transaction 2 will be written to the data cache area 1012 for storage. The multiple transaction logs stored in each data cache area can be transaction logs of the same type, such as transaction logs for recording access to a database. Alternatively, the multiple transaction logs stored in each data cache area can be transaction logs of different types, such as transaction logs for recording access to an index and transaction logs for recording access to a database.

[0062] S202: After the processor 101 executes thread A and thread B to complete writing the transaction log, thread A and thread B enter a dormant state.

[0063] Typically, during the execution of a thread, after the processor 101 writes all generated transaction logs into the data buffer, the processor 101 begins to commit the transaction and sets the thread to a dormant state to wait for the transaction to be committed.

[0064] S203 : The processor 103 obtains multiple transaction logs from the multiple data cache areas respectively, and stores the multiple transaction logs into the cache 1031 in the processor 103 .

[0065] In this embodiment, the processor 103 is configured with a cache 1031 , and the processor 103 can proactively aggregate transaction logs in multiple data cache areas into the cache 1031 .

[0066] In a specific implementation, the processor 103 may periodically scan each data cache area to detect whether a transaction log is stored in each data cache area. If not, the processor 103 may not process it; if so, the processor 103 may execute the process of writing the transaction log in the data cache area into the cache 1031.

[0067] As a first implementation example, when it is detected that transaction logs are stored in both the data cache area 1011 and the data cache area 1012 , the processor 103 may write multiple transaction logs in each data cache area into the cache 1031 one by one.

[0068] As a second implementation example, when it is detected that transaction logs are stored in both data cache 1011 and data cache 1012, processor 103 may first aggregate multiple transaction logs in each data cache and simultaneously write the aggregated multiple transaction logs to cache 1031 to form an aggregate log (or what may be called a transaction block log). This aggregate log includes the contents of multiple transaction logs in the same data cache and may record information such as the size of each transaction log. In this way, cache 1031 may store multiple aggregate logs corresponding to multiple data caches.

[0069] For example, assuming that transaction logs 1 to 4 are stored in data cache 1011 and transaction logs 5 to 7 are stored in data cache 1012, processor 103 may first aggregate the four transaction logs in data cache 1011 into aggregate log 1 and write aggregate log 1 into cache 1031. Then, processor 103 may aggregate the three transaction logs in data cache 1012 into aggregate log 2 and write aggregate log 2 into cache 1031.

[0070] In this way, by batch processing the multiple transaction logs in the data cache area, the efficiency of the processor 103 writing the multiple transaction logs into the cache 1031 can be improved.

[0071] In actual application, the number of transaction logs stored in some data cache areas is large. At this time, the processor 103 can aggregate multiple transaction logs in the data cache area into multiple aggregate logs, such as aggregating 10 transaction logs into 2 aggregate logs, each aggregate log includes the contents of 5 transaction logs, etc.

[0072] The size of the aggregate log can be pre-configured by an administrator, for example, the administrator can pre-configure the size of the aggregate log to not exceed 2KB (kilobytes). Alternatively, the size of the aggregate log can be automatically configured by the processor 103. For example, the processor 103 can automatically configure the size based on the number of data write channels supported by the storage device 102. In this way, the processor 103 can batch write multiple transaction logs in each data cache area to the cache 1031 based on the configuration result for the aggregate log size, thereby obtaining multiple aggregate logs whose sizes meet the configuration result.

[0073] At this time, the transaction log is still recorded in the buffer area and has not yet been persisted. Therefore, the thread that generates the transaction log is still in a dormant state.

[0074] S204: The processor 103 records the LSN corresponding to each thread. The LSN corresponding to each thread is the maximum value of the LSNs corresponding to multiple transaction logs generated by the processor 101 executing the thread.

[0075] In a specific implementation, the processor 103 may determine the maximum value of the log sequence numbers (LSNs) among the multiple transaction logs stored in the data cache 1011, and record the maximum value as the LSN corresponding to thread A. Similarly, the processor 103 may record the LSN corresponding to thread B.

[0076] S205 : The processor 103 persistently stores the multiple transaction logs in the cache 1031 to the storage device 102 .

[0077] In this embodiment, the following two implementation methods for persistently storing multiple transaction logs are provided.

[0078] In a first implementation, the processor 103 may serially send each transaction log to the storage device 102. In a specific implementation, the processor 103 may create a write thread and use the write thread to sequentially send each transaction log to the storage device 102 for persistent storage.

[0079] In a second implementation, the processor 103 may send multiple transaction logs to the storage device 102 in parallel.

[0080] During specific implementation, the processor 103 can create a dedicated thread and multiple write threads. The processor 103 can first execute the dedicated thread to divide the multiple transaction logs in the cache 1031 to obtain multiple log segments, each log segment including multiple transaction logs. The number of log segments can be consistent with the number of write threads created. Then, the processor 103 executes the dedicated thread to determine multiple first storage areas in the storage device 102, each first storage area is used to store multiple transaction logs included in a log segment, and adds a lock to each first storage area respectively to lock the occupation of the first storage area by each write thread. Finally, the processor 103 can execute multiple write threads at the same time to write multiple log segments in parallel to the multiple first storage areas in the storage device 102.

[0081] In this way, processor 103 can accelerate the persistent storage of transaction logs by writing multiple transaction logs in cache 1031 to storage device 102 in parallel, thereby reducing transaction latency and improving transaction execution efficiency. Furthermore, by locking the first storage area used to store transaction logs for each writing process, conflicts caused by concurrent writing of transaction logs can be avoided, supporting concurrent writing of transaction logs to disk.

[0082] For example, when multiple transaction logs generated by a thread are stored in the cache 1031 in the form of aggregated logs, the processor 103 can divide each aggregated log into a log segment and, by executing multiple write threads, transmit the multiple aggregated logs in parallel to the storage device 102 for persistent storage. At this time, the size of the aggregated log can be determined according to the number of data transmission channels between the processor 103 and the storage device 102. For example, when there are 8 data write channels between the processor 103 and the storage device 102, the size of each aggregated log can be set to 2KB; and when there are 16 data write channels between the processor 103 and the storage device 102, the size of each aggregated log can be set to 1KB. In this way, based on the underlying input / output (IO) performance of the storage device 102, the size of the aggregated log can be flexibly configured, so that the performance of the processor 103 writing transaction logs to the storage device 102 in parallel can be optimized, which is conducive to improving the overall performance of the data processing system 10.

[0083] Furthermore, during the process of writing multiple log segments to the storage device 102 in parallel, the processor 103 may also record metadata corresponding to each of the written log segments in the storage device 102. As shown in FIG3 , before writing multiple log segments to the storage device 102, the processor 103 may first use a dedicated thread to determine multiple first storage areas and multiple second storage areas in the storage device 102. Each first storage area is used to store multiple transaction logs included in a log segment, and each second storage area is used to store metadata corresponding to each log segment. The metadata may, for example, describe the storage location of the log segment in the storage device 102, and may also describe information such as the size of the log segment, the sizes corresponding to the multiple transaction logs included in the log segment, and LSN. This embodiment does not limit this.

[0084] Then, the processor 103 may add locks to each of the first storage area and the second storage area respectively to lock the occupation of the first storage area and the second storage area by each write thread. The multiple transaction logs included in the log segment may be stored in the first storage area in the form of files. As shown in FIG3 , each first storage area stores a WAL log file, and the WAL log file records the contents of the multiple transaction logs.

[0085] Finally, the processor 103 may execute multiple write threads to write the multiple log segments in parallel to the multiple first storage areas in the storage device 102, and to write the metadata corresponding to the multiple log segments in parallel to the multiple second storage areas in the storage device 102. Since the amount of metadata is usually small, for example, the amount of data of a log segment may be 1 KB, while the amount of data of the metadata may be only 16 bytes, the delay caused by the processor 103 writing the metadata to the storage device 102 is basically negligible.

[0086] Thus, when data recovery is required based on the transaction log persistently stored in the storage device 102 (e.g., restart due to a failure), the data processing system 10 can verify whether there are holes in the transaction log by using the metadata recorded in the second storage area, i.e., determine whether part of the transaction log is missing. For example, the processor 103 in the data processing system 10 can detect whether the transaction log is missing by detecting whether there is metadata corresponding to the transaction log. If it is determined that the transaction log is not missing, the processor 103 can achieve data recovery by replaying the transaction logs recorded in the multiple first storage areas. If it is determined that the transaction log is missing (e.g., due to program errors or storage device 102 failures), as shown in FIG4 , the metadata corresponding to the log file 3 is not recorded in the storage device 102, then the processor 103 can only replay the transaction logs whose LSN is less than the minimum LSN in the WAL log file 3, and the transaction logs whose LSN is greater than or equal to the minimum LSN in the WAL log file 3 are invalid. The minimum LSN refers to the minimum value among the LSNs of the multiple transaction logs included in the log file 3.

[0087] S206: The processor 103 wakes up thread A and thread B according to the recorded LSNs corresponding to the threads.

[0088] It is understood that after the multiple transaction logs corresponding to thread A and thread B are persistently stored, the executed transaction is committed and the transaction ends. Therefore, the processor 103 can wake up the dormant thread A and thread B, so that the processor 101 can execute thread A and thread B to continue processing the next transaction.

[0089] Illustratively, this embodiment provides the following implementation examples of waking up threads A and B.

[0090] In a first implementation example, when processor 103 sequentially writes multiple transaction logs in cache 1031 to storage device 102 based on a single write thread, processor 103 may continuously monitor the LSN of the transaction logs written by the write thread and compare the LSN with the pre-recorded LSNs corresponding to each thread. When the LSN of the written transaction log is greater than the LSN corresponding to thread A, processor 103 wakes up thread A; otherwise, thread A remains dormant. When the LSN of the written transaction log is greater than the LSN corresponding to thread B, processor 103 wakes up thread B; otherwise, thread B remains dormant.

[0091] In this way, by comparing the LSN corresponding to the thread with the LSN of the transaction log that has completed persistent storage, the timing of waking up the thread can be accurately determined, thereby eliminating the need to wait until all transaction logs have completed persistent storage before waking up each thread. In this way, the sleep duration of some threads can be effectively shortened, avoiding some threads being in a long sleep state and wasting the computing resources of the processor 101, improving the overall efficiency of the processor 101 in executing multiple transactions using threads, and thus improving the overall performance of the data processing system 10. In addition, the processor 103 actively wakes up thread A and thread B in the processor 101, eliminating the need for the processor 101 to consume computing resources to determine whether to change the status of each thread. In this way, the resource consumption of the processor 101 can be further saved, thereby helping to further improve the overall performance of the data processing system 10.

[0092] In a second implementation example, when processor 103 writes multiple transaction logs in cache 1031 to storage device 102 in parallel based on multiple write threads, processor 103 sends a notification of successful transaction log storage to a dedicated thread in processor 103 after writing the transaction logs to the first storage area during the execution of the write threads. Thus, when processor 103 executes the dedicated thread, it can obtain the storage status of the transaction logs based on the notification messages sent back by each write thread. Thus, when executing the dedicated thread, processor 103 can determine a target LSN based on the storage status of the multiple transaction logs. The target LSN is the maximum LSN of the transaction logs that have been continuously persistently stored. For example, suppose that the processor 103 needs to write 100 transaction logs cached in the cache 1031 to the storage device 102, and the LSNs of the 100 transaction logs are 1 to 100 respectively. In the process of writing the 100 transaction logs to the storage device 102 in parallel, the processor 103 can determine, when executing a dedicated thread, that the transaction logs with LSNs of 1 to 68, 75 to 85, 95, and 99 have all completed persistent storage based on the storage status of the transaction logs fed back by multiple writing threads, and the remaining transaction logs have not yet completed storage. Then, the processor 103 can determine that the maximum LSN of the transaction logs that have completed persistent storage continuously is 68, that is, determine the target LSN to be 68.

[0093] The processor 103 can then execute a dedicated thread to compare the LSN corresponding to thread A with the target LSN. If the LSN corresponding to thread A is not greater than the target LSN, the processor 103 can wake up thread A by executing the dedicated thread; otherwise, thread A remains dormant. Similarly, if the LSN corresponding to thread B is not greater than the target LSN, the processor 103 can wake up thread B by executing the dedicated thread; otherwise, thread B remains dormant.

[0094] Thus, during the concurrent storage of multiple transaction logs, processor 103 can precisely locate the timing for waking up threads by comparing the LSNs corresponding to the threads with the target LSNs. This effectively shortens the sleep duration of some threads, improving the overall efficiency of processor 101 in executing multiple transactions using threads, and thus enhancing the overall performance of data processing system 10. Furthermore, by proactively waking up threads A and B in processor 101 by processor 103, processor 101 can further reduce resource consumption, thereby further improving the overall performance of data processing system 10. Actual tests based on the database performance test standard TPC-C have determined that processor 101 can save at least 30% of computing resources, and the throughput of data processing system 10 can be increased by over 30%.

[0095] In actual application, when the processor 103 executes a dedicated thread, it can also use the value of the target LSN to update the global variable LSN in the processor 101, so that the processor 101 can determine the LSN corresponding to the transaction log that has completed persistent storage according to the value of the global variable LSN.

[0096] In addition to the above implementation examples, the processor 103 may also wake up each thread based on other implementations. For example, after determining that all transaction logs in the cache 1031 have completed persistent storage, the processor 103 wakes up all threads uniformly, etc. This embodiment does not limit this.

[0097] In a further possible implementation, when the data processing system 10 meets the log playback conditions, such as when the processor 101 restarts after a failure, the processor 103 (or the processor 101) can access the transaction log persistently stored in the storage device 102. For example, the processor 103 can determine the storage address of the transaction log in the storage device 102 based on the metadata recorded in the multiple second storage areas in the storage device 102, so that the processor 103 can read the transaction logs in the multiple first storage areas based on the storage address. Since the amount of metadata recorded in the second storage area is relatively small, the delay caused by the processor 103 accessing the metadata to determine the storage location of the transaction log is very small and can be basically ignored (relative to the delay caused by the processor 103 accessing the transaction log).

[0098] Then, the processor 103 may implement data recovery by replaying the transaction log in the storage device 102 .

[0099] When the processor 103 writes a transaction log to the storage device 102 in the form of an aggregate log, the processor 103 (or the processor 101) may first analyze the type of the log after accessing the log in the storage device 102, such as by analyzing the log header to determine the type of the log. If the log is determined to be an aggregate log, the processor 103 may first decompose the aggregate log into multiple transaction logs and replay each transaction log one by one.

[0100] Specifically, as shown in Figure 5, when the operation content recorded in the transaction log is a page-related operation content, the processor 103 can assign the operation content to the queue corresponding to the playback thread, and execute the playback thread in sequence to perform the operations for one or more pages in the queue. When the operation content recorded in the transaction log is a table-related operation content, the processor 103 can assign the operation content to the queue corresponding to the playback thread according to the table granularity, and execute the playback thread in sequence to perform the operations for the pages corresponding to each data in the table in the queue.

[0101] When the transaction log is a log of modifying a table, such as a data definition language (DDL) log, the processor 103 can distribute the operation content in the transaction log to the queues corresponding to all playback threads, so as to replay the transaction log based on the operations of modifying the table recorded in the transaction log by executing each playback thread.

[0102] When the operation content recorded in the transaction log indicates updating the transaction status, the processor 103 may allocate the operation content to a transaction-related queue, so as to update the transaction status by replaying the transaction log.

[0103] In this embodiment, since the specific implementation process of the processor 103 playing back the transaction log has relevant applications in actual application scenarios, it will not be elaborated here.

[0104] As shown in FIG5 , when it is determined through analysis that the log type is a single transaction log, the processor 103 can allocate the log to a corresponding queue to replay the operation content recorded in the transaction log according to the operation content recorded in the transaction log.

[0105] The transaction log playback process described above is performed by processor 103. In other possible implementations, data processing system 10 may include multiple nodes, wherein processor 101, storage device 102, and processor 103 are the first node of the multiple nodes, and the other nodes in the multiple nodes serve as disaster recovery nodes for the first node. In this case, when data processing system 10 meets the log playback conditions, the disaster recovery node in data processing system 10 may also perform data recovery by replaying the transaction log.

[0106] Taking the case where the data processing system 10 also includes a second node as an example, when the processor 103 persistently stores the transaction log in the cache 1031 to the storage device 102, it will also remotely send the transaction log to the second node for persistent storage. For example, the processor 103 can remotely send the transaction log to the memory in the second node based on RDMA technology, and then the second node will persistently store the transaction log in the memory. In this way, when the data processing system 10 meets the log replay conditions, such as when the processor 101 in the first node fails, the second node can replay the log based on its persistently stored transaction log, so that the data stored in the second node is restored to the data stored before the failure of the first node, thereby realizing data recovery of the data processing system 10.

[0107] It should be noted that the data processing method shown in FIG2 is merely an example. In actual applications, the specific implementation of persistent storage of transaction logs by data processing system 10 is not limited to the above example and may also be implemented in other ways. As some examples, the following three points are listed in this embodiment to illustrate other possible implementations of persistent storage of transaction logs.

[0108] 1. In other possible embodiments, multiple different users can input SQL statements (or other types of statements) into data processing system 10 and generate multiple different transactions in data processing system 10. In this case, thread A and thread B can be used to process transactions corresponding to different SQL statements and generate transaction logs. Furthermore, the number of threads included in data processing system 10 can be any number, not limited to thread A and thread B described above.

[0109] 2. In other possible embodiments, while the processor 101 is executing a thread to write multiple transaction logs to the data cache, the processor 103 may continue to actively pull the transaction logs in the data cache to the cache 1031. In a specific implementation, the processor 103 may continue to monitor whether the data cache corresponding to each thread stores a transaction log. When a transaction log is stored, the processor 103 directly writes all the transaction logs currently stored in the data cache into the cache 1031. At this time, the processor 101 may still be in the stage of executing the thread to process transactions and generate transaction logs, so the processor 101 will continue to execute the thread to write a new transaction log (still belonging to the currently executed transaction) into its corresponding data cache. Accordingly, the processor 103 may detect that a new transaction log is stored in the data cache within the next monitoring cycle, and continue to actively write the new transaction log into the cache 1031.

[0110] 3. In other possible embodiments, after processor 103 writes all transaction logs in the data cache area to cache 1031, it is not necessary to record the LSN corresponding to each thread. Furthermore, after processor 103 sends all transaction logs in cache 1031 to storage device 102 for persistent storage, since all transaction logs generated by threads A and B have already been persistently stored, processor 103 can directly wake up threads A and B without having to compare the LSNs corresponding to the threads with the LSNs of the persistently stored transaction logs to determine whether to wake up the threads.

[0111] The data processing method provided by the present application is described in detail above in conjunction with Figures 1 to 5. The computing device provided by the present application will be described below in conjunction with Figure 6.

[0112] Figure 6 is a schematic diagram of the structure of a computing device provided by the present application. As shown in Figure 6, the computing device 600 includes a processor 601, a memory 602, a communication interface 603 and a bus 604. The processor 601, the memory 602 and the communication interface 603 communicate through the bus 604. The bus 604 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in Figure 6, but it does not mean that there is only one bus or one type of bus. The communication interface 603 is used to communicate with the outside, such as receiving jobs (and target data) submitted by users through a terminal or client.

[0113] It should be understood that in the embodiment of the present application, the processor 601 may be a CPU, or may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete device components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0114] The memory 602 may include a read-only memory and a random access memory, and provides instructions and data to the processor 601. The memory 602 may also include a non-volatile random access memory. For example, the memory 602 may also store device type information.

[0115] The memory 602 may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0116] The memory 602 stores executable codes, and the processor 601 executes the method executed by the aforementioned processor 103 .

[0117] Specifically, when implementing the embodiment shown in Figure 2, the software or program code required to execute the functions of the processor 103 in the embodiment shown in Figure 2 is stored in the memory 602, and the processor 601 is used to execute the instructions in the memory 602 to implement the method executed by the processor 103.

[0118] It should be understood that the computing device 600 according to the embodiment of the present application may correspond to the processor 103 that executes the method shown in Figure 2 in the embodiment of the present application, and the above-mentioned and other operations and / or functions implemented by the computing device 600 are respectively for implementing the corresponding processes of each method in Figure 2. For the sake of brevity, they will not be repeated here.

[0119] In addition, an embodiment of the present application further provides a computer-readable storage medium, which stores instructions. When the computer-readable storage medium is executed on a computing device, the computing device executes the method executed by the processor 103 in the above embodiment.

[0120] In addition, embodiments of the present application further provide a computer program product. When the computer program product is executed by a computing device, the one or more computing devices perform any of the aforementioned data processing methods. The computer program product may be a software installation package. When any of the aforementioned data processing methods is required, the computer program product may be downloaded and executed on a computer.

[0121] It should also be noted that the device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment. In addition, in the drawings of the device embodiments provided in this application, the connection relationship between the modules indicates that there is a communication connection between them, which can be specifically implemented as one or more communication buses or signal lines.

[0122] The terms "first," "second," and the like in the specification and claims of this application and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate and are merely used to describe the manner in which objects with the same attributes are described in the embodiments of this application.

[0123] Through the description of the above embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course can also be implemented by special hardware including application-specific integrated circuits, special CPUs, special memories, special components, etc. In general, all functions performed by computer programs can be easily implemented with corresponding hardware, and the specific hardware structures used to implement the same function can also be diverse, such as analog circuits, digital circuits or special circuits, etc. However, for the present application, software program implementation is a better implementation method in most cases. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer's floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk or optical disk, etc., and includes a number of instructions to enable a computer device (which can be a personal computer, training equipment, or network equipment, etc.) to execute the methods described in each embodiment of the present application.

[0124] The above embodiments can be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded or executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium. The semiconductor medium can be a solid state drive (SSD).

[0125] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present application, and such modifications or substitutions should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A data processing system, characterized in that: The data processing system includes a first processor, a storage device, and a second processor, the first processor includes a plurality of threads, each thread corresponds to a data cache area, the first processor is used to execute the plurality of threads to write a plurality of transaction logs into a plurality of data cache areas respectively corresponding to the plurality of threads, and the second processor includes a cache; The second processor is used to obtain the multiple transaction logs from the multiple data cache areas respectively, store the multiple transaction logs in the cache, and persistently store the multiple transaction logs from the cache to the storage device.

2. The data processing system according to claim 1, characterized in that The second processor includes an intelligent network card or a data processing unit DPU.

3. The data processing system according to claim 1 or 2, characterized in that: When the second processor persistently stores the multiple transaction logs from the cache to the storage device, the second processor is specifically configured to: Dividing the multiple transaction logs to obtain multiple log segments, each log segment including at least one transaction log; determining a plurality of first storage areas in the storage device; The multiple log segments are written in parallel into the multiple first storage areas, each first storage area is used to store one log segment.

4. The data processing system according to claim 3, characterized in that: The second processor is further configured to: determining a plurality of second storage areas in the storage device; The metadata corresponding to the multiple log segments are written into the multiple second storage areas in parallel, each second storage area is used to store the metadata corresponding to a log segment, and the metadata corresponding to each log segment is used to describe the storage location of the log segment in the storage device.

5. The data processing system according to claim 3 or 4, characterized in that: After the first processor executes the multiple threads to write the multiple transaction logs into the multiple data buffer areas respectively, the multiple threads are in a dormant state; The second processor is further configured to: Before writing the multiple log segments into the multiple first storage areas in parallel, recording the log sequence numbers LSN corresponding to the multiple threads respectively; Determine a target LSN, where the target LSN is the maximum LSN of the transaction logs that have been continuously completed for persistent storage; Comparing the size between the LSN corresponding to the first thread among the multiple threads and the target LSN; When the LSN corresponding to the first thread is not greater than the target LSN, the first thread is awakened.

6. The data processing system according to any one of claims 1 to 5, characterized in that: The number of the multiple log segments is consistent with the number of data writing channels supported by the storage device.

7. The data processing system according to any one of claims 1 to 6, characterized in that: When the second processor obtains the multiple transaction logs from the multiple data cache areas respectively and stores the multiple transaction logs in the cache, it is specifically used to: Writing multiple transaction logs in each data cache area into the cache at the same time to obtain multiple aggregate logs, each aggregate log including multiple transaction logs in one data cache area; The multiple transaction logs in each data cache area are transaction logs of the same type, or the multiple transaction logs in each data cache area are transaction logs of different types.

8. The data processing system according to claim 7, characterized in that: The second processor is further configured to: When the log playback condition is met, the aggregate log is decomposed into multiple transaction logs, and the multiple transaction logs decomposed from the aggregate log are played back.

9. The data processing system according to claim 7 or 8, characterized in that: The second processor is further configured to: Obtaining a configuration result for the size of the aggregate log; Then, when the second processor writes multiple transaction logs in each data cache area into the cache at the same time, it is specifically used to: According to the configuration result of the size of the aggregate log, multiple transaction logs in each data cache area are simultaneously written into the cache to obtain multiple aggregate logs whose sizes meet the configuration result.

10. The data processing system according to any one of claims 1 to 9, characterized in that: The second processor is further configured to: Based on remote direct memory access (RDMA) technology, the transaction log in the cache is sent to a disaster recovery node for backup.

11. A data processing method, characterized in that: The method is applied to a second processor in a data processing system, the data processing system further comprising a first processor and a storage device, the first processor comprising a plurality of threads, each thread corresponding to a data cache area, the first processor being used to execute the plurality of threads to respectively write a plurality of transaction logs into a plurality of data cache areas respectively corresponding to the plurality of threads, the second processor comprising a cache; the method comprising: The second processor obtains the multiple transaction logs from the multiple data cache areas respectively; The second processor stores the plurality of transaction logs in the cache; The second processor persistently stores the plurality of transaction logs from the cache to the storage device.

12. The method according to claim 11, characterized in that The second processor includes an intelligent network card or a data processing unit DPU.

13. The method according to claim 11 or 12, characterized in that: The second processor persistently stores the multiple transaction logs from the cache to the storage device, including: The second processor divides the multiple transaction logs to obtain multiple log segments, each log segment includes at least one transaction log; The second processor determines a plurality of first storage areas in the storage device; The second processor writes the multiple log segments into the multiple first storage areas in parallel, and each first storage area is used to store one log segment.

14. The method according to claim 13, characterized in that The method further comprises: The second processor determines a plurality of second storage areas in the storage device; The second processor writes metadata corresponding to the multiple log segments respectively into the multiple second storage areas in parallel, each second storage area is used to store metadata corresponding to a log segment, and the metadata corresponding to each log segment is used to describe the storage location of the log segment in the storage device.

15. The method according to claim 13 or 14, characterized in that After the first processor executes the multiple threads to write the multiple transaction logs into the multiple data buffer areas respectively, the multiple threads are in a dormant state; The method further comprises: The second processor records the log sequence numbers LSNs respectively corresponding to the multiple threads before writing the multiple log segments into the multiple first storage areas in parallel; The second processor determines a target LSN, where the target LSN is the maximum LSN of the transaction logs that have been continuously completed in persistent storage; The second processor compares the size between the LSN corresponding to the first thread among the multiple threads and the target LSN; When the LSN corresponding to the first thread is not greater than the target LSN, the second processor wakes up the first thread.

16. The method according to any one of claims 11 to 15, characterized in that The number of the multiple log segments is consistent with the number of data writing channels supported by the storage device.

17. The method according to any one of claims 11 to 16, characterized in that: The second processor obtains the multiple transaction logs from the multiple data cache areas respectively, and the second processor stores the multiple transaction logs in the cache, including: The second processor writes the multiple transaction logs in each data cache area into the cache at the same time to obtain multiple aggregate logs, each aggregate log including the multiple transaction logs in one data cache area; The multiple transaction logs in each data cache area are transaction logs of the same type, or the multiple transaction logs in each data cache area are transaction logs of different types.

18. The method according to claim 17, characterized in that The method further comprises: When a log playback condition is met, the second processor decomposes the aggregate log into multiple transaction logs; The second processor plays back a plurality of transaction logs decomposed from the aggregate log.

19. The method according to claim 17 or 18, characterized in that The method further comprises: The second processor obtains a configuration result for the size of the aggregate log; Then, the second processor writes multiple transaction logs in each data cache area into the cache at the same time, including: The second processor writes multiple transaction logs in each data cache area into the cache at the same time according to the configuration result of the size of the aggregate log, so as to obtain multiple aggregate logs whose sizes meet the configuration result.

20. The method according to any one of claims 11 to 19, characterized in that The method further comprises: The second processor sends the transaction log in the cache to the disaster recovery node based on remote direct memory access (RDMA) technology. Perform a backup.

21. A computing device, characterized in that The computing device comprises a processor and a memory, wherein the memory stores program instructions, and the processor runs the program instructions to execute the method according to any one of claims 11 to 20.

22. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, which, when executed on a computing device, enable the computing device to perform the method according to any one of claims 11 to 20.

23. A computer program product comprising instructions which, when executed on a computing device, cause the computing device to perform the method according to any one of claims 11 to 20.

Images