Data processing system, data processing method and related equipment

Abstract

The invention provides a data processing system which comprises a first processor, a storage device and a second processor, the first processor comprises a plurality of threads, each thread corresponds to a data cache region, the first processor is used for executing the threads, and each thread corresponds to a data cache region. The first processor comprises a plurality of transaction logs and a plurality of threads, the plurality of transaction logs are written into a plurality of data cache regions corresponding to the plurality of threads respectively, the second processor comprises a cache, and the second processor is used for obtaining the plurality of transaction logs from the plurality of data cache regions respectively, storing the plurality of transaction logs into the cache and persistently storing the plurality of transaction logs into the storage device from the cache. Therefore, the first processor does not need to compete with each other to call and insert the threads in the process of executing the multiple threads, and does not need to execute the process of copying the transaction log, so that the first processor can process and complete the transaction in a shorter time period, and the performance of the data processing system can be effectively improved. In addition, the invention also provides a corresponding data processing method and related equipment.

Description

Technical Field

[0001] The present application relates to the field of database technology, and in particular to a data processing system, a data processing method and related equipment. Background Art

[0002] In data processing systems, write-ahead logging (WAL) is usually used to ensure the atomicity and persistence of data operations. Specifically, when data in the database needs to be modified (including adding, deleting, and replacing data, etc.), the data will not be modified directly in the database, but the data modification operation will be recorded in the WAL log and stored persistently before the data in the database is modified. In this way, during the process of modifying data, if an error occurs and the data modification fails, the data processing system can restore the data modification by replaying the persistently stored WAL log.

[0003] At present, the processor in the data processing system usually includes multiple threads for executing transactions (such as transactions for modifying data), and the threads will generate corresponding transaction logs (Xlog) in the process of executing transactions. Usually, each thread will write the generated transaction log into the data buffer allocated for the thread, and compete with other threads to call the insert thread in the processor to write the transaction log in its own buffer area into the buffer area corresponding to the WAL log, so that after the transaction log in the buffer area is persistently stored, the thread continues to execute the next transaction.

[0004] However, the competition among multiple threads to call the inserted thread in the processor will increase the latency of the thread execution transaction and the occupation of the processor's computing resources, thereby affecting the performance of the data processing system. Summary of the invention

[0005] The present application provides a data processing system for improving the performance of the data processing system. In addition, the present application also provides a data processing method, a computing device, a computer-readable storage medium, and a computer program product.

[0006] In a first aspect, the present application provides a data processing system, comprising a first processor, a storage device and a second processor, wherein the first processor comprises multiple threads, and each thread corresponds to a data cache area, which can be used to store transaction logs, and the first processor is used to execute the multiple threads to write multiple transaction logs into multiple data cache areas corresponding to the multiple threads respectively, and the multiple transaction logs can be generated when the first processor executes multiple threads to process transactions, and the second processor comprises a cache, and the second processor is used to obtain multiple transaction logs from the multiple data cache areas respectively, and store the multiple transaction logs in the cache, and persistently store the multiple transaction logs from the cache to the storage device.

[0007] Since the second processor can actively write the transaction logs in multiple data cache areas into the cache for generating WAL, the first processor does not need to compete with each other to call the insertion thread in the first processor during the execution of multiple threads (the first processor may not need to create an insertion thread), thereby avoiding the delay caused by the first processor executing multiple threads to process transactions due to the competition for calling the insertion thread. Accordingly, the first processor can complete the transaction in a shorter period of time when executing multiple threads, which can effectively reduce the occupation of these threads for the computing resources in the first processor. In this way, the performance of the data processing system can be effectively improved, such as the efficiency of the data processing system in executing transactions can be improved, and it can support the execution of more transactions at the same time. In addition, the copying and persistent storage of the transaction log are completed by the second processor, which makes it unnecessary for the first processor to copy the transaction log between different buffer areas, and it is also unnecessary to perform the process of persistently storing the transaction log, thereby effectively reducing the storage resources (specifically the cache resources required to be consumed) and computing resources consumed by the first processor during the execution of transactions, thereby further improving the performance of the data processing system.

[0008] In a possible implementation, the second processor includes a smart network card or a data processing unit DPU. Thus, by using the smart network card or the DPU, the process of copying the transaction log of the first processor can be offloaded, thereby improving the performance of the data processing system.

[0009] In a possible implementation, when the second processor persistently stores multiple transaction logs from the cache to the storage device, it can specifically divide the multiple transaction logs to obtain multiple log segments, each log segment includes at least one transaction log, and determine multiple first storage areas in the storage device, and write the multiple log segments in parallel to the multiple first storage areas, each first storage area is used to store a log segment. In this way, the second processor can accelerate the persistent storage speed of the transaction logs by writing the multiple transaction logs in the cache in parallel to the storage device, thereby reducing the transaction delay and improving the efficiency of executing transactions.

[0010] In a possible implementation, the second processor is also used to determine multiple second storage areas in the storage device, and write metadata corresponding to multiple log segments in parallel into the multiple second storage areas, each second storage area is used to store metadata corresponding to a log segment, and the metadata corresponding to each log segment is used to describe the storage location of the log segment in the storage device. In this way, when data recovery is required based on the transaction log persistently stored in the storage device, the data processing system can verify whether there are holes in the transaction log through the metadata recorded in the second storage area, that is, determine whether part of the transaction log is missing, so that when it is determined that the transaction log is not missing, the second processor can achieve data recovery by replaying the transaction logs recorded in the multiple first storage areas.

[0011] In a possible implementation, after the first processor executes multiple threads to write multiple transaction logs into multiple data cache areas respectively, the multiple threads are in a dormant state. Then, before the second processor writes multiple log segments into multiple first storage areas in parallel, it records the log sequence numbers LSN corresponding to the multiple threads respectively, and determines the target LSN, which is the maximum LSN of the transaction logs that have completed persistent storage continuously, so that the second processor can compare the size between the LSN corresponding to the first thread among the multiple threads and the target LSN, and when the LSN corresponding to the first thread is not greater than the target LSN, wake up the first thread. In this way, the second processor can accurately determine the timing of waking up the thread by comparing the LSN corresponding to the thread with the LSN of the transaction log that has completed persistent storage, so that there is no need to wait until all transaction logs have completed persistent storage before waking up each thread, so that the dormancy time of some threads can be effectively shortened, and some threads can be prevented from being in a long dormant state and wasting the computing resources of the first processor, thereby improving the overall efficiency of the first processor using threads to execute multiple transactions, thereby improving the overall performance of the data processing system. Furthermore, the second processor actively wakes up the first thread in the first processor, and the first processor does not need to consume computing resources to determine whether to change the state of each thread, thereby further saving resource consumption of the first processor and helping to further improve the overall performance of the data processing system.

[0012] In a possible implementation, the number of multiple log segments is consistent with the number of data writing channels supported by the storage device. In this way, based on the underlying IO performance of the storage device, the size of the aggregate log can be flexibly configured, that is, the number of log segments can be configured, so that the performance of the second processor writing transaction logs to the storage device in parallel can be optimized, which is beneficial to improving the overall performance of the data processing system.

[0013] In a possible implementation, when the second processor stores multiple transaction logs in the data cache area into the cache, it can specifically write multiple transaction logs in each data cache area into the cache at the same time to obtain multiple aggregate logs, each aggregate log includes multiple transaction logs in a data cache area; wherein the multiple transaction logs in each data cache area are transaction logs of the same type, or the multiple transaction logs in each data cache area are transaction logs of different types. In this way, by batch processing the multiple transaction logs in the data cache area, the efficiency of the second processor writing the multiple transaction logs into the cache can be improved.

[0014] In a possible implementation, the second processor can also decompose the aggregate log into multiple transaction logs when the log playback condition is met, and replay the multiple transaction logs decomposed from the aggregate log. In this way, the second processor can realize data recovery of the data processing system by decomposing and replaying the transaction logs in the aggregate log.

[0015] In a possible implementation, the second processor can also obtain the configuration result for the size of the aggregate log, and the size of the aggregate log can be configured by the administrator or automatically configured by the second processor. Then, when the second processor writes multiple transaction logs in each data cache area into the cache at the same time, it can specifically write multiple transaction logs in each data cache area into the cache at the same time according to the configuration result of the size of the aggregate log, and obtain multiple aggregate logs whose sizes meet the configuration result. In this way, the second processor can write multiple transaction logs in each data cache area into the cache at the same time according to the size of the aggregate log, which can not only improve the efficiency of writing transaction logs into the cache, but also realize flexible configuration of the size of the aggregate log.

[0016] In a possible implementation, the second processor can also send the transaction log in the cache to the disaster recovery node for backup based on remote direct memory access (RDMA) technology. The first processor, the storage device, and the second processor can constitute a first node in a data processing system, and the data processing system also includes a disaster recovery node, which is used as a disaster recovery for the first node. In this way, when the first node fails, the data processing system can achieve data recovery through the transaction log backed up on the disaster recovery node, thereby ensuring the reliability of the data processing system.

[0017] In a second aspect, the present application provides a data processing method, which is applied to a second processor in a data processing system. The data processing system also includes a first processor and a storage device. The first processor includes multiple threads, each thread corresponds to a data cache area, and the first processor is used to execute multiple threads to write multiple transaction logs into multiple data cache areas corresponding to the multiple threads respectively. The second processor includes a cache; the method includes: the second processor obtains multiple transaction logs from the multiple data cache areas respectively; the second processor stores the multiple transaction logs to the cache; the second processor persistently stores the multiple transaction logs from the cache to the storage device.

[0018] In a possible implementation, the second processor includes a smart network card or a data processing unit DPU.

[0019] In one possible implementation, the second processor persistently stores multiple transaction logs from a cache to a storage device, including: the second processor divides the multiple transaction logs to obtain multiple log segments, each log segment includes at least one transaction log; the second processor determines multiple first storage areas in the storage device; the second processor writes the multiple log segments in parallel to the multiple first storage areas, each first storage area is used to store a log segment.

[0020] In a possible implementation, the method also includes: a second processor determines multiple second storage areas in a storage device; the second processor writes metadata corresponding to multiple log segments into the multiple second storage areas in parallel, each second storage area is used to store metadata corresponding to a log segment, and the metadata corresponding to each log segment is used to describe the storage location of the log segment in the storage device.

[0021] In a possible implementation, after the first processor executes multiple threads to write multiple transaction logs into multiple data cache areas respectively, the multiple threads are in a dormant state; the method also includes: before the second processor writes the multiple log segments into multiple first storage areas in parallel, the second processor records the log sequence numbers LSN corresponding to the multiple threads respectively; the second processor determines the target LSN, which is the maximum LSN of the transaction logs that have been continuously completed for persistent storage; the second processor compares the size between the LSN corresponding to the first thread among the multiple threads and the target LSN; when the LSN corresponding to the first thread is not greater than the target LSN, the second processor wakes up the first thread.

[0022] In a possible implementation, the number of the multiple log segments is consistent with the number of data writing channels supported by the storage device.

[0023] In a possible implementation, the second processor obtains multiple transaction logs from multiple data cache areas respectively, and the second processor stores the multiple transaction logs in the cache, including: the second processor writes the multiple transaction logs in each data cache area into the cache at the same time to obtain multiple aggregate logs, each aggregate log includes multiple transaction logs in a data cache area; wherein the multiple transaction logs in each data cache area are transaction logs of the same type, or the multiple transaction logs in each data cache area are transaction logs of different types.

[0024] In a possible implementation, the method further includes: when a log playback condition is met, the second processor decomposes the aggregate log into multiple transaction logs; and the second processor plays back the multiple transaction logs decomposed from the aggregate log.

[0025] In a possible implementation, the method further includes: the second processor obtains a configuration result for the size of the aggregate log; then, the second processor writes multiple transaction logs in each data cache area into the cache at the same time, including: the second processor writes multiple transaction logs in each data cache area into the cache at the same time according to the configuration result of the size of the aggregate log, to obtain multiple aggregate logs whose sizes meet the configuration result.

[0026] In a possible implementation, the method further includes: the second processor sends the transaction log in the cache to the disaster recovery node for backup based on remote direct memory access (RDMA) technology.

[0027] The data processing method provided in the second aspect corresponds to the data processing system provided in the first aspect. Therefore, the technical effects of the data processing method in the second aspect and any possible implementation method of the second aspect can refer to the technical effects of the first aspect and the corresponding implementation method of the first aspect, and will not be repeated here.

[0028] In a third aspect, the present application provides a computing device, the computing device comprising a processor and a memory; the memory is used to store instructions, and when the computing device is running, the processor executes the instructions stored in the memory, so that the computing device executes the method described in the second aspect or any possible implementation of the second aspect. It should be noted that the memory can be integrated into the processor or can be independent of the processor. The computing device may also include a bus. The processor is connected to the memory via the bus. The memory may include a readable memory and a random access memory.

[0029] In a fourth aspect, the present application provides a computer-readable storage medium, wherein instructions are stored in the computer-readable storage medium, which, when executed on a computing device, enables the computing device to execute the method described in the second aspect or any one of the implementations of the second aspect.

[0030] In a fifth aspect, the present application provides a computer program product comprising instructions, which, when executed on a computing device, enables the computing device to execute the method described in the second aspect or any one of the implementations of the second aspect.

[0031] Based on the implementations provided in the above aspects, this application can also be further combined to provide more implementations. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0033] Figure 1 A schematic diagram of the structure of an exemplary data processing system provided by the present application;

[0034] Figure 2 A flowchart of a data processing method provided in this application;

[0035] Figure 3 A schematic diagram of the processor 103 writing transaction logs and metadata to the storage device 102 in parallel;

[0036] Figure 4 is a schematic diagram of metadata and transaction logs recorded in the storage device 102;

[0037] Figure 5 A schematic diagram of playing back logs according to log types provided for this application;

[0038] Figure 6 A schematic diagram of the structure of a computing device provided in this application. DETAILED DESCRIPTION

[0039] The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances. This is just a way of distinguishing objects with the same attributes when describing the embodiments of this application.

[0040] See also Figure 1 , which shows a schematic diagram of the structure of a data processing system. Figure 1 As shown, the data processing system 10 includes a processor 101, a storage device 102 and a processor 103, and the processor 101, the storage device 102 and the processor 103 can be connected via a high-speed Internet network.

[0041] The processor 101 may include multiple threads, such as Figure 1 Thread A and thread B in the data processing system 10 are used to execute one or more transactions. A transaction refers to a program execution unit that accesses and may update data items in the data processing system 10. When the processor 101 executes the multiple threads to process transactions, it will generate a corresponding transaction log to record the changes made to the persistently stored data in the data processing system 10. The processor 101 also includes a cache, and the storage area in the cache can be divided into multiple data cache areas, such as Figure 1The data cache area 1011 and the data cache area 1012 in the data cache area 1011. Each thread corresponds to a data cache area, which is used to store the transaction log generated when the processor 101 executes the thread. Specifically, the data cache area 1011 can be used to store the transaction log generated when the processor 101 executes thread A, and the data cache area 1012 can be used to store the transaction log generated when the processor 101 executes thread B.

[0042] The storage device 102 may be any device that provides persistent storage capabilities, such as a solid state disk (SSD), a shingled magnetic recording (SMR) hard disk, Intel Optane persistent memory (PMEM), a remote shared storage device, a disk array, a cloud disk, etc.

[0043] The processor 103 may be a device with data processing capability, such as a smart network interface card (SmartNIC), a data processing unit (DPU), etc., or may be other types of devices.

[0044] The data processing system 10 can provide a client to the outside. The client, for example, can be a web browser, or can be an application (application) running on a user terminal, for interacting with the user, such as receiving a statement input by the user for modifying data, which statement can be, for example, a structured query language (SQL) statement. The processor 101 can create one or more transactions according to the statement input by the user, and assign a corresponding thread to the transaction to process the transaction. Assume that the processor 101 creates transaction 1 and transaction 2 according to the statement input by the user, and assigns thread A to process transaction 1 and thread B to process transaction 2.

[0045] When the processor 101 executes thread A and thread B to process transactions, one or more transaction logs will be generated. For example, when the processor 101 executes thread A, it may access an index (such as a Btree index, etc.), thereby generating a transaction log for recording operations such as adding, deleting, and replacing the index. Similarly, when the processor 101 executes thread A, it may access a database (DB) or a heap, thereby generating corresponding transaction logs. The transaction logs generated by the processor 101 in the process of executing each thread to process transactions will be written into the data buffer area corresponding to the thread for cache.

[0046] Normally, the transaction log in the data cache area needs to be persistently stored to ensure the reliability of storage in the data processing system 10. At this time, if the processor 101 persistently stores the transaction logs in each data cache area, an insertion thread needs to be set in the processor 101, and after the processor 101 executes thread A and thread B to write the transaction log to the data cache area, it will compete to call the insertion thread, so that the transaction log written in the data cache area is inserted into the buffer area (such as WAL buffer) configured separately in the processor 101 by executing the insertion thread, so that the processor 101 can subsequently write the transaction log in the buffer area into the storage device 102 in the form of WAL. Normally, after the processor 101 executes the insertion thread to write all the transaction logs in the data cache area into the separately configured buffer area, thread A and thread B will enter a dormant state (such as a semdown state), and the thread A and thread B will be awakened by the processor 101 after all the transaction logs in the buffer area have completed persistent storage.

[0047] However, the process in which multiple threads for processing transactions compete to call the insertion thread in the processor 101 will increase the latency of the processor 101 executing threads to process transactions and increase the occupation of the computing resources of the processor 101, thereby affecting the transaction processing performance of the data processing system 10. In addition, there are a large number of copies of transaction logs between different buffer areas of the processor 101, which will limit the buffer resources and computing resources of the processor 101, so that the resource limitation of the processor 101 will further affect the performance of the data processing system 10.

[0048] Based on this, in the data processing system 10 provided by the present application, the processor 103 is used to reduce the resource occupation of the processor 101 during the execution of transactions, thereby improving the performance of the data processing system 10. Specifically, the processor 103 is configured with a cache, which can be Figure 1 The cache 1031 in the processor 103 will actively obtain multiple transaction logs in each data cache area and write the multiple transaction logs into the cache 1031. In addition, the processor 103 is also responsible for persistently storing the multiple transaction logs stored in the cache 1031 in the storage device 102.

[0049] Since the processor 103 can actively write the transaction logs in multiple data cache areas into the cache 1031 used to generate WAL, the processor 101 does not need to compete with each other to call the insertion thread in the processor 101 during the execution of thread A and thread B (the processor 101 may not need to create an insertion thread), thereby avoiding the long delay caused by the processor 101 executing thread A and thread B to process transactions due to competition for calling the insertion thread. Accordingly, the processor 101 can complete the processing of transactions in a shorter period of time when executing thread A and thread B, which can effectively reduce the occupation of computing resources in the processor 101 by these threads. In this way, the performance of the data processing system 10 can be effectively improved, such as the efficiency of the data processing system 10 in executing transactions can be improved, and it can support the execution of a larger number of transactions at the same time.

[0050] Furthermore, the copying and persistent storage of transaction logs are performed by processor 103, which eliminates the need for processor 101 to copy transaction logs between different buffer areas or to perform the process of persistently storing transaction logs. This can effectively reduce the storage resources (specifically, the cache resources) and computing resources consumed by processor 101 in the process of executing transactions, thereby further improving the performance of data processing system 10.

[0051] In actual application, after completing the persistent storage of the transaction log, the processor 103 can also wake up the dormant thread A and thread B so that thread A and thread B can continue to execute the next transaction without the need for the processor 101 to wake up thread A and thread B. This can further reduce the resource usage of the processor 101 in the process of persistent storage of the transaction log, thereby further improving the performance of the data processing system 10.

[0052] Further, the data processing system 10 may include multiple nodes. Among them, the processor 101, the storage device 102 and the processor 103 may constitute the first node in the data processing system 10, and the data processing system 10 may also include a second node, which may be used as a disaster recovery node of the first node, that is, the first node may be used as a master node (masternode), and the second node may be used as a slave node (slave node). The first node and the second node are interconnected through a network. At this time, when the processor 103 persistently stores the transaction log, it will not only write the transaction log in the cache 1031 to the storage device 102, but also send the transaction log to the second node for persistent storage. Exemplarily, the processor 103 can remotely write the transaction log to the memory in the second node based on remote direct memory access (RDMA) technology, so that the second node can persistently store the transaction log in the memory to the storage device included in the second node, etc. Among them, RDMA technology is a technology that was created to solve the data processing delay between the sender and the receiver during network transmission. It mainly transmits the data in the sender directly to the storage area of ​​the receiver through the network, eliminating multiple data copy operations during the transmission process. It does not require the intervention of the operating systems of both parties, and reduces the processor load in the sender and the receiver.

[0053] It is worth noting that the above Figure 1 The data processing system 10 shown is only used as an exemplary illustration and is not intended to limit the implementation of the data processing system 10 to Figure 1 Example shown. For example, in a Quorum scenario or a distributed consensus framework (DCF) scenario, the data processing system 10 may include three or more nodes, one of which serves as a master node and the remaining nodes serve as slave nodes. The master node includes a processor 101, a storage device 102, and a processor 103. Alternatively, the data processing system 10 may also include other types of devices to support the data processing system 10 with more functions. In addition, a storage-computing integrated architecture or a storage-computing separated architecture may be adopted between the processor 101 and the storage device 102 in the data processing system 10. The specific architecture of the data processing system is not limited in this application.

[0054] For ease of understanding, the process of data processing system 10 processing transaction logs is described in detail below with reference to the accompanying drawings.

[0055] See also Figure 2 , shows a flow chart of a data processing method provided by an embodiment of the present application, which can be applied to Figure 1The data processing system 10 shown in FIG. 1 may be used in other applicable data processing systems. Figure 1 The data processing system 10 shown in FIG. Figure 2 As shown, the method may specifically include:

[0056] S201: When executing thread A and thread B, the processor 101 writes the transaction logs generated in the process of processing transactions into their corresponding data buffers respectively.

[0057] Thread A and thread B may be created by processor 101 according to statements provided by a user.

[0058] As an implementation example, a user can input an SQL statement (or other types of statements) through a client provided by the data processing system 10, and the client sends the SQL statement to the processor 101. The processor 101 can generate a corresponding execution plan tree based on the received SQL statement, and the execution plan tree is used to describe the operations that the processor 101 needs to perform and the relationship between different operations. Then, the processor 101 can generate corresponding multiple transactions based on the execution plan tree, and create thread A and thread B for the multiple transactions, wherein thread A and thread B are used to process the multiple transactions. For ease of understanding and description, this embodiment takes thread A as an example for processing transaction 1 in the multiple transactions and thread B as an example for processing transaction 2 in the multiple transactions.

[0059] In addition, the processor 101 also allocates the following to the thread A and thread B: Figure 1 The data cache area shown. In this way, when the processor 101 executes thread A to process transaction 1, one or more transaction logs will be generated. The transaction log can be used to record the operations performed in the process of accessing the index, database or heap in the data processing system 10 when the processor 101 executes thread A, and the generated transaction log is written into the data cache area 1011 for storage. Similarly, when the processor 101 executes thread B, the transaction log generated in the process of processing transaction 2 will be written into the data cache area 1012 for storage. Among them, the multiple transaction logs stored in each data cache area can be transaction logs of the same type, such as transaction logs for recording access to a database, etc. Alternatively, the multiple transaction logs stored in each data cache area can be transaction logs of different types, such as transaction logs for recording access to an index, transaction logs for recording access to a database, etc.

[0060] S202: After the processor 101 executes thread A and thread B to complete writing of the transaction log, thread A and thread B enter a dormant state.

[0061] Typically, during the execution of a thread, after the processor 101 writes all generated transaction logs into the data buffer, the processor 101 starts to commit the transaction and sets the thread to a dormant state to wait for the transaction to be committed.

[0062] S203 : the processor 103 obtains multiple transaction logs from multiple data cache areas respectively, and stores the multiple transaction logs into the cache 1031 in the processor 103 .

[0063] In this embodiment, the processor 103 is configured with a cache 1031 , and the processor 103 can actively aggregate transaction logs in multiple data cache areas into the cache 1031 .

[0064] In specific implementation, the processor 103 may periodically scan each data cache area to detect whether there is a transaction log stored in each data cache area. If not, the processor 103 may not process it; if so, the processor 103 may execute the process of writing the transaction log in the data cache area into the cache 1031.

[0065] As a first implementation example, when it is detected that transaction logs are stored in both the data cache area 1011 and the data cache area 1012 , the processor 103 may write multiple transaction logs in each data cache area into the cache 1031 one by one.

[0066] As a second implementation example, when it is detected that transaction logs are stored in both data cache area 1011 and data cache area 1012, for each data cache area, processor 103 may first aggregate multiple transaction logs in the data cache area, and simultaneously write the aggregated multiple transaction logs into cache 1031 to form an aggregate log (or transaction block log), which includes the contents of multiple transaction logs in the same data cache area and may record information such as the size of each transaction log. In this way, cache 1031 may store multiple aggregate logs corresponding to multiple data cache areas.

[0067] For example, assuming that transaction logs 1 to 4 are stored in data cache 1011 and transaction logs 5 to 7 are stored in data cache 1012, processor 103 may first aggregate the four transaction logs in data cache 1011 into aggregate log 1, and write aggregate log 1 into cache 1031. Then, processor 103 aggregates the three transaction logs in data cache 1012 into aggregate log 2, and writes aggregate log 2 into cache 1031.

[0068] In this way, by batch processing the multiple transaction logs in the data cache area, the efficiency of the processor 103 writing the multiple transaction logs into the cache 1031 can be improved.

[0069] In actual application, the number of transaction logs stored in some data cache areas is large. At this time, the processor 103 can aggregate multiple transaction logs in the data cache area into multiple aggregate logs, such as aggregating 10 transaction logs into 2 aggregate logs, each aggregate log includes the contents of 5 transaction logs, etc.

[0070] The size of the aggregate log may be pre-configured by an administrator, such as the administrator may pre-configure the size of the aggregate log to be no more than 2KB (kilobytes). Alternatively, the size of the aggregate log may be automatically configured by the processor 103. For example, the processor 103 may perform automatic configuration according to the number of data write channels supported by the storage device 102. In this way, the processor 103 may batch write multiple transaction logs in each data cache area into the cache 1031 according to the configuration result for the size of the aggregate log, and obtain multiple aggregate logs whose sizes meet the configuration result.

[0071] At this time, the transaction log is still recorded in the buffer area and has not yet completed persistent storage. Therefore, the thread that generates the transaction log will still be in a dormant state.

[0072] S204: The processor 103 records the LSN corresponding to each thread. The LSN corresponding to each thread is the maximum value of the LSNs corresponding to multiple transaction logs generated by the processor 101 executing the thread.

[0073] In a specific implementation, the processor 103 can determine the maximum value among multiple log sequence numbers (LSN) corresponding to multiple transaction logs stored in the data cache 1011, and record the maximum value as the LSN corresponding to thread A. In a similar manner, the processor 103 can record the LSN corresponding to thread B.

[0074] S205 : The processor 103 persistently stores the multiple transaction logs in the cache 1031 to the storage device 102 .

[0075] In this embodiment, the following two implementation methods for persistently storing multiple transaction logs are provided.

[0076] In a first implementation, the processor 103 may serially send each transaction log to the storage device 102. In a specific implementation, the processor 103 may create a write thread, and use the write thread to sequentially send each transaction log to the storage device 102 for persistent storage.

[0077] In a second implementation, the processor 103 may send multiple transaction logs to the storage device 102 in parallel.

[0078] In a specific implementation, the processor 103 can create a dedicated thread and multiple write threads. The processor 103 can first execute the dedicated thread to divide the multiple transaction logs in the cache 1031 to obtain multiple log segments, each log segment including multiple transaction logs. Among them, the number of log segments can be consistent with the number of created write threads. Then, the processor 103 executes the dedicated thread to determine multiple first storage areas in the storage device 102, each first storage area is used to store multiple transaction logs included in a log segment, and a lock is added to each first storage area respectively to lock the occupation of each write thread for the first storage area. Finally, the processor 103 can execute multiple write threads at the same time to write multiple log segments in parallel to multiple first storage areas in the storage device 102.

[0079] In this way, the processor 103 can accelerate the persistent storage speed of the transaction logs by writing the multiple transaction logs in the cache 1031 in parallel to the storage device 102, thereby reducing the transaction delay and improving the efficiency of executing transactions. In addition, by locking the first storage area for storing transaction logs for each writing process, conflicts caused by concurrent writing of transaction logs can be avoided, and concurrent writing of transaction logs can be supported.

[0080] Exemplarily, when multiple transaction logs generated by threads are stored in cache 1031 in the form of aggregate logs, processor 103 can use each aggregate log as a log segment, and transmit multiple aggregate logs in parallel to storage device 102 for persistent storage by executing multiple write threads. At this time, the size of the aggregate log can be determined according to the number of data transmission channels between processor 103 and storage device 102. For example, when there are 8 data write channels between processor 103 and storage device 102, the size of each aggregate log can be set to 2KB; and when there are 16 data write channels between processor 103 and storage device 102, the size of each aggregate log can be set to 1KB. In this way, based on the underlying input / output (IO) performance of storage device 102, the size of the aggregate log can be flexibly configured, so that the performance of processor 103 writing transaction logs to storage device 102 in parallel can be optimized, which is conducive to improving the overall performance of data processing system 10.

[0081] Furthermore, in the process of writing multiple log segments to the storage device 102 in parallel, the processor 103 may also record metadata corresponding to each of the written log segments in the storage device 102. Figure 3As shown, before writing multiple log segments to the storage device 102, the processor 103 can first use a dedicated thread to determine multiple first storage areas and multiple second storage areas in the storage device 102, each first storage area is used to store multiple transaction logs included in a log segment, and each second storage area is used to store metadata corresponding to each log segment. The metadata can, for example, describe the storage location of the log segment in the storage device 102, and can also describe the size of the log segment, the sizes corresponding to the multiple transaction logs included in the log segment, and LSN and other information. This embodiment does not limit this.

[0082] Then, the processor 103 may add locks to each first storage area and the second storage area respectively to lock the occupation of each write thread for the first storage area and the second storage area. The multiple transaction logs included in the log segment may be stored in the first storage area in the form of files, such as Figure 3 As shown, each first storage area stores a WAL log file, and the WAL log file records the contents of multiple transaction logs.

[0083] Finally, the processor 103 may execute multiple write threads to write the multiple log segments in parallel to the multiple first storage areas in the storage device 102, and write the metadata corresponding to the multiple log segments in parallel to the multiple second storage areas in the storage device 102. Since the data volume of the metadata is usually small, such as the data volume of the log segment may be 1KB, while the data volume of the metadata may be only 16 bytes, etc., the delay generated by the processor 103 writing the metadata to the storage device 102 can be basically ignored.

[0084] In this way, when data recovery is required based on the transaction log persistently stored in the storage device 102 (such as restarting due to a failure, etc.), the data processing system 10 can verify whether there are holes in the transaction log through the metadata recorded in the second storage area, that is, determine whether part of the transaction log is missing. For example, the processor 103 in the data processing system 10 can detect whether the transaction log is missing by detecting whether there is metadata corresponding to the transaction log. When it is determined that the transaction log is not missing, the processor 103 can achieve data recovery by replaying the transaction logs recorded in the multiple first storage areas. When it is determined that the transaction log is missing (such as the transaction log is missing due to a program error or a failure of the storage device 102, etc.), the processor 103 can recover the data by replaying the transaction logs recorded in the multiple first storage areas. Figure 4As shown, if the metadata corresponding to the log file 3 is not recorded in the storage device 102, the processor 103 can only replay the transaction logs whose LSN is less than the minimum LSN in the WAL log file 3, and the transaction logs whose LSN is greater than or equal to the minimum LSN in the WAL log file 3 are invalid. The minimum LSN refers to the minimum value of the LSNs of multiple transaction logs included in the log file 3.

[0085] S206: The processor 103 wakes up thread A and thread B according to the recorded LSN corresponding to each thread.

[0086] It can be understood that after the multiple transaction logs corresponding to thread A and thread B are persistently stored, the executed transaction is committed and the transaction ends. Therefore, the processor 103 can wake up the dormant thread A and thread B, so that the processor 101 executes thread A and thread B to continue processing the next transaction.

[0087] Illustratively, this embodiment provides the following implementation examples of waking up threads A and B.

[0088] In the first implementation example, when the processor 103 writes multiple transaction logs in the cache 1031 to the storage device 102 in sequence based on a write thread, the processor 103 can continuously monitor the LSN of the transaction log written by the write thread, and compare the LSN with the LSN corresponding to each pre-recorded thread. When the LSN of the written transaction log is greater than the LSN corresponding to thread A, the processor 103 wakes up thread A, otherwise, thread A continues to be in a dormant state; when the LSN of the written transaction log is greater than the LSN corresponding to thread B, the processor 103 wakes up thread B, otherwise, thread B continues to be in a dormant state.

[0089] In this way, by comparing the LSN corresponding to the thread with the LSN of the transaction log that has completed persistent storage, the timing of waking up the thread can be accurately determined, so that there is no need to wait until all transaction logs have completed persistent storage before waking up each thread. In this way, the sleep time of some threads can be effectively shortened, and some threads can be prevented from being in a long-term sleep state and wasting the computing resources of the processor 101, thereby improving the overall efficiency of the processor 101 in executing multiple transactions using threads, thereby improving the overall performance of the data processing system 10. In addition, the processor 103 actively wakes up the thread A and the thread B in the processor 101, and the processor 101 does not need to consume computing resources to determine whether to change the state of each thread. In this way, the resource consumption of the processor 101 can be further saved, thereby helping to further improve the overall performance of the data processing system 10.

[0090] In the second implementation example, when the processor 103 writes multiple transaction logs in the cache 1031 to the storage device 102 in parallel based on multiple write threads, during the process of the processor 103 executing the write thread, after writing the transaction log to the first storage area, a notification of successful transaction log storage will be fed back to the dedicated thread in the processor 103, so that when the processor 103 executes the dedicated thread, the storage status of the transaction log can be obtained based on the notification message fed back by each write thread. In this way, when the processor 103 executes the dedicated thread, according to the storage status of the multiple transaction logs, the target LSN is determined, and the target LSN is the maximum LSN of the transaction logs that have completed the persistent storage continuously. For example, assuming that processor 103 needs to write 100 transaction logs cached in cache 1031 to storage device 102, and the LSNs of the 100 transaction logs are 1 to 100 respectively, in the process of writing the 100 transaction logs to storage device 102 in parallel, processor 103, when executing a dedicated thread, can determine that transaction logs with LSNs of 1 to 68, 75 to 85, 95, and 99 have all completed persistent storage, while the remaining transaction logs have not yet completed storage, based on the storage status of the transaction logs fed back by multiple write threads. Then, processor 103 can determine that the maximum LSN of the transaction logs that have completed persistent storage continuously is 68, that is, determine the target LSN to be 68.

[0091] Then, the processor 103 can execute a dedicated thread to compare the size between the LSN corresponding to thread A and the target LSN, and when the LSN corresponding to thread A is not greater than the target LSN, the processor 103 can wake up thread A by executing the dedicated thread; otherwise, thread A continues to be in a dormant state. Similarly, when the LSN corresponding to thread B is not greater than the target LSN, the processor 103 can wake up thread B by executing the dedicated thread; otherwise, thread B continues to be in a dormant state.

[0092] In this way, in the process of concurrently storing multiple transaction logs, the processor 103 can accurately locate the timing of waking up the thread by comparing the LSN corresponding to the thread with the target LSN, thereby effectively shortening the sleep time of some threads and improving the overall efficiency of the processor 101 in executing multiple transactions using threads, thereby improving the overall performance of the data processing system 10. In addition, the processor 103 actively wakes up the thread A and thread B in the processor 101, which can further save the resource consumption of the processor 101, thereby helping to further improve the overall performance of the data processing system 10. Based on the actual test of the database performance test standard TPC-C, it can be determined that the processor 101 can save at least 30% of computing resources, and the throughput of the data processing system 10 can be improved by more than 30%.

[0093] In actual application, when the processor 103 executes a dedicated thread, it can also use the value of the target LSN to update the global variable LSN in the processor 101, so that the processor 101 determines the LSN corresponding to the transaction log that has completed persistent storage according to the value of the global variable LSN.

[0094] In addition to the above implementation examples, the processor 103 may also wake up each thread based on other implementations. For example, after determining that all transaction logs in the cache 1031 have completed persistent storage, the processor 103 wakes up all threads uniformly, etc. This embodiment does not limit this.

[0095] In a further possible implementation, when the data processing system 10 meets the log playback conditions, such as when the processor 101 restarts after a failure, the processor 103 (or the processor 101) can access the persistently stored transaction log in the storage device 102. For example, the processor 103 can determine the storage address of the transaction log in the storage device 102 based on the metadata recorded in the multiple second storage areas in the storage device 102, so that the processor 103 can read the transaction logs in the multiple first storage areas according to the storage address. Since the amount of metadata recorded in the second storage area is small, the delay caused by the processor 103 accessing the metadata to determine the storage location of the transaction log is very small and can be basically ignored (relative to the delay caused by the processor 103 accessing the transaction log).

[0096] Then, the processor 103 may implement data recovery by replaying the transaction log in the storage device 102 .

[0097] When the processor 103 writes a transaction log to the storage device 102 in the form of an aggregate log, the processor 103 (or the processor 101) may first parse the type of the log after accessing the log in the storage device 102, such as determining the type of the log by parsing the log header. When it is determined that the log is an aggregate log, the processor 103 may first decompose the aggregate log into multiple transaction logs and play back each transaction log one by one.

[0098] Specifically, Figure 5 As shown, when the operation content recorded in the transaction log is a page-related operation content, the processor 103 can allocate the operation content to the queue corresponding to the playback thread, and execute the playback thread in sequence to perform operations on one or more pages in the queue. When the operation content recorded in the transaction log is a table-related operation content, the processor 103 can allocate the operation content to the queue corresponding to the playback thread according to the table granularity, and execute the playback thread in sequence to perform operations on pages corresponding to each data in the table in the queue.

[0099] When the transaction log is a log of modifying a table, such as a data definition language (DDL) log, the processor 103 may distribute the operation contents in the transaction log to queues corresponding to all playback threads, so as to replay the transaction log based on the operations of modifying the table recorded in the transaction log by executing each playback thread.

[0100] When the operation content recorded in the transaction log indicates updating the transaction status, the processor 103 may allocate the operation content to a transaction-related queue, so as to update the transaction status by replaying the transaction log.

[0101] In this embodiment, since the specific implementation process of the processor 103 playing back the transaction log has relevant applications in actual application scenarios, it will not be elaborated here.

[0102] like Figure 5 As shown, when it is determined through analysis that the type of the log is a single transaction log, the processor 103 can allocate the operation content recorded in the transaction log to a corresponding queue to replay the operation content recorded in the transaction log.

[0103] The above-mentioned process of replaying the transaction log is executed by the processor 103. In other possible implementations, the data processing system 10 may include multiple nodes, wherein the processor 101, the storage device 102, and the processor 103 belong to the first node among the multiple nodes, and the other nodes among the multiple nodes serve as disaster recovery nodes of the first node. At this time, when the data processing system 10 meets the log playback conditions, the disaster recovery node in the data processing system 10 may also realize data recovery by replaying the transaction log.

[0104] Taking the case where the data processing system 10 also includes a second node as an example, when the processor 103 persistently stores the transaction log in the cache 1031 to the storage device 102, it will also remotely send the transaction log to the second node for persistent storage. For example, based on the RDMA technology, the processor 103 can remotely send the transaction log to the memory in the second node, and then the second node will persistently store the transaction log in the memory. In this way, when the data processing system 10 meets the log playback conditions, such as the failure of the processor 101 in the first node, the second node can perform log playback based on the transaction log persistently stored by itself, so that the data stored in the second node is restored to the data stored before the failure of the first node, thereby realizing data recovery of the data processing system 10.

[0105] It should be noted that the above Figure 2The data processing method shown is only an exemplary description. In actual application, the specific implementation of the data processing system 10 for persistent storage of transaction logs is not limited to the above example, and can also be implemented in other ways. As some examples, the following three points are listed in this embodiment to illustrate other possible implementations of persistent storage of transaction logs.

[0106] 1. In other possible embodiments, multiple different users can input SQL statements (or other types of statements) to the data processing system 10 and generate multiple different transactions in the data processing system 10. In this case, thread A and thread B can be used to process transactions corresponding to different SQL statements and generate transaction logs. In addition, the number of threads included in the data processing system 10 can be any number, not limited to the above-mentioned thread A and thread B.

[0107] 2. In other possible embodiments, during the process of the processor 101 executing the thread to write multiple transaction logs to the data cache area, the processor 103 can continue to actively pull the transaction logs in the data cache area to the cache 1031. In specific implementation, the processor 103 can continue to monitor whether the data cache area corresponding to each thread stores transaction logs. When transaction logs are stored, the processor 103 directly writes all transaction logs currently stored in the data cache area into the cache 1031. At this time, the processor 101 may still be in the stage of executing the thread to process transactions and generate transaction logs, so that the processor 101 will continue to execute the thread to write new transaction logs (still belonging to the currently executed transaction) into the corresponding data cache area. Correspondingly, the processor 103 can detect that a new transaction log is stored in the data cache area in the next monitoring cycle, and continue to actively write the new transaction log into the cache 1031.

[0108] 3. In other possible embodiments, after the processor 103 writes all the transaction logs in the data cache area into the cache 1031, it is not necessary to record the LSN corresponding to each thread. In addition, after the processor 103 sends all the transaction logs in the cache 1031 to the storage device 102 for persistent storage, since all the transaction logs generated by thread A and thread B have completed persistent storage, the processor 103 can directly wake up thread A and thread B, and it is not necessary to determine whether to wake up the thread by comparing the LSN corresponding to the thread with the LSN of the transaction log that has completed persistent storage.

[0109] Combined with the above Figures 1 to 5 , describes in detail the data processing method provided by this application, and will be combined with Figure 6 , describing a computing device provided according to the present application.

[0110] Figure 6This is a schematic diagram of the structure of a computing device provided in this application. Figure 6 As shown, computing device 600 includes processor 601, memory 602, communication interface 603 and bus 604. Processor 601, memory 602 and communication interface 603 communicate through bus 604. Bus 604 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 The communication interface 603 is used for external communication, for example, receiving jobs (and target data) submitted by users through terminals or clients.

[0111] It should be understood that in the embodiment of the present application, the processor 601 may be a CPU, or may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete device components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0112] The memory 602 may include a read-only memory and a random access memory, and provide instructions and data to the processor 601. The memory 602 may also include a non-volatile random access memory. For example, the memory 602 may also store information about the device type.

[0113] The memory 602 may be a volatile memory or a nonvolatile memory, or may include both volatile and nonvolatile memories. Among them, the nonvolatile memory may be a read-only memory (ROM), a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable programmable ROM (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0114] The memory 602 stores executable codes, and the processor 601 executes the method executed by the aforementioned processor 103 .

[0115] Specifically, in implementing Figure 2 In the case of the embodiment shown, execution Figure 2 The software or program code required for the functions of the processor 103 in the illustrated embodiment is stored in the memory 602 , and the processor 601 is used to execute the instructions in the memory 602 to implement the method executed by the processor 103 .

[0116] It should be understood that the computing device 600 according to the embodiment of the present application may correspond to the execution of the embodiment of the present application. Figure 2 The processor 103 in the method shown, and the above and other operations and / or functions implemented by the computing device 600 are respectively to implement Figure 2 For the sake of brevity, the corresponding processes of each method in are not repeated here.

[0117] In addition, an embodiment of the present application further provides a computer-readable storage medium, which stores instructions. When the computer-readable storage medium is executed on a computing device, the computing device executes the method executed by the processor 103 in the above embodiment.

[0118] In addition, the embodiment of the present application further provides a computer program product, when the computer program product is executed by a computing device, the one or more computing devices execute any of the aforementioned data processing methods. The computer program product may be a software installation package, and when any of the aforementioned data processing methods is required, the computer program product may be downloaded and executed on a computer.

[0119] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0120] The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances. This is just a way of distinguishing objects with the same attributes when describing the embodiments of this application.

[0121] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0122] The above embodiments can be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded or executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium. The semiconductor medium can be a solid state drive (SSD).

[0123] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A data processing system, characterized in that: The data processing system includes a first processor, a storage device, and a second processor, the first processor includes a plurality of threads, each thread corresponds to a data cache area, the first processor is used to execute the plurality of threads to write a plurality of transaction logs into a plurality of data cache areas respectively corresponding to the plurality of threads, and the second processor includes a cache; The second processor is used to obtain the multiple transaction logs from the multiple data cache areas respectively, store the multiple transaction logs in the cache, and persistently store the multiple transaction logs from the cache to the storage device.

2. The data processing system according to claim 1, characterized in that The second processor includes an intelligent network card or a data processing unit DPU.

3. The data processing system according to claim 1 or 2, characterized in that: When the second processor persistently stores the multiple transaction logs from the cache to the storage device, the second processor is specifically configured to: Dividing the multiple transaction logs to obtain multiple log segments, each log segment including at least one transaction log; determining a plurality of first storage areas in the storage device; The multiple log segments are written in parallel into the multiple first storage areas, each first storage area is used to store one log segment.

4. The data processing system according to claim 3, characterized in that: The second processor is further configured to: determining a plurality of second storage areas in the storage device; The metadata corresponding to the multiple log segments are written into the multiple second storage areas in parallel, each second storage area is used to store the metadata corresponding to a log segment, and the metadata corresponding to each log segment is used to describe the storage location of the log segment in the storage device.

5. The data processing system according to claim 3 or 4, characterized in that: After the first processor executes the multiple threads to write the multiple transaction logs into the multiple data buffer areas respectively, the multiple threads are in a dormant state; The second processor is further configured to: Before writing the multiple log segments into the multiple first storage areas in parallel, recording the log sequence numbers LSN corresponding to the multiple threads respectively; Determine a target LSN, where the target LSN is the maximum LSN of the transaction logs that have been continuously completed for persistent storage; Comparing the size between the LSN corresponding to the first thread among the multiple threads and the target LSN; When the LSN corresponding to the first thread is not greater than the target LSN, the first thread is awakened.

6. The data processing system according to any one of claims 1 to 5, characterized in that: The number of the multiple log segments is consistent with the number of data writing channels supported by the storage device.

7. The data processing system according to any one of claims 1 to 6, characterized in that: When the second processor obtains the multiple transaction logs from the multiple data cache areas respectively and stores the multiple transaction logs in the cache, it is specifically used to: Writing multiple transaction logs in each data cache area into the cache at the same time to obtain multiple aggregate logs, each aggregate log including multiple transaction logs in one data cache area; The multiple transaction logs in each data cache area are transaction logs of the same type, or the multiple transaction logs in each data cache area are transaction logs of different types.

8. The data processing system according to claim 7, characterized in that: The second processor is further configured to: When the log playback condition is met, the aggregate log is decomposed into multiple transaction logs, and the multiple transaction logs decomposed from the aggregate log are played back.

9. The data processing system according to claim 7 or 8, characterized in that: The second processor is further configured to: Obtaining a configuration result for the size of the aggregate log; Then, when the second processor writes multiple transaction logs in each data cache area into the cache at the same time, it is specifically used to: According to the configuration result of the size of the aggregate log, multiple transaction logs in each data cache area are simultaneously written into the cache to obtain multiple aggregate logs whose sizes meet the configuration result.

10. The data processing system according to any one of claims 1 to 9, characterized in that: The second processor is further configured to: Based on remote direct memory access (RDMA) technology, the transaction log in the cache is sent to a disaster recovery node for backup.

11. A data processing method, characterized in that: The method is applied to a second processor in a data processing system, the data processing system further comprising a first processor and a storage device, the first processor comprising a plurality of threads, each thread corresponding to a data cache area, the first processor being used to execute the plurality of threads to respectively write a plurality of transaction logs into a plurality of data cache areas respectively corresponding to the plurality of threads, the second processor comprising a cache; the method comprising: The second processor obtains the multiple transaction logs from the multiple data cache areas respectively; The second processor stores the plurality of transaction logs in the cache; The second processor persistently stores the plurality of transaction logs from the cache to the storage device.

12. The method according to claim 11, characterized in that The second processor includes an intelligent network card or a data processing unit DPU.

13. The method according to claim 11 or 12, characterized in that: The second processor persistently stores the multiple transaction logs from the cache to the storage device, including: The second processor divides the multiple transaction logs to obtain multiple log segments, each log segment includes at least one transaction log; The second processor determines a plurality of first storage areas in the storage device; The second processor writes the multiple log segments into the multiple first storage areas in parallel, and each first storage area is used to store one log segment.

14. The method according to claim 13, characterized in that The method further comprises: The second processor determines a plurality of second storage areas in the storage device; The second processor writes metadata corresponding to the multiple log segments respectively into the multiple second storage areas in parallel, each second storage area is used to store metadata corresponding to a log segment, and the metadata corresponding to each log segment is used to describe the storage location of the log segment in the storage device.

15. The method according to claim 13 or 14, characterized in that After the first processor executes the multiple threads to write the multiple transaction logs into the multiple data buffer areas respectively, the multiple threads are in a dormant state; The method further comprises: The second processor records the log sequence numbers LSNs respectively corresponding to the multiple threads before writing the multiple log segments into the multiple first storage areas in parallel; The second processor determines a target LSN, where the target LSN is the maximum LSN of the transaction logs that have been continuously completed in persistent storage; The second processor compares the size between the LSN corresponding to the first thread among the multiple threads and the target LSN; When the LSN corresponding to the first thread is not greater than the target LSN, the second processor wakes up the first thread.

16. The method according to any one of claims 11 to 15, characterized in that The number of the multiple log segments is consistent with the number of data writing channels supported by the storage device.

17. The method according to any one of claims 11 to 16, characterized in that: The second processor obtains the multiple transaction logs from the multiple data cache areas respectively, and the second processor stores the multiple transaction logs in the cache, including: The second processor writes the multiple transaction logs in each data cache area into the cache at the same time to obtain multiple aggregate logs, each aggregate log including the multiple transaction logs in one data cache area; The multiple transaction logs in each data cache area are transaction logs of the same type, or the multiple transaction logs in each data cache area are transaction logs of different types.

18. The method according to claim 17, characterized in that The method further comprises: When a log playback condition is met, the second processor decomposes the aggregate log into multiple transaction logs; The second processor plays back a plurality of transaction logs decomposed from the aggregate log.

19. The method according to claim 17 or 18, characterized in that The method further comprises: The second processor obtains a configuration result for the size of the aggregate log; Then, the second processor writes multiple transaction logs in each data cache area into the cache at the same time, including: The second processor writes multiple transaction logs in each data cache area into the cache at the same time according to the configuration result of the size of the aggregate log, so as to obtain multiple aggregate logs whose sizes meet the configuration result.

20. The method according to any one of claims 11 to 19, characterized in that The method further comprises: The second processor sends the transaction log in the cache to the disaster recovery node for backup based on remote direct memory access (RDMA) technology.

21. A computing device, characterized in that The computing device comprises a processor and a memory, wherein the memory stores program instructions, and the processor runs the program instructions to execute the method according to any one of claims 11 to 20.

22. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, which, when executed on a computing device, enable the computing device to perform the method according to any one of claims 11 to 20.

23. A computer program product comprising instructions which, when executed on a computing device, cause the computing device to perform the method according to any one of claims 11 to 20.

Images