Transaction log processing method, storage medium and equipment

Abstract

The invention relates to a database technology, in particular to a transaction log processing method, a storage medium and equipment. The transaction log processing method comprises the steps of obtaining a to-be-assembled transaction log; judging whether the to-be-assembled transaction log contains page mirror image data or not; if yes, separating the page mirror image data in the to-be-assembled transaction log from the to-be-assembled transaction log to obtain a transaction log after separation of the page mirror image data; and storing the page mirror image data in a pre-created page mirror image file after the transaction log after the page mirror image data separation is subjected to disk falling operation. According to the transaction log processing method, the page mirror image data separation operation is performed on the transaction log containing the page mirror image data in the transaction log assembly stage, so that the data volume of the transaction log is reduced; the problem that flow copying between the main database and the standby database becomes a performance bottleneck of the database when a large number of transaction logs are generated due to large business pressure is avoided, and the overall performance of the database is effectively improved.

Description

Technical Field

[0001] The present invention relates to database technology, and in particular, to a method for processing transaction logs, a storage medium, and a device. Background Art

[0002] In a database, a transaction log actually refers to an XLOG log (or a WAL log). An important function of the XLOG log is to build a primary and standby database cluster. When the primary machine fails, the standby machine can immediately become the primary and take over the external service provided by the primary machine. After the primary and standby database cluster is built, the primary machine provides read and write services externally and continuously generates XLOG logs. The XLOG logs are transmitted to the standby machine through streaming replication. The standby machine continuously receives and replays these XLOGs, so as to be almost consistent with the primary machine's data. If the user hopes that the standby machine's data is strictly consistent with the primary machine's data, then the synchronous streaming replication mode needs to be enabled.

[0003] However, in the synchronous streaming replication mode, after the primary machine sends an XLOG log to the standby machine, it needs to wait for at least one standby machine to store this XLOG log locally or replay it successfully before the primary machine can continue to execute. When the streaming replication mode of the primary and standby database cluster is synchronous, and the primary database generates a large number of XLOG logs due to heavy business pressure, the streaming replication between the primary database and the standby database is likely to become a performance bottleneck of the database. Summary of the Invention

[0004] In view of the above problems, a method for processing transaction logs, a storage medium, and a device that overcome the above problems or at least partially solve the above problems are proposed.

[0005] An object of the present invention is to reduce the data of transaction logs, so as to reduce the pressure of transaction log transmission in the streaming replication between the primary machine and the standby machine.

[0006] A further object of the present invention is to improve the accuracy of the page mirror data separation operation for transaction logs.

[0007] Specifically, the present invention provides a method for processing transaction logs, which includes:

[0008] Obtaining the transaction logs to be assembled;

[0009] Determining whether the transaction logs to be assembled contain page mirror data;

[0010] If so, separating the page mirror data in the transaction logs to be assembled from the transaction logs to be assembled to obtain the transaction logs after page mirror data separation;

[0011] After performing a disk write operation on the transaction logs after page mirror data separation, storing the page mirror data in a pre-created page mirror file.

[0012] Remove the assembled linked list node for storing page mirror data from the first assembled linked list that was pre-created for storing the transaction logs to be assembled.

[0013] Optionally, removing the node for storing page mirror data from the first assembled linked list that was pre-created for storing the transaction logs to be assembled includes:

[0014] Create a page mirror linked list node for each page mirror data in the transaction logs to be assembled, and sequentially connect the page mirror linked list nodes of all the page mirror data in the transaction logs to be assembled to form a page mirror linked list;

[0015] Connect the page mirror linked list to the pre-created global page data linked list, where the global page data linked list is used to store the page mirror data in all the transaction logs generated in the database; and

[0016] Sequentially connect the remaining nodes in the first assembled linked list except the assembled linked list node for storing page mirror data to form a second assembled linked list for storing the transaction logs after the page mirror data is separated.

[0017] Optionally, after performing a disk write operation on the transaction logs after the page mirror data is separated, the step of storing the page mirror data in a pre-created page mirror file includes:

[0018] Obtain the data page to be written to disk;

[0019] Determine whether the global page data linked list contains page mirror data corresponding to the data page to be written to disk;

[0020] If it contains, write the page mirror data corresponding to the data page to be written to disk to the page mirror file;

[0021] Perform a disk write operation on the data page to be written to disk.

[0022] Optionally, the page mirror linked list node includes: page mirror data, the starting position of the transaction log where the page mirror data is located, the position information of the data page corresponding to the page mirror data, and a pointer flag, where the pointer flag is used to point to the next page mirror linked list node; and

[0023] Determining whether the global page data linked list contains page mirror data corresponding to the data page to be written to disk includes:

[0024] Obtain the position information of the data page to be written to disk;

[0025] Determine whether there is a page mirror linked list node in the global page data linked list that has the position information corresponding to the data page to be written to disk;

[0026] If it exists, determine that the global page data linked list contains page mirror data corresponding to the data page to be flushed to disk;

[0027] If it does not exist, determine that the global page data linked list does not contain page mirror data corresponding to the data page to be flushed to disk.

[0028] Optionally, flushing the page mirror data corresponding to the data page to be flushed to disk into a page mirror file includes:

[0029] Name the page mirror file according to the starting position of the transaction log where the page mirror data is located and the position information of the data page corresponding to the page mirror data, so that the page mirror data and the page mirror file are in one-to-one correspondence;

[0030] Flush the page mirror data to its corresponding page mirror file.

[0031] Optionally, after the step of determining whether the global page data linked list contains page mirror data corresponding to the data page to be flushed to disk, the processing method of the transaction log further includes:

[0032] In the case where the global page data linked list does not contain page mirror data corresponding to the data page to be flushed to disk, perform a disk flushing operation on the data page to be flushed to disk; and

[0033] After the step of flushing the page mirror data corresponding to the data page to be flushed to disk into a page mirror file, the processing method of the transaction log further includes:

[0034] Remove the page mirror linked list node of the flushed page mirror data from the global page data linked list.

[0035] Optionally, after the step of obtaining the transaction log after separating the page mirror data, the processing method of the transaction log further includes:

[0036] According to the length of the transaction log after separating the page mirror data, pre-allocate the storage location of the transaction log after separating the page mirror data in the transaction log write cache;

[0037] According to the pre-allocated storage location of the transaction log after separating the page mirror data, copy the transaction log after separating the page mirror data to the transaction log write cache;

[0038] Perform a disk flushing operation on the log page in the transaction log write cache that stores the transaction log after separating the page mirror data.

[0039] According to another aspect of the present invention, there is also provided a machine-readable storage medium, on which a machine-executable program is stored, and when the machine-executable program is executed by a processor, it implements the processing method of the transaction log as described in any one of the above.

[0040] According to another aspect of the present invention, there is also provided a computer device, including a memory, a processor, and a machine-executable program stored on the memory and running on the processor, and when the processor executes the machine-executable program, the processing method of any one of the above transaction logs is implemented.

[0041] In the processing method of the transaction log of the present invention, in the assembly stage of the transaction log, first determine whether the transaction log to be assembled contains page mirror data, and separate the page mirror data in the transaction log to be assembled containing page mirror data from the transaction log to be assembled, obtaining the transaction log after separating the page mirror data, so as to reduce the data volume of the assembled transaction log. After performing the disk write operation on the transaction log after separating the page mirror data, store the page mirror data in a pre-created page mirror file, realizing the separate disk write and independent storage of the page mirror data and the transaction log after separating the page mirror data, so that after the primary and standby database clusters are built, the host only transmits the transaction log after separating the page mirror data to the standby machine, thereby greatly reducing the pressure of transaction log transmission in the streaming replication between the host and the standby machine.

[0042] Further, in the processing method of the transaction log of the present invention, by removing the node for storing the page mirror data from the first assembly linked list for storing the transaction log to be assembled pre-created, the accurate separation of the page mirror data in the transaction log to be assembled from the transaction log to be assembled is realized, improving the accuracy of the operation of separating the page mirror data from the transaction log.

[0043] According to the following detailed description of specific embodiments of the present invention in conjunction with the drawings, those skilled in the art will become more clear about the above and other objects, advantages and features of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Some specific embodiments of the present invention will be described in detail hereinafter with reference to the drawings in an exemplary and non-limiting manner. The same reference numerals in the drawings denote the same or similar components or parts. Those skilled in the art should understand that these drawings are not necessarily drawn to scale. In the drawings:

[0045] Figure 1 is a schematic flowchart of the processing method of the transaction log according to an embodiment of the present invention;

[0046] Figure 2 is a schematic structural diagram of the transaction log in the processing method of the transaction log according to an embodiment of the present invention;

[0047] Figure 3 is a schematic diagram of transaction log transmission in the streaming replication between the host and the standby machine in the processing method of the transaction log according to an embodiment of the present invention;

[0048] Figure 4Schematic diagram of data transmission between a host and a standby machine in a method for processing a transaction log according to an embodiment of the present invention;

[0049] Figure 5 Schematic flowchart of a transaction log writing process in a method for processing a transaction log according to an embodiment of the present invention;

[0050] Figure 6 Schematic diagram of the structure of a sys_wal directory in a method for processing a transaction log according to an embodiment of the present invention;

[0051] Figure 7 Schematic diagram of the conversion from a first assembly linked list to a second assembly linked list in a method for processing a transaction log according to an embodiment of the present invention;

[0052] Figure 8 Schematic diagram of the structure of a global page data linked list in a method for processing a transaction log according to an embodiment of the present invention;

[0053] Figure 9 Schematic diagram of the structure of a page mirror linked list node in a method for processing a transaction log according to an embodiment of the present invention;

[0054] Figure 10 Schematic flowchart of a method for processing a transaction log according to an embodiment of the present invention;

[0055] Figure 11 Schematic diagram of a machine-readable storage medium according to an embodiment of the present invention; and

[0056] Figure 12 Schematic diagram of a computer device according to an embodiment of the present invention. Detailed implementation manners

[0057] Hereinafter, exemplary embodiments of the present invention will be described in more detail with reference to the accompanying drawings. Although the exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided so that this disclosure can be more thoroughly understood and the scope of the present invention can be fully conveyed to those skilled in the art.

[0058] To solve the above technical problems, an embodiment of the present invention proposes a method for processing a transaction log. Figure 1 Schematic flowchart of a method for processing a transaction log according to an embodiment of the present invention, Figure 2 Schematic diagram of the structure of a transaction log in a method for processing a transaction log according to an embodiment of the present invention. As Figure 1 shown, the method for processing a transaction log generally may include:

[0059] Step S102: Obtain the transaction log to be assembled.

[0060] Step S104: Determine whether the transaction log to be assembled contains page mirror data. If so, execute Step S106.

[0061] Step S106: Separate the page mirror data in the transaction log to be assembled from the transaction log to be assembled, obtaining the transaction log after separating the page mirror data.

[0062] Step S108: After performing a disk write operation on the transaction log after separating the page mirror data, store the page mirror data in a pre-created page mirror file.

[0063] In a database, the transaction log refers to the XLOG log, and the XLOG log details the operation process of the service process on the database. During the operation of the database, multiple XLOG logs are generated. As Figure 2 shown, an XLOG log can include an XLOG log header (denoted as head data) and an XLOG log data area. Specifically, the XLOG log data area can include multiple block data areas (denoted as block) and main data (denoted as main data), and each block can contain page mirror data (denoted as page data) and tuple data (denoted as tupledata). The XLOG log header includes the XLogRecord structure, the header data of each block, and the header data of the main data.

[0064] It should be noted that different types of XLOG logs have different compositions. Each XLOG log contains head data, but not necessarily page data, tupledata, and main data. Some types of XLOG logs only have head data and no XLOG log data area; some other types of XLOG logs only have head data and main data; and some other types of XLOG logs only have head data, page data, and tupledata, and each block may contain both page data and tupledata, or may only contain tupledata. That is to say, the XLOG log obtained in Step S102 may contain page data or may not contain page data.

[0065] During the operating system crash of the database, some operating system pages (e.g., with a page size of 4KB) may not have been written to the disk in time, which may result in a database page (e.g., composed of two operating system pages) containing a mixture of old and new data. During the recovery after the operating system crash of the database, since the information stored in the XLOG log is not complete enough, it is impossible to fully recover the database pages containing a mixture of old and new data in the database. In the prior art, in order to recover such database pages, after the redo point of the checkpoint (checkpoint log), when the database page (with a default size of 8KB) is modified (updated) for the first time, the entire database page is written to the XLOG log. At this time, the XLOG log contains page mirror data to save the complete database page mirror. It should be noted that the redo point is a special point in the XLOG log. Before this point, all the data in the database is the same as the information reflected by the disk-written XLOG log. During the creation of the checkpoint, a special XLOG log (denoted as the checkpoint log) is created. A redo point is recorded in this checkpoint log. Only when the XLOG log and user data before this redo point are both written to the disk, the checkpoint can be successfully created. When the database crashes, we can find a redo point from the nearest checkpoint log, and then replay the XLOG log from this redo point to recover the database. When encountering a broken page during the recovery of the database, the broken page is replaced according to the page mirror data in the XLOG log, thus ensuring the security and stability of the database data. Therefore, the page mirror data in the XLOG log is the same as the data of the database page, and the data volume of the XLOG log containing the page mirror data is relatively large.

[0066] For the processing method of the transaction log in this embodiment, during the assembly stage of the XLOG log, first determine whether the XLOG log to be assembled contains page mirror data, and separate the page mirror data in the XLOG log to be assembled containing page mirror data from the XLOG log to be assembled, obtaining the XLOG log after the separation of the page mirror data, which greatly reduces the data volume of the assembled XLOG log. After performing the disk write operation on the XLOG log after the separation of the page mirror data, store the page mirror data in a pre-created page mirror file, realizing the separate disk write and independent storage of the page mirror data and the XLOG log after the separation of the page mirror data, so that after the primary and standby database clusters are built, the host only transmits the XLOG log after the separation of the page mirror data to the standby machine, thereby greatly reducing the pressure of XLOG log transmission in the streaming replication between the host and the standby machine on the premise of ensuring the security and stability of the database data.

[0067] Figure 3It is a schematic diagram of transaction log transmission in the streaming replication between the host and the standby in the transaction log processing method according to an embodiment of the present invention. As Figure 3 shown, during the streaming replication between the host 10 and the standby 20, the host 10 only transmits XLOG logs without page mirror data to the standby 20.

[0068] Specifically, after the primary-standby database cluster is built, the host 10 provides read and write services externally and continuously generates XLOG logs. During the streaming replication between the host 10 and the standby 20, the host 10 only transmits the XLOG logs after separating the page mirror data to the standby 20, thus greatly reducing the streaming replication pressure of the primary-standby cluster.

[0069] Figure 4 It is a schematic diagram of data transmission between the host and the standby in the transaction log processing method according to an embodiment of the present invention. As Figure 4 shown, if a bad block 21 is found during the playback on the standby 20, a signal requesting a good block is sent to the host 10. After obtaining the good block 11 corresponding to the bad block 21 in the host 10, the good block 11 corresponding to the bad block is returned to the standby 20, and the bad block 21 in the standby 20 is updated according to the data in the returned good block 11 to ensure the data consistency of the standby 20.

[0070] Specifically, after the primary-standby database cluster is built, if a page break is found during the playback on the standby 20, a request is sent to the host 10 to obtain the corresponding page mirror data, and the broken database page is replaced according to the page mirror data, thereby ensuring the data consistency between the standby 20 and the host 10 while ensuring the streaming replication efficiency.

[0071] To more clearly illustrate the technical effect of selecting the stage of assembling XLOG log data as the timing for performing the page mirror data separation operation on the XLOG log in this embodiment, the following is combined with Figure 5 , to describe the XLOG log writing process. Figure 5 It is a schematic flowchart of the XLOG log writing process in the transaction log processing method according to an embodiment of the present invention. As Figure 5 shown, the XLOG log writing process generally may include:

[0072] Step S502, generating XLOG log data. The final data content in the XLOG log cannot be fully determined during the stage of generating the XLOG log. Therefore, this stage is not suitable for performing the page mirror data separation operation on the XLOG log.

[0073] Step S504, registering the XLOG log data. The final data content in the XLOG log also cannot be fully determined during the stage of registering the XLOG log. Therefore, this stage is not suitable for performing the page mirror data separation operation on the XLOG log.

[0074] Step S506, assemble the XLOG log data. In the stage of assembling the XLOG log data, the final data content in the XLOG log can be completely determined, and the page mirror data in the XLOG log can be conveniently obtained at this stage. In addition, at this time, the position of the XLOG log to be written in the XLOG log write cache has not been determined. It only needs to calculate the total length of the XLOG log after the page mirror data is separated, and pre-allocate the storage position of the XLOG log in the XLOG log write cache according to this length, so as to eliminate the influence of the change of the XLOG log storage position information caused by the reduction of the data size after the XLOG log page mirror data is separated. Therefore, this stage is suitable for performing the page mirror data separation operation on the XLOG log.

[0075] Step S508, pre-allocate the storage position of the XLOG log in the XLOG log write cache according to the length of the XLOG log. In this process, each process writing the XLOG log needs to obtain an exclusive lock to prevent the positions pre-allocated by each process on the XLOG log write cache from overlapping. If the page mirror data separation operation is performed during this process, the page mirror data separation tasks of each process will be executed serially, resulting in serious performance problems. Therefore, this stage is not suitable for performing the page mirror data separation operation on the XLOG log.

[0076] Step S510, put the data of each part of the XLOG log into a continuous range. At this stage, the length of the XLOG log has been determined, and it is not suitable to perform the page mirror data separation operation on the XLOG log.

[0077] Step S512, copy the XLOG data in the continuous range to the XLOG log write cache. At this stage, the length of the XLOG log has also been determined, and it is not suitable to perform the page mirror data separation operation on the XLOG log.

[0078] Step S514, write the log pages storing the XLOG log in the XLOG log write cache to the disk. At this stage, the length of the XLOG log has also been determined, and it is not suitable to perform the page mirror data separation operation on the XLOG log.

[0079] On this basis, after the above step S106, the method for processing the transaction log of the present invention may further include the following steps: pre-allocate the storage position of the XLOG log after the page mirror data is separated in the XLOG log write cache according to the length of the XLOG log after the page mirror data is separated; copy the XLOG log after the page mirror data is separated to the XLOG log write cache according to the pre-allocated storage position of the XLOG log after the page mirror data is separated; perform a disk write operation on the log pages storing the XLOG log after the page mirror data is separated in the XLOG log write cache.

[0080] In the processing method of the transaction log of this embodiment, it is selected to separate the page mirror data in the to-be-assembled XLOG log containing page mirror data from the to-be-assembled XLOG log during the assembly stage of the XLOG log, and determine the storage location information of the XLOG log after it is copied to the XLOG log write cache according to the data length of the XLOG log after the page mirror data, avoiding the change of the storage location information caused by the change of the length of the XLOG log, and maintaining the stability of the storage location information of the XLOG log in the XLOG log write cache.

[0081] Figure 6 It is a schematic diagram of the structure of the sys_wal directory in the processing method of the transaction log according to an embodiment of the present invention. As Figure 6 shown, a wal_pagedata directory 40 for storing the page mirror data separated from the XLOG log is created under the sys_wal directory 30.

[0082] In this embodiment, before the above step S102, the processing method of the transaction log of the present invention further includes the following steps: a wal_pagedata directory 40 for storing the page mirror data separated from the XLOG log is pre-created under the sys_wal directory 30. Specifically, a corresponding storage file 41 is created under the wal_pagedata directory 40 for each page mirror data to be retained in the XLOG log. That is to say, each page mirror data and its related data are stored using a separate storage file 41. In addition, in the database, the newly generated XLOG log files are stored under the sys_wal directory 30, and each storage file 41 and the XLOG log file have separate storage spaces.

[0083] Figure 7 It is a schematic diagram of the conversion of the first assembly linked list to the second assembly linked list in the processing method of the transaction log according to an embodiment of the present invention. As Figure 7 shown, the above step S106 may include the following steps: removing the assembly linked list node for storing the page mirror data from the first assembly linked list 51 pre-created for storing the to-be-assembled XLOG log.

[0084] In this embodiment, during the stage of assembling XLOG log data, first, a first assembly linked list 51 is created in memory. Each assembly linked list node in the first assembly linked list 51 stores data of different parts of the XLOG log, which can be head data, page mirror data, tuple data, and main data. Secondly, the assembly linked list nodes used to store page mirror data are removed from the first assembly linked list 51. The page mirror data in the removed assembly linked list nodes remains in memory, and the corresponding header information of the page mirror data remains in the head data in the XLOG log. After separating all the page mirror data in a single XLOG log, in memory, the page mirror data separated from each single XLOG log is organized in the form of a linked list, and the nodes of all the page mirror data in a single XLOG log are connected to form a page mirror linked list. At this time, as Figure 7 shown, the remaining nodes in the first assembly linked list 51 except for the assembly linked list nodes used to store page mirror data are sequentially connected to form a second assembly linked list 52.

[0085] The method for processing transaction logs in this embodiment completes the separation operation of the page mirror data of the XLOG log to be assembled by removing the nodes used to store page mirror data from the first assembly linked list 51 created in advance for storing the XLOG log to be assembled, improving the accuracy of the separation operation of the page mirror data of the XLOG log.

[0086] Figure 8 It is a schematic structural diagram of the global page data linked list in the method for processing transaction logs according to an embodiment of the present invention, which is a simple schematic diagram of the global page data linked list storing two page mirror linked lists. page_data_list_local represents the page mirror linked list of a database log, and page_data_list_global represents the global page data linked list. Among them, wal1 and wal2 are only used to indicate that the XLOG logs to which the two page mirror linked lists belong are different.

[0087] In this embodiment, after the nodes of all the page mirror data in a single XLOG log are connected to form a page mirror linked list 70, the method for processing transaction logs of the present invention may further include the following steps: organizing all the page mirror data separated from the XLOG logs in the database. Specifically, a linked list for storing the page mirror data separated from all the XLOG logs generated by the database is created in advance in the shared memory, denoted as page_data_list_global. That is to say, the global page data linked list 60 stores all the page mirror data separated from the XLOG logs in the database.

[0088] In one embodiment, the step of removing the node for storing page mirror data from the first assembly linked list 51 for storing the XLOG logs to be assembled created in advance may include the following steps: creating a page mirror linked list node 71 for each page mirror data in the XLOG logs to be assembled, and sequentially connecting the page mirror linked list nodes 71 of all the page mirror data in the XLOG logs to be assembled to form a page mirror linked list 70; connecting the page mirror linked list 70 to the globally created page data linked list 60, where the globally created page data linked list 60 is used to store the page mirror data in all the XLOG logs generated in the database; and sequentially connecting the remaining nodes in the first assembly linked list 51 except the assembly linked list nodes for storing page mirror data to form a second assembly linked list 52 for storing the XLOG logs after the separation of the page mirror data.

[0089] In the transaction log processing method of this embodiment, by creating a page mirror linked list node 71 for each page mirror data in the XLOG logs to be assembled, sequentially connecting the page mirror linked list nodes 71 of all the page mirror data in the XLOG logs to be assembled to form a page mirror linked list 70, connecting the page mirror linked list 70 to the globally created page data linked list 60, and sequentially connecting the remaining nodes in the first assembly linked list 51 except the assembly linked list nodes for storing page mirror data to form a second assembly linked list 52, the operation of separating the page mirror data from the XLOG logs to be assembled is completed, further improving the accuracy of the operation of separating the page mirror data from the XLOG logs.

[0090] In a specific embodiment, after the step of sequentially connecting the page mirror linked list nodes 71 of all the page mirror data in the XLOG logs to be assembled to form a page mirror linked list 70, the transaction log processing method of the present invention may further include the following steps: connecting the page mirror linked list 70 after the remaining nodes in the first assembly linked list 51 except the assembly linked list nodes for storing page mirror data.

[0091] That is to say, after sequentially connecting the page mirror linked list nodes 71 of all the page mirror data in the XLOG logs to be assembled to form a page mirror linked list 70, first connect the page mirror linked list 70 to the end of the first assembly linked list 51, then remove the page mirror linked list 70 behind the first assembly linked list 51 and connect it to the globally created page data linked list 60.

[0092] The transaction log processing method of this embodiment can orderly adjust the nodes of the page mirror data, enabling the data to be effectively managed and maintained globally, facilitating data sharing and consistency maintenance, and improving the stability of the database.

[0093] In some embodiments, step S108 may include the following steps: obtaining a data page to be disk-written; determining whether the global page data linked list 60 contains page mirror data corresponding to the data page to be disk-written; if it contains, disk-writing the page mirror data corresponding to the data page to be disk-written into a page mirror file; and performing a disk-writing operation on the data page to be disk-written.

[0094] In addition, after the step of determining whether the global page data linked list 60 contains page mirror data corresponding to the data page to be disk-written, the method for processing a transaction log of the present invention may further include the following steps: in the case where the global page data linked list 60 does not contain page mirror data corresponding to the data page to be disk-written, performing a disk-writing operation on the data page to be disk-written. In addition, after the step of disk-writing the page mirror data corresponding to the data page to be disk-written into a page mirror file, the method for processing a transaction log of the present invention may further include the following steps: removing the page mirror linked list node 71 of the disk-written page mirror data from the global page data linked list 60.

[0095] The method for processing a transaction log of this embodiment, by first disk-writing the page mirror data corresponding to the data page to be disk-written before disk-writing the data page, even if a page break occurs during the disk-writing of the data page, it can be repaired through the already disk-written page mirror file, ensuring the data consistency between the standby machine 20 and the host machine 10.

[0096] Figure 9 It is a schematic structural diagram of a page mirror linked list node in the method for processing a transaction log according to an embodiment of the present invention. As Figure 9 shown, the page mirror linked list node 71 may include: page mirror data, the starting position (denoted as lsn) of the XLOG log where the page mirror data is located, the position information of the data page corresponding to the page mirror data, and a pointer identifier (denoted as next).

[0097] In a specific embodiment, a data page refers to a data disk page. The location information of the data page corresponding to the page mirror data includes RelFileNode and BlockNumber. RelFileNode is the location information of the table where the data disk page corresponding to the page mirror data is located, and BlockNumber is the block number of the data disk page corresponding to the page mirror data in its table. RelFileNode and BlockNumber together determine the location of the data disk page in the disk, and based on these two data, a unique data disk page can be determined in the disk. When searching in the disk, these two values can be used to determine whether a version of the target page mirror data is found. The lsn corresponding to each page mirror data increases sequentially. next is used to point to the next page mirror linked list node 71 to connect the page mirror linked lists 70 corresponding to two adjacent XLOG logs.

[0098] In some embodiments, the step of determining whether the global page data linked list 60 contains page mirror data corresponding to the data page to be flushed to disk may include the following steps: obtaining the location information of the data page to be flushed to disk; determining whether there is a page mirror linked list node in the global page data linked list that has the same location information as the data page to be flushed to disk; if so, determining that the global page data linked list contains page mirror data corresponding to the data page to be flushed to disk; if not, determining that the global page data linked list does not contain page mirror data corresponding to the data page to be flushed to disk.

[0099] Specifically, the step of determining whether the global page data linked list 60 contains page mirror data corresponding to the data page to be flushed to disk may be specifically executed as follows: obtaining RelFileNode and BlockNumberpage of the data page to be flushed to disk; determining whether there is a page mirror linked list node 71 in the global page data linked list 60 that has RelFileNode and BlockNumberpage; if so, determining that the global page data linked list 60 contains page mirror data corresponding to the data page to be flushed to disk; if not, determining that the global page data linked list 60 does not contain page mirror data corresponding to the data page to be flushed to disk. That is to say, in the stage of persisting the page of the data disk, it can be checked whether each data page to be flushed to disk has corresponding page mirror data in the global page data linked list 60 through RelFileNode and BlockNumber.

[0100] The method for processing transaction logs in this embodiment utilizes the correspondence between page mirror data and data pages, improving the convenience and accuracy of searching for page mirror data corresponding to the data pages to be flushed to disk in the global page data linked list 60, and improving the processing speed in the stage of persisting the page of the data disk.

[0101] In some embodiments, the step of flushing the page mirror data corresponding to the data page to be flushed to the page mirror file may include the following steps: naming the page mirror data file according to the starting position of the XLOG log where the page mirror data is located and the position information of the data page corresponding to the page mirror data, so that the page mirror data and the page mirror data file are in one-to-one correspondence; flushing the page mirror data to its corresponding page mirror file.

[0102] Specifically, the step of naming the page mirror data file according to the starting position of the XLOG log where the page mirror data is located and the position information of the data page corresponding to the page mirror data may be specifically executed as: naming the page mirror file as RelFileNode_BlockNumber_lsn according to the RelFileNode, BlockNumber, and lsn corresponding to the page mirror data.

[0103] The method for processing the transaction log of this embodiment uses the uniqueness of the correspondence between the two data, RelFileNode and BlockNumber, and the data page, and the characteristic that the lsn corresponding to each page mirror data is increasing, names the page mirror file as RelFileNode_BlockNumber_lsn, ensuring that the names of all page mirror files are different, improving the stability of storing page mirror data and the accuracy of finding page mirror files.

[0104] Figure 10 It is a schematic flowchart of the method for processing the transaction log according to an embodiment of the present invention. The following combines Figure 10 Specifically describe the flow steps of the method for processing the transaction log of this embodiment.

[0105] Step S1002, obtain the XLOG log to be assembled.

[0106] Step S1004, determine whether the XLOG log to be assembled contains page data. If so, execute step S1006; if not, execute step S1020.

[0107] Step S1006, create a page_data_list_local linked list node for each page data in the XLOG log to be assembled, and sequentially connect the page_data_list_local linked list nodes of all page data in the XLOG log to be assembled to form a page_data_list_local linked list;

[0108] Step S1008, the remaining nodes in the first assembly linked list 51 except for the assembly linked list nodes used to store page data are sequentially connected to form a second assembly linked list 52, and the page_data_list_local linked list is connected after the second assembly linked list 52 of the XLOG log to be assembled.

[0109] Step S1010, according to the length of the XLOG log after page data separation, pre-allocate the storage location of the XLOG log after page data separation in the XLOG log write cache. It should be noted that the XLOG log stored in the second assembly linked list 52 is the XLOG log after page data separation.

[0110] Step S1012, move the page_data_list_local linked list after the second assembly linked list 52 out and connect it to the page_data_list_global linked list. It should be noted that the page_data_list_global is pre-created in the shared memory and is used to store the page data in all XLOG logs generated in the database.

[0111] Step S1014, put the data of the XLOG log after page data separation into a continuous cache area.

[0112] Step S1016, according to the pre-allocated storage location of the XLOG log after page data separation, copy the content in the continuous cache area to the XLOG log write cache.

[0113] Step S1018, perform a disk write operation on the page in the XLOG log write cache that stores the XLOG log after page data separation.

[0114] Step S1020, in the page stage of the persistent data disk, obtain the data disk page to be written to disk.

[0115] Step S1022, determine whether the page_data_list_global linked list contains the page data corresponding to the data disk page to be written to disk. If so, execute Step S1024; if not, execute Step S1030.

[0116] Step S1024, name the pagedata file as the RelFileNode_BlockNumber_lsn file according to the RelFileNode, BlockNumber, and lsn corresponding to the page data.

[0117] Step S1026, write the page data to the RelFileNode_BlockNumber_lsn file.

[0118] Step S1028, remove the page_data_list_local linked list node of the page data after writing from the page_data_list_global linked list.

[0119] Step S1030, perform the write operation on the data disk page to be written. This process ends.

[0120] Using the above method, in the assembly stage of the XLOG log, first determine whether the XLOG log to be assembled contains page data, and separate the page data in the XLOG log to be assembled that contains page data from the XLOG log to be assembled, obtaining the XLOG log after separating the page data, so as to reduce the data volume of the assembled XLOG log. After performing the write operation on the XLOG log after separating the page data, before performing the write operation on the data disk page to be written, store the page data in a pre-created page data file, realizing the separate write and independent storage of the page data and the XLOG log after separating the page data. After the primary and standby database clusters are built, the host 10 only transmits the XLOG log after separating the page data to the standby 20, thus greatly reducing the pressure of XLOG log transmission in the streaming replication between the host 10 and the standby 20.

[0121] This embodiment also provides a machine-readable storage medium and a computer device. Figure 11 It is a schematic diagram of a machine-readable storage medium 80 according to an embodiment of the present invention. Figure 12 It is a schematic diagram of a computer device 90 according to an embodiment of the present invention.

[0122] The machine-readable storage medium 80 stores a machine-executable program 81 thereon. When the machine-executable program 81 is executed by a processor, it implements the processing method of the transaction log in any of the above embodiments.

[0123] The computer device 90 may include a memory 920, a processor 910, and a machine-executable program 81 stored on the memory 920 and running on the processor 910. When the processor 910 executes the machine-executable program 81, it implements the processing method of the transaction log in any of the above embodiments.

[0124] It should be noted that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a definite sequence list of executable instructions for implementing logical functions, and can be specifically implemented in any machine-readable storage medium for use by an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other systems that can fetch and execute instructions from the instruction execution system, apparatus, or device), or in combination with these instruction execution systems, apparatuses, or devices.

[0125] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For the description of this embodiment, the machine-readable storage medium 80 can be any device that can contain, store, communicate, propagate, or transmit a program for use by an instruction execution system, apparatus, or device or in combination with these instruction execution systems, apparatuses, or devices. The computer device 90 can be, for example, a server, a desktop computer, a laptop computer, a tablet computer, or a smart phone. The computer device 90 can include a processor 910 adapted to execute the stored instructions and a memory 920 that provides temporary storage space for the operation of the instructions during operation. The processor 910 can be a single-core processor, a multi-core processor, a computing cluster, or any other number of other configurations. The memory 920 can include random access memory (RAM), read-only memory, flash memory, or any other suitable storage system.

[0126] It should be noted that in some alternative embodiments, the solution of the present invention is applicable to relational databases, and particularly applicable to the KingbaseES database (abbreviated as KES database), which enriches the functions of the database and improves the efficiency of the database. In some other alternative embodiments, the method for processing transaction logs of the present invention can also be applicable to other relational databases.

[0127] In addition, the flowchart provided in this embodiment is not intended to indicate that the operations of the method will be executed in any specific order, or that all operations of the method are included in every case. Moreover, the method can include additional operations. Within the scope of the technical concept provided by the method of this embodiment, additional changes can be made to the above method.

[0128] So far, those skilled in the art should recognize that although multiple exemplary embodiments of the present invention have been shown and described in detail herein, many other variations or modifications that conform to the principles of the present invention can still be directly determined or derived from the content disclosed in the present invention without departing from the spirit and scope of the present invention. Therefore, the scope of the present invention should be understood and determined to cover all these other variations or modifications.

Claims

1. A method for processing a transaction log, comprising: Obtaining a transaction log to be assembled; Determining whether the transaction log to be assembled contains page mirror data; If so, separating the page mirror data in the transaction log to be assembled from the transaction log to be assembled to obtain a transaction log after separation of the page mirror data; After performing a disk write operation on the transaction log after separation of the page mirror data, storing the page mirror data in a pre-created page mirror file.

2. The method for processing a transaction log according to claim 1, wherein Separating the page mirror data in the transaction log to be assembled from the transaction log to be assembled includes: Removing an assembly linked list node for storing the page mirror data from a first assembly linked list pre-created for storing the transaction log to be assembled.

3. The method for processing a transaction log according to claim 2, wherein Removing the node for storing the page mirror data from the first assembly linked list pre-created for storing the transaction log to be assembled includes: Creating a page mirror linked list node for each piece of page mirror data in the transaction log to be assembled, and sequentially connecting the page mirror linked list nodes of all the page mirror data in the transaction log to be assembled to form a page mirror linked list; Connecting the page mirror linked list to a pre-created global page data linked list for storing the page mirror data in all the transaction logs generated in the database; and Sequentially connecting the remaining nodes in the first assembly linked list except the assembly linked list node for storing the page mirror data to form a second assembly linked list for storing the transaction log after separation of the page mirror data.

4. The method for processing a transaction log according to claim 3, wherein The step of storing the page mirror data in a pre-created page mirror file after performing a disk write operation on the transaction log after separation of the page mirror data includes: Obtaining a data page to be written to disk; Determining whether the global page data linked list contains the page mirror data corresponding to the data page to be written to disk; If it contains, writing the page mirror data corresponding to the data page to be written to disk to the page mirror file; Performing a disk write operation on the data page to be written to disk.

5. The method for processing a transaction log according to claim 4, wherein The page mirror linked list node includes: page mirror data, the starting position of the transaction log where the page mirror data is located, the position information of the data page corresponding to the page mirror data, and a pointing identifier, wherein the pointing identifier is used to point to the next page mirror linked list node; and Determining whether the global page data linked list contains the page mirror data corresponding to the data page to be written to disk includes: Obtaining the position information of the data page to be written to disk; Determining whether there is a page mirror linked list node in the global page data linked list having the position information corresponding to the data page to be written to disk; If it exists, determining that the global page data linked list contains the page mirror data corresponding to the data page to be written to disk; If not, it is determined that the global page data linked list does not contain the page mirror data corresponding to the data page to be flushed to disk.

6. The method for processing a transaction log according to claim 5, wherein, Flushing the page mirror data corresponding to the data page to be flushed to disk into the page mirror file includes: Naming the page mirror file according to the starting position of the transaction log where the page mirror data is located and the position information of the data page corresponding to the page mirror data, so that the page mirror data and the page mirror file are in one-to-one correspondence; Flushing the page mirror data into the corresponding page mirror file.

7. The method for processing a transaction log according to claim 4, wherein, After the step of determining whether the global page data linked list contains the page mirror data corresponding to the data page to be flushed to disk, the method for processing the transaction log further includes: In the case where the global page data linked list does not contain the page mirror data corresponding to the data page to be flushed to disk, performing a disk flushing operation on the data page to be flushed to disk; and After the step of flushing the page mirror data corresponding to the data page to be flushed to disk into the page mirror file, the method for processing the transaction log further includes: Removing the page mirror linked list node of the flushed page mirror data from the global page data linked list.

8. The method for processing a transaction log according to claim 1, wherein, After the step of obtaining the transaction log after separating the page mirror data, the method for processing the transaction log further includes: Pre-allocating a storage location for the transaction log after separating the page mirror data in the transaction log write cache according to the length of the transaction log after separating the page mirror data; Copying the transaction log after separating the page mirror data to the transaction log write cache according to the pre-allocated storage location for the transaction log after separating the page mirror data; Performing a disk flushing operation on the log page in the transaction log write cache that stores the transaction log after separating the page mirror data.

9. A machine-readable storage medium, on which a machine-executable program is stored, and when the machine-executable program is executed by a processor, the method for processing a transaction log according to any one of claims 1 to 8 is implemented.

10. A computer device, including a memory, a processor, and a machine-executable program stored on the memory and running on the processor, and when the processor executes the machine-executable program, the method for processing a transaction log according to any one of claims 1 to 8 is implemented.

Images