Transaction log processing method, storage medium and equipment

Abstract

The invention relates to a database technology, in particular to a transaction log processing method, a storage medium and equipment. The transaction log processing method comprises the following steps: creating a global page data linked list for storing page mirror image data separated from all transaction logs generated in a database; executing at least one transaction log disk-dropping operation, each transaction log disk-dropping operation comprising: acquiring at least one piece of page mirror image data corresponding to the to-be-dropped transaction log from the global page data chain table according to the serial number range of the to-be-dropped transaction log; placing at least one piece of corresponding page mirror image data; and the transaction logs to be loaded are loaded. According to the transaction log processing method, the at least one piece of page mirror image data corresponding to the transaction log to be stored is stored, so that the storage frequency of the page mirror image data is reduced, the problem of disk IO randomization caused by storage of a large amount of small-size page mirror image data is avoided, and the storage efficiency of the page mirror image data is improved.

Description

Technical Field

[0001] The present invention relates to database technology, and particularly to a method for processing transaction logs, a storage medium, and a device. Background Art

[0002] In a database, a transaction log actually refers to an XLOG log (or a WAL log). An important function of the XLOG log is to construct a primary and standby database cluster, so that in the event of the failure of the primary machine, the standby machine can be promoted to the primary machine and continue to provide services externally on behalf of the primary machine. After the primary and standby database cluster is constructed, the primary machine provides read and write services externally and continuously generates XLOG logs. The XLOG logs are transmitted to the standby machine through streaming replication. The standby machine continuously receives and replays these XLOGs, so as to be almost consistent with the primary machine data. If a user hopes that the standby machine data is strictly consistent with the primary machine data, then the synchronous streaming replication mode needs to be enabled.

[0003] In the synchronous streaming replication mode, after the primary machine sends an XLOG log to the standby machine, it needs to wait for at least one standby machine to store this XLOG log locally or replay it successfully before the primary machine can continue to execute. When the streaming replication mode of the primary and standby database cluster is synchronous and the primary machine generates a large number of XLOG logs due to heavy business pressure, the streaming replication between the primary machine and the standby machine is likely to become a performance bottleneck of the database. Since the page mirror data in the XLOG log stores the page data of the database page and the data volume is large, separating the page mirror data in the XLOG log of the database can greatly reduce the data volume of the XLOG log.

[0004] However, when the page mirror data separated from the XLOG log is written to disk, a corresponding page mirror file will be created for each page mirror data. In actual use, the number of page mirror data in the XLOG log is extremely large, and the writing of a large number of small-sized page mirror files to disk will cause disk I / O randomization.

[0005] The above information disclosed in this background art is only used to increase the understanding of the background art of the present application. Therefore, it may include prior art that is not known to those of ordinary skill in the art. Summary of the Invention

[0006] In view of the above problems, a method for processing transaction logs, a storage medium, and a device that overcome the above problems or at least partially solve the above problems are proposed.

[0007] An object of the present invention is to reduce the number of times the page mirror files are written to disk, so as to improve the disk writing efficiency of the page mirror data separated from the transaction logs.

[0008] A further object of the present invention is to improve the accuracy of obtaining page mirror data to be flushed in the transaction log flushing operation.

[0009] In particular, the present invention provides a method for processing transaction logs, which includes:

[0010] Create a global page data linked list, which is used to store page mirror data separated from all transaction logs generated from the database;

[0011] Execute at least one transaction log flushing operation, and each transaction log flushing operation further includes:

[0012] Obtain the sequence number range of the transaction log to be flushed;

[0013] According to the sequence number range of the transaction log to be flushed, obtain at least one page mirror data corresponding to the transaction log to be flushed in the global page data linked list;

[0014] Flush at least one page mirror data corresponding to the transaction log to be flushed;

[0015] Flush the transaction log to be flushed.

[0016] Optionally, the global page data linked list includes multiple connected page mirror linked lists, each page mirror linked list includes at least one page mirror linked list node, each page mirror linked list node corresponds to a page mirror data, and the page mirror linked list node includes a page mirror data area and a pointing identifier. The page mirror data area includes at least the page mirror data and the starting position of the transaction log where the page mirror data is located. The pointing identifier is used to point to the next page mirror linked list node; and

[0017] According to the sequence number range of the transaction log to be flushed, obtaining at least one page mirror data corresponding to the transaction log to be flushed in the global page data linked list includes:

[0018] Move the page mirror linked list nodes whose starting positions of the transaction logs where the page mirror data is located are within the sequence number range of the transaction log to be flushed from the global page data linked list to a temporary linked list. The page mirror data in the temporary linked list is at least one page mirror data corresponding to the transaction log to be flushed.

[0019] Optionally, flushing at least one page mirror data corresponding to the transaction log to be flushed includes:

[0020] Sequentially obtain all page mirror data areas from the temporary linked list, and connect the sequentially obtained page mirror data areas to form a continuous data area;

[0021] Write the data of the continuous data area into the page mirror file sequentially.

[0022] Optionally, the page mirror data area further includes the total length of the page mirror data area; and

[0023] Sequentially obtaining all page mirror data areas from the temporary linked list includes:

[0024] Sequentially obtaining each page mirror data area in the temporary linked list according to the total length of the page mirror data area.

[0025] Optionally, the page mirror file includes an occupied interval and a to-be-written-to-disk interval;

[0026] Sequentially writing the data in the continuous data area into the page mirror file includes:

[0027] Sequentially writing the data in the continuous data area to a position adjacent to the occupied interval in the to-be-written-to-disk interval of the page mirror file; and

[0028] Each page mirror data area in the page mirror file is arranged sequentially and continuously.

[0029] Optionally, moving the page mirror linked list nodes whose starting position of the transaction log where the page mirror data is located is within the serial number range of the to-be-written-to-disk transaction log from the global page data linked list to the temporary linked list includes:

[0030] Moving the page mirror linked list nodes whose starting position of the transaction log where the page mirror data is located is within the serial number range of the to-be-written-to-disk transaction log out of the global page data linked list;

[0031] Sequentially storing the page mirror linked list nodes moved out of the global page data linked list into the temporary linked list according to the order of the starting position of the transaction log where the page mirror data is located in the page mirror linked list nodes.

[0032] Optionally, the page mirror data area further includes a page mirror data header and the position information of the data page corresponding to the page mirror data, and the page mirror data header is the header information corresponding to the page mirror data; and

[0033] After the step of performing at least one transaction log write-to-disk operation, the transaction log processing method further includes:

[0034] Obtaining the data page to be written to disk and the position information of the data page to be written to disk;

[0035] Judging whether there is a page mirror linked list node corresponding to the position information of the data page to be written to disk in the global page data linked list;

[0036] If not, perform a write-to-disk operation on the data page to be written to disk.

[0037] Optionally, before the step of performing at least one transaction log write-to-disk operation, the transaction log processing method further includes:

[0038] In the stage of assembling the transaction log, the page mirror data in the transaction log to be assembled is separated from the transaction log to be assembled, and the transaction log after the separation of the page mirror data is obtained; and

[0039] The transaction log to be written to disk includes the transaction log after the separation of the page mirror data.

[0040] According to another aspect of the present invention, there is also provided a machine-readable storage medium, on which a machine-executable program is stored, and when the machine-executable program is executed by a processor, the processing method of any one of the above-mentioned transaction logs is implemented.

[0041] According to still another aspect of the present invention, there is also provided a computer device, including a memory, a processor, and a machine-executable program stored on the memory and running on the processor, and when the processor executes the machine-executable program, the processing method of any one of the above-mentioned transaction logs is implemented.

[0042] In the processing method of the transaction log of the present invention, a global page data linked list for storing the page mirror data separated from all the transaction logs generated from the database is pre-created, and the transaction log to be written to disk and the page mirror data are separated. When performing the operation of writing the transaction log to disk each time, according to the serial number range of the transaction log to be written to disk this time, the corresponding page mirror data is obtained in the global page data linked list, and all the page mirror data corresponding to the transaction log to be written to disk is written to disk at one time. The processing method of the transaction log of the present invention, while realizing the separate writing to disk of the page mirror data and the transaction log, reduces the number of times of writing the page mirror data to disk, converts a large number of disk write IOs for only a single page mirror data into a small number of disk write IOs for more page mirror data, and improves the efficiency of writing the page mirror data of the transaction log to disk.

[0043] Further, the global page data linked list in the processing method of the transaction log of the present invention includes a plurality of page mirror linked lists connected, and the page mirror linked list nodes in the page mirror linked list at least include page mirror data, the starting position of the transaction log where the page mirror data is located, and a pointing identifier, realizing the sequential arrangement of the page mirror data and its related data in the global page data linked list. According to the serial number range of the transaction log to be written to disk, the page mirror linked list nodes whose starting position of the transaction log where the page mirror data is located is within the serial number range of the transaction log to be written to disk are moved from the global page data linked list to a temporary linked list, realizing the accurate acquisition of the page mirror data corresponding to the transaction log to be written to disk, and improving the accuracy of acquiring the page mirror data to be written to disk in the operation of writing the transaction log to disk.

[0044] Those skilled in the art will become more apparent about the above and other objects, advantages and features of the present invention according to the following detailed description of the specific embodiments of the present invention in conjunction with the accompanying drawings. Description of the Drawings

[0045] Some specific embodiments of the present invention will be described in detail hereinafter with reference to the accompanying drawings in an illustrative but not restrictive manner. The same reference numerals in the drawings denote the same or similar components or parts. Those skilled in the art should understand that these drawings are not necessarily drawn to scale. In the drawings:

[0046] Figure 1 is a schematic flowchart of a method for processing a transaction log according to an embodiment of the present invention;

[0047] Figure 2 is a schematic flowchart of an operation of flushing a transaction log in the method for processing a transaction log according to an embodiment of the present invention;

[0048] Figure 3 is a schematic structural diagram of a transaction log in the method for processing a transaction log according to an embodiment of the present invention;

[0049] Figure 4 is a schematic diagram of data transmission between a host and a standby machine in the method for processing a transaction log according to an embodiment of the present invention;

[0050] Figure 5 is a schematic diagram of the conversion from a first assembly linked list to a second assembly linked list in the method for processing a transaction log according to an embodiment of the present invention;

[0051] Figure 6 is a schematic structural diagram of a global page data linked list in the method for processing a transaction log according to an embodiment of the present invention;

[0052] Figure 7 is a schematic structural diagram of a page mirror linked list node in the method for processing a transaction log according to an embodiment of the present invention;

[0053] Figure 8 is a schematic structural diagram of a continuous data area in the method for processing a transaction log according to an embodiment of the present invention;

[0054] Figure 9 is a schematic structural diagram of a page mirror file in the method for processing a transaction log according to an embodiment of the present invention;

[0055] Figure 10 is a schematic structural diagram of a page mirror file in the method for processing a transaction log according to another embodiment of the present invention;

[0056] Figure 11 is a schematic flowchart of the method for processing a transaction log according to an embodiment of the present invention;

[0057] Figure 12 is a schematic diagram of a machine-readable storage medium according to an embodiment of the present invention; and

[0058] Figure 13 Schematic diagram of a computer device according to an embodiment of the present invention. Detailed implementation manners

[0059] Exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided so that the present disclosure can be more thoroughly understood and the scope of the present invention can be fully conveyed to those skilled in the art.

[0060] To solve the above technical problems, an embodiment of the present invention proposes a method for processing transaction logs. Figure 1 Schematic flowchart of a method for processing transaction logs according to an embodiment of the present invention. Figure 2 Schematic flowchart of an operation of writing a transaction log to disk in a method for processing transaction logs according to an embodiment of the present invention. Figure 3 Schematic structural diagram of a transaction log in a method for processing transaction logs according to an embodiment of the present invention. As Figure 1 shown, the method for processing transaction logs generally may include:

[0061] Step S102, create a global page data linked list, and the global page data linked list is used to store page mirror data separated from all transaction logs generated from the database.

[0062] Step S104, perform at least one operation of writing a transaction log to disk.

[0063] As Figure 2 shown, each operation of writing a transaction log to disk in the above step S104 may further include:

[0064] Step S202, obtain the serial number range of the transaction log to be written to disk.

[0065] Step S204, according to the serial number range of the transaction log to be written to disk, obtain at least one page mirror data corresponding to the transaction log to be written to disk in the global page data linked list;

[0066] Step S206, write to disk at least one page mirror data corresponding to the transaction log to be written to disk;

[0067] Step S208, write to disk the transaction log to be written to disk.

[0068] In a database, the transaction log refers to the XLOG log, which details the operation process of the service process on the database. During the operation of the database, multiple XLOG logs are generated. The serial number range of the XLOG log refers to the LSN (Log Sequence Number) range of the XLOG log, and the serial number LSN of each XLOG log is different. As Figure 3 shown, an XLOG log can include an XLOG log header (denoted as head data) and an XLOG log data area. Specifically, the XLOG log data area can include multiple block data areas (denoted as block) and main data (denoted as maindata). Each block can contain page mirror data (denoted as page data) and tuple data (denoted as tupledata). The XLOG log header includes the XLogRecord structure, the header data of each block, and the header data of main data.

[0069] It should be noted that different types of XLOG logs have different compositions. Each XLOG log contains headdata, but does not necessarily contain page data, tupledata, and main data. Some types of XLOG logs have only head data and no XLOG log data area; some other types of XLOG logs have only head data and main data; and some other types of XLOG logs have only head data, page data, and tupledata, and each block may contain both page data and tupledata, or may only contain tupledata. That is to say, an XLOG log may or may not contain page data.

[0070] During the operating system crash of the database, some operating system pages (e.g., page size is 4KB) may not have been written to the disk in time, which may result in a database page (e.g., composed of two operating system pages) containing a mixture of old and new data. During the recovery after the operating system crash of the database, since the information stored in the XLOG log is not complete enough, it is impossible to fully recover the database pages that contain a mixture of old and new data in the database. In the prior art, in order to recover such database pages, after the redo point of the checkpoint (checkpoint log), when the database page (default size is 8KB) is modified (updated) for the first time, the entire database page is written to the XLOG log. At this time, the XLOG log contains page mirror data to save the complete database page mirror. It should be noted that the redo point is a special point in the XLOG log. Before this point, all the data in the database is the same as the information reflected by the XLOG log that has been written to the disk. During the creation of the checkpoint, a special XLOG log (denoted as the checkpoint log) will be created. A redo point will be recorded in this checkpoint log. Only when the XLOG log and user data before this redo point have been written to the disk, the checkpoint will be successfully created. When the database crashes, we can find a redo point from the nearest checkpoint log, and then replay the XLOG log starting from this redo point to recover the database. When encountering a broken page during the recovery of the database, replace the broken page according to the page mirror data in the XLOG log to ensure the security and stability of the database data. Therefore, the data volume of the page mirror data in the XLOG log is equivalent to the data volume of the database page, and the data volume of the XLOG log containing the page mirror data is relatively large.

[0071] In this embodiment, the global page data linked list is used to store the page mirror data separated from all the XLOG logs generated from the database, greatly reducing the data volume of the XLOG log. After the primary and standby database clusters are built, the host only transmits the XLOG log after separating the page mirror data and the XLOG log without page mirror data to the standby machine, thus greatly reducing the pressure of XLOG log transmission in the streaming replication between the host and the standby machine on the premise of ensuring the security and stability of the database data.

[0072] The processing method of the transaction log in this embodiment pre-creates a global page data linked list for storing page mirror data separated from all XLOG logs generated from the database, and separates the XLOG logs to be flushed to disk from the page mirror data. When performing the XLOG log disk flushing operation each time, according to the serial number range of the XLOG logs to be flushed this time, the corresponding page mirror data is obtained in the global page data linked list, and all the page mirror data corresponding to the XLOG logs to be flushed is flushed to disk at one time. The processing method of the transaction log in this embodiment, while realizing the separate disk flushing of the page mirror data and the XLOG logs, reduces the number of disk flushing times of the page mirror data, converts a large number of disk write I / Os for a single page mirror data into a small number of disk write I / Os for more page mirror data, and improves the disk flushing efficiency of the XLOG log page mirror data.

[0073] Figure 4 It is a schematic diagram of data transmission between the host and the standby in the processing method of the transaction log according to an embodiment of the present invention. As Figure 4 shown, if a bad block 21 is found during playback on the standby 20, a signal requesting a good block is sent to the host 10. After obtaining the good block 11 corresponding to the bad block 21 in the host 10, the good block 11 corresponding to the bad block is returned to the standby 20, and the bad block 21 in the standby 20 is updated according to the data in the returned good block 11 to ensure the data consistency of the standby 20.

[0074] Specifically, after the primary and standby database clusters are built, if a page break is found during playback on the standby 20, the host 10 is applied to obtain the corresponding page mirror data, and the broken database page is replaced according to the page mirror data, so as to ensure the data consistency between the standby 20 and the host 10 while ensuring the streaming replication efficiency.

[0075] In some embodiments, before the above step S104, the processing method of the transaction log of the present invention may further include the following steps: during the stage of assembling the XLOG logs, the page mirror data in the XLOG logs to be assembled is separated from the XLOG logs to be assembled, and the XLOG logs after the page mirror data is separated are obtained. In this embodiment, the XLOG logs to be flushed to disk include the XLOG logs after the page mirror data is separated.

[0076] Further, after the above step of obtaining the XLOG logs after the page mirror data is separated, the processing method of the transaction log of the present invention may further include the following steps: according to the length of the XLOG logs after the page mirror data is separated, a storage location for the XLOG logs after the page mirror data is separated is pre-allocated in the XLOG log write cache; according to the pre-allocated storage location of the XLOG logs after the page mirror data is separated, the XLOG logs after the page mirror data is separated are copied to the XLOG log write cache.

[0077] The processing method of the transaction log in this embodiment separates the page mirror data in the XLOG log to be assembled, which contains page mirror data, from the XLOG log to be assembled during the assembly stage of the XLOG log, and determines the storage location information of the XLOG log after it is copied to the XLOG log write cache according to the data length of the XLOG log after the page mirror data, avoiding the change of the storage location information caused by the change of the length of the XLOG log and maintaining the stability of the storage location information of the XLOG log in the XLOG log write cache.

[0078] In some embodiments, before the above step S102, the processing method of the transaction log of the present invention further includes the following steps: A wal_pagedata directory for storing the page mirror data separated from the XLOG log is pre-created under the sys_wal directory. Specifically, a corresponding storage file is created under the wal_pagedata directory for each page mirror data to be retained in the XLOG log. That is, each page mirror data and its related data are stored using a separate storage file. In addition, in the database, newly generated XLOG log files are stored under the sys_wal directory, and each storage file and XLOG log file have separate storage spaces. A wal_pagedata directory for storing the page mirror data separated from the XLOG log is created under the sys_wal directory.

[0079] Figure 5 It is a schematic diagram of the conversion of the first assembly linked list to the second assembly linked list in the processing method of the transaction log according to an embodiment of the present invention. As Figure 5 shown, the above step of separating the page mirror data in the XLOG log to be assembled from the XLOG log to be assembled may include the following steps: Removing the assembly linked list node for storing the page mirror data from the first assembly linked list 51 pre-created for storing the XLOG log to be assembled.

[0080] In this embodiment, during the stage of assembling XLOG log data, first, a first assembly linked list 51 is created in memory. Each assembly linked list node in the first assembly linked list 51 stores data of different parts of the XLOG log. For example, it stores the XLOG log header, page mirror data, tuple data, and main data respectively. Secondly, the assembly linked list node used to store the page mirror data is removed from the first assembly linked list 51. The page mirror data in the removed assembly linked list node remains in memory, and the corresponding header information of the page mirror data remains in the XLOG log header in the XLOG log. After separating all the page mirror data in an XLOG log, in memory, the page mirror data separated from each single XLOG log is organized in a linked list manner, and the nodes of all the page mirror data in a single XLOG log are connected to form a page mirror linked list.

[0081] As Figure 5 shown, the remaining nodes in the first assembly linked list 51 except the assembly linked list node used to store the page mirror data are sequentially connected to form a second assembly linked list 52. The second assembly linked list 52 is used to store the XLOG log after separating the page mirror data.

[0082] The method for processing the transaction log in this embodiment completes the operation of separating the page mirror data of the XLOG log to be assembled by removing the node used to store the page mirror data from the first assembly linked list 51 created in advance for storing the XLOG log to be assembled, improving the accuracy of the operation of separating the page mirror data from the XLOG log.

[0083] Figure 6 According to the structural schematic diagram of the global page data linked list in the method for processing the transaction log according to an embodiment of the present invention, it is a simple schematic of the global page data linked list storing two page mirror linked lists. page_data_list_local represents the page mirror linked list of a database log, and page_data_list_global represents the global page data linked list. Among them, wal1 and wal2 are only used to indicate that the XLOG logs to which the two page mirror linked lists belong are different.

[0084] As Figure 6 shown, the global page data linked list 60 includes multiple connected page mirror linked lists 70, and each page mirror linked list 70 contains at least one page mirror linked list node 71. Each page mirror linked list node 71 corresponds to a page mirror data, and the page mirror linked list node 71 includes a page mirror data area and a pointing identifier (denoted as next).

[0085] In this embodiment, the global page data linked list 60 is a linked list pre-created in the shared memory for storing page mirror data separated from all XLOG logs generated by the database. That is to say, the global page data linked list 60 stores all page mirror data separated from the XLOG logs in the database.

[0086] Further, the step of removing the assembly linked list node for storing page mirror data from the first assembly linked list 51 pre-created for storing the XLOG logs to be assembled may include the following steps: creating a page mirror linked list node 71 for each page mirror data in the XLOG logs to be assembled, sequentially connecting the page mirror linked list nodes 71 of all page mirror data in the XLOG logs to be assembled to form a page mirror linked list 70; connecting the page mirror linked list 70 to the pre-created global page data linked list 60; and sequentially connecting the remaining nodes in the first assembly linked list 51 except the assembly linked list node for storing page mirror data to form a second assembly linked list 52 for storing the XLOG logs after the separation of page mirror data.

[0087] The method for processing the transaction log in this embodiment creates a page mirror linked list node 71 for each page mirror data in the XLOG logs to be assembled, sequentially connects the page mirror linked list nodes 71 of all page mirror data in the XLOG logs to be assembled to form a page mirror linked list 70, connects the page mirror linked list 70 to the pre-created global page data linked list 60, and sequentially connects the remaining nodes in the first assembly linked list 51 except the assembly linked list node for storing page mirror data to form a second assembly linked list 52, thus completing the operation of separating the page mirror data from the XLOG logs to be assembled, and further improving the accuracy of the operation of separating the page mirror data from the XLOG logs.

[0088] In a specific embodiment, after the step of sequentially connecting the page mirror linked list nodes 71 of all page mirror data in the XLOG logs to be assembled to form a page mirror linked list 70, the method for processing the transaction log of the present invention may further include the following steps: connecting the page mirror linked list 70 after the remaining nodes in the first assembly linked list 51 except the assembly linked list node for storing page mirror data.

[0089] That is to say, after sequentially connecting the page mirror linked list nodes 71 of all page mirror data in the XLOG logs to be assembled to form a page mirror linked list 70, first connect the page mirror linked list 70 to the end of the first assembly linked list 51, then remove the page mirror linked list 70 behind the first assembly linked list 51 and connect it to the global page data linked list 60.

[0090] The method for processing the transaction log in this embodiment can orderly adjust the nodes of the page mirror data, enabling effective management and maintenance of the data globally, facilitating data sharing and consistency maintenance, and improving the stability of the database.

[0091] Figure 7 It is a schematic structural diagram of a page mirror linked list node in the method for processing a transaction log according to an embodiment of the present invention. As Figure 7 shown, the page mirror data area at least includes page mirror data and the starting position (denoted as lsn) of the XLOG log where the page mirror data is located.

[0092] The lsn corresponding to each page mirror data increases sequentially, and the lsn can be used to determine the correspondence between the page mirror data and the XLOG log. next is used to point to the next page mirror linked list node 71 to connect the page mirror linked lists 70 corresponding to two adjacent XLOG logs.

[0093] Further, the steps of step S204 may include the following steps: Move the page mirror linked list nodes 71 whose lsn is within the serial number range of the XLOG log to be flushed to the disk from the global page data linked list 60 to a temporary linked list (not shown in the figure), and the temporary linked list can be denoted as temp_list, and the page mirror data therein is at least one page mirror data corresponding to the XLOG log to be flushed to the disk.

[0094] The global page data linked list 60 in the method for processing the transaction log in this embodiment includes multiple connected page mirror linked lists 70, and the page mirror linked list nodes 71 in the page mirror linked list 70 at least include page mirror data, lsn, and next, realizing the sequential arrangement of the page mirror data and its related data in the global page data linked list 60. According to the serial number range of the XLOG log to be flushed to the disk, moving the page mirror linked list nodes 71 whose lsn is within the serial number range of the XLOG log to be flushed to the disk from the global page data linked list 60 to the temporary linked list realizes the accurate acquisition of the page mirror data corresponding to the XLOG log to be flushed to the disk, and improves the accuracy of obtaining the page mirror data to be flushed to the disk in the XLOG log flushing operation.

[0095] Figure 8 It is a schematic structural diagram of a continuous data area in the method for processing a transaction log according to an embodiment of the present invention. Figure 9 It is a schematic structural diagram of a page mirror file in the method for processing a transaction log according to an embodiment of the present invention. As Figure 8 and Figure 9 shown, the continuous data area includes at least one sequentially and continuously arranged page mirror data area.

[0096] Further, the steps of step S206 may include the following steps: sequentially obtain all page mirror data areas from the temporary linked list, and connect the sequentially obtained page mirror data areas to form a continuous data area; sequentially write the data in the continuous data area into the page mirror file.

[0097] Using the above method, the page mirror data to be flushed to disk can be written into the page mirror file orderly, so as to convert a large number of small and random disk write I / Os into fewer and sequential disk write I / Os, further improving the disk flushing efficiency of the page mirror data.

[0098] In some embodiments, as Figures 7 - 9 shown, the page mirror data area may further include: a page mirror data header (denoted as page data header), the position information of the data page corresponding to the page mirror data, and the total length of the page mirror data area (denoted as len). In a specific embodiment, the data page refers to the data disk page. The position information of the data page corresponding to the page mirror data includes RelFileNode and BlockNumber. RelFileNode is the position information of the table where the data disk page corresponding to the page mirror data is located, and BlockNumber is the block number of the data disk page corresponding to the page mirror data in its table. RelFileNode and BlockNumber jointly determine the position of the data disk page on the disk. According to these two data of RelFileNode and BlockNumber, a unique data disk page can be determined on the disk. When searching on the disk, it can be determined whether a target page mirror data is found according to these two values. The page mirror data header is the header information corresponding to the page mirror data. len is used to calculate the start address of the next page mirror data area in the page mirror file. When traversing the page mirror file, the next page mirror data area can be quickly found according to the len of each page mirror data area.

[0099] In some specific embodiments, the step of moving the page mirror linked list node 71 whose lsn is within the serial number range of the XLOG log to be flushed to disk from the global page data linked list 60 to the temporary linked list may include the following steps: move the page mirror linked list node 71 whose lsn is within the serial number range of the XLOG log to be flushed to disk out of the global page data linked list 60; sequentially store each page mirror linked list node 71 moved out of the global page data linked list 60 into the temporary linked list according to the size order of the lsn in the page mirror linked list node 71. Specifically, each page mirror linked list node 71 moved out of the global page data linked list 60 may be sequentially stored into the temporary linked list in ascending order of the lsn in the page mirror linked list node 71.

[0100] In some other specific embodiments, the step of sequentially obtaining all page mirror data areas from the temporary linked list may include the following steps: sequentially obtain each page mirror data area in the temporary linked list according to len.

[0101] Using the above method, the data of each page mirror data area can be obtained completely and orderly according to the lsn and len of each page mirror data area, further improving the disk writing efficiency of the page mirror data.

[0102] In some embodiments, as Figure 9 shown, the page mirror file includes a used interval and a to-be-written interval. Based on this, the step of sequentially writing the data of the continuous data area into the page mirror file may include the following steps: sequentially write the data of the continuous data area into the position adjacent to the used interval in the to-be-written interval of the page mirror file. Specifically, the to-be-written interval is located after the used interval at the level of the actual physical storage space. That is to say, the step of sequentially writing the data of the continuous data area into the position adjacent to the used interval in the to-be-written interval of the page mirror file can be specifically executed as: sequentially write the data of the continuous data area after the used interval in the page mirror file.

[0103] Figure 10 FIG. is a schematic structural diagram of a page mirror file in a transaction log processing method according to another embodiment of the present invention. As Figure 10 shown, the page mirror file includes a plurality of page mirror data areas arranged continuously.

[0104] In this embodiment, after the step of sequentially writing the data of the continuous data area into the used interval in the page mirror file, the page mirror data areas in the page mirror file can be arranged sequentially and continuously.

[0105] In the subsequent page stage of the persistent data disk, if a page break occurs in the page of the to-be-written data disk, all page mirror data areas can be traversed and searched in the page mirror file according to the len of each page mirror data area until the page mirror data corresponding to the page of the to-be-written data disk is obtained, so as to perform subsequent repair operations.

[0106] In some embodiments, after the above step S104, the method for processing the transaction log of the present invention may further include the following steps: obtaining the data page to be flushed to disk and the location information of the data page to be flushed to disk; determining whether there is a page mirror linked list node corresponding to the location information of the data page to be flushed to disk in the global page data linked list; if not, performing a disk flushing operation on the data page to be flushed to disk; if so, waiting until after the next XLOG log disk flushing operation, and then executing the step of determining whether there is a page mirror linked list node 71 corresponding to the location information of the data page to be flushed to disk in the global page data linked list 60 again, until there is no page mirror linked list node 71 corresponding to the location information of the data page to be flushed to disk in the global page data linked list 60, and performing a disk flushing operation on the data page to be flushed to disk.

[0107] For the method for processing the transaction log in this embodiment, the page mirror data corresponding to the data page to be flushed to disk is flushed to disk first, and then the data page to be flushed to disk is flushed to disk. Even if a page break occurs when the data page is flushed to disk, it can be repaired through the already flushed page mirror file, further ensuring the data consistency between the standby machine 20 and the host machine 10.

[0108] Further, the step of obtaining the data page to be flushed to disk and the location information of the data page to be flushed to disk may be specifically executed as: obtaining the RelFileNode and BlockNumberpage of the data page to be flushed to disk; the step of determining whether there is a page mirror linked list node corresponding to the location information of the data page to be flushed to disk in the global page data linked list may be specifically executed as: determining whether there is a page mirror linked list node having RelFileNode and BlockNumberpage in the global page data linked list.

[0109] For the method for processing the transaction log in this embodiment, by utilizing the uniqueness of the correspondence between the two data, RelFileNode and BlockNumber, and the data page, the convenience and accuracy of finding the page mirror data corresponding to the data page to be flushed to disk in the global page data linked list 60 are improved, and the processing speed of the page stage of the persistent data disk is increased.

[0110] Figure 11 is a flowchart of the method for processing the transaction log according to an embodiment of the present invention. The following is combined with Figure 11 to specifically illustrate the flowchart steps of the method for processing the transaction log in this embodiment.

[0111] Step S1102, obtain the XLOG log to be assembled.

[0112] Step S1104, determine whether the XLOG log to be assembled contains page data. If so, execute step S1106; if not, execute step S1128.

[0113] Step S1106: Create a linked list node of page_data_list_local for each page data in the XLOG log to be assembled, and sequentially connect the linked list nodes of all page data in the XLOG log to be assembled to form a page_data_list_local linked list.

[0114] Step S1108: Sequentially connect the remaining nodes in the first assembly linked list 51 except for the assembly linked list nodes used to store page data to form a second assembly linked list 52, and connect the page_data_list_local linked list after the second assembly linked list 52 of the XLOG log to be assembled. It should be noted that the XLOG log stored in the second assembly linked list 52 is the assembled XLOG log.

[0115] Step S1110: Pre-allocate the storage location of the XLOG log after page data separation in the XLOG log write cache according to the length of the XLOG log.

[0116] Step S1112: Remove the page_data_list_local linked list after the second assembly linked list 52 and connect it to the page_data_list_global linked list. It should be noted that the global page data linked list is pre-created in the shared memory and is used to store the page data in all XLOG logs generated in the database.

[0117] Step S1114: Put the data of the XLOG log after page data separation into a continuous cache area.

[0118] Step S1116: Copy the content in the continuous cache area to the XLOG log write cache according to the pre-allocated storage location of the XLOG log after page data separation.

[0119] Step S1118: Obtain the LSN range of the XLOG log to be flushed to disk.

[0120] Step S1120: Remove the linked list nodes of page_data_list_local whose lsn is within the LSN range of the XLOG log to be flushed to disk from the page_data_list_global linked list, and store the removed page_data_list_local linked list nodes in the temp_list linked list in ascending order of lsn. The temp_list linked list can be a temporary linked list temporarily built during the XLOG log disk flushing stage.

[0121] Step S1122: Sequentially obtain all page data data areas from the temp_list linked list, and connect the sequentially obtained page data data areas to form a continuous data area.

[0122] Step S1124: Sequentially write the data in the continuous data area into the page mirror file.

[0123] Step S1126: Write the XLOG log to be flushed to disk into the XLOG log file. The XLOG log file and the page mirror file can be separately set on the disk to store the XLOG log and the page data of the XLOG log respectively. This process ends.

[0124] Step S1128: Skip the operation of separating the page data of the XLOG log. This process ends.

[0125] Using the above method, in the assembly stage of the XLOG log, first determine whether the XLOG log to be assembled contains page data, and separate the page data in the XLOG log to be assembled that contains page data from the XLOG log to be assembled, obtaining the XLOG log after page data separation, so as to reduce the data volume of the assembled XLOG log. After performing the disk flushing operation on the XLOG log after page data separation, before performing the disk flushing operation on the data pages to be flushed to disk, store the page data in a pre-created page mirror file, realizing the separate disk flushing and independent storage of the page data and the XLOG log after page data separation. So that after the primary and standby database clusters are built, the host 10 only transmits the XLOG log after page data separation to the standby 20, thereby greatly reducing the pressure of XLOG log transmission in the streaming replication between the host 10 and the standby 20.

[0126] This embodiment also provides a machine-readable storage medium and a computer device. Figure 12 It is a schematic diagram of a machine-readable storage medium 80 according to an embodiment of the present invention. Figure 13 It is a schematic diagram of a computer device 90 according to an embodiment of the present invention.

[0127] The machine-readable storage medium 80 stores a machine-executable program 81 thereon. When the machine-executable program 81 is executed by a processor, it implements the processing method of the transaction log in any of the above embodiments.

[0128] The computer device 90 may include a memory 920, a processor 910, and a machine-executable program 81 stored on the memory 920 and running on the processor 910. When the processor 910 executes the machine-executable program 81, the processing method of the transaction log in any of the above embodiments is implemented.

[0129] It should be noted that the logic and / or steps represented in the flowchart or described in other ways herein, for example, can be considered as a definite sequence list of executable instructions for implementing logical functions, and can be specifically implemented in any machine-readable storage medium for use by an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other systems that can fetch and execute instructions from the instruction execution system, apparatus, or device), or used in combination with these instruction execution systems, apparatus, or devices.

[0130] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in the memory and executed by a suitable instruction execution system. For the description of this embodiment, the machine-readable storage medium 80 can be any device that can contain, store, communicate, propagate, or transmit a program for use by an instruction execution system, apparatus, or device or in combination with these instruction execution systems, apparatus, or devices. The computer device 90 can be, for example, a server, a desktop computer, a laptop computer, a tablet computer, or a smart phone. The computer device 90 can include a processor 910 adapted to execute stored instructions and a memory 920 that provides temporary storage space for the operation of the instructions during operation. The processor 910 can be a single-core processor, a multi-core processor, a computing cluster, or any other number of other configurations. The memory 920 can include random access memory (RAM), read-only memory, flash memory, or any other suitable storage system.

[0131] It should be noted that in some alternative embodiments, the solution of the present invention is applicable to relational databases, and particularly applicable to the KingbaseES database (abbreviated as KES database), enriching the functions of the database and improving the efficiency of the database. In other alternative embodiments, the processing method of the transaction log of the present invention can also be applicable to other relational databases.

[0132] In addition, the flowchart provided in this embodiment is not intended to indicate that the operations of the method will be executed in any specific order, or that all operations of the method are included in every case. Furthermore, the method may include additional operations. Within the scope of the technical idea provided by the method in this embodiment, additional changes can be made to the above method.

[0133] At this point, those skilled in the art should recognize that although numerous exemplary embodiments of the present invention have been shown and described in detail herein, many other variations or modifications that conform to the principles of the present invention can still be directly determined or derived from the disclosure of the present invention without departing from the spirit and scope of the present invention. Therefore, the scope of the present invention should be understood and recognized to cover all such other variations or modifications.

Claims

1. A method for processing transaction logs, comprising: Creating a global page data linked list for storing page mirror data separated from all transaction logs generated from a database; Performing at least one transaction log disk write operation, and each transaction log disk write operation further includes: Obtaining a sequence number range of the transaction log to be written to disk; Obtaining at least one of the page mirror data corresponding to the transaction log to be written to disk in the global page data linked list according to the sequence number range of the transaction log to be written to disk; Writing to disk at least one of the page mirror data corresponding to the transaction log to be written to disk; Writing to disk the transaction log to be written to disk.

2. The method for processing transaction logs according to claim 1, wherein The global page data linked list includes a plurality of page mirror linked lists connected, each page mirror linked list includes at least one page mirror linked list node, each page mirror linked list node corresponds to one of the page mirror data, and the page mirror linked list node includes a page mirror data area and a pointing identifier, the page mirror data area at least includes page mirror data and the starting position of the transaction log where the page mirror data is located, and the pointing identifier is used to point to the next page mirror linked list node; and Obtaining at least one of the page mirror data corresponding to the transaction log to be written to disk in the global page data linked list according to the sequence number range of the transaction log to be written to disk includes: Moving the page mirror linked list nodes whose starting positions of the transaction logs where the page mirror data is located are within the sequence number range of the transaction log to be written to disk from the global page data linked list to a temporary linked list, and the page mirror data in the temporary linked list is at least one of the page mirror data corresponding to the transaction log to be written to disk.

3. The method for processing transaction logs according to claim 2, wherein Writing to disk at least one of the page mirror data corresponding to the transaction log to be written to disk includes: Sequentially obtaining all the page mirror data areas from the temporary linked list, and connecting the sequentially obtained page mirror data areas to form a continuous data area; Sequentially writing the data of the continuous data area to a page mirror file.

4. The method for processing transaction logs according to claim 3, wherein The page mirror data area further includes the total length of the page mirror data area; And Sequentially obtaining all the page mirror data areas from the temporary linked list includes: Sequentially obtaining each of the page mirror data areas in the temporary linked list according to the total length of the page mirror data area.

5. The method for processing transaction logs according to claim 3, wherein The page mirror file includes a used interval and a to-be-written-to-disk interval; Sequentially writing the data of the continuous data area to the page mirror file includes: Sequentially writing the data of the continuous data area to a position adjacent to the used interval in the to-be-written-to-disk interval in the page mirror file; and The page mirror data areas in the page mirror file are arranged sequentially and continuously.

6. The method for processing transaction logs according to claim 2, wherein Moving the page mirror linked list nodes whose starting positions of the transaction logs where the page mirror data is located are within the sequence number range of the transaction logs to be flushed to the disk from the global page data linked list into a temporary linked list includes: Removing the page mirror linked list nodes whose starting positions of the transaction logs where the page mirror data is located are within the sequence number range of the transaction logs to be flushed to the disk from the global page data linked list; Sequentially storing the removed page mirror linked list nodes from the global page data linked list into the temporary linked list in the order of the sizes of the starting positions of the transaction logs where the page mirror data is located in the page mirror linked list nodes.

7. The method for processing transaction logs according to claim 2, wherein The page mirror data area further includes a page mirror data header and location information of the data page corresponding to the page mirror data, and the page mirror data header is the header information corresponding to the page mirror data; And After the step of performing at least one transaction log flushing operation, the method for processing transaction logs further includes: Obtaining the data page to be flushed to the disk and the location information of the data page to be flushed to the disk; Judging whether there is a page mirror linked list node corresponding to the location information of the data page to be flushed to the disk in the global page data linked list; If not, performing a flushing operation on the data page to be flushed to the disk.

8. The method for processing transaction logs according to claim 1, wherein Before the step of performing at least one transaction log flushing operation, the method for processing transaction logs further includes: In the stage of assembling the transaction log, separating the page mirror data in the transaction log to be assembled from the transaction log to be assembled, obtaining the transaction log after the page mirror data is separated; and The transaction log to be flushed to the disk includes the transaction log after the page mirror data is separated.

9. A machine-readable storage medium, on which a machine-executable program is stored, and when the machine-executable program is executed by a processor, it implements the method for processing transaction logs according to any one of claims 1 to 8.

10. A computer device, including a memory, a processor, and a machine-executable program stored on the memory and running on the processor, and when the processor executes the machine-executable program, it implements the method for processing transaction logs according to any one of claims 1 to 8.

Images