Database log processing method, storage medium and equipment

Abstract

The invention provides a database log processing method, a storage medium and equipment. Database logs are separately stored in a cache in the form of an assembly linked list and a page data linked list, the page data linked list is stored in a global page data linked list, and a database is configured with a page data disk process. The database log processing method comprises the following steps: detecting a disk falling opening signal of a page data chain table; acquiring a log serial number range of the database logs conforming to a first preset rule by utilizing the page data disk process, and recording the log serial number range as a first range; acquiring a page data chain table with a log serial number in a first range in the global page data chain table by utilizing a page data disk process, and recording the page data chain table as a first disk-capable page data chain table; and using the page data disk process to disk the first disk-available page data chain table. The assembly chain table and the page data chain table of the database logs can be separately stored, only other data except the page mirror image data can be synchronized subsequently, and the data synchronization amount between the main database and the standby database is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of databases, and particularly to a method for processing database logs, a storage medium, and a device. Background Art

[0002] During the use of a database, database logs are recorded. An important function of database logs is to be used for constructing a primary and standby database cluster. Briefly speaking, after the primary and standby database cluster is constructed, the primary database provides read and write services externally and continuously generates database logs. The database logs are transmitted to the standby database through streaming replication, so that the standby database is nearly consistent with the primary database.

[0003] In order to make the data of the primary database and the standby database strictly consistent, a synchronous streaming replication mode is usually adopted. In the synchronous streaming replication mode, after the primary database sends a database log to the standby database, it needs to wait for the standby database to store this database log locally or replay it successfully, and then the primary database can continue to execute. Therefore, the amount of data of the database logs that the primary database needs to transmit to the standby database is an important factor affecting the performance of synchronous streaming replication. Summary of the Invention

[0004] An object of the present invention is to provide a method for processing database logs, a storage medium, and a device that can reduce the amount of data of the database logs to be transmitted.

[0005] In particular, the present invention provides a method for processing database logs. The database logs are separately stored in a cache in the form of an assembled linked list and a page data linked list. The page data linked list is a linked list composed of page mirror data in the database logs, and the assembled linked list is a linked list composed of other data except page mirror data in the database logs. The page data linked list is stored in a preset global page data linked list. The database is configured with a page data disk writing process for separately writing the page data linked list to disk. And the method for processing database logs includes:

[0006] Detecting a disk writing start signal of the page data linked list;

[0007] Using the page data disk writing process to obtain a range of log sequence numbers of database logs that meet a first preset rule, denoted as a first range. The database logs that meet the first preset rule are at least one database log that has been completely written into the cache and whose log sequence numbers are consecutive starting from the log sequence number of the last database log that has completed complete disk writing;

[0008] Using the page data disk writing process to obtain the page data linked list in the global page data linked list whose log sequence numbers are within the first range, denoted as a first disk-writable page data linked list;

[0009] Use the page data disk write process to write the first disk-writable page data linked list to disk.

[0010] Optionally, the database is also configured with a log disk write process for writing the database log to disk, and the method for processing the database log includes:

[0011] Detect the disk write start signal for the database log;

[0012] Use the log disk write process to obtain the log sequence number range of the database log that conforms to the second preset rule, denoted as the second range. The database log that conforms to the second preset rule is at least one database log that has been completely written into the cache and whose log sequence numbers are consecutive starting from the log sequence number of the last database log that has completed full disk write;

[0013] Use the log disk write process to obtain the disk-writable assembled data linked list in the cache. The disk-writable assembled data linked list is the assembled data linked list whose log sequence numbers are within the second range;

[0014] Use the log disk write process to write the disk-writable assembled data linked list to disk.

[0015] Optionally, before the step of using the log disk write process to obtain the disk-writable assembled data linked list in the cache, it includes:

[0016] Use the log disk write process to obtain the page data linked list in the global page data linked list whose log sequence numbers are within the second range, denoted as the second disk-writable page data linked list;

[0017] Use the log disk write process to write the second disk-writable page data linked list to disk.

[0018] Optionally, the global page data linked list is configured with a disk write lock;

[0019] Before the step of using the log disk write process to obtain the page data linked list in the global page data linked list whose log sequence numbers are within the second range, it includes:

[0020] Use the log disk write process to obtain the disk write lock;

[0021] And, before the step of using the page data disk write process to obtain the page data linked list in the global page data linked list whose log sequence numbers are within the first range, it includes:

[0022] Use the page data disk write process to obtain the disk write lock.

[0023] Optionally, before the step of using the log disk write process to obtain the page data linked list in the global page data linked list whose log sequence numbers are within the second range, it includes:

[0024] Detect whether the page data disk writing process is running normally. If so, use the page data disk writing process to obtain the page data linked list with log sequence numbers within the second range in the global page data linked list, and use the page data disk writing process to write the page data linked list with log sequence numbers within the second range in the global page data linked list to disk. If not, execute the step of using the log disk writing process to obtain the page data linked list with log sequence numbers within the second range in the global page data linked list.

[0025] Optionally, before the step of detecting whether the page data disk writing process is running normally, it includes:

[0026] Check whether there is a page data linked list with log sequence numbers within the second range in the global page data linked list. If so, execute the step of detecting whether the page data disk writing process is running normally. If not, execute the step of using the log disk writing process to obtain the list of assemblable data that can be written to disk in the cache.

[0027] Optionally, the step of using the page data disk writing process to write the first list of page data that can be written to disk to disk includes:

[0028] Store the first list of page data that can be written to disk into a preset page mirror data storage table;

[0029] The step of using the log disk writing process to write the second list of page data that can be written to disk to disk includes:

[0030] Store the second list of page data that can be written to disk into the page mirror data storage table.

[0031] Optionally, the page data linked lists in the global page data linked list are connected in sequence in ascending order of log sequence numbers;

[0032] The step of storing the first list of page data that can be written to disk into a preset page mirror data storage table includes:

[0033] Copy the page mirror data in the first list of page data that can be written to disk to the end of the preset page mirror data storage table in sequence according to the connection order;

[0034] The step of storing the second list of page data that can be written to disk into the page mirror data storage table includes:

[0035] Copy the page mirror data in the second list of page data that can be written to disk to the end of the page mirror data storage table in sequence according to the connection order.

[0036] In another aspect, the present invention also provides a machine-readable storage medium, on which a machine-executable program is stored. When the machine-executable program is executed by a processor, it implements the method for processing database logs according to any one of the above.

[0037] In another aspect, the present invention further provides a computer device, including a memory, a processor, and a machine-executable program stored on the memory and running on the processor. When the processor executes the machine-executable program, the processing method of the database log according to any one of the above is implemented.

[0038] In the processing method, storage medium, and device of the database log of the present invention, the database log is separately stored in the cache in the form of an assembled linked list and a page data linked list, and the page data linked list is stored in a preset global page data linked list. A page data disk writing process for the page data linked list is separately configured. After the page data disk writing process detects the disk writing start signal of the page data linked list, the page data disk writing process is used to obtain the range of log sequence numbers that can be disk written, denoted as the first range. Then, the page data linked list in the global page data linked list whose log sequence numbers are within the first range is obtained, denoted as the first page data linked list that can be disk written. Then, the first page data linked list that can be disk written is disk written. On the one hand, the page mirror data in the database log is separated and separately stored in the global page data linked list, so that in the subsequent disk writing process, the page mirror data of the database log and other data except the page mirror data can be disk written separately, that is, not disk written as a whole. Therefore, when this database synchronizes the database log to other standby databases, only the data except the page mirror data needs to be synchronized, thereby reducing the data synchronization amount between the primary and standby databases and helping to improve the synchronization efficiency. Moreover, the page mirror data of the database log is still stored on the disk, which can also ensure the security and stability of this database. On the other hand, a page data disk writing process is separately configured for the page mirror data, that is, a new page data disk writing process is separately created outside the original disk writing process of the database log. In the case where the database log is separately stored in the cache in the form of an assembled linked list and a page data linked list, the page mirror data of the database log and other data except the page mirror data can be disk written in parallel, thereby effectively improving the disk writing efficiency of the database log.

[0039] From the following detailed description of the specific embodiments of the present invention in conjunction with the accompanying drawings, those skilled in the art will become more clear about the above and other objects, advantages, and features of the present invention. Description of the Drawings

[0040] Some specific embodiments of the present invention will be described in detail hereinafter with reference to the accompanying drawings in an exemplary and non-limiting manner. The same reference numerals in the drawings denote the same or similar components or parts. Those skilled in the art should understand that these drawings are not necessarily drawn to scale. In the drawings:

[0041] Figure 1 is a schematic diagram of a database log in the prior art;

[0042] Figure 2The first schematic diagram of a database log according to an embodiment of the present invention;

[0043] Figure 3 The second schematic diagram of a database log according to an embodiment of the present invention;

[0044] Figure 4 The schematic diagram of the page data chain of a database log according to an embodiment of the present invention;

[0045] Figure 5 The schematic diagram of the page mirror data content of a database log according to an embodiment of the present invention;

[0046] Figure 6 The third schematic diagram of a database log according to an embodiment of the present invention;

[0047] Figure 7 The schematic diagram of the global page data linked list according to an embodiment of the present invention;

[0048] Figure 8 The schematic flowchart of the processing method of a database log according to an embodiment of the present invention;

[0049] Figure 9 The schematic flowchart of the processing method of a database log according to another embodiment of the present invention;

[0050] Figure 10 The schematic flowchart of the processing method of a database log according to yet another embodiment of the present invention;

[0051] Figure 11 The schematic diagram of the page mirror data storage table according to an embodiment of the present invention;

[0052] Figure 12 The schematic diagram of a machine-readable storage medium according to an embodiment of the present invention;

[0053] Figure 13 The schematic diagram of a computer device according to an embodiment of the present invention. Detailed implementation manners

[0054] Those skilled in the art should understand that the embodiments described below are only a part of the embodiments of the present invention, rather than all the embodiments of the present invention. This part of the embodiments is intended to explain the technical principles of the present invention, rather than to limit the protection scope of the present invention. Based on the embodiments provided by the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts should still fall within the protection scope of the present invention.

[0055] It should be noted that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a defined sequence list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other systems that can fetch instructions from the instruction execution system, apparatus, or device and execute the instructions), or used in combination with these instruction execution systems, apparatus, or devices.

[0056] The flowchart provided by the present invention is not intended to indicate that the operations of the method will be executed in any specific order, or that all operations of the method are included in every case. In addition, the method may include additional operations. Within the scope of the technical idea provided by the method of this embodiment, additional changes can be made to the above method.

[0057] To facilitate the understanding of this solution, the operation mode of the database in the prior art is described. First, during the process of modifying the database data, the database data on the disk is not directly modified. Instead, the data on the disk is extracted into the cache in units of data pages (the data to be modified is located in the data pages), modified in the cache, and then written to disk later.

[0058] Furthermore, when modifying the data pages in the cache, database logs will be generated. As Figure 1 shown, the existing database logs generally include two parts: a header and a data area. The data area records various modification operations on the data pages. At the same time, there is also a kind of data in the data area that is page mirror data. The page mirror data is formed after recording the complete page information of a data page in the database log. In the prior art, after the log redo point (the log redo point is a log sequence number. Before this log sequence number, the database logs have all been written to disk and the information reflected by the written logs and all the data in the actual database on the disk is the same), if the data of a data page in the cache is modified for the first time, then the entire page information of the data page after the first modification will be recorded in the database log, and then page mirror data will be generated.

[0059] Because during the process of writing a data page to disk, it is possible that a data page crashes during the write process, resulting in a situation where the written data page contains a mixture of old and new data. Therefore, by generating page mirror data, the broken page with the mixture of old and new data can be replaced with the data page information recorded in the page mirror data, and then the subsequent operations recorded in the database log can be replayed to complete the recovery of the data page. Therefore, the page mirror data is important data for ensuring the security and stability of the database.

[0060] As Figures 2 to 6As shown, in an embodiment of the present application, the database log includes an assembly linked list and a page data linked list separately stored in the cache. The page data linked list is a linked list composed of page mirror data in the database log, and the assembly linked list is a linked list composed of other data in the database log except for the page mirror data. The page data linked list is stored in a preset global page data linked list.

[0061] Specifically, in the database, the length of a database log is fixed. As the modification operations continue to increase, the capacity limit of a database log will be reached or it will be filled. At this time, the generated log information needs to be written into the next database log. For the filled database log, the complete content of the database log is determined.

[0062] In the process of generating the database log of this embodiment, when all the content of a database log is determined, the database log is assembled into a structure composed of an assembly linked list and a page data linked list, and then the assembly linked list and the page data linked list are separately stored in the cache, and subsequent disk writes can be performed separately.

[0063] Refer to Figures 2 to 6 As shown, specifically, taking a database log with two page mirror data nodes in the complete content as an example, Figure 2 is an example of the linked list form of the original database log with page mirror data, Figure 3 is Figure 2 an example of the linked list form after removing the page mirror data from the linked list in, where next represents pointing to the next node; len represents the len function, which is used to calculate the length of the corresponding data; head data represents the header data of the database log; page data represents all the page mirror data in the database log; tuple data and main data represent other data in the database log except for the page mirror data. Figure 4 is a simplified example of the linked list of two page mirror data nodes in the database log. wal1 page data1 and wal1 page data2 are only used to simply indicate two different page mirror data nodes belonging to the same database log, and do not represent specific content. Figure 5As an example of the specific content of a page mirror data node, RelFileNode represents the location information of the table to which the data page corresponding to this page mirror data belongs on the disk, and BlockNumber represents the block number of the data page corresponding to this page mirror data in the table to which it belongs. Therefore, RelFileNode and BlockNumber can determine the location of the data page corresponding to the page mirror data on the disk. lsn represents the log sequence number of the database log to which the page mirror data belongs, pagedata is the complete information of the data page recorded by this page mirror data node, and next points to the next node. Figure 6 Is an example of the finally formed database log.

[0064] During the assembly process of the database log, first remove the page mirror data in the database log, as Figures 2 to 3 shown. Then form a page data linked list with the page mirror data in the order in the database log, as Figure 4 shown. Then, connect the page data linked list behind the linked list composed of other data in the database log, so as to assemble the database log into a structure composed of an assembly linked list and a page data linked list connected after the assembly linked list, as Figure 6 shown. That is to say, Figure 5 can be regarded as Figure 4 a schematic diagram of the specific content of a page mirror data node in. Figure 6 The assembly linked list in represents Figure 3 the linked list of the database log in, that is, the linked list composed of other data in the database log except the page mirror data.

[0065] Then, during the process of writing the database log into the cache, store the assembly linked list and the page data linked list separately. And, the page data linked list is stored in the global page data linked list. That is to say, the global page data linked list is a linked list formed by connecting page data linked lists to each other.

[0066] Refer to Figure 7 shown, which is a simple schematic diagram of the global page data linked list storing two page data linked lists. page_data_list_local represents the page data linked list of a database log, and page_data_list_global represents the global page data linked list. Among them, wal1 and wal2 are only used to indicate that the database logs to which the two page data linked lists belong are different.

[0067] Refer to Figure 8 shown. In this embodiment, the database is configured with a page data disk write process that separately writes the page data linked list to disk. The processing method of the database log generally includes:

[0068] Step S101, a disk write start signal of the page data linked list is detected. Specifically, the disk write start signal can be a signal that is triggered at fixed intervals, or it can be a signal that is triggered after the number of nodes in the page data linked list in the global page data linked list reaches a certain threshold, etc. Briefly speaking, the page data disk write process monitors whether the disk write start signal of the page data linked list is triggered, and if it is triggered, it performs disk write operations on the page data linked list in the global page data linked list.

[0069] Step S102, use the log sequence number range of the database logs that meet the first preset rule by the page data disk write process, denoted as the first range. The database logs that meet the first preset rule are at least one database log that has been completely written into the cache and whose log sequence numbers are consecutive starting from the log sequence number of the last database log that has completed a complete disk write. The database log that has completed a complete disk write is the database log for which both the assembled linked list part and the page data linked list part have completed disk writes.

[0070] Since the page data disk write process is a newly created disk write process, and there is an original disk write process for the database logs in the database that is performing disk writes on the database logs, it is possible that for a database log with page mirror data, the page mirror data has been disk written but the assembled linked list part has not been disk written. The database log that has been completely written into the cache is the database log that has been or is currently written into the cache.

[0071] Specifically, the database logs that meet the first preset rule can be configured as all the database logs that have been completely written into the cache and whose log sequence numbers are consecutive starting from the log sequence number of the last database log that has completed a complete disk write; or the database logs with a set number (such as one, two, ten, one hundred, etc.) of consecutive log sequence numbers starting from the log sequence number of the last database log that has completed a complete disk write.

[0072] Exemplarily, if the currently cached database logs with log sequence numbers 3, 4, 5, 8, 9 (the numbers are only for illustration and do not represent actual complete log sequence numbers) have been completely written into the cache, and the log sequence number of the last database log that has completed a complete disk write is 2.

[0073] If the database logs that meet the first preset rule are configured as all the database logs that have been completely written into the cache and whose log sequence numbers are consecutive starting from the log sequence number of the last database log that has completed a complete disk write, then the database logs that meet the first preset rule are the database logs with log sequence numbers 3, 4, 5, and the first range is greater than or equal to 3 and less than or equal to 5.

[0074] If the database logs that meet the first preset rule are configured to start from the log sequence number of the last database log that has completed full disk write, and the log sequence numbers of two consecutive database logs that have been fully written into the cache are maintained, then the database logs that meet the first preset rule are the database logs with log sequence numbers 3 and 4, and the first range is greater than or equal to 3 and less than or equal to 4.

[0075] Step S103: Use the page data disk write process to obtain the page data linked list in the global page data linked list whose log sequence numbers are within the first range, and denote it as the first disk-writable page data linked list. Specifically, after obtaining the first range, the page data disk write process can search for the corresponding page data linked list in the global page data linked list according to the log sequence numbers within the first range. After finding the page data linked list whose log sequence numbers are within the first range, these page data linked lists are the page data linked lists to be disk-written this time.

[0076] Step S104: Use the page data disk write process to disk-write the first disk-writable page data linked list. That is, the page data disk write process disk-writes the page data linked list in the global page data linked list whose log sequence numbers are within the first range, that is, writes it into the disk for persistent storage.

[0077] In the solution of this embodiment, by storing the database logs separately in the cache in the form of an assembled linked list and a page data linked list, and storing the page data linked list in the preset global page data linked list, and separately configuring a page data disk write process for disk-writing the page data linked list. After the page data disk write process detects the disk write start signal of the page data linked list, it uses the page data disk write process to obtain the range of the log sequence numbers of the database logs that meet the first preset rule, denoted as the first range, and then obtains the page data linked list in the global page data linked list whose log sequence numbers are within the first range, denoted as the first disk-writable page data linked list, and then disk-writes the first disk-writable page data linked list. On the one hand, the page mirror data in the database logs is separated and stored separately in the global page data linked list, so that in the subsequent disk write process, the page mirror data of the database logs and other data except the page mirror data can be disk-written separately, that is, not disk-written as a whole. Therefore, when this database synchronizes the database logs to other standby databases, it can only synchronize other data except the page mirror data, thereby reducing the data synchronization volume between the master and standby databases and helping to improve the synchronization efficiency. And the page mirror data of the database logs is still stored in the disk, which can also ensure the security and stability of this database.

[0078] On the other hand, a page data disk write process is configured separately for the page mirror data, that is, a new page data disk write process is created separately outside the original database log disk write process. When the database log is stored separately in the cache in the form of an assembled linked list and a page data linked list, the page mirror data of the database log and other data except the page mirror data can be disk-written in parallel, thereby effectively improving the disk write efficiency of the database log.

[0079] Refer to Figure 9 As shown, specifically, the database is also configured with a log disk write process for disk-writing the database log, and generally, the processing method of the database log includes:

[0080] Step S201, detecting a disk write start signal for the database log. Specifically, the disk write start signal for the database log can be a signal that is triggered every fixed time, or can also be a signal that is triggered after the amount of the database log stored in the cache reaches a certain threshold, etc. Simply put, the log disk write process monitors whether the disk write start signal for the database log is triggered, and if it is triggered, it performs the disk write work on the database log.

[0081] Step S202, using the log disk write process to obtain the log sequence number range of the database log that conforms to the second preset rule, denoted as the second range. The database log that conforms to the second preset rule is at least one database log that has been completely written into the cache and whose log sequence numbers are consecutive starting from the log sequence number of the last database log that has completed complete disk writing. Specifically, the content of step S102 in the previous text can be referred to here. However, in the specific setting process, the database log that conforms to the first preset rule and the database log that conforms to the second preset rule can be set differently. For example, the database log that conforms to the first preset rule is configured as M database logs that have been completely written into the cache and whose log sequence numbers are consecutive starting from the log sequence number of the last database log that has completed complete disk writing, and the database log that conforms to the second preset rule is configured as N database logs that have been completely written into the cache and whose log sequence numbers are consecutive starting from the log sequence number of the last database log that has completed complete disk writing.

[0082] Step S203, using the log disk write process to obtain the page data linked list in the global page data linked list whose log sequence numbers are within the second range, denoted as the second disk-writable page data linked list. Specifically, after the log disk write process obtains the second range, it can search for the corresponding page data linked list in the global page data linked list according to the log sequence numbers in the second range. After finding the page data linked list whose log sequence numbers are within the second range, these page data linked lists are the page data linked lists to be disk-written this time.

[0083] Step S204: Use the log disk writing process to write the second disk-writable page data linked list to disk. That is, the log disk writing process writes the page data linked list with log sequence numbers within the second range in the global page data linked list to disk, i.e., writes it to the disk for persistent storage.

[0084] Step S205: Use the log disk writing process to obtain the disk-writable assembled data linked list in the cache. The disk-writable assembled data linked list is the assembled data linked list with log sequence numbers within the second range. Specifically, after the log disk writing process obtains the second range, it can search for the corresponding assembled data linked list in the cache according to the log sequence numbers in the second range. After finding the assembled data linked list with log sequence numbers within the second range, these assembled data linked lists are the assembled data linked lists to be written to disk this time.

[0085] Step S206: Use the log disk writing process to write the disk-writable assembled data linked list to disk. Specifically, that is, the log disk writing process writes the assembled data linked list with log sequence numbers within the second range in the cache to disk, i.e., writes it to the disk for persistent storage.

[0086] Specifically, during the working process of the log disk writing process, it is responsible for writing a complete database log to disk. Since a separate page data disk writing process is newly created to write the page data linked list to disk, when the log disk writing process writes the database log to disk, for some database logs with page mirror data, the page mirror data of this database log may have been written to disk by the page data disk writing process, thus reducing the workload of the log disk writing process. It alleviates the pressure on the log disk writing process and enables the parallel writing of the page mirror data and other data except the page mirror data of the database log to disk, thereby effectively improving the disk writing efficiency of the database log.

[0087] Furthermore, the global page data linked list is configured with a disk writing lock, which is a mutex lock. Before the step of using the log disk writing process to obtain the page data linked list with log sequence numbers within the second range in the global page data linked list, it includes: using the log disk writing process to obtain the disk writing lock, so that only the log disk writing process can operate on the global page data linked list. And, before the step of using the page data disk writing process to obtain the page data linked list with log sequence numbers within the first range in the global page data linked list, it includes: using the page data disk writing process to obtain the disk writing lock, so that only the page data disk writing process can operate on the global page data linked list.

[0088] By configuring a disk writing lock for the global page data linked list, in the case where both the log disk writing process and the page data disk writing process can operate on the global page data linked list, enabling the process operating on the global page data linked list to obtain the disk writing lock can avoid the situation where the log disk writing process and the page data disk writing process operate on the global page data linked list simultaneously, resulting in chaos in the disk writing of the page data linked list.

[0089] It can be understood that after obtaining the disk write lock by the log disk write process, after writing the second disk-writable page data linked list to disk by the log disk write process, the log disk write process releases the disk write lock. After obtaining the disk write lock by the page data disk write process, after writing the first disk-writable page data linked list to disk by the page data disk write process, the page data disk write process releases the disk write lock.

[0090] As Figure 10 shown, in one embodiment, the method for processing database logs generally includes:

[0091] Step S301, detecting a disk write start signal for the database log.

[0092] Step S302, using the log disk write process to obtain the range of log sequence numbers of the database logs that meet the second preset rule, denoted as the second range. The database logs that meet the second preset rule are at least one database log that has been completely written into the cache and whose log sequence numbers are consecutive starting from the log sequence number of the last database log that has completed a complete disk write.

[0093] Step S303, checking whether there is a page data linked list in the global page data linked list whose log sequence number is within the second range. If so, execute Step S304. If not, execute Step S309. Specifically, during the process of the page data disk write process writing the page data linked list to disk, after the disk write is completed, the log sequence number corresponding to the last page data linked list that has completed disk write will be recorded, denoted as the disk write mark sequence number.

[0094] In the step of checking whether there is a page data linked list within the second range in the global page data linked list, obtain the maximum value of the second range, denoted as the maximum sequence number. Determine whether the maximum sequence number is less than or equal to the disk write mark sequence number. If the maximum sequence number is less than or equal to the disk write mark sequence number, it means that there is no page data linked list in the global page data linked list whose log sequence number is within the second range.

[0095] If the maximum sequence number is greater than the disk write mark sequence number, obtain the minimum value of the log sequence numbers of the page data linked lists stored in the global page data linked list, denoted as the global minimum sequence number. Determine whether the maximum sequence number is less than the global minimum sequence number. If the maximum sequence number is less than the global minimum sequence number, it means that there is no page data linked list in the global page data linked list whose log sequence number is within the second range. If the maximum sequence number is greater than or equal to the global minimum sequence number, it means that there is a page data linked list in the global page data linked list whose log sequence number is within the second range.

[0096] Step S304: Detect whether the page data disk writing process is running normally. If yes, execute Step S305; if not, execute Step S307. Specifically, it is to detect whether the page data disk writing process can be used normally.

[0097] Step S305: Use the page data disk writing process to obtain the page data linked list in the global page data linked list whose log sequence numbers are within the second range. Specifically, when the data disk writing process is running normally and there is a page data linked list in the global page data linked list whose log sequence numbers are within the second range, instead of the log disk writing process performing disk writing on the page data linked list, the page data disk writing process is used to obtain the page data linked list in the global page data linked list whose log sequence numbers are within the second range.

[0098] Step S306: Use the page data disk writing process to write the page data linked list in the global page data linked list whose log sequence numbers are within the second range to disk. Specifically, the page data disk writing process writes the page data linked list in the global page data linked list whose log sequence numbers are within the second range to the disk.

[0099] Step S307: Use the log disk writing process to obtain the page data linked list in the global page data linked list whose log sequence numbers are within the second range, denoted as the second disk-writable page data linked list. If the page data disk writing process is not running normally, the log disk writing process still completes the disk writing work of the page data linked list.

[0100] Step S308: Use the log disk writing process to write the second disk-writable page data linked list to disk. Then execute Step S309. That is, the log disk writing process writes the page data linked list within the second range in the global page data linked list to disk for persistent storage.

[0101] Step S309: Use the log disk writing process to obtain the disk-writable assembled data linked list in the cache. The disk-writable assembled data linked list is the assembled data linked list whose log sequence numbers are within the second range.

[0102] Step S310: Use the log disk writing process to write the disk-writable assembled data linked list to disk.

[0103] In the solution of this embodiment, when there is a page data linked list in the database log to be written to disk by the log disk writing process, the page data disk writing process is also used to write the page data linked list to disk. In other words, all the disk writing work of the page data linked list is performed by the page data disk writing process. Because only the page data disk writing process writes the page data linked list to disk, the page data disk writing process can better manage the log sequence numbers of the written page data linked list, so that the retrieval process during the disk writing of the page data linked list can be faster, which helps improve the disk writing efficiency.

[0104] In the case of a failure in the page data disk write process, the disk write can still be completed by the log disk write process, ensuring the smooth progress of the disk write work.

[0105] Preferably, after using the page data disk write process in step S306 to write the page data linked list with the log sequence number within the second range in the global page data linked list to disk, the method for processing the database log includes ending the current disk write work. That is to say, after the log disk write process is started, when there is a page data linked list with the log sequence number within the second range in the global page data linked list and the page data disk write process is running normally, after the page data disk write process writes the page data linked list to disk, the log disk write process will not perform the disk write work on the assembled linked list anymore, but directly end the current disk write work until the next log disk write process is triggered to perform disk write again. This can avoid splitting of IO (Input / Output), that is, merge IO, thereby reducing the overhead.

[0106] Further, the step of using the page data disk write process to write the first disk-writable page data linked list to disk includes: storing the first disk-writable page data linked list in a preset page mirror data storage table. The step of using the log disk write process to write the second disk-writable page data linked list to disk includes: storing the second disk-writable page data linked list in the page mirror data storage table. The step of using the page data disk write process to write the page data linked list with the log sequence number within the second range in the global page data linked list to disk includes: using the page data disk write process to store the page data linked list with the log sequence number within the second range in the global page data linked list in the page mirror data storage table.

[0107] Specifically, a page mirror data storage table is pre-created in the disk, and all the work of writing the page data linked list to disk is to record the page data linked list in the page mirror data storage table.

[0108] Further, the page data linked lists in the global page data linked list are connected in ascending order of the log sequence number; the step of storing the first disk-writable page data linked list in a preset page mirror data storage table includes: sequentially copying the page mirror data in the first disk-writable page data linked list to the end of the page mirror data storage table in the connection order; the step of storing the second disk-writable page data linked list in the page mirror data storage table includes: sequentially copying the page mirror data in the second disk-writable page data linked list to the end of the preset page mirror data storage table in the connection order. The step of using the page data disk write process to write the page data linked list with the log sequence number within the second range in the global page data linked list to disk includes: using the page data disk write process to sequentially copy the page mirror data in the page data linked list with the log sequence number within the second range in the global page data linked list to the end of the preset page mirror data storage table in the connection order.

[0109] Specifically, when storing a page data linked list into the global page data linked list, instead of directly connecting it after the existing global page data linked list, the log sequence number of the page data linked list to be stored is compared with the log sequence numbers of the page data linked lists already stored in the global page data linked list. Then, the page data linked list to be stored is stored in the global page data linked list in ascending order of the log sequence number, so that all the page data linked lists in the global page data linked list are connected in ascending order of the log sequence number. Therefore, when flushing the page data linked list to disk, the page mirror data in the page data linked list to be flushed can be directly copied to the end of the storage table in the connection order, and the page mirror data must also be arranged in the order of the log sequence number in the page mirror data storage table.

[0110] Refer to Figure 11 As shown, it is an example of a page mirror data storage table. Since the page data linked list was arranged in the order of the log sequence number during the previous process of storing it in the global page data linked list, the page mirror data in the page data linked list to be flushed is copied to the end of the storage table in the connection order, and the page mirror data must also be arranged in the order of the log sequence number in the page mirror data storage table.

[0111] By storing the page data linked list in the page mirror data storage table, since the database can manage and search the table more conveniently, the management convenience of the page mirror data on the disk can be improved. Moreover, since the page data linked list was arranged in ascending order of the log sequence number during the process of storing it in the global page data linked list, the page mirror data in the page data linked list to be flushed can be directly copied to the end of the storage table in the connection order, without the need to perform sorting work during the flushing stage, and the efficiency is very high.

[0112] This application also provides a machine-readable storage medium and a computer device. Figure 12 It is a schematic diagram of a machine-readable storage medium 10 according to an embodiment of the present invention. Figure 13 It is a schematic diagram of a computer device 20 according to an embodiment of the present invention.

[0113] The machine-readable storage medium 10 stores a machine-executable program 11 thereon. When the machine-executable program 11 is executed by a processor, it implements the database log processing method of any of the above embodiments.

[0114] The computer device 20 may include a memory 210, a processor 220, and a machine-executable program 11 stored on the memory 210 and running on the processor 220. When the processor 220 executes the machine-executable program 11, it implements the database log processing method of any of the above embodiments.

[0115] For the description of this embodiment, the machine-readable storage medium 10 can be any device that can contain, store, communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection (electronic device) having one or more wirings, a portable computer diskette (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). Additionally, the machine-readable storage medium 10 can even be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other media, followed by editing, interpretation, or otherwise processing as appropriate, and then stored in a computer memory.

[0116] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system.

[0117] The computer device 20 can be, for example, a server, a desktop computer, a laptop computer, a tablet computer, or a smart phone. In some examples, the computer device 20 can be a cloud computing node. The computer device 20 can be described in the general context of computer system-executable instructions, such as program modules, executed by a computer system. Generally, program modules can include routines, programs, object programs, components, logic, data structures, etc. that perform specific tasks or implement specific abstract data types. The computer device 20 can be implemented in a distributed cloud computing environment where tasks are performed by remote processing devices linked through a communication network. In a distributed cloud computing environment, program modules can be located on local or remote computing system storage media including storage devices.

[0118] The computer device 20 can include a processor 220 adapted to execute stored instructions and a memory 210 that provides temporary storage space for the operation of the instructions during operation. The processor 220 can be a single-core processor, a multi-core processor, a computing cluster, or any number of other configurations. The memory 210 can include random access memory (RAM), read-only memory, flash memory, or any other suitable storage system.

[0119] The processor 220 can be connected to an I / O interface (input / output interface) suitable for connecting the computer device 20 to one or more I / O devices (input / output devices) through a system interconnect (such as PCI, PCI-Express, etc.). The I / O devices can include, for example, a keyboard and a pointing device, where the pointing device can include a touchpad or a touch screen, etc. The I / O devices can be built-in components of the computer device 20, or can be devices externally connected to the computing device.

[0120] The processor 220 can also be linked to a display interface suitable for connecting the computer device 20 to a display device through a system interconnect. The display device can include a display screen as a built-in component of the computer device 20. The display device can also include a computer monitor, a television, a projector, etc. externally connected to the computer device 20. In addition, a network interface controller (NIC) can be suitable for connecting the computer device 20 to a network through a system interconnect. In some embodiments, the NIC can use any suitable interface or protocol (such as Internet Small Computer System Interface, etc.) to transmit data. The network can be a cellular network, a radio network, a wide area network (WAN), a local area network (LAN), or the Internet, etc. A remote device can be connected to the computing device through the network.

[0121] At this point, those skilled in the art should recognize that although a number of exemplary embodiments of the present invention have been shown and described in detail herein, many other variations or modifications consistent with the principles of the present invention can still be directly determined or derived from the disclosed content of the present invention without departing from the spirit and scope of the present invention. Therefore, the scope of the present invention should be understood and determined to cover all these other variations or modifications.

Claims

1. A method for processing database logs, wherein, The database log includes an assembled linked list and a page data linked list separately stored in a cache. The page data linked list is a linked list composed of page mirror data in the database log, and the assembled linked list is a linked list composed of other data in the database log except for page mirror data. The page data linked list is stored in a preset global page data linked list. The database is configured with a page data disk write process for separately writing the page data linked list to disk. Moreover, the method for processing the database log includes: Detecting a disk write start signal for the page data linked list; Using the page data disk write process to obtain a range of log sequence numbers of the database log that conforms to a first preset rule, denoted as a first range. The database log that conforms to the first preset rule is at least one database log that has been completely written into the cache and whose log sequence numbers are consecutive starting from the log sequence number of the last database log that has completed a complete disk write; Using the page data disk write process to obtain the page data linked list in the global page data linked list whose log sequence numbers are within the first range, denoted as a first disk-writable page data linked list; Using the page data disk write process to write the first disk-writable page data linked list to disk.

2. The method for processing database logs according to claim 1, wherein, The database is further configured with a log disk write process for writing the database log to disk. Moreover, the method for processing the database log includes: Detecting a disk write start signal for the database log; Using the log disk write process to obtain a range of log sequence numbers of the database log that conforms to a second preset rule, denoted as a second range. The database log that conforms to the second preset rule is at least one database log that has been completely written into the cache and whose log sequence numbers are consecutive starting from the log sequence number of the last database log that has completed a complete disk write; Using the log disk write process to obtain the disk-writable assembled data linked list in the cache. The disk-writable assembled data linked list is an assembled data linked list whose log sequence numbers are within the second range; Using the log disk write process to write the disk-writable assembled data linked list to disk.

3. The method for processing a database log according to claim 2, wherein, Before the step of using the log disk write process to obtain the disk-writable assembled data linked list in the cache includes: Using the log disk write process to obtain the page data linked list in the global page data linked list whose log sequence numbers are within the second range, denoted as a second disk-writable page data linked list; Using the log disk write process to write the second disk-writable page data linked list to disk.

4. The method for processing database logs according to claim 3, wherein, The global page data linked list is configured with a disk write lock; Before the step of using the log disk write process to obtain the page data linked list in the global page data linked list whose log sequence numbers are within the second range includes: Using the log disk write process to obtain the disk write lock; Moreover, before the step of using the page data disk write process to obtain the page data linked list in the global page data linked list whose log sequence numbers are within the first range includes: Using the page data disk write process to obtain the disk write lock.

5. The method for processing a database log according to claim 3, wherein, Before the step of using the log disk write process to obtain the page data linked list in the global page data linked list whose log sequence numbers are within the second range includes: Detect whether the page data disk writing process is running normally. If so, use the page data disk writing process to obtain the page data linked list in the global page data linked list whose log sequence number is within the second range, and use the page data disk writing process to write the page data linked list in the global page data linked list whose log sequence number is within the second range to disk. If not, execute the step of using the log disk writing process to obtain the page data linked list in the global page data linked list whose log sequence number is within the second range.

6. The method for processing database logs according to claim 5, wherein, Before the step of detecting whether the page data disk writing process is running normally, it includes: Check whether there is a page data linked list in the global page data linked list whose log sequence number is within the second range. If so, execute the step of detecting whether the page data disk writing process is running normally. If not, execute the step of using the log disk writing process to obtain the linked list of disk-writable assembled data in the cache.

7. The method for processing database logs according to claim 3, wherein, The step of using the page data disk writing process to write the first disk-writable page data linked list to disk includes: Store the first disk-writable page data linked list into a preset page mirror data storage table; The step of using the log disk writing process to write the second disk-writable page data linked list to disk includes: Store the second disk-writable page data linked list into the page mirror data storage table.

8. The method for processing a database log according to claim 7, wherein, The page data linked lists in the global page data linked list are connected in sequence according to the ascending order of the log sequence number; The step of storing the first disk-writable page data linked list into a preset page mirror data storage table includes: Copy the page mirror data in the first disk-writable page data linked list to the last of the page mirror data storage table in sequence according to the connection order; The step of storing the second disk-writable page data linked list into the page mirror data storage table includes: Copy the page mirror data in the second disk-writable page data linked list to the last of the preset page mirror data storage table in sequence according to the connection order.

9. A machine-readable storage medium, on which a machine-executable program is stored. When the machine-executable program is executed by a processor, it implements the method for processing database logs according to any one of claims 1 to 8.

10. A computer device, including a memory, a processor, and a machine-executable program stored on the memory and running on the processor. When the processor executes the machine-executable program, it implements the method for processing database logs according to any one of claims 1 to 8.

Images