Disk refreshing method and system for database cache page

Abstract

The invention provides a disk refreshing method and system for database cache pages, and relates to the technical field of databases. A plurality of cache pages are arranged in a database cache layer, when a first cache page meets a disk refreshing condition, a second cache page is determined in a memory to serve as a data transfer page, access limitation on the first cache page is released immediately after data of the first cache page is synchronized to the second cache page, and disk refreshing operation is executed asynchronously by the second cache page. By adopting a lock replacement mechanism, the lock holding time of the source data page is shortened from a complete disk refreshing process containing disk IO to a data synchronization process only containing memory replication, and the locking time is remarkably shortened. In addition, one-time disk refreshing is carried out after the data of the multiple cache pages are merged and synchronized to the single cache page, and the IO mode of the disk is effectively optimized. According to the method, direct disk writing is replaced by memory data copying, so that the concurrency performance and throughput of a database system are greatly improved while the data consistency is ensured.

Description

Technical Field

[0001] This invention relates to a method and system for flushing database cache pages to disk, belonging to the field of database technology. Background Technology

[0002] With the continuous growth of data volume, database system performance optimization has become a key area of ​​focus in the industry. During database operation, the flushing operation of persisting dirty data in cached pages to disk is one of the critical steps affecting overall performance. Taking MySQL as an example, before writing to the actual data file, it uses a caching layer for temporary data storage. The latest data interactions are uniformly written to the page data structure of the cache layer. When flushing to disk is required, the data in the page is pre-locked. Once the lock is successfully acquired, the page is flushed to disk. The duration of the lock is the same as the flushing time. Traditional flushing mechanisms often keep data pages locked during data modification and disk writing, leading to decreased concurrency performance and making it difficult to meet the needs of high-throughput applications.

[0003] In existing technologies, such as the Chinese invention patent application CN119759867A, a method for flushing data pages to disk in a shared storage cluster is proposed. This method achieves data page flushing through cooperation between cluster nodes: when a node detects that a data page needs to be flushed, it first obtains the latest modified node information for that data page, and then forwards the flush request to the corresponding node for execution. However, the multiple request forwarding and response processes between nodes in the above-mentioned technical solution introduce significant network communication overhead, increasing the latency of the flush operation; moreover, the above-mentioned technical solution relies on a global caching service for coordination, which can easily become a performance bottleneck in high-concurrency scenarios; furthermore, complex judgment logic and multiple status queries prolong the locking time of critical data pages, reducing the system's concurrent processing capacity. These factors collectively limit the application effectiveness of the above-mentioned technical solution in database scenarios requiring low latency and high throughput.

[0004] Therefore, there is an urgent need in this field for a database cache page flushing method that can significantly reduce data page locking time, lower system overhead, and improve flushing efficiency. Summary of the Invention

[0005] To address the problems existing in the prior art, this invention proposes a method and system for flushing database cache pages to disk.

[0006] The technical solution of the present invention is as follows: On one hand, a method for flushing database cache pages to disk, the method comprising: When the first cached page in the database cache layer meets the conditions for flushing to disk, the second cached page is determined in memory; Synchronize the data in the first cache page to the second cache page; After data synchronization is completed, the access restrictions on the first cache page are released, and the second cache page performs the disk flushing operation to write the data to the persistent storage device. The disk flushing conditions are dynamically adjusted based on system load using a global coordination mechanism. This global coordination mechanism monitors system-level load indicators and, when the system-level load indicators exceed a preset threshold, proactively triggers cache pages that have not yet met the disk flushing conditions to perform disk flushing operations in advance.

[0007] Preferably, the global coordination mechanism includes monitoring the total number of dirty pages currently awaiting disk flushing in the system. When the total number of dirty pages exceeds a preset high watermark threshold, a number of dirty pages are selected based on a preset strategy, and their disk flushing process is actively triggered.

[0008] Preferably, the method further includes synchronizing the data in the first cache page and at least one other cache page that needs to be flushed to the disk to the second cache page; after the data synchronization is completed, releasing the access restrictions on the first cache page and the at least one other cache page that needs to be flushed to the disk, and having the second cache page perform a one-time merge and flush operation.

[0009] Preferably, at least one other cache page that needs to be flushed to disk is selected based on a physical proximity strategy or a transaction correlation strategy. Specifically, the physical proximity strategy prioritizes merging cache pages whose data to be flushed to disk is contiguous or adjacent on the disk physical address; the transaction correlation strategy prioritizes merging cache pages that have been modified in the same database transaction.

[0010] Preferably, determining the second cache page in memory includes: Find an idle cache page from among the multiple cache pages preset in the database cache layer to serve as the second cache page.

[0011] Preferably, when there are no free cache pages in the database cache layer, determining the second cache page in memory includes requesting and creating a new cache page from spare memory space outside the database cache layer as the second cache page.

[0012] Preferably, synchronizing data from the first cache page to the second cache page includes: A synchronization lock is applied to the first cache page and the second cache page, restricting access to the first cache page and the second cache page during data synchronization.

[0013] On the other hand, the present invention also provides a system for flushing database cache pages to disk, the system comprising: The cache page determination module is used to determine a second cache page in memory when the first cache page in the database cache layer meets the flushing conditions. This includes finding a free cache page from a set of preset cache pages in the database cache layer as the second cache page; if no free cache page exists in the database cache layer, a new cache page is requested and created from spare memory space outside the database cache layer as the second cache page. The flushing conditions are dynamically adjusted based on system load using a global coordination mechanism. The global coordination mechanism monitors system-level load indicators and, when the system-level load indicators exceed a preset threshold, proactively triggers cache pages that have not yet met the flushing conditions to perform flushing operations in advance. The data synchronization module is used to synchronize the data in the first cache page to the second cache page; during the synchronization process, a synchronization lock is applied to the first cache page and the second cache page, and access to the first cache page and the second cache page is restricted during the data synchronization; based on the physical location proximity strategy or the transaction correlation strategy, at least one other cache page that needs to be flushed to disk is selected, and the data in the first cache page is synchronized to the second cache page together with the data in the first cache page; The access restriction release module is used to release the access restriction on the first cache page after data synchronization is completed; The disk flushing operation execution module is used to execute the disk flushing operation of writing data to the persistent storage device by the second cache page; when other cache page data that needs to be flushed is synchronized to the second cache page, a one-time merge disk flushing operation is performed; based on the global coordination mechanism, the total number of dirty pages to be flushed in the system is monitored, and when the total number of dirty pages exceeds the preset high watermark threshold, several dirty pages are selected based on the preset strategy, and their disk flushing process is actively triggered.

[0014] In another aspect, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement a method for flushing database cache pages as described in the present invention.

[0015] In another aspect, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a method for flushing database cache pages as described in the present invention.

[0016] The present invention has the following beneficial effects: This invention relates to a method and system for flushing database cache pages to disk. By establishing a multi-page management mechanism and lock replacement strategy within the cache layer, the locking time of the source data page is shortened from a complete flushing process including slow disk I / O to a data synchronization stage that only includes memory copying, allowing the source data page to be released immediately for business use. By introducing a global coordination mechanism based on dynamic adjustment of system load, the flushing operation is intelligently triggered and multi-page data merging is supported for flushing. Multiple random disk writes are optimized into sequential writes, ultimately achieving a significant improvement in the concurrency performance and I / O throughput of the database system while ensuring data consistency. Attached Figure Description

[0017] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] It should be understood that the step numbers used in the text are for ease of description only and are not intended to limit the order in which the steps are performed.

[0020] It should be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0021] The terms “comprising” and “including” indicate the presence of the described feature, whole, step, operation, element and / or component, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or collections thereof.

[0022] The term “and / or” refers to any combination of one or more of the associated listed items, as well as all possible combinations, and includes these combinations.

[0023] Example 1: like Figure 1 As shown, this embodiment provides a method for flushing database cache pages to disk, the method including: S1. When the first cache page in the database cache layer meets the flushing conditions, the second cache page is determined in memory. Preferably, the database system monitors the status of each cache page in the cache layer and maintains a state machine for each cache page, which includes READ_WRITE (readable and writable), SOURCE_LOCKED (source locked), TARGET_RECEIVING (target received), and FLUSHING (flushing to disk). The state machine is used to perform state transitions during the flushing process. The flushing conditions include, but are not limited to: the amount of data in the cache page reaches the maximum page capacity, or the cache page is in a non-empty state and its continuous dirty page state exceeds a preset time limit. Furthermore: The disk flushing conditions are dynamically adjusted based on system load using a global coordination mechanism. This global coordination mechanism monitors system-level load indicators and, when the system-level load indicators exceed a preset threshold, proactively triggers cache pages that have not yet met the disk flushing conditions to perform disk flushing operations in advance. In this embodiment, the global coordination mechanism is implemented by an independent background daemon thread. This thread collects system metrics at a fixed sampling period (e.g., 100 milliseconds) and uses a sliding time window model to smooth the metrics, eliminating instantaneous peak interference. The background daemon thread continuously monitors system-level load metrics, such as the total number of dirty pages waiting to be flushed to disk. When the total number of dirty pages exceeds the high watermark threshold (e.g., 60% of the total number of pages) for three consecutive sampling periods, it indicates that too much data to be persisted has accumulated in the system memory, posing a risk of excessive I / O pressure. At this time, the global coordination mechanism will no longer passively wait for a single cache page to meet its own flushing conditions, but will instead, based on a preset selection strategy (e.g., selecting the page that became dirty earliest), specifically by scanning the LRU dirty page list, selecting the page that has not been accessed for the longest time for priority processing, and actively triggering a batch of dirty pages to perform flushing operations in advance. The dynamic adjustment mechanism in this embodiment helps to smooth I / O pressure and avoid system performance jitter caused by the concentrated flushing of all dirty pages during peak business periods. After determining that a disk flushing operation is required, the system determines a second cache page in memory to receive data from the first cache page. In this embodiment, the system first queries whether there are any spare pages in the multiple cache pages preset in the database cache layer (e.g., a cache pool of 100 pages). When there are spare pages, they are selected as the second cache page, giving priority to reusing the existing memory resources in the cache layer and avoiding additional memory consumption. Preferably, if there is no free cache page in the database cache layer, a new cache page is requested and created from the spare memory space outside the database cache layer as the second cache page; S2. Synchronize the data in the first cache page to the second cache page. Preferably, during the data synchronization process, a synchronization lock is applied to the first cache page and the second cache page. During this period, access to the first cache page and the second cache page is restricted. In this embodiment, the access restriction specifically means that the first cache page and the second cache page can only perform one-way data synchronization operations from the first cache page to the second cache page, and cannot perform other data reading or writing operations. That is, the first cache page allows reading operations but blocks writing operations in the SOURCE_LOCKED state, and the second cache page completely blocks all reading and writing operations in the TARGET_RECEIVING state, allowing only the data synchronization thread to write. Furthermore, this embodiment also includes a merge flush operation, which synchronizes the data in at least one other cache page that needs to be flushed to the second cache page. In this embodiment, the selection of the at least one other cache page that needs to be flushed is based on a specific strategy, including a physical proximity strategy. The system maintains a dirty page index sorted by disk physical address to quickly locate other dirty pages that are physically contiguous with the first cache page. That is, it prioritizes merging cache pages whose data to be flushed is contiguous or adjacent in terms of disk physical address, converting multiple random disk writes into a single sequential write, which greatly improves I / O efficiency. It also includes a transaction correlation strategy, which groups dirty pages modified within the same transaction by comparing transaction IDs. That is, it prioritizes merging cache pages that have been modified within the same database transaction. S3. After data synchronization is completed, the access restrictions on the first cache page are released by switching the state of the first cache page from SOURCE_LOCKED back to READ_WRITE. At this time, the first cache page is restored to a read-write state and can immediately respond to new business operations. Its lock holding time is greatly shortened to the time required for one memory copy. The second cache page then performs a disk flushing operation to write data to the persistent storage device. At this time, the second cache page enters the FLUSHING state, and a dedicated asynchronous I / O thread performs the disk flushing. This thread uses vectorized write operations to combine multiple non-contiguous buffers into a single I / O request. Before the disk flushing operation is completed, the second cache page is set to a non-write state and data reading is not allowed to ensure data consistency. The disk flushing operation is performed by an independent asynchronous thread, realizing the complete separation of the disk flushing process from the foreground business processing.

[0024] Example 2: This embodiment provides a database cache page flushing system, the system comprising: The cache page determination module is used to determine a second cache page in memory when the first cache page in the database cache layer meets the flushing conditions. This includes finding a free cache page from a set of preset cache pages in the database cache layer as the second cache page; if no free cache page exists in the database cache layer, a new cache page is requested and created from spare memory space outside the database cache layer as the second cache page. The flushing conditions are dynamically adjusted based on system load using a global coordination mechanism. The global coordination mechanism monitors system-level load indicators and, when the system-level load indicators exceed a preset threshold, proactively triggers cache pages that have not yet met the flushing conditions to perform flushing operations in advance. The data synchronization module is used to synchronize the data in the first cache page to the second cache page; during the synchronization process, a synchronization lock is applied to the first cache page and the second cache page, and access to the first cache page and the second cache page is restricted during the data synchronization; based on the physical location proximity strategy or the transaction correlation strategy, at least one other cache page that needs to be flushed to disk is selected, and the data in the first cache page is synchronized to the second cache page together with the data in the first cache page; The access restriction release module is used to release the access restriction on the first cache page after data synchronization is completed; The disk flushing operation execution module is used to execute the disk flushing operation of writing data to the persistent storage device by the second cache page; when other cache page data that needs to be flushed is synchronized to the second cache page, a one-time merge disk flushing operation is performed; based on the global coordination mechanism, the total number of dirty pages to be flushed in the system is monitored, and when the total number of dirty pages exceeds the preset high watermark threshold, several dirty pages are selected based on the preset strategy, and their disk flushing process is actively triggered.

[0025] Example 3: This embodiment provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements a method for flushing database cache pages to disk as described in any embodiment of the present invention.

[0026] Example 4: This embodiment provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements a method for flushing database cache pages to disk as described in any embodiment of the present invention.

[0027] It is worth noting that the system, electronic device, and computer-readable storage medium described in this invention are all based on the same inventive concept as the method described in this invention, and will not be described again here.

[0028] In this embodiment of the invention, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent the existence of A alone, A and B simultaneously, or B alone. A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects have an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of singular or plural items. For example, at least one of a, b, and c can represent: a, b, c, a and b, a and c, b and c, or a and b and c, where a, b, and c can be single or multiple.

[0029] Those skilled in the art will recognize that the units and algorithm steps described in the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of electronic hardware and software. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0030] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0031] In several embodiments provided by this invention, any function, if implemented as a software functional unit and sold or used as an independent product, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0032] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A method for flushing database cache pages to disk, characterized in that, The method includes: When the first cached page in the database cache layer meets the conditions for flushing to disk, the second cached page is determined in memory; Synchronize the data in the first cache page to the second cache page; After data synchronization is completed, the access restrictions on the first cache page are released, and the second cache page performs the disk flushing operation to write the data to the persistent storage device. The disk flushing conditions are dynamically adjusted based on system load using a global coordination mechanism. This global coordination mechanism monitors system-level load indicators and, when the system-level load indicators exceed a preset threshold, proactively triggers cache pages that have not yet met the disk flushing conditions to perform disk flushing operations in advance.

2. The method for flushing database cache pages to disk according to claim 1, characterized in that, The global coordination mechanism includes monitoring the total number of dirty pages currently awaiting flushing in the system. When the total number of dirty pages exceeds a preset high watermark threshold, a number of dirty pages are selected based on a preset strategy, and their flushing process is actively triggered.

3. The method for flushing database cache pages to disk according to claim 1, characterized in that, The method further includes synchronizing the data in the first cache page and at least one other cache page that needs to be flushed to the disk to the second cache page; after the data synchronization is completed, the access restrictions on the first cache page and the at least one other cache page that needs to be flushed to the disk are released, and the second cache page performs a one-time merge and flush operation.

4. The method for flushing database cache pages to disk according to claim 3, characterized in that, Based on a physical proximity strategy or a transaction correlation strategy, at least one other cache page that needs to be flushed to disk is selected. Specifically, the physical proximity strategy prioritizes merging cache pages whose data to be flushed to disk is contiguous or adjacent on the disk physical address. Specifically, the transaction correlation strategy prioritizes merging cache pages that have been modified in the same database transaction.

5. A method for flushing database cache pages to disk according to any one of claims 1-4, characterized in that, The step of determining the second cache page in memory includes: Find an idle cache page from among the multiple cache pages preset in the database cache layer to serve as the second cache page.

6. The method for flushing database cache pages to disk according to claim 5, characterized in that, When there are no free cache pages in the database cache layer, determining the second cache page in memory includes requesting and creating a new cache page from the spare memory space outside the database cache layer as the second cache page.

7. A method for flushing database cache pages to disk according to any one of claims 1, characterized in that, Synchronizing data from the first cache page to the second cache page includes: A synchronization lock is applied to the first cache page and the second cache page, restricting access to the first cache page and the second cache page during data synchronization.

8. A database cache page flushing system, characterized in that, The system includes: The cache page determination module is used to determine a second cache page in memory when the first cache page in the database cache layer meets the flushing conditions. This includes finding a free cache page from a set of preset cache pages in the database cache layer as the second cache page; if no free cache page exists in the database cache layer, a new cache page is requested and created from spare memory space outside the database cache layer as the second cache page. The flushing conditions are dynamically adjusted based on system load using a global coordination mechanism. The global coordination mechanism monitors system-level load indicators and, when the system-level load indicators exceed a preset threshold, proactively triggers cache pages that have not yet met the flushing conditions to perform flushing operations in advance. The data synchronization module is used to synchronize the data in the first cache page to the second cache page; during the synchronization process, a synchronization lock is applied to the first cache page and the second cache page, and access to the first cache page and the second cache page is restricted during the data synchronization; based on the physical location proximity strategy or the transaction correlation strategy, at least one other cache page that needs to be flushed to disk is selected, and the data in the first cache page is synchronized to the second cache page together with the data in the first cache page; The access restriction release module is used to release the access restriction on the first cache page after data synchronization is completed; The disk flushing operation execution module is used to execute the disk flushing operation of writing data to the persistent storage device by the second cache page; when other cache page data that needs to be flushed is synchronized to the second cache page, a one-time merge disk flushing operation is performed; based on the global coordination mechanism, the total number of dirty pages to be flushed in the system is monitored, and when the total number of dirty pages exceeds the preset high watermark threshold, several dirty pages are selected based on the preset strategy, and their disk flushing process is actively triggered.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements a method for flushing database cache pages as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by the processor, the program implements a method for flushing database cache pages as described in any one of claims 1 to 7.

Images