A method and system for handling cache failure and disk flushing
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CHINA ELECTRONICS CLOUD DIGITAL INTELLIGENCE TECH CO LTD
- Filing Date
- 2026-03-11
- Publication Date
- 2026-06-23
AI Technical Summary
Existing technologies for handling cache failures suffer from high CPU overhead, high cache resource consumption, and increased disk flushing overhead, which can easily lead to data inconsistency, especially when cache anomalies occur.
By monitoring the cache status and recording the timestamps of cache status changes, an interval management table is generated, dividing the time intervals into valid and invalid ones. Data is directly written to the disk, skipping the invalid intervals, and only flushing to disk is performed within the valid intervals.
It reduces the overhead of looking up and processing passthrough requests, improves the efficiency of cache failure handling, avoids duplicate insertion and merging processes, and enhances data consistency.
Smart Images
Figure CN122262020A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of computer technology, specifically relating to a method and system for flushing cache faults. Background Technology
[0002] For a write caching system, it's typically necessary to consider how to handle write-through requests in case of cache failure or cache breakdown. This is because conflicts between dirty data in the write cache and write-through data can lead to data inconsistency if left unhandled. Currently, there are generally two approaches to handling this:
[0003] The first method is to find the overwritten regions in the write cache and mark them for deletion; the second method is to insert a new overwritten region and merge it when the write cache is flushed to disk.
[0004] Both methods have some drawbacks. The first requires searching, increasing CPU overhead. The second, when writing to the same area multiple times, involves duplicate insertions, increasing cache resource consumption and adding merging processing overhead during disk flushing. Summary of the Invention
[0005] The purpose of this invention is to provide a method and system for flushing data to handle cache failures, thereby improving the aforementioned problems. To achieve this objective, the technical solution adopted by this invention is as follows:
[0006] Firstly, this application provides a disk flushing method for handling cache failures, including:
[0007] Monitor cache status;
[0008] Based on the cache status, record the timestamp of the cache status change, and also record the validity of the timestamp, and generate an interval management table;
[0009] Based on the interval management table, divide the valid time intervals and invalid time intervals;
[0010] Skip all invalid time intervals, traverse the disk flushing table within the valid time intervals, and extract cached data for disk flushing.
[0011] Secondly, this application also provides a disk flushing system for handling cache failures, comprising:
[0012] The first module is used to monitor cache status;
[0013] The second module is used to record the timestamps of cache status changes based on the cache status, and at the same time record the validity of the timestamps, and generate an interval management table.
[0014] The third module is used to divide valid time intervals and invalid time intervals according to the interval management table;
[0015] The fourth module is used to skip all invalid time intervals, traverse the disk flushing table within the valid time intervals, and extract cached data for disk flushing.
[0016] The beneficial effects of this invention are as follows:
[0017] This invention divides the disk flushing interval into valid and invalid intervals by recording the cache state change times. Data written to the invalid interval is directly flushed, thus eliminating the need to insert data into the disk flushing table. When a cache anomaly occurs, searching for data in the invalid interval is eliminated, reducing the search range and effectively minimizing the overhead of processing flushing requests.
[0018] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing embodiments of the invention. Attached Figure Description
[0019] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.
[0020] Figure 1 This is a flowchart of a disk flushing method for handling cache failures according to an embodiment of this application;
[0021] Figure 2 This is a schematic diagram of the partitioning of the interval management table in an embodiment of this application;
[0022] Figure 3 This is a diagram of a disk flushing device used to handle cache failures in an embodiment of this application.
[0023] Symbol explanation: 800 - Disk flushing device for handling cache faults; 801 - Processor; 802 - Memory; 803 - Multimedia component; 804 - I / O interface; 805 - Communication component. Detailed Implementation
[0024] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.
[0025] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.
[0026] In high-throughput, high-reliability distributed storage, database, and caching systems, the cache layer serves as a crucial buffer between data writing and persistence, and its stability directly determines system availability and data consistency. In actual operation, cache disks often face issues such as hardware failures, insufficient capacity, I / O anomalies, and read-only mounting. During failures, data written to the cache is prone to corruption, incompleteness, or invalidity; directly flushing to disk would lead to data pollution in the persistent layer. Existing methods for handling cache anomalies increase write overhead or lookup overhead, resulting in wasted resources.
[0027] Example 1;
[0028] See Figure 1 To address the existing problems, this application provides a disk flushing method for handling cache failures, including steps S100, S200, S300 and S400.
[0029] S100, monitor cache status;
[0030] This step monitors the cache status to identify whether the cache is writable or not. When the cache is writable, it corresponds to a valid period; data written to the cache during this period needs to be flushed to disk. When the cache is not writable, it corresponds to an invalid period; data written during this period is directly written through without requiring a cache flush. Specific monitoring details are as follows:
[0031] S110: Monitor cache disk status, cache disk capacity, and IO status, and obtain trigger events;
[0032] Monitoring the cache disk status includes: checking whether the physical device corresponding to the cache disk (such as / dev / sdb) is in the system device list, identifying whether there are situations such as disk disconnection, hot-plug removal, hardware recognition failure, etc., and triggering corresponding events; checking the hardware health status of the disk, reading information such as the number of bad sectors, the number of reallocated sectors, disk temperature, and power cycle count, and determining disk abnormality if key indicators exceed preset alarm thresholds.
[0033] Cache disk capacity monitoring includes: collecting the total capacity, used capacity, and reserved space of the cache disk, calculating the actual available remaining capacity, and determining whether an insufficient capacity event is triggered based on the remaining capacity.
[0034] Monitoring IO status includes: listening to the return codes of all business write / read cache disk IO operations, classifying and capturing error types, and triggering corresponding events;
[0035] S120. Determine whether the triggered event matches the preset invalidation event. If so, record the timestamp of the triggered event and record the corresponding cache status as invalid.
[0036] As an example, the default failure events include disk failure, insufficient capacity, IO failure, etc. The occurrence of these events indicates that the cached data written at this time is invalid;
[0037] If not, i.e., if it is another event, then record the timestamp of the cached event and record the corresponding cache status as valid; at this time, the cache function is normal.
[0038] S200. Based on the cache status, record the timestamp of the cache status change, and at the same time record the validity of the timestamp, and generate an interval management table.
[0039] Initially, an initial record is inserted into the interval management table with timestamp 0 and status "valid", meaning the cache is initially in a valid state.
[0040] When a cache invalidation event occurs, a record is inserted into the interval management table with the timestamp of the current event and the status "invalid". For example, in Table 1, a record is inserted (timestamp: 100, status: invalid).
[0041] When the cache is restored and becomes writable, a cache restore event is generated. A record is inserted into the disk flush interval management table with the timestamp of the current event and the status "valid". Taking Table 1 as an example, a record is inserted (timestamp: 200, status: valid);
[0042] Similarly, whenever the cache state changes (switching between valid and invalid), a record is inserted into the disk flushing interval management table, recording the timestamp of the event and its state, generating the interval management table as shown in Table 1. If adjacent states are identical, there is no need to insert again. For example, if the most recent record's state is invalid, and another cache invalidation event occurs, there is no need to insert again into the interval management table.
[0043] Table 1 Interval Management Table
[0044]
[0045] For caching systems implemented using Multi-Version Concurrency Control (MVCC), since each piece of data written carries a unique version number, the version number can be used to replace the timestamp in the interval management table, and the version number can be used for data lookup.
[0046] Upon receiving a write request, this method performs the following operations:
[0047] Identify the validity of the latest interval in the interval management table;
[0048] If the latest interval is an invalid time interval, then the data requested to be written within the latest invalid time interval will be pass-through; this data does not need to be inserted into the disk flush table.
[0049] If the latest interval is a valid time interval, then the data requested to be written within the latest valid time interval will be inserted into the disk flush table.
[0050] S300. Based on the interval management table, divide the valid time interval into invalid time intervals;
[0051] When flushing the cache to disk, the interval management table is read sequentially from smallest to largest timestamp. The time from the current state timestamp to the next state timestamp constitutes one flush interval, and the time from the last state timestamp to the latest current time point constitutes the last (latest) flush interval. Flushing intervals are divided into two states: valid and invalid, depending on the state corresponding to the start timestamp of the flush interval.
[0052] Table 1 and Figure 2 For example, first read (timestamp: 0; status: valid). The next status timestamp is (timestamp: 100, status: invalid). Therefore, the first disk flush interval is [0, 100), which is a valid disk flush interval. The next disk flush interval starts with (timestamp: 100, status: invalid) and ends with (timestamp: 200, status: valid), resulting in a disk flush interval of [100, 200), which is an invalid disk flush interval. This process continues until the last disk flush interval starts with (timestamp: 800, status: valid), corresponding to a disk flush interval of [800, current), which is also a valid disk flush interval.
[0053] S400: Skip all invalid time intervals, traverse the disk flushing table within the valid time interval, extract cached data, and flush the disk.
[0054] For a valid disk flushing interval, the disk flushing table is traversed according to the timestamp range specified for the disk flushing interval. The data written to the cache within the valid disk flushing interval is found, and the cache flushing process is completed.
[0055] For invalid disk flushing ranges, the entire range has already been written when a write request is received, so there is no need for cache flushing and the range can be skipped directly.
[0056] During the disk flushing process, if the flushing fails (e.g., due to a target disk I / O error), the data that has already been successfully flushed needs to be traversed from the beginning and processed repeatedly, which is inefficient and wasteful of resources. Therefore, this invention adopts the following method to optimize it:
[0057] The disk flushing table is divided into flushing intervals based on the effective time interval;
[0058] For each disk flushing interval, record the maximum offset of the flushed data and the last successful timestamp; the maximum offset of the flushed data is used to mark the position of the last data unit that has been successfully flushed within that interval up to the current time, i.e., the breakpoint position.
[0059] After a disk flushing interruption, the breakpoint is located based on the maximum offset of the flushed disk and the last successful timestamp. The flushing process resumes from the breakpoint, eliminating the need to retrace the entire valid range. This significantly reduces invalid I / O and redundant processing, improving flushing efficiency.
[0060] As an optional implementation method, the disk refresh frequency is adaptively adjusted based on multi-dimensional monitoring indicators, as follows:
[0061] The first indicator is obtained by calculating the proportion of the total duration of the effective time interval within the preset time window to the total window duration; the higher the proportion, the more stable the cache state, and the disk flushing interval can be appropriately extended.
[0062] The number of state changes within a preset time window is calculated to obtain the second indicator; high-frequency switching indicates intense multi-core competition, and the disk flushing interval should be shortened to ensure timely data persistence.
[0063] The rate at which new data items are added to the disk flushing table is calculated to obtain a third indicator; when the rate is too high, the flushing interval should be appropriately shortened to prevent the disk flushing table from overflowing.
[0064] The target refresh frequency is calculated based on the first, second, and third indicators;
[0065] Based on the target refresh frequency, the original refresh frequency is gradually adjusted at preset intervals and proportions to obtain the updated refresh frequency; the gradual adjustment, for example, increases or decreases by 15% of the current value each time, to avoid system instability caused by drastic changes.
[0066] Disk flushing is performed based on the updated flushing frequency.
[0067] In this method, the cleanup rules for the interval management table are as follows:
[0068] After each disk flush is completed, the cached data corresponding to the smallest write timestamp is deleted from the cache, and the timestamp of the disk flush table is updated to realize the space reclamation of the disk flush table.
[0069] Get the minimum write timestamp after the update of the disk flushing table, compare it with the timestamp in the interval management table, find the record whose timestamp is less than the minimum write timestamp after the update, and get the record to be cleared;
[0070] Determine whether the record to be cleared is the latest record in the interval management table. If not, delete the record to be cleared.
[0071] by Figure 2 For example, after the disk flush interval [0, 100) is flushed, the minimum write timestamp changes to be greater than 200. Therefore, the data corresponding to the records (timestamp: 0, status: valid) and (timestamp: 100, status: invalid) have been persisted or do not need to be persisted, and can be deleted to free up storage space. However, the last record in the disk flush interval management table cannot be deleted, as it is used as the basis for determining the current cache status.
[0072] A caching system based on the Multi-Version Concurrency Control (MVCC) mechanism assigns a unique version number to each data write. The flush table can be divided into multiple intervals based on the version number, and this mechanism can be used to isolate valid and invalid intervals. For non-MVCC caching systems, the method in this application can also be used to isolate valid and invalid intervals in the flush table based on the timestamp, which can reduce the search range. Data below the invalid interval can be flushed to disk without needing to be searched.
[0073] Example 2:
[0074] This embodiment provides a disk flushing system for handling cache failures, including:
[0075] The first module is used to monitor cache status;
[0076] The second module is used to record the timestamps of cache status changes based on the cache status, and at the same time record the validity of the timestamps, and generate an interval management table.
[0077] The third module is used to divide valid time intervals and invalid time intervals according to the interval management table;
[0078] The fourth module is used to skip all invalid time intervals, traverse the disk flushing table within the valid time intervals, and extract cached data for disk flushing.
[0079] As an optional implementation, the first module includes:
[0080] The first unit is used to monitor cache disk status, cache disk capacity, and I / O status, and to obtain triggering events.
[0081] The second unit is used to determine whether the triggered event matches the preset invalidation event. If so, it records the timestamp of the triggered event and records the corresponding cache status as invalid.
[0082] The third unit is used to record the timestamp of the cached event if no, and to record the corresponding cache status as valid.
[0083] As an optional implementation, the system further includes:
[0084] The fifth module is used to identify the validity of the latest interval in the interval management table;
[0085] The sixth module is used to perform pass-through writing of the data requested to be written within the latest invalid time interval if the latest interval is an invalid time interval;
[0086] The seventh module is used to insert the data requested to be written within the latest valid time interval into the disk flush table if the latest interval is a valid time interval.
[0087] As an optional implementation, the system further includes:
[0088] The eighth module is used to delete the cached data corresponding to the minimum write timestamp from the cache and update the timestamp of the disk flush table after each disk flush is completed.
[0089] The ninth module is used to obtain the minimum write timestamp after the update of the disk flushing table, compare it with the timestamp in the interval management table, find the record whose timestamp is less than the minimum write timestamp after the update, and obtain the record to be cleared.
[0090] The tenth module is used to determine whether the record to be cleared is the latest record in the interval management table. If not, the record to be cleared is deleted.
[0091] Example 3:
[0092] Corresponding to the above method embodiments, this embodiment also provides a disk flushing device for handling cache failures. The disk flushing device for handling cache failures described below and the disk flushing method for handling cache failures described above can be referred to in correspondence.
[0093] Figure 3 This is a block diagram illustrating a disk flushing device 800 for handling cache failures according to an exemplary embodiment. Figure 3 As shown, the disk flushing device 800 for handling cache failures includes a processor 801 and a memory 802. The disk flushing device 800 may also include one or more of a multimedia component 803, an input / output (I / O) interface 804, and a communication component 805. The processor 801 controls the overall operation of the disk flushing device 800 to complete all or part of the steps in the disk flushing method for handling cache failures described above. The memory 802 stores various types of data to support the operation of the disk flushing device 800. This data may include, for example, commands for any application or method operating on the disk flushing device 800, and application-related data such as contact data, sent and received messages, pictures, audio, video, etc. The memory 802 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.
[0094] Multimedia component 803 may include a screen and an audio component. The screen may be, for example, a touchscreen, and the audio component is used to output and / or input audio signals. For example, the audio component may include a microphone for receiving external audio signals.
[0095] The received audio signal can be further stored in memory 802 or transmitted via communication component 805. The audio component also includes at least one speaker for outputting audio signals. I / O interface 804 provides an interface between processor 801 and other interface modules, such as a keyboard, mouse, buttons, etc. These buttons can be virtual or physical. Communication component 805 is used for wired or wireless communication between the disk flushing device 800 (processing cache faults) and other devices. Wireless communication includes, for example, Wi-Fi, Bluetooth, Near Field Communication (NFC), 2G, 3G, or 4G, or a combination thereof; therefore, the corresponding communication component 805 may include a Wi-Fi module, a Bluetooth module, or an NFC module.
[0096] Example 4:
[0097] Corresponding to the above embodiment of the disk flushing method for handling cache failures, this embodiment also provides a readable storage medium. The readable storage medium described below can be referred to in conjunction with the disk flushing method for handling cache failures described above.
[0098] A readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-described embodiment of the disk flushing method for handling cache failures.
[0099] Specifically, the readable storage medium can be a USB flash drive, external hard drive, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk, or any other readable storage medium capable of storing program code.
[0100] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0101] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.
Claims
1. A method for flushing data to handle cache failures, characterized in that, include: Monitor cache status; Based on the cache status, record the timestamp of the cache status change, and also record the validity of the timestamp, and generate an interval management table; Based on the interval management table, divide the valid time intervals and invalid time intervals; Skip all invalid time intervals, traverse the disk flushing table within the valid time intervals, and extract cached data for disk flushing.
2. The method for flushing disks to handle cache failures according to claim 1, characterized in that, The monitoring of cache status includes: Monitor cache disk status, cache disk capacity, and I / O status, and obtain trigger events; Determine if the triggered event matches a preset invalidation event. If so, record the timestamp of the triggered event and record the corresponding cache status as invalid. If not, record the timestamp of the cached event and record the corresponding cache status as valid.
3. The method for flushing disks to handle cache failures according to claim 1, characterized in that, The method includes: Identify the validity of the latest interval in the interval management table; If the latest interval is an invalid time interval, then the data requested to be written within the latest invalid time interval will be overwritten; If the latest interval is a valid time interval, then the data requested to be written within the latest valid time interval will be inserted into the disk flush table.
4. The method for flushing disks to handle cache failures according to claim 1, characterized in that, The method includes: After each disk flush is completed, the cached data corresponding to the smallest write timestamp is deleted from the cache, and the timestamp of the disk flush table is updated. Get the minimum write timestamp after the update of the disk flushing table, compare it with the timestamp in the interval management table, find the record whose timestamp is less than the minimum write timestamp after the update, and get the record to be cleared; Determine whether the record to be cleared is the latest record in the interval management table. If not, delete the record to be cleared.
5. The method for flushing disks to handle cache failures according to claim 1, characterized in that, The method includes: The first indicator is obtained by calculating the proportion of the total duration of the effective time interval within the preset time window to the total window duration. The second indicator is obtained by calculating the number of state changes within a preset time window; The rate at which new data items are added to the disk flushing table is calculated to obtain the third indicator; The target refresh frequency is calculated based on the first, second, and third indicators; Based on the target disk refresh frequency, the original disk refresh frequency is gradually adjusted according to preset intervals and proportions to obtain the updated disk refresh frequency. Disk flushing is performed based on the updated flushing frequency.
6. The method for flushing disks to handle cache failures according to claim 1, characterized in that, The step of extracting cached data and flushing it to disk includes: The disk flushing table is divided into flushing intervals based on the effective time interval; For each disk flushing interval, record the maximum offset of the flushed data and the last successful timestamp. After the disk flushing is interrupted, the breakpoint is located based on the maximum offset of the flushed disk and the last successful timestamp, and the disk flushing is restarted from the breakpoint.
7. A disk flushing system for handling cache failures, characterized in that, include: The first module is used to monitor cache status; The second module is used to record the timestamps of cache status changes based on the cache status, and at the same time record the validity of the timestamps, and generate an interval management table. The third module is used to divide valid time intervals and invalid time intervals according to the interval management table; The fourth module is used to skip all invalid time intervals, traverse the disk flushing table within the valid time intervals, and extract cached data for disk flushing.
8. A disk flushing system for handling cache failures according to claim 7, characterized in that, The first module includes: The first unit is used to monitor cache disk status, cache disk capacity, and I / O status, and to obtain triggering events. The second unit is used to determine whether the triggered event matches the preset invalidation event. If so, it records the timestamp of the triggered event and records the corresponding cache status as invalid. The third unit is used to record the timestamp of the cached event if no, and to record the corresponding cache status as valid.
9. A disk flushing system for handling cache failures according to claim 7, characterized in that, The system also includes: The fifth module is used to identify the validity of the latest interval in the interval management table; The sixth module is used to perform pass-through writing of the data requested to be written within the latest invalid time interval if the latest interval is an invalid time interval; The seventh module is used to insert the data requested to be written within the latest valid time interval into the disk flush table if the latest interval is a valid time interval.
10. A disk flushing system for handling cache failures according to claim 7, characterized in that, The system also includes: The eighth module is used to delete the cached data corresponding to the minimum write timestamp from the cache and update the timestamp of the disk flush table after each disk flush is completed. The ninth module is used to obtain the minimum write timestamp after the update of the disk flushing table, compare it with the timestamp in the interval management table, find the record whose timestamp is less than the minimum write timestamp after the update, and obtain the record to be cleared. The tenth module is used to determine whether the record to be cleared is the latest record in the interval management table. If not, the record to be cleared is deleted.