Hard disk read-write optimization scheduling method and system based on multi-level cache queue

Abstract

This invention relates to the field of read / write scheduling technology, specifically to a disk read / write optimization scheduling method and system based on a multi-level cache queue. The method includes: constructing a multi-level cache queue, comprising a hot queue, a warm queue, a cold queue, a temporary write queue, and a persistent commit write queue; configuring corresponding initial dispatch ratios for each of the hot, warm, cold, temporary, and persistent commit write queues; receiving read / write requests; extracting data identification information from the read / write requests; and allocating an access frequency counter and a residence timestamp to each read / write request. This invention effectively distinguishes between data blocks with stable access patterns and those with bursty access patterns by performing differentiated decay processing on the counters of unaccessed requests according to queue type, and then performing cold queue upgrades and hot queue downgrades based on the decayed counter values. This dynamic heat assessment mechanism ensures that high-frequency access data resides preferentially in high-priority queues, while low-frequency data is promptly relegated to lower-priority queues.

Description

Technical Field

[0001] This invention relates to the field of read / write scheduling technology, specifically to a method and system for optimizing hard disk read / write scheduling based on a multi-level cache queue. Background Technology

[0002] Currently, traditional cache scheduling typically employs fixed partitions or a simple LRU strategy, lacking a dynamic hotness grading mechanism based on access frequency decay and residence time. This results in cold data occupying high resources for a long time while hot data is prematurely eliminated due to short-term bursts, leading to large fluctuations in cache hit rate and unstable read request response latency.

[0003] In addition, traditional write processing is mostly instant disk writing or simple merging, lacking fine detection and aggregation capabilities for overlapping and adjacent write request addresses. At the same time, it does not set up an explicit persistence priority queue and doorbell register collaborative dispatch mechanism, resulting in a large number of write I / O operations and seek overhead under high concurrency, uncontrollable persistence delay of critical data, and low overall write efficiency and consistency guarantee. Summary of the Invention

[0004] To achieve the above objectives, the present invention provides the following technical solution: a disk read / write optimization scheduling method based on a multi-level cache queue, comprising: Construct a multi-level cache queue, which includes a hot queue, a warm queue, a cold queue, a temporary write queue, and a persistent commit write queue, and configure the corresponding initial dispatch ratio for each of the hot queue, warm queue, cold queue, temporary write queue, and persistent commit write queue; Receive read and write requests, extract data identification information from the read and write requests, assign an access frequency counter and a residence timestamp to each read and write request, and assign the read request to one of the hot queue, warm queue or cold queue as the read target queue according to the initial count value of the access frequency counter and the initial value of the residence timestamp, and unconditionally store the write request in the temporary write queue. In each scheduling cycle, for target read and write requests in the multi-level cache queue that have not been accessed since the previous scheduling cycle, the access frequency counter is decayed according to a preset decay coefficient, and the decayed count value is obtained and updated to the access frequency counter. Based on the decayed count value and the dwell timestamp, perform an upgrade operation on read requests in the cold queue or a downgrade operation on read requests in the hot queue; In the temporary write queue, address overlap detection and address adjacency detection are performed on the written requests that have been stored. If an existing write request that overlaps with the address of the current write request to be processed is detected, a merged write request to be persisted is generated based on the current write request to be processed and the existing write request, and the existing write request is replaced. If a consecutive write request that is adjacent to the address of the current write request to be processed and meets the preset adjacency condition is detected, an aggregated write request to be persisted is generated based on the current write request to be processed and the consecutive write requests. When a synchronization instruction containing an explicit persistence flag is received, the synchronization instruction is parsed to obtain the file identifier and byte range associated with the persistence flag. The pending write request that matches the file identifier and byte range is searched in the temporary write queue. It is determined whether the pending write request meets the preset persistence conditions. If it does, the pending write request is determined as the target write request and transferred from the temporary write queue to the persistent commit write queue.

[0005] Preferably, after identifying the write request to be determined as the target write request and transferring it from the temporary write queue to the persistent commit write queue, the method further includes: During scheduling and dispatching, the system first checks whether there are any pending write requests in the persistent commit write queue. If there are pending write requests in the persistent commit write queue, a dispatch queue entry is generated using the pending write requests in the persistent commit write queue, the tail pointer in the first doorbell register is updated, and then the entry is submitted to the hard disk. If there are no pending write requests in the persistent commit write queue, a pending write request is retrieved from the temporary write queue, and a write-back queue entry is generated using a delayed aggregation strategy and submitted to the hard disk. At the same time, read requests are selected from the hot queue, warm queue, and cold queue according to the initial dispatch ratio, merged into the target dispatch queue, and dispatched to the hard disk in sequence. Correspondingly, if there are pending write requests in the persistent commit write queue, a dispatch queue entry is generated using the pending write requests in the persistent commit write queue, and the tail pointer in the first doorbell register is updated before being committed to the hard disk, including: Retrieve pending write requests from the persistent commit write queue and convert them into dispatch queue entries containing reserved bits, write request data, and buffer pointers. The dispatch queue entry is written to the preset send queue in the host, and the controller in the hard disk is notified by updating the tail pointer of the persistent commit write queue stored in the first doorbell register. When the controller detects that the tail pointer in the first doorbell register has been updated, it retrieves the dispatch queue entry from the preset send queue and performs the corresponding write operation on the hard disk. The first doorbell register is located inside the controller.

[0006] Preferably, before constructing the multi-level cache queue, the method further includes: Initialize the access frequency counter and the dwell timestamp. The initial count value of the access frequency counter for read requests is determined based on the historical access record corresponding to the data identification information. The initial count value of the access frequency counter for write requests is set to zero. The initial value of the dwell timestamp is the start time of the current scheduling period. Configure corresponding initial dispatch ratios for hot queues, warm queues, cold queues, temporary write queues, and persistent commit write queues. The initial dispatch ratio of hot queues is greater than that of warm queues, which in turn is greater than that of cold queues. The dispatch priority of persistent commit write queues is higher than that of temporary write queues.

[0007] Preferably, the process involves receiving read / write requests, extracting data identification information from the read / write requests, assigning an access frequency counter and a residence timestamp to each read / write request, and allocating read requests to one of a hot queue, a warm queue, or a cold queue as the read target queue based on the initial count value of the access frequency counter and the initial value of the residence timestamp. Write requests are unconditionally stored in a temporary write queue. Receive read / write requests, extract data identification information from the read / write requests, and allocate access frequency counters and dwell timestamps to the read / write requests based on the data identification information; Based on the initial count value of the access frequency counter and the initial value of the dwell timestamp of the read request, the read request is assigned to one of the hot queue, warm queue or cold queue as the read target queue. The write request is unconditionally stored in the temporary write queue. Read requests with an initial count value greater than the preset heat threshold are assigned to the hot queue, read requests with an initial count value between the preset heat threshold and the preset temperature threshold are assigned to the warm queue, and read requests with an initial count value less than the preset temperature threshold are assigned to the cold queue.

[0008] Preferably, the access frequency counter is attenuated according to a preset attenuation coefficient to obtain the attenuated count value and update it in the access frequency counter, including: At the beginning of each scheduling cycle, identify target read / write requests in the multi-level cache queue that have not been accessed since the previous scheduling cycle, and obtain the current count value of the access frequency counter in the target read / write requests; Multiply the current count value by the preset attenuation coefficient to obtain the attenuated count value, and update the access frequency counter with the attenuated count value. The preset attenuation coefficient is determined according to the type of each queue. The attenuation coefficient corresponding to the hot queue is greater than that corresponding to the warm queue, and the attenuation coefficient corresponding to the warm queue is greater than that corresponding to the cold queue.

[0009] Preferably, based on the decayed count value and the dwell timestamp, an upgrade operation is performed on read requests in the cold queue, including: In the cold queue, obtain the dwell timestamp of the read request to be upgraded, and determine whether the dwell timestamp exceeds the preset time threshold; When the dwell timestamp exceeds the time threshold, obtain the count value of the access frequency counter in the read request to be upgraded after the attenuation process. When the count value exceeds the cold-to-hot threshold, the read request to be upgraded will be moved from the cold queue to the hot queue. When the count value exceeds the cold-to-warm threshold but does not reach the cold-to-heat threshold, the read request to be upgraded will be moved from the cold queue to the warm queue.

[0010] Preferably, based on the decayed count value and the dwell timestamp, a degradation operation is performed on read requests in the hot queue, including: In the hot queue, the number of times a target read request is not accessed within a consecutive scheduling period is counted; When the number of times the target read request has not been accessed reaches a preset value of K, the target read request is moved from the hot queue to the warm queue. In the warm queue, continue to count the number of times the target read request has not been accessed within a continuous scheduling period; When the number of times a request has not been accessed reaches a preset value M, the target read request is moved from the warm queue to the cold queue, where the value K is less than the value M.

[0011] Preferably, if there are no pending write requests in the persistent commit write queue, then pending write requests are retrieved from the temporary write queue and a delayed aggregation strategy is used to generate write-back queue entries and commit them to the disk, including: Take the pending write request from the temporary write queue and check if there is an adjacent write request in the temporary write queue that is adjacent to the address of the pending write request and meets the preset aggregation condition; If there are adjacent write requests, the pending write request and the adjacent write request are converted into aggregate write queue entries using the third preset data structure. The aggregate write queue entries include the pending write request and the adjacent write request, the buffer pointers of the buffers required by the pending write request and the adjacent write request, the multi-write request identifier, and the write execution order determined based on the position of the pending write request and the adjacent write request in the temporary write queue. Submit the aggregated write-back queue entries to the hard disk for execution. The data size of the aggregated write-back queue entries shall not exceed the preset data size.

[0012] Preferably, when the controller detects an update to the tail pointer in the first doorbell register, it retrieves a dispatch queue entry from the preset dispatch queue and performs a corresponding write operation on the hard disk, including: After the controller retrieves a dispatch queue entry from the preset send queue, if the dispatch queue entry includes a multi-write request identifier, it will execute several pending write requests in the dispatch queue entry on the hard disk in sequence based on the write execution order in the dispatch queue entry. Based on the write execution order in the dispatch queue entries, the current write request to be executed is determined from several pending write requests in the dispatch queue entries. Based on the buffer pointer of the buffer required by the current write request to be executed in the dispatch queue entries, the corresponding target buffer is determined, and the data stored in the target buffer is used to execute the current write request to be executed on the hard disk. The completion queue entries containing the execution results of the write requests to be executed are written to the preset completion queue in the host, where different write requests to be executed correspond to different completion queue entries; After the controller writes the completion queue entry into the preset completion queue, it sends an interrupt command to the host so that the host can process the completion queue entry taken from the head of the preset completion queue based on the interrupt command, and notify the controller by updating the head pointer of the preset completion queue stored in the second doorbell register after processing. The second doorbell register is located inside the controller. When an update to the head pointer in the second doorbell register is detected, a second notification message indicating that the queue entry has been processed is sent to the hard disk.

[0013] A disk read / write optimization scheduling system based on multi-level cache queues, applicable to the aforementioned disk read / write optimization scheduling method based on multi-level cache queues, includes: The queue creation unit is configured to build a multi-level cache queue, which includes a hot queue, a warm queue, a cold queue, a temporary write queue, and a persistent commit write queue. The corresponding initial dispatch ratio is configured for each of the hot queue, warm queue, cold queue, temporary write queue, and persistent commit write queue. The request allocation unit is configured to receive read and write requests, extract data identification information from the read and write requests, allocate an access frequency counter and a residence timestamp to each read and write request, and allocate the read request to one of the hot queue, warm queue or cold queue as the read target queue according to the initial count value of the access frequency counter and the initial value of the residence timestamp, and unconditionally store the write request in the temporary write queue. The attenuation processing unit is configured to attenuate the access frequency counter for target read / write requests in the multi-level cache queue that have not been accessed since the previous scheduling cycle in each scheduling cycle, according to a preset attenuation coefficient, and then update the attenuated count value to the access frequency counter. The scheduling unit is optimized and configured to perform upgrade operations on read requests in the cold queue or downgrade operations on read requests in the hot queue based on the decayed count value and the dwell timestamp. The address detection unit is configured to perform address overlap detection and address adjacency detection on the write requests stored in the temporary write queue. If an existing write request that overlaps with the address of the current write request to be processed is detected, a merged write request to be persisted is generated based on the current write request to be processed and the existing write request, and the existing write request is replaced. If a consecutive write request that is adjacent to the address of the current write request to be processed and meets the preset adjacency condition is detected, an aggregated write request to be persisted is generated based on the current write request to be processed and the consecutive write requests. The queue transfer unit is configured to, upon receiving a synchronization instruction containing an explicit persistence flag, parse the synchronization instruction to obtain the file identifier and byte range associated with the persistence flag, search for a write request to be determined that matches the file identifier and byte range in the temporary write queue, determine whether the write request to be determined meets the preset persistence conditions, and if it does, determine the write request to be determined as the target write request and transfer it from the temporary write queue to the persistent commit write queue.

[0014] Compared with the prior art, the beneficial effects of the present invention are: (1) This invention realizes adaptive scheduling of read requests based on the heat perception through hierarchical queues and frequency decay mechanism. It can maintain an access frequency counter and a residence timestamp for each read request, and allocate the request to a three-level queue (hot, warm, cold) based on the initial count value. At the same time, the counter of unaccessed requests is decayed differently according to the queue type in each scheduling cycle. Then, the cold queue is upgraded and the hot queue is downgraded based on the decayed count value. This dynamic heat evaluation mechanism effectively distinguishes between data blocks with stable access patterns and bursts, so that high-frequency access data resides in high-priority queues first, and low-frequency data sinks in time. It avoids the problem of cold data occupying high resources or hot data being eliminated too early under the traditional fixed caching strategy, and significantly improves the cache hit rate and read request response speed. (2) This invention greatly improves write efficiency and data consistency by merging and aggregating write requests and using an explicit persistence priority dispatch mechanism. The method performs address overlap detection and adjacency detection on the write requests already stored in the temporary write queue, merging or aggregating overlapping or adjacent write requests into larger granularity write entries, which significantly reduces the number of I / O operations and disk seek overhead. At the same time, when a synchronization instruction with an explicit persistence flag is received, the write requests that meet the conditions can be transferred from the temporary queue to the persistence commit queue, and the persistence commit queue is given higher priority than the temporary write queue in the scheduling and dispatch, ensuring that critical data is written to disk in a timely manner. Combined with the hardware dispatch process based on the doorbell register, the software overhead and response latency are further reduced, enabling the method to have both high performance and strong consistency in high-concurrency write scenarios. Attached Figure Description

[0015] Figure 1 This is a schematic flowchart of the overall method in one embodiment of the present invention; Figure 2 This is a schematic diagram of the overall system architecture in one embodiment of the present invention.

[0016] In the diagram: 1. Queue creation unit; 2. Request allocation unit; 3. Attenuation processing unit; 4. Optimized scheduling unit; 5. Address detection unit; 6. Queue transfer unit. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] Example 1, please refer to Figure 1 This invention provides a technical solution: a disk read / write optimization scheduling method based on a multi-level cache queue, comprising: S1. Construct a multi-level cache queue, which includes a hot queue, a warm queue, a cold queue, a temporary write queue, and a persistent commit write queue. Configure the corresponding initial dispatch ratio for each of the hot queue, warm queue, cold queue, temporary write queue, and persistent commit write queue. S2. Receive read / write requests, extract data identification information from read / write requests, allocate access frequency counters and dwell timestamps to each read / write request, and allocate read requests to one of the hot queue, warm queue, or cold queue as read target queues according to the initial count value of the access frequency counter and the initial value of the dwell timestamp, and unconditionally store write requests in the temporary write queue. S3. In each scheduling cycle, for target read and write requests in the multi-level cache queue that have not been accessed since the previous scheduling cycle, the access frequency counter is attenuated according to the preset attenuation coefficient, and the attenuated count value is obtained and updated to the access frequency counter. S4. Based on the decayed count value and the dwell timestamp, perform an upgrade operation on read requests in the cold queue or a downgrade operation on read requests in the hot queue. S5. In the temporary write queue, perform address overlap detection and address adjacency detection on the written requests that have been stored. If an existing write request that overlaps with the address of the current write request to be processed is detected, generate a merged write request to be persisted based on the current write request to be processed and the existing write request, and replace the existing write request. If a consecutive write request that is adjacent to the address of the current write request to be processed and meets the preset adjacency condition is detected, generate an aggregated write request to be persisted based on the current write request to be processed and the consecutive write requests. S6. When a synchronization instruction containing a persistent flag is received, the synchronization instruction is parsed to obtain the file identifier and byte range associated with the persistent flag. The pending write request that matches the file identifier and byte range is searched in the temporary write queue. It is determined whether the pending write request meets the preset persistence conditions. If it does, the pending write request is determined as the target write request and transferred from the temporary write queue to the persistent commit write queue.

[0019] It should be noted that a multi-level cache queue is constructed in the data storage device. This queue includes three read request cache areas: a hot queue, a warm queue, and a cold queue; and two write request processing areas: a temporary write queue and a persistent commit write queue. An initial scheduling weight is assigned to each queue. When each read or write request is received, the logical address identifier in the request is extracted, and an access frequency counter and a residence timestamp are assigned to the request. The initial count is set to a fixed unit value, and the timestamp records the current scheduling round. For read requests, they are assigned to the hot queue, warm queue, or cold queue based on the initial count and timestamp. For write requests, they are unconditionally stored in the temporary write queue. Within each scheduling cycle, it iterates through all queues for requests that have not been accessed since the previous cycle, multiplicatively decays the access frequency counter according to the preset decay factor, and then updates it by rounding up; then it compares the updated count value with the preset upgrade / downgrade threshold: if the count of read requests in the cold queue exceeds the upgrade threshold, the request is promoted to the warm queue; if the count of read requests in the hot queue is lower than the downgrade threshold, the request is downgraded to the cold queue. In the temporary write queue, address overlap detection is performed on all stored write requests. If the address range of the current request to be processed overlaps with that of an existing request, the two are merged into a write request that covers a larger address range and the original request is replaced. At the same time, address adjacency detection is performed. If there are consecutive requests with consecutive addresses and an interval less than a preset adjacency threshold, and the total length after merging does not exceed the preset aggregation limit, the adjacent requests are aggregated into a single consecutive block write request. When a synchronization instruction containing an explicit persistence flag is received, the instruction is parsed to obtain the associated data range identifier and byte offset interval. The temporary write queue is searched for write requests that match the range and have no overlapping merge conflicts. The data length of the request is verified to meet the preset minimum persistence unit and have no pending dependencies. If the conditions are met, the request is transferred from the temporary write queue to the persistent write commit queue, and the background thread executes the persistence write operation in sequence. The entire scheduling process is repeated periodically, and the dispatch ratio of each queue is dynamically adjusted according to the system load.

[0020] In an optional embodiment, after identifying the write request to be determined as the target write request and transferring it from the temporary write queue to the persistent commit write queue, the method further includes: During scheduling and dispatching, the system first checks whether there are any pending write requests in the persistent commit write queue. If there are pending write requests in the persistent commit write queue, a dispatch queue entry is generated using the pending write requests in the persistent commit write queue, the tail pointer in the first doorbell register is updated, and then the entry is submitted to the hard disk. If there are no pending write requests in the persistent commit write queue, a pending write request is retrieved from the temporary write queue, and a write-back queue entry is generated using a delayed aggregation strategy and submitted to the hard disk. At the same time, read requests are selected from the hot queue, warm queue, and cold queue according to the initial dispatch ratio, merged into the target dispatch queue, and dispatched to the hard disk in sequence. Correspondingly, if there are pending write requests in the persistent commit write queue, a dispatch queue entry is generated using the pending write requests in the persistent commit write queue, and the tail pointer in the first doorbell register is updated before being committed to the hard disk, including: Retrieve pending write requests from the persistent commit write queue and convert them into dispatch queue entries containing reserved bits, write request data, and buffer pointers. The dispatch queue entry is written to the preset send queue in the host, and the controller in the hard disk is notified by updating the tail pointer of the persistent commit write queue stored in the first doorbell register. When the controller detects that the tail pointer in the first doorbell register has been updated, it retrieves the dispatch queue entry from the preset send queue and performs the corresponding write operation on the hard disk. The first doorbell register is located inside the controller.

[0021] It should be noted that after transferring the target write request from the temporary write queue to the persistent write queue, the subsequent scheduling and dispatching phase begins. At the start of each scheduling and dispatching cycle, the scheduler first checks whether there are any write requests to be dispatched in the persistent write queue. If so, it retrieves a write request from the queue, encapsulates the request's data payload, the corresponding storage buffer address pointer, and the protocol reserved fields into a standard dispatch queue entry, and writes this entry into the preset sending queue in the host memory. Simultaneously, the scheduler updates the tail pointer in the first doorbell register, which points to the end of the latest write entry in the sending queue. The first doorbell register is located inside the hard disk controller. The monitoring logic in the controller continuously polls the value of this doorbell register, and once the tail pointer is detected... When an update occurs, the scheduler actively retrieves dispatch entries sequentially from the sending queue, parses the write request data and buffer addresses within them, and performs the corresponding actual write operation on the storage medium accordingly. If there are no write requests to be dispatched in the persistent commit write queue, the scheduler retrieves write requests to be written back from the temporary write queue, applies a delayed aggregation strategy based on address proximity to these requests, merges them into a smaller number of write-back queue entries, and submits them to the hard disk. At the same time, the scheduler selects read requests from the hot queue, warm queue, and cold queue according to the pre-configured initial dispatch ratio, merges the selected read requests into a target dispatch queue, and dispatches them to the hard disk in sequence to complete the read task. The entire dispatch process is executed cyclically in each scheduling cycle to ensure that the persistence priority of write operations and the fairness of read operations are taken into account.

[0022] In an alternative embodiment, the method further includes, prior to constructing the multi-level cache queue: Initialize the access frequency counter and the dwell timestamp. The initial count value of the access frequency counter for read requests is determined based on the historical access record corresponding to the data identification information. The initial count value of the access frequency counter for write requests is set to zero. The initial value of the dwell timestamp is the start time of the current scheduling period. Configure corresponding initial dispatch ratios for hot queues, warm queues, cold queues, temporary write queues, and persistent commit write queues. The initial dispatch ratio of hot queues is greater than that of warm queues, which in turn is greater than that of cold queues. The dispatch priority of persistent commit write queues is higher than that of temporary write queues.

[0023] It should be noted that during system startup or reset, initialization is performed first. For the access frequency counter, processing is differentiated based on the request type: if the request is a read request, the cumulative number of accesses of the data identifier information carried by the request in the historical access log is queried, and this number is used as the initial count value of the access frequency counter for that read request; if the request is a write request, the initial count value of its access frequency counter is uniformly set to zero; for the dwell timestamp, the start time of the current scheduling cycle is uniformly used as the initial value of the dwell timestamp for all requests, and this start time is represented by the sequence number of the system global clock cycle. Simultaneously, an initial dispatch ratio is configured for each cache queue; hot queues are assigned the highest dispatch weight, warm queues the medium weight, and cold queues the lowest weight, with the weight decreasing sequentially; temporary write queues are configured with a separate, moderate dispatch ratio, while persistent commit write queues have a higher dispatch priority than temporary write queues, meaning that when both have pending requests, the system prioritizes dispatching entries from the persistent commit write queue; all initial ratio values ​​are stored in the configuration parameter table as relative weight coefficients and can be dynamically adjusted according to the actual load during system operation, but a preset default weight allocation is used during the initialization phase; the above initialization work is completed before receiving any read or write requests to ensure that subsequent queue operations have a consistent counting benchmark and dispatch priority basis.

[0024] In an optional embodiment, the process includes receiving read / write requests, extracting data identification information from the read / write requests, assigning an access frequency counter and a residence timestamp to each read / write request, and allocating read requests to one of a hot queue, a warm queue, or a cold queue as a read target queue based on the initial count value of the access frequency counter and the initial value of the residence timestamp. Write requests are then unconditionally stored in a temporary write queue. Receive read / write requests, extract data identification information from the read / write requests, and allocate access frequency counters and dwell timestamps to the read / write requests based on the data identification information; Based on the initial count value of the access frequency counter and the initial value of the dwell timestamp of the read request, the read request is assigned to one of the hot queue, warm queue or cold queue as the read target queue. The write request is unconditionally stored in the temporary write queue. Read requests with an initial count value greater than the preset heat threshold are assigned to the hot queue, read requests with an initial count value between the preset heat threshold and the preset temperature threshold are assigned to the warm queue, and read requests with an initial count value less than the preset temperature threshold are assigned to the cold queue.

[0025] It should be noted that when receiving each read / write request, the logical address is extracted from the request as a data identifier, and a corresponding access frequency counter and dwell timestamp are created for the request. The initial value of the counter is set according to historical records or default rules, and the timestamp is set to the start of the current period. For read requests, the initial count value of the counter is compared with a pre-set heat threshold and a temperature threshold, where the heat threshold is higher than the temperature threshold. If the count value is greater than the heat threshold, the read request is sent to the hot queue; if the count value is between the heat threshold and the temperature threshold, it is sent to the warm queue; if the count value is less than the temperature threshold, it is sent to the cold queue. For write requests, the above comparison is not performed, and they are directly stored in a temporary write queue. All requests are processed according to this process to ensure that read requests are cached hierarchically according to historical heat, and write requests are centrally stored for subsequent merging and processing.

[0026] In an optional embodiment, the access frequency counter is attenuated according to a preset attenuation coefficient to obtain an attenuated count value and update the access frequency counter, including: At the beginning of each scheduling cycle, identify target read / write requests in the multi-level cache queue that have not been accessed since the previous scheduling cycle, and obtain the current count value of the access frequency counter in the target read / write requests; Multiply the current count value by the preset attenuation coefficient to obtain the attenuated count value, and update the access frequency counter with the attenuated count value. The preset attenuation coefficient is determined according to the type of each queue. The attenuation coefficient corresponding to the hot queue is greater than that corresponding to the warm queue, and the attenuation coefficient corresponding to the warm queue is greater than that corresponding to the cold queue.

[0027] It should be noted that at the start of each scheduling cycle, the system traverses all cache queues and identifies read / write requests that have not been accessed since the end of the previous cycle. For each such request, the current value of its access frequency counter is read, multiplied by a preset attenuation coefficient, and the product is used as the new attenuated count value, which is then written back to the counter to overwrite the original value. The specific value of this attenuation coefficient varies depending on the queue type of the request: hot queues use the largest attenuation coefficient, warm queues use a medium attenuation coefficient, and cold queues use the smallest attenuation coefficient, with the values ​​decreasing sequentially to reflect the different tolerances of different popularity levels for the rate of access frequency obsolescence. After the attenuation operation is completed, the new count value is used for subsequent upgrade / downgrade determination and scheduling decisions. This process is executed independently and repeatedly for all unaccessed requests within each scheduling cycle.

[0028] In an optional embodiment, an upgrade operation is performed on read requests in the cold queue based on the decayed count value and the dwell timestamp, including: In the cold queue, obtain the dwell timestamp of the read request to be upgraded, and determine whether the dwell timestamp exceeds the preset time threshold; When the dwell timestamp exceeds the time threshold, obtain the count value of the access frequency counter in the read request to be upgraded after the attenuation process. When the count value exceeds the cold-to-hot threshold, the read request to be upgraded will be moved from the cold queue to the hot queue. When the count value exceeds the cold-to-warm threshold but does not reach the cold-to-heat threshold, the read request to be upgraded will be moved from the cold queue to the warm queue.

[0029] It should be noted that during the upgrade check phase of each scheduling cycle, the system iterates through all read requests in the cold queue. For each read request to be upgraded, its dwell timestamp is first read and compared with a preset time threshold. If the dwell timestamp does not exceed the threshold, the upgrade check for that request is skipped. If it exceeds the threshold, the current count value of the access frequency counter after attenuation processing is read. This count value is compared with two preset thresholds, where the cold-to-hot threshold is higher than the cold-to-warm threshold. If the count value is greater than the cold-to-hot threshold, the read request is directly migrated from the cold queue to the hot queue. If the count value is between the cold-to-warm threshold and the cold-to-hot threshold, the read request is migrated from the cold queue to the warm queue. If the count value is lower than the cold-to-warm threshold, no migration operation is performed. All upgrade operations are completed in batches within the same scheduling cycle, and the dwell timestamp of the corresponding request is updated to the current cycle time after migration.

[0030] In an optional embodiment, a degradation operation is performed on read requests in the hot queue based on the decayed count value and the dwell timestamp, including: In the hot queue, the number of times a target read request is not accessed within a consecutive scheduling period is counted; When the number of times the target read request has not been accessed reaches a preset value of K, the target read request is moved from the hot queue to the warm queue. In the warm queue, continue to count the number of times the target read request has not been accessed within a continuous scheduling period; When the number of times a request has not been accessed reaches a preset value M, the target read request is moved from the warm queue to the cold queue, where the value K is less than the value M.

[0031] It should be noted that, in each scheduling cycle, the system maintains a count of consecutive unaccessed cycles for each read request in the hot queue. When the request is not hit by any read operation in the current cycle, the count is incremented by one unit; if it is hit, the count is reset to zero. When the consecutive unaccessed count reaches a preset first degradation threshold, the read request is moved from the hot queue to the warm queue, and the count is reset to its initial state. After the request enters the warm queue, the system maintains another independent consecutive unaccessed count for it, which is also incremented in each unaccessed cycle and reset to zero when it is hit. When the count reaches a preset second degradation threshold, the read request is moved from the warm queue to the cold queue, where the second degradation threshold is greater than the first degradation threshold. All degradation operations are executed in batches at the end of the cycle, and after migration, the request's residence timestamp is updated to the current cycle time.

[0032] In an optional embodiment, if there are no pending write requests in the persistent commit write queue, the pending write request is retrieved from the temporary write queue and a write-back queue entry is generated using a delayed aggregation strategy and committed to the disk, including: Take the pending write request from the temporary write queue and check if there is an adjacent write request in the temporary write queue that is adjacent to the address of the pending write request and meets the preset aggregation condition; If there are adjacent write requests, the pending write request and the adjacent write request are converted into aggregate write queue entries using the third preset data structure. The aggregate write queue entries include the pending write request and the adjacent write request, the buffer pointers of the buffers required by the pending write request and the adjacent write request, the multi-write request identifier, and the write execution order determined based on the position of the pending write request and the adjacent write request in the temporary write queue. Submit the aggregated write-back queue entries to the hard disk for execution. The data size of the aggregated write-back queue entries shall not exceed the preset data size.

[0033] It should be noted that when the persistent write commit queue is empty, the scheduler retrieves a write request to be written back from the temporary write queue as the base request. Then, it iterates through the remaining unprocessed write requests in the temporary write queue, checking for adjacent write requests whose address range is continuous with the base request or whose interval is less than a preset adjacent threshold. Simultaneously, it verifies that the total data length of all candidate requests does not exceed a preset aggregation limit. If adjacent write requests meet the criteria, a third preset data structure is used to encapsulate the base request and all adjacent write requests into an aggregated write-back queue entry. This entry contains the data payload of all original write requests and their respective buffer memory address pointers, an appended multi-write request identifier to indicate that this is an aggregation operation, and the execution sequence number arranged according to the order of the original requests in the temporary write queue. After encapsulation, this aggregated entry is directly submitted to the execution queue on the hard disk. If no adjacent requests are detected, the base request is encapsulated as a single entry and submitted. The total data size of all submitted entries is controlled within the preset aggregation size limit; if the limit is exceeded, the entry is split or truncated.

[0034] In an optional embodiment, when the controller detects an update to the tail pointer in the first doorbell register, it retrieves a dispatch queue entry from a preset send queue and performs a corresponding write operation on the hard disk, including: After the controller retrieves a dispatch queue entry from the preset send queue, if the dispatch queue entry includes a multi-write request identifier, it will execute several pending write requests in the dispatch queue entry on the hard disk in sequence based on the write execution order in the dispatch queue entry. Based on the write execution order in the dispatch queue entries, the current write request to be executed is determined from several pending write requests in the dispatch queue entries. Based on the buffer pointer of the buffer required by the current write request to be executed in the dispatch queue entries, the corresponding target buffer is determined, and the data stored in the target buffer is used to execute the current write request to be executed on the hard disk. The completion queue entries containing the execution results of the write requests to be executed are written to the preset completion queue in the host, where different write requests to be executed correspond to different completion queue entries; After the controller writes the completion queue entry into the preset completion queue, it sends an interrupt command to the host so that the host can process the completion queue entry taken from the head of the preset completion queue based on the interrupt command, and notify the controller by updating the head pointer of the preset completion queue stored in the second doorbell register after processing. The second doorbell register is located inside the controller. When an update to the head pointer in the second doorbell register is detected, a second notification message indicating that the queue entry has been processed is sent to the hard disk.

[0035] It should be noted that after the controller retrieves a dispatch queue entry from the preset send queue, it first checks whether the entry contains a multi-write request identifier. If it does, it parses the write execution order in the entry and processes the multiple write requests encapsulated within it in that order. For the current write request to be executed, the controller extracts its corresponding buffer pointer from the entry, locates the target buffer in the host memory based on the pointer, reads the data from it, and performs the actual write operation on the storage medium. After each write request to be executed is completed, the controller generates a corresponding completion queue entry, which contains the execution result status of the write request, and writes the entry to the preset completion queue in the host memory. After all write requests to be executed are completed, the controller... An interrupt command is sent to the host to notify it that there are new completion entries to be processed. After receiving the interrupt, the host sequentially retrieves completion queue entries from the head of the preset completion queue for subsequent processing. After processing each entry, the host updates the completion queue head pointer stored in the second doorbell register to report the processing progress to the controller. The second doorbell register is located inside the controller. The controller continuously monitors the head pointer value of the second doorbell register. Once it detects an update to the pointer, it confirms that the corresponding completion queue entry has been processed by the host and then sends a second notification message to the hard disk storage medium to release the relevant resources and update the internal state. The entire process ensures that the execution, confirmation, and resource reclamation of each write operation are completed in a sequential and traceable manner.

[0036] Example 2, please refer to Figure 2 This invention provides a technical solution: a hard disk read / write optimization scheduling system based on a multi-level cache queue, which is applicable to the aforementioned hard disk read / write optimization scheduling method based on a multi-level cache queue, comprising: Queue creation unit 1 is configured to build a multi-level cache queue, which includes a hot queue, a warm queue, a cold queue, a temporary write queue, and a persistent commit write queue. The corresponding initial dispatch ratio is configured for each of the hot queue, warm queue, cold queue, temporary write queue, and persistent commit write queue. The request allocation unit 2 is configured to receive read and write requests, extract data identification information from the read and write requests, allocate an access frequency counter and a residence timestamp to each read and write request, and allocate the read request to one of the hot queue, warm queue or cold queue as the read target queue according to the initial count value of the access frequency counter and the initial value of the residence timestamp, and unconditionally store the write request in the temporary write queue. The attenuation processing unit 3 is configured to attenuate the access frequency counter for target read / write requests in the multi-level cache queue that have not been accessed since the previous scheduling cycle in each scheduling cycle, according to a preset attenuation coefficient, and obtain the attenuated count value and update it to the access frequency counter. Optimize scheduling unit 4 and configure it to perform upgrade operations on read requests in the cold queue or downgrade operations on read requests in the hot queue based on the decayed count value and the dwell timestamp; Address detection unit 5 is configured to perform address overlap detection and address adjacency detection on the written requests stored in the temporary write queue. If an existing write request that overlaps with the address of the current write request to be processed is detected, a merged write request to be persisted is generated based on the current write request to be processed and the existing write request, and the existing write request is replaced. If a consecutive write request that is adjacent to the address of the current write request to be processed and meets the preset adjacency condition is detected, an aggregated write request to be persisted is generated based on the current write request to be processed and the consecutive write requests. The queue transfer unit 6 is configured to, upon receiving a synchronization instruction containing an explicit persistence flag, parse the synchronization instruction to obtain the file identifier and byte range associated with the persistence flag, search for a write request to be determined that matches the file identifier and byte range in the temporary write queue, determine whether the write request to be determined meets the preset persistence conditions, and if it does, determine the write request to be determined as the target write request and transfer it from the temporary write queue to the persistent commit write queue.

[0037] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited thereto. Various changes can be made within the scope of knowledge possessed by those skilled in the art without departing from the spirit of the present invention.

Claims

1. A disk read / write optimization scheduling method based on a multi-level cache queue, characterized in that, include: Construct a multi-level cache queue, which includes a hot queue, a warm queue, a cold queue, a temporary write queue, and a persistent commit write queue, and configure the corresponding initial dispatch ratio for each of the hot queue, warm queue, cold queue, temporary write queue, and persistent commit write queue; Receive read and write requests, extract data identification information from the read and write requests, assign an access frequency counter and a residence timestamp to each read and write request, and assign the read request to one of the hot queue, warm queue or cold queue as the read target queue according to the initial count value of the access frequency counter and the initial value of the residence timestamp, and unconditionally store the write request in the temporary write queue. In each scheduling cycle, for target read and write requests in the multi-level cache queue that have not been accessed since the previous scheduling cycle, the access frequency counter is decayed according to a preset decay coefficient, and the decayed count value is obtained and updated to the access frequency counter. Based on the decayed count value and the dwell timestamp, perform an upgrade operation on read requests in the cold queue or a downgrade operation on read requests in the hot queue; In the temporary write queue, address overlap detection and address adjacency detection are performed on the written requests that have been stored. If an existing write request that overlaps with the address of the current write request to be processed is detected, a merged write request to be persisted is generated based on the current write request to be processed and the existing write request, and the existing write request is replaced. If a consecutive write request that is adjacent to the address of the current write request to be processed and meets the preset adjacency condition is detected, an aggregated write request to be persisted is generated based on the current write request to be processed and the consecutive write requests. When a synchronization instruction containing an explicit persistence flag is received, the synchronization instruction is parsed to obtain the file identifier and byte range associated with the persistence flag. The pending write request that matches the file identifier and byte range is searched in the temporary write queue. It is determined whether the pending write request meets the preset persistence conditions. If it does, the pending write request is determined as the target write request and transferred from the temporary write queue to the persistent commit write queue.

2. The disk read / write optimization scheduling method based on a multi-level cache queue according to claim 1, characterized in that, After identifying the pending write request as the target write request and transferring it from the temporary write queue to the persistent commit write queue, the method further includes: During scheduling and dispatching, the system first checks whether there are any pending write requests in the persistent commit write queue. If there are pending write requests in the persistent commit write queue, a dispatch queue entry is generated using the pending write requests in the persistent commit write queue, the tail pointer in the first doorbell register is updated, and then the entry is submitted to the hard disk. If there are no pending write requests in the persistent commit write queue, a pending write request is retrieved from the temporary write queue, and a write-back queue entry is generated using a delayed aggregation strategy and submitted to the hard disk. At the same time, read requests are selected from the hot queue, warm queue, and cold queue according to the initial dispatch ratio, merged into the target dispatch queue, and dispatched to the hard disk in sequence. Correspondingly, if there are pending write requests in the persistent commit write queue, a dispatch queue entry is generated using the pending write requests in the persistent commit write queue, and the tail pointer in the first doorbell register is updated before being committed to the hard disk, including: Retrieve pending write requests from the persistent commit write queue and convert them into dispatch queue entries containing reserved bits, write request data, and buffer pointers. The dispatch queue entry is written to the preset send queue in the host, and the controller in the hard disk is notified by updating the tail pointer of the persistent commit write queue stored in the first doorbell register. When the controller detects that the tail pointer in the first doorbell register has been updated, it retrieves the dispatch queue entry from the preset send queue and performs the corresponding write operation on the hard disk. The first doorbell register is located inside the controller.

3. The disk read / write optimization scheduling method based on a multi-level cache queue according to claim 2, characterized in that, Before building a multi-level cache queue, the method also includes: Initialize the access frequency counter and the dwell timestamp. The initial count value of the access frequency counter for read requests is determined based on the historical access record corresponding to the data identification information. The initial count value of the access frequency counter for write requests is set to zero. The initial value of the dwell timestamp is the start time of the current scheduling period. Configure corresponding initial dispatch ratios for hot queues, warm queues, cold queues, temporary write queues, and persistent commit write queues. The initial dispatch ratio of hot queues is greater than that of warm queues, which in turn is greater than that of cold queues. The dispatch priority of persistent commit write queues is higher than that of temporary write queues.

4. The disk read / write optimization scheduling method based on a multi-level cache queue according to claim 3, characterized in that, The system receives read / write requests, extracts data identification information from the requests, assigns an access frequency counter and a dwell timestamp to each request, and allocates read requests to one of the hot, warm, or cold queues as read target queues based on the initial count value of the access frequency counter and the initial value of the dwell timestamp. Write requests are unconditionally stored in a temporary write queue, including: Receive read / write requests, extract data identification information from the read / write requests, and allocate access frequency counters and dwell timestamps to the read / write requests based on the data identification information; Based on the initial count value of the access frequency counter and the initial value of the dwell timestamp of the read request, the read request is assigned to one of the hot queue, warm queue or cold queue as the read target queue. The write request is unconditionally stored in the temporary write queue. Read requests with an initial count value greater than the preset heat threshold are assigned to the hot queue, read requests with an initial count value between the preset heat threshold and the preset temperature threshold are assigned to the warm queue, and read requests with an initial count value less than the preset temperature threshold are assigned to the cold queue.

5. The disk read / write optimization scheduling method based on a multi-level cache queue according to claim 4, characterized in that, The access frequency counter is attenuated according to a preset attenuation coefficient to obtain the attenuated count value, which is then updated in the access frequency counter. This includes: At the beginning of each scheduling cycle, identify target read / write requests in the multi-level cache queue that have not been accessed since the previous scheduling cycle, and obtain the current count value of the access frequency counter in the target read / write requests; Multiply the current count value by the preset attenuation coefficient to obtain the attenuated count value, and update the access frequency counter with the attenuated count value. The preset attenuation coefficient is determined according to the type of each queue. The attenuation coefficient corresponding to the hot queue is greater than that corresponding to the warm queue, and the attenuation coefficient corresponding to the warm queue is greater than that corresponding to the cold queue.

6. The disk read / write optimization scheduling method based on a multi-level cache queue according to claim 5, characterized in that, Based on the decayed count value and the dwell timestamp, perform an upgrade operation on read requests in the cold queue, including: In the cold queue, obtain the dwell timestamp of the read request to be upgraded, and determine whether the dwell timestamp exceeds the preset time threshold; When the dwell timestamp exceeds the time threshold, obtain the count value of the access frequency counter in the read request to be upgraded after the attenuation process. When the count value exceeds the cold-to-hot threshold, the read request to be upgraded will be moved from the cold queue to the hot queue. When the count value exceeds the cold-to-warm threshold but does not reach the cold-to-heat threshold, the read request to be upgraded will be moved from the cold queue to the warm queue.

7. The disk read / write optimization scheduling method based on a multi-level cache queue according to claim 6, characterized in that, Based on the decayed count value and the dwell timestamp, a degradation operation is performed on read requests in the hot queue, including: In the hot queue, the number of times a target read request is not accessed within a consecutive scheduling period is counted; When the number of times the target read request has not been accessed reaches a preset value of K, the target read request is moved from the hot queue to the warm queue. In the warm queue, continue to count the number of times the target read request has not been accessed within a continuous scheduling period; When the number of times a request has not been accessed reaches a preset value M, the target read request is moved from the warm queue to the cold queue, where the value K is less than the value M.

8. The disk read / write optimization scheduling method based on a multi-level cache queue according to claim 7, characterized in that, If there are no pending write requests in the persistent commit write queue, then pending write requests are retrieved from the temporary write queue and a delayed aggregation strategy is used to generate write-back queue entries, which are then committed to the disk, including: Take the pending write request from the temporary write queue and check if there is an adjacent write request in the temporary write queue that is adjacent to the address of the pending write request and meets the preset aggregation condition; If there are adjacent write requests, the pending write request and the adjacent write request are converted into aggregate write queue entries using the third preset data structure. The aggregate write queue entries include the pending write request and the adjacent write request, the buffer pointers of the buffers required by the pending write request and the adjacent write request, the multi-write request identifier, and the write execution order determined based on the position of the pending write request and the adjacent write request in the temporary write queue. Submit the aggregated write-back queue entries to the hard disk for execution. The data size of the aggregated write-back queue entries shall not exceed the preset data size.

9. The disk read / write optimization scheduling method based on a multi-level cache queue according to claim 8, characterized in that, When the controller detects an update to the tail pointer in the first doorbell register, it retrieves a dispatch queue entry from the preset send queue and performs the corresponding write operation on the hard disk, including: After the controller retrieves a dispatch queue entry from the preset send queue, if the dispatch queue entry includes a multi-write request identifier, it will execute several pending write requests in the dispatch queue entry on the hard disk in sequence based on the write execution order in the dispatch queue entry. Based on the write execution order in the dispatch queue entries, the current write request to be executed is determined from several pending write requests in the dispatch queue entries. Based on the buffer pointer of the buffer required by the current write request to be executed in the dispatch queue entries, the corresponding target buffer is determined, and the data stored in the target buffer is used to execute the current write request to be executed on the hard disk. The completion queue entries containing the execution results of the write requests to be executed are written to the preset completion queue in the host, where different write requests to be executed correspond to different completion queue entries; After the controller writes the completion queue entry into the preset completion queue, it sends an interrupt command to the host so that the host can process the completion queue entry taken from the head of the preset completion queue based on the interrupt command, and notify the controller by updating the head pointer of the preset completion queue stored in the second doorbell register after processing. The second doorbell register is located inside the controller. When an update to the head pointer in the second doorbell register is detected, a second notification message indicating that the queue entry has been processed is sent to the hard disk.

10. A hard disk read / write optimization scheduling system based on a multi-level cache queue, applicable to the hard disk read / write optimization scheduling method based on a multi-level cache queue as described in any one of claims 1-9, characterized in that, include: The queue creation unit is configured to build a multi-level cache queue, which includes a hot queue, a warm queue, a cold queue, a temporary write queue, and a persistent commit write queue. The corresponding initial dispatch ratio is configured for each of the hot queue, warm queue, cold queue, temporary write queue, and persistent commit write queue. The request allocation unit is configured to receive read and write requests, extract data identification information from the read and write requests, allocate an access frequency counter and a residence timestamp to each read and write request, and allocate the read request to one of the hot queue, warm queue or cold queue as the read target queue according to the initial count value of the access frequency counter and the initial value of the residence timestamp, and unconditionally store the write request in the temporary write queue. The attenuation processing unit is configured to attenuate the access frequency counter for target read / write requests in the multi-level cache queue that have not been accessed since the previous scheduling cycle in each scheduling cycle, according to a preset attenuation coefficient, and then update the attenuated count value to the access frequency counter. The scheduling unit is optimized and configured to perform upgrade operations on read requests in the cold queue or downgrade operations on read requests in the hot queue based on the decayed count value and the dwell timestamp. The address detection unit is configured to perform address overlap detection and address adjacency detection on the write requests stored in the temporary write queue. If an existing write request that overlaps with the address of the current write request to be processed is detected, a merged write request to be persisted is generated based on the current write request to be processed and the existing write request, and the existing write request is replaced. If a consecutive write request that is adjacent to the address of the current write request to be processed and meets the preset adjacency condition is detected, an aggregated write request to be persisted is generated based on the current write request to be processed and the consecutive write requests. The queue transfer unit is configured to, upon receiving a synchronization instruction containing an explicit persistence flag, parse the synchronization instruction to obtain the file identifier and byte range associated with the persistence flag, search for a write request to be determined that matches the file identifier and byte range in the temporary write queue, determine whether the write request to be determined meets the preset persistence conditions, and if it does, determine the write request to be determined as the target write request and transfer it from the temporary write queue to the persistent commit write queue.

Images