Data flashing method, storage device, storage medium and computer program product

Abstract

The invention discloses a data flashing method, a storage device, a storage medium and a computer program product, and relates to the technical field of computers, the data flashing method comprises the steps that a cache pool is divided into at least one cache unit, and the space size of the cache unit is determined based on the stripe width and the stripe block size of a disk array; compressed data blocks obtained by compressing the original data blocks are firstly written into the cache unit to be aggregated, and when the data size of the compressed data blocks stored in the first cache unit meets the requirement of a full stripe, the compressed data blocks in the first cache unit are flashed to the disk array, so that the data flashed to the disk array are full-stripe, and the data flashed to the disk array is full-stripe. The method does not need to wait for data accumulation through lock operation, so that lock competition is avoided, storage space waste caused by filling a large amount of redundant data is avoided, the technical problems of lock competition caused by data accumulation waiting for multiple lock operations and storage space waste caused by redundant data adding can be solved, and the technical effects of avoiding lock competition and saving storage space are achieved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data flashing method, a storage device, a storage medium, and a computer program product. Background Art

[0002] As the demand for storage system performance and reliability continues to increase, Redundant Array of Independent Disks (RAID) technology has become a core solution for improving storage performance and data protection through striping, mirroring, and parity mechanisms.

[0003] To improve storage efficiency, current storage systems often introduce data compression technology. However, the size of compressed data blocks changes dynamically, making it difficult to align with the RAID stripe width. In this case, multiple lock operations are usually required to wait for data accumulation or fill in redundant data to complete alignment. The former causes lock contention and reduces concurrency, while the latter wastes storage space, with filler data accounting for up to 50%. Summary of the Invention

[0004] The present application provides a data flushing method, storage device, storage medium and computer program product to at least solve the problem in the related art of waiting for data accumulation through multiple lock operations causing lock contention and adding redundant data to waste storage space.

[0005] The present application provides a data flashing method, comprising: compressing an original data block to obtain a compressed data block; Writing the compressed data block into a first cache unit in a cache pool, wherein the cache pool includes at least one cache unit, and a space size of the cache unit is determined based on a stripe width and a stripe block size of a disk array; Detecting whether the data volume of the compressed data block stored in the first cache unit meets the full stripe requirement; When the full stripe requirement is met, the compressed data blocks in the first cache unit are flushed to the disk array.

[0006] The present application also provides a storage device, comprising: a memory for storing a computer program; and a processor for implementing the steps of any of the above-mentioned data flashing methods when executing the computer program.

[0007] The present application also provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned data flashing methods are implemented.

[0008] The present application also provides a computer program product, including a computer program, which implements the steps of any of the above-mentioned data flashing methods when executed by a processor.

[0009] Through the present application, since the cache pool is divided into at least one cache unit, the space size of the cache unit is determined based on the stripe width and stripe block size of the disk array, the compressed data blocks obtained by compressing the original data blocks are first written into the cache unit for aggregation, and when the data volume of the compressed data blocks stored in the first cache unit meets the full stripe requirement, the compressed data blocks in the first cache unit are flushed to the disk array, so that the data flushed to the disk array is full stripe, and there is no need to wait for data accumulation through lock operations, thereby avoiding lock contention. Moreover, when the data volume of the compressed data blocks stored in the first cache unit meets the full stripe requirement, they are flushed to the disk array, so that the disk array does not need to fill a large amount of redundant data for stripe alignment, thereby avoiding the waste of storage space caused by a large amount of redundant data. Therefore, the technical problems of multiple lock operations waiting for data accumulation causing lock contention and adding redundant data to waste storage space can be solved, thereby achieving the technical effect of avoiding lock contention and saving storage space. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0011] Figure 1 A schematic diagram of an application environment architecture of a data flushing method provided by an exemplary embodiment of the present application is shown; Figure 2 A flowchart of a data flashing method provided by an exemplary embodiment of the present application; Figure 3 A flowchart of a data flashing method provided by another exemplary embodiment of the present application; Figure 4 A schematic diagram of a stripe alignment-based RAID write architecture according to an exemplary embodiment of the present application is shown; Figure 5 A schematic structural diagram of a data flashing device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0012] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0013] It should be noted that, in the description of this application, the terms "comprises," "includes," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. The terms "first," "second," etc., in this application are used to distinguish similar objects, and are not used to describe a particular order or sequence.

[0014] Traditional RAID faces the following key issues in write scenarios: (1) Performance bottleneck of the stripe lock mechanism When a RAID group handles concurrent I / O, it requires exclusive control of the same logical stripe through stripe locks to prevent read / write conflicts and ensure data consistency. For example, in parity-based arrays like RAID 5 / 6, the data blocks and parity blocks of the same stripe must be updated synchronously. If multiple I / Os write to the same stripe simultaneously, lock contention will be triggered, resulting in increased latency and decreased throughput. Especially in multi-controller storage systems, cross-controller lock management further increases communication overhead, affecting system scalability.

[0015] (2) The contradiction between data compression and stripe alignment To improve storage efficiency, modern storage systems often incorporate data compression technology. However, the size of compressed data blocks changes dynamically, making it difficult to directly align them with the RAID stripe width. For example, compression may reduce a 4KB data block to 2KB, resulting in a single I / O operation with insufficient data to fill the stripe. This necessitates multiple lock operations to wait for data accumulation, or padding with redundant data to complete alignment. The former creates lock contention and reduces concurrency, while the latter wastes storage space (e.g., padding data can account for up to 50%).

[0016] (3) Limitations of relevant optimization schemes To alleviate the performance bottlenecks of the aforementioned stripe lock mechanism, related technologies have proposed dynamic stripe adjustment and full-stripe write solutions. Dynamic stripe adjustment dynamically adjusts stripes based on data block size, but frequent adjustments lead to metadata management complexity and verification computation overhead. Full-stripe writes force a wait for a full stripe of data (such as RAID5's full-stripe write strategy) before writing to avoid lock contention. However, in compression scenarios, a large amount of uncompressed data must be cached, resulting in high memory usage and limited compression efficiency. Related technologies also propose a dynamic lock granularity adjustment solution, which divides storage space into fixed blocks (such as 64KB) and allocates lock resources. However, the block size is decoupled from the RAID stripe width, resulting in complex lock management and insufficient flexibility.

[0017] (4) Challenges of reliability assurance In a solution that combines compression and caching, if a partial stripe of data fails before aggregation is complete, data loss may occur. While mirroring technologies can provide redundancy protection, directly mirroring fragmented data can significantly increase IO amplification, impacting overall performance.

[0018] In summary, current RAID technologies are difficult to balance the contradictions among compression efficiency, stripe lock overhead, storage space utilization, and data reliability.

[0019] In response to the above problems, this application provides a new data flushing solution, which is based on data compression and cache aggregation to dynamically aggregate fragmented IO data, avoid lock competition, and ensure data security through redundant design. In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0020] In conjunction with the specific application environment architecture or specific hardware architecture on which the execution of the data flushing method depends, the specific application environment architecture or specific hardware architecture is described here.

[0021] The data flushing method provided in this application can be applied to a dual-control storage device or a multi-control storage device. Taking the application to a dual-control storage device as an example, Figure 1 A schematic diagram of the application environment architecture of the data flash writing method provided by an exemplary embodiment of the present application is shown as follows: Figure 1As shown, the first and second controllers have identical structures, each including a compression module and a cache aggregation module. The cache aggregation module is pre-divided into at least one cache unit. The size of each cache unit is determined based on the disk array's stripe width and stripe block size, for example, 2MB. The cache aggregation modules of the first and second controllers mirror each other. Assuming the first controller is the primary controller and the second controller is the backup controller, user-written data is first compressed by the first controller's compression module before being written to its cache aggregation module for aggregation. When the aggregated data reaches the cache unit size or a trigger threshold (for example, 90% of a full stripe), a flush mechanism is triggered, flushing the data in the cache units that meet the flush criteria to the RAID, achieving lock-free RAID. If the first controller fails, the second controller takes over. Since the cache aggregation module of the second controller caches a mirror image of the data in the first controller's cache aggregation module, a copy of the actual data is also stored in the second controller, ensuring data loss. The second controller can then write the data in its cache aggregation module to the RAID, eliminating the need for full stripes and ensuring data reliability and security.

[0022] The embodiments of the present application provide a data flushing method. This method can be performed by a data flushing device provided in the embodiments of the present application. The device can be implemented using software and / or hardware and can be integrated into a storage device, which can be a dual-controller storage device or a multi-controller storage device. The method is described in detail below in conjunction with the execution process of the data flushing method.

[0023] Figure 2 A flow chart of a data flash writing method provided by an exemplary embodiment of the present application is shown as follows: Figure 2 As shown, the data flashing method may include the following steps: Step 101: compress the original data block to obtain a compressed data block.

[0024] The original data block is data written by the user, and the compressed data block is data generated based on the data after compressing the original data block.

[0025] Exemplarily, a preset compression algorithm may be used to compress the original data block to obtain a corresponding compressed data block.

[0026] It can be understood that the size of the compressed data block can be dynamically changed, and the compressed data obtained by compressing the original data block is the compressed data block; the compressed data block can also be of a fixed size. In this case, if the size of the data after compressing the original data block is less than the fixed size, redundant data can be filled for alignment.

[0027] Step 102: Write the compressed data block into a first cache unit in a cache pool. The cache pool includes at least one cache unit. The space size of the cache unit is determined based on the stripe width and stripe block size of the disk array.

[0028] The cache pool is divided into at least one cache unit according to a fixed stripe width. The size of a cache unit is determined based on the stripe width and stripe block size of the disk array. For example, for a 4+2 RAID6, the RAID includes 4 data disks and 2 parity disks. Assuming the stripe block size is 256KB, the size of a cache unit in the cache pool is 256KB*4=1024KB=1M. By dividing the cache pool's cache space according to a fixed stripe width, it is possible to ensure that upper-level data is aggregated in the cache pool according to the stripe width.

[0029] In this embodiment, the currently obtained compressed data block is written to the first cache unit in the cache pool. The first cache unit can be any cache unit in the cache pool. The first cache unit is used to write the current compressed data block. The first cache unit can be determined based on the cache pool's write mechanism and the compressed data block is written thereto. For example, assuming the cache pool is divided into three cache units, numbered A, B, and C, and a polling mechanism is used to write to each cache unit, if the last write was to cache unit numbered B, then cache unit numbered C is determined as the first cache unit this time, and the compressed data block is written to cache unit numbered C.

[0030] Step 103: Detect whether the data volume of the compressed data block stored in the first cache unit meets the full stripe requirement.

[0031] In this embodiment, each cache unit of the cache pool can write different compressed data blocks multiple times to aggregate the compressed data in the cache pool. The storage device can periodically detect whether the data volume of the compressed data blocks cached in each cache unit meets the full stripe requirement. Alternatively, after a compressed data block is written to a first cache unit, the storage device can detect whether the data volume of all compressed data blocks stored in the first cache unit meets the full stripe requirement.

[0032] Among them, the full stripe requirement means that the amount of cached data in a cache unit exceeds the preset value. The specific value of the preset value can be set according to actual needs. For example, assuming that the space size of a cache unit is 1M, the preset value can be set to 800KB, 900KB, etc.

[0033] Step 104 : When the full stripe requirement is met, the compressed data blocks in the first cache unit are flushed to the disk array.

[0034] In this embodiment, when the total amount of data stored in the first cache unit meets the full stripe requirement, the flush mechanism is triggered, writing all compressed data blocks in the first cache unit to the RAID. By writing data from cache units that meet the full stripe requirement to the RAID, stripe lock contention is avoided. Furthermore, because the data written to the RAID is aggregated and of sufficient size, the RAID does not need to be filled with a large amount of redundant data, thus avoiding the waste of storage space caused by filling with a large amount of redundant data.

[0035] It can be understood that in the embodiment of the present application, when the amount of data cached by any cache unit in the cache pool meets the full stripe requirement, it triggers the data of the cache unit to be flushed to the RAID.

[0036] The data flushing method of the embodiment of the present application divides the cache pool into at least one cache unit, and the space size of the cache unit is determined based on the stripe width and stripe block size of the disk array. The compressed data blocks obtained by compressing the original data blocks are first written into the cache unit for aggregation. When the data volume of the compressed data blocks stored in the first cache unit meets the full stripe requirement, the compressed data blocks in the first cache unit are flushed to the disk array, so that the data flushed to the disk array is full stripe, and there is no need to wait for data accumulation through lock operations, thereby avoiding lock contention. Moreover, when the data volume of the compressed data blocks stored in the first cache unit meets the full stripe requirement, they are flushed to the disk array, so that the disk array does not need to fill a large amount of redundant data for stripe alignment, thereby avoiding the waste of storage space caused by a large amount of redundant data. Therefore, it can solve the technical problems of multiple lock operations waiting for data accumulation causing lock contention and adding redundant data to waste storage space, thereby achieving the technical effect of avoiding lock contention and saving storage space.

[0037] In the embodiment of the present application, each time an original data block is compressed into a compressed data block, the data amount corresponding to the original data block and the compressed data block, i.e., the size of the data block, can be recorded to determine the compression ratio of this time, so that the compression algorithm can be adjusted according to the previous compression ratio in subsequent compressions, thereby achieving flexible and dynamic selection of the compression algorithm. Thus, in an optional embodiment of the present application, if Figure 3 As shown, based on the above embodiment, step 101 may include the following sub-steps: Step 201: Obtain a first data block size and a second data block size corresponding to a previous original data block, where the second data block size refers to the size of a data block obtained by compressing the previous original data block according to a first compression algorithm.

[0038] The first compression algorithm may be preset. For example, the first compression algorithm may be an LZ4 compression algorithm, which has a relatively fast compression speed.

[0039] In this embodiment, when a new original data block written by a user is received, the data block size of the previous original data block (referred to as the first data block size for ease of description and distinction) and the size of the data block obtained by compressing the previous original data block according to the first compression algorithm (referred to as the second data block size for ease of description and distinction) can be obtained first. The first compression algorithm is used to perform real-time compression on the previous original data block to obtain a data block of variable length. The data block size refers to the amount of data contained in the data block. For example, if the first data block size of the previous original data block is 16KB, after compressing the previous original data block according to the first compression algorithm, the second data block size of the compressed data block is 12KB.

[0040] It should be noted that, in this embodiment, the second data block size refers to the actual length of the data block after the original data block is compressed, rather than the length of the compressed data block after alignment according to a fixed length.

[0041] Step 202: Determine the data compression ratio of the previous original data block based on the first data block size and the second data block size.

[0042] The data compression ratio is equal to the percentage of the second data block size divided by the first data block size. Generally, the smaller the compression ratio, the better.

[0043] Step 203: When the data compression rate is higher than the compression rate threshold, the original data block is compressed according to a second compression algorithm to obtain a compressed data block.

[0044] The compression rate threshold can be set according to actual needs. For example, the compression rate threshold can be set to 90%.

[0045] In this embodiment, after obtaining the data compression rate of the previous original data block, the obtained data compression rate can be compared with a preset compression rate threshold. If the data compression rate is not higher than the compression rate threshold, the original data block received this time is still compressed according to the first compression algorithm to obtain a compressed data block. If the data compression rate is higher than the compression rate threshold, the original data block is compressed according to the second compression algorithm to obtain a compressed data block.

[0046] The second compression algorithm is different from the first compression algorithm. For example, the first compression algorithm is the LZ4 compression algorithm, and the second compression algorithm may be the Zstandard algorithm.

[0047] The data flushing method of the embodiment of the present application obtains the original data block size of the previous original data block and the compressed data block size after compression and calculates the data compression rate. When the data compression rate is higher than the compression rate threshold, the original data block to be written this time is compressed according to the second compression algorithm to obtain a compressed data block. In this way, the compression algorithm is flexibly adjusted according to the compression rate of the previous original data block, so that the compression algorithm can be dynamically updated to ensure the compression rate and save storage space.

[0048] In an optional embodiment of the present application, when detecting whether the data volume of the compressed data blocks stored in the first cache unit meets the full stripe requirement, the data volume of the compressed data blocks stored in the first cache unit can be obtained, that is, the sum of the data block sizes of all compressed data blocks stored in the first cache unit is obtained, and then, based on the obtained data volume and the space size of the first cache unit, the current space occupancy of the first cache unit is determined, that is, the current space occupancy is equal to the data volume of the compressed data blocks stored in the first cache unit divided by the percentage of the space size of the first cache unit. For example, assuming that the space size of the first cache unit is 1M and the data volume of the compressed data blocks stored in the first cache unit is 9 00KB, the current space occupancy is 900KB / 1024KB*100%≈87.89%; then, the obtained current space occupancy can be compared with the preset occupancy threshold. When the current space occupancy reaches the occupancy threshold, it is determined that the data volume of the compressed data blocks stored in the first cache unit meets the full stripe requirement, wherein the occupancy threshold can be pre-set according to actual needs. For example, the occupancy threshold can be set to 90%. The larger the value of the occupancy threshold, the more data volume of the compressed data blocks aggregated in the cache pool, the less redundant data that the RAID needs to fill after receiving the compressed data blocks flushed from the cache pool, and the higher the effective occupancy rate of the storage space.

[0049] In an embodiment of the present application, by determining the current space occupancy of the first cache unit based on the amount of compressed data blocks stored in the first cache unit and the space size of the first cache unit, and judging whether the full stripe requirement is met based on the current space occupancy, stripe lock contention can be effectively avoided and the amount of data used to fill redundant data can be reduced.

[0050] In an optional embodiment of the present application, on the basis of triggering the flushing mechanism by the full stripe requirement, the flushing mechanism can also be triggered in combination with a timer. Specifically, a timer can be set for the cache pool. When the timing duration of the timer exceeds the duration threshold, the amount of compressed data blocks stored in no cache unit meets the full stripe requirement, and the compressed data blocks of each cache unit in the cache pool are flushed to the RAID; or, a corresponding timer can be set for each cache unit in the cache pool. For any cache unit, if the timing duration of the timer exceeds the duration threshold, the amount of compressed data blocks stored in the corresponding cache unit does not meet the full stripe requirement, and the compressed data blocks stored in the cache unit are still flushed to the RAID. In this way, it is possible to avoid the situation where the data written by the user cannot be written to the disk for a long time, and the data written by the user can be flushed to the disk as promptly as possible to ensure the timeliness of data writing. Among them, the value of the timer duration threshold can be set to the minute level, reserving sufficient time for the cache unit to aggregate compressed data blocks that meet the full stripe requirement, and trying not to trigger the timer flush mechanism to ensure that the amount of data written to the RAID in a single time reaches the full stripe, thereby reducing the filling of invalid data and reducing the occupation of disk storage space by invalid data.

[0051] The data flushing method of the present application is applied to a storage device, which may be a dual-controller storage device. The storage device includes a first controller and a second controller, wherein the first controller and the second controller are mirrored. Assuming that the first controller serves as the primary controller, a data block copy of a compressed data block stored in a cache unit of the first controller's cache pool is synchronized to a cache unit of the corresponding cache unit of the second controller through cache mirroring. In other words, each time a compressed data block is written to a cache unit of the first controller's cache pool, a data block copy of the compressed data block (identical to the compressed data block) is synchronously written to a cache unit of the second controller's cache pool. The first controller's cache pool and the second controller's cache pool have the same number of cache units of the same size. Each cache unit of the first controller's cache pool has a mirrored cache unit in the second controller's cache pool, and the compressed data blocks stored in the two mirrored cache units are identical. A cache unit in the first controller's cache pool is managed by a unique thread and corresponds to a unique thread identifier. The cache unit is also associated with the mirrored cache unit in the second controller's cache pool through the thread identifier, and a mapping relationship between each cache unit and the thread identifier is stored in the second control unit. For example, the cache unit numbered D in the cache pool of the first controller and the cache unit numbered E in the cache pool of the second controller are mirror images of each other, and the cache unit numbered D is managed by thread T. Then the cache unit numbered E in the second controller is associated with thread T, and the mapping relationship between the two is stored.

[0052] Thus, in an optional embodiment of the present application, after the compressed data block in the first cache unit is flushed to the disk array, the target thread identifier of the thread to which the first cache unit belongs can also be obtained, and a cache clearing message can be generated based on the target thread identifier, wherein the cache clearing message carries the target thread identifier, which is used to instruct the second controller to send the target thread identifier and then the cache clearing message to the second controller, wherein the second controller responds to the cache clearing message, queries the mapping relationship between the thread identifier and each cache unit in its own cache pool, determines the cache unit corresponding to the target thread identifier (for ease of description and distinction, referred to as the cache unit to be cleared), and clears the data block copy in the cache unit to be cleared.

[0053] In this embodiment, for a dual-control storage device, a data block copy of a compressed data block stored in a cache unit of the cache pool of the first controller is synchronized to a cache unit of the cache pool corresponding to the second controller through cache mirroring. After the compressed data block in the first cache unit is flushed to the disk array, all data in the cache unit of the cache pool of the second controller that is mirrored with the first cache unit is cleared according to the thread identifier of the thread to which the first cache unit belongs, so that the second controller can promptly release the cache space of the cache unit where the data that has been written to the disk is located, thereby ensuring synchronization with the cache pool of the first controller.

[0054] Furthermore, in an optional embodiment of the present application, when the first controller fails, the second controller takes over the task, and the storage device flushes the data block copies stored in the cache unit of the cache pool corresponding to the second controller to the disk array through the second controller.

[0055] That is, for a dual-controller storage device, when the first of the two mirrored controllers fails, the second controller, acting as a backup, takes over. Since the cache pool of the second controller mirrors the data in the cache pool of the first controller, a copy of the actual data is also stored in the second controller, and the data is not lost. When the first controller fails, the second controller can write the data in the cache pool to the RAID, and no longer uses the full stripe method. This allows data written to the cache pool before the first controller fails to be flushed to the RAID after the first controller fails, ensuring data security and reliability.

[0056] Furthermore, in an optional embodiment of the present application, after a first controller fails, if a user continues to write new data, the storage device may store newly generated compressed data blocks in a cache unit of a cache pool corresponding to a second controller. If the amount of compressed data blocks stored in a second cache unit of the cache pool corresponding to the second controller reaches the storage capacity, the second controller may flush the compressed data blocks in the second cache unit to the disk array. The second cache unit may be any cache unit in the cache pool of the second controller.

[0057] For example, each time a new compressed data block is written to a cache unit in the second controller's cache pool, a check is performed to determine whether the total amount of all compressed data blocks stored in the cache unit reaches a preset space size, such as 1MB. If so, the compressed data blocks in the cache unit are flushed to the RAID array via the second controller. When the amount of data stored in the cache unit reaches a preset percentage (e.g., 95%) of the space size, it can be determined that the amount of data stored in the cache unit has reached the space size.

[0058] That is, in this embodiment, after a failure of the first controller, newly generated compressed data blocks are stored in the cache units of the second controller's cache pool. When the amount of compressed data blocks stored in any cache unit in the second controller's cache pool reaches the capacity, the second controller flushes the data in the corresponding cache unit to the RAID array to quickly complete the data flush. This allows for rapid data flushing to the RAID array when the cache unit is full, without waiting for a timer to expire, thus improving the timeliness of data flushing.

[0059] Furthermore, in an optional embodiment of the present application, when the amount of data of the compressed data blocks stored in the cache units of the cache pool corresponding to the second controller does not reach the space size, the current timing duration of the timer can also be obtained; when the current timing duration reaches the duration threshold, the compressed data blocks in the cache pool corresponding to the second controller are flushed to the disk array through the second controller.

[0060] The duration threshold can be set based on actual needs. Setting the duration threshold to milliseconds, for example, means setting the timer period (i.e., the duration threshold) to 10 milliseconds. Upon expiration, data in the cache pool is flushed to the RAID. By setting the duration threshold to milliseconds, data can be aggregated before being flushed to the RAID, ensuring timely data flushing while also maximizing the efficiency of the flush operation.

[0061] That is, in this embodiment, when a new compressed data block is stored in the cache unit of the second controller, if the amount of compressed data blocks in the cache unit reaches the space size, the compressed data blocks in the cache unit are quickly flushed to the disk array. If the amount of compressed data blocks stored in no cache unit reaches the space size, but the timer has reached the time threshold, all compressed data blocks cached in the cache pool are flushed to the disk array via the second controller, and the timer restarts. Thus, by setting a timer for timing, the cached data is flushed to the disk array when the cache unit is not full but the timer times out. This ensures the timeliness of data placement on disk while minimizing the use of aggregation to aggregate data before flushing to RAID, thereby reducing storage space waste.

[0062] Assuming a dual-controller storage device, if a single controller fails, the storage switches from cache mirror mode to single-controller mode. The controller that takes over the task flushes the cached data to the RAID as quickly as possible. Subsequent writes remain aggregated in the cache pool, but the cache does not return a write completion message to the upper layer before sending the cached data to the RAID. When the data stored in a cache unit in the cache pool reaches the stripe width (that is, the preset space size, such as 2MB), the data is quickly flushed to the RAID. A timer is also set to start counting. When the timer expires (the timer reaches the preset duration threshold), the data in the cache pool is immediately flushed to the RAID, regardless of whether the amount of data stored in each cache unit reaches the stripe width. Because stripe lock contention does not exist in single-controller mode, lock-freeing is not necessary.

[0063] The data flushing method of the present application can also be applied to multi-controller storage devices. In the multi-controller storage device, the two controllers are still mirrored to each other. Assuming that the multi-controller storage device includes a first controller, a second controller and a third controller, the three controllers have the same structure, the first controller and the second controller are mirrored to each other, and the data flushing logic is the same as the implementation logic described in the above embodiment. When the first controller fails, the cache pool of the second controller and the cache pool of the third controller are combined into a mirror pair, and the second controller is used as the main controller. For the newly generated compressed data blocks received by the second controller, not only are they stored in the cache unit of the cache pool corresponding to the second controller, but also the data block copies of the compressed data blocks stored in the cache unit of the cache pool corresponding to the second controller are synchronized to the cache unit of the cache pool corresponding to the third controller through cache mirroring, so as to realize the backup of the cached data in the cache pool of the second controller and ensure the security and reliability of the data. It should be noted that in this embodiment, after the cache pool of the second controller and the cache pool of the third controller are formed into a mirror pair, the data flushing logic is the same as the implementation logic when the first controller and the second controller are mirrored to each other in the aforementioned embodiment. When the second controller fails, the data storage is converted from the cache mirror mode to the single-control mode. The data flushing logic is the same as the implementation logic after the first controller fails in the dual-control storage device, which will not be repeated here.

[0064] It should be noted that, in the embodiment of the present application, the compressed data blocks cached in the cache unit may be variable-length data blocks obtained by direct compression, or may be data blocks obtained by aligning variable-length data blocks. Figure 4 A schematic diagram of a RAID write architecture based on stripe alignment according to an exemplary embodiment of the present application is shown. Figure 4As shown, after compression, a data block before compression (for example, the data volume is 8KB) is compressed to obtain a compressed data block with 320 bytes aligned. The compressed data block includes a control block, a compressed data block and a padding block. The control block is 16 bytes long and includes a 2B control header length field (used to write the length of the control block, i.e., 16B), a 2B compression algorithm field (used to write the identifier of the compression algorithm used), a 3B true length field (used to write the true length of the compressed data block), a 5B check code field (used to write a redundant check code generated based on the pre-compression data or the compressed data), and a 4B reserved field. The compressed data block is a data block obtained by compressing the pre-compression data block according to the compression algorithm. The padding block is used to write the data required for 320-byte alignment. If the sum of the length of the control block and the length of the compressed data block is less than 320 bytes, redundant data is padded according to the number of bytes that differ (xB, where the value of x is determined by the difference between 320 and the sum of the above lengths) for alignment. Compressed data blocks are aggregated in the cache units of the cache pool and flushed to the RAID when the flushing conditions are met. For example, if the space size of a cache unit in the cache pool is 512KB, when the amount of data stored in the cache unit reaches 90%, the flushing mechanism is triggered to flush the data in the cache unit to the RAID. If the full stripe flushing requirement is not met when the timer times out, the flushing mechanism is also triggered. When the RAID is flushing data to the disk, it determines whether redundant data needs to be filled based on the stripe width and the amount of data received. If redundant data needs to be filled, it is filled and then flushed to the disk. The length of the redundant data to be filled is recorded as yB, and the value of y is determined by the difference between the stripe size and the amount of data to be written to the stripe.

[0065] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method.

[0066] An embodiment of the present application also provides a data flushing device, which can be implemented using software and / or hardware and can be integrated into a storage device.

[0067] Figure 5 This is a structural diagram of a data flash writing device provided in one embodiment of the present application. Figure 5 As shown, the data flushing device 30 includes: a compression module 310 , a cache module 320 , a condition detection module 330 and a flushing module 340 .

[0068] The compression module 310 is used to compress the original data block to obtain a compressed data block; A cache module 320 is configured to write the compressed data block into a first cache unit in a cache pool, where the cache pool includes at least one cache unit, and the size of the cache unit is determined based on the stripe width and stripe block size of the disk array; A condition detection module 330 is configured to detect whether the amount of compressed data blocks stored in the first cache unit meets a full stripe requirement; The flush module 340 is configured to flush the compressed data blocks in the first cache unit to the disk array when the full stripe requirement is met.

[0069] Optionally, the compression module 310 is further configured to: Obtaining a first data block size and a second data block size corresponding to a previous original data block, where the second data block size refers to a size of a data block obtained by compressing the previous original data block according to a first compression algorithm; determining a data compression ratio of a previous original data block based on the first data block size and the second data block size; When the data compression rate is higher than the compression rate threshold, the original data block is compressed according to the second compression algorithm to obtain a compressed data block.

[0070] Optionally, the condition detection module 330 is further configured to: Obtaining the amount of compressed data blocks stored in the first cache unit; Determining a current space occupancy rate of the first cache unit based on the amount of data and the space size of the first cache unit; When the current space occupancy reaches the occupancy threshold, it is determined that the data volume of the compressed data block stored in the first cache unit meets the full stripe requirement.

[0071] Optionally, the storage device includes a first controller and a second controller, the first controller and the second controller are mirrored, and a data block copy of a compressed data block stored in a cache unit of a cache pool of the first controller is synchronized to a cache unit of a corresponding cache pool of the second controller through a cache mirroring method; The data flash writing device 30 further includes: An identifier acquisition module, configured to acquire a target thread identifier of a thread to which the first cache unit belongs; A message generation module, configured to generate a cache clearing message based on a target thread identifier; A message sending module is used to send a cache clearing message to the second controller, wherein the second controller, in response to the cache clearing message, queries the mapping relationship between the thread identifier and each cache unit in its own cache pool, determines the cache unit to be cleared corresponding to the target thread identifier, and clears the data block copy in the cache unit to be cleared.

[0072] Optionally, the flush module 340 is further configured to: in response to a failure of the first controller, flush the data block copies stored in the cache units of the cache pool corresponding to the second controller to the disk array via the second controller.

[0073] The description of the features in the embodiment corresponding to the data flushing device can be found in the relevant description of the embodiment corresponding to the data flushing method, and will not be repeated here.

[0074] An embodiment of the present application further provides a storage device comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any of the above-mentioned data flushing method embodiments.

[0075] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above-mentioned data flashing method embodiments when run.

[0076] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0077] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in any of the above-mentioned data flushing method embodiments are implemented.

[0078] An embodiment of the present application also provides another computer program product, including a non-volatile computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the steps in any of the above-mentioned data flashing method embodiments.

[0079] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0080] The above is a detailed introduction to a data flashing method, storage device, storage medium and computer program product provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of the present application.

Claims

1. A data flashing method, characterized in that: Applicable to storage devices, including: Compressing the original data block to obtain a compressed data block; Writing the compressed data block into a first cache unit in a cache pool, wherein the cache pool includes at least one cache unit, and a space size of the cache unit is determined based on a stripe width and a stripe block size of a disk array; Detecting whether the data volume of the compressed data blocks stored in the first cache unit meets the full stripe requirement; When the full stripe requirement is met, the compressed data blocks in the first cache unit are flushed to the disk array.

2. The data flashing method according to claim 1, characterized in that: The compressing the original data block to obtain the compressed data block includes: Obtaining a first data block size and a second data block size corresponding to a previous original data block, where the second data block size refers to a size of a data block obtained by compressing the previous original data block according to a first compression algorithm; determining a data compression ratio of the previous original data block based on the first data block size and the second data block size; When the data compression rate is higher than the compression rate threshold, the original data block is compressed according to a second compression algorithm to obtain the compressed data block.

3. The data flashing method according to claim 1, wherein: The detecting whether the data amount of the compressed data block stored in the first cache unit meets the full stripe requirement includes: Obtaining the amount of compressed data blocks stored in the first cache unit; Determining a current space occupancy rate of the first cache unit based on the data volume and the space size of the first cache unit; When the current space occupancy reaches an occupancy threshold, it is determined that the data volume of the compressed data blocks stored in the first cache unit meets the full stripe requirement.

4. The data flashing method according to any one of claims 1 to 3, characterized in that: The storage device includes a first controller and a second controller, wherein the first controller and the second controller are mirrored and synchronize data block copies of compressed data blocks stored in a cache unit of a cache pool of the first controller to a cache unit of a cache pool corresponding to the second controller through cache mirroring; After flushing the compressed data blocks in the first cache unit to the disk array, the method further includes: Obtain a target thread identifier of a thread to which the first cache unit belongs; Generate a clear cache message based on the target thread identifier; The cache clearing message is sent to the second controller, wherein the second controller queries the mapping relationship between the thread identifier and each cache unit in its own cache pool in response to the cache clearing message, determines the cache unit to be cleared corresponding to the target thread identifier, and clears the data block copy in the cache unit to be cleared.

5. The data flashing method according to claim 4, characterized in that: The method further comprises: In response to a failure of the first controller, the data block copies stored in the cache units of the cache pool corresponding to the second controller are flushed to the disk array through the second controller.

6. The data flashing method according to claim 5, characterized in that: The method further comprises: storing the newly generated compressed data block in a cache unit of the cache pool corresponding to the second controller; When the amount of compressed data blocks stored in the second cache unit of the cache pool corresponding to the second controller reaches the space size, the compressed data blocks in the second cache unit are flushed to the disk array through the second controller.

7. The data flashing method according to claim 6, characterized in that: The method further comprises: When the amount of compressed data blocks stored in the cache units of the cache pool corresponding to the second controller does not reach the space size, obtaining the current timing length of the timer; When the current timing duration reaches a duration threshold, the compressed data blocks in the cache pool corresponding to the second controller are flushed to the disk array through the second controller.

8. The data flashing method according to claim 5, characterized in that: The storage device further includes a third controller; The method further comprises: The cache pool of the second controller and the cache pool of the third controller form a mirror pair; The newly generated compressed data block is stored in the cache unit of the cache pool corresponding to the second controller, and the data block copy of the compressed data block stored in the cache unit of the cache pool corresponding to the second controller is synchronized to the cache unit of the cache pool corresponding to the third controller through cache mirroring.

9. A storage device, characterized in that: include: memory for storing computer programs; A processor, configured to implement the steps of the data flashing method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the data flashing method according to any one of claims 1 to 7.

Images