Garbage collection intensity prediction and compensation for storage devices

Abstract

The application relates to a storage device, in particular to a control method for garbage collection of a storage device, which comprises the following steps: comparing the difference between an actual valid data distribution and an ideal valid data distribution; initiating a garbage collection operation or adjusting the available bandwidth provided for the user to write data and / or the valid data moved by the garbage collection operation according to the difference. The garbage collection operation is controlled by the characteristics of the user writing data, so that the write performance of the storage device is improved, and the user experience of the storage device is improved.

Description

Technical Field

[0001] This application relates to storage devices, and more particularly to predicting the intensity of future garbage collection operations in storage devices, and compensating for garbage collection operations in advance based on the prediction results to reduce the impact of garbage collection operations on the performance of storage devices. Background Technology

[0002] Figure 1 A block diagram of a storage device is shown. Storage device 100 is coupled to a host computer to provide storage capabilities. The host computer and storage device 100 can be coupled in various ways, including but not limited to connections via SATA (Serial Advanced Technology Attachment), SCSI (Small Computer System Interface), SAS (Serial Attached SCSI), IDE (Integrated Drive Electronics), USB (Universal Serial Bus), PCIe (Peripheral Component Interconnect Express), NVMe (NVM Express), Ethernet, Fibre Channel, and wireless communication networks. The host computer can be an information processing device capable of communicating with the storage device via the above methods, such as a personal computer, tablet computer, server, laptop computer, network switch, router, cellular phone, or personal digital assistant. Storage device 100 includes interface 110, control unit 120, one or more NVM chips 130, and DRAM (Dynamic Random Access Memory) 140.

[0003] NAND flash memory, phase change memory, FeRAM (Ferroelectric RAM), MRAM (Magnetic Random Access Memory), and RRAM (Resistive Random Access Memory) are common types of NVM.

[0004] Interface 110 is compatible with exchanging data with the host via methods such as SATA, IDE, USB, PCIe, NVMe, SAS, Ethernet, and Fibre Channel.

[0005] Control unit 120 is used to control data transfer between interface 110, NVM chip 130, and DRAM 140. It is also used for memory management, mapping host logical addresses to NVM chip physical addresses, erase leveling, bad block management, etc. Control unit 120 can be implemented in various ways, including software, hardware, firmware, or a combination thereof. For example, control unit 120 can be in the form of an FPGA (Field-programmable gate array), ASIC (Application Specific Integrated Circuit), or a combination thereof. Control unit 120 may also include a processor or controller, in which software executes to manipulate the hardware of control unit 120 to process I / O (Input / Output) commands. Control unit 120 can also be coupled to DRAM 140 and can access data in DRAM 140. FTL tables and / or cached I / O command data are stored in DRAM.

[0006] The control unit 120 includes a flash interface controller (or media interface, media interface controller, flash channel controller), which is coupled to the NVM chip 130 and issues commands to the NVM chip 130 in accordance with the interface protocol of the NVM chip 130 to operate the NVM chip 130, and receives the command execution results output from the NVM chip 130. Known NVM chip interface protocols include "Toggle" and "ONFI".

[0007] A memory target is one or more logic units (LUNs) within a NAND flash memory package that share a chip enable (CE) signal. A NAND flash memory package includes one or more dies. Typically, a logic unit corresponds to a single die. A logic unit may include multiple planes. Multiple planes within a logic unit can be accessed in parallel, while multiple logic units within a NAND flash memory chip can execute commands and report status independently of each other. The meanings of target, logic unit (LUN), and plane are provided in the "Open NAND Flash Interface Specification (Revision 3.0)" obtained in the prior art, which is part of the prior art.

[0008] NVM storage media typically store and retrieve data in pages, while erasing data in blocks. A block (also called a physical block) on an NVM storage medium contains multiple pages. A page (called a physical page) on the storage medium has a fixed size, such as 17664 bytes. Physical pages can also have other sizes.

[0009] In storage devices, the FTL (Flash Translation Layer) is used to maintain the mapping information from logical addresses to physical addresses. Logical addresses constitute the storage space of the storage device as perceived by upper-layer software such as the operating system. Physical addresses are the addresses used to access the physical storage units of the solid-state storage device. In existing technologies, address mapping can also be implemented using intermediate address formats. For example, a logical address can be mapped to an intermediate address, and then the intermediate address can be further mapped to a physical address. Optionally, the host accessing the storage device provides the FTL.

[0010] A table structure that stores mapping information from logical addresses to physical addresses is called an FTL table. Typically, the data entries in an FTL table record the address mapping relationships in the storage device, on a per-data-page basis.

[0011] Figure 2 A schematic diagram of a large block is shown. A large block comprises physical blocks from each of multiple logic units (called logic unit groups). Preferably, each logic unit provides one physical block for a large block. As an example, large blocks are constructed over every 16 logic units (LUNs). Each large block comprises 16 physical blocks from each of the 16 logic units (LUNs). Figure 2 In the example, block 0 comprises physical blocks 0 from each of the 16 logical units (LUNs), and block 1 comprises physical blocks 1 from each logical unit (LUN). There are also many other ways to construct blocks.

[0012] As an alternative approach, page stripes are constructed within large blocks, where physical pages with the same physical address within each logical unit (LUN) constitute a "page stripe". Figure 2 In this context, physical pages P0-0, P0-1, ..., and P0-x constitute page stripe 0. Physical pages P0-0, P0-1, ..., P0-14 are used to store user data, while physical pages P0-x are used to store checksum data calculated based on all user data within the stripe. Similarly, Figure 2 In this diagram, physical pages P2-0, P2-1, ..., and P2-x together form page stripe 2. Optionally, the physical page used to store the verification data can be located at any position within the page stripe.

[0013] When a logical page is repeatedly written to, the FTL table records the correspondence between the logical page address and the latest physical page address. Data recorded at physical page addresses that have been written to but are no longer referenced (e.g., no record in the FTL table) is called "garbage" (data). Data that has been written to and referenced (e.g., has a record in the FTL table) is called valid data, while "garbage" is called dirty data. Physical blocks containing dirty data are called "dirty physical blocks," and physical blocks that have not been written to are called "free physical blocks."

[0014] Storage devices perform a garbage collection (GC) process to reclaim invalid data.

[0015] Figure 3 This diagram illustrates the garbage collection process. Data was written to physical blocks 0 and 1. Physical pages 310, 312, 314, and 316 of physical block 0, indicated by grid boxes, have no records in the FTL table; their data is dirty. Physical pages 330, 332, 334, and 336 of physical block 0, indicated by blank boxes, have records in the FTL table; their data is valid. Physical pages 320, 322, 324, and 326 of physical block 1, indicated by grid boxes, contain dirty data. Physical pages 344, 342, 346, and 348 of physical block 1, indicated by blank boxes, contain valid data. Figure 3 In this context, physical pages indicated by grid lines contain dirty data, while physical pages indicated by blank boxes contain valid data.

[0016] To perform garbage collection, dirty physical blocks (e.g., physical block 0 and physical block 1) are scanned, their valid data is read and written to free physical block 2, and the changes in the physical page addresses of the valid data are recorded in the FTL table. After all valid data in the physical blocks has been moved to physical block 2, the scanned physical blocks 0 and 1 are erased, thus making physical blocks 0 and 1 free physical blocks.

[0017] Figure 4 A diagram illustrating waste recycling methods is shown.

[0018] The dirty physical block set 410 includes dirty physical blocks of some or all of the NVM chips in the storage device. The free physical block set 420 includes free physical blocks of some or all of the NVM chips in the storage device.

[0019] To implement garbage collection, the "garbage collection" module 430 (e.g., implemented in the control unit 120 or its CPU or controller) retrieves dirty physical blocks from the dirty physical block set 410 and free physical blocks from the free physical block set 420. It scans the dirty physical blocks and writes valid data from them to the free physical blocks. After all valid data in the retrieved dirty physical blocks has been moved to the free physical blocks, the dirty physical blocks are erased, and the erased physical blocks are recorded in the free physical block set.

[0020] The dirty physical block set 410 and the free physical block set 420 can be linked lists, linear lists, or other data structures used to represent sets. The addresses of physical blocks are recorded in the sets for accessing them.

[0021] Optionally, waste can be recycled in large blocks (instead of physical blocks).

[0022] The storage device also performs wear leveling to ensure that each physical block experiences roughly the same number of erases during use, thereby reducing the adverse impact of individual physical block depletion on the lifespan of the storage device.

[0023] In the Chinese patent application with application number 2018115664573 and invention title "Garbage Collection Method and Storage Device for Multiple Streams", the provided garbage collection method distinguishes data from garbage collection from data written by the user into different streams, and different streams can be written into different physical blocks, blocks or logical unit groups. Summary of the Invention

[0024] Garbage collection operations in storage devices can be continuous or non-continuous, reclaiming enough large blocks to accommodate user-written data. When user-written data volume increases rapidly, free blocks are quickly consumed, requiring garbage collection to generate free blocks more quickly. When user-written data exhibits randomness, the amount of valid data in the written blocks decreases, reducing the amount of valid data that needs to be moved to reclaim a specified number of free blocks. Therefore, the characteristics of user-written data (bandwidth, randomness, etc.) affect the behavior of garbage collection operations. The behavior of garbage collection operations can be described, for example, by the change in bandwidth over time caused by the movement of valid data during garbage collection. Furthermore, the valid data that garbage collection operations need to move competes with user-written data for storage device resources (e.g., write bandwidth), impacting the user experience of storage device performance. Therefore, further improvements to garbage collection operations are needed to reduce the impact of garbage collection operations on user experience and storage device performance, and / or reduce the volatility of garbage collection operation behavior. The inventors also noted that changes in the characteristics of user-written data affect the behavior of garbage collection operations. For example, when a user switches from sequential write mode to random write mode, the behavior of garbage collection operations changes accordingly within a certain period of time, which leads to a short-term decrease in write performance experienced by the user, which is detrimental to the user.

[0025] According to a first aspect of this application, a first control method for garbage collection of a storage device according to the first aspect of this application is provided, comprising the steps of: comparing the difference between the actual effective data distribution and the ideal effective data distribution; initiating a garbage collection operation based on the difference, or adjusting the available bandwidth provided for effective data written by the user and / or moved by the garbage collection operation.

[0026] According to the first control method for garbage collection of storage devices according to the first aspect of this application, a second control method for garbage collection of storage devices according to the first aspect of this application is provided, wherein, in response to the effective data amount indicated by the actual effective data distribution being greater than the effective data amount indicated by the ideal effective data distribution, or the difference between the effective data amount indicated by the actual effective data distribution and the effective data amount indicated by the ideal effective data distribution being greater than a threshold, a garbage collection operation is initiated, or the available bandwidth provided for the garbage collection operation to move effective data is increased, or the ratio of the available bandwidth provided for the garbage collection operation to move effective data to the available bandwidth of user data is increased.

[0027] According to the first control method for garbage collection of storage devices according to the first aspect of this application, a third control method for garbage collection of storage devices according to the first aspect of this application is provided, wherein, in response to the effective data amount indicated by the actual effective data distribution not being greater than the effective data amount indicated by the ideal effective data distribution, or the difference between the data amount indicated by the actual effective data distribution and the effective data amount indicated by the ideal effective data distribution being less than a threshold; the available bandwidth provided for users to write data is increased, or the ratio of the available bandwidth provided for users to write data to the available bandwidth for garbage collection operations to move effective data is increased.

[0028] According to the second or third control method for garbage collection of storage devices according to the first aspect of this application, a fourth control method for garbage collection of storage devices according to the first aspect of this application is provided, wherein the available bandwidth provided for user write data is an upper limit of the bandwidth allowed for user write data; and the available bandwidth provided for garbage collection operation to move valid data is an upper limit of the bandwidth allowed for garbage collection operation to move valid data.

[0029] According to one of the first to fourth control methods for garbage collection of storage devices according to the first aspect of this application, a fifth control method for garbage collection of storage devices according to the first aspect of this application is provided, which further includes the following steps: obtaining the characteristics of the current load of the storage device; obtaining an ideal effective data distribution corresponding to the current load characteristics based on the characteristics of the current load of the storage device, so as to compare the difference between the actual effective data distribution and the ideal effective data distribution.

[0030] According to one of the first to fifth control methods for storage device waste recycling according to the first aspect of this application, a sixth control method for storage device waste recycling according to the first aspect of this application is provided, which further includes: measuring the effective data distribution on a large block of the storage device to obtain the actual effective data distribution.

[0031] According to the fifth or sixth control method for garbage collection of storage devices according to the first aspect of this application, a seventh control method for garbage collection of storage devices according to the first aspect of this application is provided, wherein the load characteristics of the storage device are sequential write load, random write load, mixed write load and / or one or more streaming loads received by the storage device.

[0032] According to the seventh control method for garbage collection of storage devices according to the first aspect of this application, an eighth control method for garbage collection of storage devices according to the first aspect of this application is provided, wherein the mixed write load is a load in which sequential writes and random writes are mixed in a specified proportion.

[0033] According to one of the sixth to eighth control methods for storage device garbage collection according to the first aspect of this application, a ninth control method for storage device garbage collection according to the first aspect of this application is provided, wherein the effective data distribution on a large block of storage device is measured at any time or periodically to obtain the actual effective data distribution.

[0034] According to one of the first to ninth control methods for garbage collection of storage devices according to the first aspect of this application, a tenth control method for garbage collection of storage devices according to the first aspect of this application is provided, wherein the ideal effective data distribution is the distribution of the proportion of effective data in a large block to the total data in a large block under steady-state random write load.

[0035] According to one of the first to tenth control methods for garbage collection of storage devices according to the first aspect of this application, an eleventh control method for garbage collection of storage devices according to the first aspect of this application is provided, wherein the ideal effective data distribution is the distribution of the large block of effective data in the storage device under steady-state random write load.

[0036] According to the tenth or eleventh control method for garbage collection of storage devices according to the first aspect of this application, a twelfth control method for garbage collection of storage devices according to the first aspect of this application is provided, wherein the steady-state random write load is a continuous and stable random write load.

[0037] According to one of the first to twelfth control methods for garbage collection of storage devices according to the first aspect of this application, a thirteenth control method for garbage collection of storage devices according to the first aspect of this application is provided, wherein the ideal effective data distribution of the storage device under steady-state random write load is measured in a laboratory environment; or the ideal effective data distribution of the storage device under steady-state random write load is measured during the use of the storage device.

[0038] According to one of the first to thirteenth control methods for garbage collection of storage devices according to the first aspect of this application, a fourteenth control method for garbage collection of storage devices according to the first aspect of this application is provided, which further includes the following steps: efficiently sorting the data of major blocks of the storage device to obtain the sorted actual effective data distribution; subtracting the sorted ideal effective data distribution from the sorted actual effective data distribution in the sorting order to obtain the difference between the actual effective data distribution and the ideal effective data distribution of each major block, wherein the sorted ideal effective data distribution is obtained by efficiently sorting the data of major blocks of the storage device under steady-state random write load; accumulating the difference between the actual effective data distribution and the ideal effective data distribution of each major block to obtain the difference between the actual effective data distribution and the ideal effective data distribution.

[0039] According to the fourteenth control method for garbage collection of storage devices according to the first aspect of this application, a fifteenth control method for garbage collection of storage devices according to the first aspect of this application is provided, wherein the data effectiveness rate is the proportion of the effective data volume of each major block of the storage device to its total data volume.

[0040] According to one of the first to fifteenth control methods for storage device waste recycling according to the first aspect of this application, a sixteenth control method for storage device waste recycling according to the first aspect of this application is provided, wherein, by means of… Obtain the difference between the actual effective data distribution and the ideal effective data distribution; where N is the number of large blocks in the storage device; array P represents the actual effective data distribution, and the elements of array P are sorted, indicating the proportion of effective data in each sorted large block to its total data volume; the array index j of element P[j] indicates the sequence number of the large block represented by element P[i]; array I represents the ideal effective data distribution, and the elements of array I are sorted, indicating the proportion of effective data in each sorted large block to its total data volume; the array index j of element I[j] indicates the sequence number of the large block represented by element I[j].

[0041] According to the sixteenth control method for waste recycling of storage devices according to the first aspect of this application, a seventeenth control method for waste recycling of storage devices according to the first aspect of this application is provided, wherein, in calculating... During the process, if for some value m of j from 1 to N, such that If the value exceeds the threshold T, a garbage collection operation is initiated, or the available bandwidth provided for the garbage collection operation to move valid data is increased, or the ratio of the available bandwidth provided for the garbage collection operation to the available bandwidth of user data is increased.

[0042] According to the seventeenth control method for garbage collection of storage devices according to the first aspect of this application, an eighteenth control method for garbage collection of storage devices according to the first aspect of this application is provided, wherein the threshold T is 0 or a specified value.

[0043] According to one of the sixteenth to eighteenth control methods for garbage collection of storage devices according to the first aspect of this application, a nineteenth control method for garbage collection of storage devices according to the first aspect of this application is provided, wherein array P and array I are sorted in ascending order according to the effective data volume or effective data volume percentage of corresponding large blocks; in the calculation During the process, if for some value m of j from 1 to N, such that If the value is greater than the threshold T, a garbage collection operation is initiated, or the available bandwidth for moving valid data in the garbage collection operation is increased, without needing to calculate the value of j from m+1 to N.

[0044] According to one of the first to nineteenth control methods for garbage collection of storage devices according to the first aspect of this application, a twentieth control method for garbage collection of storage devices according to the first aspect of this application is provided, which further includes the following steps: efficiently sorting the data of each major block of the storage device and plotting the curve of actual effective data distribution; calculating the area of ​​the portion jointly enclosed by the curve of actual effective data distribution and the curve of ideal effective data distribution, wherein the area of ​​the portion jointly enclosed by the curve of actual effective data distribution and the curve of ideal effective data distribution represents the difference between the actual effective data distribution and the ideal effective data distribution.

[0045] According to the twentieth control method for garbage collection of storage devices according to the first aspect of this application, a twenty-first control method for garbage collection of storage devices according to the first aspect of this application is provided, wherein the data effectiveness rate is the proportion of the effective data volume of each major block of the storage device to its total data volume.

[0046] According to the twentieth or twentieth-first control method for garbage collection of storage devices according to the first aspect of this application, a twentieth-second control method for garbage collection of storage devices according to the first aspect of this application is provided, wherein the data of each block of the storage device is efficiently sorted under steady-state random write load, and a curve of ideal effective data distribution is plotted.

[0047] According to the twenty-second control method for garbage collection of storage devices according to the first aspect of this application, a twenty-third control method for garbage collection of storage devices according to the first aspect of this application is provided, wherein the curve of the ideal effective data distribution is a monotonically increasing function, the horizontal axis represents the block number of the storage device, and the vertical axis represents the data effectiveness of the block.

[0048] According to one of the twentieth to twenty-third control methods for storage device garbage collection according to the first aspect of this application, a twenty-fourth control method for storage device garbage collection according to the first aspect of this application is provided, wherein the curve of the ideal effective data distribution is y = f1(x), and the curve of the actual effective data distribution is y = f2(x), then the area enclosed by the curve of the actual effective data distribution and the curve of the ideal effective data distribution is... Where M is the number of blocks in the storage device, and x is the sequence number of the blocks in the storage device.

[0049] According to the twenty-fourth control method for garbage collection of storage devices according to the first aspect of this application, a twenty-fifth control method for garbage collection of storage devices according to the first aspect of this application is provided, wherein if the area S is greater than 0 or a specified threshold T, a garbage collection operation is initiated, and the available bandwidth provided for the garbage collection operation to move valid data is increased, or the proportion of available bandwidth provided for the garbage collection operation to move valid data is increased.

[0050] According to the twenty-fourth control method for garbage collection of storage devices according to the first aspect of this application, a twenty-sixth control method for garbage collection of storage devices according to the first aspect of this application is provided, wherein if the area S is less than 0 or a specified threshold, the current garbage collection operation intensity is kept unchanged, or the ratio of the bandwidth provided to the garbage collection operation for moving effective data to the bandwidth for user-written data is reduced.

[0051] According to the twenty-fourth control method for garbage collection of storage devices according to the first aspect of this application, a twenty-seventh control method for garbage collection of storage devices according to the first aspect of this application is provided, wherein if, during the calculation of area S, there exists a value of j such that the non-final value of area S is greater than 0 or a specified threshold T, then a garbage collection operation is initiated, or the available bandwidth provided for the garbage collection operation to move valid data is increased, or the proportion of available bandwidth provided for the garbage collection operation to move valid data is increased.

[0052] According to the twenty-fourth control method for garbage collection of storage devices according to the first aspect of this application, a twenty-eighth control method for garbage collection of storage devices according to the first aspect of this application is provided, wherein if there exists f3(x)>f1(x), and according to If the final value of the calculated area S is less than 0, the current intensity of the garbage collection operation remains unchanged, or the ratio of the bandwidth provided to the effective data for garbage collection operations to the bandwidth for user-written data is reduced, where y = f3(x) is the curve of the actual effective data distribution.

[0053] According to one of the twenty-fourth to twenty-eighth control methods for garbage collection of storage devices according to the first aspect of this application, a twenty-ninth control method for garbage collection of storage devices according to the first aspect of this application is provided, wherein: the actual number of large blocks in each data efficiency interval is obtained according to the curve of the actual effective data distribution; the difference between the actual number of large blocks from the actual effective data distribution and the ideal number of large blocks from the ideal effective data distribution in each data efficiency interval is obtained according to the difference between the actual number of large blocks from the actual effective data distribution and the ideal number of large blocks from the ideal effective data distribution in the data efficiency interval and the data efficiency value of the interval, and the difference in effective data volume corresponding to the actual data efficiency distribution and the ideal data efficiency distribution in each interval is accumulated; the difference in effective data volume in each interval is obtained as the difference between the actual effective data distribution and the ideal effective data distribution of the storage device.

[0054] According to one of the first to twenty-ninth control methods for storage device waste recycling according to the first aspect of this application, a thirtieth control method for storage device waste recycling according to the first aspect of this application is provided, wherein, by curve The ideal effective data distribution is obtained; where a is the maximum data efficiency of the large blocks of the storage device, b is the minimum data efficiency of the large blocks of the storage device, and h is the number of large blocks of the storage device.

[0055] According to the thirtieth control method for waste recycling of storage devices according to the first aspect of this application, a thirty-first control method for waste recycling of storage devices according to the first aspect of this application is provided, wherein, by... and curve fitting Obtain the curve Where S is the area enclosed by the curve of the ideal effective data distribution and the coordinate axis.

[0056] According to a second aspect of this application, a first storage device according to the second aspect of this application is provided, including a control component that performs one of the control methods for garbage collection of the storage device described above. Attached Figure Description

[0057] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings.

[0058] Figure 1 A block diagram of a storage device;

[0059] Figure 2 This is a schematic diagram of a large block;

[0060] Figure 3 This is a schematic diagram of the existing waste recycling process.

[0061] Figure 4 This is a schematic diagram of existing waste recycling methods;

[0062] Figure 5 A schematic diagram illustrating waste data recycling according to an embodiment of this application is shown;

[0063] Figure 6A A schematic diagram illustrating an ideal effective data distribution of a storage device according to an embodiment of this application is provided.

[0064] Figure 6B A schematic diagram illustrating the ideal effective data distribution and the actual effective data distribution of a storage device according to an embodiment of this application is provided.

[0065] Figure 6C A schematic diagram illustrating the ideal effective data distribution and the actual effective data distribution of a storage device according to yet another embodiment of this application is provided.

[0066] Figure 7 A flowchart illustrating an embodiment of this application is shown;

[0067] Figure 8A schematic diagram illustrating an ideal data distribution according to yet another embodiment of this application is shown;

[0068] Figure 9 A schematic diagram illustrating the difference between the calculated actual effective data distribution and the ideal effective data distribution according to yet another embodiment of this application is provided.

[0069] Figure 10 A schematic diagram illustrating waste data recycling according to another embodiment of this application is shown. Detailed Implementation

[0070] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0071] Figure 5 A schematic diagram illustrating waste data recycling according to an embodiment of this application is shown.

[0072] According to an embodiment of this application, the distribution of valid data on large blocks in a storage device is obtained under a continuous and stable random write load from users (referred to as steady-state random write load). Under steady-state random write load, user data write operations continue to occur, garbage collection operations continue to occur, and valid data moved by garbage collection operations is also continuously written. The write bandwidth of user data on the storage device reaches a balance with the write bandwidth of data moved by garbage collection operations, and the distribution of valid data on large blocks in the storage device also tends to be stable. This distribution of valid data on large blocks in the storage device under steady-state random write load is called the ideal valid data distribution (510).

[0073] As an example, the distribution of the proportion of effective data in a large block to the total amount of data in that block (e.g., the sum of effective data and garbage data) under steady-state random write load is used as the ideal effective data distribution. This ideal effective data distribution under steady-state random write load is measured in a laboratory environment. Alternatively, the ideal effective data distribution under steady-state random write load is measured during the use of the storage device. As an example, the distribution of the effective data in a large block under steady-state random write load is used as the ideal effective data distribution.

[0074] During the use of the storage device, the distribution of valid data on large blocks is measured at any time or periodically to obtain the actual valid data distribution (520). For example, the actual valid data distribution is the distribution of the proportion of valid data in a large block of the storage device to the total data volume of the large block (e.g., the sum of valid data and garbage data) at the time of measurement. As another example, the actual valid data distribution is the distribution of the amount of valid data in a large block of the storage device at the time of measurement.

[0075] Because the characteristics (bandwidth, randomness, etc.) of user data received during the use of storage devices may differ from steady-state random write loads, the actual effective data distribution (520) may differ from the ideal effective data distribution (510).

[0076] The difference between the actual valid data distribution (520) and the ideal valid data distribution (510) is compared and used as a basis for initiating garbage collection operations, controlling or adjusting the bandwidth provided by the storage device for users to write data and / or for valid data to be moved by garbage collection operations (530). For example, in response to the amount of valid data indicated by the actual valid data distribution (520) being greater than the amount of valid data indicated by the ideal valid data distribution (510), a garbage collection operation is initiated, or the available bandwidth provided by the storage device for moving valid data by the garbage collection operation is increased, because the amount of valid data indicated by the actual valid data distribution (520) being greater than expected means that there is insufficient free storage space in the storage device, and more free blocks need to be generated (as soon as possible) through garbage collection operations. In response to the fact that the effective data volume indicated by the actual effective data distribution (520) is not greater than the effective data volume indicated by the ideal effective data distribution (510), the available bandwidth provided by the storage device for users to write data is increased, or the ratio of the bandwidth provided by the storage device for users to write data to the bandwidth moved by the garbage collection operation is increased. This is because the effective data volume indicated by the actual effective data distribution (520) is less than expected, which means that there is sufficient free storage space in the storage device, and it is expected that more user write data will be processed (as soon as possible) to consume the free storage space.

[0077] Understandably, the available bandwidth provided by a storage device for garbage collection operations to move valid data can, but does not necessarily represent, the maximum bandwidth allowed by the storage device for garbage collection operations to move valid data. Similarly, the available bandwidth provided by a storage device for user data can be the maximum bandwidth allowed by the storage device for user data writes.

[0078] Optionally, in response to the effective data volume indicated by the actual effective data distribution (520) being greater than the effective data volume indicated by the ideal effective data distribution (510), or the difference between the effective data volume indicated by the actual effective data distribution (520) and the effective data volume indicated by the ideal effective data distribution (510) being greater than a threshold, the ratio of the available bandwidth for garbage collection operations to the available bandwidth for user data provided by the storage device is increased. In response to the effective data volume indicated by the actual effective data distribution (520) not being greater than the effective data volume indicated by the ideal effective data distribution (510), or the difference between the effective data volume indicated by the actual effective data distribution (520) and the effective data volume indicated by the ideal effective data distribution (510) being less than a threshold, the ratio of the available bandwidth for garbage collection operations to the available bandwidth for user data provided by the storage device is decreased.

[0079] According to an embodiment of this application, to compare the difference between the actual effective data distribution (520) and the ideal effective data distribution (510), the major blocks of the storage device are sorted according to the proportion of their effective data volume to their total data volume, thereby obtaining the proportion of the effective data volume of the sorted major blocks to their total data volume. For the ideal effective data distribution, there is the proportion of the effective data volume of the sorted major blocks to their total data volume; for the actual effective data distribution, there is the proportion of the effective data volume of the sorted major blocks to their total data volume. The proportion of the effective data volume of the sorted major blocks corresponding to the ideal effective data distribution to their total data volume is subtracted from the proportion of the effective data volume of the sorted major blocks corresponding to the actual effective data distribution, in the order of sorting, to obtain the difference between the effective data volume proportion of the actual effective data distribution and the effective data volume proportion of the ideal effective data distribution. The difference between the effective data volume proportion of the actual effective data distribution and the effective data volume proportion of the ideal effective data distribution is accumulated and used as the difference between the actual effective data distribution (520) and the ideal effective data distribution (510).

[0080] For example, array P represents the actual effective data distribution. The elements of array P are sorted, and each sorted element represents the proportion of effective data in its major blocks relative to the total data volume. The array index j of element P[j] indicates the sequence number of the major block represented by element P[i]. Array I represents the ideal effective data distribution. The elements of array I are sorted, and each sorted element represents the proportion of effective data in its major blocks relative to the total data volume. The array index j of element I[j] indicates the sequence number of the major block represented by element I[j]. Therefore... This represents the difference between the actual effective data distribution (520) and the ideal effective data distribution (510), where N is the number of blocks in the storage device.

[0081] In the optional example, in the calculation During the process, if for some value m of j from 1 to N, such that If the value exceeds the threshold T, a garbage collection operation is initiated, or the available bandwidth provided by the storage device for the garbage collection operation to move valid data is increased. Optionally, the threshold T can be 0 or a specified value.

[0082] Furthermore, arrays P and I are sorted in ascending order according to the amount of valid data or the percentage of valid data in their respective large blocks. However, garbage collection typically prioritizes the collection of large blocks with lower valid data quantities or percentages, in the calculation... During the process, if for some value m of j from 1 to N, such that If the value of j exceeds the threshold T, a garbage collection operation is initiated, or the available bandwidth of the storage device for moving valid data during the garbage collection operation is increased, without having to calculate the value of j from m+1 to N, thus helping to identify the need to initiate garbage collection earlier.

[0083] Figure 6A A schematic diagram illustrating the ideal effective data distribution of a storage device according to an embodiment of this application is shown.

[0084] For example, a storage device may consist of, say, 101 large blocks. See also Figure 6A The horizontal axis represents the block number of the storage device, and the vertical axis represents the proportion of valid data in the block to the total data in the block (data validity rate). All blocks on the storage device are sorted by data validity rate, thus... Figure 6A The curve representing the ideal effective data distribution is a monotonically increasing function. The largest block on the far left of the horizontal axis has the lowest data effectiveness, and as the horizontal axis coordinate increases, the data effectiveness of the corresponding large block does not decrease at all.

[0085] Figure 6A The ideal effective data distribution shown comes from the results of derivation, from laboratory measurements, or from measurements taken after a storage device has run and processed a specified workload.

[0086] Figure 6B A schematic diagram illustrating the ideal effective data distribution and the actual effective data distribution of a storage device according to an embodiment of this application is provided.

[0087] Figure 6B In the diagram, the solid curve represents the ideal effective data distribution of the storage device, while the dashed curve represents the actual effective data distribution. The actual effective data distribution is a curve plotted based on the statistical proportion (data effectiveness rate) of the effective data volume in each major block of the storage device at a current time or a certain point in time to the total data volume in that block. The dashed curve representing the actual effective data distribution is also sorted by the data effectiveness rate of the major blocks.

[0088] Figure 6B The ideal effective data distribution curve in the middle, the same as Figure 6A The ideal effective data distribution curve is the same.

[0089] Therefore, see Figure 6B The actual effective data distribution can be compared with the ideal effective data distribution.

[0090] Figure 6C A schematic diagram illustrating the ideal effective data distribution and the actual effective data distribution of a storage device according to yet another embodiment of this application is shown.

[0091] Figure 6C The image shows the ideal valid data distribution curve and two actual valid data distribution curves. The dashed curve represents actual valid data distribution 1, while the dotted-line curve represents actual valid data distribution 2.

[0092] The two actual effective data distribution curves come from, for example, the efficiency of large blocks of data on storage devices statistically analyzed at different times.

[0093] As an example, see Figure 6B To compare the actual effective data distribution with the ideal effective data distribution, the area enclosed by the dashed and solid curves is calculated. When the ordinate of the dashed curve is less than that of the solid curve, the area enclosed by the two curves is negative; when the ordinate of the dashed curve is greater than that of the solid curve, the area enclosed by the two curves is positive. Let x represent the x-axis coordinate and y represent the y-axis coordinate. Let the solid curve be y = f1(x) and the dashed curve be y = f2(x). Then the area enclosed by the two curves is... Where M represents the number of blocks in the storage device. Understandably, at different times, the storage device can have different actual effective data distributions (dashed curve), thus resulting in different areas S.

[0094] According to one embodiment of this application, if the area S is greater than 0 or a specified threshold T, a garbage collection operation is initiated, increasing the available bandwidth provided by the storage device for the garbage collection operation to move valid data, or increasing the proportion of the available bandwidth provided by the storage device for the garbage collection operation to move valid data. If the area S is less than 0 or the specified threshold, the current garbage collection operation intensity remains unchanged, or the proportion of the bandwidth provided by the storage device for the garbage collection operation to move valid data to the bandwidth for user-written data is reduced.

[0095] According to another embodiment of this application, if during the calculation of area S, there exists a value of j such that the non-final value of area S is greater than 0 or a specified threshold T, then a garbage collection operation is initiated, increasing the available bandwidth provided by the storage device for the garbage collection operation to move valid data, or increasing the proportion of the available bandwidth provided by the storage device for the garbage collection operation to move valid data.

[0096] See again Figure 6B For example, according to During the calculation of S, when the x-coordinate j = 20, the value of area S is greater than 0. Based on this, an (additional) garbage collection operation is initiated, increasing the available bandwidth of the storage device for the garbage collection operation to move valid data, or increasing the proportion of the available bandwidth of the storage device for the garbage collection operation to move valid data.

[0097] And see Figure 6C Let the solid curve be y = f1(x), the dashed curve be y = f2(x), and the dotted-dash curve be y = f3(x). Although for some values ​​of x, f3(x) > f1(x), according to... If the final result of S is less than 0, then the current garbage collection operation intensity remains unchanged, or the ratio of the bandwidth provided by the storage device for the garbage collection operation to the bandwidth for user-written data is reduced.

[0098] Figure 7 A flowchart illustrating an embodiment of this application is shown.

[0099] Storage devices record one or more ideal effective data distributions. For example, a storage device records an ideal effective data distribution under steady-state random write load.

[0100] During storage device operation, the data efficiency of each major block of the storage device is periodically or as needed (710). The major blocks of the storage device are sorted by data efficiency (720) to obtain their actual effective data distribution. The difference between the statistical data efficiency (actual data efficiency) of each major block and a specified data efficiency (ideal data efficiency) is accumulated according to the sorting order (730). If the accumulated difference between the statistical data efficiency (actual data efficiency) of each major block and the specified data efficiency (ideal data efficiency) has an intermediate cumulative result greater than a threshold (e.g., 0) (740), it means that there is too much effective data in the storage device, and the intensity of garbage collection operations needs to be increased (760). For example, initiating garbage collection operations, increasing the available bandwidth provided by the storage device for moving effective data during garbage collection, or increasing the proportion of available bandwidth provided by the storage device for moving effective data during garbage collection. In step 740, if there is no cumulative result greater than the threshold, it means that the available storage resources of the storage device are sufficient, and the intensity of garbage collection operations can be reduced (750) to allocate more resources of the storage device to serve user I / O commands. Alternatively, in addition to reducing the bandwidth of garbage collection operations, strategies such as maintaining the current intensity of garbage collection operations or reducing the ratio of the bandwidth provided by the storage device for garbage collection operations to the bandwidth for user write data can be adopted as alternatives.

[0101] Figure 8 A schematic diagram illustrating an ideal effective data distribution according to yet another embodiment of this application is shown.

[0102] See Figure 8 The horizontal axis represents the block number of the storage device, h is the number of blocks in the storage device, and the vertical axis represents the proportion of valid data in a block to the total data in that block (data validity rate). All blocks in the storage device are sorted by data validity rate. Figure 8 The curves show the ideal effective data distribution under steady-state random write load, where b represents the minimum data effectiveness of a large block of the storage device, a represents the maximum data effectiveness of a large block of the storage device, S indicates the area enclosed by the curve of the ideal effective data distribution and the coordinate axis, indicating the sum of the storage space occupied by the effective data of each large block of the storage device under steady-state random write load, while the shaded area indicates the sum of the storage space occupied by invalid data or the storage space of data that has not been written in the storage device.

[0103] according to Figure 8 ah represents the sum of the storage space of all the major blocks of the storage device. Furthermore, the analytical expression for the curve of the ideal effective data distribution is obtained. And through curve fitting, we obtain

[0104] Under steady-state random write load, S, a, and b are measured, while h can be obtained from the storage device configuration. This allows us to determine the data efficiency of the large blocks with the ideal effective data distribution.

[0105] Figure 9 A schematic diagram illustrating the difference between the calculated actual effective data distribution and the ideal effective data distribution according to yet another embodiment of this application is shown.

[0106] In some cases, the number of large blocks in the storage device is large, leading to computational... When x has too many possible values, the computation time becomes longer and the computational cost becomes larger.

[0107] Figure 9 In the diagram, the solid curve represents the ideal effective data distribution of the storage device, while the dashed curve represents the actual effective data distribution. According to... Figure 9 The illustrated embodiment accumulates the difference between the actual effective data distribution and the ideal effective data distribution along the vertical axis. For example, the data effectiveness represented by the vertical axis is divided into N intervals, where N is, for example, equal to 10, 128, or other values, to quantify the data effectiveness into N values. As an example, Figure 9 In this context, N = 10, and the N intervals are indicated by (d10, d20, d30, d40, d50, d60, d70, d80, d90 and d100 respectively).

[0108] Based on the actual effective data distribution, the number of large blocks within each interval (corresponding to each value of the data effectiveness rate) is obtained; based on the ideal data effectiveness rate, the number of large blocks within each interval (corresponding to each value of the data effectiveness rate) is also obtained. Therefore, each interval on the vertical axis contains the number of large blocks from the actual effective data distribution and the number of large blocks from the ideal effective data distribution. Based on the difference between these two numbers and the data effectiveness rate value of the interval, the difference in effective data volume between the actual and ideal effective data distributions within each interval is obtained (denoted as Sd). Accumulating the effective data volume differences across intervals yields the difference between the actual and ideal effective data distributions of the storage device (S = ∑ d Sd, where d takes values ​​of d10, d20, d30, d40, d50, d60, d70, d80, d90, and d100. The intensity of garbage collection operations is then adjusted based on the difference between the actual effective data distribution and the ideal effective data distribution of the storage device.

[0109] Figure 10 A schematic diagram illustrating waste data recycling according to another embodiment of this application is shown.

[0110] According to embodiments of this application, an ideal effective data distribution of the storage device under various workloads is obtained, such as the ideal effective data distribution of sequential writes under a sequential write workload, the ideal effective data distribution of random writes under a random write workload, and the ideal effective data distribution of mixed writes under a mixed write workload (sequential writes and random writes are mixed in a specified ratio). Again, as an example, a stream ideal effective data distribution (1010) is obtained for one or more streams received by the storage device.

[0111] During the use of the storage device, the distribution of valid data on large blocks is measured continuously or periodically to obtain the actual valid data distribution (1020). It also obtains characteristics of the current load on the storage device, such as whether the current load is sequential write, random write, or mixed write, or identifies one or more streams currently received by the storage device.

[0112] Based on the characteristics of the current load of the storage device, an ideal effective data distribution corresponding to the current load characteristics is obtained. The difference between the actual effective data distribution (1020) and the ideal effective data distribution corresponding to the current load characteristics (1010) is compared, and this difference is used as the basis for initiating garbage collection operations, controlling or adjusting the bandwidth provided by the storage device for user data writing and / or effective data moved by garbage collection operations (1030).

[0113] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application. Clearly, those skilled in the art can make various alterations and variations to this application without departing from its spirit and scope. Thus, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method for controlling garbage collection of a storage device, comprising the steps of: comparing a difference between an actual valid data distribution and an ideal valid data distribution; initiating a garbage collection operation or adjusting an available bandwidth provided for user write data and / or an available bandwidth provided for valid data moved by the garbage collection operation according to the difference; further comprising: obtaining a characteristic of a current load of the storage device; obtaining an ideal valid data distribution corresponding to the characteristic of the current load of the storage device to compare the difference between the actual valid data distribution and the ideal valid data distribution according to the characteristic of the current load of the storage device; wherein the ideal valid data distribution is a distribution of a proportion of valid data amount to total data amount of a large block of the storage device under a steady-state random write load, or the ideal valid data distribution is a distribution of valid data amount of the large block of the storage device under the steady-state random write load; the characteristic of the current load of the storage device is a sequential write load, a random write load, a mixed write load, and / or one or more stream loads received by the storage device, and the mixed write load is a load mixed by the sequential write and the random write in a specified proportion.

2. The control method of garbage collection of a storage device according to claim 1, wherein, in response to the actual valid data amount indicated by the actual valid data distribution being greater than the actual valid data amount indicated by the ideal valid data distribution, or a difference between the actual valid data amount indicated by the actual valid data distribution and the actual valid data amount indicated by the ideal valid data distribution being greater than a threshold value, initiating the garbage collection operation, or increasing the available bandwidth provided for moving the valid data by the garbage collection operation, or increasing a proportion of the available bandwidth provided for moving the valid data by the garbage collection operation to the available bandwidth provided for the user write data.

3. The control method of garbage collection of storage devices according to claim 1, wherein, in response to the actual valid data amount indicated by the actual valid data distribution not being greater than the actual valid data amount indicated by the ideal valid data distribution, or the difference between the actual valid data amount indicated by the actual valid data distribution and the actual valid data amount indicated by the ideal valid data distribution being less than the threshold value, increasing the available bandwidth provided for the user write data, or increasing a proportion of the available bandwidth provided for the user write data to the available bandwidth provided for moving the valid data by the garbage collection operation.

4. The control method of garbage collection of storage devices according to claim 3, wherein, the available bandwidth provided for the user write data is an upper limit value of a bandwidth allowed for the user write data, and the available bandwidth provided for moving the valid data by the garbage collection operation is an upper limit value of a bandwidth allowed for moving the valid data by the garbage collection operation. 5.The method according to claim 1, further comprising: measuring the valid data distribution on the large block of the storage device to obtain the actual valid data distribution.

6. The control method of garbage collection of storage devices according to claim 5, wherein, measuring the valid data distribution on the large block of the storage device at any time or periodically to obtain the actual valid data distribution.

7. The control method of garbage collection of a storage device according to claim 1, wherein, the steady-state random write load is a continuous and stable random write load.

8. The control method of garbage collection of a storage device according to any one of claims 1-7, wherein, the ideal valid data distribution of the storage device under the steady-state random write load is measured in a laboratory environment, or the ideal valid data distribution of the storage device under the steady-state random write load is measured in a process of using the storage device. 9.The method according to claim 1 or 2, further comprising the steps of: sorting data valid rates of each large block of the storage device to obtain a sorted actual valid data distribution; subtracting the sorted ideal effective data distribution from the sorted actual effective data distribution in order to obtain the difference between the actual effective data distribution and the ideal effective data distribution of each block, wherein the sorted ideal effective data distribution is obtained by sorting the data efficiency of each block of the storage device under steady-state random write load; accumulating the difference between the actual effective data distribution and the ideal effective data distribution of each block to obtain the difference between the actual effective data distribution and the ideal effective data distribution.

10. The control method of garbage collection of storage devices according to claim 9, wherein, The data efficiency is the ratio of the amount of valid data to the total amount of data of each block of the storage device.

11. The control method of garbage collection of a storage device according to claim 1 or 2, wherein, By obtaining a difference between the actual effective data distribution and the ideal effective data distribution; wherein N is the number of blocks of the storage device; array P represents the actual effective data distribution, and each element of array P is sorted and indicates the ratio of the amount of valid data to the total amount of data of each block in order; the array subscript j of element P[j] indicates the serial number of the block represented by element P[j]; array I represents the ideal effective data distribution, and each element of array I is sorted and indicates the ratio of the amount of valid data to the total amount of data of each block in order; the array subscript j of element I[j] indicates the serial number of the block represented by element I[j].

12. The control method of garbage collection of a storage device according to claim 11, wherein, In computing the process, if for some value m of j from 1 to N, the value of is greater than a threshold T, then initiate a garbage collection operation, or increase the available bandwidth provided to the garbage collection operation to move valid data, or increase the ratio of the available bandwidth provided to the garbage collection operation to move valid data to the available bandwidth of user writes.

13. The control method of garbage collection of storage devices according to claim 12, wherein, The threshold value T is 0 or a specified value.

14. The control method of garbage collection of a storage device according to claim 13, wherein, The array P and the array I are sorted in ascending order of the corresponding chunk's valid data amount or the ratio of the valid data amount; in the process of calculating If, for some value m of j from 1 to N, the value of is greater than a threshold T, a garbage collection operation is initiated, or the available bandwidth provided to the garbage collection operation for moving valid data is increased, and there is no need to calculate the value of j from m+1 to N.

15. The control method of garbage collection of the storage device according to claim 1 or 2, further comprising the steps of: sorting the data efficiency of each block of the storage device to draw a curve of the actual effective data distribution; calculating the area of the part enclosed by the curve of the actual effective data distribution and the curve of the ideal effective data distribution, wherein the area of the part enclosed by the curve of the actual effective data distribution and the curve of the ideal effective data distribution represents the difference between the actual effective data distribution and the ideal effective data distribution.

16. The control method of garbage collection of storage devices according to claim 15, wherein, If the curve of the ideal effective data distribution is y=f1(x) and the curve of the actual effective data distribution is y=f2(x), then the area of the part enclosed by the curve of the actual effective data distribution and the curve of the ideal effective data distribution is where M is the number of the large blocks of the storage device, x is the serial number of the large blocks of the storage device, and 1≤g≤M.

17. The control method of garbage collection of the storage device according to claim 15, wherein, obtaining the number of actual blocks in each data efficiency interval according to the curve of the actual effective data distribution; obtaining the difference between the actual data efficiency distribution and the ideal data efficiency distribution corresponding to the amount of valid data in each interval according to the difference between the number of actual blocks from the actual effective data distribution and the number of ideal blocks from the ideal effective data distribution in the data efficiency interval and the data efficiency value in the interval; accumulating the amount of valid data in each interval to obtain the difference between the actual effective data distribution and the ideal effective data distribution of the storage device.

18. A storage device comprising a control component, wherein the control component performs one of the control methods of garbage collection of the storage device according to any one of claims 1-17.

Images