Data storage method and related device

Abstract

The invention discloses a data storage method and a related device, which can be applied to the technical field of storage. The method comprises the steps that when the load of the storage system is low, data received by the storage system is stored in a small check group mode, and the small check group comprises a data block and a check block; when the load of the storage system is high, data received by the storage system is stored in a large verification group mode, the large verification group comprises data blocks and verification blocks, and the number of the data blocks in the large verification group is larger than that of the data blocks in the small verification group. The problem of extra overhead caused by zero padding due to the fact that the data cannot be fully split is avoided, write amplification is reduced, and the performance and the service life of the disk are improved.

Description

Technical Field

[0001] The present application relates to the field of storage technology, and in particular to a data storage method and related devices. Background Art

[0002] In the storage media of some storage systems, such as solid state disks (SSDs), a redirection on write (ROW) mechanism is usually used to write new data. That is, when the storage system receives new data, the new data will be written to the blank blocks of the storage system, and the original data will be marked as garbage data waiting to be recycled.

[0003] When the storage system receives data, if the data cannot fill a complete stripe, it needs to fill the unfilled part of the stripe with zeros to generate verification data. However, the above operation will increase write amplification and cause additional space overhead. Summary of the invention

[0004] The present application provides a data storage method and related device, which can reduce write amplification.

[0005] In a first aspect, the present application provides a data storage method, which can be applied in the field of storage technology and mainly includes:

[0006] When the load of the storage system is low, the data received by the storage system is stored in a small check group, wherein the small check group includes data blocks and check blocks; when the load of the storage system is high, the data received by the storage system is stored in a large check group, wherein the large check group includes data blocks and check blocks, and the number of data blocks in the large check group is greater than the number of data blocks in the small check group.

[0007] In this application, the check group is dynamically adjusted according to the load of the storage system. When the load of the storage system is low, a small check group is used to store data, which avoids the problem of extra overhead caused by the need to fill zeros when the data cannot fill the stripe, reduces write amplification, and improves the overall performance and life of the storage system. When the load of the storage system is high, a large check group is used to store data to improve disk utilization. The data storage method is dynamically adjusted according to the load of the storage system, further improving the performance of the storage system and reducing system overhead.

[0008] In a possible implementation manner of the first aspect, the storage system stores data in a write redirection ROW manner.

[0009] When the storage system receives data, it stores the data in a write redirection manner. When the storage system load is low, if the data cannot fill up the full stripe of the disk, it will generally fill up the stripe by padding with zeros to generate verification data, which increases write amplification and causes additional overhead. In this implementation, the storage system dynamically adjusts the number of stored data blocks based on the received data, avoiding invalid zero padding and reducing write amplification.

[0010] In a possible implementation manner of the first aspect, storing data received by a storage system includes:

[0011] The storage system sends the data in the small check group and the large check group to a local or remote storage for storage, and different storages store different blocks.

[0012] In this implementation, the storage system uses a memory to store data, and each block in the data block and the check block corresponds to a different memory, so that the data is stored in the different memories.

[0013] In a possible implementation manner of the first aspect, the method further includes:

[0014] When performing garbage collection on the memory, the storage system migrates valid data stored in the memory in the small check group according to the large check group.

[0015] In this implementation, when garbage collection is performed on the memory, the storage system uses a large check group to migrate valid data stored in a small check group in the memory, thereby restoring the capacity utilization of the memory.

[0016] In a possible implementation manner of the first aspect, the number of check blocks in the large check group and the number of check blocks in the small check group are the same.

[0017] In this implementation, regardless of whether the storage system load is high or low, it is assumed that the number of generated check blocks is the same, so that the number of data blocks is dynamically adjusted according to the received data, and different numbers of data blocks are selected for data storage, avoiding the problem of additional overhead caused by the need to fill zeros when the data cannot fill the stripe, and reducing write amplification.

[0018] In a possible implementation manner of the first aspect, the storage system is a centralized storage device or a storage server.

[0019] In this implementation, the storage system is suitable for centralized distribution scenarios, and the storage system is a centralized storage device or a storage server.

[0020] In a possible implementation manner of the first aspect, the storage system is a distributed storage system, or a distributed storage device constituting a distributed storage system.

[0021] In this implementation, the storage system is suitable for distributed scenarios, and the storage system is a distributed storage system, or a distributed storage device in a distributed storage system.

[0022] In a second aspect, the present application provides a data storage device, and the beneficial effects can be found in the description of the first aspect, which will not be repeated here. The device has the function of implementing the behavior in the method example of the first aspect. The function can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions. In a possible design, the device includes:

[0023] A first storage module, used for selecting to store the data received by the data storage device in a small check group when the load of the data storage device is low, the small check group including a data block and a check block;

[0024] The second storage module is used to store the data received by the data storage device in a large check group when the load of the data storage device is high, wherein the large check group includes data blocks and check blocks, and the number of data blocks in the large check group is greater than the number of data blocks in the small check group.

[0025] In a possible implementation manner of the second aspect, the data storage device stores data in a write redirection ROW manner.

[0026] In a possible implementation manner of the second aspect, storing data received by a data storage device includes:

[0027] The data storage device sends the data in the small check group and the large check group to a local or remote memory for storage, and different memories store different blocks.

[0028] In a possible implementation manner of the second aspect, the method further includes:

[0029] When performing garbage collection on the memory, the data storage device migrates valid data in the memory according to the large check group.

[0030] In a possible implementation manner of the second aspect, the number of check blocks in the large check group and the number of check blocks in the small check group are the same.

[0031] In a possible implementation manner of the second aspect, the data storage device is a centralized storage device or a storage server.

[0032] In a possible implementation manner of the second aspect, the data storage device is a distributed storage system, or a distributed storage device constituting a distributed storage system.

[0033] In a third aspect, the present application provides a controller, characterized in that it includes a memory and a processor, the memory stores data and programs that can be run on the processor, and when the processor executes the program, the controller implements the method in the above-mentioned first aspect or any possible implementation method of the first aspect.

[0034] In a fourth aspect, the present application provides a storage system, comprising a plurality of storage devices, characterized in that when the load of the storage system is low, the received data is selected to be stored in the storage device of the storage system in a small check group, wherein the small check group includes a data block and a check block;

[0035] When the load of the storage system is high, the received data is stored in the storage device of the storage system in the form of a large check group. The small check group includes data blocks and check blocks. The number of data blocks in the large check group is greater than the number of data blocks in the small check group.

[0036] In a fifth aspect, a storage system is provided, wherein each storage device manages multiple memories, wherein the storage system is used for:

[0037] The blocks of the same check group are stored in multiple storage devices, and different storage devices store different blocks; or,

[0038] The blocks of the same check group are stored in multiple memories, and different memories store different blocks.

[0039] In a sixth aspect, the present application provides a computer program product, which, when executed by a computer, implements the method in the above-mentioned first aspect or any possible implementation manner of the first aspect.

[0040] In a seventh aspect, the present application provides a chip system, which includes a processor for implementing the method in the first aspect or any possible implementation of the first aspect. In one possible design, the chip system also includes a memory for storing program instructions and / or data. The chip system can be composed of a chip, or can include a chip and other discrete devices.

[0041] The solutions of the second to seventh aspects mentioned above are used to implement or cooperate with the method of the first aspect or any possible implementation method thereof, and therefore can achieve the same or corresponding beneficial effects as the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1a A schematic diagram of a storage system provided in an embodiment of the present application;

[0043] Figure 1b A structural example diagram of a controller in a storage system provided in an embodiment of the present application;

[0044] Figure 1c A schematic diagram of a flow chart of a data storage method provided in an embodiment of the present application;

[0045] Figure 2 A schematic diagram of the structure of a RAID5 disk array provided in an embodiment of the present application;

[0046] Figure 3 A flowchart of a currently used data storage method;

[0047] Figure 4a Another schematic diagram of a data storage method provided in an embodiment of the present application;

[0048] Figure 4b A schematic diagram of a process for storing received data in a small verification group manner provided in an embodiment of the present application;

[0049] Figure 4c A schematic diagram of a process for garbage collection on a solid state drive provided in an embodiment of the present application;

[0050] Figure 5 A schematic diagram of a structure of a data storage device provided in an embodiment of the present application;

[0051] Figure 6 A schematic diagram of the structure of a controller provided in an embodiment of the present application. DETAILED DESCRIPTION

[0052] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of this application. It is known to those skilled in the art that with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0053] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or modules is not necessarily limited to those steps or modules that are clearly listed, but may include other steps or modules that are not clearly listed or inherent to these processes, methods, products or devices.

[0054] The term "and / or" in this application can be a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this application generally indicates that the associated objects before and after are in an "or" relationship.

[0055] It should also be noted that, in some alternative implementations, the functions / acts noted may occur out of the order of the drawings. For example, two figures shown in succession may in fact occur substantially simultaneously or may sometimes be performed in the reverse order, depending on the functions / acts involved.

[0056] In the embodiments of the present application, unless otherwise specified, the meaning of "at least one" refers to one or more, and the meaning of "plurality" refers to two or more. It is understood that in the present application, "when", "if" and "if" all refer to the device making corresponding processing under certain objective circumstances, and do not limit the time, nor do they require that there must be a judgment action when the device is implemented, nor do they mean that there are other limitations. In addition, the special word "exemplary" means "used as an example, embodiment or illustrative". Any embodiment described as "exemplary" is not necessarily interpreted as being superior or better than other embodiments.

[0057] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0058] First, in order to better understand the solution of the embodiment of the present application, the application scenario of the embodiment of the present application is described below. The data storage method provided in the embodiment of the present application is mainly applied to a storage system to extend the availability of a service.

[0059] Combine the following Figure 1a and Figure 1b , an example is given to illustrate the application scenario of the data storage method provided in the embodiment of the present application. Figure 1a , Figure 1a A schematic diagram of a storage system provided in an embodiment of the present application is shown in FIG.

[0060] The storage system 100 is applicable to a storage system using a redirection on write (ROW) mechanism, including a centralized storage system, a distributed storage system, etc. Optionally, the storage system may be a centralized storage device or a storage server. Optionally, the storage system 100 may also be a distributed storage system, or a distributed storage device in a distributed storage system.

[0061] The controller 11 is connected to a host (not shown in the figure). The controller can be integrated with the host in the same central processing unit (CPU). The controller 11 can be a computing device, such as a server, a desktop computer, etc. A writing system and an application are installed on the controller 11. The controller 11 can receive input / output (I / O) requests from the host. The controller 11 can also store the data carried in the I / O request and write the data to the disk 12. The controller 11 and the disk 12 can be connected locally or remotely, including wired connections and wireless connections. When the disk 12 and the controller 11 are locally connected, the disk 12 and the controller 11 are located in the same device (for example, the same physical storage device, or the same physical storage server), and the controller 11 and the disk 12 are connected through a peripheral component interconnect express (PCIe) bus, or through a serial advanced technology attachment (SATA) interface, a serial small computer system interface (SAS), and an M.2 interface. When connected remotely, the disk 12 and the controller 11 are not in the same device, for example, the controller 11 is connected to the external disk 12 through a built-in network card. The disk 12 here refers to a persistent storage device that can retain data for a long time after power failure, such as a hard disk drive (HDD), a solid state disk or solid state drive (SSD), a phase change memory (PCM), etc. For the controller 11, sending data to the disk 12 executes the storage function of the controller 11.

[0062] It should be noted that Figure 1a This is merely an exemplary description, and the disk 12 is only one example of a storage device connected to the controller 11. Figure 1aThe shapes of the controller 11 and disk 12 shown in the figure do not represent the actual physical structures of the controller 11 and disk 12, and are not limited here. In addition, the implementation of this application does not limit the connection method between the controller and the disk 12, and only needs to ensure that the controller 11 and the disk 12 can communicate with each other. In actual applications, the storage system 100 may include two or more controllers, and the physical structure and function of each controller are similar to those of the controller 11, and this embodiment does not limit the connection method between the controllers, and between any controller and the disk 12. As long as the controllers and the controllers and the disks 12 can communicate with each other. In addition, in this embodiment, when the controller 11 sends instructions to the disk 12, it often sends them to multiple disks 12. For the convenience of description, the set including multiple disks 12 is called a disk group.

[0063] See also Figure 1b , Figure 1b This is a structural example diagram of a controller in a storage system provided in an embodiment of the present application. Figure 1b As shown, the controller 11 includes an interface card 110 , a memory 111 , a processor 112 , and an interface card 113 .

[0064] The interface card 110 is used to communicate with the host and receive instructions sent by the host. The controller 11 can receive write instructions from the host through the interface card 110 .

[0065] The processor 112 is a central processing unit. In the embodiment of the present application, the processor 112 can be used to receive a write instruction or a read instruction from the host and process these instructions. The processor 112 can also send the data in the write instruction to the disk 12.

[0066] Optionally, the controller 11 may further include a memory 111 .

[0067] The memory 111 includes a volatile memory, such as a random-access memory (RAM).

[0068] Optionally, the controller 11 may further include an interface card 113 .

[0069] The interface card 113 is used to communicate with the disk 12. The controller 11 can send a write data instruction to the disk 12 for storage through the interface card 113.

[0070] Understandably, Figure 1a and Figure 1b The structure of the storage system shown is only an example. The data storage method provided in the embodiment of the present application can be applied to the structure described above but is not limited to it, and is not limited here.

[0071] The embodiment of the present application provides a storage system, which includes a controller and multiple disks. The controller and the multiple disks can communicate with each other through an internal connection path to transmit control and / or data signals. The controller stores stripe data information. The stripe data information is used to record the status of each stripe unit in the disk group. The controller can execute the received data processing instruction based on the stored stripe data information.

[0072] It can be understood that the present application is applicable to any storage system that adopts the ROW mechanism. In order to simplify the description, the present application uses redundant arrays of independent disks (RAID) to represent a disk group formed by combining multiple independent physical disks in different ways, which has the same meaning as the commonly mentioned RAID group and RAID array and is not limited here.

[0073] based on Figure 1a In the storage system shown, when the storage system only supports large stripes, if the data received by the storage system in a short period of time cannot fill all the stripes of the disk, it is necessary to fill all the unfilled parts of the disk with zeros, which causes additional overhead and increases write amplification.

[0074] A stripe is a type of parity group. Each stripe consists of multiple stripe units. A stripe includes both data stripe units and parity stripe units. A large stripe is a stripe with a large number of stripe units, and a small stripe is a stripe with a small number of stripe units.

[0075] When the storage system stores data in persistent storage media (such as SSDs or HDDs), it uses a complete stripe (also called a "full stripe") as the minimum unit. For example, when the stripe granularity is 20+2, 20 data stripe units need to be gathered, and then the 2 check stripe units corresponding to the 20 data stripe units are calculated to form a "full stripe".

[0076] Assume that the amount of IO data currently received by the storage system is small and is only enough to generate 8 data stripe units. At this time, the storage system has two solutions: (1) continue to wait for the arrival of new data until 20 data stripe units are collected; (2) use "0" to fill the remaining 12 data stripe units so that 20 data stripe units can be collected as soon as possible. In the former case, these 8 data stripe units will be temporarily stored in the memory during the waiting process and will not be sent to the disk. Although the latter can make these 8 data stripe units sent to the disk as soon as possible, the additional 12 data units of all 0 are also sent to the disk for storage, wasting the disk storage space.

[0077] If the stripe size is not 20+2 but 8+2, then a "full stripe" can be assembled in a short time by relying on the user's IO, and the data can be sent to the disk as soon as possible without adding "0". Therefore, when the storage system uses large stripes, the utilization rate of the storage medium is higher, but when the storage system receives a small amount of data in a short time, there is a problem that the data cannot be sent to the disk as soon as possible or the disk utilization rate is low.

[0078] In order to solve the above problems, the present application provides a data storage method. Figure 1a For details, please refer to Figure 1c , Figure 1c A flow chart of a data storage method provided for an embodiment of the present application. The method specifically includes: S1. When the load of the storage system is low, the data received by the storage system is stored in a small check group, and the small check group includes data blocks and check blocks. S2. When the load of the storage system is high, the data received by the storage system is stored in a large check group, wherein the large check group includes data blocks and check blocks, and the number of data blocks in the large check group is greater than the number of data blocks in the small check group. The check group here refers to a RAID group, an error correction code (EC) group, or a similar technology. When the data of some blocks in the same check group is damaged, as long as the number of damaged blocks does not exceed the upper limit of the check (the upper limit is usually the amount of data in the check block), the remaining blocks can be used to recover the damaged blocks.

[0079] The embodiment of the present application uses "load" to describe the amount of data received by the storage system (or storage device, storage equipment) in a short period of time. When a large amount of data is received in a short period of time (this large amount of data can be from one IO request or from multiple IO requests, without limitation), it means that a large check group can be put together in a short period of time, and the executor of the data storage method is under high load at this time; conversely, when a small amount of data is received in a short period of time, it means that it is difficult to put together a large check group in a short period of time, and the executor of the data storage method is under low load at this time. The specific high and low load settings can be set by the user based on experience or determined by AI training. For example, when more than 100MB of data is received within 10 seconds, it is defined as high load; when less than 100MB of data is received within 10 seconds, it is defined as low load. Alternatively, when the CPU utilization of the storage device processing IO reaches 50%, it is defined as high load; otherwise it is defined as low load.

[0080] It is understandable that, at present, N and M in the disk group N+M are fixed, such as the common 23 (number of copies of data blocks) + 2 (number of copies of check blocks). These data blocks are stored in 25 disks respectively. If any two disks fail, the other 23 disks can be used for data recovery. However, if the actual data blocks cannot be gathered to 23 in a short time, it is necessary to fill the 23 data columns by adding 0s, and generate 2 check blocks of 23 data blocks to form a complete check group. After forming a check group, the data of the check group can be sent to the disk for storage (download), and these 0s will also be stored from the cache to the disk, occupying the disk space, resulting in a waste of disk space.

[0081] Correspondingly, in the present application, when the load of the storage system is low and the data received by the storage system cannot fill the stripe, the number of columns of the logical block group written to the disk can be dynamically adjusted according to the size of the received data, supporting different Ns and adaptively selecting different redundancy ratios. For example, 4+2, 10+2, and 23+3 are supported at the same time. When the load of the storage system is low, a small check group is used to store the received data, such as using 4+2, without worrying about the problem of not being able to fill up 23 data blocks. When the load of the storage system is high, a large check group is used to store the received data. This avoids the problem of additional overhead caused by the need to fill zeros when the data cannot fill the stripe, reduces write amplification, and improves the overall performance and life of the disk.

[0082] Optionally, after the data is written to the disk, when performing garbage collection on the SSD, the storage system can migrate valid data in the SSD according to the large check group, restore the disk group to the maximum capacity, and make up, for example, 23 data blocks, and directly use 23+2, thereby restoring the capacity utilization of the disk group and reducing the occurrence of zero padding. In addition, the data on the disk is more based on the user's real data, and the data on the disk is less, which means that write amplification is reduced.

[0083] The above introduces the application scenarios of the data storage method provided in the embodiments of the present application. The following will describe in detail the implementation process of the data storage method provided in the embodiments of the present application.

[0084] First of all, in order to better understand the solutions of the embodiments of the present application, the relevant terms and concepts that may be involved in the embodiments of the present application are introduced below.

[0085] (1) Redundant Array of Independent Disks

[0086] Redundant array of independent disks is a technology that combines multiple independent physical disks in different ways to form a disk group, thereby providing higher storage performance than a single disk and providing data backup. According to different data organization methods, commonly used RAIDs include RAID0, RAID1, RAID5, RAID6, RAID10, etc. Different RAID levels can provide various levels of performance and reliability, which can ensure that in most cases, when one or more disks fail, the data of the failed disk can be restored by using the data in the remaining member disks using the algorithm corresponding to the RAID level to ensure that data is not lost.

[0087] The configuration strategy of RAID5 and above needs to be configured as N+M mode, where N and M are both positive integers greater than 1, N represents the number of data disks, and M represents the number of check disks. Optionally, a hot spare disk is also configured in RAID. When a disk failure occurs in RAID, the RAID controller can restore the data on the failed disk to the hot spare disk based on the check data in the check disk and the data in the data disk, thereby improving system reliability.

[0088] In the embodiment of the present application, RAID 5 is used as an example for explanation. It should be understood that in actual application, RAID corresponds to multiple levels of N+M, and the present application does not limit it.

[0089] RAID 5 uses the data check bits to ensure data security. RAID 5 can store the data check bits on a single disk, or store the check bits of data segments alternately on each disk. The data storage structure in the RAID5 disk array is as follows: Figure 2 shown. Figure 2 This is a schematic diagram of the structure of the RAID5 disk array provided in an embodiment of the present application. Figure 2 The figure shows a RAID5 disk array consisting of four disks. Each cylinder represents a disk. P (parity) represents the check bit, and D (data) represents the data segment. The check data is calculated by XOR. Figure 2 In the example shown, the XOR result of every three data segments is used as the check bit of these three data segments. For example, the check bit P1 is the XOR result of data segments D1, D2, and D3, that is, P1 = D1⊕D2⊕D3. In this way, if any disk is damaged, the damaged data can be reconstructed based on the check bits and data segments on other disks.

[0090] According to the RAID5 technical principle described above, when one disk fails in the disk array, the data on the failed disk can be restored by reconstruction; however, when two disks fail, the RAID5 disk array fails, and no read or write operations can be performed on the remaining normal disks of the RAID5 disk array, especially data cannot be written, which may cause serious data loss accidents. Even if the data of the RAID5 disk is restored later, the business data during the RAID5 failure period will be lost because it was not written to the RAID in time.

[0091] It should be noted that the N+1 configuration of RAID5 logically means that it includes N data disks and 1 check disk. However, whether the check data is fixed on 1 check disk or rotated among multiple disks can be set according to actual needs, and this application does not limit it.

[0092] (2) Garbage Collection

[0093] Garbage Collection (GC) refers to moving valid data in a logical block group to a new logical block group, and then erasing the valid data and invalid data in the logical block group to form a new blank logical block group for storing other new data.

[0094] The above introduces the professional terms provided in the embodiments of the present application. In order to better understand the technical solutions provided in the embodiments of the present application, the currently used data storage methods are introduced.

[0095] See also Figure 3 , Figure 3 The following is a flowchart of the currently used data storage method. Figure 3 As shown, the process mainly includes the following steps 301 to 304.

[0096] Step 301, obtaining data of a write request.

[0097] In this step, the host sends data to the cache, and then the cache writes the data to the RAID. Due to overwriting, garbage may be generated.

[0098] Step 302, group the data into fixed N parts, and generate M parts of verification data based on the N parts of data.

[0099] Step 303, write N copies of data and M copies of verification data to the disk.

[0100] Step 304, move valid data on the disk and perform garbage collection.

[0101] The current data storage methods have the following problems:

[0102] (1) When the system load is low, fewer write requests are received. At this time, if the data cannot be filled into the stripe or N columns, it is necessary to perform a write-zero operation on the logical blocks where no data is written, resulting in additional overhead. In addition, writing zeros will increase redundant write operations to the disk, waste the life of the disk, increase write amplification, and affect the overall performance and life of the storage system.

[0103] (2) When the system load is low, if a large number of zeros are written, the amount of garbage data is high. If garbage collection is performed immediately at this time, it is equivalent to writing multiple times, wasting the life of the disk and increasing system overhead.

[0104] In order to solve the above problems, the present application provides a data storage method. For ease of understanding, the data storage method provided by the embodiment of the present application is specifically described below in conjunction with the accompanying drawings and application scenarios. It is known to those skilled in the art that with the development of technology and the emergence of new scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0105] See also Figure 4a , Figure 4a Another flow chart of the data storage method provided in the embodiment of the present application is shown in FIG. Figure 1a The storage system shown in the figure is executed, specifically, by a controller inside the storage system. Figure 4a As shown, the controller can execute the following steps 401 to 404 to avoid the problem of additional overhead caused by the need to fill zeros when data cannot fill the stripe, thereby reducing write amplification and improving the overall performance and life of the disk.

[0106] Step 401: The controller counts the load of the storage system.

[0107] In the embodiment of the present application, the controller receives a write request instruction issued by a user within a specific time period, and counts the size of the received write request data.

[0108] In step 402, the controller determines whether the load of the storage system is high or low. If it is low, the controller executes step 403. If it is high, the controller executes step 404.

[0109] In the embodiment of the present application, the controller adaptively selects a corresponding method to store the received data by judging the load of the storage system. The load mainly refers to whether the data received by the storage system in a short period of time can fill up a large stripe. If the load is low, the data cannot fill up the large stripe in a short time, and the controller chooses to store the received data in a small check group. If the load is high, the data can fill up the stripe in a short time, and the controller chooses to store the received data in a large check group to avoid zero padding.

[0110] Optionally, the controller may also determine the load level according to the number of data blocks and check blocks generated corresponding to the received data.

[0111] Specifically, after receiving the data, the controller calculates the number of data blocks and check blocks according to the size of the data. If the controller determines that the number of data blocks and check blocks is less than the total data columns or number of disks of the storage system, it means that the received data cannot fill the stripe, and the controller can determine that the load of the storage system is low. If the controller determines that the number of data blocks and check blocks corresponding to the received data can fill the stripe, the controller can determine that the load of the storage system is high.

[0112] It is understandable that in actual applications, the load of the storage system can also be set according to actual needs. It can be stipulated whether the stripes can be fully assembled within a short period of time, or it can be judged according to a set threshold or other methods, which are not limited here.

[0113] Step 403: The controller chooses to store the received data in a small check group manner.

[0114] In the embodiment of the present application, when the load of the storage system is low, the controller in the storage system chooses to store the data received by the storage system in a small check group, so as to dynamically adjust the number of logic blocks in the disk according to the received data to avoid invalid write zero operations. The logic block includes a data block and a check block.

[0115] For an alternative implementation, see Figure 4b , Figure 4b A schematic diagram of a flow chart of storing received data in a small check group provided in an embodiment of the present application. The step of the controller selecting to store received data in a small check group may include the following steps 4031 to 4032.

[0116] Step 4031: The controller determines the number of data blocks and check blocks according to the size of the data.

[0117] In an embodiment of the present application, after receiving the size of the data, the controller can generate the number n of data blocks according to the size of the data, assemble n data blocks, and generate m copies of verification data to write to the disk to avoid write amplification.

[0118] Optionally, the controller may perform weighted calculation on the size of all received data, and then determine the number of data blocks and check blocks according to the size of the data after the weighted calculation.

[0119] Optionally, the controller determines the number of data blocks and check blocks according to the size of the data, including:

[0120] The controller obtains the size of a single logic block in the logic block group, where the logic block includes a data block and a logic block;

[0121] The controller determines the number of copies of the data block to be written to the disk according to the size of the data and the size of a single logical block;

[0122] The controller obtains the number of copies of the check block according to the number of copies of the data block.

[0123] In this implementation, after performing weighted calculation on the size of all received data, the controller can determine the number of data blocks to be written on the disk according to the size of the weighted data and the size of a single logical block in the logical block group. The logical block includes a data block and a check block. After the controller obtains the number of data blocks on the disk, the controller can calculate the number of corresponding check blocks on the disk according to the XOR relationship between the data blocks.

[0124] It is understandable that the size of the data received by the controller in a specific time period is constantly changing according to the load. For the convenience of calculation, the controller can perform historical weighted calculation on the size of the received data to obtain an average value of the size of the data.

[0125] Optionally, the controller obtains the number of data blocks to be written to the disk according to the quotient of the size of the data and the size of a single logical block.

[0126] Step 4032: The controller sends the data to a local or remote memory for storage according to the number of data blocks and check blocks.

[0127] In the embodiment of the present application, different memories store different blocks, and the memory may be, for example, an SSD, etc. The controller may determine the number n of data blocks and the number m of check blocks to be written to the memory according to the size of the data, and write the data to the SSD.

[0128] Step 404: The controller selects to store the received data in a large check group.

[0129] In the embodiment of the present application, when the controller determines that the load is high, it can adaptively select to store the data received by the storage system in a large check group, wherein the number of data blocks in the large check group is greater than the number of data blocks in the small check group.

[0130] It is understandable that, generally in the application process, the number of data blocks and check blocks in the large check group is the number of stripes contained in the full stripe. When the storage system is heavily loaded, a large amount of data can be received in a short time and the stripe can be filled. At this time, the controller does not need to consider the problem of additional overhead caused by the need to fill zeros because the data cannot be filled into the stripe.

[0131] Based on step 401 to step 404, in an optional implementation, when garbage collection is performed on the solid state drive, the controller in the storage system migrates valid data stored in the memory in the small check group according to the large check group.

[0132] In this implementation, take the solid state drive as an example. Figure 4c , Figure 4c A schematic diagram of a process for garbage collection of a solid state drive provided in an embodiment of the present application. Figure 4c As shown, the controller executes the following steps 405 to 407.

[0133] Step 405 : The controller obtains the sizes of data received in several time periods and the number of corresponding logic blocks.

[0134] In the embodiment of the present application, when the load of the storage system is low, the controller can count the data received in several time periods and write the data to the corresponding logical blocks on the solid-state drive for comparison in subsequent garbage collection. The logical blocks include data blocks and check blocks.

[0135] Step 406 : The controller calculates the garbage collection efficiency corresponding to each of the several time periods according to the size of the data corresponding to each of the several time periods and the number of the corresponding logic blocks.

[0136] In an embodiment of the present application, when the load on the storage system is low, the controller collects the size of data in several time periods, compares the garbage collection efficiency corresponding to the data received in each time period, selects the data with the highest garbage collection efficiency for recycling, migrates the valid data in the solid-state drive according to the large check group, and restores capacity utilization.

[0137] It is understandable that in actual application, the controller can collect write requests in several different time periods and compare them. It is not required that the time of each time period is the same. The specific setting can be based on actual needs and is not limited here.

[0138] In an optional implementation, for each time period, the controller calculates the garbage collection efficiency corresponding to the time period according to the size of the corresponding data and the number of corresponding logic blocks in the time period, including:

[0139] The controller obtains the size of a single logic block in the logic block group;

[0140] The controller obtains the total size corresponding to the logic block group according to the size of the single logic block and the number of columns of the logic block group;

[0141] The controller obtains the size of valid data in the logical block group;

[0142] The controller obtains the efficiency of garbage collection according to the total size corresponding to the logical block group and the size of the valid data.

[0143] In this implementation, after the controller determines the number of columns of the corresponding logic block group according to the size of the data, the total size of the logic block group can be obtained according to the size of a single logic block and the number of columns of the logic block group. Then, the controller can obtain the efficiency of garbage collection according to the total size of the logic block group and the size of the valid data in the logic block group.

[0144] For example, assuming that the amount of garbage data is 50%, in a 4D+2P disk group, the gain is 6D, and the valid data that needs to be moved is 4D*50%, then the recovery efficiency is 6D / (4D*50%)=3; in an 8+2P disk group, the gain is 10D, and the valid data that needs to be moved is 8D*50%, then the recovery efficiency is 10D / (8D*50%)=2.5.

[0145] Step 407: The controller selects the data with the highest efficiency for garbage collection from the garbage collection efficiencies corresponding to the respective time periods.

[0146] In the embodiment of the present application, after calculating the garbage collection efficiencies corresponding to several time periods, the controller selects the data with the highest garbage collection efficiency for garbage collection to ensure maximum recycling benefits.

[0147] It is understandable that, at present, N and M in the disk group N+M are fixed, such as the common 23 (number of copies of data blocks) + 2 (number of copies of check blocks). These data blocks are stored in 25 disks respectively. If any two disks fail, the other 23 disks can be used for data recovery. However, if the actual data blocks cannot be gathered to 23 in a short time, it is necessary to fill the 23 data columns by adding 0s, and generate 2 check blocks of 23 data blocks to form a complete check group. After forming a check group, the data of the check group can be sent to the disk for storage (download), and these 0s will also be stored from the cache to the disk, occupying the disk space, resulting in a waste of disk space.

[0148] Correspondingly, in the present application, when the data of the received write request is small and the data of the write request cannot fill the stripe, the number of columns of the logical block group written to the disk can be dynamically adjusted according to the size of the data of the received write request, and different Ns are supported, and different redundancy ratios are adaptively selected. For example, 4+2, 10+2, and 23+3 are supported at the same time. When the load of the storage system is low, the received data is stored in a small check group, such as 4+2, without worrying about the problem of not being able to fill 23 data blocks. When the load of the storage system is high, the received data is stored in a large check group. This avoids the problem of additional overhead caused by the need to fill zeros when the data cannot fill the stripe, reduces write amplification, and improves the overall performance and life of the disk. After the data is written to the disk, when garbage collection is performed, since a large amount of data needs to be migrated, the valid data in the SSD can be migrated according to the large check group, and the disk group can be restored to the maximum capacity. For example, 23 data blocks can be gathered and 23+2 can be used directly, thereby restoring the capacity utilization of the disk group and reducing the occurrence of zero padding. In addition, the data on the disk is more based on the user's real data, and the data on the disk is less, which reduces the occurrence of write amplification.

[0149] In Figures 1 to Figure 4c On the basis of the corresponding embodiments, in order to better implement the above solutions of the embodiments of the present application, the following also provides related devices for implementing the above solutions. Figure 5 , Figure 5 A schematic diagram of a structure of a data storage device provided in an embodiment of the present application. The data storage device may be located in Figure 1a The data storage device 5000 includes:

[0150] The first storage module 5001 is used for selecting to store the data received by the data storage device in a small check group when the load of the data storage device is low, and the small check group includes a data block and a check block;

[0151] The second storage module 5002 is used to store the data received by the data storage device in a large check group when the load of the data storage device is high, wherein the large check group includes data blocks and check blocks, and the number of data blocks in the large check group is greater than the number of data blocks in the small check group.

[0152] In a possible implementation, the data storage device stores data in a write redirection ROW manner.

[0153] In one possible implementation, storing data received by a data storage device includes:

[0154] The data storage device sends the data in the small check group and the large check group to a local or remote memory for storage, and different memories store different blocks.

[0155] A possible implementation also includes:

[0156] When performing garbage collection on the memory, the data storage device migrates valid data in the memory according to the large check group.

[0157] In a possible implementation, the number of check blocks in the large check group and the number of check blocks in the small check group are the same.

[0158] In one possible implementation, the data storage device is a centralized storage device or a storage server.

[0159] In one possible implementation, the data storage device is a distributed storage system, or a distributed storage device constituting a distributed storage system.

[0160] In this embodiment, the operations performed by each unit in the data storage device 5000 are the same as those described above. Figure 4a The method described in the illustrated embodiment is similar and can be used to implement the functions of the controller in the above method embodiment, and can also achieve the beneficial effects possessed by the above method embodiment, which will not be described in detail here.

[0161] See also Figure 6 , Figure 6 A schematic diagram of the structure of a controller provided in an embodiment of the present application. Figure 6 As shown, the controller 6000 is implemented by a general bus architecture.

[0162] The controller 6000 includes at least one processor 6001 , a communication bus 6002 , a memory 6003 , and at least one communication interface 6004 .

[0163] The processor 6001, the memory 6003, and the communication interface 6004 communicate via the communication bus 6002, and can also communicate via other means such as wireless transmission. The memory 6003 is used to store instructions, and the processor 6001 is used to execute the instructions stored in the memory 6003. The memory 6003 stores program codes, and the processor 6001 can call the program codes stored in the memory 6003 to execute Figure 4a For specific implementation of steps 401 to 404 in the embodiment shown, please refer to Figure 4a The detailed description of steps 401 to 404 in the illustrated embodiment is not repeated here.

[0164] Optionally, the processor 6001 is a general-purpose central processing unit (CPU), and may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA), or other programmable logic devices (PLD), transistor logic devices, hardware components, or any combination thereof. The above-mentioned PLD is a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0165] The communication bus 6002 is used to transmit information between the processor 6001, the memory 6003 and the communication interface 6004. The communication bus 6002 is divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0166] Alternatively, the memory 6003 is a read-only memory (ROM) or other types of static storage devices that can store static information and instructions. Alternatively, the memory 6003 is a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions. Alternatively, the memory 6003 is an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), disk storage medium or other magnetic storage device, or can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer. Any other medium, but not limited to this. Optionally, the memory 6003 exists independently and is connected to the processor 6001 through the communication bus 6002. Optionally, the memory 6003 and the processor 6001 are integrated together.

[0167] The communication interface 6004 uses any transceiver-like device for communicating with other devices or communication networks. The communication interface 6004 includes a wired communication interface. Optionally, the communication interface 6004 also includes a wireless communication interface. Among them, the wired communication interface is, for example, an Ethernet interface. The Ethernet interface is an optical interface, an electrical interface or a combination thereof. The wireless communication interface is a wireless local area network (WLAN) interface, a cellular network communication interface or a combination thereof, etc.

[0168] In a specific implementation, as an embodiment, the processor 6001 includes one or more CPUs, such as Figure 6 CPU0 and CPU1 are shown in the figure.

[0169] In a specific implementation, as an embodiment, the controller 6000 includes multiple processors, such as Figure 6 6001 and 6005 are shown in FIG. Each of these processors is a single-CPU or a multi-CPU. A processor herein refers to one or more devices, circuits, and / or processing cores for processing data (such as computer program instructions).

[0170] In some embodiments, the memory 6003 is used to store program codes for executing the solution of the present application, and the processor 6001 executes the program codes stored in the memory 6003. In other words, the data storage device 5000 implements the above-mentioned data storage method embodiment through the processor 6001 and the program codes in the memory 6003.

[0171] It is to be understood that the method steps in the embodiments of the present application may be implemented in hardware or in software instructions executable by the processor 6001. The software instructions may be composed of corresponding software modules, and the software modules may be stored in random access memory, flash memory, read-only memory, programmable read-only memory, erasable programmable read-only memory, electrically erasable programmable read-only memory, registers, hard disks, mobile hard disks, CD-ROMs, or any other form of storage medium known in the art. In addition, the scope of the device described in the present application is not limited thereto, and the structure of the device may not be limited thereto. Figure 6 A device may be a stand-alone device or may be part of a larger device. For example, a device may be:

[0172] (1) Independent integrated circuit IC, or chip, or chip system or subsystem;

[0173] (2) having a set of one or more ICs, and optionally, the IC set may also include a storage component for storing data and / or instructions;

[0174] (3) Modules that can be embedded in other devices;

[0175] (4) Receivers, terminals, intelligent terminals, wireless devices, handheld devices, mobile units, vehicle-mounted devices, artificial intelligence devices, machinery and equipment, household equipment, medical equipment, industrial equipment, etc.;

[0176] (5)Others

[0177] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0178] An embodiment of the present application further provides a storage medium having a program stored thereon. When the program is executed by a processor, the data storage method in each of the above embodiments is implemented.

[0179] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0180] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0181] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0182] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0183] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0184] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory, a random access memory, a magnetic disk, or an optical disk.

[0185] The naming or numbering of the steps in this application does not mean that the steps in the method flow must be executed in the time / logical sequence indicated by the naming or numbering. The named or numbered process steps can change the execution order according to the technical purpose to be achieved, as long as the same or similar technical effects can be achieved. The division of the units in this application is a logical division. There may be other division methods when it is implemented in actual applications. For example, multiple units can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between each other shown or discussed can be through some interfaces, and the indirect coupling or communication connection between units can be electrical or other similar forms, which are not limited in this application. In addition, the unit or subunit described as a separate component may or may not be physically separated, may or may not be a physical unit, or may be distributed in multiple circuit units, and some or all of the units can be selected according to actual needs to achieve the purpose of the present application solution.

[0186] The above is only a specific implementation mode of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or replacements within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. The various digital numbers involved in the embodiments of the present application are only for the convenience of description and are not used to limit the scope of the embodiments of the present application. The size of the sequence number of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic.

[0187] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A data storage method, characterized in that: include: When the load of the storage system is low, the data received by the storage system is stored in a small check group, where the small check group includes a data block and a check block; When the load of the storage system is high, the data received by the storage system is stored in a large check group, wherein the large check group includes data blocks and check blocks, and the number of data blocks in the large check group is greater than the number of data blocks in the small check group.

2. The data storage method according to claim 1, characterized in that: The storage system stores the data in a write redirection ROW manner.

3. The data storage method according to claim 1 or 2, characterized in that: The storing of the data received by the storage system includes: The storage system sends the data in the small check group and the large check group to a local or remote storage for storage, and different storages store different blocks.

4. The data storage method according to claim 3, characterized in that: Also includes: When performing garbage collection on the memory, the storage system migrates valid data stored in the memory in the small check group according to the large check group.

5. The data storage method according to any one of claims 1 to 4, characterized in that: The number of check blocks in the large check group and the number of check blocks in the small check group are the same.

6. The data storage method according to any one of claims 1 to 5, characterized in that: The storage system is a centralized storage device or a storage server.

7. The data storage method according to any one of claims 1 to 5, characterized in that: The storage system is a distributed storage system, or is a distributed storage device constituting the distributed storage system.

8. A data storage device, characterized in that: include: A first storage module, configured to select to store the data received by the data storage device in a small check group when the load of the data storage device is low, wherein the small check group includes a data block and a check block; A second storage module is used to store the data received by the data storage device in a large check group when the load of the data storage device is high, wherein the large check group includes data blocks and check blocks, and the number of data blocks in the large check group is greater than the number of data blocks in the small check group.

9. The data storage device according to claim 8, characterized in that: The data storage device stores the data in a write redirection ROW manner.

10. The data storage device according to claim 8 or 9, characterized in that: The storing of the data received by the data storage device comprises: The data storage device sends the data in the small check group and the large check group to a local or remote memory for storage, and different memories store different blocks.

11. The data storage device according to claim 10, characterized in that: Also includes: When performing garbage collection on the memory, the data storage device migrates valid data stored in the memory in the small check group according to the large check group.

12. The data storage device according to any one of claims 8 to 11, characterized in that: The number of check blocks in the large check group and the number of check blocks in the small check group are the same.

13. The data storage device according to any one of claims 8 to 12, characterized in that: The data storage device is a centralized storage device or a storage server.

14. The data storage device according to any one of claims 8 to 12, characterized in that: The data storage device is a distributed storage system, or a distributed storage device constituting the distributed storage system.

15. A controller, characterized in that: The controller comprises a memory and a processor, wherein the memory stores the data and a program executable on the processor, and when the processor executes the program, the controller implements the data storage method according to any one of claims 1 to 7.

16. A storage system comprising a plurality of storage devices, characterized in that: When the load of the storage system is low, selecting to store the received data in a storage device of the storage system in a small check group, wherein the small check group includes a data block and a check block; When the load of the storage system is high, the received data is stored in the storage device of the storage system in the form of a large check group, the small check group includes data blocks and check blocks, and the number of data blocks in the large check group is greater than the number of data blocks in the small check group.

17. The storage system according to claim 16, characterized in that: Each of the storage devices manages a plurality of memories, wherein the storage system is used for: storing the same check group of blocks in a plurality of the storage devices, with different storage devices storing different blocks; or, Blocks of the same check group are stored in a plurality of the memories, and different memories store different blocks.

Images