Data storage method and related apparatus

Abstract

The present application discloses a data storage method and a related apparatus, which may be applied to the technical field of storage. The method comprises: when a storage system has low load, selecting to store data received by the storage system in a small verification group mode, the small verification group comprising data blocks and verification blocks; and when the storage system has high load, selecting to store the data received by the storage system in a large verification group mode, wherein the large verification group comprises data blocks and verification blocks, and the number of data blocks in the large verification group is greater than the number of data blocks in the small verification group. The problem that extra overhead is caused by the need of padding with zeros when data cannot completely fill all required segments is avoided, write amplification is reduced, and the performance and service life of disks is improved.

Description

Data storage method and related device

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on July 6, 2023, with application number CN202310828906.1 and invention name “A data storage method and related device”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of storage technology, and in particular to a data storage method and related devices. Background Art

[0003] Some storage systems, such as solid-state disks (SSDs), often use a redirection on write (ROW) mechanism to write new data. Specifically, when the storage system receives new data, it is written to a free block in the storage system, and the existing data is marked as garbage data and awaits recycling.

[0004] When the storage system receives data, if the data does not fill a complete stripe, it needs to fill the unfilled portion of the stripe with zeros to generate parity data. However, this operation increases write amplification and incurs additional space overhead.

[0005] Summary of the Invention

[0006] The present application provides a data storage method and related devices, which can reduce write amplification.

[0007] In a first aspect, the present application provides a data storage method, which can be applied in the field of storage technology and mainly includes:

[0008] When the load of the storage system is low, the data received by the storage system is stored in a small check group, where the small check group includes data blocks and check blocks; when the load of the storage system is high, the data received by the storage system is stored in a large check group, where the large check group includes data blocks and check blocks, and the number of data blocks in the large check group is greater than the number of data blocks in the small check group.

[0009] In this application, the parity group is dynamically adjusted based on the storage system load. When the storage system load is low, a small parity group is used to store data. This avoids the problem of additional overhead caused by the need to fill zeros when the data cannot fill the stripe, reduces write amplification, and improves the overall performance and lifespan of the storage system. When the storage system load is high, a large parity group is used to store data, improving disk utilization. This achieves dynamic adjustment of data storage methods based on the storage system load, further improving storage system performance and reducing system overhead.

[0010] In a possible implementation of the first aspect, the storage system stores data in a redirect-on-write (ROW) manner.

[0011] When the storage system receives data, it stores it using write redirection. When the storage system load is low and the data doesn't fit into a full stripe on the disk, it typically pads the stripe with zeros to generate parity data, increasing write amplification and incurring additional overhead. In this implementation, the storage system dynamically adjusts the number of stored data blocks based on the received data, avoiding ineffective zero padding and reducing write amplification.

[0012] In a possible implementation of the first aspect, storing data received by a storage system includes:

[0013] The storage system sends the data in the small check group and the large check group to a local or remote storage for storage, with different storages storing different blocks.

[0014] In this implementation, the storage system uses a memory to store data, and each block in the data block and the check block corresponds to a different memory, so that the data is stored in the different memories.

[0015] In a possible implementation of the first aspect, the method further includes:

[0016] When performing garbage collection on the memory, the storage system migrates valid data stored in the memory in the small check group according to the large check group.

[0017] In this implementation, when garbage collection is performed on the memory, the storage system uses a large parity group to migrate valid data stored in the memory using a small parity group, thereby restoring the capacity utilization of the memory.

[0018] In a possible implementation of the first aspect, the number of parity blocks in the large parity group and the number of parity blocks in the small parity group are the same.

[0019] In this implementation, regardless of whether the storage system load is high or low, it is assumed that the number of generated check blocks is the same. The number of data blocks is dynamically adjusted according to the received data, and different numbers of data blocks are selected for data storage. This avoids the problem of additional overhead caused by the need to fill zeros in the stripe when the data cannot be filled, and reduces write amplification.

[0020] In a possible implementation manner of the first aspect, the storage system is a centralized storage device or a storage server.

[0021] In this implementation, the storage system is suitable for centralized distribution scenarios, and the storage system is a centralized storage device or storage server.

[0022] In a possible implementation of the first aspect, the storage system is a distributed storage system, or a distributed storage device constituting a distributed storage system.

[0023] In this implementation, the storage system is suitable for distributed scenarios, and the storage system is a distributed storage system, or a distributed storage device in a distributed storage system.

[0024] In a second aspect, the present application provides a data storage device. The beneficial effects can be found in the description of the first aspect and will not be repeated here. The device has the function of implementing the behavior in the method example of the first aspect. The function can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions. In one possible design, the device includes:

[0025] A first storage module is configured to store data received by the data storage device in a small check group when the load of the data storage device is low, where the small check group includes a data block and a check block;

[0026] The second storage module is used to store the data received by the data storage device in a large check group when the load of the data storage device is high, wherein the large check group includes data blocks and check blocks, and the number of data blocks in the large check group is greater than the number of data blocks in the small check group.

[0027] In a possible implementation of the second aspect, the data storage device stores data in a write redirection (ROW) manner.

[0028] In a possible implementation of the second aspect, storing data received by the data storage device includes:

[0029] The data storage device sends the data in the small check group and the large check group to a local or remote memory for storage, and different memories store different blocks.

[0030] In a possible implementation of the second aspect, the method further includes:

[0031] When performing garbage collection on the memory, the data storage device migrates valid data in the memory according to the large check group.

[0032] In a possible implementation of the second aspect, the number of parity blocks in the large parity group and the number of parity blocks in the small parity group are the same.

[0033] In a possible implementation manner of the second aspect, the data storage device is a centralized storage device or a storage server.

[0034] In a possible implementation of the second aspect, the data storage device is a distributed storage system, or a distributed storage device constituting a distributed storage system.

[0035] In a third aspect, the present application provides a controller, characterized in that it includes a memory and a processor, the memory stores data and programs that can be run on the processor, and when the processor executes the program, the controller implements the method in the above-mentioned first aspect or any possible implementation of the first aspect.

[0036] In a fourth aspect, the present application provides a storage system comprising a plurality of storage devices, wherein when the load of the storage system is low, received data is stored in the storage devices of the storage system in a small parity group, where the small parity group includes a data block and a parity block;

[0037] When the load of the storage system is high, the received data is stored in the storage device of the storage system in the form of a large check group. The small check group includes data blocks and check blocks. The number of data blocks in the large check group is greater than the number of data blocks in the small check group.

[0038] In a fifth aspect, a storage system is provided, wherein each storage device manages multiple memories, wherein the storage system is used to:

[0039] Store the blocks of the same parity group in multiple storage devices, with different storage devices storing different blocks; or,

[0040] The blocks of the same check group are stored in multiple memories, and different memories store different blocks.

[0041] In a sixth aspect, the present application provides a computer program product, which, when executed by a computer, implements the method in the above-mentioned first aspect or any possible implementation manner of the first aspect.

[0042] In a seventh aspect, the present application provides a chip system, comprising a processor configured to implement the method of the first aspect or any possible implementation of the first aspect. In one possible design, the chip system further comprises a memory configured to store program instructions and / or data. The chip system may be composed of a chip alone or may include a chip and other discrete components.

[0043] The solutions of the second to seventh aspects mentioned above are used to implement or cooperate with the method of the first aspect or any possible implementation method thereof, and therefore can achieve the same or corresponding beneficial effects as the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] FIG1a is a schematic structural diagram of a storage system provided in an embodiment of the present application;

[0045] FIG1b is a structural diagram of a controller in a storage system according to an embodiment of the present application;

[0046] FIG1c is a schematic diagram of a flow chart of a data storage method provided in an embodiment of the present application;

[0047] FIG2 is a schematic diagram of the structure of a RAID 5 disk array provided in an embodiment of the present application;

[0048] FIG3 is a flow chart of a currently used data storage method;

[0049] FIG4a is another schematic flow chart of a data storage method according to an embodiment of the present application;

[0050] FIG4 b is a schematic diagram of a process for storing received data in a small check group manner according to an embodiment of the present application;

[0051] FIG4c is a schematic diagram of a process for garbage collection on a solid-state drive according to an embodiment of the present application;

[0052] FIG5 is a schematic structural diagram of a data storage device provided in an embodiment of the present application;

[0053] FIG6 is a schematic diagram of the structure of a controller provided in an embodiment of the present application. DETAILED DESCRIPTION

[0054] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of this application. It is known to those skilled in the art that with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0055] The terms "first," "second," and the like in the specification and claims of this application and in the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" and "having," as well as any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or modules is not necessarily limited to those steps or modules explicitly listed, but may include other steps or modules that are not explicitly listed or that are inherent to these processes, methods, products, or apparatus.

[0056] The term "and / or" as used in this application can be used to describe an association relationship between related objects, indicating that three possible relationships exist. For example, A and / or B can represent the following three situations: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the character " / " in this application generally indicates that the related objects are in an "or" relationship.

[0057] It should also be noted that, in some alternative implementations, the functions / acts noted may occur out of the order of the drawings. For example, two figures shown in succession may in fact occur substantially concurrently or may sometimes be executed in the reverse order, depending on the functionality / acts involved.

[0058] In the embodiments of the present application, unless otherwise specified, the meaning of "at least one" refers to one or more, and the meaning of "a plurality of" refers to two or more. It is understood that in the present application, "when...", "if" and "if" all mean that the device will make corresponding processing under certain objective circumstances, and do not limit the time, nor do they require that there must be a judgment action when the device is implemented, nor do they mean that there are other limitations. In addition, the special word "exemplary" means "used as an example, embodiment or illustrative". Any embodiment described as "exemplary" is not necessarily interpreted as being superior to or better than other embodiments.

[0059] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0060] First, in order to better understand the solution of the embodiment of the present application, the application scenario of the embodiment of the present application is described below. The data storage method provided by the embodiment of the present application is mainly used in a storage system to maintain the availability of services.

[0061] The following describes an example application scenario of the data storage method provided by the embodiment of the present application with reference to Figures 1a and 1b. Referring to Figure 1a, Figure 1a is a schematic diagram of a storage system provided by the embodiment of the present application. The storage system 100 includes several disks 12 and a controller 11.

[0062] Storage system 100 is applicable to storage systems that use a redirection on write (ROW) mechanism, including centralized storage systems and distributed storage systems. Optionally, the storage system can be a centralized storage device or a storage server. Alternatively, storage system 100 can be a distributed storage system, or a distributed storage device within a distributed storage system.

[0063] The controller 11 is connected to a host (not shown). The controller and the host can be integrated into the same central processing unit (CPU). The controller 11 can be a computing device, such as a server, desktop computer, or the like. A writing system and application programs are installed on the controller 11. The controller 11 can receive input / output (I / O) requests from the host. The controller 11 can also store data carried in the I / O requests and write the data to the disk 12. The controller 11 and the disk 12 can be connected locally or remotely, including wired and wireless connections. When the disk 12 and the controller 11 are locally connected, the disk 12 and the controller 11 are located in the same device (e.g., the same physical storage device or the same physical storage server), and the controller 11 and the disk 12 are connected via a peripheral component interconnect express (PCIe) bus, or via a serial advanced technology attachment (SATA) interface, a serial attached small computer system interface (SAS), or an M.2 interface. When connected remotely, disk 12 and controller 11 are not on the same device. For example, controller 11 is connected to external disk 12 via a built-in network card. Disk 12 here refers to persistent storage that can retain data long-term after a power outage, such as a hard disk drive (HDD), solid-state drive (SSD), or phase change memory (PCM). For controller 11, sending data to disk 12 fulfills its storage function.

[0064] It should be noted that Figure 1a is only an exemplary illustration, and the disk 12 is only one example of a storage device connected to the controller 11. The shapes of the controller 11 and disk 12 shown in Figure 1a do not represent the actual physical structure of the controller 11 and disk 12, and are not limited here. In addition, the implementation of this application does not limit the connection method between the controller and the disk 12, and only needs to ensure that the controller 11 and the disk 12 can communicate with each other. In actual applications, the storage system 100 may include two or more controllers, and the physical structure and function of each controller are similar to those of the controller 11. In addition, this embodiment does not limit the connection method between the controllers, and between any controller and the disk 12. As long as the controllers and the controllers and the disk 12 can communicate with each other. In addition, in this embodiment, when the controller 11 sends instructions to the disk 12, it often sends them to multiple disks 12. For the convenience of description, the set containing multiple disks 12 is called a disk group.

[0065] Please refer to Figure 1b, which is a structural diagram of a controller in a storage system according to an embodiment of the present application. As shown in Figure 1b, the controller 11 includes an interface card 110, a memory 111, a processor 112, and an interface card 113.

[0066] The interface card 110 is used to communicate with the host and receive instructions sent by the host. The controller 11 can receive write instructions from the host through the interface card 110 .

[0067] The processor 112 is a central processing unit. In the embodiment of the present application, the processor 112 can be used to receive write instructions or read instructions from the host and process these instructions. The processor 112 can also send the data in the write instruction to the disk 12.

[0068] Optionally, the controller 11 may further include a memory 111 .

[0069] The memory 111 includes a volatile memory, such as a random-access memory (RAM).

[0070] Optionally, the controller 11 may further include an interface card 113 .

[0071] The interface card 113 is used to communicate with the disk 12. The controller 11 can send a write data instruction to the disk 12 for storage through the interface card 113.

[0072] It can be understood that the structure of the storage system shown in Figures 1a and 1b is only an example. The data storage method provided in the embodiment of the present application can be applied to but is not limited to the structure described above and is not limited here.

[0073] An embodiment of the present application provides a storage system comprising a controller and multiple disks. The controller and the multiple disks can communicate with each other via internal connection paths, transmitting control and / or data signals. The controller stores stripe data information. The stripe data information is used to record the status of each stripe unit in the disk group. The controller can execute received data processing instructions based on the stored stripe data information.

[0074] It is understandable that the present application is applicable to any storage system that adopts the ROW mechanism. To simplify the description, the present application uses redundant arrays of independent disks (RAID) to represent a disk group formed by combining multiple independent physical disks in different ways, which has the same meaning as the commonly mentioned RAID group and RAID array and is not limited here.

[0075] Based on the storage system shown in Figure 1a, when the storage system only supports large stripes, if the data received by the storage system in a short period of time cannot fill all the stripes of the disk, all the unfilled parts of the disk need to be filled with zeros, which causes additional overhead and increases write amplification.

[0076] A stripe is a type of parity group. Each stripe consists of multiple stripe units, including both data stripe units and parity stripe units. A large stripe has a large number of stripe units, while a small stripe has a small number of stripe units.

[0077] When a storage system stores data on persistent storage media (such as SSDs or HDDs), it uses a complete stripe (also known as a "full stripe") as the minimum unit. For example, if the stripe granularity is 20+2, 20 data stripe units must be assembled, and then the two parity stripe units corresponding to the 20 data stripe units must be calculated to form a "full stripe."

[0078] Assume that the amount of IO data currently received by the storage system is small, only enough to generate 8 data stripe units. At this time, the storage system has two solutions: (1) continue to wait for the arrival of new data until 20 data stripe units are collected; (2) use "0" to fill the remaining 12 data stripe units to collect 20 data stripe units as soon as possible. In the former case, during the waiting process, these 8 data stripe units will be temporarily stored in the memory and will not be sent to the disk for a long time. Although the latter case can make these 8 data stripe units sent to the disk as soon as possible, the additional 12 data units filled with all 0s are also sent to the disk for storage, wasting disk storage space.

[0079] If the stripe size were 8+2 instead of 20+2, a full stripe could be created quickly based on user I / O, allowing data to be delivered to the disk as quickly as possible without padding with "0s." Therefore, using large stripes in a storage system improves storage media utilization. However, if the storage system receives a small amount of data in a short period of time, data may not be delivered to the disk quickly enough, or disk utilization may be low.

[0080] In order to solve the above problems, an embodiment of the present application provides a data storage method, which is applied to the controller 11 shown in Figure 1a. Specifically, please refer to Figure 1c, which is a flow chart of the data storage method provided by the embodiment of the present application. The method specifically includes: S1. When the load of the storage system is low, the data received by the storage system is selected to be stored in a small check group, and the small check group includes data blocks and check blocks. S2. When the load of the storage system is high, the data received by the storage system is selected to be stored in a large check group, wherein the large check group includes data blocks and check blocks, and the number of data blocks in the large check group is greater than the number of data blocks in the small check group. The check group here refers to a RAID group, an error correction code (EC) group or a similar technology. When the data of some blocks in the same check group is damaged, as long as the number of damaged blocks does not exceed the check upper limit (the upper limit is usually the amount of data in the check block), the remaining blocks can be used to recover the damaged blocks.

[0081] The embodiment of the present application uses "load" to describe the amount of data received by the storage system (or storage device, storage equipment) in a short period of time. When a large amount of data is received in a short period of time (this large amount of data can be from one IO request or from multiple IO requests, without limitation), it means that a large check group can be put together in a short period of time, and the executor of the data storage method is at a high load at this time; conversely, when a small amount of data is received in a short period of time, it means that it is difficult to put together a large check group in a short period of time, and the executor of the data storage method is at a low load at this time. The specific high and low load settings can be set by the user based on experience or determined by AI training. For example, when more than 100MB of data is received within 10 seconds, it is defined as a high load; when less than 100MB of data is received within 10 seconds, it is defined as a low load. Alternatively, when the CPU utilization of the storage device processing IO reaches 50%, it is defined as a high load; otherwise, it is defined as a low load.

[0082] It is understandable that, at present, the N and M in the disk group's N+M are fixed, such as the common 23 (number of data block copies) + 2 (number of check block copies). These data blocks are stored in 25 disks respectively. If any two disks fail, the other 23 disks can be used for data recovery. However, if the actual data blocks cannot be gathered in a short period of time, it is necessary to fill the 23 data columns by adding 0s and generate 2 check blocks for the 23 data blocks to form a complete check group. After forming a check group, the data of the check group can be sent to the disk for storage (download). These 0s will also be stored from the cache to the disk, occupying disk space and causing disk space waste.

[0083] Correspondingly, in the present application, when the load on the storage system is low and the data received by the storage system cannot fill the stripe, the number of columns of the logical block group written to the disk can be dynamically adjusted according to the size of the received data, supporting different Ns and adaptively selecting different redundancy ratios. For example, 4+2, 10+2, and 23+3 are supported at the same time. When the load on the storage system is low, a small check group is used to store the received data, such as 4+2, so there is no need to worry about not being able to fill up 23 data blocks. When the load on the storage system is high, a large check group is used to store the received data. This avoids the problem of additional overhead caused by the need to fill in zeros when the data cannot fill the stripe, reduces write amplification, and improves the overall performance and life of the disk.

[0084] Optionally, after the data is written to the disk, when garbage collection is performed on the solid-state drive, the storage system can migrate the valid data in the solid-state drive according to the large check group, restore the disk group to its maximum capacity, and directly use 23+2 to make up, for example, 23 data blocks, thereby restoring the capacity utilization of the disk group and reducing the occurrence of zero padding. In addition, the data downloaded to the disk is more based on the user's real data, and the data downloaded to the disk is less, which means that write amplification is reduced.

[0085] The above introduces the application scenarios of the data storage method provided in the embodiments of the present application. The following will introduce in detail the implementation process of the data storage method provided in the embodiments of the present application.

[0086] First, in order to better understand the solutions of the embodiments of the present application, the relevant terms and concepts that may be involved in the embodiments of the present application are introduced below.

[0087] (1) Redundant Array of Independent Disks

[0088] Redundant Array of Independent Disks (RAID) technology combines multiple independent physical disks in various ways to form a disk group, providing higher storage performance than a single disk and data backup. Commonly used RAID configurations include RAID 0, RAID 1, RAID 5, RAID 6, and RAID 10, depending on the data organization method. Different RAID levels offer varying levels of performance and reliability. In most cases, if one or more disks fail, the data on the failed disk can be restored using the data on the remaining member disks using the corresponding RAID level algorithm, ensuring data is not lost.

[0089] RAID configurations above 5 require an N+M configuration, where N and M are both positive integers greater than 1. N represents the number of data disks, and M represents the number of parity disks. RAID also optionally includes a hot spare disk. If a disk fails in the RAID array, the RAID controller can restore the data on the failed disk to the hot spare disk based on the parity data on the parity disk and the data on the data disks, thereby improving system reliability.

[0090] In the embodiments of the present application, RAID 5 is used as an example for description. It should be understood that in actual applications, RAID corresponds to multiple levels of N+M, which is not limited in this application.

[0091] RAID 5 uses the parity bit of data to ensure the security of data. RAID 5 can store the parity bit of data on a single disk, or it can store the parity bits of data segments alternately on each disk. The data storage structure in the RAID5 disk array is shown in Figure 2. Figure 2 is a structural schematic diagram of the RAID5 disk array provided by an embodiment of the present application. Figure 2 shows a RAID5 disk array consisting of four disks, each cylinder represents a disk, P (parity) represents the parity bit, D (data) represents the data segment, and the parity data is calculated by exclusive OR (xor). In the example shown in Figure 2, the exclusive OR result of every three data segments is used as the parity bit of these three data segments. For example, the parity bit P1 is the exclusive OR result of the data segments D1, D2, and D3, that is, P1 = D1⊕D2⊕D3. In this way, if any one disk is damaged, the damaged data can be reconstructed based on the parity bits and data segments on other disks.

[0092] According to the principles of RAID 5 technology described above, if a single drive fails in the array, the data on the failed drive can be restored through a rebuild. However, if two drives fail, the RAID 5 array becomes inoperable, and any read or write operations on the remaining healthy drives in the array become impossible, especially data writes. This can lead to serious data loss. Even if the data on the RAID 5 drives is subsequently restored, the business data from the time of the RAID 5 failure will not be written to the RAID array in a timely manner, resulting in the loss of that data.

[0093] It should be noted that the N+1 configuration of RAID 5 logically refers to N data disks and 1 parity disk. However, whether the parity data is fixed on the single parity disk or rotated among multiple disks can be set according to actual needs and is not limited in this application.

[0094] (2) Garbage Collection

[0095] Garbage Collection (GC) refers to moving valid data in a logical block group to a new logical block group, and then erasing the valid data and invalid data in the logical block group to form a new blank logical block group for storing other new data.

[0096] The above introduces the professional terms provided in the embodiments of the present application. In order to better understand the technical solutions provided in the embodiments of the present application, the currently used data storage methods are introduced.

[0097] Please refer to Figure 3, which is a flowchart of a currently used data storage method. As shown in Figure 3, the method mainly includes the following steps 301 to 304.

[0098] Step 301: Obtain write request data.

[0099] In this step, the host sends data to the cache, which then writes the data to the RAID. Due to overwriting, garbage may be generated.

[0100] Step 302: Gather the data into a fixed number of N parts, and generate M parts of verification data based on the N parts of data.

[0101] Step 303: Write N copies of data and M copies of verification data to the disk.

[0102] Step 304: Move valid data on the disk and perform garbage collection.

[0103] The current data storage methods have the following problems:

[0104] (1) When the system load is low, fewer write requests are received. If data cannot be packed into the full stripe or N columns, a write-zero operation must be performed on the unwritten logical blocks, resulting in additional overhead. Furthermore, write-zeroing increases unnecessary write operations to the disk, wasting disk life and increasing write amplification, impacting the overall performance and life of the storage system.

[0105] (2) When the system load is low, if a large number of zeros are written, the amount of garbage data will be high. If garbage collection is performed immediately at this time, it is equivalent to writing multiple times, wasting the life of the disk and increasing system overhead.

[0106] To address the above-mentioned issues, the present application provides a data storage method. For ease of understanding, the data storage method provided in the embodiments of the present application is described in detail below in conjunction with the accompanying drawings and application scenarios. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are equally applicable to similar technical problems.

[0107] Please refer to Figure 4a, which is another schematic flow diagram of a data storage method provided in an embodiment of the present application. This data processing method can be executed by the storage system shown in Figure 1a, specifically, by a controller within the storage system. As shown in Figure 4a, the controller can perform the following steps 401 to 404 to avoid the problem of additional overhead caused by the need to pad zeros when data cannot fully fit into a stripe, thereby reducing write amplification and improving the overall performance and lifespan of the disk.

[0108] Step 401: The controller counts the load of the storage system.

[0109] In an embodiment of the present application, the controller receives a write request instruction issued by a user within a specific time period, and counts the size of the received write request data.

[0110] In step 402, the controller determines whether the storage system load is high or low. If low, the controller executes step 403. If high, the controller executes step 404.

[0111] In an embodiment of the present application, the controller adaptively selects a corresponding method to store the received data by determining the load level of the storage system. The load level primarily refers to whether the data received by the storage system within a relatively short period of time can fill a large stripe. If the load is low, the data cannot fill a large stripe in a short period of time, and the controller chooses to store the received data in a small parity group. If the load is high, the data can fill a stripe in a short period of time, and the controller chooses to store the received data in a large parity group to avoid zero padding.

[0112] Optionally, the controller may also determine the load level based on the number of data blocks and check blocks generated corresponding to the received data.

[0113] Specifically, after receiving data, the controller calculates the number of data blocks and check blocks based on the data size. If the controller determines that the number of data blocks and check blocks is less than the total number of data columns or disks on the storage system, it indicates that the received data cannot fill the stripe, and the controller can determine that the storage system load is low. If the controller determines that the number of data blocks and check blocks corresponding to the received data can fill the stripe, the controller can determine that the storage system load is high.

[0114] It is understandable that in actual applications, the load of the storage system can also be set according to actual needs. It can be stipulated whether the stripes can be fully populated within a short period of time, or it can be judged based on set thresholds or other methods, which are not limited here.

[0115] In step 403, the controller chooses to store the received data in a small check group manner.

[0116] In an embodiment of the present application, when the storage system load is low, the controller within the storage system selects to store data received by the storage system in a small parity group format, thereby dynamically adjusting the number of logical blocks within the disk based on the received data and avoiding invalid write-zero operations. The logical blocks include data blocks and parity blocks.

[0117] In an optional implementation, please refer to Figure 4b, which is a flow chart of storing received data in a small check group according to an embodiment of the present application. The controller selecting the step of storing received data in a small check group may include the following steps 4031 to 4032.

[0118] In step 4031, the controller determines the number of data blocks and check blocks according to the size of the data.

[0119] In an embodiment of the present application, after receiving the size of the data, the controller can generate the number n of data blocks according to the size of the data, assemble n data blocks, and generate m pieces of verification data to write to the disk to avoid write amplification.

[0120] Optionally, the controller may perform weighted calculation on the size of all received data, and then determine the number of data blocks and check blocks according to the size of the weighted data.

[0121] Optionally, the controller determines the number of data blocks and check blocks according to the size of the data, including:

[0122] The controller obtains the size of a single logic block in the logic block group, where the logic block includes a data block and a logic block;

[0123] The controller determines the number of copies of the data block to be written to the disk based on the size of the data and the size of a single logical block;

[0124] The controller obtains the number of copies of the check block according to the number of copies of the data block.

[0125] In this implementation, after performing a weighted calculation on the sizes of all received data, the controller can determine the number of data blocks to be written to the disk based on the weighted data size and the size of a single logical block in the logical block group. A logical block includes a data block and a check block. After determining the number of data blocks on the disk, the controller can calculate the corresponding number of check blocks on the disk based on the exclusive-OR relationship between the data blocks.

[0126] It is understandable that the size of the data received by the controller in a specific time period varies depending on the load. To facilitate calculation, the controller can perform a historical weighted calculation on the size of the received data to obtain an average value of the data size.

[0127] Optionally, the controller obtains the number of data blocks to be written to the disk according to the quotient of the size of the data and the size of a single logical block.

[0128] In step 4032, the controller sends the data to a local or remote memory for storage according to the number of data blocks and check blocks.

[0129] In the embodiment of the present application, different memories store different blocks, and the memory may be, for example, an SSD, etc. The controller may determine the number n of data blocks and the number m of check blocks to be written to the memory according to the size of the data, and write the data to the SSD.

[0130] In step 404 , the controller chooses to store the received data in a large check group format.

[0131] In an embodiment of the present application, when the controller determines that the load is high, it can adaptively select to store the data received by the storage system in a large parity group, wherein the number of data blocks in the large parity group is greater than the number of data blocks in the small parity group.

[0132] It's understandable that, in general, the number of data blocks and parity blocks in a large parity group equals the number of stripes in a full stripe. When the storage system is heavily loaded, a large amount of data can be received in a short period of time and fully written to the stripe. In this case, the controller doesn't need to worry about the overhead of zero padding caused by data not being able to fully write to the stripe.

[0133] Based on steps 401 to 404, in an optional implementation, when garbage collection is performed on the solid-state drive, the controller in the storage system migrates valid data stored in the memory in the small parity group according to the large parity group.

[0134] In this implementation, taking a solid-state drive as an example, refer to FIG4c , which is a flow chart of garbage collection for a solid-state drive according to an embodiment of the present application. As shown in FIG4c , the controller executes the following steps 405 to 407 .

[0135] Step 405 : The controller obtains the size of data received in several time periods and the number of corresponding logic blocks.

[0136] In an embodiment of the present application, when the storage system load is low, the controller can count the data received in several time periods and the number of corresponding logical blocks written to the solid-state drive for comparison during subsequent garbage collection. The logical blocks include data blocks and check blocks.

[0137] In step 406 , the controller calculates the garbage collection efficiency corresponding to each of the several time periods according to the size of the data and the number of logic blocks corresponding to each of the several time periods.

[0138] In an embodiment of the present application, when the load on the storage system is low, the controller collects the size of data in several time periods, compares the garbage collection efficiency corresponding to the data received in each time period, selects the data with the highest garbage collection efficiency for recycling, migrates the valid data in the solid-state hard disk according to the large check group, and restores capacity utilization.

[0139] It is understandable that in actual application, the controller can collect write requests in several different time periods and compare them. It is not required that the time of each time period is the same. The specific setting can be based on actual needs and is not limited here.

[0140] In an optional implementation, for each time period, the controller calculates the garbage collection efficiency corresponding to the time period based on the size of the data corresponding to the time period and the number of logical blocks corresponding to the time period, including:

[0141] The controller obtains the size of a single logical block in the logical block group;

[0142] The controller obtains the total size corresponding to the logic block group according to the size of a single logic block and the number of columns of the logic block group;

[0143] The controller obtains the size of valid data in the logical block group;

[0144] The controller obtains the garbage collection efficiency according to the total size corresponding to the logical block group and the size of the valid data.

[0145] In this implementation, after the controller determines the number of columns in the corresponding logical block group based on the data size, it can then determine the total size of the logical block group based on the size of a single logical block and the number of columns in the logical block group. The controller can then determine the garbage collection efficiency based on the total size of the logical block group and the size of valid data in the logical block group.

[0146] For example, assuming that the amount of garbage data is 50%, in a 4D+2P disk group, the benefit is 6D, and the valid data that needs to be moved is 4D*50%, then the recovery efficiency is 6D / (4D*50%)=3; in an 8+2P disk group, the benefit is 10D, and the valid data that needs to be moved is 8D*50%, then the recovery efficiency is 10D / (8D*50%)=2.5.

[0147] In step 407 , the controller selects the data with the highest efficiency from the garbage collection efficiencies corresponding to the respective time periods for garbage collection.

[0148] In the embodiment of the present application, after calculating the garbage collection efficiency corresponding to each of several time periods, the controller selects the data with the highest garbage collection efficiency for garbage collection to ensure maximum recycling benefits.

[0149] It is understandable that, at present, the N and M in the disk group's N+M are fixed, such as the common 23 (number of data block copies) + 2 (number of check block copies). These data blocks are stored in 25 disks respectively. If any two disks fail, the other 23 disks can be used for data recovery. However, if the actual data blocks cannot be gathered in a short period of time, it is necessary to fill the 23 data columns by adding 0s and generate 2 check blocks for the 23 data blocks to form a complete check group. After forming a check group, the data of the check group can be sent to the disk for storage (download). These 0s will also be stored from the cache to the disk, occupying disk space and causing disk space waste.

[0150] Correspondingly, in the present application, when the data of the received write request is small and the data of the write request cannot fill the stripe, the number of columns of the logical block group written to the disk can be dynamically adjusted according to the size of the data of the received write request, and different Ns are supported, and different redundancy ratios are adaptively selected. For example, 4+2, 10+2, and 23+3 are supported at the same time. When the load of the storage system is low, the received data is stored in a small check group, such as 4+2, without worrying about the problem of not being able to fill 23 data blocks. When the load of the storage system is high, the received data is stored in a large check group. This avoids the problem of additional overhead caused by the need to fill zeros when the data cannot fill the stripe, reduces write amplification, and improves the overall performance and life of the disk. After the data is written to the disk, when garbage collection is performed, since a large amount of data needs to be migrated, the valid data in the solid-state drive can be migrated according to the large check group, and the disk group can be restored to its maximum capacity. For example, 23 data blocks can be put together and 23+2 can be used directly, thereby restoring the capacity utilization of the disk group and reducing the occurrence of zero padding. In addition, the data downloaded to the disk is more based on the user's real data, and the data downloaded to the disk is less, which reduces the occurrence of write amplification.

[0151] Based on the embodiments corresponding to Figures 1a to 4c, in order to better implement the above-mentioned solutions of the embodiments of the present application, the following also provides related devices for implementing the above-mentioned solutions. Please refer to Figure 5, which is a structural schematic diagram of a data storage device provided in the embodiments of the present application. The data storage device can be located in the controller shown in Figure 1a. The data storage device 5000 includes:

[0152] The first storage module 5001 is configured to store data received by the data storage device in a small parity group when the load of the data storage device is low, where the small parity group includes a data block and a parity block;

[0153] The second storage module 5002 is used to store the data received by the data storage device in the form of a large check group when the load of the data storage device is high, wherein the large check group includes data blocks and check blocks, and the number of data blocks in the large check group is greater than the number of data blocks in the small check group.

[0154] In one possible implementation, the data storage device stores data in a write redirection (ROW) manner.

[0155] In one possible implementation, storing data received by a data storage device includes:

[0156] The data storage device sends the data in the small check group and the large check group to a local or remote memory for storage, and different memories store different blocks.

[0157] One possible implementation further includes:

[0158] When performing garbage collection on the memory, the data storage device migrates valid data in the memory according to the large check group.

[0159] In one possible implementation, the number of parity blocks in the large parity group and the number of parity blocks in the small parity group are the same.

[0160] In one possible implementation, the data storage device is a centralized storage device or a storage server.

[0161] In one possible implementation, the data storage device is a distributed storage system, or a distributed storage device constituting a distributed storage system.

[0162] In this embodiment, the operations performed by each unit in the data storage device 5000 are similar to those described in the method embodiment shown in Figure 4a above, and can be used to implement the functions of the controller in the above method embodiment, and can also achieve the beneficial effects of the above method embodiment, which will not be repeated here.

[0163] Please refer to Figure 6, which is a schematic diagram of the structure of a controller provided in an embodiment of the present application. As shown in Figure 6, the controller 6000 is implemented by a general bus architecture.

[0164] The controller 6000 includes at least one processor 6001 , a communication bus 6002 , a memory 6003 , and at least one communication interface 6004 .

[0165] The processor 6001, memory 6003, and communication interface 6004 communicate via a communication bus 6002, and may also communicate via other means such as wireless transmission. The memory 6003 is used to store instructions, and the processor 6001 is used to execute the instructions stored in the memory 6003. The memory 6003 stores program code, and the processor 6001 can call the program code stored in the memory 6003 to execute steps 401 to 404 in the embodiment shown in Figure 4a. For specific implementation, please refer to the detailed description of steps 401 to 404 in the embodiment shown in Figure 4a, which will not be repeated here.

[0166] Optionally, processor 6001 is a general-purpose central processing unit (CPU), and may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA), other programmable logic devices (PLD), transistor logic devices, hardware components, or any combination thereof. The aforementioned PLD is a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0167] Communication bus 6002 is used to transmit information between processor 6001, memory 6003, and communication interface 6004. Communication bus 6002 is divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is used in the figure, but this does not mean that there is only one bus or only one type of bus.

[0168] Alternatively, the memory 6003 is a read-only memory (ROM) or other types of static storage devices that can store static information and instructions. Alternatively, the memory 6003 is a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions. Alternatively, the memory 6003 is an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code with the form of instructions or data structures and can be accessed by a computer, but is not limited to this. Alternatively, the memory 6003 exists independently and is connected to the processor 6001 via a communication bus 6002. Alternatively, the memory 6003 and the processor 6001 are integrated together.

[0169] The communication interface 6004 uses any transceiver-like device for communicating with other devices or communication networks. The communication interface 6004 includes a wired communication interface. Optionally, the communication interface 6004 also includes a wireless communication interface. The wired communication interface is, for example, an Ethernet interface. The Ethernet interface is an optical interface, an electrical interface, or a combination thereof. The wireless communication interface is a wireless local area network (WLAN) interface, a cellular network communication interface, or a combination thereof.

[0170] In a specific implementation, as an embodiment, the processor 6001 includes one or more CPUs, such as CPU0 and CPU1 shown in FIG6 .

[0171] In a specific implementation, as an embodiment, the controller 6000 includes multiple processors, such as processor 6001 and processor 6005 shown in FIG6 . Each of these processors is a single-core processor (single-CPU) or a multi-core processor (multi-CPU). A processor here refers to one or more devices, circuits, and / or processing cores for processing data (such as computer program instructions).

[0172] In some embodiments, the memory 6003 is used to store program codes for executing the solution of the present application, and the processor 6001 executes the program codes stored in the memory 6003. In other words, the data storage device 5000 implements the above-mentioned embodiment of the data storage method through the processor 6001 and the program codes in the memory 6003.

[0173] It is understood that the method steps in the embodiments of the present application can be implemented in hardware or in software instructions that can be executed by the processor 6001. The software instructions can be composed of corresponding software modules, and the software modules can be stored in random access memory, flash memory, read-only memory, programmable read-only memory, erasable programmable read-only memory, electrically erasable programmable read-only memory, registers, hard disks, mobile hard disks, CD-ROMs, or any other form of storage medium known in the art. In addition, the scope of the device described in the present application is not limited to this, and the structure of the device may not be limited to Figure 6. The device can be an independent device or can be part of a larger device. For example, the device can be:

[0174] (1) An independent integrated circuit (IC), or chip, or chip system or subsystem;

[0175] (2) having a set of one or more ICs, optionally including a storage component for storing data and / or instructions;

[0176] (3) Modules that can be embedded in other devices;

[0177] (4) Receivers, terminals, intelligent terminals, wireless devices, handheld devices, mobile units, vehicle-mounted devices, artificial intelligence devices, machinery and equipment, home appliances, medical equipment, industrial equipment, etc.;

[0178] (5)Others, etc.

[0179] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0180] An embodiment of the present application further provides a storage medium having a program stored thereon, which, when executed by a processor, implements the data storage method in each of the above embodiments.

[0181] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0182] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0183] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0184] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0185] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0186] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory, a random access memory, a magnetic disk, or an optical disk.

[0187] The naming or numbering of the steps in this application does not mean that the steps in the method flow must be executed in the time / logical sequence indicated by the naming or numbering. The named or numbered process steps can be changed in the execution order according to the technical purpose to be achieved, as long as the same or similar technical effects can be achieved. The division of units in this application is a logical division. There may be other division methods when implementing in actual applications. For example, multiple units can be combined into or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between each other shown or discussed can be through some interfaces, and the indirect coupling or communication connection between units can be electrical or other similar forms, which are not limited in this application. In addition, the units or subunits described as separate components may or may not be physically separated, may or may not be physical units, or may be distributed to multiple circuit units. Some or all of the units can be selected according to actual needs to achieve the purpose of the present application solution.

[0188] The above is only a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be included in the scope of protection of the present application. The various numerical numbers involved in the embodiments of the present application are only for the convenience of description and are not used to limit the scope of the embodiments of the present application. The size of the sequence number of each process does not mean the order of execution. The order of execution of each process should be determined by its function and internal logic.

[0189] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A data storage method, characterized in that: include: When the load of the storage system is low, the data received by the storage system is stored in a small check group, where the small check group includes a data block and a check block; When the load of the storage system is high, the data received by the storage system is stored in a large check group, wherein the large check group includes data blocks and check blocks, and the number of data blocks in the large check group is greater than the number of data blocks in the small check group.

2. The data storage method according to claim 1, characterized in that: The storage system stores the data in a write redirection ROW manner.

3. The data storage method according to claim 1 or 2, characterized in that: The storing of the data received by the storage system includes: The storage system sends the data in the small check group and the large check group to a local or remote storage for storage, and different storages store different blocks.

4. The data storage method according to claim 3, characterized in that: Also includes: When performing garbage collection on the memory, the storage system migrates valid data stored in the memory in the small check group according to the large check group.

5. The data storage method according to any one of claims 1 to 4, characterized in that: The number of check blocks in the large check group and the number of check blocks in the small check group are the same.

6. The data storage method according to any one of claims 1 to 5, characterized in that: The storage system is a centralized storage device or a storage server.

7. The data storage method according to any one of claims 1 to 5, characterized in that: The storage system is a distributed storage system, or is a distributed storage device constituting the distributed storage system.

8. A data storage device, characterized in that: include: A first storage module, configured to select to store the data received by the data storage device in a small check group when the load of the data storage device is low, wherein the small check group includes a data block and a check block; A second storage module is used to store the data received by the data storage device in a large check group when the load of the data storage device is high, wherein the large check group includes data blocks and check blocks, and the number of data blocks in the large check group is greater than the number of data blocks in the small check group.

9. The data storage device according to claim 8, characterized in that: The data storage device stores the data in a write redirection ROW manner.

10. The data storage device according to claim 8 or 9, characterized in that: The storing of the data received by the data storage device comprises: The data storage device sends the data in the small check group and the large check group to a local or remote memory for storage, and different memories store different blocks.

11. The data storage device according to claim 10, characterized in that: Also includes: When performing garbage collection on the memory, the data storage device migrates valid data stored in the memory in the small check group according to the large check group.

12. The data storage device according to any one of claims 8 to 11, characterized in that: The number of check blocks in the large check group and the number of check blocks in the small check group are the same.

13. The data storage device according to any one of claims 8 to 12, characterized in that: The data storage device is a centralized storage device or a storage server.

14. The data storage device according to any one of claims 8 to 12, characterized in that: The data storage device is a distributed storage system, or a distributed storage device constituting the distributed storage system.

15. A controller, characterized in that: The controller comprises a memory and a processor, wherein the memory stores the data and a program executable on the processor, and when the processor executes the program, the controller implements the data storage method according to any one of claims 1 to 7.

16. A storage system comprising a plurality of storage devices, characterized in that: When the load of the storage system is low, selecting to store the received data in a storage device of the storage system in a small check group, wherein the small check group includes a data block and a check block; When the load of the storage system is high, the received data is stored in the storage device of the storage system in the form of a large check group, the small check group includes data blocks and check blocks, and the number of data blocks in the large check group is greater than the number of data blocks in the small check group.

17. The storage system according to claim 16, characterized in that: Each of the storage devices manages a plurality of memories, wherein the storage system is used for: storing the same check group of blocks in a plurality of the storage devices, with different storage devices storing different blocks; or, Blocks of the same check group are stored in a plurality of the memories, and different memories store different blocks.

Images