Data management method and device

Abstract

The invention discloses a data management method and device, and relates to the technical field of data storage. The controller reads data from the storage medium with the small management granularity for multiple times, and when the data size of the data read for multiple times is consistent with the management granularity of the storage medium with the large management granularity, the controller writes the data read for multiple times into the storage space corresponding to one management granularity of the storage medium with the large management granularity. The data volume of the written data is consistent with the storage capacity of the received data, so that the storage space corresponding to one management granularity is completely used for storing the data. The problem that the storage resources of the storage medium cannot be effectively utilized due to the fact that the storage space for receiving the data cannot be completely filled with the data and the storage space for receiving the data has an idle storage area is solved, and the utilization rate of the storage resources of the storage medium is increased.

Description

Technical Field

[0001] The present application relates to the field of data storage, and in particular to a method and device for data management. Background Art

[0002] With the continuous development of computer technology, various types of data are growing explosively. In order to meet the storage needs of data, various types of storage media have been developed. When the storage needs of data change, the controller designs different management granularities for different storage media. The management granularity is the smallest unit for the controller to perform data operations on the storage medium.

[0003] During the data exchange between two storage media with different management granularities, the controller reads the data in the storage medium with a smaller management granularity, and writes multiple data into the storage medium with a larger management granularity in sequence according to the smaller management granularity. Since the amount of the data is consistent with the smaller management granularity and is smaller than the larger management granularity, when the controller writes the data into the storage medium with a larger management granularity, the storage space of the storage medium with a larger management granularity cannot be fully filled with data, resulting in idle areas in the storage medium with a larger management granularity, wasting the storage resources of the storage device and reducing the utilization rate of the storage resources of the storage device. Summary of the invention

[0004] The present application provides a data management method and device for solving the problem of low storage resource utilization rate caused by different management granularities between storage media with different management granularities.

[0005] In a first aspect, the present application provides a method for data management. The method is executed by a controller, and the method includes: the controller receives a first request. The first request is used to indicate: writing data stored in a first storage medium to a second storage medium. The management granularity of the first storage medium is a first granularity, the management granularity of the second storage medium is a second granularity, and the second granularity is greater than the first granularity. The controller reads multiple data from the first storage medium according to the first request, and merges the multiple data according to the second granularity to obtain one or more data to be written. The data amount of each data is consistent with the first granularity, and the data amount of each data to be written is consistent with the second granularity. The controller writes one or more data to be written to the second storage medium.

[0006] In the present application, the controller reads data multiple times from a storage medium with a smaller management granularity (such as a first storage medium). When the amount of data read multiple times is consistent with the management granularity of a storage medium with a larger management granularity (such as a second storage medium), the controller writes the data read multiple times into a storage space corresponding to a management granularity of the storage medium with a larger management granularity. Because the amount of data written is consistent with the storage space corresponding to a management granularity of the storage medium receiving the data, the storage space corresponding to a management granularity is all used to store data. This avoids the problem that the storage resources of the storage medium cannot be effectively utilized due to the fact that the data cannot completely fill the storage space corresponding to a management granularity of the storage medium receiving the data, resulting in an idle storage area in the storage space receiving the data, thereby improving the utilization rate of the storage resources of the storage medium.

[0007] In a possible implementation, the controller may use a discrete aggregation table and a linked list to merge multiple data to obtain one or more data to be written.

[0008] In another possible implementation, after the controller writes one or more data to be written into the second storage medium, the controller adds a management header to the data stored in the second storage medium. The management header is used to indicate: location information of the data stored in the second storage medium. When the controller needs to obtain data in the second storage medium, the controller parses the management header to achieve fast search and location of the data stored in the second storage medium.

[0009] In another possible implementation, the management header includes at least one of a primary management header and a secondary management header. The primary management header is used to indicate storage information of each to-be-written data in the second storage medium. The secondary management header is used to indicate storage information of each data in the second storage medium. The controller sets management headers corresponding to data of different data sizes to facilitate acquisition of data of different data sizes.

[0010] In another possible implementation, the first-level management header includes at least one of a first offset value and a first flag. The first offset value is used to indicate the storage address of the data to be written in the second storage medium. The first flag is used to indicate the data to be written. The controller sets a first-level management header for the data to be written whose data volume is consistent with the larger management granularity. When it is necessary to read the data to be written with a larger data volume as a whole, the controller parses the first-level management header and directly reads the data to be written with a larger data volume, thereby improving the data reading speed.

[0011] In another possible implementation, the secondary management header includes: at least one of a second offset value and a second flag. The second offset value is used to indicate: the storage address of the data in the second storage medium. The second flag is used to indicate: data. The controller sets a secondary management header for data with a data volume consistent with a smaller management granularity. When it is necessary to read part of the data in a larger data volume, the controller obtains all the data in the larger data volume and obtains part of the data therein based on the whole data. In the present application, the controller parses the secondary management header, directly reads the data in a smaller data volume, and obtains the part of the data based on the data in the smaller data volume, thereby improving the efficiency of obtaining part of the data in the larger data volume.

[0012] In another possible implementation, after the controller adds a management header to the data stored in the second storage medium, the controller receives a second request. The second request is used to indicate: obtain the data stored in the second storage medium. And the controller reads one or more to-be-written data in the second storage medium according to the first-level management header based on the second request.

[0013] In another possible implementation, after the controller adds a management header to the data stored in the second storage medium, the controller receives a second request. The second request is used to indicate: obtain the data stored in the second storage medium. And the controller reads one or more data in the second storage medium according to the secondary management header based on the second request.

[0014] In a second aspect, the present application provides a data management device, which includes: various modules for executing the method described in the first aspect or any possible design of the first aspect.

[0015] In a third aspect, the present application provides a controller. The controller includes: a control circuit and a power supply circuit. The power supply circuit supplies power to the control circuit. The control circuit is used to execute the operation steps of the method described in the first aspect or any possible design of the first aspect.

[0016] In a fourth aspect, the present application provides a storage device. The storage device includes a controller, a first storage medium, and a second storage medium. The management granularity of the first storage medium is a first granularity, and the granularity of the second storage medium is a second granularity, and the second granularity is greater than the first granularity. The controller is used to execute the operation steps of the method described in the first aspect or any possible implementation of the first aspect.

[0017] In a fifth aspect, the present application provides a computer-readable storage medium. The computer-readable storage medium includes: computer software instructions; when the computer software instructions are executed in a computing device, the computing device executes the operation steps of the method described in the first aspect or any possible implementation of the first aspect.

[0018] In a sixth aspect, the present application provides a computer program product. When the computer program product is executed on a computing device, the computing device is caused to execute the operation steps of the method described in the first aspect or any possible implementation of the first aspect.

[0019] The beneficial effects of the second to sixth aspects above can refer to the description of the first aspect or any implementation of the first aspect, and will not be repeated here. Based on the implementations provided by the above aspects, this application can also be further combined to provide more implementations. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 A schematic diagram of a data storage system provided for this application;

[0021] Figure 2 A schematic diagram of the structure of a storage device provided for this application;

[0022] Figure 3 A schematic diagram of the storage space of the storage medium after reading data provided by the present application;

[0023] Figure 4 A schematic diagram of the storage space of a storage medium after data is deleted provided by the present application;

[0024] Figure 5 A schematic diagram of the storage space of the first storage medium after data is written provided by the present application;

[0025] Figure 6 A schematic diagram of the storage space of the second storage medium after data is written provided by the present application;

[0026] Figure 7 A schematic diagram of the storage space of a storage medium after data is deleted provided by the present application;

[0027] Figure 8 A schematic diagram of garbage collection provided for this application;

[0028] Fig. 9 A flowchart of a data management method provided in this application;

[0029] Fig.10 A schematic diagram of the structure of the first management head provided in this application;

[0030] Fig.11 A schematic diagram of the structure of the second management head provided for this application;

[0031] Fig.12 A schematic diagram of the first type of data reading provided by this application;

[0032] Fig.13A schematic diagram of the second method of reading data provided by this application;

[0033] Fig.14 A schematic diagram of the structure of a data management device provided in this application. DETAILED DESCRIPTION

[0034] The present application provides a method for data management. The method includes: a controller reads data multiple times from a storage medium with a smaller management granularity (such as a first storage medium), and when the amount of data read multiple times is consistent with the management granularity of a storage medium with a larger management granularity (such as a second storage medium), the controller writes the data read multiple times into a storage space corresponding to a management granularity of the storage medium with a larger management granularity. Because the amount of data written is consistent with the storage capacity corresponding to a management granularity of the storage medium receiving the data, the storage space corresponding to a management granularity is all used to store data. This avoids the problem that the storage resources of the storage medium cannot be effectively utilized due to the fact that the data cannot completely fill the storage space corresponding to a management granularity of the storage medium receiving the data, resulting in an idle storage area in the storage space receiving the data, thereby improving the utilization rate of the storage resources of the storage medium.

[0035] In order to make the description of the following embodiments clear and concise, a brief introduction to the related technology is first given.

[0036] Figure 1 A data storage system diagram provided in this application, the data storage system 100 includes: a computing device 110 and a storage device 120. Figure 1 In the application scenario shown, users access data through applications. The computers running these applications can be called "computing devices". Computing device 110 can be a physical machine or a virtual machine. Physical computing devices include, but are not limited to: desktop computers, servers, laptops, and mobile devices.

[0037] In a possible scenario, the computing device 110 accesses the storage device 120 through a network to access data. The network may include one or more network devices, which may be devices with data forwarding functions such as routers or switches. For example, the network may include a switch 130.

[0038] In another possible scenario, the computing device 110 may also communicate with the storage device 120 via a wired connection. For example, the computing device 110 and the storage device 120 are connected via a bus. The bus may be a universal serial bus (USB), a peripheral component interconnect express (PCIe) bus, an extended industry standard architecture (EISA) bus, a unified bus (Ubus or UB), a compute express link (CXL), a cache coherent interconnect for accelerators (CCIX), etc.

[0039] like Figure 1 As shown, the storage device 120 can be a centralized storage system or a distributed storage system, which will be described below respectively.

[0040] In a first possible scenario, the storage device 120 is a centralized storage system.

[0041] The centralized storage system has the following characteristics: there is a unified entrance through which all data from external devices must pass, and the entrance is the centralized storage system engine 121. The engine 121 is the core component of the centralized storage system, and many advanced functions of the storage system are implemented in it.

[0042] like Figure 1 As shown, there may be one or more controllers in the engine 121 . Figure 1 The example of the engine 121 including one controller is used for explanation. In a possible example, if the engine 121 has multiple controllers, any two controllers may have a mirror channel to realize the function of the two controllers backing up each other, thereby avoiding the problem of the unavailability of the entire storage device 120 caused by hardware failure. It should be understood that if the engine 121 includes multiple controllers, the engine 121 may also be called an array controller of the storage device.

[0043] The engine 121 further includes a front-end interface 1211 and a back-end interface 1214. The front-end interface 1211 is used to communicate with the computing device 110, thereby providing data access services for the computing device 110. The back-end interface 1214 is used to communicate with an external memory to expand the capacity of the storage device 120. The external memory may include, but is not limited to: a hard disk, a USB flash drive, a floppy disk, a magnetic disk, an optical disk, a magnetic tape, etc. Figure 1Take the external memory device as a hard disk as an example. Figure 1 As shown, through the back-end interface 1214, the engine 121 can be connected to more hard disks, thereby forming a very large storage resource pool.

[0044] In hardware, such as Figure 1 As shown, the controller includes at least: a processor 1212 and a memory 1213. The processor 1212 is a central processing unit (CPU) for processing data access requests from outside the storage device 120 (server or other storage system), and is also used to process requests generated inside the storage device 120. Exemplarily, when the processor 1212 receives write data requests sent by the computing device 110 through the front-end interface 1211, the data in these write data requests will be temporarily stored in the memory 1213. When the total amount of data in the memory 1213 reaches a certain threshold, the processor 1212 sends the data stored in the memory 1213 to at least one of the mechanical hard disk 1221, the mechanical hard disk 1222, the solid state drive (SSD) 1223 or other hard disks 1224 through the back-end interface 1214 for persistent storage.

[0045] Memory 1213 refers to an internal memory that directly exchanges data with processor 1212. It can read and write data at any time, and at a very fast speed, and serves as a temporary data storage for an operating system or other running programs. Memory 1213 includes at least two types of memories, such as memory 1213 can be either a random access memory or a read-only memory (ROM). For example, the random access memory is a dynamic random access memory (DRAM) or a storage class memory (SCM). DRAM is a semiconductor memory, and like most random access memories (RAM), it is a volatile memory device. However, DRAM and SCM are only exemplary descriptions in this embodiment, and memory can also include other random access memories, such as static random access memory (SRAM), etc. As for read-only memory. For example, it can be a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), etc.

[0046] In addition, the memory 1213 may also be a dual in-line memory module or a dual-line memory module (DIMM), that is, a module composed of DRAM, or an SSD. In actual applications, the controller may be configured with multiple memories 1213 of the same type, or multiple memories 1213 of different types. This embodiment does not limit the number and type of the memory 1213. In addition, the memory 1213 may be configured to have a power-saving function. The power-saving function means that when the system loses power and then powers on again, the data stored in the memory 1213 will not be lost. A memory with a power-saving function is called a non-volatile memory.

[0047] The memory 1213 stores software programs, and the processor 1212 runs the software programs in the memory 1213 to manage the hard disk. For example, the hard disk is abstracted into a storage resource pool, and the storage resource pool is provided to the server in the form of a logical unit number (LUN). The LUN here is actually the hard disk seen on the server. Of course, some centralized storage systems are also file servers themselves, which can provide shared file services for the server.

[0048] like Figure 1 As shown, in the storage system 100, the engine 121 may not have a hard disk slot, and the hard disk needs to be placed in the hard disk frame 122, and the back-end interface 1214 communicates with the hard disk frame 122. The back-end interface 1214 exists in the form of an adapter card in the engine 121. Two or more back-end interfaces 1214 can be used simultaneously on one engine 121 to connect multiple hard disk frames. Alternatively, the adapter card can also be integrated on the motherboard, in which case the adapter card can communicate with the processor 1212 via the PCIe bus.

[0049] Figure 1 Only one engine 121 is shown in the figure. In actual applications, the storage device 120 may include two or more engines 121, and redundancy or load balancing is performed between the multiple engines 121.

[0050] The hard disk frame 122 includes: a control unit 1225 and a plurality of hard disks. The control unit 1225 can have various forms, which are described below respectively.

[0051] In one case, the hard disk enclosure 122 is an intelligent disk enclosure. The control unit 1225 includes a CPU and a memory. The CPU is used to perform operations such as address conversion and reading and writing data. The memory is used to temporarily store data to be written to the hard disk, or data read from the hard disk to be sent to the controller.

[0052] In another case, the control unit 1225 is a programmable electronic component. For example, a data processing unit (DPU). The DPU has the versatility and programmability of a CPU, but is more specialized and can run efficiently on network data packets, storage requests, or analysis requests. The DPU is distinguished from the CPU by a greater degree of parallelism (needing to process a large number of requests). Optionally, the DPU can also be replaced with a processing chip such as a graphics processing unit (GPU), an embedded neural-network processing unit (NPU), etc.

[0053] Normally, the number of control units 1225 can be one or more. The functions of the control unit 1225 can be offloaded to the network card 1226. In other words, in this embodiment, the hard disk frame 122 does not have a control unit 1225 inside, and the network card 1226 completes data reading and writing, address conversion and other computing functions. At this time, the network card 1226 is an intelligent network card. The network card 1226 may include: a CPU and a memory. The CPU is used to perform operations such as address conversion and reading and writing data. The memory is used to temporarily store data to be written to the hard disk, or data read from the hard disk to be sent to the controller. There is no ownership relationship between the network card 1226 and the hard disk in the hard disk frame 122, and the network card 1226 can access any hard disk in the hard disk frame 122 (such as Figure 1 The mechanical hard disk 1221, mechanical hard disk 1222, solid state disk 1223 and other hard disks 1224 are shown, so it is more convenient to expand the hard disk when the storage space is insufficient.

[0054] According to the type of communication protocol between the engine 121 and the hard disk frame 122, the hard disk frame 122 may be a serial attached small computer system interface (SAS) hard disk frame, or it may be an NVMe (Non-Volatile Memory express) hard disk frame and other types of hard disk frames. The SAS hard disk frame adopts the SAS3.0 protocol, and each frame supports 25 SAS hard disks. The engine 121 is connected to the hard disk frame 122 through the onboard SAS interface or the SAS interface module. The NVMe hard disk frame is more like a complete computer system, and the NVMe hard disk is inserted in the NVMe hard disk frame. The NVMe hard disk frame is then connected to the engine 121 through the RDMA port.

[0055] In an optional implementation, the storage device 120 is a centralized storage system with integrated disk and controller, and the storage device 120 does not have the above-mentioned hard disk frame 122, and the engine 121 is used to manage multiple hard disks connected through the hard disk slots. The function of the hard disk slots can be implemented by the back-end interface 1214.

[0056] In a second possible scenario, the storage device 120 is a distributed storage system.

[0057] In this case, the storage device 120 includes: a computing device cluster and a storage device cluster. The computing device cluster includes: one or more computing devices. In the case where the computing device cluster includes multiple computing devices, the multiple computing devices can communicate with each other. The computing device can be a server, a desktop computer, or a controller of a storage array, etc. In terms of hardware, the computing device can include: a processor, a memory, and a network card, etc. Among them, the processor is a CPU, which is used to process data access requests from outside the computing device, or requests generated inside the computing device. Exemplarily, when the processor receives a write data request sent by a user, the data in these write data requests will be temporarily stored in the memory. When the total amount of data in the memory reaches a certain threshold, the processor sends the data stored in the memory to the storage device for persistent storage. In addition, the processor is also used for data calculation or processing, such as metadata management, deduplication, data compression, virtualized storage space, and address conversion. Exemplarily, any computing device can access any storage device in the storage device cluster through a network. The storage device cluster includes: multiple storage devices. The storage device includes: one or more controllers and storage media. The network card is used to communicate with the computing device.

[0058] Combination of the above Figure 1 The data storage system provided by the present application is described below. Figure 2 A storage device provided by the present application is described. Figure 2 A schematic diagram of the structure of a storage device provided in this application, such as Figure 2 As shown, the storage device 200 may include: a controller 210 and one or more storage media. Figure 2 Take the storage device 200 including two storage media as an example. Figure 2 The storage device 200 shown includes a storage medium 221 and a storage medium 222 .

[0059] According to the storage requirements of the data, the controller 210 can design a minimum unit for the storage medium to perform data operations on the storage medium. In the present application, the minimum unit is also referred to as a storage unit. The storage capacity corresponding to the storage unit is the management granularity of the storage medium. The management granularity may include, but is not limited to: 8 kilobytes (kb), 4 megabytes (Mbyte, Mb), etc. The present application does not limit the size of the management granularity, and the user can set other sizes as needed, such as 512kb, etc. If the storage medium 221 includes storage units 1 to storage units M, and the storage capacity corresponding to storage units 1 to storage units M is a, then the management granularity of the storage medium 221 is a.

[0060] When the storage device 200 includes multiple storage media, the controller 210 may set the same management granularity for the multiple storage media, or may set different management granularities for the multiple storage media. For example, the controller 210 designs the management granularity of the storage medium 221 to be 8kb, and the management granularity of the storage medium 222 to be 4Mb.

[0061] The operations performed by the controller 210 on the storage medium may include, but are not limited to: write operations, read operations, update operations, and delete operations. For example, the storage device 200 includes: a controller, a storage medium 1, and a storage medium 2, and the management granularity of the storage medium 1 is 8kb, and the management granularity of the storage medium 2 is 4Mb. The following describes the usage of the storage space contained in the storage medium after the controller performs the above operations on the storage medium.

[0062] Scenario ①: After the controller performs a read operation on the storage medium, the usage of the storage space contained in the storage medium is not changed.

[0063] Take the controller performing a read operation on storage medium 1 as an example. Figure 3 This is a schematic diagram of the storage space of the storage medium after reading the data provided by this application. Figure 3 As shown, the storage medium 1 includes a storage unit 11, and the storage unit 11 stores data 1. After the controller reads the data 1 from the storage unit 11 of the storage medium 1, the storage unit 11 is still used to store the data 1.

[0064] Scenario ②: After the controller performs a deletion operation on the storage medium, the usage of the storage space contained in the storage medium is changed.

[0065] Take the controller deleting some data in storage medium 1 as an example for the deletion operation. Figure 4 This is a schematic diagram of the storage space of the storage medium after deleting data provided by this application. Figure 4As shown, the storage medium 1 includes a storage unit 12, and the storage unit 12 stores data 2. After the controller performs a deletion operation on the data 2 in the storage unit 12 of the storage medium 1, the storage space corresponding to the storage unit 2 is released, and the storage space corresponding to the storage unit 2 does not store data, so that the storage medium 1 forms a storage hole, and the usage of the storage space contained in the storage medium is changed. A storage hole refers to a storage space in the storage medium that does not store valid data.

[0066] Scenario ③: After the controller performs a write operation on the storage medium, the usage of the storage space contained in the storage medium varies depending on the relationship between the amount of data to be written and the available storage capacity of the storage medium, which are explained below.

[0067] In the first possible situation, the amount of data to be written is less than or equal to the available storage capacity of the destination storage medium, and the usage of the storage space contained in the storage medium will not be changed.

[0068] Figure 5 A schematic diagram of the storage space of the first storage medium after data is written provided by the present application, such as Figure 5 As shown, the storage device includes a storage medium 1 and a storage medium 3. The storage medium 1 includes a storage unit 11, and the storage unit 11 stores data 1 of a size of 8kb. The storage medium 3 includes a storage unit 31, and the available storage capacity of the storage unit 31 is 8kb. Since the amount of data 1 is equal to the available storage capacity of the storage unit 31 of the storage medium 3, at this time, the controller reads the data 1 from the storage unit 11 and writes the data 1 into the storage unit 31 of the storage medium 3.

[0069] In a second possible situation, the amount of data to be written is greater than the available storage capacity of the destination storage medium, and the usage of the storage space contained in the storage medium is changed.

[0070] Figure 6 A schematic diagram of the storage space of the second storage medium after data is written provided by the present application, such as Figure 6 As shown, the storage device includes a storage medium 1 and a storage medium 2. The storage medium 1 includes a storage unit 11, in which data 1 is stored, and the size of the data 1 is 8kb. The storage medium 2 includes a storage unit 21, and the available storage capacity of the storage unit 21 is 4kb. Since the amount of data 1 is greater than the available storage capacity of the storage unit 21, at this time, the controller performs a garbage collection operation on the storage unit 21, releases the storage space corresponding to the storage unit 21, and changes the usage of the storage space contained in the storage unit 21. For relevant content about garbage collection, please refer to the following Figure 8 The relevant description will not be repeated here.

[0071] When the controller performs a write operation on the storage medium, data can be read from one storage medium and written to another storage medium. In some possible examples, the storage medium from which the data is read is also referred to as the source storage medium, and the storage medium from which the data is written is also referred to as the destination storage medium. The present application does not limit the locations of the source storage medium and the destination storage medium. The source storage medium and the destination storage medium can be located in the same storage device or in different storage devices. The above is explained using the example that the source storage medium and the destination storage medium are located in the same storage device. In some possible situations, the source storage medium and the destination storage medium can also be located in different storage devices. At this time, the controller of the storage device containing the source storage medium reads data from the source storage medium and transfers the data to the storage device containing the destination storage medium. The controller of the storage device containing the destination storage medium writes the data to the destination storage medium.

[0072] Scenario ④: The controller performs an update operation on the storage medium to change the usage of the storage space contained in the storage medium.

[0073] Figure 7 A schematic diagram of the storage space of the storage medium after deleting data provided by this application, such as Figure 7 As shown, storage medium 1 includes storage unit 11, and storage unit 11 stores data 1. Storage medium 2 includes storage unit 21, and storage unit 21 stores data 2. Taking the controller adopting the database physical log append writing method as an example, the controller reads data 1 in storage unit 11 and writes data 1 to storage medium 2. After data 1 is written, data 2 stored in storage unit 21 becomes invalid. At this time, the controller can perform garbage collection operation on the storage space corresponding to storage unit 21 to release the storage space corresponding to storage unit 21. For relevant content about garbage collection, please refer to the following Figure 8 The relevant description will not be repeated here.

[0074] Figure 8 A schematic diagram of garbage collection provided for this application, such as Figure 8 As shown, the controller performs garbage collection on the storage medium including the following steps ① to ③.

[0075] Step 1: The controller determines the target storage unit according to the storage holes in each storage unit included in the storage medium. Exemplarily, the storage device includes storage units 1 to M, and the controller can determine the target storage unit in any of the following examples 1 to 4.

[0076] Example 1: The controller obtains the number of storage holes corresponding to each storage unit included in the storage medium, compares the number of storage holes corresponding to each storage unit with a preset number threshold, and determines that the storage unit with the number of storage holes greater than the preset number threshold is the target storage unit.

[0077] Example 2 is different from Example 1 in that the controller ranks each storage unit according to the number of storage holes contained in each storage unit and determines that the storage unit with a ranking less than a set ranking threshold is the target storage unit.

[0078] Example 3: The controller obtains the sum of storage capacities occupied by storage holes contained in each storage unit, and the controller determines that a storage unit whose sum of storage capacities is greater than a set capacity threshold is a target storage unit.

[0079] Example 4 is different from Example 3 in that: the controller ranks each storage unit according to the sum of the storage capacity occupied by the storage holes contained in each storage unit, and determines that the storage unit whose sum of the storage capacity occupied by the storage holes is less than the set capacity ranking threshold is the target storage unit

[0080] In actual applications, the quantity threshold, ranking threshold, capacity threshold and capacity ranking threshold in the above Examples 1 to 4 may be set as needed, and this application does not limit this.

[0081] Step ②: The controller reads the data stored in the target storage unit and writes the data into the reserved space. Exemplarily, the controller determines that storage unit 1 is the target storage unit. Storage unit 1 includes storage space storing data 1 to data j, storage space storing invalid data, and free storage space without data. The controller writes data 1 to data j into the reserved space.

[0082] Step ③: The controller releases the storage space corresponding to the target storage unit. Exemplarily, the controller releases the storage space of storage unit 1 storing data 1 to data j, the storage space storing invalid data, and the free storage space without data. For more descriptions of garbage collection in storage media, please refer to the general technology, which will not be repeated here.

[0083] During garbage collection, the controller needs to read all the data stored in the target storage unit and write the data to the reserved space. The amount of data read and written by the controller is positively correlated with the size of the storage unit, and the size of the storage unit is positively correlated with the management granularity corresponding to the storage medium. The larger the management granularity, the larger the storage capacity of the storage unit corresponding to the storage medium. During garbage collection, the controller needs to read and write more data. The larger the amount of data read and written, the more CPU resources are occupied and the more storage resources are consumed.

[0084] The above text describes the structure of the storage medium in conjunction with the accompanying drawings, and the utilization of the storage space contained in the storage medium after the controller performs operations such as writing to the storage medium. After the controller performs update and deletion operations on the storage medium, storage holes are caused; after the controller performs a write operation on the storage medium, the storage capacity of the destination storage medium may be insufficient, causing the controller to perform a garbage collection operation on the destination storage medium, occupying CPU resources and wasting storage resources. To this end, the present application provides a method for data management, Fig. 9 A flow chart of a data management method provided in this application, such as Fig. 9 As shown, this method can be Figure 1 The controller described is executed, and the method includes the following steps S910 to S930.

[0085] Step S910: The controller receives a first request.

[0086] The first request is used to instruct to write the data stored in the first storage medium to the second storage medium. The management granularity of the first storage medium is the first granularity, and the management granularity of the second storage medium is the second granularity, and the second granularity is greater than the first granularity. The first storage medium and the second storage medium can be located in the same storage device or in different storage devices, which is not limited in this application. The following example takes the first storage medium and the second storage medium as being located in the same storage device.

[0087] Exemplarily, the storage device includes a controller, a storage medium 1, and a storage medium 2. The management granularity of the storage medium 1 is 8kb, and the management granularity of the storage medium 2 is 4mb. The storage medium 1 includes storage units 11 to 1m, where m is a positive integer greater than or equal to 512. The storage units 11 to 1m store data 11 to 1m respectively. The storage medium 2 includes storage units 21 to 2n.

[0088] Step S920: The controller reads multiple data from the first storage medium according to the first request, and merges the multiple data according to the second granularity to obtain one or more data to be written.

[0089] The data volume of each data is consistent with the first granularity, and the data volume of each to-be-written data is consistent with the second granularity.

[0090] In a possible scenario, the controller may fuse multiple data to obtain one or more data to be written in a manner that includes, but is not limited to: a discrete aggregation table and a linked list. Taking the example of the controller using a discrete aggregation table to fuse multiple data to obtain one or more data to be written, the controller reads data 11 to data 1k in the storage medium 1. The number of data contained in data 11 to data 1k is greater than or equal to 512. The data size of each data in data 11 to data 1k is 8kb. The controller uses a discrete aggregation table to fuse data 11 to data 1k into data to be written, and obtains one or more data to be written, and the data size of each data to be written is 4mb.

[0091] In one possible scenario, after writing one or more data to be written into the second storage medium, the controller adds a management header to the data stored in the second storage medium. The management header is used to indicate: location information of the data stored in the second storage medium. The management header may include any of the following possible scenarios or a combination thereof.

[0092] In a first possible scenario, the management header includes a first-level management header.

[0093] The first-level management header is used to indicate the storage information of each data to be written in the second storage medium. The first-level management header may include, but is not limited to, a first offset value and a first flag. The first offset value is used to indicate the storage address of the data to be written in the second storage medium, and the first flag is used to indicate the data to be written. For example, the first-level management header includes the first offset value and the first flag. Fig.10 As shown, Fig.10 This is a schematic diagram of the structure of the first management header provided in the present application. Each data to be written stored in the second storage medium includes: a first offset value, a first identifier and the content of the data to be written. The code in Table 1 can be used to define a first-level management header.

[0094] Table 1

[0095]

[0096] In a second possible scenario, the management head includes a secondary management head.

[0097] The secondary management header is used to indicate the storage information of each data in the second storage medium. The secondary management header includes, but is not limited to, a second offset value and a second flag. The second offset value is used to indicate the storage address of the data in the second storage medium, and the second flag is used to indicate the data. For example, the secondary management header includes the second offset value and the second flag. Fig.11 As shown, Fig.11 This is a schematic diagram of the structure of the second management header provided in the present application. Each data stored in the second storage medium includes: a second offset value, a second identifier and the content of the data. The code in Table 2 can be used to define the secondary management header.

[0098] Table 2

[0099]

[0100] In actual applications, one of the primary management header and the secondary management header can be selected for use as needed, or the primary management header and the secondary management header can be used in combination. This application does not limit the number and combination of management headers. When using a combination of the primary management header and the secondary management header to manage data, the code in Table 3 can be added when defining the primary management header.

[0101] Table 3

[0102]

[0103] Step S930: The controller writes one or more data to be written into the second storage medium.

[0104] The controller may adopt any one of the following examples 1 to 2 to write the data to be written into the second storage medium.

[0105] Example 1: After acquiring each piece of data to be written, the controller immediately writes each piece of data to be written into the second storage medium.

[0106] Example 2: After acquiring all the data to be written, the controller writes all the data to be written into the second storage medium.

[0107] In actual applications, the method of writing the data to be written to the storage medium can be selected according to needs. The present application does not limit the method in which the controller writes the data to be written to the storage medium.

[0108] In the present application, when it is necessary to write data in a storage medium with a smaller management granularity (source storage medium) into a storage medium with a larger management granularity (destination storage medium), the controller can read multiple data in the source storage medium, and merge the multiple data into one or more data to be written, and the data volume of each data to be written is consistent with the management granularity of the destination storage medium. When it is necessary to perform an update or delete operation on the data stored in the destination storage medium, the data on which the update or delete operation is performed is also called the destination data. Since the controller updates or deletes all data of a management granularity corresponding to the destination data, no storage hole will be caused. And the data volume of the written data is consistent with the management granularity of the destination storage medium, and the garbage collection process will not be triggered, thereby avoiding occupying a large amount of CPU resources and storage resources.

[0109] In one possible scenario, after adding a management header to the data stored in the second storage medium, the controller receives a second request, and the controller reads one or more to-be-written data in the second storage medium according to the first-level management header based on the second request. The second request is used to indicate: obtaining the data stored in the second storage medium. Fig.12 The first schematic diagram of reading data provided by this application is as follows: Fig.12 As shown, the storage medium 2 stores data 1 to be written, the first offset value of the data 1 to be written is offset value 1, and the first mark is identification 1. The controller obtains the content of the data 1 to be written according to the storage address of the data 1 to be written indicated by offset value 1 and identification 1.

[0110] In one possible scenario, after adding a management header to the data stored in the second storage medium, the controller receives a second request, and the controller reads one or more data in the second storage medium according to the secondary management header based on the second request. The second request is used to indicate: obtaining the data stored in the second storage medium. Fig.13 The schematic diagram of the second data reading method provided in this application is as follows: Fig.13 As shown, the storage medium 2 stores the data 1 to be written obtained by merging the data 11 to the data 1k. The second offset value of the data 11 is the offset value 2, and the second mark is the mark 2. The controller obtains the content of the data 11 according to the storage address of the data 11 indicated by the offset value 2 and the mark 2. When it is necessary to read part of the data in a management granularity in a storage medium (destination storage medium) with a larger management granularity, the controller can use the second offset value and the second mark to obtain the data, thereby improving the hit rate of the data, shortening the time to read the data, and improving the efficiency of reading the data.

[0111] The data management method provided by the present application is described above with reference to the accompanying drawings. Fig.14 The data management device provided by the present application is described. Fig.14 A schematic diagram of the structure of a data management device provided in this application, such as Fig.14 As shown, the data management device 1400 includes: a transceiver module 1410 , a processing module 1420 , a writing module 1430 and a reading module 1440 .

[0112] The transceiver module 1410 is used to: receive a first request. The first request is used to instruct to write the data stored in the first storage medium to the second storage medium. The management granularity of the first storage medium is the first granularity, and the management granularity of the second storage medium is the second granularity, and the second granularity is greater than the first granularity. The processing module 1420 is used to: read multiple data from the first storage medium according to the first request, and merge the multiple data according to the second granularity to obtain one or more data to be written. The data volume of each data is consistent with the first granularity, and the data volume of each data to be written is consistent with the second granularity. The writing module 1430 is used to: write one or more data to be written to the second storage medium.

[0113] In a possible scenario, the processing module 1420 merges multiple data to obtain one or more data to be written in a manner including: at least one of a discrete aggregation table and a linked list.

[0114] In a possible scenario, the processing module 1420 is further used to add a management header to the data stored in the second storage medium. The management header is used to indicate location information of the data stored in the second storage medium.

[0115] In a possible scenario, the management header includes at least one of a primary management header and a secondary management header. The primary management header is used to indicate storage information of each to-be-written data in the second storage medium. The secondary management header is used to indicate storage information of each data in the second storage medium.

[0116] In a possible scenario, the first-level management header includes at least one of a first offset value and a first flag. The first offset value is used to indicate a storage address of the data to be written in the second storage medium, and the first flag is used to indicate the data to be written.

[0117] In a possible scenario, the secondary management header includes at least one of a second offset value and a second flag. The second offset value is used to indicate a storage address of data in the second storage medium, and the second flag is used to indicate data.

[0118] In a possible scenario, the transceiver module 1410 is used to receive a second request. The second request is used to instruct to obtain data stored in the second storage medium. The reading module 1440 is used to read one or more to-be-written data in the second storage medium according to the first-level management header based on the second request.

[0119] In a possible scenario, the transceiver module 1410 is further configured to: receive a second request. The second request is used to instruct: obtaining data stored in the second storage medium. The reading module 1440 is configured to: read one or more data in the second storage medium according to the secondary management header based on the second request.

[0120] It should be understood that the data management device 1400 of the embodiment of the present application can be implemented by a CPU, an ASIC, or a programmable logic device (PLD), and the PLD can be a complex programmable logical device (CPLD), an FPGA, a generic array logic (GAL), or any combination thereof. When the data management device 1400 implements the data management method shown in any of the above figures by software, the data management device 1400 and its various modules can also be software units.

[0121] A more detailed description of the above-mentioned data management device 1400 can be directly obtained by referring to the relevant description in the embodiment shown in the aforementioned drawings, and will not be repeated here.

[0122] For example, when the data management device 1400 is implemented by hardware, the hardware can be implemented by a controller, such as Figure 2 The controller described above includes a power supply circuit and a control circuit. The power supply circuit is used to supply power to the control circuit. The control circuit is used to implement any possible implementation method in the above embodiments through a logic circuit or execution code instructions. The beneficial effects can be found in the description of any aspect of the above embodiments, which will not be repeated here.

[0123] The present application also provides a storage device, which includes: a controller, a first storage medium, and a second storage medium. The management granularity of the first storage medium is a first granularity, and the management granularity of the second storage medium is a second granularity. The controller is used to control the first storage medium and the second storage medium to collaboratively implement the data management method provided in the above embodiment. As an optional implementation, in hardware implementation, the storage device may refer to a chip or chip system encapsulated with one or more controllers.

[0124] For example, when the storage device is used to implement Fig. 9 When the method shown is performed, the controller included in the computing device executes the method steps and possible sub-steps described in step S910 to step S930. In an optional scenario, the storage device may also include a communication interface, which may be used to receive the first request, the second request, etc.

[0125] It is worth noting that storage devices can also perform Fig.14 The functions of the data management device 1400 shown are not described in detail here.

[0126] It is understood that the processor in the embodiments of the present application may be a CPU, NPU or GPU, or other general-purpose processors, DSP, ASIC, FPGA or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. The general-purpose processor may be a microprocessor or any conventional processor.

[0127] The method steps in the embodiments of the present application can be implemented by hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, and the software modules can be stored in random access memory (RAM), flash memory, ROM, PROM, EPROM, electrically erasable programmable read-only memory (EEPROM), registers, hard disks, mobile hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC. In addition, the ASIC can be located in a network device or a terminal device. Of course, the processor and the storage medium can also be present in a network device or a terminal device as discrete components.

[0128] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instruction is loaded and executed on a computer, the process or function described in the embodiment of the present application is executed in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user device or other programmable device. The computer program or instruction may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer program or instruction may be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired or wireless means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium, for example, a floppy disk, a hard disk, a tape; it may also be an optical medium, for example, a digital video disc (DVD); it may also be a semiconductor medium, for example, a solid state drive (SSD).

[0129] In each embodiment of the present application, if there is no special explanation and logical conflict, the terms and / or descriptions between different embodiments are consistent and can be referenced to each other, and the technical features in different embodiments can be combined to form new embodiments according to their inherent logical relationships. The various digital numbers involved in the embodiments of the present application are only used to distinguish for the convenience of description and are not used to limit the scope of the embodiments of the present application. The size of the sequence number of the above-mentioned processes does not mean the order of execution, and the execution order of each process should be determined by its function and inherent logic.

Claims

1. A method for data management, characterized in that: The method comprises: receiving a first request, the first request being used to instruct writing data stored in a first storage medium into a second storage medium, the management granularity of the first storage medium being a first granularity, the management granularity of the second storage medium being a second granularity, and the second granularity being greater than the first granularity; Reading multiple data from the first storage medium according to the first request, and merging the multiple data according to the second granularity to obtain one or more data to be written, wherein the data volume of each data is consistent with the first granularity, and the data volume of each data to be written is consistent with the second granularity; The one or more data to be written are written into the second storage medium.

2. The method according to claim 1, characterized in that The method of fusing the multiple data to obtain one or more data to be written includes: at least one of a discrete aggregation table and a linked list.

3. The method according to claim 1 or 2, characterized in that: After writing the one or more data to be written into the second storage medium, the method further includes: A management header is added to the data stored in the second storage medium, where the management header is used to indicate: location information of the data stored in the second storage medium.

4. The method according to claim 3, characterized in that: The management header includes: at least one of a first-level management header and a second-level management header; the first-level management header is used to indicate: storage information of each data to be written in the second storage medium, and the second-level management header is used to indicate: storage information of each data in the second storage medium.

5. The method according to claim 4, characterized in that The first-level management header includes: at least one of a first offset value and a first flag, the first offset value is used to indicate: a storage address of the data to be written in the second storage medium, and the first flag is used to indicate: the data to be written.

6. The method according to claim 4, characterized in that The secondary management header includes at least one of a second offset value and a second flag, wherein the second offset value is used to indicate a storage address of the data in the second storage medium, and the second flag is used to indicate the data.

7. The method according to claim 4 or 5, characterized in that: After adding a management header to the data stored in the second storage medium, the method further includes: receiving a second request, where the second request is used to instruct: obtaining data stored in the second storage medium; Based on the second request, one or more data to be written in the second storage medium are read according to the first-level management header.

8. The method according to claim 4 or 6, characterized in that: After adding a management header to the data stored in the second storage medium, the method further includes: receiving a second request, where the second request is used to instruct: obtaining data stored in the second storage medium; One or more data in the second storage medium are read according to the secondary management header based on the second request.

9. A data management device, characterized in that: The device comprises: The transceiver module is used to: receive a first request, where the first request is used to instruct to write data stored in a first storage medium into a second storage medium, where the management granularity of the first storage medium is a first granularity, and the management granularity of the second storage medium is a second granularity, and the second granularity is greater than the first granularity; a processing module, configured to: read multiple data from the first storage medium according to the first request, and merge the multiple data according to the second granularity to obtain one or more data to be written, wherein the data volume of each data is consistent with the first granularity, and the data volume of each data to be written is consistent with the second granularity; A writing module is used to write the one or more data to be written into the second storage medium.

10. The device according to claim 9, characterized in that The method of fusing the multiple data to obtain one or more data to be written includes: at least one of a discrete aggregation table and a linked list.

11. The device according to claim 9 or 10, characterized in that The processing module is further used to add a management header to the data stored in the second storage medium, where the management header is used to indicate location information of the data stored in the second storage medium.

12. The device according to claim 11, characterized in that The management header includes: at least one of a first-level management header and a second-level management header; the first-level management header is used to indicate: storage information of each data to be written in the second storage medium, and the second-level management header is used to indicate: storage information of each data in the second storage medium.

13. The device according to claim 12, characterized in that The first-level management header includes: at least one of a first offset value and a first flag, the first offset value is used to indicate: a storage address of the data to be written in the second storage medium, and the first flag is used to indicate: the data to be written.

14. The device according to claim 12, characterized in that The secondary management header includes at least one of a second offset value and a second flag, wherein the second offset value is used to indicate a storage address of the data in the second storage medium, and the second flag is used to indicate the data.

15. The device according to claim 12 or 13, characterized in that The device further comprises: a reading module, The transceiver module is used to: receive a second request, where the second request is used to instruct: obtain data stored in the second storage medium; The reading module is used to read one or more to-be-written data in the second storage medium according to the first-level management header based on the second request.

16. The device according to claim 12 or 14, characterized in that The device further comprises: a reading module, The transceiver module is used to: receive a second request, where the second request is used to instruct: obtain data stored in the second storage medium; The reading module is used to read one or more data in the second storage medium according to the secondary management header based on the second request.

17. A controller, characterized in that: The controller includes: a control circuit and a power supply circuit, The power supply circuit is used to supply power to the control circuit; The control circuit is used to execute the method according to any one of claims 1 to 8.

18. A storage device, characterized in that: The storage device includes: a controller, a first storage medium and a second storage medium, the management granularity of the first storage medium is a first granularity, the management granularity of the second storage medium is a second granularity, and the controller is used to control the first storage medium and the second storage medium to execute the method described in any one of claims 1-8.

Images