Data management method and device and storage medium

Abstract

The invention discloses a data management method and device and a storage medium, and relates to the technical field of data storage, in the storage device, the storage device is divided according to data characteristics of storage data, and a storage space in the storage device is divided into a plurality of recovery groups; each recovery group is used as an independent physical space for data storage; the storage device performs read-write operation of data and recovery of the storage unit by taking the recovery group as a unit, and only operations such as addressing and recovery need to be performed in a specific recovery group, so that independent physical space storage and efficient management of different application data are realized through partitioning; the write amplification coefficient and the garbage collection operation frequency are effectively reduced, performance interference between different application data is avoided through performance isolation between different collection groups, the response speed of the storage device is increased, the performance and the service life of the storage device can be effectively guaranteed, and a data storage strategy is optimized.

Description

Technical Field

[0001] The present invention relates to the technical field of data storage, and in particular to a data management method, device and storage medium. Background Art

[0002] With the continuous development of computer technology, solid-state drives (SSDs) are becoming increasingly popular. Their advantages, such as high read and write speeds, low noise, low power consumption, and strong shock resistance, have rapidly made them a new favorite in the data storage market. The performance and lifespan of SSDs have become key indicators of the quality of storage systems. During the data writing process, traditional SSDs store data from different applications in the same physical space, resulting in a random distribution of data. This mixed storage method leads to an increase in invalid data, severe data fragmentation, increased write amplification, and frequent garbage collection operations, which in turn affects the SSD's write performance and lifespan.

[0003] It can be seen that how to adjust the data management method in storage devices such as solid-state drives to ensure the performance of the storage devices is a problem that needs to be solved by those skilled in the art. Summary of the Invention

[0004] The purpose of the embodiments of the present invention is to provide a data management method, device and storage medium, which can solve the problems of increased write amplification and excessive garbage collection that may affect the performance of storage devices.

[0005] To solve the above technical problems, an embodiment of the present invention provides a data management method, including:

[0006] Acquire data characteristics of all data currently stored in the storage device; the data characteristics include at least one of data life cycle, data read count, data write count, and data size;

[0007] Divide storage devices into several recycling groups based on data characteristics;

[0008] If there is data to be read or written in the storage device, the read or write operation is performed based on the recycling group corresponding to the data characteristics of the data to be read or written;

[0009] If the number of free units in a recycling group is less than the idle threshold, a unit recycling operation is performed on the recycling group whose number of free units is less than the idle threshold.

[0010] In some embodiments, storage devices are divided into several recycling groups based on data characteristics, including:

[0011] determining an average value of any data characteristic of all data currently stored in the storage device;

[0012] Based on the size relationship between the data characteristics of each data and the average value, all data currently stored in the storage device are divided into several partitions;

[0013] The free units of the storage device are divided into partitions to divide the storage device into several recycling groups.

[0014] In some embodiments, after dividing the storage device into a plurality of recycling groups based on data characteristics, the method further includes:

[0015] Configure several recycling unit handles corresponding to each recycling group;

[0016] For any recycling group, the storage blocks in each storage unit are logically numbered, and the initial position of the recycling unit handle is set to the storage block with the smallest logical number in any storage unit;

[0017] If the storage block where the recovery unit handle is located has data written to it, the recovery unit handle is controlled to move to the storage block with the next logical number;

[0018] If the recycling unit handle moves to the storage block with the largest logical number in any storage unit, and data has been written to the storage block with the largest logical number, the recycling unit handle is controlled to switch to the storage block with the smallest logical number in another storage unit in the recycling group;

[0019] Perform read and write operations based on the recycling group corresponding to the data characteristics of the data to be read or written, including:

[0020] Determine the data characteristic interval corresponding to each recovery group;

[0021] If there is data to be written, determine the data feature interval to which the data feature of the data to be written belongs, so as to determine the recycling group corresponding to the data to be written;

[0022] The data to be written is written based on the location of the recycling unit handle in the recycling group corresponding to the data to be written.

[0023] In some embodiments, further comprising:

[0024] For any recycling group, configure the status bits corresponding to the storage units in the recycling group one by one;

[0025] If there is data written in the storage unit, the corresponding status bit is configured as a valid value;

[0026] If the data in the storage unit is deleted or updated, the corresponding status bit is configured to an invalid value;

[0027] Perform unit recycling operations on recycling groups whose number of free units is less than the free threshold, including:

[0028] Determine the status bit corresponding to each storage unit in the recycling group whose number of free units is less than the free threshold;

[0029] Determine all storage units with invalid status bits as units to be recycled;

[0030] Release all cells to be recycled to complete the cell recycling operation for the recycling group whose number of free cells is less than the idle threshold.

[0031] In some embodiments, obtaining data characteristics of all data currently stored in the storage device includes:

[0032] The data characteristics of all data currently stored in the storage device are obtained every preset time period.

[0033] In some embodiments, further comprising:

[0034] Determine the amount of data that can be stored in each recycling group;

[0035] Adjust the division method of each recycling group based on the relationship between the data written amount of each recycling group and the corresponding storable data amount.

[0036] In some embodiments, before adjusting the division method of each recycling group based on the relationship between the amount of data written to each recycling group and the corresponding amount of storable data, the method further includes:

[0037] For any recycling group, configure the write cumulative value; the initial value of the write cumulative value is 0;

[0038] If there is data to be written, the amount of data to be written is obtained;

[0039] Add the amount of data to be written to the current cumulative value of the written cumulative value to obtain an updated written cumulative value;

[0040] The amount of data written to the reclaim group is determined based on the write accumulation value.

[0041] In some embodiments, adjusting the division of each recycling group based on the relationship between the amount of data written to each recycling group and the corresponding amount of storable data includes:

[0042] If the data write amount of a recycling group is greater than the corresponding storable data amount, the recycling group is determined to be a group to be expanded;

[0043] If the data write amount of a recycling group remains unchanged for a period longer than a preset time, the recycling group is determined to be a group to be reduced;

[0044] A preset number of idle units in the group to be reduced are reallocated to the group to be expanded.

[0045] To solve the above technical problems, an embodiment of the present invention further provides an electronic device, including:

[0046] memory for storing computer programs;

[0047] The processor is configured to execute a computer program to implement the steps of the aforementioned data management method.

[0048] To solve the above technical problems, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the data management method as described above are implemented.

[0049] As can be seen from the above technical solution, in a storage device, the storage device is divided according to the data characteristics of the stored data, and the storage space in the storage device is divided into several recycling groups, each recycling group being used as an independent physical space for data storage; the storage device performs data read and write operations and storage unit recycling in recycling groups, and only needs to perform addressing and recycling operations in a specific recycling group. The beneficial effect of the present invention is that it realizes the independent physical space storage and efficient management of different application data through partitioning, effectively reduces the write amplification factor and the frequency of garbage collection operations, utilizes the performance isolation between different recycling groups to avoid performance interference between different application data, improves the response speed of the storage device, can effectively ensure the performance and life of the storage device, and optimizes the data storage strategy. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the embodiments of the present invention, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0051] Figure 1 A flowchart of a data management method provided by an embodiment of the present invention;

[0052] Figure 2 A schematic diagram of partitions in a storage device provided in an embodiment of the present invention;

[0053] Figure 3 A schematic diagram of a recycling group provided in an embodiment of the present invention;

[0054] Figure 4 A schematic diagram of a process flow for processing data to be written provided by an embodiment of the present invention;

[0055] Figure 5 A schematic structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0056] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0057] The terms "including" and "having," as used in the present description and accompanying drawings, and any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or elements is not limited to the listed steps or elements and may include steps or elements that are not listed.

[0058] In order to enable those skilled in the art to better understand the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0059] Next, a data management method provided by an embodiment of the present invention is described in detail. Figure 1 As shown, Figure 1 A flowchart of a data management method provided in an embodiment of the present invention; the data management method includes:

[0060] S11: Acquire data characteristics of all data currently stored in the storage device; the data characteristics include at least one of data life cycle, data read times, data write times, and data size.

[0061] It is understood that to avoid fragmentation and scattered distribution of data during storage, for storage devices that already store data, data features can be extracted from all currently stored data in the storage device to obtain the data features of each data. This allows the currently stored data to be grouped according to the different states of the data features, and the storage device to be intelligently partitioned based on the grouping. Data features of stored data include data lifecycle, data access frequency, data size, and other types of features. The data lifecycle specifically includes the data creation time, modification time, and expected retention time. Data access frequency specifically includes the number of data reads and data writes. This application does not specifically limit the specific types and implementation methods of data features, and is not limited to this embodiment. This application does not specifically limit the methods for obtaining each data feature, and it can be directly obtained through methods such as recorded I / O (Input / Output) statistics of the storage device. This application does not specifically limit the specific types and implementation methods of storage devices, and can include SSDs (Solid State Drives) and NAND (NAND Flash Memory).

[0062] S12: Divide the storage device into several recycling groups based on data characteristics.

[0063] It is not difficult to understand that after obtaining the data characteristics of each data, all data currently stored in the storage device can be grouped based on this data characteristic. The specific grouping basis can be based on the length of the data life cycle (L), the number of data reads (R), the number of data writes (W), the different data sizes (S), etc., and then the storage device is divided into several partitions according to the data grouping situation. Dividing into several partitions means dividing the physical space of the storage device for storage into several independent areas, and each partition has an independent physical storage space. After the division is completed, the partition is combined with the FDP (Flexible Data Placement) function to define each partition as an independent RG (Reclaim Group). The implementation of the FDP function can be achieved by integrating the FDP function module in the controller of the storage device. This application does not make any special restrictions on the basis and method of partitioning. The specific number of partitions obtained by division and the specific size of each partition can be set according to actual application requirements or data characteristics.

[0064] It should be noted that a storage device is divided into multiple storage units during use. Each storage unit is an independently erasable physical unit and contains several storage blocks. When partitioning a storage device, the division can be performed on a per-storage-unit basis. That is, if a piece of data is assigned to the first partition, the storage unit containing that data is also assigned to the first partition. Each resulting partition contains several storage units. A partition is defined as a reclaim group, and a storage unit within a partition is defined as a RU (Reclaim Unit). Multiple RUs form an RG. Reclaim groups are independent of each other, so the performance and reliability of each reclaim group do not affect the other. In actual applications, multiple reclaim groups can be grouped into one partition as needed.

[0065] As a specific embodiment, taking NAND disk as an example, NAND disk will be divided into multiple LUNs for application. Therefore, after partitioning the NAND disk based on data characteristics, there will be multiple LUNs in one partition, that is, one LUN is a recycling unit, multiple LUNs constitute a recycling group, and there will be multiple blocks in one LUN. Taking solid state disk as an example, see Figure 2 As shown, Figure 2 A schematic diagram of partitions in a storage device provided in an embodiment of the present invention; the entire solid-state drive is divided into eight areas of different sizes, PART-0 to PART-7, each area serving as a recycling group, and each recycling group having independent storage space for storing data with similar data characteristics.

[0066] Furthermore, for ease of application, after the recycling group division is completed, it is necessary to determine and record the status parameters of each recycling group. The status parameters of the recycling group specifically include the amount of data that can be stored, data_size, the cumulative write value, write_size, the number of storage units contained in the recycling group, and the data feature interval corresponding to the recycling group. The amount of data that can be stored, data_size, refers to the total amount of data that can be accommodated in the recycling group, and the cumulative write value, write_size, refers to the total amount of data written to the recycling group. The data feature interval corresponding to the recycling group refers to the corresponding interval of the data characteristics of the data stored in the recycling group, including the threshold interval consisting of the maximum value Lmax and minimum value Lmin of the data life cycle, the threshold interval consisting of the maximum value Rmax and minimum value Rmin of the number of data reads, the threshold interval consisting of the maximum value Wmax and minimum value Wmin of the number of data writes, and the threshold interval consisting of the maximum value Smax and minimum value Smin of the data size. It can be set based on the data characteristics of all data divided into the recycling group, or it can be determined based on the specific classification basis when dividing the partition. For example, when the storage data with a data life cycle greater than a first preset value and less than a second preset value is divided into the first partition, the data feature threshold corresponding to the first partition includes the data life cycle threshold interval consisting of the first preset value and the second preset value.

[0067] S13: If the storage device has data to be read or written, a read or write operation is performed based on the recycling group corresponding to the data characteristics of the data to be read or written.

[0068] It is understandable that after the recovery group in the storage device is divided, if there are data that need to be read and written by the storage device in the follow-up, the read and write operations will be performed directly by the recovery group corresponding to the data. Specifically, if there are data that need to be written to the storage device, the data characteristics of the data are first obtained, then the target recovery group to which the data belongs is judged according to its data characteristics, then the idle cells are determined in the target recovery group, and the data are written into the idle cells of the target recovery group, so that data with similar characteristics are stored in the same recovery group. If there are data that need to be read from the storage device, the target recovery group to which the data belongs is first determined, and then the data is directly addressed in the target recovery group to read the data. For the specific type and implementation of read and write operations, the application is not particularly limited here, is not limited to simple read or write operations, and can also be complex read and write operations such as data update and data deletion realized by read and write.

[0069] S14: If the number of free units in a recycling group is less than the idle threshold, a unit recycling operation is performed on the recycling group whose number of free units is less than the idle threshold.

[0070] It is not difficult to understand that when recycling storage units, the unit recycling operation will also be directly performed in units of recycling groups. For any recycling group, the number of idle units therein is determined in real time or periodically. When the number of idle units is very small, a unit recycling operation (garbage collection) is performed to release the recyclable storage units to an idle state, ensuring that there are a sufficient number of idle units in the recycling group that can be used. Idle units refer to storage units that are in an idle state and not occupied. The specific value and implementation method of the idle threshold are not specifically limited in this application. The specific identification method and identification frequency of idle units, that is, whether the storage units are idle, are not specifically limited in this application.

[0071] It should be noted that when data stored in an occupied storage unit is deleted or updated, the storage unit will be defined as a to-be-reclaimed unit, and its corresponding physical space will be defined as physical space that needs to be reclaimed. If the unit recycling operation is performed on the entire storage device as a unit, as long as the number of free units in the storage device is less than the free threshold, all to-be-reclaimed units in the entire storage device will be recycled. The number of units recycled each time will be relatively large, and the storage device will need to be recycled as long as new data is written, resulting in frequent garbage collection operations, affecting the performance of the storage device. When the unit recycling operation is performed on a recycling group basis, the number of free units in each recycling group is determined independently, and the number of free units can be determined when new data is about to be written. If some recycling groups do not need to write new data, there is no need to recycle the units temporarily. Therefore, the recycling group to be used can be recycled first. In this way, the recycling amount of the unit recycling operation of the entire storage device is relatively small, the recycling efficiency is fast, and new data can also be written quickly. Therefore, when data is deleted or updated, when the corresponding physical space that needs to be recycled is identified, it is only necessary to erase and recycle the physical space related to the corresponding recycling group according to the partition information.

[0072] The present invention provides an intelligent partition management method based on the FDP function in a storage device. The method combines an intelligent partitioning strategy with the FDP function. Through the intelligent partitioning strategy, data is divided into different partitions according to data characteristics such as the data life cycle and access frequency, realizing independent physical space storage for data with different characteristics. Data addressing and movement only need to be performed within specific partitions, thereby improving the access efficiency of the storage device, ensuring efficient access to high-priority data, reducing the number of unnecessary flash memory writes, and thus effectively reducing write amplification and the amount of data moved when the storage device performs garbage collection. At the same time, the FDP function is used to manage data reading, writing, and recycling in different partitions using recycling groups as units, realizing intelligent partition storage and flexible data placement, avoiding performance interference between different applications, improving the response speed of the storage device, and further improving the performance and lifespan of the storage device. In addition, the storage device can also dynamically adjust the partitions according to application requirements, dynamically adjusting the partition size and number according to the usage of the storage device and changes in the acquired data characteristics, thereby achieving high flexibility and a wide range of applications.

[0073] In some embodiments, storage devices are divided into several recycling groups based on data characteristics, including:

[0074] determining an average value of any data characteristic of all data currently stored in the storage device;

[0075] Based on the size relationship between the data characteristics of each data and the average value, all data currently stored in the storage device are divided into several partitions;

[0076] The free units of the storage device are divided into partitions to divide the storage device into several recycling groups.

[0077] It is not difficult to understand that dividing a storage device into several recycling groups specifically includes two processes. First, a preliminary division is performed on the data already stored in the storage device. The preliminary division is to divide the storage unit where the data is located into corresponding partitions, so that all the data already stored in the storage device is divided into several partitions; then, for the free units in the storage device, the free units are allocated to each partition, thereby dividing the entire storage device into several partitions and realizing several independent recycling groups. This application does not make any special restrictions on the allocation method of the free units. Specifically, all the free units in the storage device can be evenly distributed to each partition by an average distribution method; the free units can also be allocated according to the situation of the preliminary division, such as allocating more free units to partitions with higher data access frequency. During the preliminary division, the average value of the feature can be used as the basis for partitioning, for example, data with a data life cycle greater than the average value and data with a data life cycle less than the average value are divided into two partitions; in actual application, in order to ensure the similarity of data features in the same partition, a better embodiment is to use the size relationship between multiple data features and the corresponding average values ​​as the basis for partitioning.

[0078] As a specific embodiment, taking data features including data life cycle, data read count, data write count and data size as an example, each feature has an average value, and division is performed based on the size relationship between the four data features and their corresponding average values. Based on the average value corresponding to the data life cycle, all data are divided into two groups: data life cycle is greater than the average value and data life cycle is less than the average value. Then, for the two groups, the data in one group are further divided into two groups: data read count is greater than the average value and data read count is less than the average value based on the data read count, thereby obtaining four groups of data. Then, for the four groups, the data in one group are further divided into two groups: data write count is greater than the average value and data write count is less than the average value based on the data write count, thereby obtaining eight groups of data. Then, for the eight groups, the data in one group are further divided into two groups: data size is greater than the average value and data size is less than the average value based on the data size. The final sixteen partitions are obtained, and the number of partitions can also be controlled according to actual conditions. For example, if the number of partitions is 8, the first partition is L greater than the average, R greater than the average, W greater than the average, and S greater than the average; the second partition is L less than the average, R greater than the average, W greater than the average, and S greater than the average; the third partition is L greater than the average, R less than the average, W greater than the average, and S greater than the average; the fourth partition is L greater than the average, R greater than the average, W less than the average, and S greater than the average; the fifth partition is L greater than the average, R greater than the average, W greater than the average, and S less than the average; the sixth partition is L less than the average, R less than the average, W greater than the average, and S greater than the average; the seventh partition is L less than the average, R greater than the average, W less than the average, and S greater than the average; and the eighth partition is L less than the average, R greater than the average, W greater than the average, and S less than the average.

[0079] Specifically, partitions can be divided by comparing with the average value, so that the final partitions can support data storage of all different data characteristics. When performing partition division, the occupied storage units are first divided using the data characteristics, and then the allocation number of free units is adjusted according to the actual application of each partition, so that the partitions and recycling groups obtained by division are more reasonable; data with similar data characteristics are stored in the same recycling group, which is convenient for management and application.

[0080] In some embodiments, after dividing the storage device into a plurality of recycling groups based on data characteristics, the method further includes:

[0081] Configure several recycling unit handles corresponding to each recycling group;

[0082] For any recycling group, the storage blocks in each storage unit are logically numbered, and the initial position of the recycling unit handle is set to the storage block with the smallest logical number in any storage unit;

[0083] If the storage block where the recovery unit handle is located has data written to it, the recovery unit handle is controlled to move to the storage block with the next logical number;

[0084] If the recycling unit handle moves to the storage block with the largest logical number in any storage unit, and data has been written to the storage block with the largest logical number, the recycling unit handle is controlled to switch to the storage block with the smallest logical number in another storage unit in the recycling group;

[0085] Perform read and write operations based on the recycling group corresponding to the data characteristics of the data to be read or written, including:

[0086] Determine the data characteristic interval corresponding to each recovery group;

[0087] If there is data to be written, determine the data feature interval to which the data feature of the data to be written belongs, so as to determine the recycling group corresponding to the data to be written;

[0088] The data to be written is written based on the location of the recycling unit handle in the recycling group corresponding to the data to be written.

[0089] It is understandable that the independent management of data in each partition is achieved through the RUH (Reclaim Unit Handle) in each recycling group. For each recycling group, a corresponding RUH is configured in the storage device. RUH is a logical pointer, serving as the write pointer of RU, always pointing to the RU that can currently write data, used to indicate the RU location where the current data is written. When the RU is full or needs to be recycled, the RUH location will switch to the next available RU. See Figure 3 As shown, Figure 3 A schematic diagram of a recovery group provided in an embodiment of the present invention; see Figure 4 As shown, Figure 4 A schematic diagram of a processing flow for data to be written is provided in an embodiment of the present invention; when data writing occurs, it is necessary to select the RUH of the corresponding partition to write the data to be written based on the data characteristics of the data to be written, that is, the L, R, W, and S values ​​of the data to be written. When one RU is full, the RUH will automatically switch to another RU.

[0090] It should be noted that for an RU, that is, a storage unit, there are multiple storage blocks. Therefore, to achieve more efficient data management, multiple storage blocks in a storage unit can be logically numbered. For example, a storage unit contains a total of blk_cnt storage blocks. The position of the recycling unit handle will start from the storage block with the smallest logical number and automatically switch by increasing the logical number, so that data is written starting from the storage block with the logical number blk-0 and continues until the recycling unit handle points to the storage block with the logical number blk_cnt. If data is also written to the storage block with the logical number blk_cnt, it means that the storage unit is full. At this time, the recycling unit handle will switch to the next storage unit and continue to write the input to the other storage unit starting from the storage block with the logical number blk-0 in another storage unit. Alternatively, the storage units in a recycling group can be numbered, and the initial position of the recycling unit handle can be set to the storage block with the smallest logical number in the smallest numbered storage unit. Starting from the first storage block in the first storage unit, when data is written to a storage block, the handle switches to the next logically numbered storage block. When the storage unit is full, the handle switches to the first storage block in the next logically numbered storage unit. The specific numbering sequence of each storage block and storage unit can be set and adjusted based on actual application requirements and the application of the storage device.

[0091] Furthermore, each partition or recycling group obtained by division has a corresponding data feature interval, and the data feature interval of each partition can be saved in a preset specific location, such as an area in the memory specifically for recording partition features. When there is data to be written, the data features of the data to be written are compared with each data feature interval to determine the recycling group corresponding to the data to be written, so as to find the recycling unit handle of the corresponding partition to determine the write position of the written data, so as to write the data.

[0092] Specifically, by configuring the recycling unit handle, the storage device can flexibly manage data writing in different partitions. The independent recycling unit handle and independent partition writing strategy ensure physical isolation and efficient access to data between partitions.

[0093] In some embodiments, further comprising:

[0094] For any recycling group, configure the status bits corresponding to the storage units in the recycling group one by one;

[0095] If there is data written in the storage unit, the corresponding status bit is configured as a valid value;

[0096] If the data in the storage unit is deleted or updated, the corresponding status bit is configured to an invalid value;

[0097] Perform unit recycling operations on recycling groups whose number of free units is less than the free threshold, including:

[0098] Determine the status bit corresponding to each storage unit in the recycling group whose number of free units is less than the free threshold;

[0099] Determine all storage units with invalid status bits as units to be recycled;

[0100] Release all cells to be recycled to complete the cell recycling operation for the recycling group whose number of free cells is less than the idle threshold.

[0101] It is not difficult to understand that, in order to facilitate subsequent unit recycling operations, a state table can also be configured in each recycling group. For any recycling group, its state table includes a one-to-one corresponding storage unit and a status bit of the storage unit. The status bit is used to indicate whether the value on the physical space corresponding to the storage unit is valid. If there is written storage data in the storage unit, it means that the storage unit is in an occupied state, and its corresponding status bit is configured as a valid value; if the data stored in the storage unit is deleted or updated, the status bit of the storage unit will be configured as an invalid value, indicating that the data stored in the storage unit is invalid data. For the units to be recycled whose status bits are invalid values, the storage device can restore these units to be recycled to free units through the unit recycling operation, freeing up the storage space of the recycling group. Accordingly, the number of free units in the recycling group and the determination of the units to be recycled can be specifically indicated by identifying each status bit in the state table. The specific types and implementation methods of valid and invalid values ​​are not particularly limited in this application. For example, 1 can be used as a valid value and 0 as an invalid value.

[0102] Specifically, the status table configuration can be used to indicate whether the unit recycling operation is executed and the specific storage units that need to be recycled, thereby achieving visual management of the unit recycling operation and improving the operational efficiency of the unit recycling operation.

[0103] In some embodiments, obtaining data characteristics of all data currently stored in the storage device includes:

[0104] The data characteristics of all data currently stored in the storage device are obtained every preset time period.

[0105] It can be understood that in order to improve the accuracy and reliability of data management, the partitioning operation can be set as an automatic periodic operation, that is, the data characteristics of all data currently stored in the storage device are obtained every preset time period, and after each acquisition, the operation of dividing the storage device into several recycling groups based on the data characteristics is re-executed. The specific value of the preset time period is not particularly limited in this application.

[0106] Specifically, the storage device is re-partitioned every preset time period, so as to realize automatic dynamic adjustment of the partition according to the real-time storage status of the storage device at different times, adapt to the demand fluctuation of the storage device, and dynamically respond to the business needs of the storage device at different time periods.

[0107] In some embodiments, further comprising:

[0108] Determine the amount of data that can be stored in each recycling group;

[0109] Adjust the division method of each recycling group based on the relationship between the data written amount of each recycling group and the corresponding storable data amount.

[0110] It is not difficult to understand that not only can the partition of the storage device be dynamically adjusted through periodic operations, but the dynamic adjustment of the partition can also be achieved by detecting the read-write status of the data in the storage device. The storage data volume and storage unit usage of each partition are continuously detected. The partitioning method of determining whether it is necessary to adjust the partition is determined by evaluation. The partition, that is, the partitioning method of the recycling group includes the size of each recycling group and the number of recycling groups in the storage device. The size of the recycling group refers to the amount of data that can be stored in the recycling group. Specifically, for any recycling group, the data write amount of the recycling group can be continuously detected, and then compared with the amount of data that can be stored in the recycling group. If the data write amount is about to be greater than the amount of data that can be stored, it means that the recycling group needs a larger storage space and the size of the recycling group needs to be increased. It is also possible to determine whether the size of each recycling group needs to be adjusted by other means, not limited to the method of judging by the data write amount in the present embodiment. After the partition adjustment, the state parameters related to each recycling group need to be updated.

[0111] Furthermore, in addition to adjusting the size of the recycling group, the number of recycling groups can be adjusted based on changes in data characteristics and storage device application requirements, and recycling groups can be merged or split. For example, if the number of data reads in a recycling group suddenly increases, or the number of data writes suddenly increases, the threshold interval corresponding to the number of data writes or data reads in the data characteristic interval of the recycling group can be further subdivided, and the recycling group can be split into multiple partitions based on the subdivided threshold intervals. For example, if the basis for dividing a recycling group is that the number of data writes is between a first preset value and a second preset value, a third preset value can be further set between the first preset value and the second preset value, and the number of data writes between the first preset value and the third preset value and the number of data writes between the third preset value and the second preset value can be used as the basis for splitting the recycling group into two recycling groups. For another example, as data is read and written, the number of data reads and the number of data writes also change. If the data characteristics of data in two or more recycling groups intersect, for example, if the Rmax of some data in one partition is greater than the Rmin of data in another partition, the two partitions can be merged into one, forming a partition with the same data characteristics.

[0112] Specifically, storage devices can achieve dynamic adjustment of recycling groups through continuous detection of data characteristics. Reasonable partitioning operations combined with dynamic adjustment of partitions can make full use of the storage space of storage devices, help optimize the utilization of storage resources and improve the overall performance of storage devices, and help storage devices meet high-performance storage requirements.

[0113] In some embodiments, before adjusting the division method of each recycling group based on the relationship between the amount of data written to each recycling group and the corresponding amount of storable data, the method further includes:

[0114] For any recycling group, configure the write cumulative value; the initial value of the write cumulative value is 0;

[0115] If there is data to be written, the amount of data to be written is obtained;

[0116] Add the amount of data to be written to the current cumulative value of the written cumulative value to obtain an updated written cumulative value;

[0117] The amount of data written to the reclaim group is determined based on the write accumulation value.

[0118] It is understandable that in order to facilitate real-time determination of the amount of data written by the recycling group, a write cumulative value write_size can also be configured for each recycling group. The initial value of the write cumulative value write_size is zero. When data is written to the recycling group, the write cumulative value will be increased by the amount of data written. The amount of data written each time the recycling group writes data will change the write cumulative value write_size. This application does not make any special restrictions on the specific implementation method of the write cumulative value. The data write amount can be recorded in bytes. The write cumulative value can be implemented by setting a counter in the storage device.

[0119] Specifically, by configuring a write cumulative value for each recycling group, the data writing status of each recycling group is obtained in real time, so that the storage device can dynamically adjust the partition size of the recycling group according to the data writing status of each recycling group, thereby realizing partition self-adjustment.

[0120] In some embodiments, adjusting the division of each recycling group based on the relationship between the amount of data written to each recycling group and the corresponding amount of storable data includes:

[0121] If the data write amount of a recycling group is greater than the corresponding storable data amount, the recycling group is determined to be a group to be expanded;

[0122] If the data write amount of a recycling group remains unchanged for a period longer than a preset time, the recycling group is determined to be a group to be reduced;

[0123] A preset number of idle units in the group to be reduced are reallocated to the group to be expanded.

[0124] It is not difficult to understand that when the recycling group needs to perform a data write operation, the recycling group will first receive a write command. At this time, the data has not been written yet. The cumulative write value of the recycling group will obtain the amount of data to be written at this time, and first calculate the cumulative write value of the recycling group after the data is written, so as to predict the storage space occupancy of the recycling group after the data is written. If it is found in this prediction process that the data write amount write_size of a certain recycling group is greater than its own storable data amount data_size, it means that the current remaining storage space of the recycling group cannot support the complete writing of the data. At this time, it is necessary to expand the remaining storage space of the recycling group by adjusting the partition size to support the effective and complete writing of the data.

[0125] It is understood that partition size adjustment can be achieved through diversion, whereby free cells in a recycling group with relatively large remaining storage space or relatively low remaining storage space utilization are diverted to a recycling group that needs to expand storage space, thereby improving storage space utilization. Therefore, a recycling group with a data write volume greater than the corresponding storable data volume can be identified as a group to be expanded, i.e., a recycling group awaiting storage space expansion. Then, a recycling group can be selected from other recycling groups as a group to be reduced, and space cells in the group to be reduced can be diverted to the group to be expanded, thereby expanding the partition size of the group to be expanded and reducing the partition size of the group to be reduced. The specific method for determining the group to be reduced is not specifically limited herein. The group to be reduced can be determined by determining the time over which the data write volume remains constant. When the data write volume of a recycling group remains constant for a long period of time, the free cells in that recycling group remain idle for a long period of time, resulting in low utilization. Therefore, free cells can be diverted to an expansion group to improve utilization. Alternatively, the number of free cells in other recycling groups can be determined, and the recycling group with the largest number of free cells (e.g., the largest number) can be identified as the group to be reduced. The specific value of the preset time duration can be set and adjusted based on actual application circumstances. Once it detects that the amount of data written to a recycling group is greater than the corresponding storable data volume, the storage device automatically performs a partition adjustment operation to automatically expand the partition size of the group to be expanded. After the partition adjustment is completed, the storable data volume data_size of the group to be expanded and the group to be reduced also needs to be updated.

[0126] Specifically, this application combines intelligent partitioning with FDP functions to achieve intelligent partition storage and flexible data placement, effectively reducing the write amplification factor and improving the performance and lifespan of solid-state drives. At the same time, a dynamic partition adjustment strategy is set up in the intelligent partitioning to flexibly adjust the partition size when there is insufficient remaining storage space in the recycling group. Through the intelligent partitioning strategy, FDP functional module, and dynamic partition adjustment strategy, independent physical space storage and efficient management of different application data are achieved, meeting high-performance storage requirements.

[0127] See also Figure 5 As shown, Figure 5 This is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. To solve the above technical problems, an embodiment of the present invention further provides an electronic device, including:

[0128] Memory 60, for storing computer programs;

[0129] The processor 61 is configured to execute a computer program to implement the steps of the aforementioned data management method.

[0130] The electronic device provided in this embodiment may include but is not limited to a smart phone, a tablet computer, a laptop computer, or a desktop computer.

[0131] The processor 61 may include one or more processing cores, such as a quad-core processor or an octa-core processor. The processor 61 may be implemented using at least one of the following hardware forms: a digital signal processing (DSP), a field-programmable gate array (FPGA), or a programmable logic array (PLA). The processor 61 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a central processing unit (CPU); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 61 may be integrated with a graphics processing unit (GPU), which is responsible for rendering and drawing content required to be displayed on the display screen. In some embodiments, the processor 61 may also include an artificial intelligence (AI) processor for handling computational operations related to machine learning.

[0132] The memory 60 may include one or more computer-readable storage media, which may be non-transitory. The memory 60 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In this embodiment, the memory 60 is at least used to store the following computer program 601, wherein, after the computer program is loaded and executed by the processor 61, it can implement the relevant steps of the data management method disclosed in any of the aforementioned embodiments. In addition, the resources stored in the memory 60 may also include an operating system 602 and data 603, etc., and the storage method may be temporary storage or permanent storage. Among them, the operating system 602 may include Windows, Unix, Linux, etc. The data 603 may include but is not limited to the data in the data management method, etc.

[0133] In some embodiments, the electronic device may further include a display screen 62 , an input / output interface 63 , a communication interface 64 , a power supply 65 , and a communication bus 66 .

[0134] Those skilled in the art will understand that Figure 5 The structure shown in the figure does not constitute a limitation of the electronic device, and may include more or fewer components than shown in the figure.

[0135] For descriptions of features in the electronic device provided by the embodiments of the present invention, reference may be made to the relevant descriptions of the embodiments of the data management method, which will not be described in detail here.

[0136] It is understood that if the data management method in the above-mentioned embodiment is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the current technology, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and executes all or part of the steps of the various embodiments of the present invention. The aforementioned storage medium includes: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, a magnetic disk, or an optical disk, and other media that can store program code.

[0137] To solve the above technical problems, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the data management method as described above are implemented.

[0138] For descriptions of features in the computer-readable storage medium provided in the embodiments of the present invention, reference may be made to the relevant descriptions of the embodiments of the data management method, which will not be detailed here.

[0139] An embodiment of the present invention further provides a computer program product, including a computer program / instruction, which implements the steps of the data management method described in the above embodiment when executed by a processor.

[0140] For descriptions of features in the computer program product provided by the embodiments of the present invention, reference may be made to the relevant descriptions of the embodiments of the data management method, which will not be detailed here.

[0141] To solve the above technical problems, an embodiment of the present invention further provides a data management device, comprising:

[0142] A feature acquisition unit, configured to acquire data features of all data currently stored in the storage device; the data features include at least one of data life cycle, number of data reads, number of data writes, and data size;

[0143] A partition unit, configured to divide a storage device into a number of recycling groups based on data characteristics;

[0144] A read / write unit, configured to perform read / write operations based on a recycling group corresponding to data characteristics of the data to be read / written if there is data to be read / written in the storage device;

[0145] The recycling unit is used to perform a unit recycling operation on the recycling group in which the number of free units is less than the idle threshold if the number of free units in the recycling group is less than the idle threshold.

[0146] In some embodiments, the partition unit includes:

[0147] an average value calculation unit, configured to determine an average value of any data feature of all data currently stored in the storage device;

[0148] A first partitioning unit is configured to partition all data currently stored in the storage device into a plurality of partitions based on a size relationship between a data feature of each data and an average value;

[0149] The second partitioning unit is configured to partition the free units of the storage device into partitions, so as to divide the storage device into a plurality of recycling groups.

[0150] In some embodiments, further comprising:

[0151] A handle configuration unit configures a number of recycling unit handles corresponding to each recycling group;

[0152] A numbering unit is used to logically number the storage blocks in each storage unit in any recycling group, and set the initial position of the recycling unit handle to the storage block with the smallest logical number in any storage unit;

[0153] A first control unit is configured to control the recycling unit handle to move to a storage block with a next logical number if data has already been written to the storage block where the recycling unit handle is located;

[0154] A second control unit is configured to control the recycling unit handle to switch to a storage block with a smallest logical number in another storage unit in the recycling group if the recycling unit handle moves to the storage block with the largest logical number in any storage unit and data has been written to the storage block with the largest logical number;

[0155] The read-write unit includes:

[0156] An interval determination unit, used to determine the data feature interval corresponding to each recycling group;

[0157] a target recycling group determining unit, configured to determine, if there is data to be written, a data feature interval to which the data feature of the data to be written belongs, so as to determine a recycling group corresponding to the data to be written;

[0158] The writing unit is used to write the data to be written based on the location of the recycling unit handle in the recycling group corresponding to the data to be written.

[0159] In some embodiments, further comprising:

[0160] A state configuration unit, configured to configure, for any recycling group, a state bit corresponding one-to-one to a storage unit in the recycling group;

[0161] A first configuration unit is configured to configure a corresponding status bit to a valid value if data is written into the storage unit;

[0162] A second configuration unit is configured to configure a corresponding status bit to an invalid value if the data in the storage unit is deleted or updated;

[0163] The recycling unit includes:

[0164] A state determination unit, configured to determine a state bit corresponding to each storage unit in a recycling group in which the number of free units is less than an idle threshold;

[0165] A recyclable determination unit, configured to determine all storage units with invalid status bits as units to be recycled;

[0166] The recycling subunit is used to release all cells to be recycled to complete the cell recycling operation of the recycling group whose number of free cells is less than the idle threshold.

[0167] In some embodiments, the feature acquisition unit includes:

[0168] The feature acquisition subunit is used to acquire data features of all data currently stored in the storage device every preset time period.

[0169] In some embodiments, further comprising:

[0170] a storable data volume determination unit, configured to determine the storable data volume of each recycling group;

[0171] The partition adjustment unit is used to adjust the partitioning method of each recycling group according to the relationship between the data writing amount of each recycling group and the corresponding storable data amount.

[0172] In some embodiments, further comprising:

[0173] A write configuration unit is used to configure a write cumulative value for any recycling group; wherein the initial value of the write cumulative value is 0;

[0174] A write data amount acquisition unit, configured to acquire the amount of data to be written if there is data to be written;

[0175] an accumulation unit, configured to perform an addition operation on the amount of data to be written and the current accumulated value of the accumulated value of writing to obtain an updated accumulated value of writing;

[0176] The write amount determining unit is configured to determine the data write amount of the recycling group based on the write cumulative value.

[0177] In some embodiments, the partition adjustment unit includes:

[0178] an expansion determination unit, configured to determine a recycling group as a group to be expanded if the amount of data written to the recycling group is greater than the corresponding amount of storable data;

[0179] a reduction determining unit, configured to determine a recycling group as a group to be reduced if the amount of data written to the recycling group remains unchanged for a period greater than a preset time period;

[0180] The partition adjustment subunit is used to reallocate a preset number of idle units in the group to be reduced to the group to be expanded.

[0181] For descriptions of features of the data management device provided in the embodiments of the present invention, reference may be made to the relevant descriptions of the embodiments of the data management method, which will not be described in detail here.

[0182] The above describes in detail a data management method, device, and storage medium provided by embodiments of the present invention. The various embodiments are described in a progressive manner throughout this specification, with each embodiment focusing on its differences from other embodiments. Reference can be made to the descriptions of the embodiments for similar or identical parts. The devices disclosed in the embodiments are described briefly because they correspond to the methods disclosed in the embodiments. For relevant details, refer to the description of the methods.

[0183] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0184] The above is a detailed introduction to a data management method, device, and storage medium provided by the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only intended to help understand the method and core ideas of the present invention. It should be pointed out that, for those skilled in the art, without departing from the principles of the present invention, several improvements and modifications may be made to the present invention, and these improvements and modifications also fall within the scope of protection of the present invention.

Claims

1. A data management method, characterized in that: include: Acquire data characteristics of all data currently stored in the storage device; the data characteristics include at least one of data life cycle, data read count, data write count, and data size; dividing the storage device into a plurality of recycling groups based on the data characteristics; If the storage device has data to be read or written, performing a read or write operation based on the recycling group corresponding to the data characteristics of the data to be read or written; If the number of free units in a recycling group is less than the idle threshold, a unit recycling operation is performed on the recycling group whose number of free units is less than the idle threshold.

2. The data management method according to claim 1, wherein: Dividing the storage device into a plurality of recycling groups based on the data characteristics, including: determining an average value of any data characteristic of all data currently stored in the storage device; Based on the size relationship between the data characteristics of each data and the average value, all data currently stored in the storage device are divided into a plurality of partitions; The free units of the storage device are divided into the partitions, so as to divide the storage device into a plurality of recycling groups.

3. The data management method according to claim 2, characterized in that: After dividing the storage device into a plurality of recycling groups based on the data characteristics, the method further includes: Configuring a plurality of recycling unit handles corresponding one to one to each of the recycling groups; For any recycling group, logically number the storage blocks in each storage unit, and set the initial position of the recycling unit handle to the storage block with the smallest logical number in any storage unit; If data has been written to the storage block where the recycling unit handle is located, control the recycling unit handle to move to the storage block with the next logical number; If the recycling unit handle moves to the storage block with the largest logical number in any storage unit, and data has been written to the storage block with the largest logical number, then control the recycling unit handle to switch to the storage block with the smallest logical number in another storage unit in the recycling group; Performing a read / write operation based on a recycling group corresponding to the data characteristics of the data to be read / written includes: Determining the data characteristic interval corresponding to each of the recycling groups; If there is data to be written, determining the data feature interval to which the data feature of the data to be written belongs, so as to determine the recycling group corresponding to the data to be written; The data to be written is written based on the location of the recycling unit handle in the recycling group corresponding to the data to be written.

4. The data management method according to claim 1, wherein: Also includes: For any recycling group, configure the status bits corresponding to the storage units in the recycling group one by one; If there is data written in the storage unit, the corresponding status bit is configured as a valid value; If the data in the storage unit is deleted or updated, the corresponding status bit is configured to an invalid value; Perform unit recycling operations on recycling groups whose number of free units is less than the free threshold, including: Determine the status bit corresponding to each storage unit in the recycling group whose number of free units is less than the free threshold; Determine all storage units with invalid status bits as units to be recycled; All the units to be recycled are released to complete the unit recycling operation of the recycling group whose number of free units is less than the idle threshold.

5. The data management method according to claim 1, wherein: Get the data characteristics of all data currently stored in the storage device, including: The data characteristics of all data currently stored in the storage device are obtained every preset time period.

6. The data management method according to any one of claims 1 to 5, characterized in that: Also includes: Determining the amount of data that can be stored in each of the recycling groups; The division method of each recycling group is adjusted according to the relationship between the data writing amount of each recycling group and the corresponding storable data amount.

7. The data management method according to claim 6, characterized in that: Before adjusting the division method of each recycling group according to the relationship between the data written amount of each recycling group and the corresponding storable data amount, the method further includes: For any recycling group, configure a write cumulative value; wherein the initial value of the write cumulative value is 0; If there is data to be written, the amount of data to be written is obtained; Adding the amount of data to be written to the current cumulative value of the write cumulative value to obtain an updated write cumulative value; The data write amount of the reclaim group is determined based on the write cumulative value.

8. The data management method according to claim 6, characterized in that: Adjusting the division method of each recycling group according to the relationship between the data written amount of each recycling group and the corresponding storable data amount includes: If the data write amount of a recycling group is greater than the corresponding storable data amount, the recycling group is determined to be a group to be expanded; If the data write amount of a recycling group remains unchanged for a period longer than a preset time, the recycling group is determined to be a group to be reduced; A preset number of idle units in the group to be reduced are reallocated to the group to be expanded.

9. An electronic device, characterized in that: include: memory for storing computer programs; A processor, configured to execute the computer program to implement the steps of the data management method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the data management method according to any one of claims 1 to 8 are implemented.

Images