A data migration method, system, device and storage medium

Abstract

The application discloses a data migration method, system, device and storage medium, and is applied to a data storage system. The data storage system comprises an SSD storage unit and an HDD storage unit. The data migration method comprises the following steps: acquiring the size, creation time and storage capacity usage of the SSD storage unit; updating the data migration task list of the SSD storage unit based on the size, creation time and storage capacity usage of each file in the SSD storage unit; and migrating the target file in the SSD storage unit to the HDD storage unit based on the updated data migration task list. The application can effectively reduce the influence of small files on the HDD storage unit read bandwidth of the data storage system, and reduce the probability of small files being read after entering the HDD storage unit of the data storage system.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data storage, and in particular to a data migration method, system, device and storage medium. BACKGROUND

[0002] Currently, the mainstream storage media are DRAM (Dynamic Random Access Memory), SSD (Solid State Drive) and HDD (Hard Disk Drive), which differ in cost and read-write speed. In order to balance performance and cost, most data storage systems consider using two storage media at the same time, i.e. multi-media hybrid storage, which uses flash storage (such as SSD hard disk) and higher cost-effective storage media (HDD hard disk) as the media of the data storage system.

[0003] Considering the feature that the throughput of HDD hard disk increases with the increase of the size of the read block, the IO read-write request of small files on the HDD hard disk will affect the overall throughput of the hard disk, which requires developers to reduce the IO read-write request of small files on the HDD hard disk as much as possible. The SSD hard disk has the characteristics of high IOPS (Input / Output Operations Per Second) and good performance per unit time. Therefore, the common practice of hybrid storage is to store user data on the SSD hard disk and periodically compress data to the HDD hard disk through the background, i.e. to migrate the data stored on the SSD hard disk to the HDD hard disk through data migration, so as to reduce the storage space occupied by the data.

[0004] Currently, in the data storage system, data migration is usually performed according to data heat or without distinction according to the time of creation. In the data migration method according to data heat, in order to determine the data heat, the number of data read needs to be recorded in real time. Some data storage systems, such as the Ceph distributed data storage system, cannot record the number of file read, so an additional storage system is needed to record the heat file. In the data migration method without distinction according to the time of creation, many small files are also migrated to the HDD hard disk, and due to the defect of seek, small file read will cause the bandwidth of the HDD hard disk to decrease. In addition, relevant research shows that the probability of accessing many data after a period of time in the data storage system is relatively low.

[0005] Therefore, how to reduce the influence of small files on the read bandwidth of the HDD storage unit of the data storage system and reduce the probability of small files being read after entering the HDD storage unit of the data storage system is a problem to be solved at present. SUMMARY

[0006] To solve the above technical problems, the present application provides a data migration method, which can effectively reduce the influence of small files on the read bandwidth of the HDD storage unit of the data storage system and reduce the probability of small files being read after entering the HDD storage unit of the data storage system. The present application also provides a data migration system, a device and a storage medium, which have the same technical effect.

[0007] The first object of the present application is to provide a data migration method.

[0008] The above-mentioned application object of the present application is achieved by the following technical scheme:

[0009] A data migration method applied to a data storage system, wherein the data storage system comprises an SSD storage unit and an HDD storage unit, and the data migration method comprises:

[0010] S1, obtaining the size, creation time of each file in the SSD storage unit and the storage capacity utilization rate of the SSD storage unit;

[0011] S2, updating the data migration task list of the SSD storage unit based on the size, creation time of each file in the SSD storage unit and the storage capacity utilization rate of the SSD storage unit;

[0012] S3, migrating the target file in the SSD storage unit to the HDD storage unit based on the updated data migration task list.

[0013] Preferably, the S2 comprises:

[0014] sequentially judging whether each file in the SSD storage unit meets the first preset migration condition based on the size, creation time of each file in the SSD storage unit and the storage capacity utilization rate of the SSD storage unit, and inserting the metadata corresponding to the file meeting the first preset migration condition into the data migration task list.

[0015] Preferably, the sequentially judging whether each file in the SSD storage unit meets the first preset migration condition based on the size, creation time of each file in the SSD storage unit and the storage capacity utilization rate of the SSD storage unit and inserting the metadata corresponding to the file meeting the first preset migration condition into the data migration task list comprises:

[0016] S21, sequentially judging whether the size of each file in the SSD storage unit is less than or equal to a first size threshold, if yes, executing step S22, if no, inserting the metadata of the corresponding file into the data migration task list;

[0017] S22, sequentially judging whether the creation time of each file is before a preset time, if yes, inserting the metadata of the corresponding file into the data migration task list, if no, executing step S23;

[0018] S23, sequentially judging whether the size of each file is greater than or equal to a second size threshold, if yes, executing step S24, if no, executing step S25, wherein the second size threshold is less than the first size threshold;

[0019] S24, judging whether the storage capacity usage rate of the SSD storage unit is higher than a first usage rate threshold, if yes, inserting the metadata corresponding to the current file into the data migration task list, if no, judging the next file;

[0020] S25, judging whether the storage capacity usage rate of the SSD storage unit is higher than a second usage rate threshold, if yes, inserting the metadata corresponding to the current file into the data migration task list, if no, judging the next file, wherein the second usage rate threshold is greater than the first usage rate threshold.

[0021] Preferably, the S3 comprises:

[0022] S31, obtaining a data migration task from the updated data migration task list, wherein the data migration task comprises the metadata of at least one target file;

[0023] S32, obtaining the storage capacity usage rate of the SSD storage unit, the current network bandwidth, and the storage capacity usage rate of the HDD storage unit;

[0024] S33, sequentially judging whether the storage capacity usage rate of the SSD storage unit, the current network bandwidth, and the storage capacity usage rate of the HDD storage unit satisfy a second preset migration condition, if yes, migrating the corresponding target file to the HDD storage unit based on the metadata of the at least one target file in the data migration task.

[0025] Preferably, the S33 comprises:

[0026] S331, determine whether the storage capacity utilization rate of the SSD storage unit is higher than the third utilization rate threshold. If yes, migrate the corresponding target file to the HDD storage unit based on the metadata of the at least one target file in the data migration task. If no, execute step S332.

[0027] S332, determine whether the current network bandwidth has reached the target network bandwidth; if so, proceed to step S333.

[0028] S333, determine whether the storage capacity utilization rate of the HDD storage unit is lower than the fourth utilization rate threshold. If so, migrate the corresponding target file to the HDD storage unit based on the metadata of the at least one target file in the data migration task.

[0029] Preferably, migrating the corresponding target file to the HDD storage unit based on the metadata of at least one target file in the data migration task includes:

[0030] Based on the metadata of at least one target file in the data migration task, the corresponding target file is read from the SSD storage unit;

[0031] Copy the target file from the SSD storage unit to the HDD storage unit;

[0032] Delete the target file that has been successfully copied from the SSD storage unit;

[0033] Remove the data migration task corresponding to the target file that has been successfully deleted from the data migration task list.

[0034] Preferably, obtaining the size and creation time of each file in the SSD storage unit includes:

[0035] Iterate through the file list of all files in the SSD storage unit;

[0036] Read the size and creation time information of each file from the file list.

[0037] The second objective of this application is to provide a data migration system.

[0038] The second objective of this application is achieved through the following technical solution:

[0039] A data migration system is applied to a data storage system, the data storage system including SSD storage units and HDD storage units, the data migration system comprising:

[0040] The information acquisition module is used to acquire the size, creation time, and storage capacity utilization rate of each file in the SSD storage unit.

[0041] The task list update module is used to update the data migration task list of the SSD storage unit based on the size and creation time of each file in the SSD storage unit and the storage capacity utilization of the SSD storage unit.

[0042] The file migration module is used to migrate target files from the SSD storage unit to the HDD storage unit based on the updated data migration task list.

[0043] Preferably, the task list update module is specifically used for:

[0044] The system sequentially determines whether the size, creation time, and storage capacity utilization of each file in the SSD storage unit meet the first preset migration conditions. If so, the metadata corresponding to the file that meets the first preset migration conditions is inserted into the data migration task list.

[0045] Preferably, the step of sequentially determining whether the size, creation time, and storage capacity utilization of each file in the SSD storage unit meet the first preset migration condition, and if so, inserting the metadata corresponding to the files that meet the first preset migration condition into the data migration task list includes:

[0046] S21, sequentially determine whether the size of each file in the SSD storage unit is less than or equal to the first size threshold. If yes, proceed to step S22. If no, insert the metadata of the corresponding file into the data migration task list.

[0047] S22, sequentially determine whether the creation time of each file is before the preset time. If yes, insert the metadata of the corresponding file into the data migration task list. If not, execute step S23.

[0048] S23, sequentially determine whether the size of each file is greater than or equal to the second size threshold. If yes, proceed to step S24; otherwise, proceed to step S25, wherein the second size threshold is less than the first size threshold.

[0049] S24, determine whether the storage capacity utilization rate of the SSD storage unit is higher than the first utilization rate threshold. If yes, insert the metadata corresponding to the current file into the data migration task list. If no, determine the next file.

[0050] S25, determine whether the storage capacity utilization rate of the SSD storage unit is higher than the second utilization rate threshold. If yes, insert the metadata corresponding to the current file into the data migration task list. If no, determine the next file, wherein the second utilization rate threshold is greater than the first utilization rate threshold.

[0051] Preferably, the file migration module performs the migration of target files from the SSD storage unit to the HDD storage unit based on the updated data migration task list, specifically including the following steps:

[0052] S31, Obtain a data migration task from the updated data migration task list, wherein the data migration task includes metadata of at least one target file;

[0053] S32, obtain the storage capacity utilization rate of the SSD storage unit, the current network bandwidth, and the storage capacity utilization rate of the HDD storage unit;

[0054] S33, sequentially determine whether the storage capacity utilization rate of the SSD storage unit, the current network bandwidth, and the storage capacity utilization rate of the HDD storage unit meet the second preset migration condition. If so, migrate the corresponding target file to the HDD storage unit based on the metadata of the at least one target file in the data migration task.

[0055] Preferably, S33 includes:

[0056] S331, determine whether the storage capacity utilization rate of the SSD storage unit is higher than the third utilization rate threshold. If yes, migrate the corresponding target file to the HDD storage unit based on the metadata of the at least one target file in the data migration task. If no, execute step S332.

[0057] S332, determine whether the current network bandwidth has reached the target network bandwidth; if so, proceed to step S333.

[0058] S333, determine whether the storage capacity utilization rate of the HDD storage unit is lower than the fourth utilization rate threshold. If so, migrate the corresponding target file to the HDD storage unit based on the metadata of the at least one target file in the data migration task.

[0059] Preferably, migrating the corresponding target file to the HDD storage unit based on the metadata of at least one target file in the data migration task includes:

[0060] Based on the metadata of at least one target file in the data migration task, the corresponding target file is read from the SSD storage unit;

[0061] Copy the target file from the SSD storage unit to the HDD storage unit;

[0062] Delete the target file that has been successfully copied from the SSD storage unit;

[0063] Remove the data migration task corresponding to the target file that has been successfully deleted from the data migration task list.

[0064] Preferably, obtaining the size and creation time of each file in the SSD storage unit includes:

[0065] Iterate through the file list of all files in the SSD storage unit;

[0066] Read the size and creation time information of each file from the file list.

[0067] The third objective of this application is to provide an electronic device.

[0068] The aforementioned objective three of this application is achieved through the following technical solution:

[0069] An electronic device, comprising:

[0070] The data migration method described in any one of the first objectives of this application includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the data migration method described in any one of the first objectives of this application.

[0071] The fourth objective of this application is to provide a computer-readable storage medium.

[0072] The fourth objective of this application is achieved through the following technical solution:

[0073] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the data migration method described in any one of the first objectives of this application.

[0074] In summary, this application discloses a data migration method, system, device, and storage medium. When data needs to be migrated from an SSD storage unit to an HDD storage unit in a data storage system, the method first obtains the size, creation time, and storage capacity utilization of each file in the SSD storage unit; then, it updates the data migration task list of the SSD storage unit based on the size, creation time, and storage capacity utilization of each file; finally, it migrates the target files from the SSD storage unit to the HDD storage unit based on the updated data migration task list. This application comprehensively considers the size, creation time, and storage capacity utilization of the migrated files, thus prioritizing the migration of large files and files with longer creation times when migrating files from the SSD storage unit to the HDD storage unit, while also considering the storage capacity utilization status of the SSD storage unit. This effectively reduces the impact of small files on the read bandwidth of the HDD storage unit in the data storage system, lowers the probability of small files being read after entering the HDD storage unit, and ensures that the storage capacity of the SSD storage unit is in a healthy state. Attached Figure Description

[0075] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0076] Figure 1 A graph showing the relationship between the throughput of random read / write operations on an HDD and the block size during a single read / write operation;

[0077] Figure 2 A graph showing the relationship between the throughput of random read / write operations on an SSD and the block size during a single read / write operation;

[0078] Figure 3 This is a flowchart illustrating a data migration method according to an embodiment of this application;

[0079] Figure 4 This is a flowchart illustrating the process of updating the data migration task list of SSD storage units in an embodiment of this application.

[0080] Figure 5 This is a schematic diagram illustrating the data migration process based on the updated data migration task list in an embodiment of this application.

[0081] Figure 6 This is a schematic diagram of the structure of a data migration system according to an embodiment of this application;

[0082] Figure 7 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0083] To enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be clearly and completely described below. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0084] In the embodiments provided in this application, it should be understood that the disclosed methods and systems can be implemented in other ways. The system embodiments described below are merely illustrative. For example, the division of units and modules is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or modules can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, and can be electrical, mechanical, or other forms.

[0085] In addition, each functional unit in the various embodiments of this application can be integrated into a single processor, or each unit can be a separate device, or two or more units can be integrated into a single device; each functional unit in the various embodiments of this application can be implemented in hardware or in the form of hardware plus software functional units.

[0086] Those skilled in the art will understand that all or part of the steps of the following method embodiments can be implemented by program instructions and related hardware. The aforementioned program instructions can be stored in a computer-readable storage medium. When the program instructions are executed, they perform the steps of the following method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), magnetic disks, or optical disks.

[0087] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "a plurality of" or "several" means two or more, unless otherwise explicitly specified.

[0088] 1. Research on the characteristics of HDD storage media:

[0089] Hard drive service time: This refers to the time required to complete a single read / write request, primarily composed of three parts: seek time (A), rotational latency (X), and data transfer time (T). Data transfer time is on the order of microseconds and is negligible compared to the other two latencies (seek time and rotational latency). The time spent on each part typically varies depending on the hard drive's specifications. One important metric for evaluating hard drive performance is IOPS (Input / Output Per Second), which is the number of read / write operations per second. The formula for calculating IOPS is as follows (Formula...).

[0090] As shown in Equation 1), the higher the IOPS of the hard drive, the better the hard drive performance.

[0091]

[0092] Table 1 shows the average latency and IOPS for three hard drive specifications: 7200rpm, 10000rpm, and 15000rpm.

[0093] Table 1. Average time and IPOS for different hard drive specifications

[0094] Hard disk specifications Average seek time Average rotational latency IOPS 7200 rpm 9 ms 4.17 ms 76 10000 rpm 6 ms 3 ms 111 15000 rpm 4 ms 2 ms 166

[0095] This study selected a Western Digital HDD (model WUH721818ALE6L4) for performance testing. The random write and random read throughput, IOPS, and latency were tested under different block sizes (the size of a single data block requested). The testing tool used was fio. Table 2 shows the actual random read and write performance of the HDD under different block sizes. It can be observed that the larger the single data block size in the read / write operation, the higher the throughput, but the higher the latency. In this study, the maximum random write performance of the HDD reached 256.0 MB / s, and the random read performance reached 240.0 MB / s.

[0096] Table 2 Random Read / Write Performance of HDD Hard Drives at Different Block Sizes

[0097]

[0098]

[0099] The relationship between HDD random read / write throughput and the block size in a single read / write operation is shown in the graph below. Figure 1 As shown, in the random read performance test, the throughput improvement from 128KB to 1M random reads is significant. Therefore, IOPS amplification will seriously affect the throughput of HDD hard drives. These test results will also serve as the theoretical basis for setting the data stripe size and prefetch data block size of the HDD hard drive in this application's research on data storage system.

[0100] 2. Research on the characteristics of SSD storage media

[0101] Early SSDs used SATA or SAS interfaces, both with a theoretical maximum bandwidth of only a few hundred MB / s. HDDs could meet their physical maximum bandwidth requirements using these protocols. However, for SSDs, this protocol became ineffective. To fully utilize the hardware bandwidth capabilities of SSDs, PCIe interface hard drives were introduced, and based on the PCIe interface, NVMe (Non-Volatile Memory Express) was developed. NVMe-based SSDs feature high bandwidth and low latency. This application investigated the throughput performance of Samsung's MZQL21T9HCJR-00B7C SSD at different block sizes. Table 3 shows the actual random read / write performance of the SSD at different block sizes. It can be seen that this model achieves a maximum random write throughput of 2656MB / s and a maximum random write throughput of 3280MB / s. The SSD's read / write latency can reach the hundreds of microseconds level, and with the use of technologies such as SPDK and io uring, the latency may reach the 10 microseconds level.

[0102] Table 3. Random read / write performance of SSDs at different block sizes.

[0103]

[0104]

[0105] The relationship between SSD random read / write throughput and block size in a single read / write operation is shown in the graph below. Figure 2As shown, in the random read performance test, the throughput improvement was significant from 4KB to 65KB. SSDs also exhibited a trend where larger block sizes resulted in higher read / write bandwidth throughput. It is worth noting that once the SSD block size reached 4MB, the increase in read / write bandwidth throughput tended to plateau as the block size increased. These test results will serve as the theoretical basis for setting the data stripe size and prefetch data block size of the SSD in the data storage system in this application.

[0106] like Figure 3 As shown in the figure, this application provides a data migration method applied to a data storage system, the data storage system including SSD storage units and HDD storage units, the data migration method may include the following steps:

[0107] S1, obtain the size, creation time, and storage capacity utilization of each file in the SSD storage unit;

[0108] When it is necessary to migrate files stored in an SSD storage unit to an HDD storage unit, the size, creation time, and storage capacity utilization of each file in the SSD storage unit must first be obtained. This information is then used to comprehensively consider the size, creation time, and storage capacity utilization of the SSD storage unit to perform the data migration of the corresponding files in the SSD storage unit.

[0109] Specifically, in this embodiment, the size and creation time of each file in the SSD storage unit can be obtained using the following method:

[0110] First, iterate through the file list of all files in the SSD storage unit; then, read the file attribute information such as the size and creation time of each file from the file list.

[0111] Specifically, the storage capacity utilization rate of an SSD storage unit can be obtained from a monitoring system that monitors the usage status of each storage unit in the data storage system. In essence, the storage capacity utilization rate of an SSD storage unit refers to the ratio of the used storage capacity of each SSD storage unit in the data storage system to the total storage capacity of the SSD storage units.

[0112] It should be noted that an SSD storage unit refers to a data storage unit that uses an SSD hard drive to store data, and an SSD storage unit can be composed of one or more SSD hard drives as needed; an HDD storage unit refers to a data storage unit that uses an HDD hard drive to store data, and an HDD storage unit can be composed of one or more HDD hard drives as needed.

[0113] S2 updates the data migration task list of the SSD storage unit based on the size and creation time of each file in the SSD storage unit and the storage capacity utilization of the SSD storage unit;

[0114] After obtaining the size, creation time, and storage capacity utilization of each file in the SSD storage unit, the data migration task list needs to be updated by comprehensively considering the size, creation time, and storage capacity utilization of the files to be migrated. That is, the corresponding data migration tasks are generated and the data migration task list of the SSD storage unit is updated based on the size, creation time, and storage capacity utilization of each file in the SSD storage unit.

[0115] Specifically, in this embodiment, the size, creation time, and storage capacity utilization of each file in the SSD storage unit can be used to determine whether each file in the SSD storage unit meets the first preset migration condition. The metadata corresponding to the file that meets the first preset migration condition is then inserted into the data migration task list, thereby completing the update of the data migration task list.

[0116] S3 migrates target files from SSD storage units to HDD storage units based on the updated data migration task list.

[0117] After updating the data migration task list of the SSD storage unit, the target files in the SSD storage unit are finally migrated to the HDD storage unit according to the updated data migration task list, thus completing the data migration of the target files from the SSD storage unit to the HDD storage unit in the data storage system.

[0118] In summary, the above embodiments, by comprehensively considering the size of the migrated files, their creation time, and the storage capacity utilization of the SSD storage units, can prioritize the migration of large files and files with longer creation times when migrating files stored in SSD storage units to HDD storage units. Simultaneously, the storage capacity utilization status of the SSD storage units is considered, thereby effectively reducing the impact of small files on the read bandwidth of the HDD storage units in the data storage system, lowering the probability of small files being read after entering the HDD storage units of the data storage system, and ensuring that the storage capacity of the SSD storage units remains healthy.

[0119] like Figure 4 As shown, based on the above embodiments, step S2 may specifically include the following steps:

[0120] S21, sequentially determine whether the size of each file in the SSD storage unit is less than or equal to the first size threshold. If yes, proceed to step S22. If no, insert the metadata of the corresponding file into the data migration task list.

[0121] Based on the test results of the HDD and SSD storage media characteristic studies in this application, it is known that both HDDs and SSDs exhibit a trend where the larger the block size requested in a single transaction, the higher the read / write bandwidth throughput. Therefore, during data migration, to reduce the impact of small files on the read bandwidth of HDD storage units in the data storage system, large files are first selected. HDD storage units prioritize reading and migrating large files. That is, when the size of a single file in an SSD storage unit does not meet the condition of being less than or equal to a first size threshold, the metadata of that file is directly inserted into the data migration task list, thus making that file a candidate file to be migrated.

[0122] It should be noted that the first size threshold is set according to the device performance of the data storage system and user needs. In this embodiment, since the increase in read and write bandwidth tends to be gradual as the block size increases after the SSD block size reaches 4MB, the first size threshold is set to 4MB. That is, when the size of a single file in the SSD storage unit is greater than 4MB, the metadata of the file is directly inserted into the data migration task list. Correspondingly, if the size of a single file in the SSD storage unit is less than or equal to 4MB, the subsequent step S22 is executed.

[0123] S22, sequentially determine whether the creation time of each file is before the preset time. If yes, insert the metadata of the corresponding file into the data migration task list. If not, proceed to step S23.

[0124] Related studies have shown that a lot of data has a low probability of being accessed after entering the data storage system for a period of time. Therefore, for files with a size of less than or equal to 4MB (i.e., small files), in order to reduce the probability of small files being read after entering the HDD storage unit of the data storage system, this application also uses the creation time attribute of small files as a reference parameter to determine the priority of data migration. That is, for small files, it is determined whether their creation time is before the preset time. If the creation time of the small file is before the preset time, it indicates that the data is cold data, and the metadata of the corresponding file can be inserted into the data migration task list, that is, the file is regarded as a candidate file to be migrated. Correspondingly, if the creation time of the file is after the preset time, the subsequent step S23 is executed.

[0125] It should be noted that the preset time is set according to the device performance of the data storage system and user needs. In this embodiment, the preset time is set to 10 days. That is, if the file was created more than 10 days ago, the corresponding file's metadata is inserted into the data migration task list; otherwise, step S23 is executed.

[0126] S23, sequentially determine whether the size of each file is greater than or equal to the second size threshold. If yes, proceed to step S24; otherwise, proceed to step S25. Wherein, the second size threshold is less than the first size threshold.

[0127] For small files whose file size is less than or equal to the first size threshold and whose creation time is after the preset time, it is necessary to further determine whether the file size is greater than or equal to the second size threshold, that is, to determine whether the file size is between the first size threshold and the second size threshold. If it is, then proceed to step S24; otherwise, proceed to step S25.

[0128] It should be noted that the second size threshold is set according to the device performance of the data storage system and user needs. In this embodiment, the second size threshold is set to 128KB.

[0129] S24, determine whether the storage capacity utilization rate of the SSD storage unit is higher than the first utilization rate threshold. If yes, insert the metadata corresponding to the current file into the data migration task list. If no, determine the next file.

[0130] When the file size is between the first and second size thresholds, and the SSD storage unit's storage capacity utilization rate is higher than the first utilization rate threshold (indicating that the remaining storage space of the SSD storage unit is tight, affecting the data processing performance of the SSD storage unit, and data migration is required to free up some storage space), the metadata corresponding to the current file is inserted into the data migration task list; otherwise, the next file is checked.

[0131] It should be noted that the first utilization threshold is set according to the device performance of the data storage system and user needs. In this embodiment, the first utilization threshold is set to 60%. That is, when the file size is between 128KB and 4MB and the storage capacity utilization of the SSD storage unit is higher than 60%, in order to reduce the storage capacity utilization of the SSD storage unit and improve the data processing performance of the SSD storage unit, the metadata of small files with a file size between 128KB and 4MB can also be inserted into the data migration task list, that is, the file is used as a candidate file to be migrated.

[0132] S25, determine whether the storage capacity utilization rate of the SSD storage unit is higher than the second utilization rate threshold. If yes, insert the metadata corresponding to the current file into the data migration task list. If no, determine the next file, where the second utilization rate threshold is greater than the first utilization rate threshold.

[0133] When the file size is less than the second utilization threshold and the storage capacity utilization of the SSD storage unit is higher than the second utilization threshold (indicating that the remaining storage space of the SSD storage unit is very tight, which seriously affects the data processing performance of the SSD storage unit and data migration is urgently needed to free up some storage space), the metadata corresponding to the current file is inserted into the data migration task list; otherwise, the next file is checked.

[0134] It should be noted that the second utilization threshold is set according to the device performance of the data storage system and user needs. In this embodiment, the second utilization threshold is set to 80%. That is, when the file size is less than 128KB (tiny file) and the storage capacity utilization of the SSD storage unit is higher than 80%, in order to reduce the storage capacity utilization of the SSD storage unit and improve the data processing performance of the SSD storage unit, the metadata of tiny files with a file size of less than 128KB can also be inserted into the data migration task list, that is, the file is used as a candidate file to be migrated.

[0135] like Figure 5 As shown, based on the above embodiments, step S3 may specifically include the following steps:

[0136] S31, retrieve data migration tasks from the updated data migration task list, wherein the data migration task includes metadata of at least one target file;

[0137] S32, obtain the storage capacity utilization rate of the SSD storage unit, the current network bandwidth, and the storage capacity utilization rate of the HDD storage unit;

[0138] S33, sequentially determine whether the storage capacity utilization rate of the SSD storage unit, the current network bandwidth, and the storage capacity utilization rate of the HDD storage unit meet the second preset migration conditions. If so, migrate the corresponding target file to the HDD storage unit based on the metadata of at least one target file in the data migration task.

[0139] Specifically, in this embodiment, step S33 may include the following steps:

[0140] S331, determine whether the storage capacity utilization rate of the SSD storage unit is higher than the third utilization rate threshold. If yes, migrate the corresponding target file to the HDD storage unit based on the metadata of at least one target file in the data migration task. If no, proceed to step S332.

[0141] S332, determine whether the current network bandwidth has reached the target network bandwidth; if so, proceed to step S333.

[0142] S333, determine whether the storage capacity utilization rate of the HDD storage unit is lower than the fourth utilization rate threshold. If so, migrate the corresponding target file to the HDD storage unit based on the metadata of at least one target file in the data migration task.

[0143] It should be noted that when the current network bandwidth reaches the target network bandwidth, it means that the current network bandwidth meets the data migration conditions. In other words, there is enough network bandwidth to support data migration. Data migration at this time will not affect the normal operation of other performance aspects of the data storage system.

[0144] Specifically, in this embodiment, the third utilization threshold, the target network bandwidth, and the fourth utilization threshold are set according to the device performance of the data storage system and user needs. In this embodiment, the third utilization threshold is set to 60%, and the fourth utilization threshold is set to 60%.

[0145] In other words, when migrating target files from an SSD storage unit to an HDD storage unit based on the updated data migration task list, the data migration task is first removed from the updated data migration task list. Then, it is determined whether the current storage capacity utilization of the SSD storage unit is too high (above 60%). If it is too high, the data migration operation is performed directly. Otherwise, the current network bandwidth and the storage capacity utilization of the HDD storage unit need to be determined. Only when the current network bandwidth is higher than the target network bandwidth and the storage capacity utilization of the HDD storage unit is less than 60%, will the data migration operation begin.

[0146] It should be noted that the thresholds in this application can be dynamically configured and take effect in real time.

[0147] When performing a data migration operation, you can copy and migrate the data, delete the data, and finally delete the migration task.

[0148] Specifically, in this embodiment, migrating the corresponding target file to the HDD storage unit based on the metadata of at least one target file in the data migration task includes:

[0149] Based on the metadata of at least one target file in the data migration task, the corresponding target file is read from the SSD storage unit;

[0150] Copy the target file from the SSD storage unit to the HDD storage unit;

[0151] Delete the target file that has been successfully copied from the SSD storage unit;

[0152] Remove the data migration task corresponding to the target file that has been successfully deleted from the data migration task list.

[0153] like Figure 6 As shown in the illustration, this application provides a data migration system applied to a data storage system. The data storage system includes SSD storage units and HDD storage units. The data migration system may include:

[0154] The information acquisition module 201 is used to acquire the size, creation time, and storage capacity utilization of each file in the SSD storage unit.

[0155] The task list update module 202 is used to update the data migration task list of the SSD storage unit based on the size of each file in the SSD storage unit, the creation time, and the storage capacity utilization of the SSD storage unit.

[0156] File migration module 203 is used to migrate target files from SSD storage units to HDD storage units based on an updated data migration task list.

[0157] Based on the above embodiments, the task list update module 202 is specifically used for:

[0158] The system sequentially determines whether the size, creation time, and storage capacity utilization of each file in the SSD storage unit meet the first preset migration conditions. If so, the metadata corresponding to the files that meet the first preset migration conditions is inserted into the data migration task list.

[0159] Based on the above embodiments, the step of sequentially determining whether the size, creation time, and storage capacity utilization of each file in the SSD storage unit meet the first preset migration condition, and if so, inserting the metadata corresponding to the file that meets the first preset migration condition into the data migration task list, may include the following steps:

[0160] S21, sequentially determine whether the size of each file in the SSD storage unit is less than or equal to the first size threshold. If yes, proceed to step S22. If no, insert the metadata of the corresponding file into the data migration task list.

[0161] S22, sequentially determine whether the creation time of each file is before the preset time. If yes, insert the metadata of the corresponding file into the data migration task list. If not, proceed to step S23.

[0162] S23, sequentially determine whether the size of each file is greater than or equal to the second size threshold. If yes, proceed to step S24; otherwise, proceed to step S25. Wherein, the second size threshold is less than the first size threshold.

[0163] S24, determine whether the storage capacity utilization rate of the SSD storage unit is higher than the first utilization rate threshold. If yes, insert the metadata corresponding to the current file into the data migration task list. If no, determine the next file.

[0164] S25, determine whether the storage capacity utilization rate of the SSD storage unit is higher than the second utilization rate threshold. If yes, insert the metadata corresponding to the current file into the data migration task list. If no, determine the next file, where the second utilization rate threshold is greater than the first utilization rate threshold.

[0165] Based on the above embodiments, the file migration module 203 performs the migration of target files from the SSD storage unit to the HDD storage unit based on the updated data migration task list, specifically including the following steps:

[0166] S31, retrieve data migration tasks from the updated data migration task list, wherein the data migration task includes metadata of at least one target file;

[0167] S32, obtain the storage capacity utilization rate of the SSD storage unit, the current network bandwidth, and the storage capacity utilization rate of the HDD storage unit;

[0168] S33, sequentially determine whether the storage capacity utilization rate of the SSD storage unit, the current network bandwidth, and the storage capacity utilization rate of the HDD storage unit meet the second preset migration conditions. If so, migrate the corresponding target file to the HDD storage unit based on the metadata of at least one target file in the data migration task.

[0169] Based on the above embodiments, S33 may include the following steps:

[0170] S331, determine whether the storage capacity utilization rate of the SSD storage unit is higher than the third utilization rate threshold. If yes, migrate the corresponding target file to the HDD storage unit based on the metadata of at least one target file in the data migration task. If no, proceed to step S332.

[0171] S332, determine whether the current network bandwidth has reached the target network bandwidth; if so, proceed to step S333.

[0172] S333, determine whether the storage capacity utilization rate of the HDD storage unit is lower than the fourth utilization rate threshold. If so, migrate the corresponding target file to the HDD storage unit based on the metadata of at least one target file in the data migration task.

[0173] Based on the above embodiments, migrating the corresponding target file to the HDD storage unit based on the metadata of at least one target file in the data migration task includes:

[0174] Based on the metadata of at least one target file in the data migration task, the corresponding target file is read from the SSD storage unit;

[0175] Copy the target file from the SSD storage unit to the HDD storage unit;

[0176] Delete the target file that has been successfully copied from the SSD storage unit;

[0177] Remove the data migration task corresponding to the target file that has been successfully deleted from the data migration task list.

[0178] Based on the above embodiments, obtaining the size and creation time of each file in the SSD storage unit includes:

[0179] Iterate through the file list of all files in the SSD storage unit;

[0180] Read the size and creation time information of each file from the file list.

[0181] The data migration system in the above embodiments has the same working principle and technical effect as the data migration method in the above embodiments, and will not be described again here.

[0182] like Figure 7 As shown, this application provides an electronic device 3, which includes a memory 301, a processor 302, and a computer program 303 stored in the memory 301 and executable on the processor 302. When the processor 302 executes the computer program 303, it implements the steps of the data migration method as described in the above method embodiment of this application.

[0183] Specifically, the electronic device 3 can be an intelligent device with memory and processor, such as an industrial control computer, PC, or smart mobile terminal, or a computer component with memory and processor, such as a CPU or GPU.

[0184] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the data migration method as described in the above-described method embodiments of this application.

[0185] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0186] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0187] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly using hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0188] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A data migration method applied to a data storage system, the data storage system comprising SSD storage units and HDD storage units, characterized in that, The data migration method includes: S1, obtain the size, creation time, and storage capacity utilization rate of each file in the SSD storage unit; S2, update the data migration task list of the SSD storage unit based on the size and creation time of each file in the SSD storage unit and the storage capacity utilization of the SSD storage unit; S3, based on the updated data migration task list, migrate the target files in the SSD storage unit to the HDD storage unit; Specifically, S2 includes: S21, sequentially determine whether the size of each file in the SSD storage unit is less than or equal to the first size threshold. If yes, proceed to step S22. If no, insert the metadata of the corresponding file into the data migration task list. S22, sequentially determine whether the creation time of each file is before the preset time. If yes, insert the metadata of the corresponding file into the data migration task list. If not, proceed to step S23. S23, sequentially determine whether the size of each file is greater than or equal to the second size threshold. If yes, proceed to step S24; otherwise, proceed to step S25, wherein the second size threshold is less than the first size threshold. S24, determine whether the storage capacity utilization rate of the SSD storage unit is higher than the first utilization rate threshold. If yes, insert the metadata corresponding to the current file into the data migration task list. If no, determine the next file. S25, determine whether the storage capacity utilization rate of the SSD storage unit is higher than the second utilization rate threshold. If yes, insert the metadata corresponding to the current file into the data migration task list. If no, determine the next file, wherein the second utilization rate threshold is greater than the first utilization rate threshold.

2. The data migration method according to claim 1, characterized in that, S3 includes: S31, Obtain a data migration task from the updated data migration task list, wherein the data migration task includes metadata of at least one target file; S32, obtain the storage capacity utilization rate of the SSD storage unit, the current network bandwidth, and the storage capacity utilization rate of the HDD storage unit; S33, sequentially determine whether the storage capacity utilization rate of the SSD storage unit, the current network bandwidth, and the storage capacity utilization rate of the HDD storage unit meet the second preset migration condition. If so, migrate the corresponding target file to the HDD storage unit based on the metadata of the at least one target file in the data migration task.

3. The data migration method according to claim 2, characterized in that, S33 includes: S331, determine whether the storage capacity utilization rate of the SSD storage unit is higher than the third utilization rate threshold. If yes, migrate the corresponding target file to the HDD storage unit based on the metadata of the at least one target file in the data migration task. If no, execute step S332. S332, determine whether the current network bandwidth has reached the target network bandwidth; if so, proceed to step S333. S333, determine whether the storage capacity utilization rate of the HDD storage unit is lower than the fourth utilization rate threshold. If so, migrate the corresponding target file to the HDD storage unit based on the metadata of the at least one target file in the data migration task.

4. The data migration method according to claim 2 or 3, characterized in that, The step of migrating the corresponding target file to the HDD storage unit based on the metadata of at least one target file in the data migration task includes: Based on the metadata of at least one target file in the data migration task, the corresponding target file is read from the SSD storage unit; Copy the target file from the SSD storage unit to the HDD storage unit; Delete the target file that has been successfully copied from the SSD storage unit; Remove the data migration task corresponding to the target file that has been successfully deleted from the data migration task list.

5. The data migration method according to any one of claims 1-3, characterized in that, The process of obtaining the size and creation time of each file in the SSD storage unit includes: Iterate through the file list of all files in the SSD storage unit; Read the size and creation time information of each file from the file list.

6. A data migration system applied to a data storage system, the data storage system comprising SSD storage units and HDD storage units, characterized in that, The data migration system includes: The information acquisition module is used to acquire the size, creation time, and storage capacity utilization rate of each file in the SSD storage unit. The task list update module is used to update the data migration task list of the SSD storage unit based on the size and creation time of each file in the SSD storage unit and the storage capacity utilization of the SSD storage unit. The file migration module is used to migrate target files from the SSD storage unit to the HDD storage unit based on the updated data migration task list; Specifically, when the task list update module updates the data migration task list of the SSD storage unit based on the size and creation time of each file in the SSD storage unit and the storage capacity utilization of the SSD storage unit, it is used for: S21, sequentially determine whether the size of each file in the SSD storage unit is less than or equal to the first size threshold. If yes, proceed to step S22. If no, insert the metadata of the corresponding file into the data migration task list. S22, sequentially determine whether the creation time of each file is before the preset time. If yes, insert the metadata of the corresponding file into the data migration task list. If not, proceed to step S23. S23, sequentially determine whether the size of each file is greater than or equal to the second size threshold. If yes, proceed to step S24; otherwise, proceed to step S25, wherein the second size threshold is less than the first size threshold. S24, determine whether the storage capacity utilization rate of the SSD storage unit is higher than the first utilization rate threshold. If yes, insert the metadata corresponding to the current file into the data migration task list. If no, determine the next file. S25, determine whether the storage capacity utilization rate of the SSD storage unit is higher than the second utilization rate threshold. If yes, insert the metadata corresponding to the current file into the data migration task list. If no, determine the next file, wherein the second utilization rate threshold is greater than the first utilization rate threshold.

7. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the data migration method as described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the data migration method as described in any one of claims 1-5.

Images