Data migration method and device, computer equipment and storage medium

Abstract

The invention relates to a data migration method and device, computer equipment, a storage medium and a computer program product. The method comprises the steps that after a migration instruction is received, a target hard disk corresponding to the migration instruction is determined, and all to-be-migrated data blocks corresponding to the target hard disk are determined; through the target data table, querying a garbage amount proportion and a cold and hot data type of each to-be-migrated data block; determining a target migration sequence of the to-be-migrated data blocks according to the garbage amount proportion of the to-be-migrated data blocks and the cold and hot data types; and according to the target migration sequence, migrating the to-be-migrated data blocks to the target data space, and after the data blocks are migrated, releasing the original data space where the to-be-migrated data blocks are located. By adopting the method, the migration sequence of each data block can be determined based on the garbage amount proportion and the cold and hot data type, write amplification in a storage system can be reduced, the data processing efficiency can be improved, the resource consumption can be reduced, and the service life of storage equipment can be prolonged.

Description

Technical Field

[0001] The present application relates to the field of storage technology, and in particular to a data migration method, apparatus, computer equipment, storage medium and computer program product. Background Art

[0002] With the continuous development of multi-core technology in the storage field, the multi-threaded tasks in the background of the storage system have increased, resulting in more resource consumption. Background tasks such as load balancing, data repair, and garbage collection all involve data migration, which involves the migration of multiple data blocks.

[0003] In the related art, when performing data migration, the data is divided into several small data, each small data to be migrated is locked, the locking timer module is used to calculate the locking time, and then the small data blocks are sorted according to the time, and the small data blocks with longer locking time are migrated first. The migration data is only migrated sequentially by locking and calculating the locking time, which reduces the efficiency of data migration. Summary of the invention

[0004] Based on this, it is necessary to provide a data migration method, apparatus, computer equipment, computer-readable storage medium and computer program product that can improve efficiency in response to the above technical problems.

[0005] In a first aspect, the present application provides a data migration method. The method comprises:

[0006] After receiving the migration instruction, determining a target hard disk corresponding to the migration instruction, and determining each to-be-migrated data block corresponding to the target hard disk;

[0007] Through the target data table, query the garbage volume ratio and hot and cold data types of each of the data blocks to be migrated;

[0008] Determining a target migration order of each of the data blocks to be migrated according to the garbage amount ratio and the hot and cold data types of each of the data blocks to be migrated;

[0009] Migrate each of the to-be-migrated data blocks to the target data space according to the target migration order, and release the original data space where each of the to-be-migrated data blocks is located after the data blocks are migrated.

[0010] In this embodiment, by maintaining data structures such as arrays and linked lists, the garbage ratio of each data block in the target data table and the hot and cold data types of each data block can be maintained, and the migration order of each data block can be determined based on the garbage ratio and the hot and cold data types. This can reduce the write amplification in the storage system, improve data processing efficiency, reduce resource consumption, and extend the service life of storage devices.

[0011] In one embodiment, the method further comprises:

[0012] Obtaining the number of used objects of the target hard disk, and obtaining the average number of used objects of the storage space corresponding to the target hard disk;

[0013] Calculate the difference between the number of used objects of the target hard disk and the average number of used objects of the storage space; if the difference is greater than or equal to a preset difference, determine that the target hard disk meets the preset migration condition, and generate a migration instruction based on the identification information of the target hard disk.

[0014] In this embodiment, by determining whether the preset migration condition is met according to the used objects of each data block contained in the hard disk, the hard disk that needs data migration can be detected in time, thereby improving the timeliness and accuracy of data migration in the hard disk.

[0015] In one embodiment, the method further comprises:

[0016] For each data block, detect the amount of junk data in the data block, and calculate the junk amount ratio of the data block based on the amount of junk data and the total data space; and determine the hot and cold data types of the data block based on the difference between the latest access time of the data block and a preset time;

[0017] Based on the garbage amount ratio of each data block and the hot and cold data types of each data block, each data block is arranged to obtain a target data table, wherein the target data table includes multiple garbage amount levels arranged in order of garbage amount size, and each garbage amount level includes multiple data blocks divided into cold data block type and hot data block type.

[0018] In this embodiment, statistics are collected on the garbage data ratio of each data block and the hot and cold data types to which each data block belongs, and a target data table is generated, which can provide a stable data basis for data migration in subsequent processes and improve data processing efficiency.

[0019] In one embodiment, after the step of migrating each of the to-be-migrated data blocks to the target data space according to the target migration order and releasing the original data space where each of the to-be-migrated data blocks is located after the data block migration, the method further includes:

[0020] Eliminate each of the data blocks to be migrated from the target data table;

[0021] The querying of the garbage volume ratio and hot and cold data types of each of the to-be-migrated data blocks through the target data table includes:

[0022] The target data table is queried for each module to be migrated. If the module to be migrated is not found in the target data table, the garbage volume ratio and the hot and cold data types of the next module to be migrated are queried.

[0023] In this embodiment, the data table can be updated in real time by promptly proposing the data blocks that have completed data migration in the data table. The data table is updated based on the results of the data balancing task and the garbage collection task, avoiding multiple background tasks from processing the same data block at the same time, improving data processing efficiency and reducing task delays.

[0024] In one embodiment, determining the target migration order of each of the data blocks to be migrated according to the garbage amount ratio and the hot and cold data types of each of the data blocks to be migrated includes:

[0025] Determining the garbage level of each of the data blocks to be migrated based on the garbage amount ratio of each of the data blocks to be migrated;

[0026] Sorting the data blocks to be migrated in order from low to high according to the garbage amount level to obtain an initial migration order;

[0027] In each of the garbage amount levels, the data blocks to be migrated are sorted according to the sequence of the data blocks to be migrated from cold data blocks to hot data blocks to obtain a target migration sequence.

[0028] In this embodiment, the data blocks required for balancing are sorted by the amount of garbage, so that the balancing task prioritizes the migration of cold data with a small amount of garbage, and then migrates the hot data with a large amount of garbage, thereby reducing write amplification, reducing the waste of hard disk read and write IO, and effectively extending the service life of the storage array.

[0029] In one embodiment, the migrating each of the to-be-migrated data blocks to the target data space according to the target migration order includes:

[0030] Migrate each of the to-be-migrated data blocks to the target data space according to the target migration order and the maximum number of single migrations.

[0031] In this embodiment, the data blocks that need to be migrated for balancing are sorted by the amount of garbage, so that the balancing task prioritizes migrating cold data with a small amount of garbage, and then migrates hot data with a large amount of garbage. Based on the maximum number of single migrations, data migration is performed in batches, which can ensure a balance between data migration efficiency and computing resource consumption.

[0032] In one embodiment, the method further comprises:

[0033] After receiving the data recovery instruction, querying the garbage amount ratio and hot and cold data types of each to-be-recovered data block corresponding to the data recovery instruction through the target data table;

[0034] Based on the garbage amount ratio of each of the to-be-reclaimed data blocks, the garbage amount level of each of the to-be-reclaimed data blocks is determined; in the order of the garbage amount levels from high to low, the to-be-reclaimed data blocks are sorted to obtain an initial recycling order; in each of the garbage amount levels, in the order of the to-be-reclaimed data blocks from cold data blocks to hot data blocks, the to-be-reclaimed data blocks are sorted to obtain a target recycling order;

[0035] According to the target recovery order, migration processing is performed on the valid data in each of the to-be-recovered data blocks respectively, and after the migration processing is completed, the original data space where each of the to-be-recovered data blocks is located is released.

[0036] In this embodiment, the data blocks to be recycled are sorted by the amount of garbage, so that the garbage collection task prioritizes migrating valid data in cold data with a large amount of garbage, and then migrates valid data in hot data with a small amount of garbage. Based on the maximum number of single recyclings, data recycling is performed in batches to ensure a balance between data recovery efficiency and computing resource consumption.

[0037] In a second aspect, the present application also provides a data migration device. The device comprises:

[0038] A first determination module is used to determine, after receiving a migration instruction, a target hard disk corresponding to the migration instruction and determine each to-be-migrated data block corresponding to the target hard disk;

[0039] A first query module is used to query the garbage volume ratio and hot and cold data types of each of the data blocks to be migrated through the target data table;

[0040] A second determination module is used to determine the target migration order of each of the data blocks to be migrated according to the garbage amount ratio and the hot and cold data types of each of the data blocks to be migrated;

[0041] The migration module is used to migrate each of the to-be-migrated data blocks to the target data space according to the target migration sequence, and release the original data space where each of the to-be-migrated data blocks is located after the data blocks are migrated.

[0042] In one embodiment, the data migration device further includes:

[0043] A first acquisition module, used to acquire the number of used objects of a target hard disk, and to acquire an average value of the number of used objects of a storage space corresponding to the target hard disk;

[0044] The first calculation module is used to calculate the difference between the number of used objects of the target hard disk and the average number of used objects of the storage space; if the difference is greater than or equal to a preset difference, it is determined that the target hard disk meets the preset migration conditions, and a migration instruction is generated based on the identification information of the target hard disk.

[0045] In one embodiment, the data migration device further includes:

[0046] A detection module is used to detect the amount of junk data in each data block, and calculate the proportion of the junk data in the data block based on the amount of junk data and the total data space; and determine the hot and cold data types of the data block based on the difference between the latest access time of the data block and a preset time;

[0047] The arrangement module is used to arrange each data block based on the garbage amount ratio of each data block and the hot and cold data types of each data block to obtain a target data table, wherein the target data table includes multiple garbage amount levels arranged in order of garbage amount size, and each garbage amount level includes multiple data blocks divided into cold data block type and hot data block type.

[0048] In one embodiment, the data migration device further includes:

[0049] A removal module, used for removing each of the data blocks to be migrated from the target data table;

[0050] The first query module is specifically used for:

[0051] The target data table is queried for each module to be migrated. If the module to be migrated is not found in the target data table, the garbage volume ratio and the hot and cold data types of the next module to be migrated are queried.

[0052] In one embodiment, the second determining module is specifically configured to:

[0053] Determining the garbage level of each of the data blocks to be migrated based on the garbage amount ratio of each of the data blocks to be migrated;

[0054] Sorting the data blocks to be migrated in order from low to high according to the garbage amount level to obtain an initial migration order;

[0055] In each of the garbage amount levels, the data blocks to be migrated are sorted according to the sequence of the data blocks to be migrated from cold data blocks to hot data blocks to obtain a target migration sequence.

[0056] In one embodiment, the migration module is specifically used for:

[0057] Migrate each of the to-be-migrated data blocks to the target data space according to the target migration order and the maximum number of single migrations.

[0058] In one embodiment, the data migration device further includes:

[0059] A second query module is used to query the garbage amount ratio and hot and cold data types of each to-be-reclaimed data block corresponding to the data recovery instruction through the target data table after receiving the data recovery instruction;

[0060] The third determination module is used to determine the garbage level of each of the to-be-recovered data blocks based on the garbage level ratio of each of the to-be-recovered data blocks; sort the to-be-recovered data blocks in order from high to low according to the garbage level level to obtain an initial recovery order; in each of the garbage level levels, sort the to-be-recovered data blocks in order from cold data blocks to hot data blocks to obtain a target recovery order;

[0061] The valid data migration module is used to migrate the valid data in each of the to-be-reclaimed data blocks according to the target recovery order, and release the original data space where each of the to-be-reclaimed data blocks is located after the migration process is completed.

[0062] In a third aspect, the present application further provides a computer device. The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0063] After receiving the migration instruction, determining a target hard disk corresponding to the migration instruction, and determining each to-be-migrated data block corresponding to the target hard disk;

[0064] Through the target data table, query the garbage volume ratio and hot and cold data types of each of the data blocks to be migrated;

[0065] Determining a target migration order of each of the data blocks to be migrated according to the garbage amount ratio and the hot and cold data types of each of the data blocks to be migrated;

[0066] Migrate each of the to-be-migrated data blocks to the target data space according to the target migration order, and release the original data space where each of the to-be-migrated data blocks is located after the data blocks are migrated.

[0067] In a fourth aspect, the present application further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0068] After receiving the migration instruction, determining a target hard disk corresponding to the migration instruction, and determining each to-be-migrated data block corresponding to the target hard disk;

[0069] Through the target data table, query the garbage volume ratio and hot and cold data types of each of the data blocks to be migrated;

[0070] Determining a target migration order of each of the data blocks to be migrated according to the garbage amount ratio and the hot and cold data types of each of the data blocks to be migrated;

[0071] Migrate each of the to-be-migrated data blocks to the target data space according to the target migration order, and release the original data space where each of the to-be-migrated data blocks is located after the data blocks are migrated.

[0072] In a fifth aspect, the present application further provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0073] After receiving the migration instruction, determining a target hard disk corresponding to the migration instruction, and determining each to-be-migrated data block corresponding to the target hard disk;

[0074] Through the target data table, query the garbage volume ratio and hot and cold data types of each of the data blocks to be migrated;

[0075] Determining a target migration order of each of the data blocks to be migrated according to the garbage amount ratio and the hot and cold data types of each of the data blocks to be migrated;

[0076] Migrate each of the to-be-migrated data blocks to the target data space according to the target migration order, and release the original data space where each of the to-be-migrated data blocks is located after the data blocks are migrated.

[0077] The above-mentioned data migration method, device, computer equipment, storage medium and computer program product, the method includes: after receiving the migration instruction, determining the target hard disk corresponding to the migration instruction, determining each data block to be migrated corresponding to the target hard disk; querying the garbage volume ratio and hot and cold data type of each data block to be migrated through the target data table; determining the target migration order of each data block to be migrated according to the garbage volume ratio and hot and cold data type of each data block to be migrated; migrating each data block to be migrated to the target data space according to the target migration order, and releasing the original data space where each data block to be migrated is located after the data block is migrated. By adopting this method, the garbage volume ratio of each data block in the target data table and the hot and cold data type of each data block can be maintained by maintaining data structures such as arrays and linked lists, and the migration order of each data block can be determined based on the garbage volume ratio and the hot and cold data type, which can reduce the write amplification in the storage system, improve the efficiency of data processing, reduce resource consumption, and extend the service life of storage devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] Figure 1 A schematic diagram of a data migration method in one embodiment;

[0079] Figure 2 A schematic diagram of a flow chart of steps for generating a migration instruction in one embodiment;

[0080] Figure 3 A schematic diagram of a target data table step in an embodiment;

[0081] Figure 4 A flowchart of target migration sequence steps in one embodiment;

[0082] Figure 5 is a flowchart of a garbage collection step in one embodiment;

[0083] Figure 6 A schematic diagram of the overall process of data migration and garbage collection in one embodiment;

[0084] Figure 7 is a schematic diagram of the structure of a target data table in an embodiment;

[0085] Figure 8 is a flow chart of a data migration method in another embodiment;

[0086] Fig. 9 is a structural block diagram of a data migration device in one embodiment;

[0087] Fig.10 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0088] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0089] In one embodiment, Figure 1 As shown, a data migration method is provided. This embodiment uses the method applied to a terminal as an example for illustration. It is understandable that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. The above-mentioned terminal can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, etc. The server can be implemented as an independent server or a server cluster composed of multiple servers. In this embodiment, the data migration method includes the following steps:

[0090] Step 102: after receiving the migration instruction, determine the target hard disk corresponding to the migration instruction, and determine the data blocks to be migrated corresponding to the target hard disk.

[0091] Among them, the migration instruction is an instruction used to instruct data balancing processing, for example, it can be a migration instruction generated when it is determined that the current hard disk meets the preset migration conditions; the target hard disk corresponding to the migration instruction can be a hard disk that meets the preset migration conditions, and the storage system can include multiple hard disks, among which the hard disk that meets the preset migration conditions can be the target hard disk corresponding to the migration instruction; the data blocks to be migrated corresponding to the target hard disk can be multiple data blocks to be migrated that need to be migrated determined from the multiple data blocks contained in the target hard disk, for example, they can be multiple data blocks to be migrated determined based on the data addresses carried by the migration instruction.

[0092] Specifically, after detecting the migration instruction, the terminal can parse the migration instruction to obtain the hard disk information to be migrated and the data block information to be migrated carried by the migration instruction. For example, the hard disk information to be migrated can be the identification information of the hard disk or the logical block address information. The terminal can determine the target hard disk based on the identification information or the logical block address information in the hard disk information to be migrated carried by the migration instruction; accordingly, the data block information to be migrated carried in the migration instruction can be the address information of the data block, and the terminal can determine multiple data blocks to be migrated based on the address information. In one example, the process of the terminal detecting the migration instruction can be that the terminal receives the migration instruction sent by the management module.

[0093] Step 104 , querying the garbage ratio and hot and cold data types of each data block to be migrated through the target data table.

[0094] The target data table may include the garbage volume ratio of multiple data blocks, the garbage volume level corresponding to the garbage volume ratio, and the hot and cold data types of the data blocks. The data blocks may be arranged in order from high to low according to the garbage volume levels, and in each garbage volume level, the data blocks may be divided and arranged according to the hot and cold data types; the target data table may be updated based on the data blocks involved in the data balancing task and the data blocks involved in the garbage collection task.

[0095] Specifically, after determining each data block to be migrated, the terminal can query the garbage volume ratio of each data block to be migrated and the hot and cold data types of each data block to be migrated through the target data table. In one example, the target data table can be a data table stored in the management module. After receiving the migration instruction through the balancing module, the terminal can send a query request to the management module, and query the garbage volume ratio and hot and cold data types corresponding to each data block to be migrated in the target data table through the management module.

[0096] Step 106 : determining the target migration order of each data block to be migrated according to the garbage amount ratio and the hot and cold data types of each data block to be migrated.

[0097] Specifically, the terminal can sort the data blocks to be migrated based on the obtained garbage ratio of each data block to be migrated and the hot and cold data types corresponding to each data block to be migrated, and obtain the target migration order of each data block to be migrated. In other words, the terminal can determine the order of data migration for each data block to be migrated based on the garbage ratio of each data block to be migrated and the hot and cold data types. Among them, the target migration order represents the order of data migration for each data block to be migrated.

[0098] Step 108 , migrate each data block to be migrated to the target data space according to the target migration order, and release the original data space where each data block to be migrated is located after the data block is migrated.

[0099] Specifically, after the terminal determines the target migration order of each data block to be migrated, the terminal can apply for a new data block storage space and determine the newly applied data block storage space as the target data space. Based on this, the terminal can migrate each data block to be migrated to the target data space in sequence according to the target migration order of each data block to be migrated. After the terminal determines that each data block to be migrated has been migrated to the target data space, it can determine that the data migration of the current data block has been completed, and a migration completion response message can be generated, and the migration completion response message can be returned to the management module; accordingly, the terminal can also release the original data space where each data block to be migrated is located after determining that the migration of the current data block has been completed.

[0100] In one example, the target data space may be a data storage space that belongs to the same solid-state hard disk as the original data space, or the target data space may be a data storage space that does not belong to the same solid-state hard disk as the original data space. In another example, if the terminal receives a migration completion response message through the management module, the management module may determine the data blocks that have been migrated based on the migration completion response message, and delete the migrated data blocks in the target data table.

[0101] In the above data migration method, after receiving the migration instruction, the target hard disk corresponding to the migration instruction is determined, and each data block to be migrated corresponding to the target hard disk is determined; through the target data table, the garbage volume ratio and the hot and cold data types of each data block to be migrated are queried; according to the garbage volume ratio and the hot and cold data types of each data block to be migrated, the target migration order of each data block to be migrated is determined; according to the target migration order, each data block to be migrated is migrated to the target data space, and after the data block is migrated, the original data space where each data block to be migrated is located is released. By adopting this method, the garbage volume ratio of each data block and the hot and cold data types of each data block in the target data table can be maintained by maintaining data structures such as arrays and linked lists, and the migration order of each data block can be determined based on the garbage volume ratio and the hot and cold data types, which can reduce the write amplification in the storage system, improve the efficiency of data processing, reduce resource consumption, and extend the service life of storage devices.

[0102] In one embodiment, Figure 2 As shown, the data migration method also includes:

[0103] Step 202, obtaining the number of used objects of the target hard disk, and obtaining the average number of used objects of the storage space corresponding to the target hard disk.

[0104] Specifically, the storage space may include multiple target hard disks, the storage space may be a storage pool including multiple storage hard disks, the storage space corresponding to the target hard disk may be the storage space to which the target hard disk belongs, for example, it may be the storage pool where the target hard disk is located; the average used number may be the average number of used objects of each target hard disk in the storage pool.

[0105] Specifically, the terminal may obtain the number of used objects of each target hard disk included in the storage pool, and calculate the average of the used numbers corresponding to the storage pool based on the number of used objects of each target hard disk and the number of target hard disks.

[0106] Step 204, calculate the difference between the number of used objects of the target hard disk and the average number of used objects of the storage space. If the difference is greater than or equal to the preset difference, determine that the target hard disk meets the preset migration condition, and generate a migration instruction based on the identification information of the target hard disk.

[0107] The preset number difference may be a pre-configured number difference threshold, which represents the standard deviation range. The specific value of the preset number difference may be determined by a person skilled in the art based on the actual application scenario, and the present disclosure does not specifically limit this. The preset migration condition is used to determine whether the target hard disk needs data balancing or data migration under the current situation; the content of the preset migration condition may be that the number difference corresponding to the hard disk is greater than or equal to the preset number difference. The identification information of the target hard disk may be the logical block address information of the target hard disk, or the name information of the target hard disk, etc.

[0108] Specifically, the terminal can detect each target hard disk. For each target hard disk, the terminal can calculate the difference between the average used number and the number of used objects of the target hard disk, and compare the difference with the preset number difference as the number difference; if the terminal determines that the number difference is greater than or equal to the preset number difference, the terminal can determine that the target hard disk is a target hard disk that meets the preset migration conditions; in this way, the terminal can generate a migration instruction based on the identification information of the target hard disk that meets the preset migration conditions, and send the migration instruction to the balancing module.

[0109] In one example, after determining that the target hard disk meets the preset migration conditions, the terminal can determine multiple data blocks to be migrated based on the multiple data blocks contained in the target hard disk. For example, the data blocks can be screened based on the garbage ratio of each data block and the data block type of each data block, and the target number of cold data blocks can be determined as the data blocks to be migrated in order of the garbage ratio of the data blocks from small to large.

[0110] In another example, after the terminal sends the migration instruction to the balancing module, the balancing module can execute the above 102 based on the received migration instruction, determine the target hard disk corresponding to the migration instruction after receiving the migration instruction, and determine the data blocks to be migrated corresponding to the target hard disk.

[0111] In this embodiment, by determining whether the preset migration condition is met according to the used objects of each data block contained in the hard disk, the hard disk that needs data migration can be detected in time, thereby improving the timeliness and accuracy of data migration in the hard disk.

[0112] In one embodiment, Figure 3 As shown, the data migration method also includes:

[0113] Step 302: for each data block, detect the amount of junk data in the data block, and calculate the junk amount ratio of the data block based on the amount of junk data and the total data space, and determine the hot and cold data types of the data block based on the difference between the latest access time of the data block and the preset time.

[0114] Among them, the amount of garbage data in the data block can be the amount of invalid data in the data block at present, the total data space can be the total data capacity of the data block, the garbage amount ratio represents the proportion of the amount of garbage in the current data block, the latest access time can be the time when the data block was last accessed, or it can be the closing time of the data block, and the preset time difference can be a pre-configured threshold.

[0115] Specifically, for each data block, the terminal can respectively obtain the total data capacity corresponding to the total data space of the data block and the capacity of the junk data in the current data block, and calculate the ratio of the capacity of the junk data to the total data capacity, and use the ratio to determine the garbage amount ratio of the data block; accordingly, the terminal can obtain the last access time of the data block, and determine whether the data block is a cold or hot data type through the difference between the last access time and the preset time. The data block can be a cold data block or a hot data block.

[0116] In one example, the latest access time may be a shutdown time, the preset time difference may be a preset time difference threshold, the terminal may obtain the current time and calculate a first difference between the current time and the shutdown time; if the first difference is greater than or equal to the preset time difference threshold, the terminal may determine that the data block may be a cold data block; if the first difference is less than the preset time difference threshold, the terminal may determine that the data block may be a hot data block. A cold data block indicates that the user's access is relatively infrequent, and a hot data block indicates that the user's access is relatively frequent.

[0117] Step 304 , based on the garbage amount ratio of each data block and the hot and cold data types of each data block, the data blocks are arranged to obtain a target data table.

[0118] The target data table includes a plurality of garbage volume levels arranged in order of garbage volume, and each garbage volume level includes a plurality of data blocks divided into a cold data block type and a hot data block type.

[0119] Specifically, the terminal can divide each data block according to the garbage amount ratio of each data block and multiple gear thresholds, and obtain multiple garbage amount gears and the garbage amount gear to which each data block belongs. In one example, the gear threshold can be a first gear threshold and a second gear threshold, and the first gear threshold can be greater than the second gear threshold. The terminal can determine that the data block with a garbage amount ratio greater than or equal to the first gear threshold is the first garbage amount gear; the terminal can also determine that the data block with a garbage amount ratio less than the first gear threshold and greater than or equal to the second gear threshold is the second garbage amount gear, and the data block with a garbage amount ratio less than the second gear threshold is determined as the third garbage amount gear.

[0120] Based on this, the terminal can arrange the data blocks in order from high to low according to the garbage level. In each garbage level, the data blocks contained in the garbage level can be divided and arranged according to the hot and cold data types; in addition, the target data table can be updated based on the data blocks involved in the data balancing task and the data blocks involved in the garbage collection task. The terminal can delete the data blocks that have completed data migration in the data balancing task from the target data table; the terminal can also delete the data blocks that have completed garbage collection in the garbage collection task from the target data table.

[0121] In this embodiment, statistics are collected on the garbage data ratio of each data block and the hot and cold data types to which each data block belongs, and a target data table is generated, which can provide a stable data basis for data migration in subsequent processes and improve data processing efficiency.

[0122] In one embodiment, after the steps of migrating each data block to be migrated to the target data space according to the target migration order and releasing the original data space where each data block to be migrated is located after the data block migration, the data migration method further includes:

[0123] Eliminate each data block to be migrated in the target data table.

[0124] Specifically, after receiving the migration completion response message, the terminal may determine the migrated data blocks to be migrated based on the migration completion response message. Based on this, the terminal may delete the migrated data blocks to be migrated from the target data table.

[0125] In another example, after the terminal determines that the steps of migrating each data block to be migrated to the target data space according to the target migration order and releasing the original data space where each data block to be migrated is located after the data block migration, the terminal can delete the migrated data block in the target data table.

[0126] Based on this, step 104, through the target data table, queries the garbage volume ratio of each data block to be migrated and the specific implementation method of the hot and cold data types include:

[0127] The target data table is queried for each module to be migrated. If the module to be migrated is not found in the target data table, the garbage volume ratio and hot and cold data types of the next module to be migrated are queried.

[0128] Specifically, after determining multiple data blocks to be migrated, the terminal can query the garbage amount ratio and hot and cold data types of each data block to be migrated in turn through the target data table; for each data block to be migrated, if the terminal can query the garbage amount ratio and hot and cold data types of the data block to be migrated in the target data table, the terminal will extract the garbage amount ratio and hot and cold data type information corresponding to the data block to be migrated; if the terminal does not query the garbage amount ratio and hot and cold data type information of the data block to be migrated in the target data table, the terminal can skip the step of obtaining the garbage amount ratio and hot and cold data type information of the data table to be migrated, and continue to query the target data table for the garbage amount ratio and hot and cold data type information of the next data to be migrated of the data block to be migrated.

[0129] In this embodiment, the data table can be updated in real time by promptly proposing the data blocks that have completed data migration in the data table. The data table is updated based on the results of the data balancing task and the garbage collection task, avoiding multiple background tasks from processing the same data block at the same time, improving data processing efficiency and reducing task delays.

[0130] In one embodiment, Figure 4 As shown, the specific processing process of step 106 "determining the target migration order of each data block to be migrated according to the garbage amount ratio of each data block to be migrated and the type of hot and cold data" includes:

[0131] Step 402: determining the garbage level of each data block to be migrated based on the garbage level ratio of each data block to be migrated.

[0132] Among them, each garbage volume level can be divided based on multiple level thresholds, and the specific values ​​of the multiple level thresholds can be determined based on actual application scenarios.

[0133] Specifically, the terminal may determine multiple garbage volume levels according to multiple level thresholds, and determine the garbage volume level to which each data block belongs based on the matching relationship between the garbage volume ratio of each data block and each level threshold. In one example, the level threshold may include a first level threshold and a second level threshold, and the corresponding garbage volume levels may include a first garbage volume level, a second garbage volume level, and a third garbage volume level. The first level threshold may be greater than the second level threshold, and the terminal may determine that the garbage volume level of a data block whose garbage volume ratio is greater than or equal to the first level threshold is the first garbage volume level; the terminal may also determine that the garbage volume level of a data block whose garbage volume ratio is less than the first level threshold and greater than or equal to the second level threshold is the second garbage volume level, and the garbage volume level of a data block whose garbage volume ratio is less than the second level threshold is the third garbage volume level.

[0134] Step 404: sort the data blocks to be migrated in order from low to high according to the garbage level to obtain an initial migration order.

[0135] The order of the garbage amount levels from low to high may be the order of the garbage amount ratios from low to high.

[0136] Specifically, the terminal may sort the data blocks to be migrated in order from low to high according to the garbage level they belong to, and obtain the initial migration order of the data blocks to be migrated, that is, the initial migration order is from low garbage level to high garbage level.

[0137] Step 406 , in each garbage level, sort the data blocks to be migrated according to the sequence from cold data blocks to hot data blocks to obtain a target migration sequence.

[0138] Specifically, among the multiple data blocks to be migrated belonging to the same garbage amount level, the multiple data blocks to be migrated belonging to the same garbage amount level are sorted in order from cold data blocks to hot data blocks to obtain a target migration order of each data block to be migrated.

[0139] In this embodiment, the data blocks required for balancing are sorted by the amount of garbage, so that the balancing task prioritizes the migration of cold data with a small amount of garbage, and then migrates the hot data with a large amount of garbage, thereby reducing write amplification, reducing the waste of hard disk read and write IO, and effectively extending the service life of the storage array.

[0140] In one embodiment, the specific processing of step 108 of "migrating each to-be-migrated data block to the target data space according to the target migration order" includes:

[0141] Migrate each data block to be migrated to the target data space according to the target migration order and the maximum number of single migrations.

[0142] The maximum number of single migrations may be the maximum number of data blocks that the terminal simultaneously migrates to the target data space, for example, 3 or 5, etc. The present disclosure does not limit the specific value of the maximum number of single migrations.

[0143] Specifically, the terminal can migrate multiple data blocks to be migrated to the target number space in batches based on the maximum number of single migrations and the predetermined target migration order. In the case that the number of data blocks to be migrated contained in the first garbage volume gear determined based on the target migration order is greater than or equal to the maximum number of single migrations, the terminal can extract the maximum number of data blocks to be migrated in the first garbage volume gear, and migrate each data block to be migrated to the target data space. If the first number of data blocks to be migrated contained in the first garbage volume gear is less than the maximum number of single migrations, the terminal can extract the target number of cold data blocks in the second garbage volume gear, migrate the first number of data blocks to be migrated and the target number of cold data blocks to the target data space, and continue to migrate each data block to be migrated to the target data space according to the target migration order; wherein the target number is determined based on the difference between the first number and the maximum number of single migrations.

[0144] In one example, the terminal may migrate in a target order. In one example, the maximum number of single migrations may be 3, the number of data blocks to be migrated may be 7, the first garbage amount gear may include the first cold data block and the second cold data block, the second garbage amount gear may include the first hot data block, the second hot data block and the third cold data block, and the third garbage amount gear may include the fourth cold data block and the third hot data block; in this way, the terminal may determine that the first batch of data blocks migrated to the target data space may be the first cold data block and the second cold data block in the first garbage amount gear, and the third cold data block in the second garbage amount gear; the terminal may determine that the second batch of data blocks migrated to the target data space may be the first hot data block, the second hot data block in the second garbage amount gear, and the fourth cold data block in the third garbage amount gear, and determine that the third batch of data blocks migrated to the target data space are the third hot data blocks.

[0145] In this embodiment, the data blocks that need to be migrated for balancing are sorted by the amount of garbage, so that the balancing task prioritizes migrating cold data with a small amount of garbage, and then migrates hot data with a large amount of garbage. Based on the maximum number of single migrations, data migration is performed in batches, which can ensure a balance between data migration efficiency and computing resource consumption.

[0146] In one embodiment, Figure 5 As shown, the data migration method also includes:

[0147] Step 502, after receiving the data recovery instruction, query the garbage amount ratio and hot and cold data types of each to-be-recovered data block corresponding to the data recovery instruction through the target data table.

[0148] The data recovery instruction may be an instruction generated after determining that there is a data block to be recovered that meets a preset recovery condition, and is used to instruct a data recovery operation to be performed.

[0149] Specifically, in response to the data recovery instruction, the terminal can parse the data recovery instruction to determine the multiple data blocks to be recovered corresponding to the data recovery instruction. In this way, the terminal can query the garbage volume ratio of each data block to be recovered and the hot and cold data types corresponding to each data block to be recovered through the target data table.

[0150] Step 504, based on the garbage amount ratio of each data block to be recovered, determine the garbage amount level of each data block to be recovered. Sort the data blocks to be recovered in order from high to low according to the garbage amount level to obtain an initial recovery order. In each garbage amount level, sort the data blocks to be recovered in order from cold data blocks to hot data blocks to obtain a target recovery order.

[0151] Specifically, the terminal can determine multiple garbage volume gears according to multiple gear thresholds, and determine the garbage volume gear to which each data block to be recovered belongs based on the matching relationship between the garbage volume ratio of each data block to be recovered and each gear threshold. In one example, the gear threshold may include a first gear threshold and a second gear threshold, and the corresponding garbage volume gear may include a first garbage volume gear, a second garbage volume gear, and a third garbage volume gear. The first gear threshold may be greater than the second gear threshold, and the terminal may determine that the garbage volume gear of the data block to be recovered whose garbage volume ratio is greater than or equal to the first gear threshold is the first garbage volume gear; the terminal may also determine that the garbage volume gear of the data block to be recovered whose garbage volume ratio is less than the first gear threshold and greater than or equal to the second gear threshold is the second garbage volume gear, and the garbage volume gear of the data block to be recovered whose garbage volume ratio is less than the second gear threshold is the third garbage volume gear.

[0152] The terminal can sort each data block to be recycled in order from high to low according to the garbage amount level to which each data block to be recycled belongs, and obtain the initial recycling order of each data block to be recycled, and among the multiple data blocks to be recycled that belong to the same garbage amount level, sort the multiple data blocks to be recycled that belong to the same garbage amount level in order from cold data blocks to hot data blocks, and obtain the target recycling order of each data block to be recycled.

[0153] Step 506 , according to the target recovery order, respectively migrate the valid data in each to-be-reclaimed data block, and after the migration process is completed, release the original data space where each to-be-reclaimed data block is located.

[0154] Specifically, the terminal can perform data recovery on each data block to be recovered in sequence based on the target recovery order. For each data block to be recovered, the terminal can obtain the valid data in the data block to be recovered and migrate the valid data in the data block to be recovered. After determining that the valid data in the current data block to be recovered has been migrated to the target data space, the original data space where the data block to be recovered is located can be released.

[0155] In one example, the terminal can perform data recovery processing on each data block to be recovered in batches based on the target recovery order and the maximum number of single recovery. The terminal can obtain the data blocks to be recovered with the previous maximum number of single recovery according to the target recovery order, and migrate the valid data in the data blocks to be recovered with the previous maximum number of single recovery. After determining that the valid data has been migrated to the target data space, the original data space where each data block to be recovered is located can be released. In this way, the terminal can obtain the next data block to be recovered with the previous maximum number of single recovery according to the target recovery order, and migrate the valid data until the garbage collection of each data block to be recovered corresponding to the target recovery order is completed.

[0156] In this embodiment, the data blocks to be recycled are sorted by the amount of garbage, so that the garbage collection task prioritizes migrating valid data in cold data with a large amount of garbage, and then migrates valid data in hot data with a small amount of garbage. Based on the maximum number of single recyclings, data recycling is performed in batches to ensure a balance between data recovery efficiency and computing resource consumption.

[0157] The specific implementation process of the above data migration method is described in detail below in conjunction with a specific embodiment:

[0158] With the continuous development and increasing maturity of multi-core technology, the characteristics of modern storage systems have gradually become richer, which has led to an increase in background multi-threaded tasks and more resource consumption. In related technologies, data is divided into several small data. When several small data need to be migrated, the small data to be migrated is first locked, and the lock timer module is used to calculate the lock time. Then, the small data blocks are sorted according to the time, and the small data blocks with longer lock time are migrated first. The data blocks that have been moved are unlocked, which will cause write amplification, reduce the performance and service life of the storage device, and also affect the efficiency of data migration.

[0159] The data balancing method provided in this embodiment can reduce the write amplification that exists during data migration and improve the efficiency of data processing. Specifically, it is based on data structures such as arrays and linked lists, adopts a method of mutual cooperation between balancing and GC, formulates a data migration strategy, and proposes a data migration method based on a cooperation strategy. The data to be migrated is divided according to the amount of garbage and the hot and cold types, and the migration of data blocks is completed, thereby reducing resource consumption and extending the service life of storage devices.

[0160] like Figure 6 As shown, it can be a data migration system based on a cooperative strategy, which includes a management module, a balancing module and a GC module (garbage collection module). When a user operates a file, writing a file is an overwrite, while in the background system it is an append write. When a user overwrites existing data, the system will write the content to other locations on the disk, which will cause the space previously occupied by the data to become garbage space. The management module is responsible for maintaining these garbage spaces according to the amount of garbage and access time, and providing the background task module with information on the data to be migrated, including but not limited to the disk ID to be migrated, the corresponding data block information of the disk to be migrated, and the public data type table; the balancing module and the GC module query the management module for information on the data blocks to be migrated and the garbage level and data type of each data block, perform data migration in sequence, and return the migration results to the management module, and the management module deletes the migrated data blocks from the public data table. The balancing module and the GC module can also query the target data table (public data table) maintained by the management module for migration data, etc., that is, it can include the garbage level and data type of each data block.

[0161] Specifically, Figure 7 As shown, it can be a storage form of a public data table in a memory, and the public data table includes multiple garbage amount gears arranged in order from high to low, for example, it can include garbage amount gear 1, garbage amount gear 2, garbage amount gear 3, ..., garbage amount gear n. In each garbage amount gear, each data block contained in the garbage amount gear can be divided and arranged according to the type of hot and cold data; the balancing module can query the migration data from the public data table according to the order of the garbage amount gears from low to high, and the GC module can query the migration data from the public data table according to the order of the garbage amount gears from high to low.

[0162] like Figure 8 As shown, the overall migration process based on the cooperation strategy can be:

[0163] The process of the management module maintaining the data table may include: regularly detecting the amount of garbage in each data block; maintaining a data block classification table for joint query by GC and load balancing; traversing all data blocks; calculating the garbage ratio of the current data block, and calculating the formula: ((total space - used space) * 100) / total space; judging whether the current data block is a cold data block according to the closing time and the set threshold; placing the data blocks into the data block classification table according to the gear and type; when the migration conditions are met, sending the data migration task to the background task module. Among them, the migration condition may be that data migration is required when the number of used objects on the hard disk contained in the storage pool deviates greatly from the average value. The migration process in an example may include:

[0164] Step 1: The business initiates the load balancing process;

[0165] Step 2: query the management module for information about the hard disk to be migrated and information about the data blocks to be migrated;

[0166] Step 3, obtaining the garbage amount ratio and type of the data block to be migrated according to the data block classification table obtained from the management module;

[0167] Step 4: Sort the migrated data blocks by garbage ratio and type

[0168] Step 5: Prioritize cold data blocks with the least amount of garbage;

[0169] Step 6, apply for new data block space from the management module;

[0170] Step 7: Perform data migration;

[0171] Step 8: Return the migration result to the management module;

[0172] Step 9: Release the old data space after migration;

[0173] Step 10: Complete balanced data migration.

[0174] In an example, the GC module moves valid data from data blocks stored in the garbage space to new data blocks, so that all old data blocks become garbage data and are released. The purpose is to clean up the garbage. The data migration process in the GC module is as follows:

[0175] Step 1: The business initiates the GC process;

[0176] Step 2, query the data block information to be processed;

[0177] Step 3, obtaining the garbage volume ratio and type of each data block;

[0178] Step 4: sort the data blocks to be migrated;

[0179] Step 5, apply for new data block space;

[0180] Step 6: Migrate valid data in the order of large amount of garbage, small amount of garbage, and small amount of garbage.

[0181] Step 7, returning the processing result to the management module;

[0182] Step 8, clean up the junk space;

[0183] Step 9: Complete GC data migration.

[0184] Specifically, when a background task has migrated a certain data block and another background task also needs to migrate the data block, the data will not be found in the data block classification table. At this time, the data block is skipped and the next data block is directly migrated. In the related technology, the migration technology sorts the data to be migrated by locking behavior and timing, which can reduce the impact of the migrated data on the user's write operation, but it does not cooperate with the background task. After the balancing module migrates the data to the destination space, the data in the destination space may be recycled by GC, resulting in a waste of read and write IO; the data balancing method provided by the present application can sort the data to be migrated by the amount of garbage before migrating the data. First, the data blocks are sorted according to the amount of garbage, and then the cold and hot gears are distinguished according to the closing clock of the data blocks. The cold data blocks in the gears with less garbage are preferentially selected for migration, which effectively reduces resource consumption, prevents repeated data processing, improves data processing efficiency, and extends the service life of storage devices.

[0185] That is to say, the data balancing method provided by the present application can sort the data required for balancing according to the amount of garbage, so that the balancing task prioritizes the migration of data with a small amount of garbage, and then migrates the data with a large amount of garbage. This is the opposite of GC, and reduces write amplification. By formulating a balancing strategy and cooperating with the GC task, it can reduce the waste of hard disk read and write IO and effectively extend the service life of the storage array.

[0186] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0187] Based on the same inventive concept, the embodiment of the present application also provides a data migration device for implementing the data migration method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more data migration device embodiments provided below can refer to the limitations on the data migration method above, and will not be repeated here.

[0188] In one embodiment, Fig. 9 As shown, a data migration device 900 is provided, comprising:

[0189] A first determining module 902 is used to determine the target hard disk corresponding to the migration instruction and determine the data blocks to be migrated corresponding to the target hard disk after receiving the migration instruction;

[0190] The first query module 904 is used to query the garbage volume ratio and hot and cold data types of each data block to be migrated through the target data table;

[0191] The second determination module 906 is used to determine the target migration order of each data block to be migrated according to the garbage amount ratio and the hot and cold data types of each data block to be migrated;

[0192] The migration module 908 is used to migrate each data block to be migrated to the target data space according to the target migration order, and release the original data space where each data block to be migrated is located after the data block is migrated.

[0193] In one embodiment, the data migration device further includes:

[0194] A first acquisition module, used to acquire the number of used objects of a target hard disk, and to acquire an average value of the number of used objects of a storage space corresponding to the target hard disk;

[0195] The first calculation module is used to calculate the difference between the number of used objects of the target hard disk and the average number of used objects of the storage space; if the difference is greater than or equal to a preset difference, it is determined that the target hard disk meets the preset migration conditions, and a migration instruction is generated based on the identification information of the target hard disk.

[0196] In one embodiment, the data migration device further includes:

[0197] A detection module is used to detect the amount of junk data in each data block, and calculate the proportion of the junk data in the data block based on the amount of junk data and the total data space; and determine the hot and cold data types of the data block based on the difference between the latest access time of the data block and a preset time;

[0198] The arrangement module is used to arrange each data block based on the garbage amount ratio of each data block and the hot and cold data types of each data block to obtain a target data table, wherein the target data table includes multiple garbage amount levels arranged in order of garbage amount size, and each garbage amount level includes multiple data blocks divided into cold data block type and hot data block type.

[0199] In one embodiment, the data migration device further includes:

[0200] A removal module, used for removing each of the data blocks to be migrated from the target data table;

[0201] The first query module is specifically used for:

[0202] The target data table is queried for each module to be migrated. If the module to be migrated is not found in the target data table, the garbage volume ratio and the hot and cold data types of the next module to be migrated are queried.

[0203] In one embodiment, the second determining module is specifically configured to:

[0204] Determining the garbage level of each of the data blocks to be migrated based on the garbage amount ratio of each of the data blocks to be migrated;

[0205] Sorting the data blocks to be migrated in order from low to high according to the garbage amount level to obtain an initial migration order;

[0206] In each of the garbage amount levels, the data blocks to be migrated are sorted according to the sequence of the data blocks to be migrated from cold data blocks to hot data blocks to obtain a target migration sequence.

[0207] In one embodiment, the migration module is specifically used for:

[0208] Migrate each of the to-be-migrated data blocks to the target data space according to the target migration order and the maximum number of single migrations.

[0209] In one embodiment, the data migration device further includes:

[0210] A second query module is used to query the garbage amount ratio and hot and cold data types of each to-be-reclaimed data block corresponding to the data recovery instruction through the target data table after receiving the data recovery instruction;

[0211] The third determination module is used to determine the garbage level of each of the to-be-recovered data blocks based on the garbage level ratio of each of the to-be-recovered data blocks; sort the to-be-recovered data blocks in order from high to low according to the garbage level level to obtain an initial recovery order; in each of the garbage level levels, sort the to-be-recovered data blocks in order from cold data blocks to hot data blocks to obtain a target recovery order;

[0212] The valid data migration module is used to migrate the valid data in each of the to-be-reclaimed data blocks according to the target recovery order, and release the original data space where each of the to-be-reclaimed data blocks is located after the migration process is completed.

[0213] Each module in the above data migration device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0214] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Fig.10 As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store relevant data of the target data table. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a data migration method is implemented.

[0215] Those skilled in the art will understand that Fig.10 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0216] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0217] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0218] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0219] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0220] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0221] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0222] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A data migration method, It is characterized in that The method comprises: After receiving the migration instruction, determining a target hard disk corresponding to the migration instruction, and determining each to-be-migrated data block corresponding to the target hard disk; Through the target data table, query the garbage volume ratio and hot and cold data types of each of the data blocks to be migrated; Determining a target migration order of each of the data blocks to be migrated according to the garbage amount ratio and the hot and cold data types of each of the data blocks to be migrated; Migrate each of the to-be-migrated data blocks to the target data space according to the target migration order, and release the original data space where each of the to-be-migrated data blocks is located after the data blocks are migrated.

2. The method according to claim 1, It is characterized in that The method further comprises: Obtaining the number of used objects of the target hard disk, and obtaining the average number of used objects of the storage space corresponding to the target hard disk; Calculate the difference between the number of used objects of the target hard disk and the average number of used objects of the storage space; if the difference is greater than or equal to a preset difference, determine that the target hard disk meets the preset migration condition, and generate a migration instruction based on the identification information of the target hard disk.

3. The method according to claim 1, It is characterized in that The method further comprises: For each data block, detect the amount of junk data in the data block, and calculate the junk amount ratio of the data block based on the amount of junk data and the total data space; and determine the hot and cold data types of the data block based on the difference between the latest access time of the data block and a preset time; Based on the garbage amount ratio of each data block and the hot and cold data types of each data block, each data block is arranged to obtain a target data table, wherein the target data table includes multiple garbage amount levels arranged in order of garbage amount size, and each garbage amount level includes multiple data blocks divided into cold data block type and hot data block type.

4. The method according to claim 3, It is characterized in that After the step of migrating the data blocks to be migrated to the target data space according to the target migration order and releasing the original data space where the data blocks to be migrated are located after the data blocks are migrated, the method further includes: Eliminate each of the data blocks to be migrated from the target data table; The querying of the garbage volume ratio and hot and cold data types of each of the to-be-migrated data blocks through the target data table includes: The target data table is queried for each module to be migrated. If the module to be migrated is not found in the target data table, the garbage volume ratio and the hot and cold data types of the next module to be migrated are queried.

5. The method according to claim 1, It is characterized in that The step of determining the target migration order of each of the data blocks to be migrated according to the garbage amount ratio and the hot and cold data types of each of the data blocks to be migrated includes: Determining the garbage level of each of the data blocks to be migrated based on the garbage amount ratio of each of the data blocks to be migrated; Sorting the data blocks to be migrated in order from low to high according to the garbage amount level to obtain an initial migration order; In each of the garbage amount levels, the data blocks to be migrated are sorted according to the sequence of the data blocks to be migrated from cold data blocks to hot data blocks to obtain a target migration sequence.

6. The method according to claim 1, It is characterized in that The step of migrating each of the to-be-migrated data blocks to the target data space according to the target migration order includes: Migrate each of the to-be-migrated data blocks to the target data space according to the target migration order and the maximum number of single migrations.

7. The method according to claim 3, It is characterized in that The method further comprises: After receiving the data recovery instruction, querying the garbage amount ratio and hot and cold data types of each to-be-recovered data block corresponding to the data recovery instruction through the target data table; Based on the garbage amount ratio of each of the to-be-reclaimed data blocks, the garbage amount level of each of the to-be-reclaimed data blocks is determined; in the order of the garbage amount levels from high to low, the to-be-reclaimed data blocks are sorted to obtain an initial recycling order; in each of the garbage amount levels, in the order of the to-be-reclaimed data blocks from cold data blocks to hot data blocks, the to-be-reclaimed data blocks are sorted to obtain a target recycling order; According to the target recovery order, migration processing is performed on the valid data in each of the to-be-recovered data blocks respectively, and after the migration processing is completed, the original data space where each of the to-be-recovered data blocks is located is released.

8. A data migration device, It is characterized in that The device comprises: A first determination module is used to determine, after receiving a migration instruction, a target hard disk corresponding to the migration instruction and determine each to-be-migrated data block corresponding to the target hard disk; A first query module is used to query the garbage volume ratio and hot and cold data types of each of the data blocks to be migrated through the target data table; A second determination module is used to determine the target migration order of each of the data blocks to be migrated according to the garbage amount ratio and the hot and cold data types of each of the data blocks to be migrated; The migration module is used to migrate each of the to-be-migrated data blocks to the target data space according to the target migration sequence, and release the original data space where each of the to-be-migrated data blocks is located after the data blocks are migrated.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program. It is characterized in that When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images