Data migration method and device based on deduplication, computer equipment and program product

Abstract

The invention relates to a data migration method and device based on deduplication, computer equipment, a storage medium and a computer program product. The method comprises the steps of performing recovery analysis on target data, and determining initial migration data in the target data; and carrying out deduplication analysis on the initial migration data, determining actual migration data corresponding to the initial migration data, and carrying out migration processing on the actual migration data. By the adoption of the method, after recovery analysis is conducted, the data is subjected to deduplication check operation, that is, the actually moved data needing to actually execute the movement action can be further screened out according to whether the data meets the deduplication requirement or not, the data size actually needing to be moved is reduced, the storage space in a storage system is saved, the deduplication rate is increased, and the data recovery efficiency is improved. In addition, due to the fact that data actually moved is reduced, write amplification is correspondingly reduced, and the service life of the flash memory is prolonged.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer storage, and in particular to a data migration method and device based on deduplication, a computer device, a storage medium, and a computer program product. BACKGROUND

[0002] With the continuous development of storage technology, a non-volatile memory, flash memory, has emerged, which can ensure high operation speed when erasing or programming. Flash memory is widely used in various devices, such as solid state drives (SSDs), mobile devices (such as smartphones and tablets), etc. All-flash storage refers to a storage system that completely uses flash memory technology without mechanical hard disk drives. It contains both disk drives and flash memory, which can provide higher performance and lower access latency. In all-flash storage, due to the characteristics of flash memory, garbage collection needs to be performed.

[0003] In the conventional technology, when garbage collection is performed, all the data to be migrated is migrated, which leads to an increase in write amplification and a decrease in the performance of the storage system. SUMMARY

[0004] Therefore, it is necessary to provide a data migration method and device based on deduplication that can reduce write amplification in order to solve the above technical problems.

[0005] In a first aspect, the present application provides a data migration method based on deduplication, applied to a terminal, the method comprising:

[0006] performing a recycling analysis on target data to determine initial migration data in the target data;

[0007] performing a deduplication analysis on the initial migration data to determine actual migration data corresponding to the initial migration data, and performing migration processing on the actual migration data.

[0008] In this embodiment, the actual migration data that needs to be actually executed is further screened out by whether the data meets the deduplication requirement, which reduces the amount of data that needs to be actually migrated, saves storage space in the storage system, improves the deduplication rate, and correspondingly reduces the write amplification due to the reduction in the data that is actually migrated, thereby prolonging the service life of the flash memory.

[0009] In one of the embodiments, the performing a recycling analysis on target data to determine initial migration data in the target data comprises:

[0010] determining valid data in the target data based on the state of the target data;

[0011] The effective data is determined as initial migration data.

[0012] In the embodiment, the data is analyzed, the initial migration data is determined based on the current state of the data, the storage space of the data block where the target data is located is released in time, and the storage performance of the storage system is ensured.

[0013] In one of the embodiments, the initial migration data includes multiple pieces of data; the initial migration data is analyzed by deduplication to determine the actual migration data corresponding to the initial migration data, including:

[0014] Each piece of data included in the initial migration data is analyzed by deduplication to determine the fingerprint value of each piece of data;

[0015] If there is a fingerprint value that meets the preset similarity condition, the data corresponding to the fingerprint value is determined as duplicate data;

[0016] The actual migration data is determined based on the initial migration data and the duplicate data.

[0017] In the embodiment, after the recycling analysis, the data is checked by deduplication, and the actual migration data that needs to be actually migrated is further screened out by whether the data meets the deduplication requirement, so as to reduce the amount of data that needs to be actually migrated. By determining the data with the same hash value as duplicate data, the integrity of the data during data migration can be ensured, and data loss can be avoided.

[0018] In one of the embodiments, the initial migration data is located in a first data block; the actual migration data is migrated, including:

[0019] The actual migration data is migrated from the first data block to a second data block.

[0020] In the embodiment, after the actual migration data is moved from the first data block to the second data block, the first data block is emptied, the storage space occupied by the invalid data is released, and the storage performance of the storage system where the data block is located is improved.

[0021] In one of the embodiments, the initial migration data is analyzed by deduplication to determine the actual migration data corresponding to the initial migration data, including:

[0022] The fingerprint table corresponding to the data stored in the second data block is obtained;

[0023] The initial migration data is analyzed by deduplication based on the data fingerprint table to determine the actual migration data.

[0024] In the embodiment, the fingerprint value duplication checking can be performed based on the fingerprint table of the data block to be received when determining the actual migration data, so as to further reduce the migration processing of the duplicate data, reduce the actual data amount to be migrated, and reduce the write amplification.

[0025] In one of the embodiments, the initial migration data includes a plurality of data; and the deduplication analysis on the initial migration data based on the data fingerprint table to obtain the actual migration data includes:

[0026] The fingerprint value of each data included in the initial migration data is calculated respectively;

[0027] The data whose fingerprint value is not found in the data fingerprint table is determined as the actual migration data.

[0028] In the embodiment, the fingerprint value duplication checking can be performed based on the fingerprint table of the data block to be received when determining the actual migration data, so as to further reduce the actual data amount to be migrated.

[0029] In one of the embodiments, the threads running on the terminal include a first thread and a second thread; and the garbage collection analysis on the target data to determine the initial migration data in the target data includes:

[0030] The garbage collection analysis on the target data is performed by the first thread to determine the initial migration data in the target data;

[0031] The deduplication analysis on the initial migration data to determine the actual migration data corresponding to the initial migration data includes:

[0032] A second thread satisfying a preset same core condition with the first thread is determined;

[0033] The deduplication analysis on the initial migration data is performed by the second thread to determine the actual migration data corresponding to the initial migration data.

[0034] In the embodiment, the threads belonging to the same CPU core perform the garbage collection analysis and the deduplication analysis, so as to improve the data transmission efficiency.

[0035] In a second aspect, the application further provides a data migration device based on deduplication, applied to a terminal, and the device includes:

[0036] A first determining module is configured to perform garbage collection analysis on target data to determine initial migration data in the target data;

[0037] The second determining module is configured to perform deduplication analysis on the initial migration data, determine actual migration data corresponding to the initial migration data, and perform migration processing on the actual migration data.

[0038] In one of the embodiments, the first determining module is specifically configured to:

[0039] determine valid data in the target data based on a state of the target data;

[0040] determine the valid data as initial migration data.

[0041] In one of the embodiments, the initial migration data includes a plurality of data; and the second determining module is specifically configured to:

[0042] perform deduplication analysis on each piece of data included in the initial migration data to determine a fingerprint value of each piece of data;

[0043] if there is a fingerprint value satisfying a preset similarity condition, determine data corresponding to the fingerprint value as repeated data;

[0044] determine actual migration data based on the initial migration data and the repeated data.

[0045] In one of the embodiments, the initial migration data is located in a first data block; and the second determining module is specifically configured to:

[0046] migrate the actual migration data from the first data block to a second data block.

[0047] In one of the embodiments, the second determining module is specifically configured to:

[0048] obtain a fingerprint table corresponding to data stored in the second data block;

[0049] perform deduplication analysis on initial migration data based on the data fingerprint table to determine actual migration data.

[0050] In one of the embodiments, the initial migration data includes a plurality of data; and the second determining module is specifically configured to:

[0051] respectively calculate fingerprint values of each piece of data included in the initial migration data;

[0052] determine data for which no fingerprint value is queried in the data fingerprint table as the actual migration data.

[0053] In one of the embodiments, the threads running on the terminal include a first thread and a second thread; and the first determining module is specifically configured to:

[0054] determining initial move data in the target data by the first thread;

[0055] The second determining module is specifically configured to:

[0056] determine a second thread that satisfies a preset same kernel condition with the first thread;

[0057] performing re-deletion analysis on the initial move data by the second thread to determine actual move data corresponding to the initial move data.

[0058] In a third aspect, the present application further provides a computer device, comprising a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0059] performing re-deletion analysis on the initial move data to determine actual move data corresponding to the initial move data, and performing move processing on the actual move data.

[0060] performing re-deletion analysis on the initial move data to determine actual move data corresponding to the initial move data, and performing move processing on the actual move data.

[0061] In a fourth aspect, the present application further provides a computer readable storage medium, which stores a computer program, and the computer program implements the following steps when executed by a processor:

[0062] performing re-deletion analysis on the initial move data to determine actual move data corresponding to the initial move data, and performing move processing on the actual move data.

[0063] performing re-deletion analysis on the initial move data to determine actual move data corresponding to the initial move data, and performing move processing on the actual move data.

[0064] In a fifth aspect, the present application further provides a computer program product, comprising a computer program, and the computer program implements the following steps when executed by a processor:

[0065] performing re-deletion analysis on the initial move data to determine actual move data corresponding to the initial move data, and performing move processing on the actual move data.

[0066] performing re-deletion analysis on the initial move data to determine actual move data corresponding to the initial move data, and performing move processing on the actual move data.

[0067] The method, device, computer device, storage medium and computer program product based on deduplication for data migration, the method comprising: performing recycling analysis on target data, determining initial migration data in the target data; performing deduplication analysis on the initial migration data, determining actual migration data corresponding to the initial migration data, and performing migration processing on the actual migration data. By using the method, after recycling analysis, the data is subjected to deduplication checking operation, that is, the actual migration data that needs to be actually executed is further screened out by whether the data meets the deduplication requirement, the amount of data that needs to be actually migrated is reduced, the storage space in the storage system is saved, the deduplication rate is improved, and since the data that needs to be actually migrated is reduced, the write amplification is also reduced, and the service life of the flash memory is prolonged. BRIEF DESCRIPTION OF DRAWINGS

[0068] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related art, the drawings needed to be used in the embodiments or the related art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0069] Figure 1 A flowchart of a data migration method based on deduplication in an embodiment;

[0070] Figure 2 A flowchart of a step of determining initial migration data in an embodiment;

[0071] Figure 3 A flowchart of a step of determining actual migration data in an embodiment;

[0072] Figure 4 A flowchart of a step of actual migration data in an embodiment;

[0073] Figure 5 A flowchart of a step of actual migration data in an embodiment;

[0074] Figure 6 A flowchart of a step of actual migration data in an embodiment;

[0075] Figure 7 A flowchart of a data migration method based on deduplication in another embodiment;

[0076] Figure 8 A structural block diagram of a data migration device based on deduplication in an embodiment;

[0077] Figure 9 An internal structure diagram of a computer device in an embodiment. DETAILED DESCRIPTION

[0078] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not intended to limit the present application.

[0079] In one exemplary embodiment, as shown in Figure 1 A data migration method based on deduplication is provided, which is applied to a terminal for example. It can be understood that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction of the terminal and the server. The terminal can be, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers, Internet of Things devices and portable wearable devices. The Internet of Things device can be a smart speaker, a smart television, a smart air conditioner, a smart vehicle device, etc. The portable wearable device can be a smart watch, a smart bracelet, a head-mounted device, etc. The server can be implemented by an independent server or a server cluster composed of multiple servers. In this embodiment, the data migration method based on deduplication includes the following steps:

[0080] In step 102, the target data is analyzed for garbage collection, and initial migration data is determined in the target data.

[0081] The target data can be data that needs to be garbage collected, for example, data in a data block that has data collection requirements. The initial migration data can be part of the target data, for example, data that can be migrated when the target data needs garbage collection processing, etc. It can also be data that can be migrated based on the valid state of the data in the target data.

[0082] Specifically, the terminal can obtain target data and analyze the target data for garbage collection to determine initial migration data in the target data that needs to be migrated. Optionally, the target data block can store multiple pieces of data, and the target data can include the target data block. The terminal can determine the initial migration data in the target data block based on the current state of each piece of data. In one example, the terminal can perform garbage collection on the target data to determine the initial migration data in the target data.

[0083] In step 104, the initial migration data is analyzed for deduplication, actual migration data corresponding to the initial migration data is determined, and the actual migration data is migrated.

[0084] The actual migration data corresponding to the initial migration data can be data that meets the deduplication condition in the initial migration data.

[0085] Specifically, after the terminal performs the recycling analysis on the target data and determines the initial migration data to be moved based on the current state of each piece of data, the terminal can perform deduplication analysis on the initial migration data, determine the data satisfying the deduplication condition as the actual migration data. Based on this, the terminal can perform migration processing on the actual migration data in the target data, for example, can move the actual migration data to other data blocks, and perform emptying processing on the data block where the target data is located, that is, perform erasing processing on the data block where the target data is located, to release the storage space occupied by the target data.

[0086] In the above data migration method based on deduplication, the target data is analyzed for recycling, and the initial migration data is determined in the target data; the initial migration data is analyzed for deduplication, the actual migration data corresponding to the initial migration data is determined, and the actual migration data is moved. By using the method, after the recycling analysis, the deduplication checking operation is performed on the data, that is, the actual migration data that needs to be actually moved can be further screened out by whether the data meets the deduplication requirement, the amount of data that needs to be actually moved is reduced, the storage space in the storage system is saved, the deduplication rate is improved, and since the amount of data that needs to be actually moved is reduced, the write amplification is also reduced, and the service life of the flash memory is prolonged.

[0087] In an exemplary embodiment, as shown in Figure 2 The specific implementation process of the step of "performing recycling analysis on the target data to determine the initial migration data in the target data" can include:

[0088] In step 202, valid data is determined in the target data based on the state of the target data.

[0089] The state of the target data can be the current state of the target data, which can include a valid state or an invalid state. The target data can include multiple pieces of data.

[0090] Specifically, the process of performing recycling analysis on the target data by the terminal can actually be that the terminal acquires the target data and determines the current state of each piece of data included in the target data. The current state of each piece of data can be a valid state or an invalid state. The terminal can determine the data with the valid state as valid data; that is, the terminal performs garbage analysis on the target data to determine the valid data in the panel data. Alternatively, the terminal can perform recycling analysis on the target data, determine invalid data in the target data based on the current state of each piece of data included in the target data, and determine valid data in the target data based on the invalid data.

[0091] In step 204, the valid data is determined as the initial migration data.

[0092] Specifically, the valid data is data that needs to be moved, based on which the terminal can determine the valid data in the target data as the initial moving data.

[0093] In this embodiment, the data can be recycled and analyzed, and the initial moving data that needs to be moved can be determined based on the current state of the data, so that the storage space of the data block where the target data is located can be released in time, and the storage performance of the storage system can be ensured.

[0094] In one example embodiment, the initial moving data includes a plurality of data.

[0095] Correspondingly, as shown in Figure 3 The specific implementation process of the step of "performing deduplication analysis on the initial moving data to determine the actual moving data corresponding to the initial moving data" can include:

[0096] In step 302, deduplication analysis is performed on each piece of data included in the initial moving data to determine the fingerprint value of each piece of data.

[0097] The fingerprint value of the data is used to determine the similarity between the data, and the fingerprint value of the data can be obtained by hashing calculation through a hash algorithm.

[0098] Specifically, for each piece of data included in the initial moving data, the terminal can perform hashing calculation on the piece of data through a hash algorithm to obtain a hash value, and use the hash value as the fingerprint value of the piece of data.

[0099] In step 304, if there is a fingerprint value that meets the preset similarity condition, the data corresponding to the fingerprint value is determined as duplicate data.

[0100] The preset similarity condition is used to determine whether the data is similar, and the content of the preset similarity condition can be that the hash values of the two pieces of data are the same.

[0101] Specifically, the terminal can filter based on the fingerprint values of the pieces of data included in the initial moving data to determine the data that meets the preset similarity condition, and determine the data that meets the preset similarity condition as duplicate data; alternatively, based on the fingerprint values of the pieces of data included in the initial moving data, the terminal can filter the fingerprint values of the pieces of data to filter out the same fingerprint values, and determine the data corresponding to the same fingerprint values as duplicate data.

[0102] In one example, for any one of the fingerprint values of the data contained in the initial migration data, the terminal can traverse the fingerprint values of other data contained in the initial migration data to determine whether there is a fingerprint value identical to the fingerprint value, and if there is a fingerprint value identical to the fingerprint value, the fingerprint value is determined as the same fingerprint value. The terminal can determine the data corresponding to the same fingerprint value as the repeated data.

[0103] Optionally, the process of determining the data corresponding to the fingerprint value as the repeated data can include that the terminal can determine the plurality of data with the fingerprint value being the same fingerprint value as the repeated data.

[0104] Step 306, determining the actual migration data based on the initial migration data and the repeated data.

[0105] The repeated data contains a plurality of data with the same fingerprint value, or the repeated data can also contain a plurality of groups of data with the same fingerprint value, and the fingerprint values of each group are different.

[0106] Specifically, the terminal can filter the initial migration data based on the repeated data to obtain the actual migration data that needs to be actually migrated. In one example, the repeated data contains a plurality of data with the same fingerprint value, the terminal can randomly extract one data from the repeated data, and determine the data and other data in the initial migration data except the repeated data as the actual migration data; in one example, the repeated data can contain a plurality of groups of data with the same fingerprint value, the terminal can randomly extract one data from each group of data with the same fingerprint value, and determine the extracted data of each group and other data in the initial migration data except the repeated data as the actual migration data.

[0107] For example, the initial migration data can contain first data, second data, third data, fourth data and fifth data, the terminal can determine that the fingerprint values of the first data and the second data are the same, and the fingerprint values of the third data and the fourth data are the same through the duplicate analysis on the above data. Based on this, the terminal can determine the first data, the second data, the third data and the fourth data as the repeated data, and the repeated data contains two groups of data with the same fingerprint value. The terminal can randomly extract one data from the first data and the second data, for example, it can be the first data, and the terminal can also randomly extract one data from the third data and the fourth data, for example, it can be the third data. In this way, the terminal can determine the first data, the third data and other data in the initial migration data except the repeated data (i.e. the fifth data) as the actual migration data.

[0108] In another example, the terminal can update the actual migration data based on the data fingerprint table of the second data block to obtain updated actual migration data. Specifically, for each piece of data included in the actual migration data, the terminal can perform traversal processing on the data fingerprint table corresponding to the second data block based on the fingerprint value of the piece of data, determine whether there is a fingerprint value that meets the preset similarity condition with the fingerprint value of the piece of data, that is, determine whether there is a fingerprint value that is the same as the fingerprint value of the piece of data, and if there is a fingerprint value that is the same as the fingerprint value of the piece of data, determine that the piece of data is existing data.

[0109] In this way, after traversal processing is performed on each piece of data included in the actual migration data, the terminal can determine the existing data in the actual migration data, and the terminal can determine other data in the actual migration data except the existing data as the updated actual migration data. Based on this, the terminal can perform migration data on the updated actual migration data, move the updated actual migration data to the second data block, and perform erasure processing on the first data block.

[0110] In this embodiment, after the recycling analysis is performed, the data is subjected to the re-deletion checking operation, and whether the data meets the re-deletion requirement can further filter out the actual migration data that needs to be actually executed. The amount of data that actually needs to be migrated is reduced, and by determining the data with the same hash value as the duplicate data, the integrity of the data retention during data migration can be ensured, and data loss is avoided.

[0111] In an exemplary embodiment, the initial migration data is located in the first data block. That is, the target data can be located in the first data block, that is, the terminal can perform recycling analysis on the target data in the first data block to determine the initial migration data.

[0112] Correspondingly, the specific implementation process of the step "performing migration processing on the actual migration data" can include:

[0113] Migrating the actual migration data from the first data block to the second data block.

[0114] The second data block can be a new data block for storing data.

[0115] Specifically, after the terminal determines the actual migration data corresponding to the target data, the terminal can perform migration processing on the actual migration data, move the actual migration data from the first data block to the second data block, and after moving the actual migration data from the first data block to the second data block, perform data emptying processing on the first data block, that is, perform erasure processing on the first data block, to release the storage space occupied by the first data block.

[0116] In this embodiment, the data emptying processing can be performed on the first data block after the actual moving data is moved from the first data block to the second data block, the release of the storage space occupied by the invalid data is completed, and the storage performance of the storage system where the data block is located is improved.

[0117] In one exemplary embodiment, as shown in Figure 4 The specific implementation process of the step of "performing deduplication analysis on the initial moving data to determine the actual moving data corresponding to the initial moving data" can include:

[0118] In step 402, a fingerprint table corresponding to the data stored in the second data block is obtained.

[0119] The second data block can be a data block to which the actual moving data needs to be moved. The fingerprint table can be obtained by performing a hash calculation on the data stored in the second data block by using a hash algorithm. The fingerprint table includes fingerprint values corresponding to each piece of data stored in the second data block.

[0120] Specifically, after determining that the actual moving data needs to be moved to the second data block, the terminal can obtain the fingerprint table corresponding to the second data block, which can be directly reading the fingerprint table corresponding to the second data block, or performing a hash calculation on each piece of data stored in the second data block to obtain the fingerprint values corresponding to each piece of data.

[0121] In step 404, based on the data fingerprint table, deduplication analysis is performed on the initial moving data to determine the actual moving data.

[0122] Specifically, the terminal can perform a hash calculation on each piece of data included in the initial moving data by using a hash algorithm to obtain hash values corresponding to each piece of data, and use the hash values corresponding to each piece of data as the fingerprint values corresponding to each piece of data. Based on the fingerprint values corresponding to each piece of data included in the initial moving data, the terminal can perform deduplication analysis based on the data fingerprint table in the second data block, and extract the actual moving data from the initial moving data.

[0123] In this embodiment, when the actual moving data is determined, the fingerprint value duplication checking processing can be performed based on the fingerprint table of the data block to be received, which further reduces the moving processing of the duplicate data, reduces the amount of data that actually needs to be moved, and reduces the write amplification.

[0124] In one exemplary embodiment, the initial moving data includes multiple pieces of data.

[0125] Correspondingly, as shown in Figure 5 The specific implementation process of the step of "performing deduplication analysis on the initial moving data to determine the actual moving data corresponding to the initial moving data" can include:

[0126] Step 502, respectively calculate the fingerprint value of each data contained in the initial migration data.

[0127] Specifically, the terminal can perform hash calculation on each data contained in the initial migration data through a hash algorithm, obtain the hash value corresponding to each data respectively, and take the hash value corresponding to each data respectively as the fingerprint value corresponding to each data respectively.

[0128] Step 504, determine the data whose fingerprint value is not queried in the data fingerprint table as the actual migration data.

[0129] Specifically, for each data contained in the initial migration data, the terminal can perform traversal processing in the data fingerprint table corresponding to the second data block based on the fingerprint value of the data, judge whether there is a fingerprint value that satisfies the preset similarity condition with the fingerprint value of the data, that is, judge whether there is a fingerprint value that is the same as the fingerprint value of the data, if there is no fingerprint value that is the same as the fingerprint value of the data in the data fingerprint table, determine the data as the actual migration data. After traversal processing of each data contained in the initial migration data, the terminal can obtain the actual migration data.

[0130] Optionally, the terminal judges whether there is a fingerprint value that is the same as the fingerprint value of the data, if there is a fingerprint value that is the same as the fingerprint value of the data in the data fingerprint table, determines the data as the existing data. In this way, after traversal processing of each data contained in the initial migration data, the terminal can determine the existing data in the initial migration data, and the terminal can determine the data other than the existing data in the initial migration data as the actual migration data.

[0131] In this embodiment, the fingerprint value duplication checking processing can be performed through the fingerprint table of the data block to be received, and the actual migration data can be determined by whether there is a fingerprint value that is the same as the fingerprint value of the initial migration data in the data fingerprint table, which can further reduce the amount of data that actually needs to be migrated.

[0132] In one exemplary embodiment, the threads running on the terminal include a first thread and a second thread.

[0133] Correspondingly, the specific implementation process of the step of "performing garbage collection analysis on the target data, and determining the initial migration data in the target data" can include:

[0134] Perform garbage collection analysis on the target data through the first thread, and determine the initial migration data in the target data.

[0135] Specifically, the terminal can perform garbage collection analysis on the target data through the first thread, and determine the initial migration data in the target data.

[0136] Accordingly, as shown in Figure 6 the step of performing deduplication analysis on the initial migration data to determine actual migration data corresponding to the initial migration data can include:

[0137] In step 602, a second thread that satisfies a preset same core condition with the first thread is determined.

[0138] The preset same core condition can be that the same CPU core is used.

[0139] Specifically, the terminal can perform garbage collection analysis on the target data through the first thread to determine the initial migration data in the target data. The terminal can filter threads belonging to the same CPU core as the first thread from the plurality of threads currently running, and determine the threads belonging to the same CPU core as the first thread as the second thread.

[0140] Optionally, if a plurality of threads belonging to the same CPU core as the first thread are filtered from the plurality of threads currently running, the terminal can determine the thread with the lowest usage rate as the second thread based on the usage rates of the threads.

[0141] In step 604, the initial migration data is deduplicated by the second thread to determine actual migration data corresponding to the initial migration data.

[0142] Specifically, the terminal can perform deduplication analysis on the initial migration data through the second thread to determine actual migration data corresponding to the initial migration data.

[0143] Optionally, after the terminal performs garbage collection analysis on the target data through the first thread to determine the initial migration data, the second thread can be determined, and a message containing the initial migration data can be sent to the second thread. The second thread of the terminal can perform deduplication analysis on the initial migration data based on the message to determine actual migration data corresponding to the initial migration data after receiving the message containing the initial migration data.

[0144] In this embodiment, performing collection analysis and deduplication analysis through threads belonging to the same CPU core can improve data transfer efficiency.

[0145] The execution process of the deduplication-based data migration method will be described in detail below in conjunction with a detailed embodiment. The method includes:

[0146] The data migration method based on deduplication provided by the embodiment is to add a deduplication checking operation to the data to be migrated before starting the actual data migration in the garbage collection, and the data block meeting the deduplication requirement can be subjected to a deduplication process, so that the data migration method based on deduplication can reduce the amount of data to be actually executed in the migration operation. As shown in Figure 7 The data migration method based on deduplication includes the following steps.

[0147] Step 1, garbage amount analysis is performed, and data can be read, for example, the read data can include data0, data1 and data2; that is, the data before migration can include data0, data1 and data2.

[0148] Step 2, deduplication analysis is initiated for the data to be migrated, and is input to the background deduplication for data analysis; specifically, the data to be migrated (initial migration data) determined in the garbage collection analysis can be data0 and data2.

[0149] Steps 3 and 4, calculate the fingerprint and compare it with the fingerprint table to find the data that can be deduplicated; data analysis is performed in the background deduplication, for example, comparison can be made based on the fingerprint and the fingerprint table, and it is determined that the above data is duplicate data in data0 and data2, so only one of the duplicate data needs to be migrated, that is, one of data0 and data2 can be randomly determined as the actual migration data.

[0150] Step 5, data reduction and metadata modification are performed.

[0151] Step 6, return the actual migration data (actual migration data) after deduplication.

[0152] Step 7, perform the migration operation based on the actual migration data, for example, in the case of data0 as the actual migration data, data0 can be migrated from the first data block to the second data block, and the storage space in the first data block is released, that is, the first data block is subjected to erasure processing.

[0153] Step 8, data migration and metadata modification.

[0154] In the data migration method based on deduplication provided in this embodiment, the data volume input of the current garbage collection can be data that needs to be processed by garbage collection, and data0 and data2 can be determined as valid data (initial migration data) that needs to be migrated. Before performing the migration operation, the terminal can initiate a deduplication process for the valid data (deduplication analysis is performed on the initial migration data). After the deduplication analysis, it is found that data0 and data2 are duplicate data, and only one copy needs to be migrated. Then, the actual data (actual migration data) that needs to be migrated is returned to the garbage collection module, so as to complete the recovery of the storage space occupied by the invalid data data1 and the duplicate data data2. Alternatively, if the same fingerprint as data0 / data2 already exists in the fingerprint table, the current garbage collection does not need to migrate data, and only the metadata needs to be modified.

[0155] The data migration method based on deduplication provided in this embodiment can reduce the data migration volume and reduce the write amplification effect by introducing deduplication in the garbage collection migration process. The data migration method based on deduplication provided in this embodiment increases the deduplication application scenario, increases deduplication in the garbage collection process, performs deduplication analysis on the data before migration, and only performs the migration operation on the data returned after deduplication execution, thereby reducing the actual migration data volume, saving storage space, and improving the deduplication rate. The data migration method based on deduplication provided in this embodiment can improve the deduplication rate while improving the garbage collection efficiency. Due to the reduction of the migration data volume, the write amplification is reduced, and the service life of the flash SSD is prolonged.

[0156] It should be understood that, although each step in the flowchart involved in each embodiment as described above is displayed in sequence according to the arrow, these steps are not necessarily executed in sequence according to the arrow. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other sequences. Moreover, at least part of the steps in the flowchart involved in each embodiment as described above can include multiple steps or stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution sequence of these steps or stages is not necessarily sequential, but can be alternately or alternately executed with at least part of other steps or steps or stages in other steps.

[0157] Based on the same inventive concept, the embodiments of the present application also provide a deduplication-based data migration device for implementing the deduplication-based data migration method described above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme described in the above method, and therefore the specific limitations in one or more deduplication-based data migration device embodiments provided below can refer to the limitations of the deduplication-based data migration method described above, which will not be repeated here.

[0158] In one example embodiment, as shown in Figure 8 A data migration device 800 based on deduplication is provided, applied to a terminal and comprising:

[0159] A first determining module 802 is configured to perform recycle analysis on target data, and determine initial migration data in the target data.

[0160] A second determining module 804 is configured to perform deduplication analysis on the initial migration data, determine actual migration data corresponding to the initial migration data, and perform migration processing on the actual migration data.

[0161] In one example embodiment, the first determining module is specifically configured to:

[0162] determine valid data in the target data based on a state of the target data;

[0163] determine the valid data as the initial migration data.

[0164] In one example embodiment, the initial migration data comprises a plurality of data; and the second determining module is specifically configured to:

[0165] perform deduplication analysis on each piece of data contained in the initial migration data, and determine a fingerprint value of each piece of data;

[0166] if there is a fingerprint value satisfying a preset similarity condition, determine data corresponding to the fingerprint value as repeated data;

[0167] determine actual migration data based on the initial migration data and the repeated data.

[0168] In one example embodiment, the initial migration data is located in a first data block; and the second determining module is specifically configured to:

[0169] migrate the actual migration data from the first data block to a second data block.

[0170] In one example embodiment, the second determining module is specifically configured to:

[0171] obtain a fingerprint table corresponding to data stored in the second data block;

[0172] perform deduplication analysis on the initial migration data based on the data fingerprint table, and determine actual migration data.

[0173] In one example embodiment, the initial migration data comprises a plurality of data; and the second determining module is specifically configured to:

[0174] calculate a fingerprint value of each piece of data included in the initial moving data respectively;

[0175] determine data for which no fingerprint value is queried in the data fingerprint table as the actual moving data.

[0176] In one of the embodiments, the threads running on the terminal include a first thread and a second thread; the first determining module is specifically configured to:

[0177] perform garbage collection analysis on the target data through the first thread, and determine initial moving data in the target data;

[0178] the second determining module is specifically configured to:

[0179] determine a second thread that satisfies a preset same kernel condition as the first thread;

[0180] perform deduplication analysis on the initial moving data through the second thread, and determine actual moving data corresponding to the initial moving data.

[0181] The modules in the deduplication-based data moving apparatus described above can be realized by software, hardware, and combinations thereof, in whole or in part. The modules described above can be embedded in or independent of a processor in a computer device in hardware form, or can be stored in a memory in a computer device in software form, so as to be called and executed by a processor to perform operations corresponding to the modules.

[0182] In one exemplary embodiment, a computer device, which can be a server, is provided, and an internal structure diagram of the computer device can be as shown in Figure 9 The computer device includes a processor, a memory, an input / output interface (I / O), and a communication interface. The processor, the memory, and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for running the operating system and the computer program in the non-volatile storage medium. The database of the computer device is configured to store target data. The input / output interface of the computer device is configured to exchange information between the processor and external devices. The communication interface of the computer device is configured to communicate with terminals outside through a network connection. The computer program is executed by the processor to implement a deduplication-based data moving method.

[0183] Those skilled in the art can understand that, Figure 9The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0184] In one exemplary embodiment, a computer device is provided, comprising a memory and a processor, the memory storing a computer program, and the processor implementing the following steps when executing the computer program:

[0185] Performing recycle analysis on target data, determining initial migration data in the target data;

[0186] Performing deduplication analysis on the initial migration data, determining actual migration data corresponding to the initial migration data, and performing migration processing on the actual migration data.

[0187] In one embodiment, the processor further implements the following steps when executing the computer program:

[0188] Determining valid data in the target data based on the state of the target data;

[0189] Determining the valid data as the initial migration data.

[0190] In one embodiment, the processor further implements the following steps when executing the computer program:

[0191] Performing deduplication analysis on each piece of data contained in the initial migration data, determining a fingerprint value of each piece of data;

[0192] If there is a fingerprint value that meets a preset similarity condition, determining the data corresponding to the fingerprint value as duplicate data;

[0193] Determining actual migration data based on the initial migration data and the duplicate data.

[0194] In one embodiment, the processor further implements the following steps when executing the computer program:

[0195] Migrating the actual migration data from the first data block to a second data block.

[0196] In one embodiment, the processor further implements the following steps when executing the computer program:

[0197] Obtaining a fingerprint table corresponding to the data stored in the second data block;

[0198] Performing deduplication analysis on initial migration data based on the data fingerprint table, and determining actual migration data.

[0199] In one embodiment, the processor, when executing the computer program, further implements the following steps:

[0200] Respectively calculating the fingerprint value of each piece of data contained in the initial move data;

[0201] Determining the data for which no fingerprint value is found in the data fingerprint table as the actual move data.

[0202] In one embodiment, the processor, when executing the computer program, further implements the following steps:

[0203] Through the first thread, performing garbage collection analysis on the target data to determine the initial move data in the target data;

[0204] In one embodiment, the processor, when executing the computer program, further implements the following steps:

[0205] Determining a second thread that meets a preset same kernel condition with the first thread;

[0206] Through the second thread, performing re-deletion analysis on the initial move data to determine the actual move data corresponding to the initial move data.

[0207] In one embodiment, a computer readable storage medium is provided, and the computer readable storage medium stores a computer program, which, when executed by a processor, implements the steps in the above method embodiments.

[0208] In one embodiment, a computer program product is provided, and the computer program product includes a computer program, which, when executed by a processor, implements the steps in the above method embodiments.

[0209] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant regulations.

[0210] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiments of the methods can be included. Any reference to memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., without being limited thereto.

[0211] Any combination of the technical features of the above embodiments can be made. In order to make the description simple, all possible combinations of the technical features in the above embodiments are not described, however, as long as the combination of the technical features does not exist, it should be considered as the scope of the present application.

[0212] The above embodiments only express several implementation manners of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the patent of the present application. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are all within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A data migration method based on deduplication, characterized in that, Applied to a terminal, the method includes: The target data is recycled and analyzed to determine the initial relocation data within the target data. The initial relocation data is subjected to deduplication analysis to determine the actual relocation data corresponding to the initial relocation data, and the actual relocation data is then processed for relocation.

2. The method according to claim 1, characterized in that, The step of performing recovery analysis on the target data, and determining the initial relocation data within the target data, includes: Based on the state of the target data, determine the valid data in the target data; The valid data is determined as the initial relocation data.

3. The method according to claim 1, characterized in that, The initial relocation data includes multiple data entries; the deduplication analysis of the initial relocation data to determine the actual relocation data corresponding to the initial relocation data includes: The initial relocation data is subjected to deduplication analysis to determine the fingerprint value of each piece of data. If a fingerprint value that meets the preset similarity conditions exists, the data corresponding to the fingerprint value is determined to be duplicate data. Based on the initial relocation data and the repeated data, the actual relocation data is determined.

4. The method according to claim 1, characterized in that, The initial moved data is located in the first data block; the moving process of the actual moved data includes: The actual data to be moved is moved from the first data block to the second data block.

5. The method according to claim 4, characterized in that, The step of performing deduplication analysis on the initial relocation data to determine the actual relocation data corresponding to the initial relocation data includes: Obtain the fingerprint table corresponding to the data stored in the second data block; Based on the data fingerprint table, deduplication analysis is performed on the initial relocation data to determine the actual relocation data.

6. The method according to claim 5, characterized in that, The initial relocation data includes multiple data entries; the deduplication analysis of the initial relocation data based on the data fingerprint table to obtain the actual relocation data includes: Calculate the fingerprint value of each data entry contained in the initial relocation data; Data for which no fingerprint value is found in the data fingerprint table is identified as the actual relocation data.

7. The method according to claim 1, characterized in that, The threads running on the terminal include a first thread and a second thread; the process of reclaiming and analyzing the target data, and determining the initial data to be moved from the target data, includes: The first thread performs garbage collection analysis on the target data to determine the initial relocation data in the target data. The step of performing deduplication analysis on the initial relocation data to determine the actual relocation data corresponding to the initial relocation data includes: Identify a second thread that meets the same preset kernel conditions as the first thread; The second thread performs deduplication analysis on the initial relocation data to determine the actual relocation data corresponding to the initial relocation data.

8. A data transfer device based on deduplication, characterized in that, Applied to a terminal, the device includes: The first determining module is used to perform recovery analysis on the target data and determine the initial relocation data in the target data; The second determining module is used to perform deduplication analysis on the initial moving data, determine the actual moving data corresponding to the initial moving data, and perform moving processing on the actual moving data.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images