Data processing method and device, computer equipment, storage medium and program product

Abstract

The invention relates to a data processing method and device, computer equipment, a storage medium and a program product. The method comprises the following steps: according to fingerprint information of each piece of data to be deduplicated on a disk, determining a preset deduplication duration corresponding to each piece of data to be deduplicated; and performing deduplication operation on the to-be-deduplicated data on the disk every preset deduplication duration. According to the fingerprint information of each to-be-deduplicated data, the preset deduplication duration corresponding to each to-be-deduplicated data is flexibly determined, the matching degree of the preset deduplication duration and the to-be-deduplicated data is improved, further, the to-be-deduplicated data is deduplicated based on the preset deduplication duration, high-frequency deduplication operation on all the to-be-deduplicated data is avoided, and the data deduplication efficiency is improved. The efficiency of deduplication operation on the to-be-deduplication data is improved.

Description

Technical Field

[0001] The present application relates to the technical field of computer equipment, and in particular to a data processing method, apparatus, computer equipment, storage medium and program product. Background Art

[0002] In order to increase the capacity available to users, storage arrays use data deduplication technology. Based on this storage technology, only one copy of multiple identical data is saved on the hard disk, and duplicate data is mapped to the location of the data to achieve the purpose of increasing the capacity available to users.

[0003] When deleting duplicate data, traditional technologies usually need to traverse all data blocks in the storage space, read all the data blocks for comparison, and select the same data blocks for deduplication operations until a copy of the data is retained on the hard disk.

[0004] However, the above data deduplication method has the problem of low deduplication efficiency. Summary of the invention

[0005] Based on this, it is necessary to provide a data processing method, apparatus, computer equipment, storage medium and program product that can improve the efficiency of data deduplication in response to the above technical problems.

[0006] In a first aspect, the present application provides a data processing method, comprising:

[0007] Determine the preset deduplication duration corresponding to each piece of data to be deduplicated according to the fingerprint information of each piece of data to be deduplicated on the disk;

[0008] At each preset deduplication time interval, deduplication operations are performed on each piece of data to be deduplicated on the disk.

[0009] The above-mentioned data processing method, device, computer equipment, storage medium and program product determine the preset deduplication time corresponding to each data to be deduplicated according to the fingerprint information of each data to be deduplicated on the disk; and perform deduplication operation on each data to be deduplicated on the disk at each interval of the preset deduplication time. According to the fingerprint information of each data to be deduplicated, the preset deduplication time corresponding to each data to be deduplicated is flexibly determined, thereby improving the matching degree between the preset deduplication time and the data to be deduplicated. Furthermore, deduplication operation is performed on the data to be deduplicated based on the preset deduplication time, thereby avoiding high-frequency deduplication operation on all data to be deduplicated, thereby improving the efficiency of deduplication operation on the data to be deduplicated.

[0010] In one embodiment, determining a preset deduplication duration corresponding to each to-be-deduplicated data according to fingerprint information of each to-be-deduplicated data on the disk includes:

[0011] According to the fingerprint information of each to-be-deduplicated data on the disk, the popularity and number of duplicates of each to-be-deduplicated data are determined; the number of duplicates indicates the number of data that is consistent with the fingerprint information of each to-be-deduplicated data;

[0012] The preset deduplication duration corresponding to each piece of data to be deduplicated is determined according to the popularity and number of duplicates of each piece of data to be deduplicated.

[0013] In the above-mentioned application embodiment, the popularity and number of duplications of the data to be deduplicated are determined based on the fingerprint information of the data to be deduplicated, and only the fingerprint information is read and analyzed, without the need to read and analyze the data to be deduplicated, thereby reducing the data processing time, thereby improving the efficiency of determining the popularity and number of duplications of the data to be deduplicated, and further improving the efficiency of deduplication operations on the data to be deduplicated.

[0014] In one embodiment, determining a preset deduplication duration corresponding to each to-be-deduplicated data according to the popularity and the number of duplicates of each to-be-deduplicated data includes:

[0015] Determine the initial deduplication duration of each piece of data to be deduplicated according to its popularity;

[0016] According to the number of repetitions of each to-be-deduplicated data, the initial deduplication duration of each to-be-deduplicated data is modified to obtain a preset deduplication duration corresponding to each to-be-deduplicated data.

[0017] In the above application embodiment, the initial deduplication time of the data to be deduplicated is first determined based on the popularity of the data to be deduplicated, and then the initial deduplication time is corrected based on the duplicate data of the data to be deduplicated. Through the two determination processes, the preset deduplication time is obtained more accurately, thereby improving the reliability of the preset deduplication time.

[0018] In one embodiment, determining the heat of each to-be-deduplicated data according to the fingerprint information of each to-be-deduplicated data on the disk includes:

[0019] Determine the storage location of each piece of data to be deduplicated based on the fingerprint information of each piece of data to be deduplicated on the disk;

[0020] The heat of each piece of data to be deduplicated is determined according to the storage location of each piece of data to be deduplicated.

[0021] In the above application embodiment, the storage location of each data to be deduplicated is determined based on the fingerprint information of each data to be deduplicated, thereby determining the heat of each data to be deduplicated based on each storage location, and determining the heat of the data to be deduplicated based on the heat of the storage location, thereby improving the accuracy of the heat of each data to be deduplicated.

[0022] In one embodiment, at each preset deduplication time interval, deduplication operation is performed on each to-be-deduplicated data on the disk, including:

[0023] Obtain the attributes of each data to be deleted;

[0024] For each piece of data to be deduplicated, if the attribute of the data to be deduplicated is a retain attribute, the data to be deduplicated is retained, and other data that is the same as the data to be deduplicated is deleted;

[0025] If the data to be deduplicated has a non-retained attribute, the data to be deduplicated is deleted, and target other data is selected from other data with the same attributes as the data to be deduplicated for retention, and other data except the target other data is deleted; the attribute of the target other data is a retaining attribute.

[0026] In the above-mentioned application embodiment, deduplication operations are performed on the data to be deduplicated according to the attributes of the data to be deduplicated, thereby retaining the same data stored in the optimal location, or retaining the same data with the highest reliability, thereby improving the storage performance and read-write performance of the server, and at the same time improving the life and balance of the disk array corresponding to the storage space.

[0027] In one embodiment, the method further comprises:

[0028] The number of deduplication times of the data to be deduplicated is recorded, and when the number of deduplication times reaches a preset threshold, the preset deduplication time corresponding to the data to be deduplicated is optimized and updated.

[0029] In the above-mentioned application embodiment, based on the number of deduplications of the data to be deleted, the preset deduplication time corresponding to the data to be deleted is optimized and updated, so that the preset deduplication time and the data to be deleted have a higher matching degree, and the deduplication operation is performed on the data to be deleted based on the optimized and updated preset deduplication time, thereby further improving the effect of the deduplication operation.

[0030] In one embodiment, the method further comprises:

[0031] Traverse the fingerprint information of all data on the disk, and determine the candidate deduplication data based on the fingerprint information of each data;

[0032] According to the consistency of the fingerprint information of each candidate deduplication data, the data to be deduplicated is determined from the candidate deduplication data.

[0033] In the above application embodiment, the data to be deduplicated is determined from all the data according to the fingerprint information, avoiding deduplication of all the data, thereby improving the efficiency of deduplication operation on the data to be deduplicated and improving the effect of the deduplication operation.

[0034] In a second aspect, the present application further provides a data processing device, comprising:

[0035] A first determination module is used to determine a preset deduplication duration corresponding to each piece of data to be deduplicated according to fingerprint information of each piece of data to be deduplicated on the disk;

[0036] The deduplication module is used to perform deduplication operations on each piece of data to be deduplicated on the disk at each preset deduplication time interval.

[0037] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0038] Determine the preset deduplication duration corresponding to each piece of data to be deduplicated according to the fingerprint information of each piece of data to be deduplicated on the disk;

[0039] At each preset deduplication time interval, deduplication operations are performed on each piece of data to be deduplicated on the disk.

[0040] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the following steps are implemented:

[0041] Determine the preset deduplication duration corresponding to each piece of data to be deduplicated according to the fingerprint information of each piece of data to be deduplicated on the disk;

[0042] At each preset deduplication time interval, deduplication operations are performed on each piece of data to be deduplicated on the disk.

[0043] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0044] Determine the preset deduplication duration corresponding to each piece of data to be deduplicated according to the fingerprint information of each piece of data to be deduplicated on the disk;

[0045] At each preset deduplication time interval, deduplication operations are performed on each piece of data to be deduplicated on the disk.

[0046] The above-mentioned data processing method, device, computer equipment, storage medium and program product determine the preset deduplication time corresponding to each data to be deduplicated according to the fingerprint information of each data to be deduplicated on the disk; and perform deduplication operation on each data to be deduplicated on the disk at each interval of the preset deduplication time. According to the fingerprint information of each data to be deduplicated, the preset deduplication time corresponding to each data to be deduplicated is flexibly determined, thereby improving the matching degree between the preset deduplication time and the data to be deduplicated. Furthermore, deduplication operation is performed on the data to be deduplicated based on the preset deduplication time, thereby avoiding high-frequency deduplication operation on all data to be deduplicated, thereby improving the efficiency of deduplication operation on the data to be deduplicated. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0048] Figure 1 An application environment diagram of a data processing method in an embodiment;

[0049] Figure 2 is a flow chart of a data processing method in one embodiment;

[0050] Figure 3 is a flow chart of a data processing method in another embodiment;

[0051] Figure 4 is a flow chart of a data processing method in another embodiment;

[0052] Figure 5 is a flow chart of a data processing method in another embodiment;

[0053] Figure 6 is a flow chart of a data processing method in another embodiment;

[0054] Figure 7 is a flow chart of a data processing method in another embodiment;

[0055] Figure 8 is a flow chart of a data processing method in another embodiment;

[0056] Fig. 9 is a flow chart of a data processing method in another embodiment;

[0057] Fig.10 FIG. 4 is a structural block diagram of a data processing device in one embodiment. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0059] In traditional technology, the storage system performs hash calculation on the original data written by the host, and stores the hash value in the fingerprint table, which together with the storage location information of the original data forms a fingerprint record. The fingerprint table sorts the fingerprint information according to the results, and puts the fingerprint information with the same fingerprint value (i.e., hash value) together. Since the hash value itself may conflict, different data may calculate the same hash value. Therefore, to confirm whether it is duplicate data, it is necessary to read the original data for final confirmation. Only when the original data has the same content can the deduplication operation be performed. In the process of accumulating data in the fingerprint table, the data storage location recorded in the fingerprint information may be rewritten by the host, that is, the fingerprint information no longer matches the actual situation, that is, it is invalid. There are two ways to deal with the invalid fingerprint record information:

[0060] 1. Delete it from the fingerprint table. When performing subsequent deduplication operations based on the fingerprint record, the record will not be read. However, the fingerprint table is generally sorted based on the fingerprint value. It will be extremely time-consuming to find the record through the storage location of the record.

[0061] 2. When reading next, first check whether the fingerprint record is invalid. If it is found that its information is invalid, abandon the record and do not add it to the deduplication process. Although this method will be more complicated in the process, the overall time consumption is shorter than the previous method, and this method is generally used for processing.

[0062] In addition, for scenarios where data is written frequently, since the original data will change more frequently, a large number of invalid records will be generated in the fingerprint table. In this case, since the frequent checks on fingerprint records return invalid results, the deduplication rate will be reduced. At the same time, these invalid fingerprint address records also occupy a large amount of fingerprint table space, further leading to a decrease in deduplication efficiency. For scenarios where data is written very little, most fingerprint records are valid, and more fingerprint records can be accumulated to release the space occupied by the original data and store more user data.

[0063] It can be seen that the actual effect of deduplication through the fingerprint table is very dependent on the current host's IO module and the hot and cold conditions of the data. If it cannot be adjusted, the deduplication effect will not be very good. In addition, the characteristics of the fingerprint itself will also affect the deduplication effect. For the same original data, the same fingerprint value will definitely be generated. If there are multiple records of certain fingerprint values ​​in the fingerprint table, deduplication for this series of fingerprint records will achieve a good deduplication effect. If there are multiple identical data, in principle, only one record can be retained. By default, the first written data will be retained. However, in actual use, due to differences in the hard disks or hard disk combinations (such as RAID groups) storing data, retaining data on specific hard disks will have better effects, such as faster reading speeds. At this time, relevant configuration information is also needed for guidance.

[0064] In summary, the deduplication method that usually uses fingerprint values ​​and storage records to indicate data features can achieve data deduplication, but there is still room for improvement in deduplication effect and efficiency.

[0065] The data processing method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown in FIG. 1 , the computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 1 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data processing data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a data processing method is implemented.

[0066] Those skilled in the art will understand that Figure 1 The structure shown in the figure is only a block diagram of a part of the structure related to the present application scheme, and does not constitute a limitation on the computer device to which the present application scheme is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0067] In one embodiment, Figure 2As shown, a data processing method is provided, which is applied to Figure 1 The following is an example of a server in the example, including:

[0068] S201, determining a preset deduplication duration corresponding to each piece of data to be deduplicated according to fingerprint information of each piece of data to be deduplicated on the disk.

[0069] The data to be deduplicated may be any data stored on a disk, and the fingerprint data may be used to store the storage location and data identifier of each data. The preset deduplication durations corresponding to each data to be deduplicated may be the same or different.

[0070] In an embodiment of the present application, the server can read the fingerprint information of the data to be deduplicated when storing the data to be deduplicated, and determine the preset deduplication time of the data to be deduplicated based on the fingerprint information; or, the server can first select the data identifier of the data to be deduplicated from all data identifiers, and then read the fingerprint information corresponding to the data to be deduplicated based on the data identifier of the data to be deduplicated, thereby determining the preset deduplication time of the data to be deduplicated based on the fingerprint information, and recording the preset deduplication time of the data to be deduplicated in the corresponding fingerprint information, or, store the preset deduplication time of the data to be deduplicated and the corresponding fingerprint information in the same storage location.

[0071] Optionally, a correspondence between a storage location and a deduplication duration may be pre-constructed, so that the deduplication duration of each to-be-deduplicated data may be determined according to the storage location in the fingerprint information of each to-be-deduplicated data and the above correspondence.

[0072] S202, performing a deduplication operation on each to-be-deduplicated data on the disk at each preset deduplication time interval.

[0073] In the embodiment of the present application, for each data to be deduplicated, at each interval of the preset deduplication time corresponding to the data to be deduplicated, it is determined whether duplicate data identical to the data to be deduplicated is stored on the disk. If duplicate data identical to the data to be deduplicated is stored on the disk, the data to be deduplicated or the duplicate data is deleted so that only one copy of the same data is retained on the disk; if duplicate data identical to the duplicate data is not stored on the disk, the deduplication operation is terminated. Optionally, when duplicate data identical to the data to be deduplicated is stored on the disk, either the data to be deduplicated or the duplicate data can be deleted so that only one copy of the same data is retained on the disk.

[0074] In the above data processing method, the preset deduplication time corresponding to each data to be deduplicated is determined according to the fingerprint information of each data to be deduplicated on the disk; and the deduplication operation is performed on each data to be deduplicated on the disk at each interval of the preset deduplication time. According to the fingerprint information of each data to be deduplicated, the preset deduplication time corresponding to each data to be deduplicated is flexibly determined, which improves the matching degree between the preset deduplication time and the data to be deduplicated. Furthermore, the deduplication operation is performed on the data to be deduplicated based on the preset deduplication time, which avoids high-frequency deduplication operations on all the data to be deduplicated, and improves the efficiency of the deduplication operation on the data to be deduplicated.

[0075] In one embodiment, an implementation of the above S201 is provided, such as Figure 3 As shown, the above “determining the preset deduplication duration corresponding to each piece of data to be deduplicated according to the fingerprint information of each piece of data to be deduplicated on the disk” includes:

[0076] S301, determining the popularity and the number of duplicates of each to-be-deduplicated data on the disk according to the fingerprint information of each to-be-deduplicated data; the number of duplicates indicates the number of data consistent with the fingerprint information of each to-be-deduplicated data.

[0077] In the embodiment of the present application, the fingerprint information of each data to be deleted is traversed to determine the number of identical fingerprint information in all fingerprint information, that is, the number of data consistent with the fingerprint information of each data to be deleted. Optionally, the data identifier in each fingerprint information can be obtained to determine the number of identical data identifiers in the data identifiers of all fingerprint information. Further, the fingerprint information of each data to be deleted is parsed to obtain the heat of each data to be deleted.

[0078] As an optional implementation, the fingerprint information may include the number of times each data to be deduplicated is read, the heat of the data to be deduplicated may be expressed as a numerical value, and the number of reads may be used as the heat of the data to be deduplicated; or, when the number of reads is greater than a preset number threshold, the heat of the data to be deduplicated represents hot data, and when the number of reads is not greater than the preset number threshold, the heat of the data to be deduplicated represents cold data.

[0079] As another optional implementation, the fingerprint information may include the heat of the data to be deduplicated. By parsing the fingerprint information of each data to be deduplicated, the heat of each data to be deduplicated may be directly obtained.

[0080] S302: Determine a preset deduplication duration corresponding to each piece of data to be deduplicated according to the popularity and the number of duplicates of each piece of data to be deduplicated.

[0081] In the embodiment of the present application, the preset deduplication time length corresponding to each piece of data to be deduplicated is determined according to the heat of the data to be deduplicated, the duplicate data and the preset time length determination logic.

[0082] Optionally, the preset duration determination logic may be a preset relational expression, and the heat of the to-be-deduplicated data and the duplicate data are substituted into the preset relational expression to obtain a preset deduplication duration corresponding to the to-be-deduplicated data.

[0083] In the above-mentioned application embodiment, the popularity and number of duplications of the data to be deduplicated are determined based on the fingerprint information of the data to be deduplicated, and only the fingerprint information is read and analyzed, without the need to read and analyze the data to be deduplicated, thereby reducing the data processing time, thereby improving the efficiency of determining the popularity and number of duplications of the data to be deduplicated, and further improving the efficiency of deduplication operations on the data to be deduplicated.

[0084] In one embodiment, an implementation of the above S302 is provided, such as Figure 4 As shown, the above “determining the preset deduplication time corresponding to each piece of data to be deduplicated according to the popularity and the number of duplicates of each piece of data to be deduplicated” includes:

[0085] S401, determining an initial deduplication duration of each to-be-deduplicated data according to the heat of each to-be-deduplicated data.

[0086] The initial deduplication duration may be a candidate duration range, or the initial deduplication duration may include one or more candidate deduplication durations.

[0087] In the embodiment of the present application, a first correspondence between heat and initial deduplication duration is pre-established, so that the initial deduplication duration of each data to be deduplicated is determined according to the heat of each data to be deduplicated and the first correspondence. Exemplarily, when the heat of the data to be deduplicated represents hot data, the first initial duration is determined as the initial deduplication duration of the data to be deduplicated; when the heat of the data to be deduplicated represents cold data, the second initial duration is determined as the initial deduplication duration of the data to be deduplicated.

[0088] S402: According to the number of repetitions of each to-be-deduplicated data, the initial deduplication duration of each to-be-deduplicated data is modified to obtain a preset deduplication duration corresponding to each to-be-deduplicated data.

[0089] In an embodiment of the present application, if the number of duplicates of the data to be deduplicated is less than a preset threshold, more identical data can be accumulated to improve the deduplication effect and save storage space, and the initial deduplication duration can be extended to obtain a preset deduplication duration; if the number of duplicates of the data to be deduplicated is not less than the preset threshold, and the proportion of identical data to be deduplicated is high, the deduplication duration can be shortened to improve the deduplication effect and save storage space, and the initial deduplication duration can be shortened to obtain a preset deduplication duration. Optionally, a second correspondence between the number of duplicates and the modified duration can be established in advance, wherein the modified duration can include an increase duration and a decrease duration. If the initial deduplication duration is a candidate deduplication duration, the modified duration corresponding to the data to be deduplicated is determined based on the number of duplicates of the data to be deduplicated and the preset second correspondence, thereby determining the preset deduplication duration corresponding to each data to be deduplicated based on the initial deduplication duration and the modified duration.

[0090] As another optional implementation, a third correspondence between the number of repetitions and the preset deduplication time is pre-established. If the initial deduplication time is a candidate time range, the preset deduplication time corresponding to the data to be deduplicated can be determined from the candidate time range based on the number of repetitions of the data to be deduplicated and the third correspondence. Alternatively, if the initial deduplication time is a plurality of candidate deduplication time ranges, the preset deduplication time corresponding to the data to be deduplicated can be determined from a plurality of candidate deduplication time ranges based on the number of repetitions of the data to be deduplicated and the third correspondence.

[0091] In the above application embodiment, the initial deduplication time of the data to be deduplicated is first determined based on the popularity of the data to be deduplicated, and then the initial deduplication time is corrected based on the duplicate data of the data to be deduplicated. Through the two determination processes, the preset deduplication time is obtained more accurately, thereby improving the reliability of the preset deduplication time.

[0092] In one embodiment, an implementation of the above S301 is provided, such as Figure 5 As shown, the above “determining the heat of each to-be-deduplicated data according to the fingerprint information of each to-be-deduplicated data on the disk” includes:

[0093] S501, determining the storage location of each to-be-deduplicated data according to the fingerprint information of each to-be-deduplicated data on the disk.

[0094] In an embodiment of the present application, the fingerprint information may include the storage location of each data to be deleted, thereby extracting the storage location of each data to be deleted from the fingerprint data of the majority of data to be deleted; or, data processing may be performed on each data to be deleted to obtain the storage location of each data to be deleted; or, the storage location of each data to be deleted may be determined based on the correspondence between the preset fingerprint information and the storage location.

[0095] S502: Determine the heat of each piece of data to be deduplicated according to the storage location of each piece of data to be deduplicated.

[0096] In an embodiment of the present application, for each data to be deduplicated, a fourth correspondence between the storage location and the heat can be pre-established, so as to determine the heat of the data to be deduplicated based on the storage location of the data to be deduplicated and the fourth correspondence; or, the number of read and write times of the storage location of the data to be deduplicated can be determined, and the heat of the data to be deduplicated can be determined based on the number of read and write times of the storage location. Exemplarily, for example, the number of read and write times of the storage location is determined as the heat of the data to be deduplicated; or, the number of read and write times of the storage location is substituted into a preset relationship to determine the heat of the data to be deduplicated.

[0097] In the above application embodiment, the storage location of each data to be deduplicated is determined based on the fingerprint information of each data to be deduplicated, thereby determining the heat of each data to be deduplicated based on each storage location, and determining the heat of the data to be deduplicated based on the heat of the storage location, thereby improving the accuracy of the heat of each data to be deduplicated.

[0098] In one embodiment, an implementation of the above S202 is provided, such as Figure 6 As shown, the above “performing a deduplication operation on each to-be-deduplicated data on the disk at each preset deduplication time interval” includes:

[0099] S601, obtaining attributes of each data to be deduplicated.

[0100] The attributes of the data to be deduplicated include reserved attributes and non-reserved attributes.

[0101] In the embodiment of the present application, the attribute of each data to be deduplicated can be determined according to the storage location of each data to be deduplicated, and optionally, the disk performance where the data to be deduplicated is located is obtained, and the attribute of each data to be deduplicated is determined according to the performance of each disk, if the disk performance is excellent or good, the attribute of the data to be deduplicated is determined as a reserved attribute, if the disk performance is average or poor, the attribute of the data to be deduplicated is determined as a non-reserved attribute. Alternatively, the attribute of each data to be deduplicated can be obtained according to the indication information received by the server, wherein the indication information can be carried in the read and write request corresponding to the data to be deduplicated.

[0102] S602: For each piece of data to be deduplicated, if the attribute of the data to be deduplicated is a retain attribute, retain the data to be deduplicated and delete other data that is the same as the data to be deduplicated.

[0103] In an embodiment of the present application, when the deduplication time is preset at each interval, it is determined whether there is other data in the storage space that is the same as the data to be deduplicated. If there is other data corresponding to the data to be deduplicated in the storage space and the attribute of the data to be deduplicated is a retention attribute, the data to be deduplicated is retained and the other data is deleted.

[0104] S603, if the data to be deduplicated is of a non-retained attribute, the data to be deduplicated is deleted, and target other data is selected from other data that is the same as the data to be deduplicated for a retaining operation, and other data except the target other data is deleted; the attribute of the target other data is a retaining attribute.

[0105] In an embodiment of the present application, at each interval of a preset deduplication time, it is determined whether there is other data identical to the data to be deduplicated in the storage space. When there is other data corresponding to the data to be deduplicated in the storage space and the data to be deduplicated is non-retained data, the data to be deduplicated is deleted, and attributes of the other data are determined. Target data whose attributes are retainable attributes are determined from the other data. Furthermore, the target data is retained, and other data except the target data is deleted.

[0106] In the above-mentioned application embodiment, deduplication operations are performed on the data to be deduplicated according to the attributes of the data to be deduplicated, thereby retaining the same data stored in the optimal location, or retaining the same data with the highest reliability, thereby improving the storage performance and read-write performance of the server, and at the same time improving the life and balance of the disk array corresponding to the storage space.

[0107] In one embodiment, Figure 7 As shown, the above data processing method further includes:

[0108] S203, recording the number of deduplication times of the data to be deduplicated, and when the number of deduplication times reaches a preset threshold, optimizing and updating the preset deduplication time corresponding to the data to be deduplicated.

[0109] The number of deduplication times is the number of times the deduplication operation is performed on the data to be deduplicated.

[0110] In an embodiment of the present application, after each deduplication operation is performed on the data to be deduplicated, if the data to be deduplicated is retained, the number of deduplication times of the data to be deduplicated is updated, and the number of deduplication times reaches the preset number threshold. When the number of deduplication times reaches the preset number threshold, it indicates that the number of deduplication operations on the data to be deduplicated is large, or the fingerprint data corresponding to the data to be deduplicated is stored in the fingerprint table for a long time, and the probability of the data to be deduplicated being deleted is small. Not participating in the deduplication operation can reduce resource usage, and the preset deduplication time corresponding to the data to be deduplicated is extended; when the number of deduplication times does not reach the preset number threshold, it indicates that the number of deduplication operations on the data to be deduplicated is small, or the fingerprint data corresponding to the data to be deduplicated is stored in the fingerprint table for a short time, and the probability of the data to be deduplicated being deleted is high, and the preset deduplication time corresponding to the data to be deduplicated is shortened.

[0111] Optionally, when the number of deduplication times has not reached a preset threshold, the frequency of deduplication operations on the data to be deduplicated can be increased, for example, a deduplication operation is performed on the data to be deduplicated each time deduplication is started; when the number of deduplication times has reached a preset threshold, the frequency of deduplication operations on the data to be deduplicated can be reduced, for example, a deduplication operation is not performed on the data to be deduplicated each time deduplication is started.

[0112] In the above-mentioned application embodiment, based on the number of deduplications of the data to be deleted, the preset deduplication time corresponding to the data to be deleted is optimized and updated, so that the preset deduplication time and the data to be deleted have a higher matching degree, and the deduplication operation is performed on the data to be deleted based on the optimized and updated preset deduplication time, thereby further improving the effect of the deduplication operation.

[0113] In one embodiment, Figure 8 As shown, the above data processing method further includes:

[0114] S204, traverse the fingerprint information of all data on the disk, and determine candidate deduplication data according to the fingerprint information of each data.

[0115] In an embodiment of the present application, fingerprint information can be stored in a preset fingerprint table, and optionally, the storage location of the fingerprint table can be a cache space. Traverse all fingerprint information in the fingerprint table, and determine the data corresponding to all fingerprint information as candidate deduplication data; or, determine the fingerprint information that meets the preset conditions in the fingerprint information as candidate fingerprint information, and determine the data corresponding to the candidate fingerprint information as candidate deduplication data. Optionally, the preset conditions can be the time of writing the fingerprint information, or the size of the data corresponding to the fingerprint information, etc.

[0116] S205 , determining the data to be deduplicated from the candidate deduplication data according to the consistency of the fingerprint information of each candidate deduplication data.

[0117] In an embodiment of the present application, for each fingerprint information, it is determined whether there is duplicate fingerprint information identical to the fingerprint information. If there is duplicate fingerprint information identical to the fingerprint information, the candidate deduplication data corresponding to the fingerprint information is determined as the data to be deduplicated; if there is no duplicate fingerprint information identical to the fingerprint information, it is determined whether there is duplicate fingerprint information identical to the next fingerprint information, until the fingerprint information of all candidate deduplication data is traversed and all the data to be deduplicated is determined.

[0118] In the above application embodiment, the data to be deduplicated is determined from all the data according to the fingerprint information, avoiding deduplication of all the data, thereby improving the efficiency of deduplication operation on the data to be deduplicated and improving the effect of the deduplication operation.

[0119] In one embodiment, a complete data processing method is provided, such as Fig. 9As shown, the method includes:

[0120] S1, traverse the fingerprint information of all data on the disk, and determine the candidate deduplication data according to the fingerprint information of each data.

[0121] S2: Determine the data to be deduplicated from the candidate deduplication data according to the consistency of the fingerprint information of each candidate deduplication data.

[0122] S3, determining the storage location of each to-be-deduplicated data according to the fingerprint information of each to-be-deduplicated data on the disk.

[0123] S4, determining the heat of each to-be-deduplicated data according to the storage location of each to-be-deduplicated data.

[0124] S5, determining the number of duplicates of each to-be-deduplicated data on the disk according to the fingerprint information of each to-be-deduplicated data; the number of duplicates represents the number of data consistent with the fingerprint information of each to-be-deduplicated data.

[0125] S6: Determine the initial deduplication duration of each piece of data to be deduplicated according to the popularity of each piece of data to be deduplicated.

[0126] S7, according to the number of repetitions of each to-be-deduplicated data, modify the initial deduplication duration of each to-be-deduplicated data to obtain a preset deduplication duration corresponding to each to-be-deduplicated data.

[0127] S8, obtaining the attributes of each data to be deduplicated.

[0128] S9: For each piece of data to be deduplicated, if the attribute of the data to be deduplicated is a retain attribute, retain the data to be deduplicated and delete other data that is the same as the data to be deduplicated.

[0129] S10, if the data to be deduplicated is of a non-retained attribute, the data to be deduplicated is deleted, and target other data is selected from other data that is the same as the data to be deduplicated for a retaining operation, and other data except the target other data is deleted; the attribute of the target other data is a retaining attribute.

[0130] S11, recording the number of deduplication times of the data to be deduplicated, and when the number of deduplication times reaches a preset threshold, optimizing and updating the preset deduplication time corresponding to the data to be deduplicated.

[0131] In the above data processing method, the preset deduplication time corresponding to each data to be deduplicated is determined according to the fingerprint information of each data to be deduplicated on the disk; and the deduplication operation is performed on each data to be deduplicated on the disk at each interval of the preset deduplication time. According to the fingerprint information of each data to be deduplicated, the preset deduplication time corresponding to each data to be deduplicated is flexibly determined, which improves the matching degree between the preset deduplication time and the data to be deduplicated. Furthermore, the deduplication operation is performed on the data to be deduplicated based on the preset deduplication time, which avoids high-frequency deduplication operations on all the data to be deduplicated, and improves the efficiency of the deduplication operation on the data to be deduplicated.

[0132] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0133] Based on the same inventive concept, the embodiment of the present application also provides a data processing device for implementing the data processing method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the one or more data processing device embodiments provided below can refer to the limitations on the data processing method above, and will not be repeated here.

[0134] In one embodiment, Fig.10 As shown, a data processing device is provided, including: a first determination module 10 and a deduplication module 11, wherein:

[0135] A first determination module 10 is used to determine a preset deduplication duration corresponding to each to-be-deduplicated data according to fingerprint information of each to-be-deduplicated data on the disk;

[0136] The deduplication module 11 is used to perform a deduplication operation on each to-be-deduplicated data on the disk at each preset deduplication time interval.

[0137] In one embodiment, the first determination module 10 includes: a first determination unit and a second determination unit, wherein:

[0138] The first determination unit is used to determine the heat and the number of duplicates of each to-be-deduplicated data according to the fingerprint information of each to-be-deduplicated data on the disk; the number of duplicates represents the number of data consistent with the fingerprint information of each to-be-deduplicated data.

[0139] The second determining unit is used to determine the preset deduplication time corresponding to each piece of data to be deduplicated according to the popularity and the number of repetitions of each piece of data to be deduplicated.

[0140] In one embodiment, the above-mentioned second determination unit is specifically used to determine the initial deduplication time of each data to be deduplicated according to the heat of each data to be deduplicated; according to the number of repetitions of each data to be deduplicated, the initial deduplication time of each data to be deduplicated is corrected to obtain the preset deduplication time corresponding to each data to be deduplicated.

[0141] In one embodiment, the first determining unit is specifically configured to determine the storage location of each to-be-deduplicated data according to the fingerprint information of each to-be-deduplicated data on the disk; and determine the heat of each to-be-deduplicated data according to the storage location of each to-be-deduplicated data.

[0142] In one embodiment, the deduplication module 11 includes: an acquisition unit, a first deleting unit, and a second deleting unit, wherein:

[0143] The acquisition unit is used to acquire the attributes of each data to be deduplicated.

[0144] The first deleting unit is used for retaining each to-be-deduplicated data if the attribute of the to-be-deduplicated data is a retain attribute, and deleting other data identical to the to-be-deduplicated data.

[0145] The second deleting unit is used to delete the data to be deleting if it is a non-retained attribute, select target other data from other data that are the same as the data to be deleting for retention, and delete other data except the target other data; the attribute of the target other data is a retaining attribute.

[0146] In one embodiment, the above-mentioned data processing device further includes: an updating module, which is used to record the number of deduplication times of the data to be deduplicated, and optimize and update the preset deduplication time corresponding to the data to be deduplicated when the number of deduplication times reaches a preset threshold.

[0147] In one embodiment, the data processing device further includes: a traversal module and a second determination module, wherein:

[0148] The traversal module is used to traverse the fingerprint information of all data on the disk and determine the candidate deduplication data according to the fingerprint information of each data.

[0149] The second determination module is used to determine the to-be-deduplicated data from the candidate deduplication data according to the consistency of the fingerprint information of each candidate deduplication data.

[0150] Each module in the above data processing device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0151] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0152] Determine the preset deduplication duration corresponding to each piece of data to be deduplicated according to the fingerprint information of each piece of data to be deduplicated on the disk;

[0153] At each preset deduplication time interval, deduplication operations are performed on each piece of data to be deduplicated on the disk.

[0154] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0155] According to the fingerprint information of each to-be-deduplicated data on the disk, the popularity and number of duplicates of each to-be-deduplicated data are determined; the number of duplicates indicates the number of data that is consistent with the fingerprint information of each to-be-deduplicated data;

[0156] The preset deduplication duration corresponding to each piece of data to be deduplicated is determined according to the popularity and number of duplicates of each piece of data to be deduplicated.

[0157] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0158] Determine the initial deduplication duration of each piece of data to be deduplicated according to its popularity;

[0159] According to the number of repetitions of each to-be-deduplicated data, the initial deduplication duration of each to-be-deduplicated data is modified to obtain a preset deduplication duration corresponding to each to-be-deduplicated data.

[0160] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0161] Determine the storage location of each piece of data to be deduplicated based on the fingerprint information of each piece of data to be deduplicated on the disk;

[0162] The heat of each piece of data to be deduplicated is determined according to the storage location of each piece of data to be deduplicated.

[0163] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0164] Obtain the attributes of each data to be deleted;

[0165] For each piece of data to be deduplicated, if the attribute of the data to be deduplicated is a retain attribute, the data to be deduplicated is retained, and other data that is the same as the data to be deduplicated is deleted;

[0166] If the data to be deduplicated has a non-retained attribute, the data to be deduplicated is deleted, and target other data is selected from other data with the same attributes as the data to be deduplicated for retention, and other data except the target other data is deleted; the attribute of the target other data is a retaining attribute.

[0167] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0168] The number of deduplication times of the data to be deduplicated is recorded, and when the number of deduplication times reaches a preset threshold, the preset deduplication time corresponding to the data to be deduplicated is optimized and updated.

[0169] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0170] Traverse the fingerprint information of all data on the disk, and determine the candidate deduplication data based on the fingerprint information of each data;

[0171] According to the consistency of the fingerprint information of each candidate deduplication data, the data to be deduplicated is determined from the candidate deduplication data.

[0172] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0173] Determine the preset deduplication duration corresponding to each piece of data to be deduplicated according to the fingerprint information of each piece of data to be deduplicated on the disk;

[0174] At each preset deduplication time interval, deduplication operations are performed on each piece of data to be deduplicated on the disk.

[0175] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0176] According to the fingerprint information of each to-be-deduplicated data on the disk, the popularity and number of duplicates of each to-be-deduplicated data are determined; the number of duplicates indicates the number of data that is consistent with the fingerprint information of each to-be-deduplicated data;

[0177] The preset deduplication duration corresponding to each piece of data to be deduplicated is determined according to the popularity and number of duplicates of each piece of data to be deduplicated.

[0178] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0179] Determine the initial deduplication duration of each piece of data to be deduplicated according to its popularity;

[0180] According to the number of repetitions of each to-be-deduplicated data, the initial deduplication duration of each to-be-deduplicated data is modified to obtain a preset deduplication duration corresponding to each to-be-deduplicated data.

[0181] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0182] Determine the storage location of each piece of data to be deduplicated based on the fingerprint information of each piece of data to be deduplicated on the disk;

[0183] The heat of each piece of data to be deduplicated is determined according to the storage location of each piece of data to be deduplicated.

[0184] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0185] Obtain the attributes of each data to be deleted;

[0186] For each piece of data to be deduplicated, if the attribute of the data to be deduplicated is a retain attribute, the data to be deduplicated is retained, and other data that is the same as the data to be deduplicated is deleted;

[0187] If the data to be deduplicated has a non-retained attribute, the data to be deduplicated is deleted, and target other data is selected from other data with the same attributes as the data to be deduplicated for retention, and other data except the target other data is deleted; the attribute of the target other data is a retaining attribute.

[0188] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0189] The number of deduplication times of the data to be deduplicated is recorded, and when the number of deduplication times reaches a preset threshold, the preset deduplication time corresponding to the data to be deduplicated is optimized and updated.

[0190] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0191] Traverse the fingerprint information of all data on the disk, and determine the candidate deduplication data based on the fingerprint information of each data;

[0192] According to the consistency of the fingerprint information of each candidate deduplication data, the data to be deduplicated is determined from the candidate deduplication data.

[0193] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps:

[0194] Determine the preset deduplication duration corresponding to each piece of data to be deduplicated according to the fingerprint information of each piece of data to be deduplicated on the disk;

[0195] At each preset deduplication time interval, deduplication operations are performed on each piece of data to be deduplicated on the disk.

[0196] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0197] According to the fingerprint information of each to-be-deduplicated data on the disk, the popularity and number of duplicates of each to-be-deduplicated data are determined; the number of duplicates indicates the number of data that is consistent with the fingerprint information of each to-be-deduplicated data;

[0198] The preset deduplication duration corresponding to each piece of data to be deduplicated is determined according to the popularity and number of duplicates of each piece of data to be deduplicated.

[0199] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0200] Determine the initial deduplication duration of each piece of data to be deduplicated according to its popularity;

[0201] According to the number of repetitions of each to-be-deduplicated data, the initial deduplication duration of each to-be-deduplicated data is modified to obtain a preset deduplication duration corresponding to each to-be-deduplicated data.

[0202] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0203] Determine the storage location of each piece of data to be deduplicated based on the fingerprint information of each piece of data to be deduplicated on the disk;

[0204] The heat of each piece of data to be deduplicated is determined according to the storage location of each piece of data to be deduplicated.

[0205] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0206] Obtain the attributes of each data to be deleted;

[0207] For each piece of data to be deduplicated, if the attribute of the data to be deduplicated is a retain attribute, the data to be deduplicated is retained, and other data that is the same as the data to be deduplicated is deleted;

[0208] If the data to be deduplicated has a non-retained attribute, the data to be deduplicated is deleted, and target other data is selected from other data with the same attributes as the data to be deduplicated for retention, and other data except the target other data is deleted; the attribute of the target other data is a retaining attribute.

[0209] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0210] The number of deduplication times of the data to be deduplicated is recorded, and when the number of deduplication times reaches a preset threshold, the preset deduplication time corresponding to the data to be deduplicated is optimized and updated.

[0211] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0212] Traverse the fingerprint information of all data on the disk, and determine the candidate deduplication data based on the fingerprint information of each data;

[0213] According to the consistency of the fingerprint information of each candidate deduplication data, the data to be deduplicated is determined from the candidate deduplication data.

[0214] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0215] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0216] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A data processing method, characterized in that: The method comprises: Determine the preset deduplication duration corresponding to each of the to-be-deduplicated data according to the fingerprint information of each to-be-deduplicated data on the disk; At each interval of the preset deduplication time, a deduplication operation is performed on each of the to-be-deduplicated data on the disk.

2. The method according to claim 1, characterized in that The step of determining, according to the fingerprint information of each to-be-deduplicated data on the disk, a preset deduplication duration corresponding to each to-be-deduplicated data, comprises: Determine the heat and the number of duplicates of each of the to-be-deduplicated data on the disk according to the fingerprint information of each of the to-be-deduplicated data; the number of duplicates indicates the number of data consistent with the fingerprint information of each of the to-be-deduplicated data; According to the popularity and the number of repetitions of each of the to-be-deduplicated data, a preset deduplication time duration corresponding to each of the to-be-deduplicated data is determined.

3. The method according to claim 2, characterized in that Determining the preset deduplication duration corresponding to each of the to-be-deduplicated data according to the heat and the number of repetitions of each of the to-be-deduplicated data includes: Determining an initial deduplication duration of each of the to-be-deduplicated data according to the heat of each of the to-be-deduplicated data; According to the number of repetitions of each of the to-be-deduplicated data, the initial deduplication duration of each of the to-be-deduplicated data is modified to obtain a preset deduplication duration corresponding to each of the to-be-deduplicated data.

4. The method according to claim 2, characterized in that: The step of determining the heat of each of the to-be-deduplicated data according to the fingerprint information of each of the to-be-deduplicated data on the disk includes: Determine the storage location of each of the to-be-deduplicated data according to the fingerprint information of each of the to-be-deduplicated data on the disk; The heat of each of the to-be-deduplicated data is determined according to the storage location of each of the to-be-deduplicated data.

5. The method according to any one of claims 1 to 4, characterized in that: The deduplication operation is performed on each of the to-be-deduplicated data on the disk at intervals of the preset deduplication time, including: Obtaining attributes of each of the to-be-deduplicated data; For each of the to-be-deduplicated data, if the attribute of the to-be-deduplicated data is a reserved attribute, the to-be-deduplicated data is reserved, and other data identical to the to-be-deduplicated data is deleted; If the data to be deduplicated is of a non-retained attribute, the data to be deduplicated is deleted, and target other data is selected from other data that is the same as the data to be deduplicated for a retaining operation, and other data except the target other data is deleted; the attribute of the target other data is a retaining attribute.

6. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: The number of deduplication times of the to-be-deduplicated data is recorded, and when the number of deduplication times reaches a preset number threshold, a preset deduplication time length corresponding to the to-be-deduplicated data is optimized and updated.

7. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: Traversing fingerprint information of all data on the disk, and determining candidate deduplication data according to the fingerprint information of each data; The data to be deduplicated is determined from the candidate deduplication data according to the consistency of the fingerprint information of each candidate deduplication data.

8. A data processing device, characterized in that: The device comprises: A first determination module, configured to determine a preset deduplication duration corresponding to each of the to-be-deduplicated data according to fingerprint information of each to-be-deduplicated data on the disk; The deduplication module is used to perform a deduplication operation on each of the to-be-deduplicated data on the disk at intervals of the preset deduplication time.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images