Data processing method and device, electronic equipment, storage medium and program product

Abstract

The invention provides a data processing method and device, electronic equipment, a storage medium and a program product. The method comprises the steps that in a one-time deduplication task, the storage device can determine the current load of the storage device and the number of feature values to be deduplication, and a target deletion algorithm and a target compression algorithm are determined according to the current load and the number of the feature values to be deduplication. The method comprises the steps that data to be deduplicated are acquired, then the data to be deduplicated are subjected to deduplication processing according to a target deletion algorithm to acquire data after deduplication processing, the data to be deduplicated comprise feature values to be deduplicated and original data corresponding to the feature values, and the data after deduplication processing are compressed according to a target compression algorithm to acquire processed data. According to the current load of the storage device and the number of the feature values to be deduplicated, the appropriate deletion algorithm and compression algorithm are adaptively selected, the stability of the storage device can be ensured under the condition that the load of the storage device is high, and the deduplication rate can be effectively ensured under the condition that the load of the storage device is low.

Description

Technical Field

[0001] This application relates to the field of storage technology, and in particular, to a data processing method, apparatus, electronic device, storage medium, and program product. Background Art

[0002] With the advent of the information age, the amount of data has grown exponentially, leading to huge challenges in data storage, such as rising storage costs and reduced storage efficiency. Since there are a large number of duplicate data in the stored data, deleting duplicate data can effectively reduce the amount of data, relieve the storage pressure, and save storage costs.

[0003] The deduplication ratio is an important indicator to measure the effect of duplicate data deletion technology. In the related art, a data disk can be divided into multiple deduplication domains, and multiple deduplication domains have different priorities and different deduplication strategies. By applying different deduplication strategies in different deduplication domains, the deduplication ratio can be significantly improved.

[0004] However, the above method consumes a large amount of resources of the storage device. For example, when the storage device is under high load, this method may cause the storage device to freeze. Summary of the Invention

[0005] This application provides a data processing method, apparatus, electronic device, storage medium, and program product to solve the technical problem that the deduplication method in the related art consumes a large amount of resources of the storage device.

[0006] In a first aspect, this application provides a data processing method applied to a storage device. The storage device includes a data domain and a deduplication domain. Multiple storage areas in the data domain are respectively used to store multiple groups of original data, and multiple storage areas in the deduplication domain are respectively used to store the characteristic values corresponding to the multiple groups of original data. The method includes:

[0007] Determine the current load of the storage device and the number of characteristic values to be deduplicated;

[0008] Determine a target deletion algorithm and a target compression algorithm according to the current load and the number;

[0009] Perform deletion processing on the data to be deduplicated according to the target deletion algorithm to obtain the data after deletion processing. The data to be deduplicated includes the characteristic values to be deduplicated and the original data corresponding to the characteristic values;

[0010] Perform compression processing on the data after deletion processing according to the target compression algorithm to obtain the processed data.

[0011] In this embodiment, in a deduplication task, the storage device can determine the current load of the storage device and the number of eigenvalues to be deduplicated, and determine a target deletion algorithm and a target compression algorithm according to the current load and the number of eigenvalues to be deduplicated. Then, the data to be deduplicated is processed by the target deletion algorithm to obtain the data after deduplication processing. The data to be deduplicated includes the eigenvalues to be deduplicated and the original data corresponding to the eigenvalues, and the data after deduplication processing is compressed by the target compression algorithm to obtain the processed data. According to the current load of the storage device and the number of eigenvalues to be deduplicated, this application adaptively selects appropriate deletion and compression algorithms, so as to ensure the stability of the storage device under high load conditions and effectively ensure the deduplication rate under low load conditions of the storage device.

[0012] Optionally, for the method described above, determining the current load of the storage device and the number of eigenvalues to be deduplicated includes:

[0013] In the deduplication domain, obtain the eigenvalues of multiple subtypes in the target data type;

[0014] Among the eigenvalues of the multiple subtypes, determine multiple target eigenvalues;

[0015] Determine the number of the multiple target eigenvalues as the number of eigenvalues to be deduplicated;

[0016] Obtain the operating parameters of the storage device, where the operating parameters include the CPU usage rate of the storage device and / or the input / output usage rate of the storage device;

[0017] Determine the current load according to the operating parameters.

[0018] In this embodiment, the eigenvalues to be deduplicated can be determined based on the eigenvalues of multiple subclasses. Determining the current load based on the CPU usage rate and the I / O usage rate can enable subsequent selection of appropriate deletion and compression algorithms according to the current load and the number of eigenvalues to be deduplicated.

[0019] Optionally, for the method described above, determining the target deletion algorithm and the target compression algorithm according to the current load and the number includes:

[0020] Determine the load level of the current load and the number level of the number. The load level is the first load level or the second load level, and the number level is the first number level or the second number level. Among them, the first load level is higher than the second load level, and the first number level is higher than the second number level;

[0021] Determine the target deletion algorithm and the target compression algorithm in the algorithm library according to the load level and the quantity level.

[0022] In this embodiment, determine the target deletion algorithm and the target compression algorithm in the algorithm library according to the load level of the current load and the quantity level of the number of eigenvalues to be deduplicated, so that the algorithms used can meet the requirements of the current actual situation.

[0023] Optionally, for the method as described above, determining the target deletion algorithm and the target compression algorithm in the algorithm library according to the load level and the quantity level includes:

[0024] If the load level is the first load level and the quantity level is the first quantity level, determine that the target deletion algorithm is the first deletion algorithm and the target compression algorithm is the first compression algorithm. The first deletion algorithm belongs to the category of deletion algorithms with the lowest resource consumption among the multiple types of deletion algorithms, and the first compression algorithm belongs to the category of compression algorithms with the lowest compression ratio among the multiple types of compression algorithms;

[0025] If the load level is the first load level and the quantity level is the second quantity level, or if the load level is the second load level and the quantity level is the first quantity level, determine that the target deletion algorithm is the second deletion algorithm and the target compression algorithm is the second compression algorithm. The storage device resources consumed by the second deletion algorithm are higher than those consumed by the first deletion algorithm, and the compression ratio of the second compression algorithm is greater than that of the first compression algorithm;

[0026] If the load level is the second load level and the quantity level is the second quantity level, determine that the target deletion algorithm is the third deletion algorithm and the target compression algorithm is the third compression algorithm. The storage device resources consumed by the third deletion algorithm are higher than those consumed by the second deletion algorithm, and the compression ratio of the third compression algorithm is greater than that of the second compression algorithm.

[0027] In this embodiment, determine the target deletion algorithm and the target compression algorithm among multiple types of deletion algorithms and multiple types of compression algorithms according to the load level of the current load and the quantity level of the number of eigenvalues to be deduplicated, so that the algorithms used can meet the requirements of the current actual situation.

[0028] Optionally, in each type of deletion algorithm, there are multiple deletion algorithms, and the multiple deletion algorithms respectively correspond to different data types; in each type of compression algorithm, there are multiple compression algorithms, and the multiple compression algorithms respectively correspond to different data types;

[0029] The data type corresponding to the target deletion algorithm is the data type of the data to be deduplicated; the data type corresponding to the target compression algorithm is the data type of the data to be deduplicated.

[0030] In this embodiment, by setting deletion algorithms applicable to multiple data types for each type of deletion algorithm, and compression algorithms applicable to multiple data types, the data type corresponding to the selected target deletion algorithm is the data type of the data to be deduplicated, and the data type corresponding to the selected target compression algorithm is the data type of the data to be deduplicated, which can improve the effect of data processing.

[0031] Optionally, in the method as described above, the eigenvalues to be deduplicated include N eigenvalues, and the original data corresponding to the eigenvalues to be deduplicated includes N groups of original data; where N is a positive integer.

[0032] Performing deletion processing on the data to be deduplicated according to the target deletion algorithm to obtain the data after deletion processing, including:

[0033] Based on the target deletion algorithm, deleting N - 1 groups of the N groups of original data, and storing a first index in the storage area in the data domain where the N - 1 groups of original data are located, the first index pointing to the one group of original data that is not deleted among the N groups of original data;

[0034] Based on the target deletion algorithm, deleting the eigenvalues corresponding to the N - 1 groups of original data among the N eigenvalues, and storing a second index in the storage area in the deduplication domain where the eigenvalues corresponding to the N - 1 groups of original data are located, the second index pointing to the one eigenvalue that is not deleted among the N eigenvalues.

[0035] In this embodiment, storing corresponding indexes at the positions where the deleted data is located can point to the corresponding copies, so that the corresponding data can be obtained according to the index when obtaining data subsequently, and the data here can include original data and eigenvalues.

[0036] In a second aspect, the present application provides a data processing device, which is applied to a storage device. The storage device includes a data domain and a deduplication domain. Multiple storage areas in the data domain are respectively used to store multiple groups of original data, and multiple storage areas in the deduplication domain are respectively used to store the eigenvalues corresponding to the multiple groups of original data. The device includes:

[0037] A first determination module, configured to determine the current load of the storage device and the number of eigenvalues to be deduplicated;

[0038] A second determination module, configured to determine a target deletion algorithm and a target compression algorithm according to the current load and the number;

[0039] The first processing module is configured to perform deletion processing on the data to be deduplicated according to the target deletion algorithm to obtain the data after the deletion processing, where the data to be deduplicated includes the eigenvalue to be deduplicated and the original data corresponding to the eigenvalue;

[0040] The second processing module is configured to perform compression processing on the data after the deletion processing according to the target compression algorithm to obtain the processed data.

[0041] Optionally, for the device as described above, the first determination module is specifically configured to:

[0042] In the deduplication domain, obtain the eigenvalues of multiple subtypes in the target data type;

[0043] Among the eigenvalues of the multiple subtypes, determine multiple target eigenvalues;

[0044] Determine the number of the multiple target eigenvalues as the quantity of the eigenvalues to be deduplicated;

[0045] Obtain the operating parameters of the storage device, where the operating parameters include the CPU usage rate of the storage device and / or the input / output usage rate of the storage device;

[0046] Determine the current load according to the operating parameters.

[0047] Optionally, for the device as described above, the second determination module is specifically configured to:

[0048] Determine the load level of the current load and the quantity level of the quantity, where the load level is the first load level or the second load level, and the quantity level is the first quantity level or the second quantity level. Among them, the first load level is higher than the second load level, and the first quantity level is higher than the second quantity level;

[0049] Determine the target deletion algorithm and the target compression algorithm in the algorithm library according to the load level and the quantity level.

[0050] Optionally, for the device as described above, the second determination module is specifically configured to:

[0051] If the load level is the first load level and the quantity level is the first quantity level, then determine the target deletion algorithm as the first deletion algorithm and the target compression algorithm as the first compression algorithm. The first deletion algorithm belongs to the category of deletion algorithms with the lowest resource consumption among the multiple types of deletion algorithms, and the first compression algorithm belongs to the category of compression algorithms with the lowest compression ratio among the multiple types of compression algorithms;

[0052] If the load level is the first load level and the quantity level is the second quantity level, or the load level is the second load level and the quantity level is the first quantity level, then determine that the target deletion algorithm is the second deletion algorithm and the target compression algorithm is the second compression algorithm. The storage device resources consumed by the second deletion algorithm are higher than those consumed by the first deletion algorithm, and the compression ratio of the second compression algorithm is greater than that of the first compression algorithm.

[0053] If the load level is the second load level and the quantity level is the second quantity level, then determine that the target deletion algorithm is the third deletion algorithm and the target compression algorithm is the third compression algorithm. The storage device resources consumed by the third deletion algorithm are higher than those consumed by the second deletion algorithm, and the compression ratio of the third compression algorithm is greater than that of the second compression algorithm.

[0054] Optionally, in the device as described above, each type of deletion algorithm includes multiple deletion algorithms, and the multiple deletion algorithms respectively correspond to different data types; each type of compression algorithm includes multiple compression algorithms, and the multiple compression algorithms respectively correspond to different data types.

[0055] The data type corresponding to the target deletion algorithm is the data type of the data to be deduplicated; the data type corresponding to the target compression algorithm is the data type of the data to be deduplicated.

[0056] Optionally, in the device as described above, the feature values to be deduplicated include N feature values, and the original data corresponding to the feature values to be deduplicated includes N groups of original data; where N is a positive integer.

[0057] The first processing module is specifically configured to:

[0058] Based on the target deletion algorithm, delete N - 1 groups of the N groups of original data, and store a first index in the storage area in the data domain where the N - 1 groups of original data are located. The first index points to the group of original data that is not deleted among the N groups of original data.

[0059] Based on the target deletion algorithm, delete the feature values corresponding to the N - 1 groups of original data among the N feature values, and store a second index in the storage area in the deduplication domain where the feature values corresponding to the N - 1 groups of original data are located. The second index points to the feature value that is not deleted among the N feature values.

[0060] The data processing device provided by the embodiments of the present application has beneficial effects similar to those in the first aspect, and will not be elaborated here.

[0061] In a third aspect, the present application provides an electronic device, including: a processor, and a memory communicatively connected to the processor;

[0062] The memory stores computer-executable instructions;

[0063] The processor executes the computer-executable instructions stored in the memory to implement the method described in the first aspect.

[0064] In a fourth aspect, the present application provides a computer-readable storage medium storing computer-executable instructions, which are used to implement the method described in the first aspect when executed by a computer.

[0065] The computer-readable storage medium provided by the embodiments of the present application can execute the technical solutions in the above method embodiments, and the beneficial effects are similar, so details are not described herein again.

[0066] In a fifth aspect, the present application provides a computer program product including a computer program, which is used to implement the method described in the first aspect when executed by a computer.

[0067] The computer program product provided by the embodiments of the present application can execute the technical solutions in the above method embodiments, and the beneficial effects are similar, so details are not described herein again.

[0068] For the data processing method, device, electronic device, storage medium and program product provided by the present application, in a single deduplication task, the storage device can determine the current load of the storage device and the number of feature values to be deduplicated, and determine a target deletion algorithm and a target compression algorithm according to the current load and the number of feature values to be deduplicated. Then, the data to be deduplicated is processed by the target deletion algorithm to obtain the deduplicated data. The data to be deduplicated includes the feature values to be deduplicated and the original data corresponding to the feature values, and the deduplicated data is compressed by the target compression algorithm to obtain the processed data. The present application adaptively selects a suitable deletion algorithm and compression algorithm according to the current load of the storage device and the number of feature values to be deduplicated, so as to ensure the stability of the storage device under high load conditions of the storage device, and effectively ensure the deduplication rate under low load conditions of the storage device. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] The drawings herein are incorporated into the specification and form a part of the specification, showing embodiments consistent with the present application, and are used together with the specification to explain the principles of the present application.

[0070] Figure 1 It is a schematic diagram of the structure of the storage device according to an example of the present application and the process of a single deduplication task based on this structure;

[0071] Figure 2 It is a schematic flowchart of a data processing method provided by an embodiment of the present application;

[0072] Figure 3 It is a schematic flowchart of another data processing method provided by an embodiment of the present application;

[0073] Figure 4 It is a schematic flowchart of yet another data processing method provided by an embodiment of the present application;

[0074] Figure 5 It is a schematic diagram for determining a target deletion algorithm and a target compression algorithm according to a load level and a quantity level in an example of the present application;

[0075] Figure 6 It is a schematic flowchart of yet another data processing method provided by an embodiment of the present application;

[0076] Figure 7 It is a schematic diagram for storing a first index after data deletion in an example of the present application;

[0077] Figure 8 It is a schematic flowchart of yet another data processing method provided by an embodiment of the present application;

[0078] Figure 9 It is a schematic structural diagram of a data processing device provided by an embodiment of the present application;

[0079] Figure 10 It is a schematic structural diagram of an electronic device provided by an embodiment of the present application.

[0080] Through the above-mentioned drawings, specific embodiments of the present application have been shown, and there will be more detailed descriptions hereinafter. These drawings and textual descriptions are not intended to limit the scope of the concept of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. Specific Embodiments

[0081] Here, exemplary embodiments will be described in detail, and the examples are shown in the drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0082] Since there is a large amount of duplicate data in the stored data, deleting duplicate data can effectively reduce the amount of data. Then, different deduplication methods have different impacts on the overall stability of the storage device and the final data reduction effect. In the following text, the deletion of duplicate data is referred to as deduplication.

[0083] In related technologies, deduplication can be divided into real-time deduplication and offline deduplication. Real-time deduplication means that deduplication is performed when data is written to the storage device. The advantage is that deduplication can be completed before the data is written to the disk, reducing the number of data writes to the disk. However, it has a greater impact on the input / output (I / O) stability of the storage device, and the deduplication rate is relatively low. Offline deduplication means that after the original data is written to the disk, offline deduplication is performed when certain conditions are met. Since it is executed in the background, it has less impact on the I / O stability of the storage device, and an algorithm with a higher deduplication rate can be selected for deduplication.

[0084] Generally, in deduplication processing, in addition to data deletion, it is often necessary to compress the deleted data. Both of these steps require a large amount of computing resources, such as central processing unit (CPU) resources.

[0085] The following are the deduplication algorithms in related technologies:

[0086] (1) Hybrid duplicate data deletion scheduling method: Deduplication is performed based on whether the service data type is delay-sensitive or bandwidth-sensitive; or the ratio of online deduplication and offline deduplication is set for deduplication.

[0087] (2) Execution strategy of hierarchical compression: A threshold is set for the similarity degree of the data to be compressed. For the data to be compressed with a similarity degree higher than the threshold, differential compression is used, and for the compressed data with a similarity degree lower than the threshold, another compression method is selected to make full use of the computing resources of the storage device.

[0088] (3) Execution strategy of multiple deduplication domains: The storage device is divided into multiple deduplication domains. One address can belong to different deduplication domains at the same time. Multiple deduplication domains have different priorities, and different priorities correspond to different deduplication strategies. When data is written to the disk, the corresponding deduplication strategy is determined according to the priority of the current deduplication domain, that is, deduplication is performed with this deduplication strategy. Since deduplication is performed according to the deduplication strategies corresponding to different priorities, a higher deduplication rate can be obtained.

[0089] However, the factors considered in the above methods are relatively single. For example, methods (1) and (2) can reduce the resource consumption of the storage device, but the deduplication rate is relatively low; method (3) can achieve a higher deduplication rate, but it consumes more resources of the storage device, and it will cause the storage device to freeze when the storage device is in a high-load state.

[0090] However, storage devices do not always operate under high load conditions and may also be in a low load operating state; moreover, different amounts of data to be deduplicated consume different amounts of resources of the storage device. Therefore, how to perform deduplication based on the actual operating conditions of the storage device and the amount of data to be deduplicated to achieve a high deduplication rate or high stability of the storage device is a problem that needs to be solved.

[0091] Therefore, the present application provides a data processing method. In a single deduplication task, according to the current load of the storage device and the number of eigenvalue data to be deduplicated, an appropriate deletion algorithm and compression algorithm are adaptively selected, so as to ensure the stability of the storage device under high load conditions of the storage device, and effectively ensure the deduplication rate under low load conditions of the storage device.

[0092] It can be understood that in the embodiments of the present application, the stability of the storage device can be understood as whether the storage device has phenomena such as freezing. For example, if the resource consumption of the storage device is large or the bandwidth usage during data transmission of the storage device is unstable, the stability of the storage device will decrease.

[0093] For ease of understanding, the present application Figure 1 describes the process of a single deduplication task involved in the present application. Figure 1 The deduplication process can be understood as an offline deduplication process.

[0094] Figure 1 FIG. is a schematic diagram of the structure of the storage device according to an example of the present application and the process of a single deduplication task based on this structure. As Figure 1 shown, the storage device 10 includes a controller 101, a data domain 102, and a deduplication domain 103. The data domain 102 is used to store original data, and the deduplication domain 103 is used to store eigenvalues corresponding to the total original data.

[0095] Specifically, the data domain 102 includes multiple storage areas. Exemplarily, the size of each storage area can be 8KB (8 kilobytes), and the multiple storage areas are respectively used to store multiple groups of original data in the total original data.

[0096] In step A, when the storage device 10 writes data, it can store data in units of storage areas, and each storage area stores 8KB of data. Exemplarily, the total original data may include, for example, one or more different data types such as pictures, audio, video, and text.

[0097] In step B, the controller 101 can calculate a feature value for the original data of each storage area and store the feature value in the deduplication domain 103. It can be understood that in the deduplication domain 103, there are also multiple storage areas, which are respectively used to store different feature values. For the original data of any storage area, the feature value of the original data can be used to compare with the feature values of the original data of other storage areas to describe the similarity of these two original data. For example, if the original data stored in two storage areas are similar or the same, then the feature values calculated from the data of these two storage areas are also the same.

[0098] In the embodiments of the present application, the feature value can also be referred to as a fingerprint, which can be understood as a mapping of the original data of a storage area.

[0099] In a possible implementation manner, when the controller 101 stores the feature values in the deduplication domain 103, it can classify these feature values. That is to say, there are feature values of multiple data types stored in the deduplication domain 103, and each data type includes multiple subtypes. For the original data stored in the data domain 102 that is of the picture type, these data include pictures of different subtypes such as cats, dogs, and trees. The controller 101 can classify the feature values based on the subtypes. For example, the feature values corresponding to the original data with the subtype of cat can be classified into one category, the feature values corresponding to the original data with the subtype of dog can be classified into one category, and the feature values corresponding to the original data with the subtype of tree can be classified into one category, so as to obtain the feature values of multiple subtypes in the picture type data type.

[0100] For the original data stored in the data domain 102 that is of the video type, these data include videos of different subtypes such as sports, entertainment, and news. The controller 101 can classify the feature values based on the subtypes. For example, the feature values corresponding to the original data with the subtype of sports can be classified into one category, the feature values corresponding to the original pictures with the subtype of entertainment can be classified into one category, and the feature values corresponding to the original pictures with the subtype of news can be classified into one category, so as to obtain the feature values of multiple subtypes in the video type data type.

[0101] For the original data stored in the data domain 102 that is of the audio type, these data include videos of different subtypes such as music, voice, and environment. The controller 101 can classify the feature values based on the subtypes. For example, the feature values corresponding to the original data with the subtype of music can be classified into one category, the feature values corresponding to the original pictures with the subtype of voice can be classified into one category, and the feature values corresponding to the original pictures with the subtype of environment can be classified into one category, so as to obtain the feature values of multiple subtypes in the audio type data type.

[0102] For ease of understanding, the above classifications are exemplified below through Table 1. It should be noted that the classifications in Table 1 are only examples. For example, the subtypes of videos can be further divided into smaller types according to sports, entertainment, news, etc. For example, sports can be divided into running, gymnastics, etc. The form of data classification in this application is not limited.

[0103] In step C, the controller 101 can obtain eigenvalue of multiple subtypes from the deduplication domain 103. Exemplarily, it is possible to further analyze the eigenvalue that can be deduplicated among the eigenvalue of these subtypes.

[0104] Steps D, E, and F are the data processing stages of a deduplication task. In step D, the eigenvalue of multiple subtypes obtained from the deduplication domain 103 derived in step C can be analyzed and filtered, etc., to select the eigenvalue that can be deduplicated.

[0105] Exemplarily, for any one subtype, the same eigenvalue among the eigenvalue of this subtype can be determined as the eigenvalue that can be deduplicated.

[0106] In step E, the original data corresponding to the eigenvalue that can be deduplicated selected in step D can be deduplicated.

[0107] Step F is executed after step E. Step F can deduplicate the eigenvalue that can be deduplicated.

[0108] So far, the process of a deduplication task is completed.

[0109] Table 1

[0110]

[0111] In the embodiment of the present application, deduplication means retaining one copy of the original data in the same storage area among the original data in multiple storage areas, deleting the original data in the remaining storage areas, and storing an index on the storage area after deleting the original data, and this index can point to the unique copy.

[0112] The technical solution of the present application and how the technical solution of the present application solves the above technical problems will be described in detail below with specific embodiments. These several specific embodiments below can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below with reference to the accompanying drawings.

[0113] Figure 2 It is a schematic flowchart of a data processing method provided by an embodiment of the present application. This method can be applied to a storage device, specifically to the controller 101 in the storage device 10 as Figure 1 shown. As Figure 2As shown, the method includes the following steps:

[0114] S201. Determine the current load of the storage device and the number of eigenvalues to be deduplicated.

[0115] When performing a deduplication task once, the storage device can determine the current load and the number of eigenvalues to be deduplicated.

[0116] Exemplarily, the way for the storage device to determine the current load can be, for example: determine the current load according to the resource operation parameters of the storage device, and the resource operation parameters can include, for example, CPU usage rate, I / O usage rate, etc.

[0117] In a possible implementation manner, the current load can be represented by a load level. The higher the load level, the greater the current load of the storage device at this time, and the lower the load level, the smaller the current load of the storage device at this time.

[0118] In a possible implementation manner, the number of eigenvalues to be deduplicated can be represented by a quantity level. The higher the quantity level, the more the number of eigenvalues to be deduplicated at this time, and the lower the quantity level, the fewer the number of eigenvalues to be deduplicated at this time.

[0119] Exemplarily, the way for the storage device to determine the number of eigenvalues to be deduplicated can be, for example: determine the target eigenvalues to be deleted among various types of eigenvalues in the deduplication domain, and use the number of the target eigenvalues as the number of eigenvalues to be deduplicated.

[0120] S202. Determine the target deletion algorithm and the target compression algorithm according to the current load and the number of eigenvalues to be deduplicated.

[0121] After the storage device determines the current load and the eigenvalues to be deduplicated, it can determine the target deletion algorithm and the target compression algorithm according to the current load and the number of eigenvalues to be deduplicated.

[0122] Exemplarily, the storage device can determine the target deletion algorithm and the target compression algorithm in the algorithm library, and the algorithm library includes various deletion algorithms and various compression algorithms.

[0123] It can be understood that different loads of the storage device and different numbers of eigenvalues to be deduplicated can determine different target deletion algorithms and target compression algorithms, and the target deletion algorithm and the target compression algorithm can meet the current load situation of the storage device and the requirements of the deduplication task.

[0124] S203. Perform deletion processing on the data to be deduplicated according to the target deletion algorithm to obtain the data after deletion processing. The data to be deduplicated includes the eigenvalues to be deduplicated and the original data corresponding to the eigenvalues.

[0125] After determining the target deletion algorithm and the target compression algorithm, the storage device can perform deletion processing on the data to be deduplicated according to the target deletion algorithm. Among them, the data to be deduplicated includes the eigenvalues to be deduplicated and the original data corresponding to the eigenvalues.

[0126] Exemplarily, the storage device can delete the original data corresponding to the eigenvalues to be deduplicated, and then delete the eigenvalues to be deduplicated. For example Figure 1 Steps F and E in

[0127] S204. According to the target compression algorithm, perform compression processing on the data after deletion processing to obtain the processed data.

[0128] After performing deletion processing on the data to be deduplicated according to the target compression algorithm, the storage device can perform compression processing on the data after deletion processing according to the target compression algorithm, and the processed data can be obtained.

[0129] In this embodiment, in a deduplication task, the storage device can determine the current load of the storage device and the number of eigenvalues to be deduplicated, and determine the target deletion algorithm and the target compression algorithm according to the current load and the number of eigenvalues to be deduplicated. Then, perform deduplication processing on the data to be deduplicated according to the target deletion algorithm to obtain the data after deduplication processing. The data to be deduplicated includes the eigenvalues to be deduplicated and the original data corresponding to the eigenvalues, and perform compression processing on the data after deduplication processing according to the target compression algorithm to obtain the processed data. This application adaptively selects a suitable deletion algorithm and compression algorithm according to the current load of the storage device and the number of eigenvalues to be deduplicated, so as to ensure the stability of the storage device under high load of the storage device, and effectively ensure the deduplication rate under low load of the storage device.

[0130] Next, a detailed description will be given of how the storage device determines the current load and the number of eigenvalues to be deduplicated.

[0131] Figure 3 FIG. is a schematic flowchart of another data processing method provided by an embodiment of the present application. This method can be applied to a storage device, specifically to a controller 101 in a storage device 10 as shown in Figure 1 As shown in Figure 3 As shown, the method includes the following steps:

[0132] S301. In the deduplication domain, obtain the eigenvalues of multiple subtypes in the target data type.

[0133] Among them, the target data type can be pictures, videos, voices, etc. The eigenvalues of multiple subtypes can be the eigenvalues after the controller classifies the eigenvalues of the original data corresponding to the target data type when storing the eigenvalues in the deduplication domain. Specifically, reference can be made toFigure 1 Corresponding classification examples. For each subtype, the characteristic values of the subtype may include one or more characteristic values.

[0134] S302. Determine multiple target characteristic values among the characteristic values of multiple subtypes.

[0135] The storage device can determine multiple target characteristic values among the characteristic values of multiple subtypes.

[0136] Exemplarily, among the characteristic values of multiple subtypes, for the characteristic values of any one subclass, all the characteristic values in this class of characteristic values can be compared, and the characteristic values with the same value (or the difference in values is less than a set threshold) can be determined as target characteristic values.

[0137] S303. Determine the number of multiple target characteristic values as the quantity of characteristic values to be deduplicated.

[0138] S304. Obtain the operating parameters of the storage device, where the operating parameters include the CPU usage rate of the storage device and / or the I / O usage rate of the storage device.

[0139] The storage device can obtain the operating parameters of the storage device such as the current CPU usage rate and I / O usage rate of the storage device. Among them, the CPU usage rate refers to the proportion of the time for the CPU to execute tasks within a unit time, reflecting the busy degree of CPU resources. The I / O usage rate refers to the load intensity of the storage device for processing I / O operations within a unit time, reflecting the data throughput efficiency.

[0140] It can be understood that the operating parameters of the storage device may also include other parameters that can be used to measure the load of the storage device, and this application does not limit this.

[0141] S305. Determine the current load according to the operating parameters.

[0142] After obtaining the operating parameters, the storage device can determine the current load according to the CPU usage rate and / or the I / O usage rate.

[0143] In a possible implementation manner, the storage device can determine the load level of the current load, and the load level is the first load level or the second load level. Among them, the first load level is higher than the second load level. The first load level can indicate that the storage device is currently in a high-load state, and the second load level can indicate that the storage device is currently in a low-load state.

[0144] Specifically, the storage device can determine the current load according to the following method:

[0145] Method 1

[0146] The storage device can determine the current load according to the I / O usage rate. If the I / O usage rate is greater than the first threshold, the current load can be determined to be the first load level. If the I / O usage rate is less than or equal to the first threshold, the current load can be determined to be the second load level.

[0147] Method 2

[0148] The storage device can determine the current load according to the CPU usage rate. If the CPU usage rate is greater than the second threshold, the current load can be determined to be the first load level. If the CPU usage rate is less than or equal to the second threshold, the current load can be determined to be the second load level.

[0149] Method 3

[0150] The storage device can determine the current load according to the CPU usage rate and the I / O usage rate. If the sum of the CPU usage rate and the I / O usage rate is greater than the second threshold, the current load can be determined to be the first load level. If the sum of the CPU usage rate and the I / O usage rate is less than or equal to the second threshold, the current load can be determined to be the second load level.

[0151] Method 4

[0152] Generally, in a storage device, when the I / O usage rate continuously exceeds a certain threshold (e.g., 70%), the response time of the storage device increases significantly, which may lead to backlog in the data read / write queue, directly affecting the throughput and stability of the storage system. The impact of the CPU usage rate on the stability of the storage device is relatively small compared to the impact of the I / O usage rate on the stability of the storage device.

[0153] Based on Method 3, the storage device can set different scaling factors for the CPU usage rate and the I / O usage rate. For example, the scaling factor for the I / O usage rate is set to 70%, and the scaling factor for the CPU usage rate is set to 30%. When determining the current load, the storage device can determine the current load according to the sum of the product of the CPU usage rate and the scaling factor of 30% and the product of the I / O usage rate and the scaling factor of 70%.

[0154] It can be understood that the above methods for determining the current load are only examples, and the current load can also be determined through other operating parameters and other methods. This application does not limit this.

[0155] In Figure 3 In the corresponding embodiment, the characteristic value to be deduplicated can be determined based on the characteristic values of multiple subclasses. Determining the current load based on the CPU usage rate and the I / O usage rate can enable subsequent selection of appropriate deletion algorithms and compression algorithms according to the current load and the number of characteristic values to be deduplicated.

[0156] Next, through Figure 4Describe in detail how the storage device determines the target deletion algorithm and the target compression algorithm according to the current load and the number of eigenvalues to be deduplicated.

[0157] Figure 4 It is a schematic flowchart of another data processing method provided by an embodiment of the present application. This method can be applied to a storage device, specifically to a controller 101 in a storage device 10 as shown in Figure 1 shown. As shown in Figure 4 shown, this method includes the following steps:

[0158] S401. Determine the load level of the current load and the quantity level of the number of eigenvalues to be deduplicated.

[0159] For the storage device to determine the load level of the current load, reference can be made to the above embodiments, which will not be elaborated here.

[0160] The storage device can determine the quantity level according to the number of eigenvalues to be deduplicated. Exemplarily, if the number of eigenvalues to be deduplicated is greater than the first quantity threshold, the quantity level can be determined as the first quantity level; if the number of eigenvalues to be deduplicated is less than or equal to the first quantity threshold, the quantity level can be determined as the second quantity level, where the first quantity level is higher than the second quantity level.

[0161] The first quantity level can indicate that there are more eigenvalues to be deduplicated in the storage device, that is, there are a large number of eigenvalues to be deduplicated in the storage device that need to be deduplicated currently. The second quantity level can indicate that there are fewer eigenvalues to be deduplicated in the storage device, that is, there are a small number of eigenvalues to be deduplicated in the storage device that need to be deduplicated currently.

[0162] S402. Determine the target deletion algorithm and the target compression algorithm in the algorithm library according to the load level and the quantity level.

[0163] The storage device can determine the target deletion algorithm and the target compression algorithm in the algorithm library according to the load level and the quantity level. Among them, the algorithm library includes multiple types of deletion algorithms and multiple types of compression algorithms. Among them, the storage device resources consumed by multiple types of deletion algorithms are different, and the compression ratios of multiple types of compression algorithms are different. Each type of deletion algorithm includes multiple deletion algorithms, and each type of compression algorithm includes multiple compression algorithms.

[0164] In a possible implementation manner, the deletion algorithms can be classified according to the amount of resource consumption. Multiple types of deletion algorithms include three types of deletion algorithms with low resource consumption, medium resource consumption, and high resource consumption. Each type of deletion algorithm includes multiple deletion algorithms. It can be understood that the resources mentioned here refer to the resources of the storage device, such as computing resources (i.e., CPU resources, etc.).

[0165] It should be noted that the three types of low resource consumption, medium resource consumption, and high resource consumption can be determined according to the amount of resource consumption of the storage device. For example, threshold 1 and threshold 2 can be set, where threshold 1 is less than threshold 2. Taking a certain deletion algorithm as an example, if the amount of resources of the storage device consumed by the deletion algorithm during operation is less than or equal to threshold 1, it can be determined that the deletion algorithm is a deletion algorithm of the low resource consumption type; if the amount of resources of the storage device consumed by the deletion algorithm during operation is greater than threshold 1 and less than threshold 2, it can be determined that the deletion algorithm is a deletion algorithm of the medium resource consumption type; if the amount of resources of the storage device consumed by the deletion algorithm during operation is greater than or equal to threshold 2, it can be determined that the deletion algorithm is a deletion algorithm of the high resource consumption type.

[0166] In a possible implementation, the compression algorithms can be classified according to the compression ratio. The multiple types of deletion algorithms include three types of compression algorithms with low compression ratio, medium compression ratio, and high compression ratio. Each type of compression algorithm includes multiple compression algorithms.

[0167] Exemplarily, the multiple types of deletion algorithms and multiple types of compression algorithms in the algorithm library are shown in Table II.

[0168] The storage device can determine the target deletion algorithm and the target compression algorithm in the algorithm library shown in Table II according to the load level and the quantity level.

[0169] Table II

[0170]

[0171] In a possible implementation, the storage device can determine the target deletion algorithm and the target compression algorithm in the following manner:

[0172] If the load level is the first load level and the quantity level is the first quantity level, then determine that the target deletion algorithm is the first deletion algorithm, and the target compression algorithm is the first compression algorithm. The first deletion algorithm belongs to the category of deletion algorithms with the lowest resource consumption among the multiple types of deletion algorithms, and the first compression algorithm belongs to the category of compression algorithms with the lowest compression ratio among the multiple types of compression algorithms. Taking Table II as an example, the first deletion algorithm can be any one of deletion algorithms a1, a2,... and the first compression algorithm can be any one of compression algorithms d1, d2,...

[0173] It can be understood that when the load level is the first load level and the quantity level is the first quantity level, it indicates that the current load of the storage device is relatively high, and the number of feature values and original data to be deduplicated in this deduplication task is relatively large. This deduplication task puts the storage device in a high-pressure state. At this time, selecting the first deletion algorithm and the first compression algorithm, that is, the deletion algorithm with relatively low resource consumption and the compression algorithm with a low compression ratio, can ensure the stability of the storage device. For example, it can ensure the stability of the bandwidth of the storage device.

[0174] If the load level is the first load level and the quantity level is the second quantity level, or the load level is the second load level and the quantity level is the first quantity level, then determine that the target deletion algorithm is the second deletion algorithm and the target compression algorithm is the second compression algorithm. The storage device resources consumed by the second deletion algorithm are higher than those consumed by the first deletion algorithm, and the compression ratio of the second compression algorithm is greater than that of the first compression algorithm. Taking Table 2 as an example, the second deletion algorithm can be any one of deletion algorithms b1, b2,... and the first compression algorithm can be any one of compression algorithms e1, e2,...

[0175] It can be understood that when the load level is the first load level and the quantity level is the second quantity level, it indicates that the current load of the storage device is relatively high, but the number of feature values and original data to be deduplicated in this deduplication task is relatively small. At this time, a relatively balanced deletion algorithm and compression algorithm can be selected, that is, select the second deletion algorithm and the second compression algorithm, that is, the deletion algorithm with medium resource consumption and the compression algorithm with a medium compression ratio, which can not only ensure the stability of the storage device but also ensure that the obtained deduplication rate is at a relatively high level.

[0176] Or, if the load level is the second load level and the quantity level is the first quantity level, it indicates that the current load of the storage device is relatively low, but the number of feature values and original data to be deduplicated in this deduplication task is relatively large. Then the selection of the deletion algorithm and the compression algorithm can be similar to the case where the load level is the first load level and the quantity level is the second quantity level as described above, which will not be elaborated here.

[0177] If the load level is the second load level and the quantity level is the second quantity level, then determine that the target deletion algorithm is the third deletion algorithm and the target compression algorithm is the third compression algorithm. The storage device resources consumed by the third deletion algorithm are higher than those consumed by the second deletion algorithm, and the compression ratio of the third compression algorithm is greater than that of the second compression algorithm. Taking Table 2 as an example, the third deletion algorithm can be any one of deletion algorithms c1, c2,... and the first compression algorithm can be any one of compression algorithms f1, f2,...

[0178] It can be understood that when the load level is the second load level and the quantity level is the second quantity level, it indicates that the current load of the storage device is relatively low, and the number of feature values and original data to be deduplicated in this deduplication task is relatively small. At this time, a deletion algorithm with relatively high resource consumption and a compression algorithm with a high compression ratio can be adopted, that is, the third deletion algorithm and the third compression algorithm are selected, so as to obtain a relatively high deduplication rate.

[0179] In a possible implementation, each type of deletion algorithm includes multiple deletion algorithms, and the multiple deletion algorithms respectively correspond to different data types; each type of compression algorithm includes multiple compression algorithms, and the multiple compression algorithms respectively correspond to different data types. The data type corresponding to the target deletion algorithm is the data type of the data to be deduplicated; the data type corresponding to the target compression algorithm is the data type of the data to be deduplicated.

[0180] Taking the data types including pictures, videos, and audios as an example, the algorithm library can be as shown in Table 3.

[0181] Taking the data type of the data to be deduplicated as pictures, the load level as the first load level, the quantity level as the first quantity level, the target deletion algorithm as the picture type deletion algorithm a1, and the target compression algorithm as the picture type compression algorithm d1.

[0182] By setting deletion algorithms applicable to multiple data types for each type of deletion algorithm, and compression algorithms applicable to multiple data types, and the data type corresponding to the selected target deletion algorithm is the data type of the data to be deduplicated, and the data type corresponding to the selected target compression algorithm is the data type of the data to be deduplicated, the data processing effect can be improved.

[0183] Table 3

[0184]

[0185] For the sake of easy understanding, for the above determination of the target deletion algorithm and the target compression algorithm according to the load level and the quantity level, specific reference can be made to Figure 5 .

[0186] In Figure 4 the corresponding embodiment, according to the load level of the current load and the quantity level of the number of feature values to be deduplicated, the target deletion algorithm and the target compression algorithm are determined from multiple types of deletion algorithms and multiple types of compression algorithms in the algorithm library, so that the algorithms used can meet the requirements of the current actual situation.

[0187] Next, a detailed description will be given on how the storage device deletes the feature values to be deduplicated and the original data corresponding to the feature values to be deduplicated.

[0188] Figure 6The flowchart of another data processing method provided by an embodiment of this application. This method can be applied to a storage device, specifically to the controller 101 in the storage device 10 as shown in Figure 1 as follows. As shown in Figure 6 the method includes the following steps:

[0189] S601: Based on the target deletion algorithm, delete N - 1 groups of the N groups of original data, and store a first index in the storage area in the data domain where the N - 1 groups of original data are located. The first index points to the one group of original data that is not deleted among the N groups of original data.

[0190] The storage device can, based on the target deletion algorithm, delete N - 1 groups of the N groups of original data, and store a first index in the storage area in the data domain where the N - 1 groups of original data are located. Exemplarily, Figure 7 a schematic diagram of storing the first index after data deletion in an example of this application can be referred to Figure 7 . Where N is a positive integer.

[0191] It can be understood that the one group of original data that is not deleted among the N groups of original data is the only copy of the N groups of original data.

[0192] Exemplarily, the first index can be the address of the storage area where the one group of original data that is not deleted among the N groups of original data is located, or it can be an identifier corresponding to the address, etc. This application does not limit this. When the original data group a in the N - 1 groups of original data is accessed, the one group of original data that is not deleted among the N groups of original data can be accessed based on the first index, and this original data is used as the data of the original data group a.

[0193] S602: Based on the target deletion algorithm, delete the eigenvalues corresponding to the N - 1 groups of original data among the N eigenvalues, and store a second index in the storage area in the deduplication domain where the eigenvalues corresponding to the N - 1 groups of original data are located. The second index points to the one eigenvalue that is not deleted among the N eigenvalues.

[0194] After deleting N - 1 groups of the N groups of original data, the storage device can, based on the target deletion algorithm, delete the eigenvalues corresponding to the N - 1 groups of original data among the N eigenvalues, and store a second index in the storage area in the deduplication domain where the eigenvalues corresponding to the N - 1 groups of original data are located. The second index points to the one eigenvalue that is not deleted among the N eigenvalues.

[0195] It can be understood that the one eigenvalue that is not deleted among the N eigenvalues is the only copy of the N eigenvalues.

[0196] Exemplarily, the second index may be the address of the storage area where one of the undeleted eigenvalues among the N eigenvalues is located, or may be an identifier corresponding to the address, etc., and the present application does not limit this. When the eigenvalue b among the eigenvalues corresponding to the N - 1 groups of original data is accessed, one of the undeleted eigenvalues among the N eigenvalues can be accessed based on the second index, and this eigenvalue is used as the eigenvalue b.

[0197] In this embodiment, corresponding indexes are stored at the positions of the data after deletion, which can point to the corresponding copies, so that the corresponding data can be obtained according to the indexes when obtaining data subsequently. The data here may include original data and eigenvalues.

[0198] Next, another data processing method provided by the embodiments of the present application will be described.

[0199] Figure 8 It is a schematic flowchart of another data processing method provided by the embodiments of the present application. This method can be applied to a storage device, specifically to the controller 101 in the storage device 10 as shown in Figure 1 As shown. As shown in Figure 8 As shown, the method includes the following steps:

[0200] S801. In the deduplication domain, obtain the eigenvalues of multiple subtypes in the target data type.

[0201] S802. Determine multiple target eigenvalues among the eigenvalues of multiple subtypes.

[0202] S803. Determine the number of multiple target eigenvalues as the number of eigenvalues to be deduplicated.

[0203] S804. Obtain the operating parameters of the storage device. The operating parameters include the CPU usage rate of the storage device and / or the I / O usage rate of the storage device.

[0204] S805. Determine the current load according to the operating parameters.

[0205] S806. Determine the load level of the current load and the quantity level of the number of eigenvalues to be deduplicated.

[0206] S807. Determine the target deletion algorithm and the target compression algorithm in the algorithm library according to the load level and the quantity level.

[0207] S808. Based on the target deletion algorithm, delete N - 1 groups of the N groups of original data, and store the first index in the storage area in the data domain where the N - 1 groups of original data are located.

[0208] The first index points to the group of original data that is not deleted among the N groups of original data.

[0209] S809. Based on the target deletion algorithm, delete the eigenvalues corresponding to N - 1 groups of original data among the N eigenvalues, and store the second index in the storage area in the duplicate deletion domain where the eigenvalues corresponding to the N - 1 groups of original data are located.

[0210] The second index points to the one eigenvalue that is not deleted among the N eigenvalues.

[0211] S810. According to the target compression algorithm, perform compression processing on the data after the deletion processing to obtain the processed data.

[0212] In this embodiment, the specific implementation manner and technical effects are similar to those of the above embodiments, and will not be elaborated here.

[0213] Figure 9 The following is a schematic structural diagram of a data processing device provided by an embodiment of the present application. This device is applied to a storage device, such as Figure 9 As shown, the device 90 includes: a first determination module 901, a second determination module 902, a first processing module 903, and a second processing module 904.

[0214] The first determination module 901 is used to determine the current load of the storage device and the number of eigenvalues to be duplicate - deleted.

[0215] The second determination module 902 is used to determine the target deletion algorithm and the target compression algorithm according to the current load and the number.

[0216] The first processing module 903 is used to perform deletion processing on the data to be duplicate - deleted according to the target deletion algorithm to obtain the data after the deletion processing. The data to be duplicate - deleted includes the eigenvalues to be duplicate - deleted and the original data corresponding to the eigenvalues.

[0217] The second processing module 904 is used to perform compression processing on the data after the deletion processing according to the target compression algorithm to obtain the processed data.

[0218] Optionally, for the above - mentioned device 90, the first determination module 901 is specifically used for:

[0219] In the duplicate deletion domain, obtain the eigenvalues of multiple subtypes in the target data type.

[0220] Among the eigenvalues of the multiple subtypes, determine multiple target eigenvalues.

[0221] Determine the number of the multiple target eigenvalues as the number of eigenvalues to be duplicate - deleted.

[0222] Obtain the operating parameters of the storage device. The operating parameters include the CPU usage rate of the storage device and / or the input / output usage rate of the storage device.

[0223] Determine the current load according to the operating parameters.

[0224] Optionally, for the above device 90, the second determination module 902 is specifically configured to:

[0225] Determine the load level of the current load and the quantity level of the quantity, where the load level is the first load level and the second load level, and the quantity level is the first quantity level or the second quantity level. Among them, the first load level is higher than the second load level, and the first quantity level is higher than the second quantity level.

[0226] According to the load level and the quantity level, determine the target deletion algorithm and the target compression algorithm in the algorithm library.

[0227] Optionally, for the above device 90, the second determination module 902 is specifically configured to:

[0228] If the load level is the first load level and the quantity level is the first quantity level, then determine that the target deletion algorithm is the first deletion algorithm, and the target compression algorithm is the first compression algorithm. The first deletion algorithm belongs to the category of deletion algorithms with the lowest resource consumption among multiple types of deletion algorithms, and the first compression algorithm belongs to the category of compression algorithms with the lowest compression ratio among multiple types of compression algorithms.

[0229] If the load level is the first load level and the quantity level is the second quantity level, or the load level is the second load level and the quantity level is the first quantity level, then determine that the target deletion algorithm is the second deletion algorithm, and the target compression algorithm is the second compression algorithm. The storage device resources consumed by the second deletion algorithm are higher than those consumed by the first deletion algorithm, and the compression ratio of the second compression algorithm is greater than that of the first compression algorithm.

[0230] If the load level is the second load level and the quantity level is the second quantity level, then determine that the target deletion algorithm is the third deletion algorithm, and the target compression algorithm is the third compression algorithm. The storage device resources consumed by the third deletion algorithm are higher than those consumed by the second deletion algorithm, and the compression ratio of the third compression algorithm is greater than that of the second compression algorithm.

[0231] Optionally, for the above device 90, each type of deletion algorithm includes multiple deletion algorithms, and the multiple deletion algorithms respectively correspond to different data types; each type of compression algorithm includes multiple compression algorithms, and the multiple compression algorithms respectively correspond to different data types;

[0232] The data type corresponding to the target deletion algorithm is the data type of the data to be deduplicated; the data type corresponding to the target compression algorithm is the data type of the data to be deduplicated.

[0233] Optionally, for the above device 90, the eigenvalues to be deduplicated include N eigenvalues, and the original data corresponding to the eigenvalues to be deduplicated includes N groups of original data; where N is a positive integer.

[0234] The first processing module 903 is specifically configured to:

[0235] Based on the target deletion algorithm, delete N - 1 groups of the N groups of original data, and store a first index in the storage area in the data domain where the N - 1 groups of original data are located, and the first index points to the one group of original data that is not deleted among the N groups of original data.

[0236] Based on the target deletion algorithm, delete the eigenvalues corresponding to N - 1 groups of the N eigenvalues, and store a second index in the storage area in the duplicate deletion domain where the eigenvalues corresponding to the N - 1 groups of original data are located, and the second index points to the one eigenvalue that is not deleted among the N eigenvalues.

[0237] The device in this embodiment can be used to execute the technical solutions of the above - mentioned method embodiments. The specific implementation manners and technical effects are similar and will not be elaborated here.

[0238] Figure 10 The following is a schematic structural diagram of an electronic device provided by an embodiment of the present application. As Figure 10 shown, the electronic device 100 may include: at least one processor 1001 and a memory 1002.

[0239] The memory 1002 is used to store a program. Specifically, the program may include program code, and the program code includes computer - executable instructions.

[0240] The memory 1002 may include a random access memory (RAM), and may also include a non - volatile memory, such as at least one disk memory.

[0241] The processor 1001 is used to execute the computer - executable instructions stored in the memory 1002 to implement the method described in the foregoing method embodiments. Among them, the processor 1001 may be a CPU, or an application - specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0242] Optionally, the electronic device 100 may further include: a communication interface 1003. In a specific implementation, if the communication interface 1003, the memory 1002, and the processor 1001 are implemented independently, the communication interface 1003, the memory 1002, and the processor 1001 may be interconnected through a bus and communicate with each other. The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The bus may be divided into an address bus, a data bus, a control bus, etc., but it does not mean that there is only one bus or one type of bus.

[0243] Optionally, in a specific implementation, if the communication interface 1003, the memory 1002, and the processor 1001 are integrated on a single chip, the communication interface 1003, the memory 1002, and the processor 1001 may communicate through an internal interface.

[0244] The electronic device 100 may be a storage device.

[0245] The electronic device of this embodiment can be used to execute the technical solutions of the above method embodiments. The specific implementation manners and technical effects are similar and will not be elaborated here.

[0246] This application embodiment provides a computer-readable storage medium, which may include: various media such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a RAM, a magnetic disk, or an optical disc that can store computer-executable instructions. Specifically, the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a computer, the technical solutions shown in the above method embodiments are executed. The specific implementation manners and technical effects are similar and will not be elaborated here.

[0247] This application embodiment provides a computer program product, including a computer program. When the computer program is executed by a computer, the technical solutions shown in the above method embodiments are executed. The specific implementation manners and technical effects are similar and will not be elaborated here.

[0248] It should be noted that, for the foregoing method embodiments, for the sake of simple description, they are all expressed as a series of action combinations. However, those skilled in the art should know that this application is not limited by the described action sequence, because according to this application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this application.

[0249] Furthermore, it should be noted that although the steps in the flowchart are shown in sequence according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless there is a clear description in this article, the execution of these steps has no strict order restriction, and these steps can be executed in other sequences. Moreover, at least a part of the steps in the flowchart may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed alternately or in turn with at least a part of other steps or sub-steps or stages of other steps.

[0250] It should be understood that the foregoing device embodiments are only illustrative, and the devices of this application can also be implemented in other ways. For example, the division of units / modules in the foregoing embodiments is only a logical function division, and there can be other division methods in actual implementation. For example, multiple units, modules or components can be combined, or can be integrated into another system, or some features can be ignored or not executed.

[0251] In addition, without special instructions, in each embodiment of this application, each functional unit / module can be integrated in one unit / module, or each unit / module can exist physically alone, or two or more units / modules can be integrated together. The above integrated unit / module can be implemented in the form of hardware or in the form of a software program module.

[0252] When the integrated unit / module is implemented in the form of hardware, the hardware can be a digital circuit, an analog circuit, etc. The physical implementation of the hardware structure includes but is not limited to transistors, memristors, etc. Unless otherwise specified, the processor can be any suitable hardware processor, such as a CPU, a graphics processing unit (GPU), a field programmable gate array (FPGA), a digital signal processing (DSP) chip, and an application-specific integrated circuit (ASIC), etc. Unless otherwise specified, the storage unit can be any suitable magnetic storage medium or magneto-optical storage medium, such as a resistive random access memory (RRAM), a dynamic random access memory (DRAM), a static random-access memory (SRAM), an enhanced dynamic random access memory (EDRAM), a high-bandwidth memory (HBM), a hybrid memory cube (HMC), etc.

[0253] When the integrated unit / module is implemented in the form of a software program module and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods in various embodiments of this application. And the aforementioned memory includes: various media that can store program codes, such as a USB flash drive, a ROM, a RAM, a mobile hard disk, a magnetic disk, or an optical disc.

[0254] In the above embodiments, the descriptions of the various embodiments have their own emphases. For the parts not detailed in a certain embodiment, reference can be made to the relevant descriptions of other embodiments. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered to be within the scope described in this specification.

[0255] Other embodiments of the present application will be readily contemplated by those skilled in the art upon considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present application, which follow the general principles of the present application and include known common knowledge or conventional technical means in the technical field not disclosed in the present application. The specification and examples are only illustrative, and the true scope and spirit of the present application are pointed out by the following claims.

[0256] It should be understood that the present application is not limited to the exact structures described above and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of the present application is only limited by the appended claims.

Claims

1. A data processing method, characterized in that, Applied to a storage device, the storage device including a data domain and a deduplication domain, multiple storage areas in the data domain are respectively used to store multiple groups of original data, and multiple storage areas in the deduplication domain are respectively used to store the eigenvalue corresponding to the multiple groups of original data. The method includes: Determine the current load of the storage device and the number of eigenvalues to be deduplicated; Determine a target deletion algorithm and a target compression algorithm according to the current load and the number; Delete the data to be deduplicated according to the target deletion algorithm to obtain the data after deletion processing. The data to be deduplicated includes the eigenvalue to be deduplicated and the original data corresponding to the eigenvalue; Compress the data after deletion processing according to the target compression algorithm to obtain the processed data.

2. The method according to claim 1, characterized in that The deduplication domain includes eigenvalues of multiple data types, and each data type includes multiple subtypes; Determine the current load of the storage device and the number of eigenvalues to be deduplicated, including: In the deduplication domain, obtain the eigenvalues of multiple subtypes in the target data type; Determine multiple target eigenvalues among the eigenvalues of the multiple subtypes; Determine the number of the multiple target eigenvalues as the number of eigenvalues to be deduplicated; Obtain the operating parameters of the storage device, where the operating parameters include the CPU usage rate of the storage device and / or the input / output usage rate of the storage device; Determine the current load according to the operating parameters.

3. The method according to claim 1 or 2, characterized in that, Determine a target deletion algorithm and a target compression algorithm according to the current load and the number, including: Determine the load level of the current load and the number level of the number. The load level is the first load level or the second load level, and the number level is the first number level or the second number level. Among them, the first load level is higher than the second load level, and the first number level is higher than the second number level; Determine the target deletion algorithm and the target compression algorithm in the algorithm library according to the load level and the number level.

4. The method according to claim 3, characterized in that, Determine the target deletion algorithm and the target compression algorithm in the algorithm library according to the load level and the number level, including: If the load level is the first load level and the number level is the first number level, then determine the target deletion algorithm as the first deletion algorithm and the target compression algorithm as the first compression algorithm. The first deletion algorithm belongs to a type of deletion algorithm with the lowest resource consumption among multiple types of deletion algorithms, and the first compression algorithm belongs to a type of compression algorithm with the lowest compression rate among multiple types of compression algorithms; If the load level is the first load level and the number level is the second number level, or the load level is the second load level and the number level is the first number level, then determine the target deletion algorithm as the second deletion algorithm and the target compression algorithm as the second compression algorithm. The second deletion algorithm consumes more storage device resources than the first deletion algorithm, and the compression rate of the second compression algorithm is greater than that of the first compression algorithm; If the load level is the second load level and the quantity level is the second quantity level, determine that the target deletion algorithm is the third deletion algorithm and the target compression algorithm is the third compression algorithm. The storage device resources consumed by the third deletion algorithm are higher than those consumed by the second deletion algorithm, and the compression ratio of the third compression algorithm is greater than that of the second compression algorithm.

5. The method according to claim 3 or 4, characterized in that, Each type of deletion algorithm includes multiple deletion algorithms, and the multiple deletion algorithms respectively correspond to different data types; each type of compression algorithm includes multiple compression algorithms, and the multiple compression algorithms respectively correspond to different data types; The data type corresponding to the target deletion algorithm is the data type of the data to be deduplicated; the data type corresponding to the target compression algorithm is the data type of the data to be deduplicated.

6. The method according to any one of claims 1-5, characterized in that, The eigenvalues to be deduplicated include N eigenvalues, and the original data corresponding to the eigenvalues to be deduplicated includes N groups of original data; where N is a positive integer; Performing deletion processing on the data to be deduplicated according to the target deletion algorithm to obtain the data after deletion processing, including: Based on the target deletion algorithm, delete N - 1 groups of the N groups of original data, and store a first index in the storage area in the data domain where the N - 1 groups of original data are located. The first index points to the group of original data that is not deleted among the N groups of original data; Based on the target deletion algorithm, delete the eigenvalues corresponding to the N - 1 groups of original data among the N eigenvalues, and store a second index in the storage area in the deduplication domain where the eigenvalues corresponding to the N - 1 groups of original data are located. The second index points to the eigenvalue that is not deleted among the N eigenvalues.

7. A data processing device, characterized in that, Applied to a storage device, the storage device includes a data domain and a deduplication domain. Multiple storage areas in the data domain are respectively used to store multiple groups of original data, and multiple storage areas in the deduplication domain are respectively used to store the eigenvalues corresponding to the multiple groups of original data. The device includes: A first determination module, configured to determine the current load of the storage device and the quantity of the eigenvalues to be deduplicated; A second determination module, configured to determine a target deletion algorithm and a target compression algorithm according to the current load and the quantity; A first processing module, configured to perform deletion processing on the data to be deduplicated according to the target deletion algorithm to obtain the data after deletion processing. The data to be deduplicated includes the eigenvalues to be deduplicated and the original data corresponding to the eigenvalues; A second processing module, configured to perform compression processing on the data after deletion processing according to the target compression algorithm to obtain the processed data.

8. An electronic device, characterized in that, Including: A processor and a memory communicatively connected to the processor; The memory stores computer execution instructions; The processor executes the computer execution instructions stored in the memory to implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, Computer execution instructions are stored in the computer-readable storage medium, and when the computer execution instructions are executed by a processor, they are used to implement the method according to any one of claims 1 to 6.

10. A computer program product, characterized in that, Comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1-6.

Images