Data processing method and device, electronic equipment, storage medium and program product

Abstract

The application provides a data processing method and device, electronic equipment, storage medium and program product. The method comprises the following steps: in a deduplication task, the storage device can determine the current load of the storage device and the number of feature values to be deduplicated, and determine a target deletion algorithm and a target compression algorithm according to the current load and the number of feature values to be deduplicated. Then, the data to be deduplicated is deduplicated according to the target deletion algorithm to obtain deduplicated data, and the deduplicated data comprises the feature values to be deduplicated and the original data corresponding to the feature values, and the deduplicated data is compressed according to the target compression algorithm to obtain processed data. According to the current load of the storage device and the number of feature values to be deduplicated, the application adaptively selects a suitable deletion algorithm and a compression algorithm, which can ensure the stability of the storage device in the case of high load of the storage device, and effectively ensure the deduplication rate in the case of low load of the storage device.

Description

Technical Field

[0001] This application relates to the field of storage technology, and in particular to a data processing method, apparatus, electronic device, storage medium, and program product. Background Technology

[0002] With the advent of the information age, the volume of data has grown exponentially, leading to enormous challenges for data storage, such as rising storage costs and reduced storage efficiency. Since a large amount of duplicate data exists in the stored data, deleting duplicate data can effectively reduce the data volume, alleviate storage pressure, and save storage costs.

[0003] The deduplication ratio is an important metric for measuring the effectiveness of deduplication technologies. In related technologies, the data disk can be divided into multiple deduplication domains, each with different priorities and deduplication strategies. By applying different deduplication strategies to different domains, the deduplication ratio can be significantly improved.

[0004] However, the above method consumes a lot of storage device resources. For example, when the storage device is under high load, this method can cause the storage device to lag. Summary of the Invention

[0005] This application provides a data processing method, apparatus, electronic device, storage medium, and program product to solve the technical problem that deduplication methods in related technologies consume a large amount of storage device resources.

[0006] In a first aspect, this application provides a data processing method applied to a storage device, the storage device including a data domain and a deduplication domain, wherein multiple storage areas in the data domain are respectively used to store multiple sets of original data, and multiple storage areas in the deduplication domain are respectively used to store feature values ​​corresponding to the multiple sets of original data, the method comprising:

[0007] Determine the current load of the storage device and the number of feature values ​​to be deduplicated;

[0008] Based on the current load and the quantity, determine the target deletion algorithm and the target compression algorithm;

[0009] The data to be deleted is processed according to the target deletion algorithm to obtain the data after deletion. The data to be deleted includes the feature value to be deleted and the original data corresponding to the feature value.

[0010] The data after deletion is compressed according to the target compression algorithm to obtain the processed data.

[0011] In this embodiment, during a deduplication task, the storage device can determine its current load and the number of feature values ​​to be deduplicated. Based on these factors, it determines the target deletion algorithm and the target compression algorithm. Then, the data to be deduplicated is processed according to the target deletion algorithm, resulting in deduplicated data. The data to be deduplicated includes the feature values ​​to be deduplicated and the corresponding original data. Finally, the deduplicated data is compressed according to the target compression algorithm, resulting in processed data. This application adaptively selects appropriate deletion and compression algorithms based on the storage device's current load and the number of feature values ​​to be deduplicated, ensuring storage device stability under high load and effectively guaranteeing the deduplication rate under low load.

[0012] Optionally, the method described above, determining the current load of the storage device and the number of feature values ​​to be deduplicated, includes:

[0013] Within the deduplication domain, feature values ​​of multiple subtypes of the target data type are obtained;

[0014] Among the feature values ​​of the various subtypes, multiple target feature values ​​are determined;

[0015] The number of the plurality of target feature values ​​is determined as the number of feature values ​​to be deleted;

[0016] Obtain the operating parameters of the storage device, including the CPU utilization and / or the input / output utilization of the storage device;

[0017] The current load is determined based on the operating parameters.

[0018] In this embodiment, the feature values ​​to be deleted can be determined based on the feature values ​​of multiple subclasses. Determining the current load based on CPU utilization and I / O utilization allows for the selection of appropriate deletion and compression algorithms based on the current load and the number of feature values ​​to be deleted.

[0019] Optionally, the method described above, determining the target deletion algorithm and the target compression algorithm based on the current load and the quantity, includes:

[0020] Determine the load level of the current load and the quantity level of the quantity, wherein the load level is a first load level and a second load level, and the quantity level is a first quantity level or a second quantity level, wherein the first load level is higher than the second load level, and the first quantity level is higher than the second quantity level;

[0021] Based on the load level and the quantity level, the target deletion algorithm and the target compression algorithm are determined in the algorithm library.

[0022] In this embodiment, the target deletion algorithm and target compression algorithm are determined from the algorithm library based on the current load level and the number of feature values ​​to be deleted, so that the algorithm used can meet the needs of the current actual situation.

[0023] Optionally, the method described above, determining the target deletion algorithm and the target compression algorithm in the algorithm library based on the load level and the quantity level, includes:

[0024] If the load level is the first load level and the quantity level is the first quantity level, then the target deletion algorithm is determined to be the first deletion algorithm and the target compression algorithm is determined to be the first compression algorithm. The first deletion algorithm belongs to the deletion algorithm with the lowest resource consumption among the multiple deletion algorithms, and the first compression algorithm belongs to the compression algorithm with the lowest compression ratio among the multiple compression algorithms.

[0025] If the load level is a first load level and the quantity level is a second quantity level, or if the load level is a second load level and the quantity level is a first quantity level, then the target deletion algorithm is determined to be a second deletion algorithm, and the target compression algorithm is determined to be a second compression algorithm. The storage device resources consumed by the second deletion algorithm are higher than those consumed by the first deletion algorithm, and the compression ratio of the second compression algorithm is greater than that of the first compression algorithm.

[0026] If the load level is the second load level and the quantity level is the second quantity level, then the target deletion algorithm is determined to be the third deletion algorithm, and the target compression algorithm is determined to be the third compression algorithm. The storage device resources consumed by the third deletion algorithm are higher than those consumed by the second deletion algorithm, and the compression ratio of the third compression algorithm is greater than that of the second compression algorithm.

[0027] In this embodiment, the target deletion algorithm and the target compression algorithm are determined from multiple deletion algorithms and multiple compression algorithms based on the current load level and the number of feature values ​​to be deleted, so that the algorithm used can meet the needs of the current actual situation.

[0028] Optionally, in the method described above, each type of deletion algorithm includes multiple deletion algorithms, each corresponding to a different data type; each type of compression algorithm includes multiple compression algorithms, each corresponding to a different data type.

[0029] The data type corresponding to the target deletion algorithm is the same as the data type of the data to be deleted; the data type corresponding to the target compression algorithm is the same as the data type of the data to be deleted.

[0030] In this embodiment, by setting multiple deletion algorithms applicable to various data types and multiple compression algorithms applicable to various data types for each type of deletion algorithm, and by selecting the data type corresponding to the target deletion algorithm as the data type of the data to be deleted, and selecting the data type corresponding to the target compression algorithm as the data type of the data to be deleted, the data processing efficiency can be improved.

[0031] Optionally, in the method described above, the feature values ​​to be deduplicated include N feature values, and the original data corresponding to the feature values ​​to be deduplicated includes N sets of original data; wherein, N is a positive integer.

[0032] The data to be deleted is processed according to the target deletion algorithm to obtain the data after deletion, including:

[0033] Based on the target deletion algorithm, delete N-1 sets of original data from the N sets of original data, and store a first index in the storage area of ​​the data domain where the N-1 sets of original data are located. The first index points to a set of original data that has not been deleted from the N sets of original data.

[0034] Based on the target deletion algorithm, the feature value corresponding to the N-1 groups of original data among the N feature values ​​is deleted, and the storage area in the deduplication domain where the feature value corresponding to the N-1 groups of original data is located is stored with a second index, the second index pointing to a feature value among the N feature values ​​that has not been deleted.

[0035] In this embodiment, an index is stored at the location of the deleted data, which can point to the corresponding copy, so that the corresponding data can be retrieved according to the index when data is retrieved later. The data here can include the original data and feature values.

[0036] Secondly, this application provides a data processing apparatus applied to a storage device, the storage device including a data domain and a deduplication domain, wherein multiple storage areas in the data domain are respectively used to store multiple sets of original data, and multiple storage areas in the deduplication domain are respectively used to store feature values ​​corresponding to the multiple sets of original data, the apparatus comprising:

[0037] The first determining module is used to determine the current load of the storage device and the number of feature values ​​to be deduplicated.

[0038] The second determining module is used to determine the target deletion algorithm and the target compression algorithm based on the current load and the quantity;

[0039] The first processing module is used to perform deletion processing on the data to be deleted according to the target deletion algorithm to obtain the data after deletion processing. The data to be deleted includes the feature value to be deleted and the original data corresponding to the feature value.

[0040] The second processing module is used to compress the deleted data according to the target compression algorithm to obtain the processed data.

[0041] Optionally, in the apparatus described above, the first determining module is specifically used for:

[0042] Within the deduplication domain, feature values ​​of multiple subtypes of the target data type are obtained;

[0043] Among the feature values ​​of the various subtypes, multiple target feature values ​​are determined;

[0044] The number of the plurality of target feature values ​​is determined as the number of feature values ​​to be deleted;

[0045] Obtain the operating parameters of the storage device, including the CPU utilization and / or the input / output utilization of the storage device;

[0046] The current load is determined based on the operating parameters.

[0047] Optionally, in the apparatus described above, the second determining module is specifically used for:

[0048] Determine the load level of the current load and the quantity level of the quantity, wherein the load level is a first load level and a second load level, and the quantity level is a first quantity level or a second quantity level, wherein the first load level is higher than the second load level, and the first quantity level is higher than the second quantity level;

[0049] Based on the load level and the quantity level, the target deletion algorithm and the target compression algorithm are determined in the algorithm library.

[0050] Optionally, in the apparatus described above, the second determining module is specifically used for:

[0051] If the load level is the first load level and the quantity level is the first quantity level, then the target deletion algorithm is determined to be the first deletion algorithm and the target compression algorithm is determined to be the first compression algorithm. The first deletion algorithm belongs to the deletion algorithm with the lowest resource consumption among the multiple deletion algorithms, and the first compression algorithm belongs to the compression algorithm with the lowest compression ratio among the multiple compression algorithms.

[0052] If the load level is a first load level and the quantity level is a second quantity level, or if the load level is a second load level and the quantity level is a first quantity level, then the target deletion algorithm is determined to be a second deletion algorithm, and the target compression algorithm is determined to be a second compression algorithm. The storage device resources consumed by the second deletion algorithm are higher than those consumed by the first deletion algorithm, and the compression ratio of the second compression algorithm is greater than that of the first compression algorithm.

[0053] If the load level is the second load level and the quantity level is the second quantity level, then the target deletion algorithm is determined to be the third deletion algorithm, and the target compression algorithm is determined to be the third compression algorithm. The storage device resources consumed by the third deletion algorithm are higher than those consumed by the second deletion algorithm, and the compression ratio of the third compression algorithm is greater than that of the second compression algorithm.

[0054] Optionally, in the apparatus described above, each type of deletion algorithm includes multiple deletion algorithms, each of which corresponds to a different data type; each type of compression algorithm includes multiple compression algorithms, each of which corresponds to a different data type.

[0055] The data type corresponding to the target deletion algorithm is the same as the data type of the data to be deleted; the data type corresponding to the target compression algorithm is the same as the data type of the data to be deleted.

[0056] Optionally, in the apparatus described above, the feature values ​​to be deduplicated include N feature values, and the original data corresponding to the feature values ​​to be deduplicated includes N sets of original data; wherein, N is a positive integer.

[0057] The first processing module is specifically used for:

[0058] Based on the target deletion algorithm, delete N-1 sets of original data from the N sets of original data, and store a first index in the storage area of ​​the data domain where the N-1 sets of original data are located. The first index points to a set of original data that has not been deleted from the N sets of original data.

[0059] Based on the target deletion algorithm, the feature value corresponding to the N-1 groups of original data among the N feature values ​​is deleted, and the storage area in the deduplication domain where the feature value corresponding to the N-1 groups of original data is located is stored with a second index, the second index pointing to a feature value among the N feature values ​​that has not been deleted.

[0060] The data processing device provided in this application embodiment has similar beneficial effects to the first aspect, and will not be described again here.

[0061] Thirdly, this application provides an electronic device, including: a processor, and a memory communicatively connected to the processor;

[0062] The memory stores computer-executed instructions;

[0063] The processor executes computer execution instructions stored in the memory to implement the method as described in the first aspect.

[0064] Fourthly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a computer, are used to implement the method described in the first aspect.

[0065] The computer-readable storage medium provided in this application embodiment can execute the technical solutions in the above method embodiments, and its beneficial effects are similar, so they will not be described again here.

[0066] Fifthly, this application provides a computer program product, including a computer program, which, when executed by a computer, is used to implement the method described in the first aspect.

[0067] The computer program product provided in this application embodiment can execute the technical solutions in the above method embodiments, and its beneficial effects are similar, so they will not be described again here.

[0068] The data processing method, apparatus, electronic device, storage medium, and program product provided in this application, in a deduplication task, allow the storage device to determine its current load and the number of features to be deduplicated, and to determine a target deletion algorithm and a target compression algorithm based on the current load and the number of features to be deduplicated. Then, the data to be deduplicated is processed according to the target deletion algorithm to obtain deduplicated data. The data to be deduplicated includes the features to be deduplicated and the original data corresponding to the features. Finally, the deduplicated data is compressed according to the target compression algorithm to obtain processed data. This application adaptively selects appropriate deletion and compression algorithms based on the current load of the storage device and the number of features to be deduplicated, thus ensuring the stability of the storage device under high load and effectively guaranteeing the deduplication rate under low load. Attached Figure Description

[0069] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0070] Figure 1 A schematic diagram illustrating the structure of a storage device as an example of this application and the process of a deduplication task based on this structure;

[0071] Figure 2 A flowchart illustrating a data processing method provided in an embodiment of this application;

[0072] Figure 3 A flowchart illustrating another data processing method provided in an embodiment of this application;

[0073] Figure 4 A flowchart illustrating another data processing method provided in an embodiment of this application;

[0074] Figure 5 This is a schematic diagram illustrating how the target deletion algorithm and target compression algorithm are determined based on load level and quantity level, as an example of this application.

[0075] Figure 6 A flowchart illustrating another data processing method provided in an embodiment of this application;

[0076] Figure 7 This is a schematic diagram illustrating how the first index is stored after data deletion, as an example of this application.

[0077] Figure 8 A flowchart illustrating another data processing method provided in an embodiment of this application;

[0078] Figure 9 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application;

[0079] Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0080] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0081] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0082] Since a large amount of duplicate data exists in the stored data, deleting duplicate data can effectively reduce the data volume. Different deleting methods have different impacts on the overall stability of the storage device and the final data reduction effect. In the following text, the deletion of duplicate data will be referred to as deleting.

[0083] In related technologies, deduplication can be divided into real-time deduplication and offline deduplication. Real-time deduplication refers to deduplication performed when data is written to the storage device. Its advantage is that deduplication can be completed before the data is written to disk, reducing the number of disk writes. However, it has a significant impact on the input / output (I / O) stability of the storage device, and the deduplication rate is relatively low. Offline deduplication refers to deduplication performed offline after the original data has been written to disk, when certain conditions are met. Because it is executed in the background, it has less impact on the I / O stability of the storage device, and algorithms with higher deduplication rates can be selected.

[0084] In addition to data deletion, deduplication often requires compressing the deleted data. Both of these steps consume a lot of computing resources, including central processing unit (CPU) resources.

[0085] The following are deduplication algorithms in related technologies:

[0086] (1) Hybrid deduplication scheduling method: Deduplication is performed based on whether the business data type is latency-sensitive or bandwidth-sensitive; or deduplication is performed by setting the ratio of online deduplication to offline deduplication.

[0087] (2) Execution strategy of hierarchical compression: Set a threshold for the similarity of the data to be compressed. Data to be compressed that is higher than the threshold is compressed using differential compression, while data to be compressed that is lower than the threshold is compressed using another compression method, so as to make full use of the computing resources of the storage device.

[0088] (3) Execution strategy for multiple deduplication domains: The storage device is divided into multiple deduplication domains. An address can belong to different deduplication domains simultaneously. These domains have different priorities, and each priority corresponds to a different deduplication strategy. When data is written to disk, the corresponding deduplication strategy is determined based on the priority of the current deduplication domain, and deduplication is performed according to that strategy. By performing deduplication according to the deduplication strategies corresponding to different priorities, a higher deduplication rate can be achieved.

[0089] However, the factors considered in the above methods are relatively simple. For example, methods (1) and (2) can reduce the resource consumption of storage devices, but the deduplication rate is low; method (3) can achieve a higher deduplication rate, but it consumes more resources of storage devices, and the storage devices will lag when they are under high load.

[0090] However, storage devices do not always operate under high load; they also experience periods of low load. Furthermore, the resource consumption of storage devices varies depending on the amount of data to be deduplicated. Therefore, determining how to perform deduplication based on the actual operating conditions of the storage device and the amount of data to be deduplicated, in order to achieve a high deduplication rate or high stability of the storage device, is a problem that needs to be solved.

[0091] Therefore, this application provides a data processing method that, in a single deduplication task, adaptively selects appropriate deletion and compression algorithms based on the current load of the storage device and the number of feature values ​​to be deduplicated, so as to ensure the stability of the storage device under high load and effectively guarantee the deduplication rate under low load.

[0092] It is understood that, in the embodiments of this application, the stability of the storage device can be understood as whether the storage device experiences phenomena such as lag. For example, if the storage device consumes a lot of resources or the bandwidth usage of the storage device is unstable during data transmission, it will lead to a decrease in the stability of the storage device.

[0093] For ease of understanding, this application is approved. Figure 1 The process of a single deleting task involved in this application is described. Figure 1 The deduplication process can be understood as an offline deduplication process.

[0094] Figure 1 The diagram illustrates the structure of a storage device as an example in this application and the process of a single deduplication task based on this structure. Figure 1 As shown, the storage device 10 includes a controller 101, a data field 102, and a deduplication field 103. The data field 102 is used to store the original data, and the deduplication field 103 is used to store the feature values ​​corresponding to the total original data.

[0095] Specifically, data domain 102 includes multiple storage areas, for example, each storage area may be 8KB (8 kilobytes) in size, and the multiple storage areas are used to store multiple sets of raw data in the total raw data.

[0096] In step A, when writing data, storage device 10 can store data in basic units of storage areas, with each storage area storing 8KB of data. For example, the total raw data may include one or more different data types such as images, audio, video, and text.

[0097] In step B, controller 101 can calculate a feature value for the raw data in each storage area and store the feature value in deduplication field 103. It can be understood that deduplication field 103 is also divided into multiple storage areas, each used to store different feature values. For any given storage area's raw data, the feature value of that raw data can be compared with the feature values ​​of the raw data in other storage areas to describe the similarity between the two sets of raw data. For example, if the raw data stored in two storage areas are similar or identical, then the feature values ​​calculated for the data in these two storage areas will also be the same.

[0098] In this embodiment of the application, the feature value can also be called a fingerprint, which can be understood as a mapping of the original data of a storage area.

[0099] In one possible implementation, when storing feature values ​​in the deduplication field 103, the controller 101 can classify these feature values. That is, the deduplication field 103 stores feature values ​​of multiple data types, each data type including multiple subtypes. For the original data stored in the data field 102, which is image data, including images of different subtypes such as cats, dogs, and trees, the controller 101 can classify the feature values ​​based on subtype. For example, it can group the feature values ​​corresponding to the original data of the subtype "cat" into one category, the feature values ​​corresponding to the original data of the subtype "dog" into another category, and the feature values ​​corresponding to the original data of the subtype "tree" into yet another category, thus obtaining feature values ​​of multiple subtypes within the image data type.

[0100] For the raw data stored in data domain 102, which is video data, including videos of different subtypes such as sports, entertainment, and news, controller 101 can classify feature values ​​based on subtype. For example, it can classify the feature values ​​corresponding to the raw data of the sports subtype into one category, the feature values ​​corresponding to the raw images of the entertainment subtype into another category, and the feature values ​​corresponding to the raw images of the news subtype into another category, thus obtaining feature values ​​for multiple subtypes of the video data type.

[0101] For the raw data stored in data domain 102, which is audio-type data, including videos of different subtypes such as music, speech, and environment, controller 101 can classify feature values ​​based on subtype. For example, it can classify the feature values ​​corresponding to the raw data of the music subtype into one category, the feature values ​​corresponding to the raw images of the speech subtype into another category, and the feature values ​​corresponding to the raw images of the environment subtype into another category, thus obtaining feature values ​​for multiple subtypes of the audio data type.

[0102] To facilitate understanding, Table 1 below provides an example of the above classifications. It should be noted that the classifications in Table 1 are merely examples. For instance, video subcategories can be further subdivided into smaller categories such as sports, entertainment, and news. For example, sports can be divided into running, gymnastics, etc. This application does not impose any restrictions on the form of data classification.

[0103] In step C, controller 101 can obtain feature values ​​of various subtypes from deduplication domain 103, for example, so that feature values ​​of these subtypes can be further analyzed to determine feature values ​​that can be deduplicated.

[0104] Stages D, E, and F constitute the data processing phases of a deduplication task. In stage D, the feature values ​​of various subtypes obtained from the deduplication domain 103 derived in stage C can be analyzed and filtered to select the feature values ​​that can be deduplicated.

[0105] For example, for any subtype, the same feature values ​​among the feature values ​​of that subtype can be identified as feature values ​​that can be deduplicated.

[0106] Step E can perform deduplication on the original data corresponding to the feature values ​​selected in Step D that can be deduplicated.

[0107] Step F is executed after step E. Step F can delete features that can be deleted.

[0108] This completes the process of one deleting task.

[0109] Table 1

[0110]

[0111] In the embodiments of this application, deduplication refers to retaining a copy of the original data in the same storage area among the original data in multiple storage areas, deleting the original data in the remaining storage areas, and storing an index in the storage area after deleting the original data, which can point to a unique copy.

[0112] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0113] Figure 2 This is a flowchart illustrating a data processing method provided in an embodiment of this application. This method can be applied to storage devices, specifically to applications such as... Figure 1 The controller 101 in the storage device 10 shown. For example... Figure 2As shown, the method includes the following steps:

[0114] S201. Determine the current load of the storage device and the number of features to be deduplicated.

[0115] When performing a deduplication task, the storage device can determine the current load and the number of features to be deduplicated.

[0116] For example, the storage device may determine its current load by determining the current load based on the storage device's resource operating parameters, which may include, for example, CPU utilization, I / O utilization, etc.

[0117] In one possible implementation, the current load can be represented by a load level. A higher load level indicates a greater current load on the storage device, while a lower load level indicates a smaller current load on the storage device.

[0118] In one possible implementation, the number of feature values ​​to be deduplicated can be represented by a magnitude. The higher the magnitude, the more feature values ​​to be deduplicated, and the lower the magnitude, the fewer feature values ​​to be deduplicated.

[0119] For example, the storage device may determine the number of feature values ​​to be deleted by identifying target feature values ​​that can be deleted from among multiple types of feature values ​​in the deletion domain, and using the number of such target feature values ​​as the number of feature values ​​to be deleted.

[0120] S202. Determine the target deletion algorithm and target compression algorithm based on the current load and the number of feature values ​​to be deleted.

[0121] After determining the current load and the features to be deduplicated, the storage device can determine the target deletion algorithm and the target compression algorithm based on the current load and the number of features to be deduplicated.

[0122] For example, the storage device may determine the target deletion algorithm and the target compression algorithm from an algorithm library, which includes a variety of deletion algorithms and a variety of compression algorithms.

[0123] It is understandable that the target deletion algorithm and target compression algorithm can be different depending on the different loads of the storage device and the different numbers of features to be deduplicated. These target deletion and target compression algorithms can meet the current load conditions of the storage device and the needs of the deduplication task.

[0124] S203. According to the target deletion algorithm, the data to be deleted is processed to obtain the data after deletion. The data to be deleted includes the feature value to be deleted and the original data corresponding to the feature value.

[0125] After determining the target deletion algorithm and the target compression algorithm, the storage device can perform deletion processing on the data to be deleted according to the target deletion algorithm. The data to be deleted includes the feature value to be deleted and the original data corresponding to that feature value.

[0126] For example, the storage device can delete the original data corresponding to the feature value to be deleted, and then delete the feature value to be deleted. Figure 1 The links F and E in the process.

[0127] S204. Based on the target compression algorithm, compress the data after deletion to obtain the processed data.

[0128] After deleting the data to be deleted according to the target compression algorithm, the storage device can compress the deleted data according to the target compression algorithm to obtain the processed data.

[0129] In this embodiment, during a deduplication task, the storage device can determine its current load and the number of feature values ​​to be deduplicated. Based on these factors, it determines the target deletion algorithm and the target compression algorithm. Then, the data to be deduplicated is processed according to the target deletion algorithm, resulting in deduplicated data. The data to be deduplicated includes the feature values ​​to be deduplicated and the corresponding original data. Finally, the deduplicated data is compressed according to the target compression algorithm, resulting in processed data. This application adaptively selects appropriate deletion and compression algorithms based on the storage device's current load and the number of feature values ​​to be deduplicated, ensuring storage device stability under high load and effectively guaranteeing the deduplication rate under low load.

[0130] The following section details how storage devices determine their current load and the number of features to be deduplicated.

[0131] Figure 3 This is a flowchart illustrating another data processing method provided in an embodiment of this application. This method can be applied to storage devices, specifically to applications such as... Figure 1 The controller 101 in the storage device 10 shown. For example... Figure 3 As shown, the method includes the following steps:

[0132] S301. In the deduplication field, obtain the feature values ​​of multiple subtypes in the target data type.

[0133] The target data type can be images, videos, or audio, etc. The feature values ​​for various subtypes can be the feature values ​​obtained by classifying the feature values ​​of the original data corresponding to the target data type when the controller stores feature values ​​in the deduplication domain. For details, please refer to... Figure 1 Corresponding classification examples. For each subtype, the feature values ​​of that subtype can include one or more feature values.

[0134] S302. Among the feature values ​​of various subtypes, determine multiple target feature values.

[0135] Storage devices can determine multiple target feature values ​​from feature values ​​of various subtypes.

[0136] For example, among the feature values ​​of multiple subtypes, for any feature value of a subclass, all feature values ​​of that subclass can be compared, and feature values ​​with the same value (or with a value difference of less than a set threshold) can be determined as the target feature value.

[0137] S303. The number of multiple target feature values ​​is determined as the number of feature values ​​to be deleted.

[0138] S304. Obtain the operating parameters of the storage device, including the CPU utilization and / or I / O utilization of the storage device.

[0139] Storage devices can obtain operating parameters such as current CPU utilization and I / O utilization. CPU utilization refers to the percentage of time the CPU spends executing tasks per unit of time, reflecting the CPU's workload. I / O utilization refers to the load intensity of I / O operations processed by the storage device per unit of time, reflecting data throughput efficiency.

[0140] It is understood that the operating parameters of a storage device may also include other parameters that can be used to measure the load on the storage device, and this application does not impose any restrictions on this.

[0141] S305. Determine the current load based on the operating parameters.

[0142] After obtaining the operating parameters, the storage device can determine the current load based on CPU utilization and / or I / O utilization.

[0143] In one possible implementation, the storage device can determine the current load level, which is either a first load level or a second load level, wherein the first load level is higher than the second load level. The first load level may indicate that the storage device is currently under high load, and the second load level may indicate that the storage device is currently under low load.

[0144] Specifically, the storage device can determine its current load in the following ways:

[0145] Method 1

[0146] The storage device can determine the current load based on the I / O utilization rate. If the I / O utilization rate is greater than a first threshold, the current load can be determined as the first load level. If the I / O utilization rate is less than or equal to the first threshold, the current load can be determined as the second load level.

[0147] Method 2

[0148] The storage device can determine the current load based on CPU utilization. If the CPU utilization is greater than a second threshold, the current load can be determined as the first load level. If the CPU utilization is less than or equal to the second threshold, the current load can be determined as the second load level.

[0149] Method 3

[0150] The storage device can determine the current load based on CPU utilization and I / O utilization. If the sum of CPU utilization and I / O utilization is greater than a second threshold, the current load can be determined as the first load level. If the sum of CPU utilization and I / O utilization is less than or equal to the second threshold, the current load can be determined as the second load level.

[0151] Method 4

[0152] Generally, in storage devices, when I / O utilization consistently exceeds a certain threshold (e.g., 70%), the response time of the storage device increases significantly, potentially leading to data read / write queue backlog and directly impacting the throughput and stability of the storage system. In contrast, CPU utilization has a smaller impact on the stability of storage devices compared to I / O utilization.

[0153] Building upon Method 3, storage devices can set different scaling factors for CPU utilization and I / O utilization. For example, the I / O utilization scaling factor can be set to 70%, and the CPU utilization scaling factor can be set to 30%. When determining the current load, the storage device can determine the current load by summing the product of CPU utilization and scaling factor 30% and the product of I / O utilization and scaling factor 70%.

[0154] It is understood that the above method of determining the current load is only an example, and the current load can also be determined by other operating parameters and other methods. This application does not limit this.

[0155] exist Figure 3 In a corresponding embodiment, the feature values ​​to be deleted can be determined based on the feature values ​​of multiple subclasses. Determining the current load based on CPU utilization and I / O utilization allows for the selection of appropriate deletion and compression algorithms based on the current load and the number of feature values ​​to be deleted.

[0156] Below, through Figure 4This section provides a detailed explanation of how storage devices determine the target deletion algorithm and the target compression algorithm based on the current load and the number of features to be deduplicated.

[0157] Figure 4 This is a flowchart illustrating another data processing method provided in an embodiment of this application. This method can be applied to storage devices, specifically to applications such as... Figure 1 The controller 101 in the storage device 10 shown. For example... Figure 4 As shown, the method includes the following steps:

[0158] S401. Determine the load level of the current load and the number level of the feature values ​​to be deduplicated.

[0159] The storage device determines the load level of the current load, which can be referred to in the above embodiments and will not be repeated here.

[0160] The storage device can determine the quantity level based on the number of feature values ​​to be deduplicated. For example, if the number of feature values ​​to be deduplicated is greater than a first quantity threshold, the quantity level can be determined as the first quantity level. If the number of feature values ​​to be deduplicated is less than or equal to the first quantity threshold, the quantity level can be determined as the second quantity level. The first quantity level is higher than the second quantity level.

[0161] The first order of magnitude indicates that the storage device has a large number of features to be deduplicated, meaning that the storage device currently has a large number of features to be deduplicated. The second order of magnitude indicates that the storage device has a small number of features to be deduplicated, meaning that the storage device currently has a small number of features to be deduplicated.

[0162] S402. Based on the load level and quantity level, determine the target deletion algorithm and target compression algorithm in the algorithm library.

[0163] Storage devices can determine target deletion and compression algorithms from an algorithm library based on load and quantity levels. The algorithm library includes multiple types of deletion and compression algorithms. The deletion algorithms consume different amounts of storage resources, and the compression algorithms have different compression ratios. Each type of deletion algorithm includes multiple deletion algorithms, and each type of compression algorithm includes multiple compression algorithms.

[0164] In one possible implementation, deletion algorithms can be categorized based on resource consumption. These categorizations include low-resource-consumption, medium-resource-consumption, and high-resource-consumption algorithms, with each category containing multiple algorithms. It's understood that "resources" here refers to storage device resources, such as computing resources (i.e., CPU resources).

[0165] It should be noted that the three types of resource consumption—low, medium, and high—can be determined based on the amount of storage device resources consumed. For example, thresholds 1 and 2 can be set, with threshold 1 being less than threshold 2. Taking a certain deletion algorithm as an example, if the amount of storage device resources consumed by the deletion algorithm during operation is less than or equal to threshold 1, then the deletion algorithm can be determined as a low-resource-consumption type. If the amount of storage device resources consumed by the deletion algorithm during operation is greater than threshold 1 but less than threshold 2, then the deletion algorithm can be determined as a medium-resource-consumption type. If the amount of storage device resources consumed by the deletion algorithm during operation is greater than or equal to threshold 2, then the deletion algorithm can be determined as a high-resource-consumption type.

[0166] In one possible implementation, compression algorithms can be classified according to their compression ratio. The multiple compression algorithms include three types: low compression ratio, medium compression ratio, and high compression ratio. Each type of compression algorithm includes multiple compression algorithms.

[0167] For example, the multi-class deletion algorithm and multi-class compression algorithm in the algorithm library are shown in Table 2.

[0168] Storage devices can determine the target deletion algorithm and target compression algorithm from the algorithm library shown in Table 2, based on the load level and quantity level.

[0169] Table 2

[0170]

[0171] In one possible implementation, the storage device can determine the target deletion algorithm and the target compression algorithm in the following way:

[0172] If the load level is the first load level and the quantity level is the first quantity level, then the target deletion algorithm is determined to be the first deletion algorithm, and the target compression algorithm is determined to be the first compression algorithm. The first deletion algorithm belongs to the category of deletion algorithms with the lowest resource consumption among multiple deletion algorithms, and the first compression algorithm belongs to the category of compression algorithms with the lowest compression ratio among multiple compression algorithms. Taking Table 2 as an example, the first deletion algorithm is any one of deletion algorithm a1, deletion algorithm a2, ..., and the first compression algorithm is any one of compression algorithm d1, compression algorithm d2, ...

[0173] It is understandable that a load level of 1 and a quantity level of 1 indicate that the current load on the storage device is high, and the deduplication task requires a large number of feature values ​​and original data to be deduplicated. This deduplication task puts the storage device under high pressure. At this time, selecting the first deletion algorithm and the first compression algorithm, that is, the deletion algorithm with low resource consumption and the compression algorithm with low compression ratio, can ensure the stability of the storage device, such as ensuring the stability of the storage device's bandwidth.

[0174] If the load level is the first load level and the quantity level is the second quantity level, or vice versa, then the target deletion algorithm is determined to be the second deletion algorithm, and the target compression algorithm is determined to be the second compression algorithm. The storage device resources consumed by the second deletion algorithm are higher than those consumed by the first deletion algorithm, and the compression ratio of the second compression algorithm is higher than that of the first compression algorithm. Taking Table 2 as an example, the second deletion algorithm is any one of deletion algorithm b1, deletion algorithm b2, ..., and the first compression algorithm is any one of compression algorithm e1, compression algorithm e2, ...

[0175] It is understandable that a load level of 1 and a quantity level of 2 indicate that the current load on the storage device is high. However, the number of features and original data to be deleted in this deduplication task is relatively small. At this time, a balanced deletion algorithm and compression algorithm can be compared. That is, the second deletion algorithm and the second compression algorithm can be selected. This means that the deletion algorithm with moderate resource consumption and the compression algorithm with moderate compression rate can ensure both the stability of the storage device and the high deduplication rate.

[0176] Alternatively, if the load level is the second load level and the quantity level is the first quantity level, it indicates that the current load of the storage device is low, but the deduplication task requires a large number of feature values ​​and original data to be deduplicated. In this case, the selection of deletion and compression algorithms can be similar to that of the first load level and the second quantity level, which will not be elaborated here.

[0177] If the load level is the second load level and the quantity level is the second quantity level, then the target deletion algorithm is determined to be the third deletion algorithm, and the target compression algorithm is determined to be the third compression algorithm. The third deletion algorithm consumes more storage device resources than the second deletion algorithm, and the compression ratio of the third compression algorithm is greater than that of the second compression algorithm. Taking Table 2 as an example, the third deletion algorithm is any one of deletion algorithm c1, deletion algorithm c2, ..., and the first compression algorithm is any one of compression algorithm f1, compression algorithm f2, ...

[0178] It is understandable that a load level of the second load level and a quantity level of the second quantity level indicate that the current load on the storage device is low, and the number of feature values ​​and original data to be deleted in this deduplication task is small. At this time, a deletion algorithm with higher resource consumption and a compression algorithm with a high compression ratio can be used, that is, the third deletion algorithm and the third compression algorithm can be selected to obtain a higher deduplication rate.

[0179] In one possible implementation, each type of deletion algorithm includes multiple deletion algorithms, each corresponding to a different data type; similarly, each type of compression algorithm includes multiple deletion algorithms, each corresponding to a different data type. The data type corresponding to the target deletion algorithm is the same as the data type of the data to be deleted; the data type corresponding to the target compression algorithm is also the same as the data type of the data to be deleted.

[0180] Taking data types including images, videos, and audio as examples, the algorithm library can be shown in Table 3.

[0181] The data to be deleted is categorized as image, load level as first load level, quantity level as first quantity level, target deletion algorithm as image deletion algorithm a1, and target compression algorithm as image compression algorithm d1.

[0182] By setting multiple deletion algorithms applicable to various data types and multiple compression algorithms applicable to various data types for each type of deletion algorithm, and ensuring that the data type corresponding to the selected target deletion algorithm is the same as the data type of the data to be deleted, and the data type corresponding to the selected target compression algorithm is the same as the data type of the data to be deleted, the efficiency of data processing can be improved.

[0183] Table 3

[0184]

[0185] For ease of understanding, the above describes determining the target deletion algorithm and target compression algorithm based on load level and quantity level. For details, please refer to [reference needed]. Figure 5 .

[0186] exist Figure 4 In the corresponding embodiment, based on the current load level and the number of feature values ​​to be deduplicated, the target deletion algorithm and target compression algorithm are determined from the multiple deletion algorithms and multiple compression algorithms in the algorithm library, so that the algorithm used can meet the needs of the current actual situation.

[0187] The following section provides a detailed explanation of how storage devices delete features to be deleted and the original data corresponding to those features.

[0188] Figure 6This is a flowchart illustrating another data processing method provided in an embodiment of this application. This method can be applied to storage devices, specifically to applications such as... Figure 1 The controller 101 in the storage device 10 shown. For example... Figure 6 As shown, the method includes the following steps:

[0189] S601. Based on the target deletion algorithm, delete N-1 sets of original data from N sets of original data, and store a first index in the storage area of ​​the data domain where the N-1 sets of original data are located. The first index points to a set of original data that has not been deleted from the N sets of original data.

[0190] The storage device can delete N-1 sets of original data out of N sets of original data based on a target deletion algorithm, and store a first index in the storage area of ​​the data domain where the N-1 sets of original data are located. For example, Figure 7 This is a schematic diagram illustrating how the first index is stored after data deletion, as exemplified in this application. (See attached diagram.) Figure 7 Where N is a positive integer.

[0191] It can be understood that the set of original data that has not been deleted from the N sets of original data is a unique copy of the N sets of original data.

[0192] For example, the first index may be the address of the storage area where the undelete set of original data in the N sets of original data is located, or it may be an identifier corresponding to that address, etc., and this application does not limit it in this way. When the original data group a in the N-1 sets of original data is accessed, the undelete set of original data in the N sets of original data can be accessed based on the first index, and that original data can be used as the data of the original data group a.

[0193] S602. Based on the target deletion algorithm, delete the feature values ​​corresponding to N-1 sets of original data among N feature values, and store the second index in the storage area of ​​the deleting domain where the feature values ​​corresponding to N-1 sets of original data are located. The second index points to one of the N feature values ​​that has not been deleted.

[0194] After deleting N-1 sets of original data from N sets of original data, the storage device can delete the feature values ​​corresponding to N-1 sets of original data from N feature values ​​based on the target deletion algorithm, and store a second index in the storage area of ​​the deduplication domain where the feature values ​​corresponding to N-1 sets of original data are located. The second index points to one of the feature values ​​that has not been deleted from the N feature values.

[0195] It can be understood that the one feature value that is not deleted out of N feature values ​​is a unique copy of the N feature values.

[0196] For example, the second index can be the address of the storage area where one of the N feature values ​​that has not been deleted is located, or it can be the identifier corresponding to that address, etc., and this application does not limit it in this way. When feature value b in the feature values ​​corresponding to the N-1 sets of original data is accessed, one of the N feature values ​​that has not been deleted can be accessed based on the second index, and that feature value can be used as feature value b.

[0197] In this embodiment, an index is stored at the location of the deleted data, which can point to the corresponding copy, so that the corresponding data can be retrieved according to the index when data is retrieved later. The data here can include the original data and feature values.

[0198] The following describes another data processing method provided in the embodiments of this application.

[0199] Figure 8 This is a flowchart illustrating another data processing method provided in an embodiment of this application. This method can be applied to storage devices, specifically to applications such as... Figure 1 The controller 101 in the storage device 10 shown. For example... Figure 8 As shown, the method includes the following steps:

[0200] S801. In the deduplication field, obtain the feature values ​​of multiple subtypes in the target data type.

[0201] S802. Among the feature values ​​of various subtypes, determine multiple target feature values.

[0202] S803. The number of multiple target feature values ​​is determined as the number of feature values ​​to be deleted.

[0203] S804. Obtain the operating parameters of the storage device, including the CPU utilization and / or I / O utilization of the storage device.

[0204] S805. Determine the current load based on the operating parameters.

[0205] S806. Determine the load level of the current load and the number level of the feature values ​​to be deduplicated.

[0206] S807. Based on the load level and quantity level, determine the target deletion algorithm and target compression algorithm in the algorithm library.

[0207] S808. Based on the target deletion algorithm, delete N-1 sets of original data from N sets of original data, and store the first index in the storage area of ​​the data domain where the N-1 sets of original data are located.

[0208] The first index points to the set of original data that was not deleted from the N sets of original data.

[0209] S809. Based on the target deletion algorithm, delete the feature values ​​corresponding to N-1 sets of original data from N feature values, and store the second index in the storage area of ​​the deduplication domain where the feature values ​​corresponding to N-1 sets of original data are located.

[0210] The second index points to one of the N feature values ​​that has not been deleted.

[0211] S810. Based on the target compression algorithm, compress the data after deletion to obtain the processed data.

[0212] The specific implementation method and technical effects in this embodiment are similar to those in the above embodiments, and will not be repeated here.

[0213] Figure 9 This is a schematic diagram of a data processing apparatus provided in an embodiment of this application. The apparatus is applied to a storage device, such as... Figure 9 As shown, the device 90 includes: a first determining module 901, a second determining module 902, a first processing module 903, and a second processing module 904.

[0214] The first determining module 901 is used to determine the current load of the storage device and the number of feature values ​​to be deduplicated.

[0215] The second determining module 902 is used to determine the target deletion algorithm and the target compression algorithm based on the current load and quantity.

[0216] The first processing module 903 is used to perform deletion processing on the data to be deleted according to the target deletion algorithm, and obtain the data after deletion processing. The data to be deleted includes the feature value to be deleted and the original data corresponding to the feature value.

[0217] The second processing module 904 is used to compress the deleted data according to the target compression algorithm to obtain the processed data.

[0218] Optionally, as in the above device 90, the first determining module 901 is specifically used for:

[0219] In the deduplication field, retrieve the feature values ​​of multiple subtypes in the target data type.

[0220] Among the eigenvalues ​​of various subtypes, multiple target eigenvalues ​​are determined.

[0221] The number of multiple target feature values ​​is determined as the number of feature values ​​to be deleted.

[0222] Obtain the operating parameters of the storage device, including the CPU utilization and / or I / O utilization of the storage device.

[0223] Determine the current load based on the operating parameters.

[0224] Optionally, as in the above device 90, the second determining module 902 is specifically used for:

[0225] Determine the load level and quantity level of the current load. The load level is designated as the first load level and the second load level, and the quantity level is designated as either the first quantity level or the second quantity level. The first load level is higher than the second load level, and the first quantity level is higher than the second quantity level.

[0226] Based on the load level and quantity level, target deletion algorithms and target compression algorithms are determined from the algorithm library.

[0227] Optionally, as in the above device 90, the second determining module 902 is specifically used for:

[0228] If the load level is the first load level and the quantity level is the first quantity level, then the target deletion algorithm is determined to be the first deletion algorithm, and the target compression algorithm is determined to be the first compression algorithm. The first deletion algorithm belongs to the deletion algorithm with the lowest resource consumption among multiple deletion algorithms, and the first compression algorithm belongs to the compression algorithm with the lowest compression ratio among multiple compression algorithms.

[0229] If the load level is the first load level and the quantity level is the second quantity level, or if the load level is the second load level and the quantity level is the first quantity level, then the target deletion algorithm is determined to be the second deletion algorithm, and the target compression algorithm is determined to be the second compression algorithm. The storage device resources consumed by the second deletion algorithm are higher than those consumed by the first deletion algorithm, and the compression ratio of the second compression algorithm is greater than that of the first compression algorithm.

[0230] If the load level is the second load level and the quantity level is the second quantity level, then the target deletion algorithm is determined to be the third deletion algorithm, and the target compression algorithm is determined to be the third compression algorithm. The storage device resources consumed by the third deletion algorithm are higher than those consumed by the second deletion algorithm, and the compression ratio of the third compression algorithm is higher than that of the second compression algorithm.

[0231] Optionally, as in the above device 90, each type of deletion algorithm includes multiple deletion algorithms, each corresponding to a different data type; each type of compression algorithm includes multiple compression algorithms, each corresponding to a different data type.

[0232] The data type corresponding to the target deletion algorithm is the same as the data type of the data to be deleted; the data type corresponding to the target compression algorithm is the same as the data type of the data to be deleted.

[0233] Optionally, as in the above device 90, the feature values ​​to be deduplicated include N feature values, and the original data corresponding to the feature values ​​to be deduplicated includes N sets of original data; where N is a positive integer.

[0234] The first processing module 903 is specifically used for:

[0235] Based on the target deletion algorithm, delete N-1 sets of original data from N sets of original data, and store the first index in the storage area of ​​the data domain where the N-1 sets of original data are located. The first index points to the set of original data that has not been deleted from the N sets of original data.

[0236] Based on the target deletion algorithm, delete the feature values ​​corresponding to N-1 sets of original data from N feature values, and store the second index in the storage area of ​​the deleting domain where the feature values ​​corresponding to N-1 sets of original data are located. The second index points to one of the N feature values ​​that has not been deleted.

[0237] The apparatus in this embodiment can be used to execute the technical solutions of the above method embodiments. The specific implementation methods and technical effects are similar, and will not be described again here.

[0238] Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application, such as... Figure 10 As shown, the electronic device 100 may include at least one processor 1001 and a memory 1002.

[0239] The memory 1002 is used to store programs. Specifically, the program may include program code, which includes computer-executable instructions.

[0240] The memory 1002 may include random access memory (RAM) and may also include non-volatile memory, such as at least one disk storage device.

[0241] The processor 1001 is used to execute computer execution instructions stored in the memory 1002 to implement the method described in the foregoing method embodiments. The processor 1001 may be a CPU, an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application.

[0242] Optionally, the electronic device 100 may also include a communication interface 1003. In specific implementations, if the communication interface 1003, memory 1002, and processor 1001 are implemented independently, they can be interconnected via a bus to complete communication. The bus can be an industry-standard architecture (ISA) bus, a peripheral component (PCI) bus, or an extended industry standard architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc., but this does not imply that there is only one bus or one type of bus.

[0243] Optionally, in a specific implementation, if the communication interface 1003, memory 1002 and processor 1001 are integrated on a single chip, then the communication interface 1003, memory 1002 and processor 1001 can communicate through an internal interface.

[0244] Electronic device 100 can be a storage device.

[0245] The electronic device in this embodiment can be used to execute the technical solutions of the above method embodiments. The specific implementation methods and technical effects are similar, and will not be repeated here.

[0246] This application provides a computer-readable storage medium, which may include various media capable of storing computer-executable instructions, such as USB flash drives, portable hard drives, read-only memory (ROM), RAM, disks, or optical discs. Specifically, the computer-readable storage medium stores computer-executable instructions, which, when executed by a computer, cause the technical solution shown in the above method embodiment to be executed. The specific implementation and technical effects are similar and will not be repeated here.

[0247] This application provides a computer program product, including a computer program. When the computer program is executed by a computer, the technical solution shown in the above method embodiment is executed. The specific implementation method and technical effect are similar, and will not be repeated here.

[0248] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this application.

[0249] It should be further noted that although the steps in the flowchart are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0250] It should be understood that the above-described device embodiments are merely illustrative, and the device of this application can also be implemented in other ways. For example, the division of units / modules in the above embodiments is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple units, modules, or components may be combined, or integrated into another system, or some features may be ignored or not executed.

[0251] Furthermore, unless otherwise specified, the functional units / modules in the various embodiments of this application can be integrated into one unit / module, or each unit / module can exist physically separately, or two or more units / modules can be integrated together. The integrated units / modules described above can be implemented in hardware or as software program modules.

[0252] When an integrated unit / module is implemented in hardware, the hardware can be digital circuits, analog circuits, etc. The physical implementation of the hardware structure includes, but is not limited to, transistors, memristors, etc. Unless otherwise specified, the processor can be any suitable hardware processor, such as a CPU, graphics processing unit (GPU), field-programmable gate array (FPGA), digital signal processing (DSP) chip, and application-specific integrated circuit (ASIC), etc. Unless otherwise specified, the storage unit can be any suitable magnetic or magneto-optical storage medium, such as resistive random access memory (RRAM), dynamic random access memory (DRAM), static random-access memory (SRAM), enhanced dynamic random access memory (EDRAM), high-bandwidth memory (HBM), hybrid memory cube (HMC), etc.

[0253] If the integrated unit / module is implemented as a software program module and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned memory includes various media capable of storing program code, such as USB flash drives, ROM, RAM, portable hard drives, magnetic disks, or optical disks.

[0254] In the above embodiments, the descriptions of each embodiment have their own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification.

[0255] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0256] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A data processing method, characterized in that, The method is applied to a storage device, which includes a data domain and a deduplication domain. Multiple storage areas in the data domain are used to store multiple sets of original data, and multiple storage areas in the deduplication domain are used to store feature values ​​corresponding to the multiple sets of original data. The method includes: Determine the current load of the storage device and the number of feature values ​​to be deduplicated; Based on the current load and the quantity, determine the target deletion algorithm and the target compression algorithm; The data to be deleted is processed according to the target deletion algorithm to obtain the data after deletion. The data to be deleted includes the feature value to be deleted and the original data corresponding to the feature value. The data after deletion is compressed according to the target compression algorithm to obtain the processed data.

2. The method according to claim 1, characterized in that, The deduplication field includes feature values ​​of various data types, and each data type includes multiple subtypes; Determining the current load of the storage device and the number of features to be deduplicated includes: Within the deduplication domain, feature values ​​of multiple subtypes of the target data type are obtained; Among the feature values ​​of the various subtypes, multiple target feature values ​​are determined; The number of the plurality of target feature values ​​is determined as the number of feature values ​​to be deleted; Obtain the operating parameters of the storage device, including the CPU utilization and / or the input / output utilization of the storage device; The current load is determined based on the operating parameters.

3. The method according to claim 1, characterized in that, Based on the current load and the quantity, the target deletion algorithm and the target compression algorithm are determined, including: Determine the load level of the current load and the quantity level of the quantity, wherein the load level is a first load level and a second load level, and the quantity level is a first quantity level or a second quantity level, wherein the first load level is higher than the second load level, and the first quantity level is higher than the second quantity level; Based on the load level and the quantity level, the target deletion algorithm and the target compression algorithm are determined in the algorithm library.

4. The method according to claim 3, characterized in that, Based on the load level and the quantity level, the target deletion algorithm and the target compression algorithm are determined from the algorithm library, including: If the load level is the first load level and the quantity level is the first quantity level, then the target deletion algorithm is determined to be the first deletion algorithm and the target compression algorithm is determined to be the first compression algorithm. The first deletion algorithm belongs to the deletion algorithm with the lowest resource consumption among multiple deletion algorithms, and the first compression algorithm belongs to the compression algorithm with the lowest compression ratio among multiple compression algorithms. If the load level is a first load level and the quantity level is a second quantity level, or if the load level is a second load level and the quantity level is a first quantity level, then the target deletion algorithm is determined to be a second deletion algorithm, and the target compression algorithm is determined to be a second compression algorithm. The storage device resources consumed by the second deletion algorithm are higher than those consumed by the first deletion algorithm, and the compression ratio of the second compression algorithm is greater than that of the first compression algorithm. If the load level is the second load level and the quantity level is the second quantity level, then the target deletion algorithm is determined to be the third deletion algorithm, and the target compression algorithm is determined to be the third compression algorithm. The storage device resources consumed by the third deletion algorithm are higher than those consumed by the second deletion algorithm, and the compression ratio of the third compression algorithm is greater than that of the second compression algorithm.

5. The method according to claim 4, characterized in that, Each type of deletion algorithm includes multiple deletion algorithms, each corresponding to a different data type; each type of compression algorithm includes multiple compression algorithms, each corresponding to a different data type. The data type corresponding to the target deletion algorithm is the same as the data type of the data to be deleted; the data type corresponding to the target compression algorithm is the same as the data type of the data to be deleted.

6. The method according to any one of claims 1-5, characterized in that, The feature values ​​to be deleted include N feature values, and the original data corresponding to the feature values ​​to be deleted includes N sets of original data; where N is a positive integer; The data to be deleted is processed according to the target deletion algorithm to obtain the data after deletion, including: Based on the target deletion algorithm, delete N-1 sets of original data from the N sets of original data, and store a first index in the storage area of ​​the data domain where the N-1 sets of original data are located. The first index points to a set of original data that has not been deleted from the N sets of original data. Based on the target deletion algorithm, the feature value corresponding to the N-1 groups of original data among the N feature values ​​is deleted, and the storage area in the deduplication domain where the feature value corresponding to the N-1 groups of original data is located is stored with a second index, the second index pointing to a feature value among the N feature values ​​that has not been deleted.

7. A data processing apparatus, characterized in that, An apparatus for use in a storage device, the storage device comprising a data domain and a deduplication domain, wherein multiple storage areas in the data domain are respectively used to store multiple sets of original data, and multiple storage areas in the deduplication domain are respectively used to store feature values ​​corresponding to the multiple sets of original data, the apparatus comprising: The first determining module is used to determine the current load of the storage device and the number of feature values ​​to be deduplicated. The second determining module is used to determine the target deletion algorithm and the target compression algorithm based on the current load and the quantity; The first processing module is used to perform deletion processing on the data to be deleted according to the target deletion algorithm to obtain the data after deletion processing. The data to be deleted includes the feature value to be deleted and the original data corresponding to the feature value. The second processing module is used to compress the deleted data according to the target compression algorithm to obtain the processed data.

8. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1 to 6.

10. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method described in any one of claims 1-6.

Images