Hierarchical storage management system, duplicated data deletion method, equipment and medium

Abstract

The invention relates to the technical field of data storage and computers and the technical field of videos, and provides a hierarchical storage management system, a repeated data deletion method, equipment and a medium. Each storage layer in the quantity management storage system is used for storing reference fingerprint information of a corresponding grade of a reference data block obtained according to a preset variable-length fingerprint structure, and calculating target fingerprint information of a corresponding grade of a target data block according to the variable-length fingerprint structure; and performing incremental updating and storage on the reference fingerprint information based on the position comparison result. According to the quantity management storage system, a variable-length fingerprint structure is adopted, so that each storage layer can dynamically adjust the fingerprint information precision according to the characteristics and storage requirements of the data blocks, and the flexibility and performance of the system are remarkably improved. Meanwhile, through an incremental updating mechanism, the system only needs to compare and store different difference data contents between the target data block and the reference data block, repeated storage of repeated data is reduced, and the storage overhead is reduced.

Description

Technical Field

[0001] The present application relates to the fields of data storage, computer technology, and video technology, and in particular to a hierarchical storage management system, a method, a device, and a medium for deleting duplicate data. Background Art

[0002] By identifying and eliminating duplicate data in storage management systems, data deduplication technology can significantly reduce the storage space occupied by the data being stored, lowering storage costs and improving the overall efficiency of the storage management system. For large-scale data storage environments, particularly backup, archiving, and cloud storage scenarios, data deduplication not only optimizes storage resource utilization but also reduces data transmission bandwidth requirements, accelerating backup and recovery speeds. This enables more efficient and cost-effective storage management while ensuring data integrity and availability.

[0003] In the early stages of deduplication technology, hierarchical storage management systems were not yet popular. Therefore, most of the research results in the first ten years of development focused on pre-fetching algorithms, hash function design, and deduplication timing selection to improve the efficiency of deduplication or data reduction rate within a single storage layer. Existing technologies also use a hierarchical storage management system to store data blocks corresponding to files. However, each storage layer uses a fixed-length fingerprint structure to obtain the fingerprint information of the data block, resulting in the same fingerprint information accuracy for each storage layer. The following problems may occur when completing the deduplication and storage of duplicate data under the same structural fingerprint information: (1) It is unable to adapt to the performance requirements and data characteristics of different storage layers during data deletion and storage, limiting the flexibility of the hierarchical storage management system; (2) It is unable to optimize for different data types (such as hot data and cold data), resulting in the storage space utilization and system performance of the hierarchical storage management system not being optimal; (3) It is difficult to support fine-grained duplicate content identification, especially when processing large-scale cold data, which may result in insufficient deduplication of duplicate data, wasting the storage space of the hierarchical storage management system.

[0004] In summary, the existing hierarchical storage management system has low flexibility and performance when performing data deduplication and data storage, and also causes a problem of high system storage overhead when performing data deduplication and data storage. Summary of the Invention

[0005] The present application provides a hierarchical storage management system, a method for deduplicating data, a device, and a medium to solve the problems of low flexibility and performance of the existing hierarchical storage management system when performing deduplication and data storage, and high system storage overhead when performing deduplication and data storage.

[0006] A first aspect of the present application provides a hierarchical storage management system, the hierarchical storage management system of the present application comprising:

[0007] A management device for performing hierarchical prediction on a target data block to obtain a predicted storage location of the target data block, and comparing the predicted storage location of the target data block with an actual storage location of a reference data block to obtain a position comparison result;

[0008] A storage device includes multiple storage layers, each storage layer is used to store reference fingerprint information of a corresponding level of a reference data block obtained according to a preset variable-length fingerprint structure, calculate target fingerprint information of a corresponding level of a target data block according to the variable-length fingerprint structure, and incrementally update and store the reference fingerprint information based on a position comparison result, wherein the target fingerprint information includes a target fingerprint and its corresponding data content, the reference fingerprint information includes a reference fingerprint and its corresponding data content, the variable-length fingerprint structure specifies different levels of fingerprint information in different storage layers, the incremental update indicates comparing the reference fingerprint information with the target fingerprint information, deleting consistent data content in the target fingerprint information that is identical to the reference fingerprint information, and storing differential data content in the target fingerprint information that is different from the reference fingerprint information.

[0009] In one embodiment of the present application, the management device includes a fingerprint level lifting layer, and,

[0010] The fingerprint level promotion layer is configured to: when the predicted storage location is different from the actual storage location, obtain updated reference fingerprint information and perform level transformation on it, and migrate the level-transformed reference fingerprint information to the storage layer corresponding to the predicted storage location;

[0011] Each storage layer is configured to: store the updated reference fingerprint information when the predicted storage location is the same as the actual storage location, or delete the updated reference fingerprint information of the storage layer after the fingerprint level promotion layer obtains the updated reference fingerprint information stored in the storage layer when the predicted storage location is different from the actual storage location.

[0012] In one embodiment of the present application, the management device further includes:

[0013] Intelligent stratification, which uses a pre-trained intelligent stratification model to perform stratification prediction on the target data block to obtain the predicted storage location of the target data block;

[0014] The storage position comparison layer is used to compare the predicted storage position of the target data block with the actual storage position of the reference data block to obtain a position comparison result.

[0015] In some embodiments of the present application, a variable-length fingerprint structure includes: a primary fingerprint, a secondary fingerprint, and a tertiary fingerprint; reference fingerprint information includes: primary reference fingerprint information, secondary reference fingerprint information, and tertiary reference fingerprint information; a storage device includes a performance layer, an intermediate layer, and a capacity layer; and:

[0016] The performance layer is configured to: store updated first-level reference fingerprint information when both the predicted storage location and the actual storage location are the performance layer; or, when the predicted storage location is the middle layer or the capacity layer and the actual storage location is the performance layer, delete the updated first-level reference fingerprint information after the fingerprint level promotion layer obtains the updated first-level reference fingerprint information stored in the performance layer;

[0017] Alternatively, the middle layer is configured to: store the updated secondary reference fingerprint information when both the predicted storage location and the actual storage location are the middle layer; or, when the predicted storage location is the performance layer or the capacity layer and the actual storage location is the middle layer, delete the updated secondary reference fingerprint information after the fingerprint level promotion layer obtains the updated secondary reference fingerprint information stored in the middle layer;

[0018] Alternatively, the capacity layer is configured to: store the updated third-level reference fingerprint information when the predicted storage location and the actual storage location are both the capacity layer; or delete the updated third-level reference fingerprint information after the fingerprint level promotion layer obtains the updated third-level reference fingerprint information stored in the capacity layer when the predicted storage location is the performance layer or the middle layer and the actual storage location is the capacity layer.

[0019] In some embodiments of the present application, the fingerprint level lifting layer includes:

[0020] A fingerprint upgrade module is configured to, when the predicted storage location is the performance layer and the actual storage location is the middle layer or the capacity layer, obtain the updated secondary reference fingerprint information or tertiary reference fingerprint information of the middle layer or the capacity layer, upgrade the information, and migrate the upgraded secondary reference fingerprint information or tertiary reference fingerprint information to the performance layer; or, when the predicted storage location is the middle layer and the actual storage location is the capacity layer, obtain the updated tertiary reference fingerprint information of the capacity layer, upgrade the information, and migrate the upgraded tertiary reference fingerprint information to the middle layer;

[0021] The fingerprint downgrade module is used to obtain the updated first-level reference fingerprint information or second-level reference fingerprint information of the performance layer or the intermediate layer and downgrade it when the predicted storage location is the capacity layer and the actual storage location is the performance layer or the intermediate layer, and migrate the downgraded first-level reference fingerprint information or second-level reference fingerprint information to the capacity layer; or to obtain the updated first-level reference fingerprint information of the performance layer and downgrade it when the predicted storage location is the intermediate layer and the actual storage location is the performance layer, and migrate the downgraded first-level reference fingerprint information to the intermediate layer.

[0022] In some embodiments of the present application,

[0023] The performance layer includes:

[0024] A first-level fingerprint calculation module, used to calculate first-level target fingerprint information using a preset first hash function;

[0025] The middle layer includes:

[0026] a secondary fingerprint calculation module, configured to calculate first intermediate fingerprint information of the target data block using a preset second hash function, and concatenate the primary target fingerprint information and the first intermediate fingerprint information to obtain secondary target fingerprint information;

[0027] The capacity tier includes:

[0028] The third-level fingerprint calculation module is used to calculate the second intermediate fingerprint information of the target data block using the pre-trained fingerprint feature generation model, and to splice the second-level target fingerprint information and the second intermediate fingerprint information to obtain the third-level target fingerprint information.

[0029] In some embodiments of the present application, the secondary target fingerprint information includes the secondary target fingerprint and its corresponding data content, the tertiary target fingerprint information includes the tertiary target fingerprint and its corresponding data content, and:

[0030] The middle layer also includes:

[0031] A first compression module is used to compress and store the difference data content between the secondary target fingerprint information and the secondary reference fingerprint information using a preset first data structure;

[0032] The capacity tier also includes:

[0033] The second compression module is configured to compress and store the difference data content between the third-level target fingerprint information and the third-level reference fingerprint information using a preset second data structure.

[0034] A second aspect of the present application provides a method for deduplicating data. The method for deduplicating data of the present application includes:

[0035] The hierarchical storage management system as described in any of the above embodiments responds to the data block access request and obtains the target data block;

[0036] The hierarchical storage management system incrementally updates and stores the reference data blocks stored in itself based on the target data blocks.

[0037] The third aspect of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the method described in the second aspect of the above embodiment are implemented.

[0038] A fourth aspect of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method described in the second aspect of the above embodiment is implemented.

[0039] This application has the following beneficial effects:

[0040] In the quantity management storage system of the above embodiment of the present application, each storage layer is used to store the reference fingerprint information of the corresponding level of the reference data block obtained according to the preset variable-length fingerprint structure, and calculate the target fingerprint information of the corresponding level of the target data block according to the variable-length fingerprint structure, and incrementally update and store the reference fingerprint information based on the position comparison result. The quantity management storage system of the present application adopts a variable-length fingerprint structure, so that each storage layer can dynamically adjust the accuracy of the fingerprint information according to the characteristics and storage requirements of the data block, and achieve an optimized balance between performance and storage space, thereby significantly improving the flexibility and performance of the quantity management storage system. At the same time, through the incremental update mechanism, the system only needs to compare and store the different differential data content between the target data block and the reference data block, reducing the repeated storage of duplicate data and reducing storage overhead. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The drawings herein are incorporated into and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and, together with the specification, are used to illustrate the technical solutions of the present application.

[0042] Figure 1 This is a schematic diagram of the framework of multiple embodiments of the hierarchical storage management system provided by this application;

[0043] Figure 2 This is a flow chart of an embodiment of the intelligent hierarchical model training method provided by this application;

[0044] Figure 3 This is a schematic diagram of the framework of an embodiment of a variable-length fingerprint structure provided by this application;

[0045] Figure 4 This is a schematic diagram of a framework of an embodiment of the first data structure and the second data structure provided by this application;

[0046] Figure 5 This is a flow chart of an embodiment of a fingerprint feature generation model training method provided by this application;

[0047] Figure 61 is a flow chart of an embodiment of a method for constructing a second training data set provided by the present application;

[0048] Figure 7 This is a schematic diagram of an example of experimental results after clustering the data set provided by this application;

[0049] Figure 8 This is a schematic diagram of the framework of an embodiment of the fingerprint feature generation model provided by this application;

[0050] Figure 9 This is a schematic diagram of the framework of an embodiment of an electronic device provided by the present application;

[0051] Figure 10 It is a schematic diagram of a framework of an embodiment of a computer-readable storage medium provided by this application. DETAILED DESCRIPTION

[0052] The following describes the embodiments of the present application in detail with reference to the accompanying drawings.

[0053] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0054] The term "and / or" in this article is simply a description of the association relationship of associated objects, indicating that three relationships can exist. For example, A and / or B can represent three situations: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects are in an "or" relationship. In addition, "many" in this article means two or more than two. In addition, the term "at least one" in this article means any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0055] As described in the background art, existing hierarchical storage management systems have low flexibility and performance when performing data deduplication and data storage, and also cause a problem of high system storage overhead when performing data deduplication and data storage.

[0056] In order to solve the above problems, this application proposes a hierarchical storage management system, such as Figure 1As shown, the hierarchical storage management system of the present application includes: a management device for performing hierarchical prediction on a target data block to obtain a predicted storage location of the target data block, and comparing the predicted storage location of the target data block with the actual storage location of the reference data block to obtain a position comparison result; a storage device, which includes multiple storage layers, each storage layer for storing reference fingerprint information of a corresponding level of the reference data block obtained according to a preset variable-length fingerprint structure, and calculating target fingerprint information of a corresponding level of the target data block according to the variable-length fingerprint structure, and incrementally updating and storing the reference fingerprint information based on the position comparison result, wherein the target fingerprint information includes a target fingerprint and its corresponding data content, the reference fingerprint information includes a reference fingerprint and its corresponding data content, the variable-length fingerprint structure specifies different fingerprint information levels for different storage layers, and the incremental update indicates comparing the reference fingerprint information with the target fingerprint information, deleting consistent data content in the target fingerprint information that is identical to the reference fingerprint information, and storing differential data content in the target fingerprint information that is different from the reference fingerprint information.

[0057] As can be seen from the foregoing description, the quantity management storage system of the aforementioned embodiment of the present application utilizes a variable-length fingerprint structure, enabling each storage layer to dynamically adjust the accuracy of fingerprint information based on the characteristics of the data block and storage requirements, achieving an optimal balance between performance and storage space, thereby significantly improving the flexibility and performance of the quantity management storage system. Furthermore, through an incremental update mechanism, the system only needs to compare and store the different data content between the target data block and the reference data block, reducing the repeated storage of duplicate data and lowering storage overhead.

[0058] The following is an introduction to the functions performed by the management device and storage device of the hierarchical storage management system proposed in this application.

[0059] 1. Management Device

[0060] In addition, through research, the inventors found that even if the existing hierarchical storage management system maximizes the data deduplication rate of each storage layer and ensures that there is no duplicate data in each storage layer, the fixed-length fingerprint information cannot adapt to the data access patterns and storage requirements of different storage layers, making it difficult to effectively identify and eliminate cross-layer duplicate data, thereby resulting in a waste of storage resources and a decline in system performance.

[0061] The functions performed by the management device are described in detail below.

[0062] (1) Fingerprint level lifting layer

[0063] According to one embodiment of the present application, still referring to Figure 1As shown, the management device includes a fingerprint level promotion layer, and the fingerprint level promotion layer is configured to: when the predicted storage location is different from the actual storage location, obtain the updated reference fingerprint information and perform level transformation on it, and migrate the reference fingerprint information after level transformation to the storage layer corresponding to the predicted storage location; each storage layer is configured to: when the predicted storage location is the same as the actual storage location, store the updated reference fingerprint information, or when the predicted storage location is different from the actual storage location, after the fingerprint level promotion layer obtains the updated reference fingerprint information stored in the storage layer, delete the updated reference fingerprint information of the storage layer.

[0064] As can be seen from the above description, the above embodiments of the present application utilize the properties of the variable-length fingerprint structure to dynamically adjust the fingerprint information level of the data block and combine the comparison of the predicted storage location with the actual storage location to identify and delete the same data blocks in different storage layers at the fingerprint information level, thereby effectively eliminating duplicate data across storage layers and improving the utilization efficiency of storage resources.

[0065] The following is a detailed introduction to the functions performed by the fingerprint level lifting layer

[0066] 1. Fingerprint upgrade module

[0067] According to one embodiment of the present application, still referring to Figure 1 As shown, the fingerprint level promotion layer includes: a fingerprint upgrading module, which is used to obtain updated reference fingerprint information and perform promotion and promotion processing on it when the fingerprint information level of the storage layer corresponding to the predicted storage location is higher than the fingerprint information level of the storage layer corresponding to the actual storage location.

[0068] Among them, according to one embodiment of the present application, the variable-length fingerprint structure includes: a first-level fingerprint, a second-level fingerprint and a third-level fingerprint, the reference fingerprint information includes: a first-level reference fingerprint information, a second-level reference fingerprint information, and a third-level reference fingerprint information, the storage device includes a performance layer, an intermediate layer, and a capacity layer, and: the fingerprint upgrade module is also used to obtain the updated second-level reference fingerprint information or the third-level reference fingerprint information of the intermediate layer or the capacity layer and upgrade it when the predicted storage location is the performance layer and the actual storage location is the intermediate layer or the capacity layer, and migrate the upgraded second-level reference fingerprint information or the third-level reference fingerprint information to the performance layer; or when the predicted storage location is the intermediate layer and the actual storage location is the capacity layer, obtain the updated third-level reference fingerprint information of the capacity layer and upgrade it, and migrate the upgraded third-level reference fingerprint information to the intermediate layer.

[0069] As can be seen from the above description, the above-mentioned embodiments of the present application realize dynamic upgrading and downgrading of fingerprint information levels by introducing a variable-length fingerprint structure (including first-level fingerprint, second-level fingerprint and third-level fingerprint) and combining it with the hierarchical architecture of the storage device (performance layer, middle layer, capacity layer). This approach significantly improves the flexibility and accuracy of data management in the hierarchical storage management system, optimizes the allocation efficiency of storage resources, and reduces redundant data storage and system load.

[0070] According to one embodiment of the present application, the fingerprint upgrade module is further configured to upgrade the updated secondary reference fingerprint information or tertiary reference fingerprint information in the following manner when the predicted storage location is the performance layer and the actual storage location is the middle layer or the capacity layer: the updated secondary reference fingerprint information or tertiary reference fingerprint information is divided into the first-level reference fingerprint information stored in the performance layer, and the first-level reference fingerprint information obtained by the division is migrated to the performance layer for storage. At the same time, the fingerprint upgrade module is further configured to upgrade the updated third-level reference fingerprint information in the following manner when the predicted storage location is the middle layer and the actual storage location is the capacity layer: the updated third-level reference fingerprint information is divided into the second-level reference fingerprint information stored in the middle layer, and the second-level reference fingerprint information obtained by the division is migrated to the middle layer for storage.

[0071] As can be seen from the above description, the above embodiment of the present application dynamically adjusts the fingerprint information storage level of the reference data block through the fingerprint upgrade module, and intelligently divides and migrates the fingerprint information between the performance layer, the middle layer and the capacity layer, which can effectively optimize the allocation and utilization of storage resources. This method not only improves the efficiency of data access and reduces the storage pressure of the performance layer and the middle layer, but also achieves a rapid response to hot data through a hierarchical storage strategy, while taking into account the low-cost storage of cold data, thereby improving the performance and resource utilization of the hierarchical storage management system as a whole.

[0072] 2. Fingerprint downgrade module

[0073] According to one embodiment of the present application, still referring to Figure 1 As shown, the fingerprint level promotion layer includes: a fingerprint degradation module, which is used to obtain updated reference fingerprint information and perform degradation processing on it when the fingerprint information level of the storage layer corresponding to the predicted storage location is lower than the fingerprint information level of the storage layer corresponding to the actual storage location.

[0074] Among them, according to one embodiment of the present application, the fingerprint degradation module is also used to obtain the updated first-level reference fingerprint information or second-level reference fingerprint information of the performance layer or the intermediate layer and downgrade it when the predicted storage location is the capacity layer and the actual storage location is the performance layer or the intermediate layer, and migrate the downgraded first-level reference fingerprint information or second-level reference fingerprint information to the capacity layer; or when the predicted storage location is the intermediate layer and the actual storage location is the performance layer, obtain the updated first-level reference fingerprint information of the performance layer and downgrade it, and migrate the downgraded first-level reference fingerprint information to the intermediate layer.

[0075] According to one embodiment of the present application, the fingerprint degradation module is further configured to, when the predicted storage location is the capacity layer and the actual storage location is the performance layer or the intermediate layer, use the updated first-level reference fingerprint information or the second-level reference fingerprint information for degradation processing in the following manner: using a second hash function to supplement the updated first-level reference fingerprint information into second-level reference fingerprint information, and then using a fingerprint feature generation model to supplement the supplemented second-level reference fingerprint information again into third-level reference fingerprint information stored in the capacity layer, and migrate the supplemented third-level reference fingerprint information to the capacity layer for storage, or using a fingerprint feature generation model to supplement the updated second-level reference fingerprint information into third-level reference fingerprint information stored in the capacity layer, and migrate the supplemented third-level reference fingerprint information to the capacity layer for storage. According to one embodiment of the present application, when supplementing the first-level reference fingerprint information and the second-level reference fingerprint information into third-level reference fingerprint information, the second-level fingerprint calculation module and the third-level fingerprint calculation module of the intermediate layer are called to complete the process. At the same time, according to one embodiment of the present application, the fingerprint degradation module is further configured to, when the predicted storage location is the middle tier and the actual storage location is the performance tier, use the updated first-level reference fingerprint information and perform degradation processing on it in the following manner: using a second hash function to supplement the updated first-level reference fingerprint information as the second-level reference fingerprint information stored in the middle tier, and migrating the supplemented second-level reference fingerprint information to the middle tier for storage. According to one embodiment of the present application, when supplementing the first-level reference fingerprint information to the second-level reference fingerprint information, the second-level fingerprint calculation module in the middle tier is called to complete the process.

[0076] As can be seen from the above description, the above embodiment of the present application dynamically adjusts the fingerprint information storage level of the reference data block through the fingerprint degradation module, and uses the hash function and fingerprint feature generation model to supplement the first or second level fingerprint information of the performance layer or middle layer with the second or third level fingerprint information of the middle layer or capacity layer, and migrates it to the corresponding layer for storage. This method can effectively release the storage resources of the performance layer and the middle layer, reduce storage costs, and ensure the reasonable storage of cold data through a hierarchical degradation strategy. While improving the overall efficiency of the hierarchical storage management system, it achieves efficient utilization of storage resources and cost optimization.

[0077] Based on the above analysis, it can be concluded that the above embodiments of the present application configure a fingerprint upgrade module and a fingerprint downgrade module in the hierarchical storage management system, and can dynamically adjust the fingerprint information level of the reference data block according to the difference in fingerprint information level between the predicted storage location of the target data block and the actual storage location of the reference data block. This method effectively optimizes the flexibility and accuracy of fingerprint information management in the hierarchical storage management system, reduces redundant data storage, and improves system performance and resource utilization.

[0078] (2) Intelligent stratification

[0079] According to one embodiment of the present application, still referring to Figure 1 As shown, the management device of the present application also includes: intelligent stratification, which is used to use a pre-trained intelligent stratification model to perform stratified prediction on the target data block to obtain the predicted storage location of the target data block.

[0080] It can be seen from the above embodiments that the above embodiments of the present application use a pre-trained intelligent hierarchical model to perform hierarchical prediction on the target data block to obtain its predicted storage location, thereby optimizing the distribution of data between the performance layer, the middle layer and the capacity layer.

[0081] According to one embodiment of the present application, Figure 2 As shown, the pre-trained intelligent tiering model is trained in the following manner: S1. Obtain a first data block and construct a first training data set based thereon, wherein the first data block includes access features, and the access features include: inherent attributes, temporal locality features, and spatial locality features; S2. Construct an initial intelligent tiering model, and use the first data block as input and the predicted storage location of the first data block as output; S3. Use the first training data set to perform multiple iterative training convergences on the initial intelligent tiering model to obtain a pre-trained intelligent tiering model. According to one embodiment of the present application, the first training data set includes a block-level deduplication data set (e.g., FIUIODedup traces) and a server user data set (e.g., Inotify).

[0082] From the above description, it can be seen that the embodiment of the present application constructs a first training data set based on the access characteristics of the first data block (including inherent attributes, temporal locality characteristics and spatial locality characteristics), and finally obtains a pre-trained intelligent tiering model. This training method enables the intelligent tiering model to fully capture the access pattern and storage requirements of the first data block, thereby significantly improving the accuracy of the intelligent tiering model's prediction of the storage location of the target data block, optimizing the data allocation efficiency of the tiered storage management system, and reducing the waste of storage resources.

[0083] (3) Storage location comparison layer

[0084] According to one embodiment of the present application, still referring to Figure 1 As shown, the management device of the present application further includes: a storage position comparison layer, which is used to compare the predicted storage position of the target data block with the actual storage position of the reference data block to obtain a position comparison result.

[0085] It can be seen from the above embodiments that the storage location comparison layer of the above embodiments of the present application compares the predicted storage location of the target data block with the actual storage location of the reference data block to generate a location comparison result, which provides a decision basis for subsequent fingerprint information management and duplicate data deletion.

[0086] Based on the above analysis, the configuration of intelligent tiering and storage location comparison layers in the tiered storage management system of the above-mentioned embodiment of the present application enables accurate prediction and efficient management of the storage location of target data blocks. This approach significantly enhances the intelligence level of the tiered storage management system, optimizes data storage efficiency and resource utilization, and reduces storage costs and system management complexity.

[0087] 2. Storage Device

[0088] According to one embodiment of the present application, the performance layer is configured to: when the predicted storage location and the actual storage location are both the performance layer, store the updated first-level reference fingerprint information; or when the predicted storage location is the middle layer or the capacity layer and the actual storage location is the performance layer, after the fingerprint level promotion layer obtains the updated first-level reference fingerprint information stored in the performance layer, delete the updated first-level reference fingerprint information; or the middle layer is configured to: when the predicted storage location and the actual storage location are both the middle layer, store the updated second-level reference fingerprint information; or when the predicted storage location is the performance layer or the capacity layer and the actual storage location is the middle layer, after the fingerprint level promotion layer obtains the updated second-level reference fingerprint information stored in the middle layer, delete the updated second-level reference fingerprint information; or the capacity layer is configured to: when the predicted storage location and the actual storage location are both the capacity layer, store the updated third-level reference fingerprint information; or when the predicted storage location is the performance layer or the middle layer and the actual storage location is the capacity layer, after the fingerprint level promotion layer obtains the updated third-level reference fingerprint information stored in the capacity layer, delete the updated third-level reference fingerprint information.

[0089] From the above description, it can be seen that the above embodiments of the present application realize intra-layer deduplication and cross-layer deduplication by designing a variable-length fingerprint structure (first-level fingerprint information, second-level fingerprint information, and third-level fingerprint information) and combining the upgrade and downgrade deduplication management mechanism of the hierarchical storage management system (performance layer, middle layer, and capacity layer). This upgrade and downgrade deduplication management mechanism also optimizes the storage resource utilization of the hierarchical storage management system and improves the accuracy and efficiency of deduplication.

[0090] The following is an introduction to the design ideas of the variable-length fingerprint structure of this application.

[0091] The inventors have found through research that the types of storage media are becoming increasingly diverse. Storage media such as Flash and DRAM are sensitive to delays and have limited resources. They are usually used to store frequently accessed hot data (hot data refers to data that is frequently accessed or used in the system). When these storage media are integrated into corresponding storage layers, the need for deduplication in such storage layers is to try to identify redundancy without introducing too much additional overhead. On the contrary, storage media such as tapes, hard disks, optical storage, and cloud storage environments have sufficient capacity and usually do not require high throughput. They are usually used to store archived data. When these storage media are integrated into corresponding storage layers, the need for deduplication in such storage layers is to pursue the ultimate data reduction rate. Since the accuracy of fingerprint information directly affects the space overhead required for storing data in a hierarchical storage management system and the time overhead for executing fingerprint information generation and query, the diverse requirements of different storage media for deduplication operations determine that fixed-length fingerprint structures are difficult to achieve satisfactory results in hierarchical storage management systems. To address the problem of poor compatibility between fixed-length fingerprint structures and hierarchical storage management systems, this application designs a variable-length fingerprint structure (VLFP) that supports flexible expansion and segmentation in different storage layers to improve the accuracy of the system's deduplication and storage space utilization.

[0092] Considering that data deduplication will identify data blocks with identical fingerprint information as redundant data and retain only a unique backup to reduce the storage space overhead of the hierarchical storage management system, the cost of misjudgment caused by hash collisions is extremely high, and it is often necessary to restore the accidentally deleted data blocks from the system's secure zone backup. Therefore, in order to accurately eliminate duplicate data and store duplicate data with low overhead, the inventors conducted the following research, resulting in the variable-length fingerprint structure of this application:

[0093] This application uses part of the Linux kernel source code to form a new data set. The data set used in this application contains 17,777,962 4KB data blocks and 4,424,710 unique data blocks (unique data blocks are data blocks with different data contents). Since the differences between Linux kernel versions are small, the proportion of duplicate data blocks in the entire data set is high, which can more clearly reflect the space saving effect under different deduplication operations and also facilitate subsequent similarity evaluation.

[0094] According to one embodiment of the present application, collision rate detection is performed on different hash functions using different versions of Linux kernel files. According to one embodiment of the present application, collision rate detection is performed on different hash functions using the following method:

[0095]

[0096] Among them, N is the size of the fingerprint space. For fingerprint information with a length of L, N=2 L , k is the number of unique data blocks in the dataset. If we want all the fingerprint information generated by the unique data blocks to avoid collision, the collision probability threshold can be determined as 2.26E-7.

[0097] According to one embodiment of the present application, the collision detection formula is used to perform collision rate detection on hash functions such as xxhash, 32LSH, and 64LSH.

[0098] The results show that the collision probability of 32 locality sensitive hash functions (LSH) on the data set constructed by this application is as high as 88%, which exceeds the proportion of duplicate data blocks (data blocks with the same or similar data) in the data set, and may cause an extremely high misjudgment rate. Therefore, the secondary fingerprint information in the variable-length fingerprint structure of this application takes into account the storage overhead and computational delay, and selects the 64-bit LSH hash value corresponding to the 64LSH hash function for combination. In order to subsequently evaluate the deduplication performance under the guidance of the variable-length fingerprint structure, this application limits the length of the tertiary fingerprint information in the variable-length fingerprint structure to 256 bits, that is, it can be compared with the most widely used 256SHA fingerprint information in one length. Therefore, the length of the feature value that can be used for the similarity recognition part of the capacity layer is determined to be 128 bits.

[0099] According to the calculation of collision detection rates of different hash functions, according to one embodiment of the present application, Figure 3 As shown, the first-level fingerprint information in the variable-length fingerprint structure is the 64-bit xxhash, the second-level fingerprint information is the concatenation result of 64-bit xxhash and 64-bit LSH, and the third-level fingerprint information is the concatenation result of 64-bit xxhash, 64-bit LSH, and 128-bit similar feature values.

[0100] From the above description, it can be seen that the variable-length fingerprint structure designed in the above embodiment of the present application can enable the performance layer of the hierarchical storage management system to enjoy high-speed, efficient and low-cost deduplication of data, and enable the capacity layer to achieve high-precision and high deduplication rate data management, which can comprehensively optimize the global resource allocation of the hierarchical storage management system.

[0101] The following describes the functions performed by the performance layer, middle layer, and capacity layer.

[0102] Through the aforementioned verification of different hash functions, the inventors discovered that the length and computation speed of fingerprint information at different levels in the variable-length fingerprint structure of this application differ by several times. Furthermore, the highly collision-resistant hash function ensures that deduplication will not mistakenly delete data with identical fingerprint information but different content, thus ensuring security.

[0103] (1) Performance layer

[0104] According to one embodiment of the present application, still referring to Figure 1 As shown, the performance layer includes: a first-level fingerprint calculation module for calculating the first-level target fingerprint information using a preset first hash function. According to one embodiment of the present application, the first hash function is the xxhash hash function. It should be noted that with the continuous iteration and update of the hash function, the first-level fingerprint information in the variable-length fingerprint structure of the present application can use a hash function with better performance. The above-mentioned xxhash hash function is only one embodiment of the present application and is not specifically limited here.

[0105] From the above description, it can be seen that the above embodiment of the present application uses a hash function of appropriate fingerprint information length to directly calculate the first-level fingerprint information, which can save storage space and achieve rapid deduplication. Therefore, for the first-level fingerprint information, the variable-length fingerprint structure is directly calculated using the xxhash hash function, which combines the characteristics of xxhash's high computing rate, low storage overhead, and sufficiently low collision probability, significantly improving the efficiency of fingerprint information calculation in the hierarchical storage management system, saving storage space, and ensuring the accuracy and reliability of data deduplication.

[0106] (2) Middle layer

[0107] The locality-sensitive hash function (LSH hash function) can additionally identify similar data blocks through the Hamming distance of the text's feature vectors. The collision does not mean that the data blocks are exactly the same, but rather that the data blocks are similar in content. Combined with the incremental compression method, it can further achieve finer-grained data reduction.

[0108] According to one embodiment of the present application, still referring to Figure 1 As shown, the secondary fingerprint calculation module is used to calculate the first intermediate fingerprint information of the target data block using a preset second hash function, and to concatenate the primary target fingerprint information and the first intermediate fingerprint information to obtain the secondary target fingerprint information. According to one embodiment of the present application, the second hash function is 64LSH. It should be noted that the secondary fingerprint information in the variable-length fingerprint structure of the present application can use a hash function with better performance. The above-mentioned 64LSH hash function is only one embodiment of the present application and is not specifically limited here.

[0109] As can be seen from the above description, the above embodiment of the present application uses the 64LSH hash function as the second hash function to calculate the first intermediate fingerprint information of the target data block in the secondary fingerprint calculation module, and then splices the first-level fingerprint information with the first intermediate fingerprint information to generate the second-level fingerprint information. This method fully utilizes the high efficiency and low collision probability of the 64LSH hash function, and further enhances the uniqueness and reliability of the second-level fingerprint through fingerprint information splicing, providing more accurate fingerprint information support for data deduplication and storage optimization, and improving the overall performance and data management efficiency of the hierarchical data management system.

[0110] The inventor has found that by splicing 64LSH on the basis of the primary fingerprint information to generate the secondary fingerprint information, a higher data reduction rate can be explored with additional similarity recognition overhead. To this end, according to one embodiment of the present application, still referring to Figure 1 As shown, the middle layer further includes: a first compression module, configured to compress and store the differential data content in the secondary target fingerprint information and the secondary reference fingerprint information using a preset first data structure.

[0111] According to one embodiment of the present application, reference is still made to Figure 1 As shown, the fingerprint information includes the fingerprint and its corresponding data content. When the data content corresponding to the fingerprint in the target data block is the same as that in the reference data block, the data content corresponding to the fingerprint is set to 0, otherwise it is set to 1. After the target data block and the reference data block are calculated using the xxhash hash function, the first-level fingerprint of the target data block and the reference data block and the data content corresponding to the fingerprint are obtained, and as shown Figure 4As shown, the first data structure includes: a 1-byte short compressed data structure, which is used to compress and store the secondary fingerprint information in the target data block that is different from the reference data block based on a preset first offset standard, wherein the 1-byte short compressed data structure includes: a flag bit 1 (also called flag1), which is used to identify that the data content 1 corresponding to the fingerprint appears alone when a single 1 appears, and to record the fingerprint of the first 1 when consecutive 1s appear; a value bit 1 (also called value1), which is used to record the fingerprint corresponding to the data content, that is, the offset of the data content, when a single 1 appears, and to record the offset after the first 1 when consecutive 1s appear. The number of 1s on the surface; a 2-byte long compressed data structure, used to compress and store secondary fingerprint information in the target data block that is different from the reference data block based on a preset second offset standard, wherein the first long compressed data structure includes: a flag bit 2 (also called flag2), used to identify that when a single 1 appears, it identifies that the data content 1 corresponding to the fingerprint appears alone, and when consecutive 1s appear, it records the fingerprint of the first 1; a value bit 2 (also called value2), used to record the fingerprint corresponding to the data content, that is, the offset of the data content, when a single 1 appears, and when consecutive 1s appear, it records the number of 1s after the first 1. According to one embodiment of the present application, the first offset standard is that the offset is less than 7 bits, and the second offset standard is that the offset is equal to or greater than 7 bits. The short compressed data structure and the long compressed data structure are collectively referred to as a compressed data structure (OPTICS based Similarity Compression, OSC)

[0112] For each single occurrence of 1, the 1-bit flag is set to 0, and the subsequent numerical bits record the relative offset from the position of the previous 1 (the fingerprint indicates the position index of the data content). If the offset is less than 128, the 1-byte first short compression structure is used to represent the data content; because the maximum value of the offset of a 4KB data block does not exceed 32768, the 2-byte first long compression structure is used for storage when the offset is greater than 128. For consecutive occurrences of 1, the first 1 is first stored in the format of a single 1, and then the number of consecutive 1s n is counted and stored using a "flag=1, value=n" structure, which occupies an additional 1 or 2 bytes. In particular, the index subscript of the first occurrence of 1 after comparison between the target data block and the reference data block is represented by an absolute offset.

[0113] The above examples show that the more consecutive 1s there are after comparing the target data block with the reference data block, the stronger the compression capability provided by OSC. In the worst case, it appears that each 1 after comparing the target data block with the reference data block requires two bytes to represent, and incremental compression requires additional storage of the reference data block's secondary fingerprint information to facilitate the decompression process.

[0114] The inventors conducted the following experiments to verify the effectiveness of the data structure proposed in the above-mentioned embodiments of this application for storing differential data. Using 2040 similar data blocks, they conducted a rigorous mathematical analysis of three special cases with the worst incremental compression results. Data blocks with sufficiently small Hamming distances from the target data block were designated as reference data blocks for incremental compression. After performing an XOR operation on the target and reference data blocks to obtain differential fingerprint information, compressing and storing this differential fingerprint information effectively achieves the purpose of incremental compression. The binary representation of the differential fingerprint information, in principle, consists of a large number of 0s and a small number of 1s. The following results were obtained:

[0115] A: 2040 ones are evenly distributed across the 4KB data block, meaning there are no consecutive ones. The average distance between any two ones is about 16. Each one can be represented using a 1-byte short compressed data structure. The storage overhead of the difference between the target data block and the reference data block is 2040 bytes.

[0116] B: When the spacing between 1s increases to more than 128, a 15-bit value2 is required to represent the relative offset. Each 1 requires 2 bytes to store. However, due to the total length of the subsequent data block, a large number of 1s accumulate at the end, inevitably resulting in a continuous sequence of 1s, significantly improving the effectiveness of incremental compression. Considering the worst-case scenario, where a single 1 appears every other bit, after 1815 cycles, a 1 appears every 129 bits until the end of the data block. At this point, the storage overhead for the difference between the target and reference data blocks is approximately 2265 bytes.

[0117] C: Based on B, since the overhead of representing two consecutive 1s is basically the same as the overhead of representing two single 1s, a small sequence of two 1s appears every other bit. After 787 cycles, a small sequence of two 1s appears every 129 bits until the end of the data block. At this time, the storage overhead of the difference data content after comparing the target data block and the reference data block is approximately 2273 bytes.

[0118] Theoretically, there is no situation worse than C, and such accidental data content distribution rarely occurs in actual data sets. It is expected that the final similarity threshold allowed when incrementally compressing 4KB data blocks using the OSC structure will be much higher than 2040. The specific value will be determined through subsequent experiments.

[0119] In specific implementation, the result of comparing the incrementally compressed target data block and the reference data block with the difference data content will be an array formed by a series of OSC objects. Each array element can be represented as a uint8 or uint16 type, which is convenient for computer processing. During decompression, each time an object is traversed, its flag field is checked. If it is 0, the flag of the next object is used to determine whether the current object is a single 1 or the beginning of a continuous sequence of 1s. Under normal circumstances, if the flag of the current object is 1, it indicates that the compression process is abnormal. The difference data content after comparing the decompressed target data block and the reference data block is XORed with the data content of the reference data block obtained through the fingerprint of the stored reference data block to obtain the original content of the data block. This application uses a processed list to store the index of the data blocks that have been processed by incremental compression. The nearest neighbor search will skip the elements in this list to ensure that the compressed data blocks will not be selected as reference data blocks for other data blocks, avoiding possible circular dependencies and unbearable performance losses caused by recursive compression and decompression operations of too great a depth. After obtaining a compressed representation of the difference between the target data block and the reference data block through the OSC structure, Huffman coding is used for secondary processing to further improve the overall compression rate of the corresponding storage layer, thereby reducing the space overhead of storing data in the hierarchical storage management system.

[0120] As can be seen from the above description, the design of the first data structure in the above embodiment of the present application has been verified to be effective and reasonable through a large number of experiments. The use of the first data structure to compress and store the differential data content in the secondary target fingerprint information that is different from the secondary reference fingerprint information can reduce the space overhead of storing data in the hierarchical storage management system.

[0121] (3) Capacity layer

[0122] According to one embodiment of the present application, still referring to Figure 1 As shown, the capacity layer includes a three-level fingerprint calculation module, which is used to calculate the second intermediate fingerprint information of the target data block using a pre-trained fingerprint feature generation model, and to splice the secondary target fingerprint information and the second intermediate fingerprint information to obtain the third-level target fingerprint information.

[0123] As can be seen from the above description, the above embodiment of the present application configures a three-level fingerprint calculation module within a hierarchical storage management system, uses a pre-trained fingerprint feature generation model to calculate the second intermediate fingerprint information of the target data block, and then splices the second-level target fingerprint information and the second intermediate fingerprint information to generate more refined and discriminative third-level target fingerprint information. This multi-level, fine-grained fingerprint calculation method not only improves the accuracy of duplicate content identification, but also effectively reduces the storage space occupied by cold data, thereby achieving efficient utilization of storage resources.

[0124] According to one embodiment of the present application, Figure 5 As shown, the pre-trained fingerprint feature generation model is trained in the following manner: P1, obtaining a second data block and constructing a second training data set based on it; P2, constructing an initial fingerprint feature generation model, and taking the second data block as input, and the fingerprint information feature value of the second data block as output; P3, using the second training data set to perform multiple iterative training on the initial fingerprint feature generation model until convergence, so as to obtain a pre-trained fingerprint feature generation model.

[0125] From the above description, it can be seen that the training method of the above embodiment of the present application effectively improves the generalization ability and feature extraction accuracy of the fingerprint feature generation model, ensures that the generated fingerprint information features are highly reliable and unique, thereby providing more accurate fingerprint information support for data deduplication and storage optimization, and significantly improving the performance and data processing efficiency of the hierarchical storage management system.

[0126] According to one embodiment of the present application, Figure 6 As shown, the following steps are used to construct the second training data set: T1, clustering the obtained multiple second data blocks using a clustering method to obtain multiple clusters, wherein a point in each cluster represents a second data block (such as Figure 7 As shown); T2, any two points in each cluster constitute a positive sample pair, and any two points between different clusters, and / or any point in any cluster and any point outside the cluster, and / or any two points outside the cluster respectively constitute a negative sample pair; T3, all positive sample pairs and negative sample pairs are used to construct a second training data set.

[0127] According to one embodiment of the present application, P3 includes: in each iterative training, updating the parameters of the fingerprint feature generation model according to the preset total comparison loss. The total comparison loss is:

[0128] loss = pos_loss + neg_loss

[0129] Among them, loss is the total loss of comparison, pos_loss is the positive sample loss, and neg_loss is the negative sample loss.

[0130] According to one embodiment of the present application, the positive sample loss is:

[0131]

[0132] Where ReLU(·) is the activation function, N is the number of positive sample pairs, X is the number of eigenvalues ​​of the second data block, I(·) is the indicator function (1 when unequal, 0 when equal), sketch0 (i,j)Indicates the jth fingerprint information feature value of the second data block in the i-th positive sample pair, sketch1 (i,j) represents the jth fingerprint information feature value of the other second data block in the i-th positive sample pair, p is the preset positive sample Hamming distance threshold, where p=8.

[0133] According to one embodiment of the present application, the negative sample loss is:

[0134]

[0135] Among them, log(·) is the logarithmic function, exp(·) is the exponential function, sketch0 (k,t) represents the tth fingerprint information feature value of the second data block in the kth negative sample pair, sketch1 (k,t) represents the t-th fingerprint information feature value of another second data block in the k-th positive sample pair, m is the preset negative sample Hamming distance threshold, where m=64.

[0136] As can be seen from the above description, the above embodiment of the present application constructs positive and negative sample pairs, and the second data blocks similar within the cluster constitute positive sample pairs to shorten the distance between positive samples. At the same time, the second data blocks between different clusters or outside the cluster constitute negative sample pairs to increase the distance between negative samples. The positive sample loss and negative sample loss are calculated separately, and the total loss is synthesized to update the model parameters. This method can effectively enhance the fingerprint feature generation model's ability to aggregate similar data and distinguish dissimilar data, thereby improving the fingerprint feature generation model's representation effect on data features, improving the generalization performance and robustness of the fingerprint feature generation model, and ultimately obtaining more accurate fingerprint information feature values.

[0137] According to one embodiment of the present application, Figure 8As shown, the fingerprint feature generation model consists of an input layer, a fully connected layer, and an output layer. The fully connected layers are designed to better preserve the complex similarity relationships between the original data. LeakyReLU layers are added between the fully connected layers to prevent gradient vanishing and neuron death. Dropout layers are introduced to randomly drop units to prevent overfitting. For any non-noise sample (i.e., a data block within a cluster; samples outside the cluster are considered noise samples), 1% (at least one) of the samples with the same cluster label are randomly selected to form the positive sample set. A similar 1% of the non-noise samples with different labels are also selected, and a random sample from the noise samples is added to form the negative sample set. Although each noise sample is essentially a separate class, including all noise samples as negative samples reduces the generalization ability of the fingerprint feature generation model, increases instability, and impairs performance. Based on the relationship between the number of noise points and the number of true cluster samples, the number of noise samples selected is proportionally reduced, ensuring that the number of samples provided by each label is roughly the same. Since there are no positive samples in the noise, 0.1% of the data blocks are randomly sampled from all labels as negative samples, emphasizing the loss of negative samples and learning the difference between noise and other data. The loss is measured by the average Hamming distance between the 128-bit fingerprint information eigenvalue of the current sample and the 128-bit fingerprint information eigenvalue generated by each positive and negative sample. For the positive sample set, the loss is proportional to the Hamming distance. For the negative sample set, a margin value is initialized to guide the fingerprint feature generation model to maximize the distance between the target data block and the negative sample data block. The margin value is dynamically increased during the iteration process to avoid gradient explosion caused by an initial excessively large margin value in the early stages of training. After training, VLFP can generate similar fingerprint information for data blocks with originally small Hamming distances, and can also distinguish data blocks with originally large differences as much as possible through similar fingerprint information.

[0138] According to one embodiment of the present application, the capacity layer further includes: a second compression module for compressing and storing the differential data content in the third-level target fingerprint information and the third-level reference fingerprint information using a preset second data structure. It should be noted that the second data structure has the same structure and storage principle as the first data structure. For the specific structure, refer to Figure 4 As shown, no further details will be given here.

[0139] It can be seen from the above embodiments that the design of the second data structure in the above embodiments of the present application has been verified to be effective and rational through a large number of experiments. The use of the second data structure to compress and store the differential data content in the third-level target fingerprint information that is different from the third-level reference fingerprint information can reduce the space overhead of storing data in the hierarchical storage management system.

[0140] The inventors have found through research that, based on the inspiration of DeepSketch and SimEnc, which respectively tried to use DK-Clustering and DBSCAN clustering methods for the identification of similar data blocks, this application adopts the OPTICS clustering method to identify similar data blocks at the capacity layer, and measures the gap between different data blocks by Hamming distance. The OPTICS clustering method starts from a random point (the point here refers to a data block), and each time selects the unprocessed point with the smallest reachable distance to the current point as the next expansion point until all core points are processed. After the method is completed, a reachable distance graph will be generated. At this time, the neighborhood radius or inclination degree can be flexibly set to perform clustering divisions with different standards, thereby enhancing the scalability of similarity identification.

[0141] To this end, according to one embodiment of the present application, still refer to Figure 1 As shown, the capacity layer includes a clustering module for calculating similarity metrics between the three-level target fingerprint information and the three-level reference fingerprint information using the OPTICS clustering method, and transmitting the similarity metrics to the three-level fingerprint calculation module for processing.

[0142] For example, still referring to Figure 8 As shown in the figure, after the clustering module determines the similarity measure between the three-level target fingerprint information and the three-level reference fingerprint information, the fingerprint feature generation model projects the similarity between 4KB data blocks onto 128-bit eigenvalues. After clustering, the data blocks generate 128-bit fingerprint information eigenvalues ​​through hash learning. The k fingerprint information eigenvalues ​​with the closest Hamming distance to each 128-bit fingerprint information eigenvalue are obtained through the nearest neighbor method. The data blocks containing these k fingerprint information eigenvalues ​​are then reverse-indexed to the data blocks. The data block with the smallest Hamming distance between the two blocks is selected as the reference data block to achieve the highest data compression rate. It is important to note that incremental compression is not applicable to all data blocks. When the Hamming distance between the reference data block and the target data block is greater than the threshold D, it indicates that there is no data block in the capacity layer that is sufficiently similar to the target data block. In this case, forcing incremental compression will actually increase the actual storage overhead. The Hamming distance is the number of data differences between the target fingerprint information and the reference fingerprint information.

[0143] As can be seen from the above description, the above embodiment of the present application achieves efficient compression of the different fingerprint information between the third-level target fingerprint information and the third-level reference fingerprint information by projecting the similarity between 4KB data blocks onto 128-bit feature values ​​and using the Hamming distance to filter out the reference data block closest to the target data block. This method not only significantly improves the data compression rate, but also avoids the problem of increased storage overhead caused by forced compression by setting the threshold D, ensuring the accuracy and effectiveness of incremental compression, and ultimately optimizing the utilization of storage resources while ensuring data integrity.

[0144] In addition, based on the hierarchical storage management system of the above-mentioned embodiments, according to one embodiment of the present application, the present application proposes a method for deduplicating data. The method for deduplicating data of the present application includes: the hierarchical storage management system as described in any of the above-mentioned embodiments responds to the data block access request and obtains the target data block; the hierarchical storage management system incrementally updates and stores the reference data blocks stored in itself based on the target data blocks.

[0145] When the tiered storage management system receives a data block access request, Intelligent Tiering uses the pre-trained Intelligent Tiering model to determine the ideal storage location for the data block in the tiered storage management system. It then performs one of the following three deduplication operations based on the location:

[0146] (1) Intra-layer deduplication

[0147] When the predicted storage location obtained by the intelligent tiering model is the same as the actual storage location, the accessed data block does not need to be migrated across layers. According to one embodiment of the present application, the access request includes: a read request and a write request. When a read request occurs, the data content does not change, and its access request and the priority of the elimination queue are updated, and it continues to remain in the original storage layer; when a write request occurs, the data content of the data block changes, and the corresponding fingerprint changes. It is necessary to update the fingerprint information in the fingerprint calculation module of the corresponding storage layer, and then perform intra-layer deduplication.

[0148] Intra-layer deduplication in the performance layer is performed directly by comparing the xxhash values ​​calculated by the first-level fingerprint calculation module. If the same fingerprint information is found, the redundant data is deleted; otherwise, a new data block is stored.

[0149] Intra-layer deduplication in the middle layer first relies on a comprehensive evaluation of the secondary fingerprint information. If the entire secondary fingerprint information is identical, it indicates a confirmed duplicate data block. If the 64LSH value calculated by the secondary fingerprint calculation module differs despite the same xxhash value, it indicates a hash collision and requires full storage of the data block. If the 64LSH values ​​are the same but the xxhash values ​​are different, the two data blocks have a certain degree of similarity, enabling additional data reduction through processing in the first compression module. To ensure accurate and efficient deduplication, data blocks with similar LSH values ​​are not considered candidates for compression, so the additional fingerprint information search process for LSH does not introduce significant overhead.

[0150] Since the anti-collision probability of xxhash+LSH is sufficient to ensure that there will be no misjudgment in the fingerprint deduplication of data blocks, the capacity layer can directly determine that the deduplication is redundant when the first half of the three-level fingerprint information is the same (i.e., the first 128 bits). If the first half is different, the last 128 bits of similar fingerprint information feature values ​​are compared first. If an entry is found in the fingerprint index table whose distance to the fingerprint information feature value similar to that of the data block is less than a given threshold c, the data block corresponding to the entry is determined to be a candidate for an incremental compression reference block. If there are no sufficiently similar fingerprint information feature values, the data block is treated as a unique data block.

[0151] In addition, although relying solely on LSH values ​​for similarity detection has an extremely high probability of misjudgment and missed judgment, the multi-layer storage management system of this application still considers relaxing the standard for similar hash values ​​generated intelligently with reference to LSH values ​​at the capacity layer. Specifically, when the LSH values ​​of two data blocks are the same, the multi-layer storage management system tends to recognize that they are worth implementing incremental compression. At this time, the threshold c will be appropriately enlarged so as not to filter out too many potential candidates. Such a setting not only weakens the missed judgment that may be caused by the overly strict similarity conditions of "clustering + hash learning", but also allows the LSH value to be fully utilized, and can explore more expected benefits.

[0152] (2) Upgrade and delete

[0153] When the fingerprint information level of the storage layer corresponding to the predicted storage location is higher than the fingerprint information level of the storage layer corresponding to the actual storage location, the data block undergoes an upward migration process. Read requests do not trigger intra-layer deduplication. When the fingerprint information of the data block migrates upward, the current fingerprint information is divided into the length of the fingerprint information index field of the storage layer corresponding to the predicted storage location, and then cross-layer deduplication is performed. After the write request modifies the data content, the current fingerprint is first updated, and intra-layer deduplication is performed once. The new fingerprint information is then divided and migrated upward for cross-layer deduplication. After cross-layer deduplication is completed, the data content in the storage layer corresponding to the actual storage location is deleted, ensuring that redundant data in the entire multi-layer storage management system only appears in the storage layer corresponding to the predicted storage location.

[0154] (3) Downgrade and delete

[0155] When the fingerprint information level of the storage layer corresponding to the predicted storage location is lower than that of the storage layer corresponding to the actual storage location, the data block undergoes a downward migration process. Read requests do not trigger intra-layer deduplication. When migrating downward, the data block's fingerprint information requires additional processing by the fingerprint calculation module in the storage layer corresponding to the actual storage location. Redundancy detection is then performed using the processed fingerprint information. After modifying the data content, write requests first update the current fingerprint, perform intra-layer deduplication, and then migrate the updated data content downward for cross-layer deduplication.

[0156] Additionally, when the data block storage layer is upgraded and data replacement occurs in the upper layer, the eliminated data will also passively trigger a series of downward deduplication operations. Although the additional fingerprint information generation overhead is usually borne by the relatively low storage layer, it can alleviate the resource pressure of the performance layer to a certain extent, but every time the data is migrated downward, the LSH value is calculated in real time and even the clustering and fingerprint feature generation model is run to generate fingerprint information. This usually significantly increases the response time of the tiered storage management system, for scenarios with sufficient capacity and latency sensitivity. Therefore, it is possible to consider maintaining a global fingerprint index table in the capacity layer to store the three-level fingerprint corresponding to each data block. This can optimize the performance of the tiered storage management system through batch processing. When a write request causes a data block to be updated, the cache area can also be introduced to asynchronously perform delayed fingerprint updates.

[0157] In order to verify the effectiveness of the deduplication and storage scheme in the above-mentioned hierarchical storage management system, this application evaluated the proportion of duplicate content between different storage layers in the hierarchical storage management system on block-level deduplication datasets (e.g., FIU IODedup traces) and server user datasets (e.g., Inotify). The results show that, in general, the smaller the difference in the amount of data stored between different storage layers, the more obvious the impact on performance when cross-layer redundancy occurs.

[0158] Based on the above analysis, it can be concluded that the hierarchical storage management system designed in this application uses a variable-length fingerprint structure to discover potential similar data blocks and complete incremental compression, thereby improving the storage efficiency of hot data and warm data in the hierarchical storage management system, allowing the storage layer to maintain more valuable data, thereby reducing the overall access latency of the hierarchical storage management system and achieving overall space savings when the hierarchical storage management system stores data.

[0159] Based on the inventive concept of the above embodiments, the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method described in the above embodiments when executing the computer program. Figure 9 Provide detailed explanation.

[0160] like Figure 9 As shown in FIG, the electronic device 100 of the present application is shown, which may specifically include a processor 110 and a memory 120. The memory 120 is coupled to the processor 110.

[0161] The processor 110 is used to control the operation of the electronic device. The processor 110 may also be referred to as a CPU (Central Processing Unit). The processor 110 may be an integrated circuit chip with signal processing capabilities. The processor 110 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. A general-purpose processor may be a microprocessor, or the processor 110 may be any conventional processor.

[0162] The memory 120 is used to store computer programs and can be RAM, ROM, or other types of storage terminals. Specifically, the memory 120 may include one or more computer-readable storage media, which may be non-transitory or transient. The memory 120 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage terminals and flash memory storage terminals. In some embodiments, the non-transitory computer-readable storage medium in the memory 120 is used to store at least one program code.

[0163] The processor 110 is configured to execute the computer program stored in the memory 120 to implement the methods described in the various method embodiments of the present application.

[0164] In some embodiments, the electronic device may further include a peripheral terminal interface 130 and at least one peripheral terminal. The processor 110, memory 120, and peripheral terminal interface 130 may be connected via a bus or signal lines. Each peripheral terminal may be connected to the peripheral terminal interface 130 via a bus, signal lines, or circuit boards. Specifically, the peripheral terminal includes at least one of a radio frequency circuit 140, a display screen 150, an audio circuit 160, and a power supply 170.

[0165] The peripheral terminal interface 130 can be used to connect at least one peripheral terminal related to I / O (Input / Output) to the processor 110 and the memory 120. In some embodiments, the processor 110, the memory 120, and the peripheral terminal interface 130 are integrated on the same chip or circuit board; in some other implementations, any one or two of the processor 110, the memory 120, and the peripheral terminal interface 130 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0166] The RF circuit 140 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 140 communicates with communication networks and other IoT devices via electromagnetic signals, and is therefore the communication circuitry of the electronic device. The RF circuit 140 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. Optionally, the RF circuit 140 includes an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, an operator identification module card, and the like. The RF circuit 140 can communicate with other terminals via at least one wireless communication protocol. Such wireless communication protocols include, but are not limited to, the World Wide Web, metropolitan area networks, intranets, various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks, and / or WiFi (Wireless Fidelity) networks. In some embodiments, the RF circuit 140 may also include circuitry related to Near Field Communication (NFC), although this application does not limit this.

[0167] The display screen 150 is used to display a user interface (UI). The UI may include graphics, text, icons, videos, or any combination thereof. When the display screen 150 is a touch screen display, the display screen 150 is also capable of collecting touch signals on or above the surface of the display screen 150. The touch signals can be input as control signals to the processor 110 for processing. In this case, the display screen 150 can also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, there can be one display screen 150, disposed on the front panel of the electronic device; in other embodiments, there can be at least two display screens 150, disposed on different surfaces of the electronic device or in a foldable design; in still other embodiments, the display screen 150 can be a flexible display, disposed on a curved or foldable surface of the electronic device. Furthermore, the display screen 150 can be configured as a non-rectangular irregular shape, i.e., a special-shaped screen. The display screen 150 can be made of materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).

[0168] The audio circuit 160 may include a microphone and a speaker. The microphone is used to collect sound waves from the operator and the environment, and convert the sound waves into electrical signals and input them into the processor 110 for processing, or input them into the radio frequency circuit 140 to achieve voice communication. For the purpose of stereo acquisition or noise reduction, there can be multiple microphones, which are respectively set in different parts of the electronic device. The microphone can also be an array microphone or an omnidirectional acquisition microphone. The speaker is used to convert the electrical signal from the processor 110 or the radio frequency circuit 140 into sound waves. The speaker can be a traditional thin film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert the electrical signal into sound waves audible to humans, but also convert the electrical signal into sound waves inaudible to humans for purposes such as ranging. In some embodiments, the audio circuit 160 may also include a headphone jack.

[0169] Power supply 170 is used to power various components in the electronic device. Power supply 170 can be AC ​​power, DC power, a disposable battery, or a rechargeable battery. When power supply 170 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery that is charged via a wired line, while a wireless rechargeable battery is a battery that is charged via a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0170] For a detailed description of the functions and execution processes of the various functional modules or components in the electronic device embodiments of the present application, reference can be made to the descriptions in the above-mentioned method embodiments of the present application, which will not be repeated here.

[0171] In the several embodiments provided in this application, it should be understood that the disclosed electronic devices and methods can be implemented in other ways. For example, the various embodiments of the electronic devices described above are merely illustrative. For example, the division of modules or units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some data can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0172] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0173] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0174] Based on the inventive concept of the above embodiments, the present application further provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the steps of the method described in any of the above embodiments are described. Figure 10 The execution process of the above embodiment in a computer-readable storage medium is described.

[0175] like Figure 10 As shown, it shows the computer-readable storage medium of the present application. If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium 200. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions / computer programs for making an IoT device (which can be a personal computer, server, or network terminal, etc.) or a processor (processor) perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes various media such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, as well as electronic terminals such as computers, mobile phones, laptops, tablet computers, cameras, etc. having the above-mentioned storage medium.

[0176] The description of the execution process of the program data in the computer-readable storage medium can refer to the description in the above-mentioned method embodiments of the present application, and will not be repeated here.

[0177] The above description is merely an embodiment of the present application and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

[0178] Those skilled in the art will understand that in the above-mentioned method of the specific implementation method, the writing order of each step does not mean a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

Claims

1. A hierarchical storage management system, characterized in that: The system comprises: A management device, configured to perform hierarchical prediction on a target data block to obtain a predicted storage location of the target data block, and compare the predicted storage location of the target data block with an actual storage location of a reference data block to obtain a position comparison result; A storage device comprising a plurality of storage layers, each of which is used to store reference fingerprint information of a corresponding level of a reference data block obtained according to a preset variable-length fingerprint structure, calculate target fingerprint information of a corresponding level of a target data block according to the variable-length fingerprint structure, and incrementally update and store the reference fingerprint information based on the position comparison result, wherein the target fingerprint information includes a target fingerprint and its corresponding data content, the reference fingerprint information includes a reference fingerprint and its corresponding data content, the variable-length fingerprint structure specifies different fingerprint information levels for different storage layers, and the incremental update indicates comparing the reference fingerprint information with the target fingerprint information, deleting consistent data content in the target fingerprint information that is identical to that in the reference fingerprint information, and storing differential data content in the target fingerprint information that is different from that in the reference fingerprint information.

2. The hierarchical storage management system according to claim 1, wherein: The management device includes a fingerprint level lifting layer, and The fingerprint level promotion layer is configured to: when the predicted storage location is different from the actual storage location, obtain the updated reference fingerprint information and perform level transformation on it, and migrate the level-transformed reference fingerprint information to the storage layer corresponding to the predicted storage location; Each of the storage layers is configured to: store the updated reference fingerprint information when the predicted storage location is the same as the actual storage location, or delete the updated reference fingerprint information of the storage layer after the fingerprint level promotion layer obtains the updated reference fingerprint information stored in the storage layer when the predicted storage location is different from the actual storage location.

3. The hierarchical storage management system according to claim 1, wherein: The management device further includes: Intelligent stratification, for performing stratified prediction on target data blocks using a pre-trained intelligent stratification model to obtain a predicted storage location of the target data blocks; The storage position comparison layer is used to compare the predicted storage position of the target data block with the actual storage position of the reference data block to obtain a position comparison result.

4. The hierarchical storage management system according to claim 2, wherein: The variable-length fingerprint structure includes: a primary fingerprint, a secondary fingerprint, and a tertiary fingerprint; the reference fingerprint information includes: primary reference fingerprint information, secondary reference fingerprint information, and tertiary reference fingerprint information; the storage device includes a performance layer, an intermediate layer, and a capacity layer; and: The performance layer is configured to: store the updated first-level reference fingerprint information when both the predicted storage location and the actual storage location are the performance layer, or delete the updated first-level reference fingerprint information after the fingerprint level promotion layer obtains the updated first-level reference fingerprint information stored in the performance layer when the predicted storage location is the intermediate layer or the capacity layer and the actual storage location is the performance layer; Alternatively, the intermediate layer is configured to: store the updated secondary reference fingerprint information when both the predicted storage location and the actual storage location are in the intermediate layer; or, when the predicted storage location is the performance layer or the capacity layer and the actual storage location is in the intermediate layer, delete the updated secondary reference fingerprint information after the fingerprint level promotion layer obtains the updated secondary reference fingerprint information stored in the intermediate layer; Alternatively, the capacity layer is configured to: store the updated third-level reference fingerprint information when the predicted storage location and the actual storage location are both the capacity layer; or delete the updated third-level reference fingerprint information after the fingerprint level promotion layer obtains the updated third-level reference fingerprint information stored in the capacity layer when the predicted storage location is the performance layer or the middle layer and the actual storage location is the capacity layer.

5. The hierarchical storage management system according to claim 4, characterized in that: Fingerprint level lifting layers include: a fingerprint upgrade module for, when the predicted storage location is the performance layer and the actual storage location is the intermediate layer or the capacity layer, obtaining the updated secondary reference fingerprint information or the tertiary reference fingerprint information of the intermediate layer or the capacity layer, performing upgrade processing on the same, and migrating the upgraded secondary reference fingerprint information or the tertiary reference fingerprint information to the performance layer; or, when the predicted storage location is the intermediate layer and the actual storage location is the capacity layer, obtaining the updated tertiary reference fingerprint information of the capacity layer, performing upgrade processing on the same, and migrating the upgraded tertiary reference fingerprint information to the intermediate layer; A fingerprint downgrade module is used to obtain the updated first-level reference fingerprint information or the second-level reference fingerprint information of the performance layer or the intermediate layer and downgrade it when the predicted storage location is the capacity layer and the actual storage location is the performance layer or the intermediate layer, and migrate the downgraded first-level reference fingerprint information or the second-level reference fingerprint information to the capacity layer; or to obtain the updated first-level reference fingerprint information of the performance layer and downgrade it when the predicted storage location is the intermediate layer and the actual storage location is the performance layer, and migrate the downgraded first-level reference fingerprint information to the intermediate layer.

6. The hierarchical storage management system according to claim 5, characterized in that: The target fingerprint information includes: first-level target fingerprint information, second-level target fingerprint information, and third-level target fingerprint information, and The performance layer includes: A first-level fingerprint calculation module, configured to calculate the first-level target fingerprint information using a preset first hash function; The intermediate layer includes: a secondary fingerprint calculation module, configured to calculate first intermediate fingerprint information of the target data block using a preset second hash function, and concatenate the primary target fingerprint information and the first intermediate fingerprint information to obtain the secondary target fingerprint information; The capacity layer includes: The third-level fingerprint calculation module is used to calculate the second intermediate fingerprint information of the target data block using a pre-trained fingerprint feature generation model, and to concatenate the second-level target fingerprint information and the second intermediate fingerprint information to obtain the third-level target fingerprint information.

7. The hierarchical storage management system according to claim 4, wherein: The secondary target fingerprint information includes the secondary target fingerprint and its corresponding data content, and the tertiary target fingerprint information includes the tertiary target fingerprint and its corresponding data content, and: The intermediate layer further comprises: a first compression module, configured to compress and store differential data content in the secondary target fingerprint information that is different from the secondary reference fingerprint information using a preset first data structure; The capacity layer further includes: The second compression module is configured to compress and store the differential data content in the third-level target fingerprint information and the third-level reference fingerprint information using a preset second data structure.

8. A method for deleting duplicate data, characterized in that: The method comprises: The hierarchical storage management system according to claims 1 to 7 responds to a data block access request and obtains a target data block; The hierarchical storage management system incrementally updates and stores the reference data blocks stored in itself based on the target data blocks.

9. An electronic device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the steps of the method according to claim 8 are implemented when the processor executes the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to claim 8 are implemented.

Images