A tiered storage and deduplication method and system

Abstract

The application provides a layered storage and deduplication method and system, which comprises the following steps: monitoring characteristic information of each file; obtaining historical characteristic information and historical layering level of the file in the last period, training and updating an intelligent layering model to obtain an intelligent layering classifier; obtaining target characteristic information of the file in the current period, and predicting a target layering level corresponding to each file; querying data blocks of each file, deleting duplicate data blocks, and migrating the remaining data blocks of each file to the corresponding storage layer. The model has low complexity, short training time, good layering effect and high layering efficiency. By reasonably layering the files, the missing rate of data access and the delay of data access are reduced; by global deduplication, cross-layer data redundancy is eliminated, and storage space is further saved; and the application can update the training model according to the changing workload, adapt to different access modes, and has strong expansibility.

Description

Technical Field

[0001] This invention relates to the fields of information storage and computer technology, and in particular to a hierarchical storage and deduplication method and system. Background Technology

[0002] To manage increasingly complex and valuable data assets, tiered storage can achieve a good trade-off between performance and cost-effectiveness by managing the entire data lifecycle. To cope with data overload, a data tiering strategy is needed to integrate with a modern tiered storage infrastructure.

[0003] At the same time, deduplication has become a popular feature in modern storage systems, reducing the need for large storage spaces by eliminating duplicate data. However, the granularity, precision, and complexity of deduplication can vary depending on the application scenario. Summary of the Invention

[0004] This invention provides a hierarchical storage and deduplication method and system, which solves the problems of low efficiency, reduced system performance, unbalanced hierarchical effect and cost, wasted storage space and ineffective deduplication in the prior art.

[0005] To address the aforementioned technical problems, the present invention provides a hierarchical storage and deduplication method, comprising the following steps: real-time monitoring of all files in the storage module, recording and storing feature information corresponding to each file; obtaining historical feature information and historical hierarchical levels corresponding to each file in the previous period, constructing an intelligent hierarchical model, training and updating the intelligent hierarchical model based on the historical feature information and historical hierarchical levels to obtain a corresponding intelligent hierarchical classifier; obtaining target feature information corresponding to each file in the current period, predicting the target hierarchical level corresponding to each file in the current period using the intelligent hierarchical classifier based on the target feature information; querying the data blocks of each file in the storage module, deleting duplicate data blocks, and migrating the remaining data blocks of each file to the corresponding storage layer in the storage module according to the target hierarchical level.

[0006] In some embodiments, the real-time monitoring of all files in the storage module, recording and storing the characteristic information corresponding to each file, specifically includes: monitoring all file opening operations, recording and storing the characteristic information corresponding to each file in key-value (KV) format; wherein, the characteristic information includes the absolute file path, file size, file access frequency, time interval between the two most recent file accesses, access frequency of the parent folder where the file is located, and file type.

[0007] In some embodiments, after the step of recording and storing the feature information corresponding to each file, the method further includes the step of designing a hierarchical standard for the files based on the inherent information of the storage module and the feature information.

[0008] In some embodiments, designing the file layering criteria based on the inherent information of the storage module and the feature information specifically includes: calculating the average data block content of all files in the storage module; defining the re-access distance of the file based on the inherent information of the storage module, the feature information, and the average data block content; calculating the re-access level threshold for dividing different storage layers in the storage module; if the re-access distance of the file is greater than the re-access level threshold of the storage layer, it means that the file does not belong to the corresponding storage layer.

[0009] In some embodiments, training and updating the intelligent stratification model based on the historical feature information and historical stratification levels specifically includes: predicting the predicted stratification level corresponding to each file in the previous period using the intelligent stratification model based on the historical feature information; comparing the predicted stratification level with the historical stratification level; adjusting the parameters of the intelligent stratification model based on the comparison result; and updating the intelligent stratification model.

[0010] In some embodiments, predicting the target hierarchical level corresponding to each file in the current period using the intelligent hierarchical classifier based on the target feature information specifically includes: inputting the target feature information into the intelligent hierarchical classifier, predicting the target hierarchical level corresponding to each file in the current period, obtaining the data hierarchical levels corresponding to multiple data blocks of each file; and using an error correction table to update and save the data hierarchical levels corresponding to the incorrectly predicted data blocks.

[0011] In some embodiments, querying the data blocks of each file in the storage module and deleting duplicate data blocks specifically includes: performing a hash calculation on each data block to obtain fingerprint information corresponding to the data block; for multiple data blocks with the same fingerprint information, retaining the physical address corresponding to the storage of one of the data blocks, and establishing an index pointing to the physical address for the remaining data blocks respectively.

[0012] In some embodiments, migrating the remaining data blocks of each file to the corresponding storage layer in the storage module according to the target hierarchical level specifically includes: querying multiple storage layers in the storage module, migrating the data blocks from the storage layer where the previous cycle was located to the storage layer indicated by the target hierarchical level; using the migrated storage layer as the unique address of the data blocks, traversing all the storage layers, deleting data blocks with the same fingerprint information as the migrated data blocks, and establishing an index pointing to the migrated storage layer.

[0013] This invention also provides a hierarchical storage and deduplication system, comprising: a feature unit for real-time monitoring of all files in a storage module and recording feature information corresponding to each file; a hierarchical unit for acquiring historical feature information and historical hierarchical levels corresponding to each file in the previous period, constructing an intelligent hierarchical model, training and updating the intelligent hierarchical model based on the historical feature information and historical hierarchical levels to obtain a corresponding intelligent hierarchical classifier; further configured to acquire target feature information corresponding to each file in the current period, and predict the target hierarchical level corresponding to each file in the current period using the intelligent hierarchical classifier based on the target feature information; a storage unit for storing the files and the feature information corresponding to each file; and a post-processing unit for querying data blocks of each file in the storage module, deleting duplicate data blocks, and migrating the remaining data blocks of each file to the corresponding storage layer in the storage module according to the target hierarchical level.

[0014] In some embodiments, the hierarchical unit includes an acquisition module, a training module, an update module, and a prediction module; the storage unit includes a storage module and a management module, the storage module including a first storage layer, a second storage layer, and a third storage layer, the second and third storage layers each including an error correction table, the management module including a metadata management submodule and a data block management submodule; the post-processing unit includes a deduplication module and a migration module; the acquisition module is used to acquire the feature information corresponding to each file; the training module is used to train the intelligent hierarchical model; the update module is used to update the intelligent hierarchical model; the prediction module is used to predict the target hierarchical level corresponding to each file; the storage module is used to store all the files and the feature information corresponding to each file; the management module is used to manage all the files and the feature information corresponding to each file; the deduplication module is used to query the data blocks of each file in the storage module and delete duplicate data blocks; the migration module is used to migrate the remaining data blocks of each file to the corresponding storage layer in the storage module according to the target hierarchical level.

[0015] The beneficial effects of this invention are as follows: This invention discloses a hierarchical storage and deduplication method and system. The method includes the following steps: real-time monitoring of all files in the storage module, recording and storing the feature information corresponding to each file; obtaining historical feature information and historical hierarchical levels corresponding to each file in the previous period, constructing an intelligent hierarchical model, training and updating the intelligent hierarchical model based on the historical feature information and historical hierarchical levels to obtain a corresponding intelligent hierarchical classifier; obtaining target feature information corresponding to each file in the current period, predicting the target hierarchical level corresponding to each file in the current period based on the target feature information and through the intelligent hierarchical classifier; querying the data blocks of each file in the storage module, deleting duplicate data blocks, and migrating the remaining data blocks of each file to the corresponding storage layer in the storage module according to the target hierarchical level. The intelligent hierarchical model of this application has low complexity and short training time, resulting in not only good hierarchical effect but also high hierarchical efficiency, further improving the overall performance of the system. By rationally layering file data, the missing data rate and latency of data access are effectively reduced. At the same time, global deduplication eliminates cross-layer data redundancy, ensuring that data blocks occupy a unique physical address in all storage layers, further saving storage space. Moreover, this application can update the training model according to changing workloads to adapt to different access patterns. By monitoring the feature information of files, this method can be extended to local storage systems with any number of layers, demonstrating strong scalability. Attached Figure Description

[0016] Figure 1This is a flowchart illustrating the hierarchical storage and deduplication method of this application;

[0017] Figure 2 This is a schematic diagram of the hierarchical storage and deduplication system of this application. Detailed Implementation

[0018] To facilitate understanding of the present invention, a more detailed description is provided below with reference to the accompanying drawings and specific embodiments. Preferred embodiments of the invention are shown in the drawings. However, the invention can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided to provide a thorough and complete understanding of the disclosure of the invention.

[0019] It should be noted that, unless otherwise defined, all technical and scientific terms used in this specification have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. The term "and / or" as used in this specification includes any and all combinations of one or more of the associated listed items.

[0020] The data tiering strategy involved in this application stores frequently accessed critical data on high-end storage media, while archiving less critical data to more economical solutions.

[0021] Artificial intelligence-driven adaptive intelligent hierarchical storage utilizes machine learning models to identify patterns, summarize rules, and make predictions, continuously improving performance with minimal human intervention. This application aims to solve the challenges faced by intelligent hierarchical storage methods, such as the trade-off between hierarchical effects and costs, as well as the difficulties in deduplication. Figure 1 The flowchart illustrating the implementation of the hierarchical storage and deduplication method provided in this application is shown, including the following steps:

[0022] S1: Monitor all files in the storage module in real time, and record and store the characteristic information corresponding to each file.

[0023] S2: Design file hierarchical standards based on the inherent and characteristic information of the storage module.

[0024] S3: Obtain the historical feature information and historical stratification level of each file in the previous period, construct an intelligent stratification model, train and update the intelligent stratification model based on the historical feature information and historical stratification level, and obtain the corresponding intelligent stratification classifier.

[0025] S4: Obtain the target feature information corresponding to each file in the current period, and predict the target hierarchical level corresponding to each file in the current period based on the target feature information through an intelligent hierarchical classifier.

[0026] S5: Query the data blocks of each file in the storage module, delete duplicate data blocks, and migrate the remaining data blocks of each file to the corresponding storage layer in the storage module according to the target layer level.

[0027] The intelligent hierarchical model of this application has low complexity and short training time, achieving both good hierarchical effect and high efficiency, further improving the overall performance of the system. By rationally hierarchically layering file data, it effectively reduces the missing data rate and data access latency; simultaneously, global deduplication eliminates cross-layer data redundancy, ensuring that data blocks occupy a unique physical address across all storage layers, further saving storage space; moreover, this application can update the training model according to changing workloads, adapting to different access patterns; by monitoring file feature information, the method can be extended to local storage systems with arbitrarily multiple layers, demonstrating strong scalability.

[0028] The following is combined Figures 1 to 2 The present application will be further described in detail with reference to specific embodiments.

[0029] like Figure 1 As shown in the embodiments of this application, the hierarchical storage and deduplication method provided is described in detail below:

[0030] S1: Monitor all files in the storage module in real time, and record and store the characteristic information corresponding to each file.

[0031] Combination Figure 2 As shown, the storage module 31 of this application includes a first storage layer 301, a second storage layer 302, and a third storage layer 303. Both the second storage layer 302 and the third storage layer 303 include error correction tables 306. In this embodiment, from the first storage layer 301 to the second storage layer 302, and then to the third storage layer 303, the storage capacity gradually increases, while the data access rate gradually decreases. As one optional embodiment, the first storage layer 301, the second storage layer 302, and the third storage layer 303 of this application can be DRAM (Dynamic Random Access Memory), NVMe SSD (Non-Volatile Memory), and SATA SSD (SATA Interface Solid State Drive), respectively.

[0032] Specifically, through feature unit 1, the monitoring tool (inotify-tools) is used to monitor the opening operations of all files in storage module 31, record and store the feature information corresponding to each file in KV format.

[0033] The feature information includes the file's absolute path, file size, file access frequency, the time interval between the file's last two accesses, the access frequency of the file's parent folder, and file type. It should be noted that the file's absolute path refers to the complete path describing the file's location. The file size refers to the amount of space occupied by the file in storage module 31. The file access frequency refers to the number of times the file is accessed within a certain period (e.g., within a week). The access frequency of the file's parent folder refers to the number of times the file's parent folder is accessed within a certain period. The file type refers to a file's content, format, and purpose.

[0034] The above-mentioned feature information takes into account both temporal limitations and spatial locality, and reflects the characteristics of the document more comprehensively.

[0035] Specifically, the file's characteristic information is stored in storage module 31 in key-value (KV) format. KV format refers to K (absolute file path) - V (file size, file access frequency, time interval between the last two accesses, access frequency of the file's parent folder, and file type). KV format is a data storage method where "K" represents the key and "V" represents the value. This data storage method organizes information in key-value pairs, where each key uniquely identifies a file, and the associated value is the actual file content.

[0036] S2: Design file hierarchical standards based on the inherent and characteristic information of the storage module.

[0037] The inherent information of a storage module includes storage capacity, access speed, power consumption, reliability, and cost. A file within a storage module can be divided into multiple data blocks.

[0038] Specifically, in combination Figure 2 As shown, the average data block content of all files in storage module 31 is calculated. The above process is expressed by the following formula:

[0039]

[0040] Where b represents the average data block content, F j Indicates the size of file j, weight j The weight of file j is represented by , k represents the size of the data block in the storage layer, interval represents the time interval between the two most recent accesses, freq represents the access frequency, w1 represents the weight of the time interval between the two most recent accesses, w2 represents the weight of the access frequency freq, t represents the t-th file, and n represents the total number of files.

[0041] Furthermore, based on the inherent information, characteristic information, and average data block content of storage module 31, the file re-access distance is defined, and the re-access level threshold for different storage layers in storage module 31 is calculated. Here, the re-access distance refers to the total number of file accesses occurring in the entire system between two consecutive accesses of the same file. The re-access level threshold can be expressed by the following formula:

[0042]

[0043] Among them, C i Let k represent the storage capacity of the i-th storage layer, k represent the size of the data blocks in the storage layer, b represent the average data block content, and h represent the storage capacity of the i-th storage layer. i denoted by , where represents the hit rate of the i-th storage layer, and p represents the proportion of files accessed no more than once within the statistical period.

[0044] Furthermore, the tiering criterion for files is: if the reaccess distance of a file is greater than the reaccess level threshold of the storage layer, then the file does not belong to the corresponding storage layer.

[0045] For example, if the reaccess level threshold of the second storage layer 302 is 3, and the reaccess distance of one of the files is 5, which is greater than the reaccess level threshold of the second storage layer 302, then it means that the file does not belong to the second storage layer 302 and should not be allocated to this layer.

[0046] S3: Obtain the historical feature information and historical stratification level of each file in the previous period, construct an intelligent stratification model, train and update the intelligent stratification model based on the historical feature information and historical stratification level, and obtain the corresponding intelligent stratification classifier.

[0047] Specifically, in combination Figure 2 As shown, the historical feature information and historical hierarchical level corresponding to each file in the previous cycle in the storage module 31 are obtained through the acquisition module 21 in the hierarchical unit 2.

[0048] The "previous cycle" refers to the last time a file was stratified. A cycle can be one week or one month, depending on the need. The "historical stratification level" refers to the storage stratification level to which the file was assigned in the previous cycle; for example, if the file was assigned to the first storage stratum 301, its historical stratification level is 1; if the file was assigned to the second storage stratum 302, its historical stratification level is 2.

[0049] Furthermore, an intelligent hierarchical model is constructed. In this embodiment, the intelligent hierarchical model is a lightweight decision tree model. The decision tree model is a simple and easy-to-use nonparametric classifier that does not require any prior assumptions about the data, has a fast computation speed, easy-to-interpret results, and strong robustness, balancing high prediction accuracy with low latency.

[0050] In this embodiment, the complete training set (such as the historical feature information corresponding to each file in the previous cycle) is used as the root node of the decision tree model. Each time a child node is split, the feature with the largest information gain among all historical feature information is selected as the splitting criterion for this layer. The number of child nodes is reduced by using a binary splitting method to reduce the probability of overfitting. The splitting continues until the maximum decision tree height is defined or a certain child node belongs to the same layer level.

[0051] For example, in a training set D, file access frequency is used as the feature with the highest information gain. A threshold T is chosen as the classification criterion to maximize the information gain of the partitioned subsets D1 and D2. The training set D will then be divided into a subset D2 with file access frequencies greater than the threshold T and a subset D1 with file access frequencies less than the threshold T, where subsets D1 and D2 can distinguish different hierarchical levels to the greatest extent possible. This process is recursively repeated until the decision tree reaches its maximum height or a subset has identical hierarchical levels.

[0052] Training module 22 inputs historical feature information corresponding to each file from the previous period into the intelligent hierarchical model. The intelligent hierarchical model then predicts the predicted hierarchical level for each file from the previous period. The predicted hierarchical level is compared with the historical hierarchical level, and the parameters of the intelligent hierarchical model are adjusted based on the comparison results. Finally, the intelligent hierarchical model is updated via update module 23.

[0053] In this embodiment, the parameters of the intelligent hierarchical model include the ratio of training data to prediction data, the hierarchical standard adopted, the decision tree height, the minimum number of samples that can be divided at a node, and the minimum number of samples at a leaf node.

[0054] For example, based on historical feature information, the intelligent stratification model predicts that a file in the previous period will have a predicted stratification level of 2 (meaning the file is predicted to be allocated to the second storage layer 302), while its historical stratification level is 1 (meaning the file was actually allocated to the first storage layer 301). The comparison shows that the predicted stratification level and the historical stratification level are inconsistent, indicating that the intelligent stratification model's prediction is incorrect. Therefore, the parameters of the intelligent stratification model need to be adjusted, and the model needs to be updated.

[0055] After multiple training, adjustments, and updates, a corresponding intelligent hierarchical classifier is obtained. At this point, the intelligent hierarchical classifier can accurately predict the most suitable storage layer level to which a file should be allocated.

[0056] S4: Obtain the target feature information corresponding to each file in the current period, and predict the target hierarchical level corresponding to each file in the current period based on the target feature information through an intelligent hierarchical classifier.

[0057] Specifically, in combination Figure 2 As shown, the target feature information corresponding to each file in the current period in the storage module 31 is obtained through the acquisition module 21 in the hierarchical unit 2.

[0058] Furthermore, the prediction module 24 inputs the target feature information corresponding to each file in the current period into the intelligent hierarchical classifier, and the intelligent hierarchical classifier predicts the target hierarchical level corresponding to each file in the current period.

[0059] For example, if the intelligent hierarchical classifier predicts that the target hierarchical level of one of the files in the current period is 2 based on the input target feature information, then the file needs to be allocated to the second storage layer 302.

[0060] Furthermore, based on the target hierarchical level corresponding to each file, the data hierarchical level corresponding to multiple data blocks of each file is obtained. In this embodiment, the target hierarchical level corresponding to each file is used as the data hierarchical level corresponding to multiple data blocks of each file. For example, if the target hierarchical level of one file is 1, then the data hierarchical level of multiple data blocks of that file is also 1.

[0061] Furthermore, since the prediction results of the intelligent hierarchical classifier do not completely match the actual situation, and the access of data blocks and files is not strictly equal, an error correction table 306 is set up for both the second storage layer 302 and the third storage layer 303 to give data blocks that are misjudged as not qualified to enter the first storage layer 301 the opportunity to upgrade the storage layer. That is, only data blocks with a data hierarchical level of 2 or 3 are corrected.

[0062] Specifically, using error correction table 306, a FIFO (First Input First Output) sequence is maintained to record the fingerprint information and corresponding data hierarchical level of the data blocks that are predicted incorrectly. The data hierarchical level corresponding to the data blocks that are predicted incorrectly is sent to post-processing unit 4 for further processing.

[0063] The target hierarchical level corresponding to each file in the current period and the data hierarchical level corresponding to multiple data blocks of each file are sent to the post-processing unit 4 for further processing.

[0064] S5: Query the data blocks of each file in the storage module, delete duplicate data blocks, and migrate the remaining data blocks of each file to the corresponding storage layer in the storage module according to the target layer level.

[0065] Specifically, in combination Figure 2 As shown, the deduplication module 41 in the post-processing unit 4 performs hash calculation on each data block in the storage module 31 using the hash function (SHA-256) to obtain the fingerprint information corresponding to the data block.

[0066] For multiple data blocks with the same fingerprint information, one physical address in the storage module 31 is reserved to store the actual data of one of the data blocks. For the remaining data blocks, an index pointing to the physical address is created respectively, and the duplicate actual data of the remaining data blocks is deleted to save the corresponding storage space.

[0067] Furthermore, the system queries multiple storage layers in storage module 31 and migrates data blocks from the storage layer where the previous cycle was located to the storage layer indicated by the target layer level.

[0068] Specifically, when accessing the first storage layer 301 using a data block, if a match is found, it means that the storage layer where the data block was located in the previous cycle is the first storage layer 301. Therefore, there is no need to perform intelligent stratification through the intelligent stratification classifier, and the data block will still reside in the first storage layer 301. At the same time, the data block management submodule 305 updates the data blocks in the first storage layer 301 with an LRU strategy, and migrates data blocks with low access frequency out of the first storage layer 301.

[0069] It's important to note that the LRU (Least Recently Used) policy is a cache eviction strategy. Its basic principle is to evict the least recently used data. When the cache is full, it will first discard the least recently used item to make room for new elements.

[0070] Furthermore, if the same data block is accessed in the first storage layer 301 and a miss occurs, it means that the storage layer in the previous cycle of the data block was not the first storage layer 301. Then, further processing is performed according to the data layer level corresponding to the data block obtained in S4.

[0071] If the data layer level is 1, the data block is migrated to the first storage layer 301, and the first storage layer 301 is used as the unique address of the data block. All storage layers are traversed, and data blocks with the same fingerprint information as the data block are deleted. An index pointing to the first storage layer 301 is created for data blocks with the same fingerprint information.

[0072] If the data stratification level is 2, then query the error correction table 306 of the second storage layer 302. If it matches, it means that the data stratification level predicted by the intelligent stratification classifier is incorrect. Then, the data block is migrated to the first storage layer 301, and the first storage layer 301 is used as the unique address of the data block. Traverse all storage layers, delete data blocks with the same fingerprint information as the data block, and create an index pointing to the first storage layer 301 for data blocks with the same fingerprint information.

[0073] If the data stratification level is 2 and the error correction table 306 of the second storage layer 302 is not found, it means that the data stratification level predicted by the intelligent stratification classifier is correct. Therefore, the error correction table 306 of the second storage layer 302 is updated. Further querying the second storage layer 302, if a match is found, it means that the storage layer in the previous period where the data block was located was the second storage layer 302. Therefore, the data block is not migrated. Simultaneously, the data block management submodule 305 updates the data blocks in the second storage layer 302 using an LRU strategy, migrating data blocks with low access frequency out of the second storage layer 302. If the query for the second storage layer 302 is not found, it means that the storage layer in the previous period where the data block was located was not the second storage layer 302. Therefore, the data block is migrated to the second storage layer 302, and the second storage layer 302 is used as the unique address of the data block. All storage layers are traversed, and data blocks with the same fingerprint information as the data block are deleted. An index pointing to the second storage layer 302 is created for data blocks with the same fingerprint information.

[0074] If the data stratification level is 3, then query the error correction table 306 of the second storage layer 302. If it is a match, it means that the data stratification level predicted by the intelligent stratification classifier is incorrect. Then, the data block is migrated to the first storage layer 301, and the first storage layer 301 is used as the unique address of the data block. Traverse all storage layers, delete data blocks with the same fingerprint information as the data block, and create an index pointing to the first storage layer 301 for data blocks with the same fingerprint information.

[0075] If the data tier level is 3 and the error correction table 306 of the second storage layer 302 is not found, then the error correction table 306 of the second storage layer 302 is updated; further query the second storage layer 302. If it is found, it means that the storage layer where the data block was located in the previous cycle is the second storage layer 302. Then the data block is not migrated. At the same time, the data block management submodule 305 updates the data blocks in the second storage layer 302 with the LRU strategy and migrates the data blocks with low access frequency out of the second storage layer 302.

[0076] If the query for the second storage layer 302 fails, the error correction table 306 of the third storage layer 303 is further queried. If a match is found, it means the intelligent hierarchical classifier's predicted data hierarchical level is incorrect. The data block is then migrated to the second storage layer 302, and the second storage layer 302 is used as the unique address of the data block. All storage layers are traversed, and data blocks with the same fingerprint information as the data block are deleted. An index pointing to the second storage layer 302 is created for data blocks with the same fingerprint information. If the query for the error correction table 306 of the third storage layer 303 fails, it means the intelligent hierarchical classifier's predicted data hierarchical level is correct, and it also means that the storage layer in the previous cycle where the data block was located was the third storage layer 303, so no migration is needed. Simultaneously, the data block in the third storage layer 303 is updated using an LRU strategy through the data block management submodule 305, and the error correction table 306 of the third storage layer 303 is also updated.

[0077] Corresponding to the hierarchical storage and deduplication method in the above embodiments, this application also provides a hierarchical storage and deduplication system. For ease of explanation, only the parts related to the embodiments of this application are shown.

[0078] like Figure 2 As shown, this tiered storage and deduplication system includes:

[0079] Feature unit 1 is used to monitor all files in the storage module 31 in real time and record the feature information corresponding to each file.

[0080] Hierarchical Unit 2 is used to obtain historical feature information and historical hierarchical level corresponding to each file in the previous period, construct an intelligent hierarchical model, train and update the intelligent hierarchical model based on historical feature information and historical hierarchical level to obtain the corresponding intelligent hierarchical classifier; it is also used to obtain target feature information corresponding to each file in the current period, and predict the target hierarchical level corresponding to each file in the current period through the intelligent hierarchical classifier based on the target feature information.

[0081] Storage unit 3 is used to store files and the characteristic information corresponding to each file.

[0082] The post-processing unit 4 is used to query the data blocks of each file in the storage module 31, delete duplicate data blocks, and migrate the remaining data blocks of each file to the corresponding storage layer in the storage module 31 according to the target hierarchical level.

[0083] Furthermore, the hierarchical unit 2 includes an acquisition module 21, a training module 22, an update module 23, and a prediction module 24; the storage unit 3 includes a storage module 31 and a management module 32. The storage module 31 includes a first storage layer 301, a second storage layer 302, and a third storage layer 303. Both the second storage layer 302 and the third storage layer 303 include an error correction table 306. The management module 32 includes a metadata management submodule 304 and a data block management submodule 305; the post-processing unit 4 includes a deduplication module 41 and a migration module 42; the acquisition module 21 is used to acquire the feature information corresponding to each file. The training module 22 is used to train the intelligent hierarchical model; the update module 23 is used to update the intelligent hierarchical model; the prediction module 24 is used to predict the target hierarchical level corresponding to each file; the storage module 31 is used to store all files and the feature information corresponding to each file; the management module 32 is used to manage all files and the feature information corresponding to each file; the deduplication module 41 is used to query the data blocks of each file in the storage module 31 and delete duplicate data blocks; the migration module 42 is used to migrate the remaining data blocks of each file to the corresponding storage layer in the storage module 31 according to the target hierarchical level.

[0084] Other technical features in the above-described hierarchical storage and deduplication system are the same as those disclosed in the above-described embodiments. They can be referred to the descriptions in the corresponding method embodiments above, and will not be repeated here.

[0085] Therefore, this invention discloses a hierarchical storage and deduplication method and system. The method includes the following steps: real-time monitoring of all files in the storage module, recording and storing the feature information corresponding to each file; obtaining the historical feature information and historical hierarchical level of each file in the previous period, constructing an intelligent hierarchical model, training and updating the intelligent hierarchical model based on the historical feature information and historical hierarchical level to obtain the corresponding intelligent hierarchical classifier; obtaining the target feature information of each file in the current period, predicting the target hierarchical level of each file in the current period using the intelligent hierarchical classifier based on the target feature information; querying the data blocks of each file in the storage module, deleting duplicate data blocks, and migrating the remaining data blocks of each file to the corresponding storage layer in the storage module according to the target hierarchical level. The intelligent hierarchical model of this application has low complexity and short training time, and not only has good hierarchical effect but also high hierarchical efficiency, further improving the overall performance of the system. By rationally layering file data, the missing data rate and latency of data access are effectively reduced. At the same time, global deduplication eliminates cross-layer data redundancy, ensuring that data blocks occupy a unique physical address in all storage layers, further saving storage space. Moreover, this application can update the training model according to changing workloads to adapt to different access patterns. By monitoring the feature information of files, this method can be extended to local storage systems with any number of layers, demonstrating strong scalability.

[0086] The above are merely embodiments of the present invention and do not limit the patent scope of the present invention. Any equivalent structural transformations made based on the content of the present invention specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A hierarchical storage and deduplication method, characterized in that, Including the following steps: All files in the storage module are monitored in real time, and the characteristic information corresponding to each file is recorded and stored. Obtain the historical feature information and historical hierarchical level of each file in the previous period, construct an intelligent hierarchical model, train and update the intelligent hierarchical model according to the historical feature information and historical hierarchical level, and obtain the corresponding intelligent hierarchical classifier. Obtain target feature information corresponding to each file in the current period, and predict the target hierarchical level corresponding to each file in the current period based on the target feature information and the intelligent hierarchical classifier, specifically including: The target feature information is input into the intelligent hierarchical classifier to predict the target hierarchical level corresponding to each file in the current period, and to obtain the data hierarchical level corresponding to multiple data blocks of each file; Using an error correction table, the fingerprint information and corresponding data hierarchy level of the data blocks that are predicted incorrectly are recorded, and the data hierarchy level corresponding to the data blocks that are predicted incorrectly is updated and saved. The data blocks of each file in the storage module are queried, duplicate data blocks are deleted, and the remaining data blocks of each file are migrated to the corresponding storage layer in the storage module according to the target hierarchical level.

2. The hierarchical storage and deduplication method according to claim 1, characterized in that, The real-time monitoring of all files in the storage module, and the recording and storage of characteristic information corresponding to each file, specifically includes: All file opening operations are monitored, and the characteristic information corresponding to each file is recorded and stored in key-value format. The feature information includes the absolute path of the file, file size, file access frequency, time interval between the two most recent accesses of the file, access frequency of the parent folder where the file is located, and file type.

3. The hierarchical storage and deduplication method according to claim 1, characterized in that, After the step of recording and storing the feature information corresponding to each file, the method further includes the following step: Based on the inherent information of the storage module and the feature information, the hierarchical standard for the file is designed.

4. The hierarchical storage and deduplication method according to claim 3, characterized in that, The step of designing the file hierarchy standard based on the inherent information of the storage module and the feature information specifically includes: Calculate the average data block content of all files in the storage module; Based on the inherent information of the storage module, the feature information, and the average data block content, the re-access distance of the file is defined, and the re-access level threshold for different storage layers in the storage module is calculated. If the re-access distance of the file is greater than the re-access level threshold of the storage layer, it means that the file does not belong to the corresponding storage layer.

5. The hierarchical storage and deduplication method according to claim 1, characterized in that, The step of training and updating the intelligent hierarchical model based on the historical feature information and historical hierarchical levels specifically includes: Based on the historical feature information, the intelligent hierarchical model is used to predict the predicted hierarchical level of each file in the previous period; The predicted stratification level is compared with the historical stratification level, and the parameters of the intelligent stratification model are adjusted and updated based on the comparison results.

6. The hierarchical storage and deduplication method according to claim 1, characterized in that, The step of querying the data blocks of each file in the storage module and deleting duplicate data blocks specifically includes: Perform a hash calculation on each of the data blocks to obtain the fingerprint information corresponding to the data block; For multiple data blocks with identical fingerprint information, retain the physical address corresponding to one of the data blocks, and for the remaining data blocks, establish an index pointing to the physical address.

7. The hierarchical storage and deduplication method according to claim 1, characterized in that, The step of migrating the remaining data blocks of each file to the corresponding storage layer in the storage module according to the target hierarchical level specifically includes: Query multiple storage layers in the storage module, and migrate the data block from the storage layer where the previous cycle was located to the storage layer pointed to by the target layer level; Using the migrated storage layer as the unique address of the data block, traverse all the storage layers, delete the data blocks that have the same fingerprint information as the migrated data blocks, and create an index pointing to the migrated storage layer.

8. A hierarchical storage and deduplication system, characterized in that, include: The feature unit is used to monitor all files in the storage module in real time and record the feature information corresponding to each file; The hierarchical unit is used to acquire historical feature information and historical hierarchical levels corresponding to each file in the previous period, construct an intelligent hierarchical model, train and update the intelligent hierarchical model based on the historical feature information and historical hierarchical levels, and obtain a corresponding intelligent hierarchical classifier; it is also used to acquire target feature information corresponding to each file in the current period, and predict the target hierarchical level corresponding to each file in the current period based on the target feature information and the intelligent hierarchical classifier, specifically including: The target feature information is input into the intelligent hierarchical classifier to predict the target hierarchical level corresponding to each file in the current period, and to obtain the data hierarchical level corresponding to multiple data blocks of each file; Using an error correction table, the fingerprint information and corresponding data hierarchy level of the data blocks that are predicted incorrectly are recorded, and the data hierarchy level corresponding to the data blocks that are predicted incorrectly is updated and saved. Storage unit, used to store the files and the feature information corresponding to each file; The post-processing unit is used to query the data blocks of each file in the storage module, delete duplicate data blocks, and migrate the remaining data blocks of each file to the corresponding storage layer in the storage module according to the target hierarchical level.

9. The hierarchical storage and deduplication system according to claim 8, characterized in that, The hierarchical unit includes an acquisition module, a training module, an update module, and a prediction module; the storage unit includes a storage module and a management module, the storage module includes a first storage layer, a second storage layer, and a third storage layer, both of which include error correction tables; the management module includes a metadata management submodule and a data block management submodule; the post-processing unit includes a deduplication module and a migration module. The acquisition module is used to acquire the feature information corresponding to each file; The training module is used to train the intelligent hierarchical model; The update module is used to update the intelligent hierarchical model; The prediction module is used to predict the target hierarchical level corresponding to each file; The storage module is used to store all the files and the feature information corresponding to each file; The management module is used to manage all the files and the feature information corresponding to each file; The deduplication module is used to query the data blocks of each file in the storage module and delete duplicate data blocks. The migration module is used to migrate the remaining data blocks of each file to the corresponding storage layer in the storage module according to the target hierarchical level.

Images