Hierarchical storage and duplicated data deletion method and system

Abstract

The invention provides a hierarchical storage and duplicated data deletion method and system. The method comprises the following steps: monitoring feature information of each file; historical feature information and historical layering levels of the files in the last period are obtained, the intelligent layering model is trained and updated, and an intelligent layering classifier is obtained; obtaining target feature information of the files in the current period, and predicting a target hierarchical level corresponding to each file; and querying the data blocks of each file, deleting repeated data blocks, and migrating the remaining data blocks of each file to the corresponding storage layer. The model complexity is low, the training time is short, the layering effect is good, and the layering efficiency is high. The files are reasonably layered, so that the missing rate of data access and the delay of data access are reduced; cross-layer data redundancy is eliminated through global repeated data deletion, and the storage space is further saved; moreover, the method can update the training model according to the changed workload, is suitable for different access modes, and is high in expansibility.

Description

Technical Field

[0001] The present invention relates to the field of information storage and computer technology, and in particular to a hierarchical storage and duplicate data deletion method and system. Background Art

[0002] To manage increasingly complex and valuable data assets, tiered storage can achieve a good trade-off between performance and cost-effectiveness by managing the entire data lifecycle. To cope with data flooding, a data tiering strategy is needed to be combined with a modern tiered storage infrastructure.

[0003] At the same time, data deduplication has become a popular feature in modern storage systems, reducing the need for large storage space by eliminating duplicate data. However, the granularity, accuracy, and complexity of data deduplication will also change depending on the scenario. Summary of the invention

[0004] The present invention provides a tiered storage and duplicate data deletion method and system, which solves the problems in the prior art of low efficiency of data tiered storage, reduced system performance, unbalanced tiered effect and tiered cost, wasted storage space and being unfavorable for duplicate data deletion.

[0005] In order to solve the above technical problems, the technical solution adopted by the present invention is to provide a hierarchical storage and duplicate data deletion method, including the steps of: real-time monitoring of all files in the storage module, recording and storing feature information corresponding to each of the files; obtaining historical feature information and historical hierarchical levels corresponding to each of the files in the previous cycle, building an intelligent hierarchical model, training and updating the intelligent hierarchical model according to the historical feature information and historical hierarchical levels, and obtaining a corresponding intelligent hierarchical classifier; obtaining target feature information corresponding to each of the files in the current cycle, and predicting the target hierarchical level corresponding to each of the files in the current cycle through the intelligent hierarchical classifier according to the target feature information; querying the data blocks of each of the files in the storage module, deleting the duplicated data blocks, and migrating the remaining data blocks of each of the files to the corresponding storage layer in the storage module according to the target hierarchical level.

[0006] In some embodiments, all files in the storage module are monitored in real time, and characteristic information corresponding to each of the files is recorded and stored, specifically including: monitoring all opening operations of the files, recording and storing the characteristic information corresponding to each of the files in KV form; wherein the characteristic information includes the absolute path of the file, the file size, the file access frequency, the time interval between the two most recent file accesses, the access frequency of the parent folder where the file is located, and the file type.

[0007] In some embodiments, after the step of recording and storing the characteristic information corresponding to each of the files, the method further includes the step of designing a layering standard for the files based on the inherent information of the storage module and the characteristic information.

[0008] In some embodiments, the layering standard of the file is designed based on the inherent information of the storage module and the characteristic information, specifically including: calculating the average data block content of all files in the storage module; defining the reaccess distance of the file based on the inherent information of the storage module, the characteristic information and the average data block content, and calculating the reaccess level thresholds of the different storage layers in the storage module; if the reaccess distance of the file is greater than the reaccess level threshold of the storage layer, it means that the file does not belong to the corresponding storage layer.

[0009] In some embodiments, the intelligent stratification model is trained and updated based on the historical feature information and the historical stratification level, specifically including: predicting the predicted stratification level corresponding to each of the files in the previous cycle through the intelligent stratification model based on the historical feature information; comparing the predicted stratification level with the historical stratification level, adjusting the parameters of the intelligent stratification model according to the comparison result, and updating the intelligent stratification model.

[0010] In some embodiments, the target hierarchical level corresponding to each of the files in the current cycle is predicted through the intelligent hierarchical classifier based on the target feature information, specifically including: inputting the target feature information into the intelligent hierarchical classifier, predicting the target hierarchical level corresponding to each of the files in the current cycle, and obtaining the data hierarchical levels corresponding to the multiple data blocks of each of the files; using an error correction table to update and save the data hierarchical levels corresponding to the data blocks that are incorrectly predicted.

[0011] In some embodiments, the querying of the data blocks of each file in the storage module and the deletion of duplicate data blocks specifically include: performing hash calculation on each data block to obtain fingerprint information corresponding to the data block; for multiple data blocks with the same fingerprint information, retaining the physical address corresponding to the storage of one of the data blocks, and establishing indexes pointing to the physical addresses for the remaining data blocks.

[0012] In some embodiments, the remaining data blocks of each file are migrated to the corresponding storage layer in the storage module according to the target hierarchical level, specifically including: querying multiple storage layers in the storage module, migrating the data blocks from the storage layer in the previous cycle to the storage layer indicated by the target hierarchical level; using the storage layer after migration as the unique address of the data block, traversing all the storage layers, deleting the data blocks with the same fingerprint information as the data blocks after migration, and establishing an index pointing to the storage layer after migration.

[0013] The present invention also provides a hierarchical storage and deduplication system, comprising: a feature unit, used to monitor all files in a storage module in real time and record feature information corresponding to each of the files; a hierarchical unit, used to obtain historical feature information and historical hierarchical levels corresponding to each of the files in the previous cycle, build an intelligent hierarchical model, train and update the intelligent hierarchical model according to the historical feature information and historical hierarchical levels, and obtain a corresponding intelligent hierarchical classifier; and also used to obtain target feature information corresponding to each of the files in the current cycle, and predict the target hierarchical level corresponding to each of the files in the current cycle through the intelligent hierarchical classifier according to the target feature information; a storage unit, used to store the files and the feature information corresponding to each of the files; a post-processing unit, used to query the data blocks of each of the files in the storage module, delete the duplicated data blocks, and migrate the remaining data blocks of each of the files to the corresponding storage layer in the storage module according to the target hierarchical level.

[0014] In some embodiments, the stratification unit includes an acquisition module, a training module, an update module and a prediction module; the storage unit includes a storage module and a management module, the storage module includes a first storage layer, a second storage layer and a third storage layer, the second storage layer and the third storage layer both include error correction tables, and the management module includes a metadata management submodule and a data block management submodule; the post-processing unit includes a deduplication module and a migration module; the acquisition module is used to acquire the feature information corresponding to each of the files; the training module is used to train the intelligent stratification model; the update module is used to update the intelligent stratification model; the prediction module is used to predict the target stratification level corresponding to each of the files; the storage module is used to store all the files and the feature information corresponding to each of the files; the management module is used to manage all the files and the feature information corresponding to each of the files; the deduplication module is used to query the data blocks of each of the files in the storage module and delete the duplicated data blocks; the migration module is used to migrate the remaining data blocks of each of the files to the corresponding storage layer in the storage module according to the target stratification level.

[0015] The beneficial effects of the present invention are as follows: the present invention discloses a hierarchical storage and deduplication method and system, the method comprising the steps of: real-time monitoring of all files in a storage module, recording and storing feature information corresponding to each file; obtaining historical feature information and historical hierarchical level corresponding to each file in the previous cycle, constructing an intelligent hierarchical model, training and updating the intelligent hierarchical model according to the historical feature information and historical hierarchical level, and obtaining a corresponding intelligent hierarchical classifier; obtaining target feature information corresponding to each file in the current cycle, and predicting the target hierarchical level corresponding to each file in the current cycle through an intelligent hierarchical classifier according to the target feature information; querying the data blocks of each file in the storage module, deleting duplicate data blocks, and migrating the remaining data blocks of each file to the corresponding storage layer in the storage module according to the target hierarchical level. The intelligent hierarchical model of the present application has low complexity and short training time, and not only has good hierarchical effect, but also high hierarchical efficiency, further improving the overall performance of the system. By rationally stratifying file data, the data access miss rate and data access latency are effectively reduced; at the same time, cross-layer data redundancy is eliminated through global data deduplication to ensure that data blocks only occupy unique physical addresses in all storage layers, further saving storage space; and the present application can update the training model according to the changing workload to adapt to different access modes; by monitoring the characteristic information of the file, the method can be extended to any number of layers of local storage systems, with strong scalability. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1It is a flowchart of the hierarchical storage and deduplication method of the present application;

[0017] Figure 2 It is a schematic diagram of the composition of the hierarchical storage and deduplication system of the present application. DETAILED DESCRIPTION

[0018] In order to facilitate the understanding of the present invention, the present invention is described in more detail below in conjunction with the accompanying drawings and specific embodiments. Preferred embodiments of the present invention are provided in the accompanying drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described in this specification. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present invention more thorough and comprehensive.

[0019] It should be noted that, unless otherwise defined, all technical and scientific terms used in this specification have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used in this specification includes any and all combinations of one or more related listed items.

[0020] The data tiering strategy involved in this application stores frequently accessed critical data on premium storage media while archiving less critical data to more economical solutions.

[0021] Artificial intelligence-driven adaptive intelligent tiered storage uses machine learning models to identify patterns, summarize rules to make predictions, and continuously improve performance with minimal human intervention. This application strives to solve the problem of balancing tiered effects and tiered costs, as well as the problem of being unfavorable to duplicate data deletion, faced by intelligent tiered storage methods. Figure 1 The flowchart of the implementation of the hierarchical storage and deduplication method provided by the present application is shown, including the steps of:

[0022] S1: Monitor all files in the storage module in real time, and record and store feature information corresponding to each file.

[0023] S2: Design the hierarchical standard of the file based on the inherent information and characteristic information of the storage module.

[0024] S3: Obtain the historical feature information and historical stratification level corresponding to each file in the previous cycle, build an intelligent stratification model, train and update the intelligent stratification model according to the historical feature information and historical stratification level, and obtain the corresponding intelligent stratification classifier.

[0025] S4: Obtain target feature information corresponding to each file in the current cycle, and predict the target hierarchical level corresponding to each file in the current cycle through an intelligent hierarchical classifier based on the target feature information.

[0026] S5: query the data blocks of each file in the storage module, delete duplicate data blocks, and migrate the remaining data blocks of each file to the corresponding storage layer in the storage module according to the target tier level.

[0027] The intelligent tiering model of this application has low complexity and short training time. It not only has good tiering effect, but also high tiering efficiency, which further improves the overall performance of the system. By rationally tiering file data, the missing rate and latency of data access are effectively reduced; at the same time, cross-layer data redundancy is eliminated through global data deduplication, ensuring that data blocks only occupy unique physical addresses in all storage layers, further saving storage space; and this application can update the training model according to the changing workload to adapt to different access modes; by monitoring the characteristic information of the file, the method can be extended to any number of local storage systems with strong scalability.

[0028] Combine the following Figure 1 to Figure 2 , the present application is further described in detail with specific embodiments.

[0029] like Figure 1 As shown, the hierarchical storage and data deduplication method provided by the embodiment of the present application is described in detail as follows:

[0030] S1: Monitor all files in the storage module in real time, and record and store feature information corresponding to each file.

[0031] Combination Figure 2 As shown, the storage module 31 of the present application includes a first storage layer 301, a second storage layer 302 and a third storage layer 303, and the second storage layer 302 and the third storage layer 303 both include an error correction table 306. In this embodiment, from the first storage layer 301 to the second storage layer 302, and then to the third storage layer 303, the storage capacity gradually increases and the data access rate gradually slows down. As one of the optional embodiments, the first storage layer 301, the second storage layer 302 and the third storage layer 303 of the present application can be DRAM (dynamic random access memory), NVMe SSD (non-volatile memory) and SATA SSD (solid state drive with SATA interface), respectively.

[0032] Specifically, through the feature unit 1, a monitoring tool (inotify-tools) is used to monitor the opening operation of all files in the storage module 31, and the feature information corresponding to each file is recorded and stored in the form of KV.

[0033] Among them, the characteristic information includes the absolute path of the file, the file size, the file access frequency, the time interval between the two most recent file accesses, the access frequency of the parent folder where the file is located, and the file type. It should be noted that the absolute path of the file refers to the complete path describing the location of the file. The file size refers to the size of the space occupied by the file in the storage module 31. The file access frequency refers to the number of times the file is accessed within a certain period (such as a set week). The access frequency of the parent folder where the file is located refers to the number of times the parent folder where the file is located is accessed within a certain period. The file type refers to the content, format and purpose of a file.

[0034] The above-mentioned characteristic information takes into account both temporal limitations and spatial locality, and reflects the characteristics of the file more comprehensively.

[0035] The characteristic information of the file is stored in the storage module 31 in the form of KV. The KV form refers to K (absolute file path)-V (file size, file access frequency, time interval between the last two file accesses, access frequency of the parent folder where the file is located, and file type). The KV form is a data storage method, where "K" represents key and "V" represents value. This form of data storage organizes information in the form of "key-value pairs", each "key" uniquely identifies a file, and the "value" associated with it is the specific file content.

[0036] S2: Design the hierarchical standard of the file based on the inherent information and characteristic information of the storage module.

[0037] The inherent information of the storage module includes storage capacity, access speed, power consumption, reliability and cost. A file in the storage module can be divided into multiple data blocks.

[0038] Specific, combined Figure 2 As shown, the average data block content of all files in the storage module 31 is calculated. The above process is expressed by the following formula:

[0039]

[0040] Where b represents the average data block content, F j Indicates the size of file j, weight j represents the weight of file j, k represents the size of the data block in the storage layer, interval represents the time interval between the two most recent accesses, freq represents the access frequency, w1 represents the weight of the time interval between the two most recent accesses, w2 represents the weight of the access frequency freq, t represents the tth file, and n represents the total number of files.

[0041] Further, based on the inherent information, characteristic information and average data block content of the storage module 31, the revisit distance of the file is defined, and the revisit level thresholds for different storage layers in the storage module 31 are calculated. The revisit distance refers to the total number of file accesses in the entire system between two consecutive accesses to the same file. The revisit level threshold can be expressed by the following formula:

[0042]

[0043] Among them, C i represents the storage capacity of the i-th storage layer, k represents the size of the data block in the storage layer, b represents the average data block content, and h i It represents the hit rate of the i-th storage layer, and p represents the proportion of files that are accessed no more than once during the statistical period.

[0044] Furthermore, the tiering standard of the file is: if the reaccess distance of the file is greater than the reaccess level threshold of the storage layer, it means that the file does not belong to the corresponding storage layer.

[0045] For example, if the reaccess level threshold of the second storage layer 302 is 3, and the reaccess distance of a file is 5, which is greater than the reaccess level threshold of the second storage layer 302, it means that the file does not belong to the second storage layer 302 and should not be allocated to this layer.

[0046] S3: Obtain the historical feature information and historical stratification level corresponding to each file in the previous cycle, build an intelligent stratification model, train and update the intelligent stratification model according to the historical feature information and historical stratification level, and obtain the corresponding intelligent stratification classifier.

[0047] Specific, combined Figure 2 As shown, the historical feature information and the historical hierarchical level corresponding to each file in the previous cycle in the storage module 31 are obtained through the acquisition module 21 in the hierarchical unit 2.

[0048] The last period refers to the period of the last tiering of the file, which can be a week or a month, and can be set as needed. The historical tier level refers to the level of the storage layer to which the file was assigned in the last period; if the file is assigned to the first storage layer 301, its historical tier level is 1; if the file is assigned to the second storage layer 302, its historical tier level is 2.

[0049] Furthermore, an intelligent hierarchical model is constructed. In this embodiment, the intelligent hierarchical model is a lightweight decision tree model. The decision tree model is a simple and easy-to-use non-parametric classifier that does not require any prior assumptions about the data, has a fast calculation speed, is easy to interpret the results, and is robust, and can take into account both high prediction accuracy and low latency.

[0050] In this embodiment, the complete training set (such as the historical feature information corresponding to each file in the previous cycle) is used as the root node of the decision tree model. Each time the sub-nodes are divided, the feature with the largest information gain among all the historical feature information is selected as the division standard of this layer, and the number of sub-nodes is reduced by dichotomy to reduce the probability of overfitting. The division continues until the maximum decision tree height is limited or all sub-nodes belong to the same hierarchical level.

[0051] For example, in a training set D, the file access frequency is used as the feature with the largest information gain. The threshold T that can maximize the information gain of the divided subsets D1 and D2 is selected as the classification standard. After that, the training set D will be divided into the subset D2 with a file access frequency greater than the threshold T and the subset D1 with a file access frequency less than the threshold T, and the subsets D1 and D2 can achieve the maximum degree of distinction between different hierarchical levels. A similar process is recursively performed until the decision tree reaches the specified maximum decision tree height or the hierarchical levels of a certain divided subset are exactly the same.

[0052] The historical feature information corresponding to each file in the previous cycle is input into the intelligent stratification model through the training module 22, and the predicted stratification level corresponding to each file in the previous cycle is predicted through the intelligent stratification model. The predicted stratification level is compared with the historical stratification level, and the parameters of the intelligent stratification model are adjusted according to the comparison result, and the intelligent stratification model is updated through the updating module 23.

[0053] In this embodiment, the parameters of the intelligent stratification model include the ratio of training data to prediction data, the adopted stratification standard, the height of the decision tree, the minimum node divisible sample size and the minimum leaf node sample size.

[0054] For example, based on the input historical feature information, the intelligent tiering model predicts that the predicted tier level corresponding to one of the files in the last cycle is 2 (indicating that the file is predicted to be allocated to the second storage layer 302), while its historical tier level is 1 (indicating that the file is actually allocated to the first storage layer 301). According to the comparison, the predicted tier level and the historical tier level are inconsistent, indicating that the predicted tier level predicted by the intelligent tiering model is incorrect. Therefore, the parameters of the intelligent tiering model need to be adjusted and the intelligent tiering model needs to be updated.

[0055] After multiple trainings, adjustments, and updates, the corresponding intelligent hierarchical classifier is obtained. At this time, the intelligent hierarchical classifier can accurately predict the hierarchical level of the most suitable storage layer to which a file should be assigned.

[0056] S4: Obtain target feature information corresponding to each file in the current cycle, and predict the target hierarchical level corresponding to each file in the current cycle through an intelligent hierarchical classifier based on the target feature information.

[0057] Specific, combined Figure 2 As shown, the target feature information corresponding to each file in the current cycle in the storage module 31 is obtained through the acquisition module 21 in the layering unit 2.

[0058] Furthermore, the target feature information corresponding to each file in the current cycle is input into the intelligent hierarchical classifier through the prediction module 24, and the target hierarchical level corresponding to each file in the current cycle is predicted through the intelligent hierarchical classifier.

[0059] For example, the intelligent hierarchical classifier predicts that the target hierarchical level corresponding to one of the files in the current cycle is 2 based on the input target feature information, which means that the file needs to be allocated to the second storage layer 302 .

[0060] Further, according to the target hierarchical level corresponding to each file, the data hierarchical level corresponding to the multiple data blocks of each file is obtained. In this embodiment, the target hierarchical level corresponding to each file is used as the data hierarchical level corresponding to the multiple data blocks of each file. For example, if the target hierarchical level of one of the files is 1, the data hierarchical level of the multiple data blocks of the file is also 1.

[0061] Furthermore, since the prediction results of the intelligent hierarchical classifier are not completely consistent with the actual situation, and the access to data blocks and files is not strictly equal, an error correction table 306 is established for the second storage layer 302 and the third storage layer 303 to give data blocks that are mistakenly judged as not qualified to enter the first storage layer 301 an opportunity to upgrade the storage layer, that is, only data blocks with data hierarchical levels 2 or 3 are corrected.

[0062] Specifically, the error correction table 306 is used to maintain a FIFO (First Input First Output) sequence to record the fingerprint information and corresponding data hierarchy level of the incorrectly predicted data block, and the data hierarchy level corresponding to the incorrectly predicted data block is sent to the post-processing unit 4 for further processing.

[0063] The predicted target hierarchical level corresponding to each file in the current cycle and the data hierarchical levels corresponding to the multiple data blocks of each file are sent to the post-processing unit 4 for further processing.

[0064] S5: query the data blocks of each file in the storage module, delete duplicate data blocks, and migrate the remaining data blocks of each file to the corresponding storage layer in the storage module according to the target tier level.

[0065] Specific, combined Figure 2 As shown, the deduplication module 41 in the post-processing unit 4 performs hash calculation on each data block in the storage module 31 using a hash function (SHA-256) to obtain fingerprint information corresponding to the data block.

[0066] For multiple data blocks with the same fingerprint information, a physical address in the storage module 31 is reserved for storing the actual data of one of the data blocks. For the remaining data blocks, indexes pointing to the physical address are respectively established, and the duplicate actual data of the remaining data blocks are deleted to save the corresponding storage space.

[0067] Furthermore, multiple storage layers in the storage module 31 are queried to migrate the data block from the storage layer in the previous cycle to the storage layer indicated by the target hierarchical level.

[0068] Specifically, a data block is used to access the first storage layer 301. If a hit is found, it means that the storage layer where the data block was located in the previous cycle is the first storage layer 301. In this case, there is no need to perform intelligent tiering through the intelligent tiering classifier, and the data block still resides in the first storage layer 301. At the same time, through the data block management submodule 305, the data blocks in the first storage layer 301 are updated with the LRU strategy, and the data blocks with too low access frequency are migrated out of the first storage layer 301.

[0069] It should be noted that the LRU (Least Recently Used) strategy is a cache elimination strategy, and its basic principle is to eliminate the least recently used data. When the cache is full, it will first discard the least recently used items to make room for new elements.

[0070] Furthermore, when the same data block is used to access the first storage layer 301, if there is no hit, it means that the storage layer where the data block was located in the previous cycle is not the first storage layer 301, and further processing is performed according to the data layer level corresponding to the data block obtained in S4.

[0071] If the data layering level is 1, the data block is migrated to the first storage layer 301, and the first storage layer 301 is used as the unique address of the data block. All storage layers are traversed, and the data blocks with the same fingerprint information as the data block are deleted, and an index pointing to the first storage layer 301 is established for the data blocks with the same fingerprint information.

[0072] If the data hierarchy level is 2, the error correction table 306 of the second storage layer 302 is queried. If a hit is found, it means that the data hierarchy level predicted by the intelligent hierarchy classifier is incorrect. In this case, the data block is migrated to the first storage layer 301, and the first storage layer 301 is used as the unique address of the data block. All storage layers are traversed, and the data blocks with the same fingerprint information as the data block are deleted, and an index pointing to the first storage layer 301 is established for the data blocks with the same fingerprint information.

[0073] If the data tier level is 2, and the query of the error correction table 306 of the second storage layer 302 does not hit, it means that the data tier level predicted by the intelligent tier classifier is correct, and the error correction table 306 of the second storage layer 302 is updated; the second storage layer 302 is further queried, and if it hits, it means that the storage layer where the data block was located in the previous cycle is the second storage layer 302, and the data block is not migrated. At the same time, the data blocks in the second storage layer 302 are updated by the data block management submodule 305 with the LRU strategy, and the data blocks with too low access frequency are migrated out of the second storage layer 302. If the query of the second storage layer 302 does not hit, it means that the storage layer where the data block was located in the previous cycle is not the second storage layer 302, and the data block is migrated to the second storage layer 302, and the second storage layer 302 is used as the unique address of the data block, and all storage layers are traversed to delete the data blocks with the same fingerprint information as the data block, and an index pointing to the second storage layer 302 is established for the data blocks with the same fingerprint information.

[0074] If the data hierarchy level is 3, the error correction table 306 of the second storage layer 302 is queried. If a hit is found, it means that the data hierarchy level predicted by the intelligent hierarchy classifier is incorrect. In this case, the data block is migrated to the first storage layer 301, and the first storage layer 301 is used as the unique address of the data block. All storage layers are traversed, and the data blocks with the same fingerprint information as the data block are deleted, and an index pointing to the first storage layer 301 is established for the data blocks with the same fingerprint information.

[0075] If the data layering level is 3, and the error correction table 306 of the second storage layer 302 is not hit when querying, the error correction table 306 of the second storage layer 302 is updated; the second storage layer 302 is further queried. If a hit is found, it means that the storage layer where the data block was located in the previous cycle is the second storage layer 302. In this case, the data block will not be migrated. At the same time, the data blocks in the second storage layer 302 are updated with the LRU strategy through the data block management submodule 305, and the data blocks with too low access frequency are migrated out of the second storage layer 302.

[0076] If the query of the second storage layer 302 does not hit, the error correction table 306 of the third storage layer 303 is further queried. If it hits, it means that the data hierarchical level predicted by the intelligent hierarchical classifier is incorrect, and the data block is migrated to the second storage layer 302, and the second storage layer 302 is used as the unique address of the data block, and all storage layers are traversed to delete the data blocks with the same fingerprint information as the data block, and an index pointing to the second storage layer 302 is established for the data blocks with the same fingerprint information. If the query of the error correction table 306 of the third storage layer 303 does not hit, it means that the data hierarchical level predicted by the intelligent hierarchical classifier is correct, and it also means that the storage layer where the data block was located in the last cycle is the third storage layer 303, and there is no need to migrate; at the same time, through the data block management submodule 305, the data blocks in the third storage layer 303 are updated with the LRU strategy, and the error correction table 306 of the third storage layer 303 is updated at the same time.

[0077] Corresponding to the tiered storage and deduplication method of the above embodiment, the embodiment of the present application further provides a tiered storage and deduplication system. For ease of description, only the parts related to the embodiment of the present application are shown.

[0078] like Figure 2 As shown, the tiered storage and data deduplication system includes:

[0079] The feature unit 1 is used to monitor all files in the storage module 31 in real time and record feature information corresponding to each file.

[0080] Hierarchical unit 2 is used to obtain the historical feature information and historical hierarchical level corresponding to each file in the previous cycle, build an intelligent hierarchical model, train and update the intelligent hierarchical model according to the historical feature information and historical hierarchical level, and obtain the corresponding intelligent hierarchical classifier; it is also used to obtain the target feature information corresponding to each file in the current cycle, and predict the target hierarchical level corresponding to each file in the current cycle through the intelligent hierarchical classifier according to the target feature information.

[0081] The storage unit 3 is used to store files and feature information corresponding to each file.

[0082] The post-processing unit 4 is used to query the data blocks of each file in the storage module 31, delete duplicate data blocks, and migrate the remaining data blocks of each file to the corresponding storage layer in the storage module 31 according to the target layer level.

[0083] Further, the hierarchical unit 2 includes an acquisition module 21, a training module 22, an update module 23 and a prediction module 24; the storage unit 3 includes a storage module 31 and a management module 32, the storage module 31 includes a first storage layer 301, a second storage layer 302 and a third storage layer 303, the second storage layer 302 and the third storage layer 303 both include an error correction table 306, and the management module 32 includes a metadata management submodule 304 and a data block management submodule 305; the post-processing unit 4 includes a deduplication module 41 and a migration module 42; the acquisition module 21 is used to obtain the feature information corresponding to each file ; The training module 22 is used to train the intelligent tiering model; the update module 23 is used to update the intelligent tiering model; the prediction module 24 is used to predict the target tiering level corresponding to each file; the storage module 31 is used to store all files and the feature information corresponding to each file; the management module 32 is used to manage all files and the feature information corresponding to each file; the deduplication module 41 is used to query the data blocks of each file in the storage module 31 and delete the duplicate data blocks; the migration module 42 is used to migrate the remaining data blocks of each file to the corresponding storage layer in the storage module 31 according to the target tiering level.

[0084] Other technical features of the above-mentioned tiered storage and deduplication system are the same as those disclosed in the above-mentioned embodiment methods, and can be referred to the description in the above-mentioned corresponding method embodiments, which will not be repeated here.

[0085] It can be seen that the present invention discloses a hierarchical storage and deduplication method and system, the method comprising the steps of: real-time monitoring of all files in a storage module, recording and storing feature information corresponding to each file; obtaining historical feature information and historical hierarchical level corresponding to each file in the previous cycle, constructing an intelligent hierarchical model, training and updating the intelligent hierarchical model according to the historical feature information and historical hierarchical level, and obtaining a corresponding intelligent hierarchical classifier; obtaining target feature information corresponding to each file in the current cycle, and predicting the target hierarchical level corresponding to each file in the current cycle through an intelligent hierarchical classifier according to the target feature information; querying the data blocks of each file in the storage module, deleting duplicate data blocks, and migrating the remaining data blocks of each file to the corresponding storage layer in the storage module according to the target hierarchical level. The intelligent hierarchical model of the present application has low complexity and short training time, and not only has good hierarchical effect, but also high hierarchical efficiency, further improving the overall performance of the system. By rationally stratifying file data, the data access miss rate and data access latency are effectively reduced; at the same time, cross-layer data redundancy is eliminated through global data deduplication to ensure that data blocks only occupy unique physical addresses in all storage layers, further saving storage space; and the present application can update the training model according to the changing workload to adapt to different access modes; by monitoring the characteristic information of the file, the method can be extended to any number of layers of local storage systems, with strong scalability.

[0086] The above are only embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structural transformations made using the contents of the present invention's specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A hierarchical storage and data deduplication method, characterized in that: Includes steps: Monitor all files in the storage module in real time, and record and store feature information corresponding to each file; Obtaining historical feature information and historical hierarchical levels corresponding to each of the files in the previous cycle, building an intelligent hierarchical model, and training and updating the intelligent hierarchical model according to the historical feature information and historical hierarchical levels to obtain a corresponding intelligent hierarchical classifier; Obtaining target feature information corresponding to each of the files in the current cycle, and predicting the target hierarchical level corresponding to each of the files in the current cycle through the intelligent hierarchical classifier according to the target feature information; The data blocks of each of the files in the storage module are queried, duplicate data blocks are deleted, and the remaining data blocks of each of the files are migrated to the corresponding storage layer in the storage module according to the target tier level.

2. The hierarchical storage and data deduplication method according to claim 1, characterized in that: The real-time monitoring of all files in the storage module and recording and storing characteristic information corresponding to each file specifically include: Monitor all opening operations of the files, record and store the characteristic information corresponding to each file in KV format; The characteristic information includes the absolute path of the file, the size of the file, the frequency of access to the file, the time interval between the two most recent accesses to the file, the frequency of access to the parent folder where the file is located, and the file type.

3. The hierarchical storage and data deduplication method according to claim 1, characterized in that: After the step of recording and storing the characteristic information corresponding to each of the files, the step further includes: The layering standard of the file is designed according to the inherent information of the storage module and the characteristic information.

4. The hierarchical storage and data deduplication method according to claim 3, characterized in that: The step of designing the layering standard of the file according to the inherent information of the storage module and the characteristic information specifically includes: Calculate and obtain the average data block content of all files in the storage module; Defining the reaccess distance of the file according to the inherent information of the storage module, the characteristic information and the average data block content, and calculating the reaccess level thresholds of the different storage layers in the storage module; If the reaccess distance of the file is greater than the reaccess level threshold of the storage layer, it means that the file does not belong to the corresponding storage layer.

5. The hierarchical storage and data deduplication method according to claim 1, characterized in that: The training and updating of the intelligent stratification model according to the historical feature information and the historical stratification level specifically includes: According to the historical feature information, the predicted hierarchical level corresponding to each of the files in the previous cycle is predicted by the intelligent hierarchical model; The predicted stratification level is compared with the historical stratification level, and the parameters of the intelligent stratification model are adjusted according to the comparison result to update the intelligent stratification model.

6. The hierarchical storage and data deduplication method according to claim 1, characterized in that: The predicting, based on the target feature information, through the intelligent hierarchical classifier, the target hierarchical level corresponding to each of the files in the current cycle specifically includes: Inputting the target feature information into the intelligent hierarchical classifier, predicting the target hierarchical level corresponding to each of the files in the current cycle, and obtaining the data hierarchical levels corresponding to the multiple data blocks of each of the files; The data hierarchical level corresponding to the data block predicted incorrectly is updated and saved using the error correction table.

7. The hierarchical storage and data deduplication method according to claim 1, characterized in that: The querying of the data blocks of each file in the storage module and deleting the duplicated data blocks specifically includes: Performing hash calculation on each of the data blocks to obtain fingerprint information corresponding to the data block; For the multiple data blocks with the same fingerprint information, the physical address corresponding to one of the data blocks is retained, and for the remaining data blocks, indexes pointing to the physical addresses are respectively established.

8. The hierarchical storage and data deduplication method according to claim 1, characterized in that: Migrating the remaining data blocks of each file to a corresponding storage layer in the storage module according to the target layer level specifically includes: querying the plurality of storage layers in the storage module, and migrating the data block from the storage layer in the previous cycle to the storage layer indicated by the target hierarchical level; The storage layer after migration is used as the unique address of the data block, all the storage layers are traversed, the data blocks having the same fingerprint information as the data block after migration are deleted, and an index pointing to the storage layer after migration is established.

9. A hierarchical storage and data deduplication system, characterized in that: include: A feature unit, used to monitor all files in the storage module in real time and record feature information corresponding to each file; The stratification unit is used to obtain the historical feature information and historical stratification level corresponding to each of the files in the previous cycle, build an intelligent stratification model, train and update the intelligent stratification model according to the historical feature information and historical stratification level, and obtain a corresponding intelligent stratification classifier; and is also used to obtain the target feature information corresponding to each of the files in the current cycle, and predict the target stratification level corresponding to each of the files in the current cycle through the intelligent stratification classifier according to the target feature information; A storage unit, used for storing the files and the characteristic information corresponding to each of the files; A post-processing unit is used to query the data blocks of each file in the storage module, delete the duplicated data blocks, and migrate the remaining data blocks of each file to the corresponding storage layer in the storage module according to the target hierarchical level.

10. The hierarchical storage and data deduplication system according to claim 9, characterized in that: The hierarchical unit includes an acquisition module, a training module, an update module and a prediction module; the storage unit includes a storage module and a management module, the storage module includes a first storage layer, a second storage layer and a third storage layer, the second storage layer and the third storage layer both include an error correction table, the management module includes a metadata management submodule and a data block management submodule; the post-processing unit includes a deduplication module and a migration module; The acquisition module is used to acquire the feature information corresponding to each of the files; The training module is used to train the intelligent hierarchical model; The updating module is used to update the intelligent hierarchical model; The prediction module is used to predict the target hierarchical level corresponding to each of the files; The storage module is used to store all the files and the feature information corresponding to each of the files; The management module is used to manage all the files and the feature information corresponding to each file; The deduplication module is used to query the data blocks of each file in the storage module and delete the duplicate data blocks; The migration module is used to migrate the remaining data blocks of each file to the corresponding storage layer in the storage module according to the target layer level.

Images