Data management method and system for storage device

Abstract

The application relates to the technical field of data storage, in particular to a data management method and system of a storage device. The steps of the data management method comprise the following: classifying data in the storage device, dividing the data into hot data and cold data, the hot data being frequently accessed data, and the cold data being long-time non-accessed data; applying differential privacy technology to compress and deduplicate all the data, so as to reduce storage space and ensure that data privacy is not infringed. The application realizes three-dimensional optimization of privacy protection strength, storage resource utilization rate and access performance in the field of storage systems, and provides a theoretically complete engineering solution for the operation of a super-large-scale data center.

Description

Technical Field

[0001] The present invention relates to the technical field of data storage, and in particular to a data management method and system for a storage device. Background Art

[0002] Data storage devices usually adopt static storage strategies and are unable to dynamically adjust storage media and compression algorithms based on the actual access frequency, importance, and type of data. However, static storage mechanisms are difficult to adapt to dynamic access patterns, resulting in a continuous degradation of data classification accuracy and storage efficiency. In the data compression and deduplication process, existing methods have coupling defects between the encryption strength of cold data and the privacy protection mechanism in cross-media migration scenarios, and are unable to meet the strong constraints of differential privacy while reducing storage redundancy. In addition, the coordination problem between metadata management and cold data recovery efficiency has not been broken through for a long time, and the state switching between regular queries and hidden data lacks dynamic consistency guarantees, making it difficult to balance system throughput and response latency. The core contradiction that current technology has not yet solved is: how to build a dynamic and adaptive privacy protection mechanism under a multi-layer heterogeneous storage architecture to achieve full-link collaborative optimization of hot and cold data classification, compression and deduplication, encryption migration and intelligent recovery, while meeting the dual rigid requirements of storage resource efficiency and data security. Summary of the Invention

[0003] The present disclosure provides a data management method and system for a storage device, aiming to overcome at least one drawback in the prior art.

[0004] To achieve the above objectives, the technical solutions disclosed in the present invention are as follows:

[0005] According to one aspect of the present disclosure, a data management method for a storage device is provided, the steps of the data management method comprising:

[0006] Classifying data in a storage device into hot data and cold data, wherein the hot data is frequently accessed data and the cold data is data that has not been accessed for a long time;

[0007] Apply differential privacy technology to compress and deduplicate all data to reduce storage space and ensure data privacy;

[0008] Migrating the cold data to low-cost storage media and performing encryption and differential privacy processing on the cold data to ensure the security of the data during transmission and storage;

[0009] In the storage device, the cold data is marked as "hidden" so as to be invisible in conventional queries, and a metadata table is maintained to record the location and status of the cold data so as to be retrieved when needed;

[0010] When a user requests to access the cold data, the intelligent scheduling system automatically detects the location and status of the cold data, reads the data from the low-cost storage medium, decrypts and verifies its integrity, and restores the data to the storage device for user access.

[0011] Furthermore, a machine learning model is used to classify the data, identify the cold data and hot data, and dynamically adjust the data classification criteria based on access frequency and importance. The steps include: collecting data access logs; using a support vector machine (SVM) algorithm to classify the data; dynamically adjusting the classification threshold to adapt to changes in data access patterns; and updating the classification results to ensure the accuracy of data classification.

[0012] The data is compressed using the LZ77 compression algorithm, and the Bloom filter is used to identify and remove redundant data to free up storage space. The steps include: dividing the data into multiple blocks; performing LZ77 compression on each block; calculating the hash value of each block; using the Bloom filter to check whether the hash value exists, and if so, the block is considered redundant data; removing redundant data and retaining unique data blocks.

[0013] Furthermore, cold data encryption uses the AES-256 encryption algorithm and combines differential privacy technology to ensure data security during transmission and storage. The steps include: generating a random key K; encrypting the cold data using the AES-256 algorithm to obtain encrypted data E; storing the encrypted data E together with the key K in a low-cost storage medium;

[0014] The differential privacy technology protects the privacy of keys or data by adding noise, and the steps include: calculating the sensitivity Δf of the key K or data; generating noise that conforms to the Laplace distribution , where ϵ is the privacy budget; noise is added to the key K or data to increase key security and protect data privacy.

[0015] Furthermore, the maintained metadata table includes the identification, location information, encryption key and timestamp of the cold data, so as to quickly locate and restore the data when needed. The steps include: creating a metadata table to record the identification, location information, encryption key and timestamp of the cold data; regularly updating the metadata table to ensure that the data is complete and consistent; when a user requests to access the cold data, quickly locate the location and status of the cold data through the metadata table.

[0016] Furthermore, the intelligent scheduling system temporarily stores recently accessed cold data in a cache through a caching mechanism to improve the response speed of subsequent accesses. The steps include: creating a cache to store recently accessed cold data; when a user requests access to cold data, first checking whether the data exists in the cache; if so, reading the data directly from the cache; if not, reading the data from a low-cost storage medium and adding it to the cache.

[0017] Furthermore, the steps of the data compression algorithm include:

[0018] Divide the data into multiple blocks B1, B2, ..., Bn;

[0019] For each block B i Perform preliminary compression to obtain C i ;

[0020] Use hash function H to calculate the hash value h of each block i =H(B i );

[0021] Find duplicate blocks through hash values ​​and merge them to obtain the merged data set;

[0022] Perform final compression on the merged data set to obtain the final compression result;

[0023] The expression of the data compression algorithm is:

[0024] , where m is the length of the block, and p and q are large prime numbers.

[0025] Furthermore, the deduplication process also uses a deduplication algorithm, which includes the following steps: constructing a similarity graph G=(V,E) between data blocks, where V is a set of data blocks and E is a set of edges; calculating the similarity of each node v i Degree d i ; Select the node v with the highest degree max As the central node; perform deduplication operation through the neighbor nodes of the central node, let N(v max ) is the set of neighbor nodes of the central node; update the similarity graph G, and repeat the above steps until all nodes are processed;

[0026] The expression of the deduplication algorithm is:

[0027] , where I(v i ,v j ) is the indicator function, indicating that node v i and v j Is there an edge between them?

[0028] Further, the step of the data management method further comprises: presetting a time threshold; monitoring the last access date of each data; obtaining a time difference value by subtracting the current date from the last access date, and converting the data from hot data to cold data if the time difference value is greater than the time threshold; updating the metadata table to record the state change of the data; migrating the converted cold data to a low-cost storage medium and performing encryption and differential privacy processing; and marking the cold data as a hidden state and recording its location and state in the metadata table.

[0029] Further, if the monitored data does not have a last access date, the date of storing the data in the storage device is used as the last access date.

[0030] According to another aspect of the present disclosure, a data management system of a storage device is provided for implementing the data management method of the storage device as described above, and the data management system comprises:

[0031] a data classification module for classifying data into hot data and cold data;

[0032] a data compression and deduplication module for compressing and deduplicating all data and applying differential privacy technology;

[0033] a data migration module for migrating cold data to a low-cost storage medium and performing encryption and differential privacy processing;

[0034] a data hiding module for marking cold data as a hidden state and maintaining a metadata table;

[0035] an intelligent scheduling module for automatically detecting and restoring data when a user requests to access cold data.

[0036] The present application has the following advantages:

[0037] The present application realizes adaptive adjustment of the cold and hot data boundary by fusing the dynamic classification model of support vector machine and the differential privacy enhanced compression algorithm, embeds a redundancy elimination mechanism driven by a Bloom filter in the LZ77 compression process, and reduces the storage redundancy. The composite encryption strategy of AES-256 and Laplace noise superposition is adopted, differential privacy protection is injected in the key generation stage, so that the cold data achieves the security strength of ε-differential privacy when migrated to a low-cost medium, and the compatibility problem of traditional encryption mechanism and privacy budget allocation is solved. The state-aware architecture based on the metadata table and the intelligent cache scheduling system work cooperatively to shorten the cold data recovery delay to milliseconds, and improve the regular query performance through the hidden state marking technology. In particular, the graph theory deduplication algorithm for constructing a data block similarity graph controls the hash collision probability to 10-7 This approach achieves a Pareto-optimal solution for storage efficiency and data integrity. This invention achieves three-dimensional optimization of privacy protection strength, storage resource utilization, and access performance in the storage system field, providing a theoretically complete engineering solution for the operation of ultra-large-scale data centers.

[0038] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention and implement it according to the contents of the specification, the following is a detailed description of the preferred embodiments of the present invention with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 is a flow chart of a data management method for a storage device according to an embodiment of the present invention;

[0040] Figure 2 A heat map of data access frequency in one embodiment of the present invention;

[0041] Figure 3 This is a distribution diagram of compression ratios of different types of data in one embodiment of the present invention;

[0042] Figure 4 A graph showing the relationship between data access frequency and storage cost in one embodiment of the present invention;

[0043] Figure 5 A diagram of the data classification and migration process in one embodiment of the present invention;

[0044] Figure 6 This is a surface diagram of intelligent cache parameter optimization in one embodiment of the present invention;

[0045] Figure 7 This is a heat map of the spatiotemporal distribution of metadata tables in one embodiment of the present invention;

[0046] Figure 8 This is a differential privacy noise superposition distribution field diagram in one embodiment of the present invention;

[0047] Figure 9 1. Bloom filter misjudgment rate parameter surface diagram in one embodiment of the present invention;

[0048] Figure 10 This is a surface diagram of LZ77 compression parameter optimization in one embodiment of the present invention;

[0049] Figure 11 This is a decision surface diagram for hot and cold data classification in one embodiment of the present invention. DETAILED DESCRIPTION

[0050] With reference to the drawings of the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts are within the scope of the present application.

[0051] The term "comprising" and any variation thereof when used in the specification and claims of the present application, is intended to cover a non-exclusive inclusion, for example, a process, method, system, product or apparatus that includes a list of steps or units not necessarily limited to those explicitly listed, but can include other steps or units not expressly listed or inherent to such processes, methods, products or apparatus. In addition, the use of "and / or" in the specification and claims indicates at least one of the connected objects, for example, A and / or B, which includes separate A, separate B, and all three cases where A and B exist.

[0052] In the embodiments of the present application, the words such as "exemplary" or "for example" are used to mean serving as an example, instance or illustration. Any embodiment or design presented as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or advantageous than other embodiments or design solutions. Rather, the use of "exemplary" or "for example" is intended to present relevant concepts in a concrete manner.

[0053] The present application provides the following preferred embodiments:

[0054] Embodiment one

[0055] In order to solve the problem of continuous deterioration of data classification accuracy and storage efficiency caused by static storage strategy in existing data storage devices, the present embodiment provides a data management method for a storage device, as shown in Figure 1 The steps of the data management method include:

[0056] S100, classifying the data in the storage device, and dividing the data into hot data and cold data, the hot data being frequently accessed data, and the cold data being long-time non-accessed data.

[0057] S200, applying differential privacy technology to compress and deduplicate all data to reduce storage space and ensure data privacy is not infringed.

[0058] S300, migrating the cold data to a low-cost storage medium, and encrypting and differentially processing the cold data to ensure the security of the data in the transmission and storage process.

[0059] S400: In a storage device, mark the cold data as hidden so that it is not visible in regular queries, and maintain a metadata table to record the location and status of the cold data so that it can be retrieved when needed.

[0060] S500. When a user requests to access cold data, the intelligent scheduling system automatically detects the location and status of the cold data, reads the data from the low-cost storage medium, decrypts and verifies its integrity, and restores the data to the storage device for user access.

[0061] In this embodiment, the data in the storage device is first classified. Specifically, data access logs are collected and the access frequency and time decay characteristics of each data item are calculated. A support vector machine (SVM) algorithm is then used to classify the data, marking frequently accessed data as hot data and data that has not been accessed for a long time as cold data. It is important to understand that the SVM algorithm can construct a decision boundary based on the access frequency and time decay characteristics of the data, thereby achieving high-precision data classification. To adapt to changes in data access patterns, the classification threshold is dynamically adjusted, and the classification results are regularly updated to ensure accurate data classification.

[0062] Furthermore, differential privacy technology is applied to compress and deduplicate all data. Specifically, the LZ77 compression algorithm is used to perform preliminary data compression, dividing the data into multiple blocks. LZ77 compression is performed on each block, and a hash value is calculated for each block. A Bloom filter is used to check whether the hash value exists. If it exists, the block is considered redundant data and removed. Otherwise, it is added to the Bloom filter. In this way, storage space can be effectively reduced and data privacy is ensured. It should be understood that the LZ77 compression algorithm combined with the Bloom filter can significantly reduce data redundancy while maintaining the compression rate.

[0063] Furthermore, cold data can be migrated to low-cost storage media, such as tapes or cloud storage, and encrypted and differentially private processed. Figure 5 As shown in the figure. Specifically, a random key K is generated, and the cold data is encrypted using the AES-256 algorithm to obtain encrypted data E. The encrypted data E is stored together with the key K on a low-cost storage medium. To further protect data privacy, the sensitivity Δf of the key K or the data is calculated, noise conforming to the Laplace distribution is generated, and the noise is added to the key K or the data. In this way, data security can be ensured during transmission and storage. It should be understood that the AES-256 encryption algorithm combined with differential privacy technology can ensure data security while meeting the strong constraints of differential privacy.

[0064] Furthermore, in the storage device, cold data is marked as "hidden" to prevent it from being viewed in regular queries. A metadata table is maintained to record the location and status of the cold data. Specifically, a metadata table is created to record the cold data's identification, location information, encryption key, and timestamp. The metadata table is regularly updated to ensure data integrity and consistency. When a user requests access to cold data, the metadata table can be used to quickly locate the cold data's location and status. It is important to understand that the metadata table enables rapid location and recovery of cold data, thereby improving system responsiveness.

[0065] Furthermore, when a user requests access to cold data, the intelligent scheduling system automatically detects the location and status of the cold data. Specifically, the data is read from low-cost storage media, decrypted and its integrity verified, and restored to the storage device for user access. It should be understood that the intelligent scheduling system's ability to automatically detect and restore cold data improves overall system performance and user experience.

[0066] The benefit of this embodiment lies in achieving efficient data storage management and security through dynamic data classification, differentially privacy-enhanced compression algorithms, cold data encryption migration, and an intelligent scheduling system. These optimization measures not only improve data classification accuracy and storage efficiency, but also ensure data security during transmission and storage, thus meeting the dual requirements of storage resource efficiency and data security. Example 2

[0067] To address the problem of identifying and removing redundant data during data classification and compression, this embodiment further optimizes the data classification and compression steps in the data management method. Specifically, this embodiment uses a machine learning model to classify data and employs the LZ77 compression algorithm combined with a Bloom filter to identify and remove redundant data.

[0068] Furthermore, collecting data access logs is the basis for data classification. By recording the time, frequency, and importance of each data access, it can provide the necessary input for subsequent data classification. It can be understood that the support vector machine (SVM) algorithm is applied to the data classification process. The SVM algorithm can classify data into cold data and hot data based on access frequency and time decay characteristics, such as Figure 2 and Figure 11 The figure shows the decision surface for hot and cold data classification. It's important to understand that dynamic adjustment of the classification threshold ensures accurate classification results. By regularly analyzing changes in data access patterns, the system can automatically adjust the classification threshold to adapt to new data access behavior.

[0069] Furthermore, in order to effectively utilize storage space, this embodiment adopts the LZ77 compression algorithm. LZ77 is a dictionary-based lossless compression algorithm that achieves compression by finding repeated data blocks. Figure 10 As shown, the LZ77 compression parameter optimization surface shows the compression ratio under different block sizes and data redundancy. It can be understood that the data is divided into multiple blocks of fixed size, and each block is compressed by LZ77. Furthermore, in order to identify and remove redundant data, Bloom filter is introduced. Bloom filter is a probabilistic data structure with extremely high space efficiency, which is used to determine whether an element is in a set. In this embodiment, the hash value of each data block is calculated and stored in the Bloom filter. When a new data block arrives, its hash value will first be checked by the Bloom filter to see if it exists. If it exists, the block is considered to be redundant data; otherwise, it is added to the Bloom filter and the unique data block is retained. This mechanism can significantly reduce the storage space occupied while maintaining data integrity.

[0070] Furthermore, in order to improve the adaptability of the system, this embodiment also takes into account the misjudgment rate of the Bloom filter. Figure 9 As shown in Figure 2, the Bloom filter false positive rate parameter surface shows the false positive probability for different numbers of elements and hash functions. By properly choosing the number of hash functions and the size of the Bloom filter, the false positive rate can be controlled within an acceptable range.

[0071] The benefit of this embodiment is that by combining machine learning classification and the LZ77 compression algorithm, it not only improves the accuracy of data classification, but also effectively reduces storage space usage. The introduction of Bloom filters further enhances the system's deduplication capabilities, thereby improving overall storage efficiency. Example 3

[0072] To address the security issues of cold data during transmission and storage, this embodiment further refines the encryption and privacy protection methods for cold data. Specifically, this embodiment uses the AES-256 encryption algorithm combined with differential privacy technology to ensure data security and privacy.

[0073] Specifically, generating a random key K is the first step in the encryption process. AES-256 is an advanced encryption standard with high security and reliability. Cold data is encrypted using the AES-256 algorithm to generate encrypted data E. Encrypted data E is then stored along with the key K on low-cost storage media. It's important to understand that the management and protection of the key K impacts the security of the entire encryption system. Therefore, the key K must be transmitted over a secure channel and protected during storage.

[0074] To further enhance the security and privacy of data, this embodiment introduces differential privacy technology. Differential privacy is a mathematical framework designed to protect individual privacy during data analysis. As shown in Figure 8 the differential privacy noise superposition distribution chart demonstrates the relationship between the original data distribution and the amount of noise injected. The core idea of differential privacy technology is to obscure data by adding noise, making it impossible for attackers to infer individual information from statistical results. The specific steps include: calculating the sensitivity of the key K or data Δf; generating noise that conforms to the Laplace distribution, where ε is the privacy budget; adding noise to the key K or data. In this way, even if an attacker obtains part of the data, it is impossible to accurately restore the original data, thus protecting the privacy of the data.

[0075] Further, to ensure the integrity and consistency of data, this embodiment also considers data verification mechanisms. That is, a message authentication code (MAC) can be used to verify the integrity of the data. MAC is a key-based hash function that can detect whether data has been tampered with. By attaching a MAC value after encrypting the data, the integrity of the data can be verified when decrypted, thus preventing data tampering.

[0076] The benefits of this embodiment are that by combining the AES-256 encryption algorithm and differential privacy technology, not only is the security of the data improved, but the privacy of the data is also effectively protected. By introducing a data verification mechanism, the reliability of the system is further enhanced. Embodiment Four

[0077] To solve the problem of fast positioning and recovery of cold data, this embodiment further optimizes the maintenance and management method of the metadata table. That is, this embodiment ensures that cold data can be quickly located and recovered when needed by creating and maintaining a detailed metadata table.

[0078] Specifically, creating a metadata table is the basis of data management. The metadata table records key information such as the identification, location information, encryption key, and timestamp of cold data. As shown in Figure 7 the metadata table time-space distribution heat map demonstrates the relationship between the timestamp and access frequency. By recording this information, the system can quickly locate the location and status of cold data when a user requests access to cold data. Further, the metadata table needs to be updated regularly to ensure the integrity and consistency of the data. Regularly updating the metadata table can prevent inconsistencies caused by data changes.

[0079] Further, to improve the query efficiency of the metadata table, the embodiment adopts an indexing technique. By establishing an index on the metadata table, the query speed can be significantly improved. Common index types include B-tree index and hash index. B-tree index is suitable for range queries, while hash index is suitable for point queries. By reasonably selecting the index type, different query requirements can be met, thereby improving the response speed of the system.

[0080] In addition, to ensure the security of the metadata table, the embodiment also considers data backup and recovery mechanisms. Regularly backing up the metadata table can prevent data loss due to unexpected circumstances. Backup data can be stored in different physical locations to increase data security. In the event of data loss or damage, the metadata table can be quickly recovered through backup data, thereby ensuring the normal operation of the system.

[0081] The benefits of the embodiment are that by creating and maintaining a detailed metadata table, not only does it improve the positioning and recovery efficiency of cold data, but also enhances the reliability and security of the system. Embodiment Five

[0082] To solve the problem of cold data access delay, the embodiment further optimizes the cache mechanism in the intelligent scheduling system. That is, the embodiment creates a cache to temporarily store recently accessed cold data in the cache, thereby improving the response speed of subsequent access.

[0083] Specifically, creating a cache is the basis of the cache mechanism. Cache usually uses high-speed storage media such as DRAM or SSD to provide faster read and write speeds. By storing recently accessed cold data in the cache, the delay of data access can be significantly reduced. Further, when a user requests to access cold data, the system first checks whether the data exists in the cache. If it exists, the data is read directly from the cache; if it does not exist, the data is read from the low-cost storage medium and added to the cache. This mechanism can significantly improve the response speed of data access, especially in the case of frequent access to the same data.

[0084] Further, to improve the hit rate of the cache, the embodiment adopts an intelligent cache replacement strategy. Common cache replacement strategies include LRU (Least Recently Used), LFU (Least Frequently Used), and FIFO (First In First Out). As shown in Figure 6 The intelligent cache parameter optimization surface shows the influence of cache capacity and temporal locality factor on hit rate. By reasonably selecting the cache replacement strategy, the hit rate of the cache can be improved, thereby further improving the performance of the system.

[0085] Furthermore, to ensure cache consistency, this embodiment also considers a cache synchronization mechanism. When data changes in the low-cost storage medium, the data in the cache needs to be updated promptly to avoid data inconsistency issues. Common cache synchronization mechanisms include write-back and write-through. The write-back method only updates the cache when data is modified, and writes the data back to the low-cost storage medium only when the data is replaced from the cache; the write-through method writes the data back to the low-cost storage medium immediately when the data is modified. By properly selecting a cache synchronization mechanism, data consistency can be ensured, thereby improving system reliability.

[0086] The benefit of this embodiment is that by creating a cache and storing recently accessed cold data in the cache, it not only improves the response speed of data access but also enhances system performance. By introducing an intelligent cache replacement strategy and cache synchronization mechanism, the efficiency of the system is further improved. Example 6

[0087] In order to solve the problem of low data storage space utilization, this embodiment further optimizes the data compression algorithm. Specifically, first, the data to be compressed is divided into multiple blocks B1, B2, ..., Bn. The size of each block can be adjusted according to actual needs. Next, for each block B i Perform preliminary compression and obtain preliminary compression result C i In order to detect and merge duplicate data blocks, a hash function H is used to calculate the hash value h of each block. i =H(B i The hash function can be a common one like MD5 or SHA-256, ensuring the calculated hash value is highly unique and collision-resistant. Duplicate blocks are found and merged using the hash value, reducing redundant data. Finally, the merged data set is compressed to produce the final result.

[0088] Furthermore, the expression of the hash function is:

[0089] ;

[0090] Here, m is the block length, and p and q are large prime numbers. The hash function design effectively reduces the probability of hash collisions and enables rapid calculation of the hash value for each block. It's important to understand that this approach significantly improves data compression efficiency and accuracy. Furthermore, different hash functions and parameters can be selected based on actual needs to suit data compression requirements in different scenarios.

[0091] Furthermore, to further improve compression effectiveness, multiple compression algorithms, such as LZ77 and DEFLATE, can be introduced during the initial compression phase. These algorithms have different advantages when processing different types of data. For example, LZ77 is suitable for text data, while DEFLATE is suitable for binary data. By combining multiple compression algorithms, more efficient data compression can be achieved.

[0092] Furthermore, for the combined data set, more advanced compression algorithms, such as Huffman coding or arithmetic coding, can be used to further reduce the data volume. These algorithms generate the optimal coding scheme by statistically analyzing the frequency of occurrence of each symbol in the data, thereby achieving a higher compression ratio. Figure 3 shown.

[0093] The benefit of this embodiment is that through multiple steps such as block segmentation, preliminary compression, hash deduplication and final compression, the data compression rate can be effectively improved, the storage space occupied can be reduced, and the integrity and recoverability of the data can be guaranteed at the same time. Example 7

[0094] In order to solve the problem of high computational complexity in the data deduplication process, this embodiment further refines the deduplication algorithm. Specifically, first construct a similarity graph G = (V, E) between data blocks, where V is the set of data blocks and E is the set of edges. Each edge represents the similarity between two data blocks. Then, calculate the similarity of each node v i Degree d i , which is the number of edges connected to the node. Select the node v with the highest degree max As the central node. Deduplication is performed through the neighboring nodes of the central node. Let N(v max ) is the set of neighbor nodes of the central node. Update the similarity graph G and repeat the above steps until all nodes are processed.

[0095] Furthermore, the expression of the deduplication algorithm is:

[0096] ;

[0097] Among them, I(v i ,v j ) is the indicator function, indicating that node v i and v j Whether there are edges between them. This graph-theory-based approach can effectively identify and remove duplicate data blocks, reducing storage space waste. It's important to understand that by selecting the node with the highest degree as the central node, the part most likely to contain duplicate data can be prioritized, thereby improving deduplication efficiency.

[0098] Furthermore, to improve the efficiency of similarity graph construction, efficient graph data structures such as adjacency lists or adjacency matrices can be used. These data structures enable rapid insertion and querying of nodes and their adjacent nodes, accelerating the similarity graph construction process. Furthermore, parallel computing techniques can be introduced to utilize multi-core processors or multiple computers to process data blocks in parallel, further accelerating deduplication.

[0099] Furthermore, during deduplication, the similarity between data blocks can be determined by comparing their contents. Methods such as Jaccard similarity and cosine similarity can be used to calculate the similarity between two data blocks. By setting a threshold, when the similarity exceeds the threshold, the two data blocks are considered duplicates and merged. This can avoid false positives and improve deduplication accuracy.

[0100] Through this embodiment, the computational complexity in the data deduplication process can be effectively reduced, the deduplication efficiency can be improved, and the storage space occupied can be reduced. Example 8

[0101] In order to solve the storage cost problem caused by the change in data access frequency, this embodiment further optimizes the data management method. Specifically, a time threshold is preset to monitor the last access date of each data. The time difference is obtained by subtracting the current date from the last access date. If the time difference is greater than the time threshold, the data is converted from hot data to cold data. The metadata table is updated to record the state change of the data. The converted cold data is migrated to a low-cost storage medium and encrypted and differentially privately processed, such as Figure 5 As shown in the figure, mark the cold data as hidden and record its location and status in the metadata table.

[0102] Furthermore, the time threshold can be adjusted based on actual business needs. For example, a shorter time threshold can be set for frequently accessed data, while a longer time threshold can be set for less frequently accessed data. This allows for flexible adjustment of data classification criteria to ensure accurate classification. It is important to understand that this approach allows for dynamic adjustment of data storage locations, thereby optimizing storage costs.

[0103] Furthermore, cold data migration can be automated. Scripts can be written or existing data migration tools can be used to automatically migrate eligible cold data to low-cost storage media. During the migration process, incremental migration can be employed to migrate only newly added cold data, reducing the amount of data transferred. Furthermore, data verification can be performed during the migration process to ensure data consistency and integrity.

[0104] Furthermore, to protect the security of cold data, various encryption algorithms, such as AES and RSA, can be used. These algorithms provide strong data protection, preventing unauthorized access during data transmission and storage. Furthermore, to protect user privacy, differential privacy techniques can be applied. By adding noise and other methods, attackers cannot infer specific user information through data analysis.

[0105] Furthermore, after marking cold data as hidden, its location and status can be recorded in a metadata table. This way, when a user requests access to cold data, the system can quickly locate the corresponding data and perform recovery operations. It's important to understand that maintaining a detailed metadata table facilitates data management and retrieval, improving system responsiveness.

[0106] Through this embodiment, the life cycle of data can be effectively managed, storage costs can be reduced, and data security and privacy can be ensured. Embodiment 9

[0107] To address the issue of newly stored data lacking access history, this embodiment further optimizes the data management method. Specifically, if the monitored data does not have a last access date, the date the data was stored on the storage device is used as the last access date. This ensures that each piece of data has an initial access date, facilitating subsequent data classification and management.

[0108] Furthermore, the data storage date can be obtained from the creation timestamp in the file system or database. When data is first stored, the system automatically records its creation time and uses it as the last access date. It is important to understand that this approach can simplify data management and avoid data classification errors caused by a lack of access history.

[0109] Furthermore, to ensure the accuracy of data storage dates, you can regularly synchronize the time of your storage devices. You can use a time synchronization service such as NTP (Network Time Protocol) to ensure consistent time across all storage devices. This prevents data classification errors caused by time deviations and improves data management reliability.

[0110] Furthermore, for newly stored data, its creation time and initial status can be recorded in the metadata table. This allows for easy querying and updating of data status during subsequent data management. For example, when data is accessed, its last access date can be updated; when data has not been accessed for a long time, it can be converted to cold data and processed accordingly.

[0111] Furthermore, to increase data management flexibility, different time thresholds can be set to categorize different types of data. For example, a shorter time threshold can be set for critical business data to ensure it remains hot data, while a longer time threshold can be set for non-critical data to allow it to transition to cold data more quickly. This allows flexible adjustment of data classification standards to meet diverse business needs.

[0112] Through this embodiment, effective management of newly stored data can be ensured, data classification errors caused by lack of access history records can be avoided, and the accuracy of data management can be improved. Example 10

[0113] In order to solve the problem of module collaboration in the data management system, this embodiment further optimizes the architecture of the data management system. Specifically, the data management system includes the following modules:

[0114] The data classification module is used to classify data into hot data and cold data. This module dynamically adjusts the data classification by monitoring the access frequency and last access date of the data. Machine learning algorithms such as support vector machines (SVM) can be used to classify the data. Figure 11 As shown in the figure, the results of data classification can be intuitively displayed through the 3D decision surface diagram.

[0115] The data compression and deduplication module is used to compress and dedupe all data and apply differential privacy technology. This module uses a variety of compression algorithms, such as LZ77, DEFLATE, etc., combined with hash functions for deduplication. Figure 10 As shown in Figure 2, the compression effect can be intuitively demonstrated by optimizing the LZ77 compression parameter surface. In addition, differential privacy technology is applied to protect user privacy by adding noise and other methods.

[0116] Data migration module: used to migrate cold data to low-cost storage media and perform encryption and differential privacy processing. This module uses an automated method to migrate data and can use incremental migration to migrate only newly added cold data. Figure 4 As shown in the figure, the effect of data migration can be intuitively demonstrated through the relationship graph between data access frequency and storage cost.

[0117] The data hiding module is used to mark cold data as hidden and maintain the metadata table. This module records the location and status of cold data in the metadata table so that the data can be quickly located and restored when the user requests access. Figure 7 As shown in the figure, the temporal and spatial distribution heat map of the metadata table can intuitively display the distribution of metadata.

[0118] The intelligent scheduling module is used for automatically detecting and restoring data when a user requests to access cold data. The module automatically detects and restores the required cold data by monitoring the user's access requests in real time. Figure 6 As shown in FIG. 6, the optimization effect of cache hit rate can be intuitively displayed by the intelligent cache parameter optimization curve.

[0119] Further, the modules communicate with each other through standardized interfaces to ensure seamless data flow. The data classification module transmits the classification results to the data compression and deduplication module, which compresses and deduplicates the data and then transmits the results to the data migration module. After completing data migration, the data migration module transmits relevant information to the data hiding module, which records the location and status of the data in the metadata table. The intelligent scheduling module automatically detects and restores the data requested by the user by reading the metadata table.

[0120] Further, to improve the reliability and fault tolerance of the system, a distributed architecture can be used. The modules can be deployed on multiple servers, and tasks can be distributed through load balancing technology to ensure high availability of the system. In addition, a backup mechanism can be introduced to regularly back up important data to prevent data loss.

[0121] Further, to improve the security and privacy protection capabilities of the system, multi-level security measures can be used. In addition to data encryption and differential privacy technology, an access control mechanism can be introduced to limit user access to sensitive data. At the same time, regular security audits can be conducted to check for system security vulnerabilities and promptly fix potential risks.

[0122] In this way, efficient, secure, and reliable data management can be achieved.

[0123] Although the present application has been described above with reference to the preferred embodiments thereof, it is to be understood that the application is not limited to the above-described embodiments, but various modifications and changes can be made by those skilled in the art without departing from the spirit of the present application, and such modifications and changes should fall within the scope of the appended claims and their equivalents.

Claims

1. A data management method for a storage device, characterized in that: The steps of the data management method include: Classifying data in a storage device into hot data and cold data, wherein the hot data is frequently accessed data and the cold data is data that has not been accessed for a long time; Apply differential privacy technology to compress and deduplicate all data to reduce storage space and ensure data privacy; Migrating the cold data to low-cost storage media and performing encryption and differential privacy processing on the cold data to ensure the security of the data during transmission and storage; In the storage device, the cold data is marked as hidden so as not to be visible in regular queries, and a metadata table is maintained to record the location and status of the cold data so as to be retrieved when needed; When a user requests to access the cold data, the intelligent scheduling system automatically detects the location and status of the cold data, reads the data from the low-cost storage medium, decrypts and verifies its integrity, and restores the data to the storage device for user access; The metadata table maintained includes the cold data's identification, location information, encryption key, and timestamp, so as to quickly locate and restore the data when needed. The steps include: creating a metadata table to record the cold data's identification, location information, encryption key, and timestamp; regularly updating the metadata table to ensure data integrity and consistency; and quickly locating the cold data's location and status through the metadata table when a user requests access to the cold data. The deduplication process also uses a deduplication algorithm, which includes the following steps: constructing a similarity graph G = (V, E) between data blocks, where V is a set of data blocks and E is a set of edges; calculating the similarity of each node v i Degree d i ; Select the node v with the highest degree max As the central node; perform deduplication operation through the neighbor nodes of the central node, let N(v max ) is the set of neighbor nodes of the central node; update the similarity graph G, and repeat the above steps until all nodes are processed; The expression of the deduplication algorithm is: , where I(v i ,v j ) is the indicator function, indicating that node v i and v j Is there an edge between them? 2. The data management method of a storage device according to claim 1, wherein: Using a machine learning model to classify data, identify the cold data and hot data, and dynamically adjust data classification standards based on access frequency and importance, the steps include: collecting data access logs; using a support vector machine algorithm to classify data; dynamically adjusting classification thresholds to adapt to changes in data access patterns; and updating classification results to ensure accuracy of data classification. The data is compressed using the LZ77 compression algorithm, and the Bloom filter is used to identify and remove redundant data to free up storage space. The steps include: dividing the data into multiple blocks; performing LZ77 compression on each block; calculating the hash value of each block; using the Bloom filter to check whether the hash value exists, and if so, the block is considered redundant data; removing redundant data and retaining unique data blocks.

3. The data management method of a storage device according to claim 1, wherein: Cold data encryption uses the AES-256 encryption algorithm and combines differential privacy technology to ensure data security during transmission and storage. The steps include: generating a random key K; encrypting the cold data using the AES-256 algorithm to obtain encrypted data E; storing the encrypted data E together with the key K in a low-cost storage medium; The differential privacy technology protects the privacy of keys or data by adding noise, and the steps include: calculating the sensitivity Δf of the key K or data; generating noise that conforms to the Laplace distribution , where ϵ is the privacy budget; noise is added to the key K or data to increase key security and protect data privacy.

4. The data management method of a storage device according to claim 1, wherein: The intelligent scheduling system uses a caching mechanism to temporarily store recently accessed cold data in a cache to improve the response speed of subsequent access. The steps include: creating a cache to store recently accessed cold data; when a user requests access to cold data, first checking whether the data exists in the cache; if so, reading the data directly from the cache; if not, reading the data from a low-cost storage medium and adding it to the cache.

5. The data management method of a storage device according to claim 2, wherein: The steps of the LZ77 compression algorithm include: Divide the data into multiple blocks B1, B2, ..., B n ; For each block B i Perform preliminary compression to obtain C i ; Use hash function H to calculate the hash value h of each block i =H(B i ); Find duplicate blocks through hash values ​​and merge them to obtain the merged data set; Perform final compression on the merged data set to obtain the final compression result; The expression of the LZ77 compression algorithm is: , where m is the length of the block, and p and q are large prime numbers.

6. The data management method of a storage device according to claim 1, wherein: The data management method further includes the following steps: presetting a time threshold; monitoring the last access date of each piece of data; subtracting the current date from the last access date to obtain a time difference, and if the time difference is greater than the time threshold, converting the data from hot data to cold data; updating a metadata table to record data status changes; migrating the converted cold data to a low-cost storage medium, and performing encryption and differential privacy processing; marking the cold data as hidden, and recording its location and status in the metadata table.

7. The data management method of a storage device according to claim 1, wherein: If the monitored data does not have a last access date, the date on which the data is stored in the storage device is used as the last access date.

8. A data management system for a storage device, used to implement the data management method for a storage device according to any one of claims 1 to 7, characterized in that: The data management system includes: Data classification module, used to classify data into hot data and cold data; Data compression and deduplication module, used to compress and deduplicate all data and apply differential privacy technology; Data migration module, used to migrate cold data to low-cost storage media and perform encryption and differential privacy processing; Data hiding module, used to mark cold data as hidden and maintain metadata tables; The intelligent scheduling module is used to automatically detect and restore data when users request access to cold data.

Images