Large-scale data storage deduplication optimization method based on machine learning model

Through the data indexing and deduplication method based on hash functions and multi-level machine learning model, the problem of scalability and risk of error deletion in large-scale data storage is solved, and efficient data deduplication and storage is achieved Optimize to meet data security and privacy protection needs.

CN119003504AInactive Publication Date: 2024-11-22天津华信惠悦科技有限公司

Patent Information

Application Number
CN202411075734.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-07
Publication Date
2024-11-22
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

The existing bitmap data structures have problems such as limited scalability, risk of accidentally deleting data, high processing delays, large resource consumption, and difficulty in compatibility and privacy protection in large-scale data storage.

Method used

Using hash function-based data indexing and locally sensitive hashing technology, combined with multi-level machine learning model and rule engine, designing partitioning strategies and dynamic partition adjustment mechanisms based on data features, and through multimodal similarity calculation and heuristic optimization algorithm Perform data deduplication.

Benefits of technology

It significantly improves data storage efficiency, accurately identify and remove duplicate data, reduces storage space requirements, optimizes data processing speed and response speed, and meets data security and privacy protection requirements.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119003504A_ABST
    Figure CN119003504A_ABST
Patent Text Reader

Abstract

The invention relates to a large-scale data storage deduplication optimization method based on a machine learning model. Constructing a data index based on a hash function, mapping the data into a hash table with a fixed size, and indexing similar data by using a locality sensitive hash (LSH) technology; a partitioning strategy based on data features is designed, similar data is distributed to the same or similar partitions, cross-partition comparison operation is controlled and reduced, and a partitioning scheme is adjusted according to data changes; in combination with a rule engine and a multi-level machine learning model, performing preliminary de-duplication screening based on a predefined rule through the rule engine, inputting the screened data into the machine learning model, and performing similarity calculation and de-duplication processing; then designing a multi-modal similarity calculation method, learning feature representation of data by using a deep learning model, and calculating multi-modal similarity; a heuristic optimization algorithm is adopted to optimize parameters of similarity calculation, data is expressed as a graph structure, and duplicate removal processing is carried out through graph matching and clustering.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to a large-scale data storage deduplication optimization method, and specifically to a large-scale data storage deduplication optimization method based on a machine learning model. Background Art

[0002] In modern large-scale data storage systems, data deduplication technology is one of the key methods to improve storage efficiency and reduce costs. China Invention Publication No. CN115794788A describes a method for data deduplication using a bitmap data structure, which maps and classifies data subsets to reduce resource usage and increase deduplication speed. Although this technology provides some significant advantages, including reducing memory usage and improving processing speed, it still has some shortcomings and potential drawbacks. First, although the bitmap data structure is relatively efficient in space usage, its scalability is limited by the size of the bitmap and processing power. When processing extremely large-scale data, the size of the bitmap and the corresponding processing requirements will grow rapidly, resulting in a decrease in processing efficiency. Especially in cloud storage and big data environments, the dynamic changes in data require the deduplication system to have higher flexibility and scalability. In addition, due to the characteristics of the bitmap, that is, each bit can only indicate whether it exists or not and cannot record additional information, this will lead to the occurrence of erroneous deletion of data. When two different data blocks are accidentally mapped to the same bit value, the original method cannot distinguish the difference and mistakenly regards it as duplicate data. Although this method improves processing speed by reducing the representation of data, in practice, each data deduplication process needs to traverse the entire bitmap and check the state of each bit, which will cause significant processing delays when the amount of data is large. In addition, the formation and deduplication of each mapping set also consumes computing resources, especially in an environment where data is frequently updated.

[0003] Secondly, although bitmaps are more space-saving when storing a single data state, as the types and complexity of data increase, in order to maintain an effective deduplication effect, more bitmaps or more complex data structures need to be introduced. This not only increases the complexity of management, but also leads to increased resource consumption, especially in scenarios where bitmaps need to be updated frequently. Then, although the bitmap structure simplifies the representation of data, this method has low transparency and is difficult to meet specific data security and privacy protection requirements. When processing sensitive data, how to ensure that the data is not accessed by unauthorized or processed incorrectly is an important aspect that must be considered when designing a deduplication system. Finally, this method relies on specific data structures and processing logic, and there are compatibility issues with the integration of existing data storage and management systems. In a diverse technical environment, how to effectively integrate new deduplication technologies and how to deal with interfaces and data exchanges with other data processing processes are technical challenges that need to be solved during implementation. Summary of the invention

[0004] The purpose of the present invention is to provide a large-scale data storage deduplication optimization method based on a machine learning model, thereby solving some of the drawbacks and shortcomings pointed out in the background technology.

[0005] The present invention solves the above-mentioned technical problems by adopting the following technical solutions:

[0006] S1. Build a data index based on hash function, map the data into a fixed-size hash table, and use locality sensitive hashing (LSH) technology to index similar data.

[0007] S2. Design a partitioning strategy based on data features, assign similar data to the same or similar partitions, control and reduce cross-partition comparison operations, and adjust the partitioning scheme according to data changes;

[0008] S3. In the process of data deduplication, the rule engine is combined with the multi-level machine learning model. First, the rule engine performs preliminary deduplication screening based on predefined rules, and the screened data is input into the machine learning model for similarity calculation and deduplication processing;

[0009] S3.1, wherein the multi-level machine learning model includes a shallow model and a deep model, and the prediction results of multiple models are combined by an integrated learning method to control and improve the accuracy of deduplication;

[0010] S3.2, then design a multimodal similarity calculation method, combine the similarities of multiple data types such as text, image, and audio, use a deep learning model to learn the feature representation of the data, and calculate the multimodal similarity;

[0011] S3.3. A heuristic optimization algorithm is used to optimize the parameters of similarity calculation. By designing a deduplication method based on a graph structure, the data is represented as a graph structure, and deduplication is performed through graph matching and clustering.

[0012] Furthermore, the steps of indexing similar data include:

[0013] S1. First, a hash function with adaptive parameters is used:

[0014]

[0015] where x=(x1,x2,…,x n ) is the data vector, a i is the adaptive coefficient, γ i is an exponent that changes dynamically according to the data, and p is a large prime number used to control the uniformity of data mapping;

[0016] S2. Then calculate the size N of the hash table using the expression:

[0017]

[0018] Where D is the size of the data set, S is the expected query speed, and k is the adjustment factor, which reflects the logarithmic relationship between data size and query speed;

[0019] S3. Introduce local sensitive hashing (LSH) technology and design the LSH function family:

[0020]

[0021] where h j (x) is the sub-hash function, c j is the weight, δ j is the data sensitivity parameter, and q is a large prime number that controls the high probability aggregation of similar data.

[0022] Furthermore, the partition strategy construction process based on data features is as follows:

[0023] First, machine learning models including autoencoders or deep neural networks are used to extract features from stored data, and the extracted high-dimensional feature vectors are converted into data fingerprints. The feature vectors are simplified through hash functions to generate fingerprints for subsequent deduplication comparison.

[0024] Then, an improved clustering algorithm including spectral clustering or Gaussian mixture models is used to process data fingerprints, classify data items with high similarity into the same cluster partition, perform deduplication within the cluster partition, and control and reduce cross-partition comparison operations;

[0025] Finally, a dynamic partition adjustment mechanism is integrated to adjust the partition structure in real time according to the data inflow speed and query frequency. When the partition data grows too fast, the partition splitting is controlled.

[0026] Furthermore, the means of converting the high-dimensional feature vector into a data fingerprint process include:

[0027] S1. First, use an autoencoder or deep neural network to extract features from the input data, using nonlinear activation functions and advanced mathematical transformations through the formula:

[0028]

[0029] To learn the latent space representation of the data, where x is the input vector and w i ,u ij , and α j are the learning weights and nonlinear degree parameters of the model respectively;

[0030] S2. Next, the extracted high-dimensional feature vector is passed through a hash function:

[0031]

[0032] Converted to data fingerprint, where x j is a component of the eigenvector, c j is the selected coefficient, and p is a large prime number used for modular operations to generate data fingerprints with high information preservation;

[0033] S3. Finally, use the fingerprint to perform duplicate comparison by calculating the weighted Hamming distance between two fingerprints:

[0034]

[0035] Where x and y are two data fingerprints, γ k is the weight coefficient based on the position, l is the length of the fingerprint, and the d(x,y) function is used to control the identification and removal of duplicate data.

[0036] Furthermore, the dynamic partition adjustment mechanism construction includes the following steps:

[0037] First, use the autoencoder or deep learning network to extract features from the stored data, using the formula:

[0038]

[0039] Generate data fingerprint, where x represents the input data vector, β i The nonlinear adjustment parameter representing the feature;

[0040] Then monitor the number and changes of data fingerprints in each partition in real time through the change rate function:

[0041]

[0042] Where N is the number of data fingerprints in the partition and t is the time to evaluate the growth and diversity of the data;

[0043] Then, based on the similarity of data fingerprints and real-time monitoring information, a prediction model is used:

[0044]

[0045] Where y represents the deduplication efficiency, Z represents the data growth rate and query frequency characteristics, and predicts the partition performance and decides whether to adjust the partition;

[0046] Finally, when the data in a partition grows too fast, resulting in a decrease in deduplication efficiency, cluster analysis is performed:

[0047]

[0048] where x j is the data fingerprint, ξ is the cluster center, k is the number of clusters, which triggers partition splitting, controls similar data to remain in similar partitions, and optimizes subsequent deduplication operations.

[0049] Furthermore, the multi-level machine learning model construction includes:

[0050] S1. First, preprocess the input data, extract features using autoencoders or deep neural networks, and generate high-dimensional feature vectors through the function:

[0051]

[0052] Converted to data fingerprint, where x is the input data vector, w i , α, β i is the learning parameter;

[0053] S2. Then train multiple shallow models and deep models. The shallow models focus on extracting basic features by optimizing the function:

[0054]

[0055] Optimization, where θ is the model parameter, γ and φ j is the regularization coefficient, and the deep model is calculated by the formula:

[0056]

[0057] Extract complex patterns and nonlinear relationships, where z is the input feature and w k and bx are weights and biases, σ is the activation function;

[0058] S3, then through the integrated learning methods including Bagging, Boosting and Stacking, the prediction results of the shallow model and the deep model are integrated, using the integrated formula:

[0059]

[0060] Combine the results, where h i (x) is the prediction result of the i-th model, α j is the weight factor and N is the number of models.

[0061] Furthermore, the multimodal similarity calculation method comprises the following steps:

[0062] S1. First, preprocess the input data and extract features. For text data, use the pre-trained natural language processing model to extract features and obtain a high-dimensional text vector representation:

[0063]

[0064] Where x is the input text data, α i and β i are model parameters;

[0065] S2. For image data, use convolutional neural networks (CNNs), including ResNet and VGGNet, to extract features and convert images into feature vector representations:

[0066]

[0067] Where x is the input image data, γ j and δ j are model parameters;

[0068] For audio data, use convolutional neural network or recurrent neural network RNN ​​including LSTM to process and extract audio feature vectors:

[0069]

[0070] Where x is the input audio data, ∈ k and η k are model parameters;

[0071] S3. Then, the data of different modalities are normalized and aligned to control the feature vectors of different data types to have the same dimension, and the autoencoder or variational autoencoder VAE is used to map the features of each modality to a common embedding space through the function:

[0072]

[0073] To achieve feature fusion, where x is the input feature vector, λ i is the fusion parameter;

[0074] S4. Finally, a multimodal similarity calculation model is designed. By combining different modal features, the similarity between data is calculated. The deep learning model including Siamese network is used to learn the similarity between different modal features, and the multimodal similarity function is defined:

[0075]

[0076] Among them, α, β and γ are weight parameters of different modes, which control the comprehensive similarity calculation of multimodal data.

[0077] Furthermore, the heuristic optimization algorithm optimizes the similarity calculation process and includes the following steps:

[0078] S1. First, heuristic optimization algorithms including genetic algorithms are used to optimize similarity calculation parameters, set the initial parameter set, and use the fitness function:

[0079]

[0080] Evaluate the effect of parameter combination, where α, β, γ are weight parameters in the similarity function, S(x i ,y i ) is the similarity calculation result, label i For the true label, the parameter set is iteratively updated to obtain the optimal parameter;

[0081] S2. Then represent the data as a graph structure:

[0082] G=(V,E,w)

[0083] Where V is the node set, E is the edge set, and w is the weight of the edge, which represents the similarity between data items;

[0084] Through the function:

[0085]

[0086] Calculate the edge weight, where x i and x j is the feature vector of the data item, and λ is the adjustment parameter;

[0087] S3. Then use the graph matching and clustering algorithm to perform deduplication processing, identify the predicted duplicate data items through the maximum matching algorithm, and define the maximum matching function:

[0088]

[0089] Where m ij is the matching indicator function;

[0090] S4. Finally, the spectral clustering algorithm is used to group the graphs, and the eigendecomposition is performed through the graph Laplacian matrix L = DA, where D is the degree matrix and A is the adjacency matrix, through the function:

[0091]

[0092] Group the data, where λ j is the eigenvalue, φ j is a feature vector to further remove duplicate data.

[0093] Beneficial effects of the present invention:

[0094] 1. The present invention can significantly improve data storage efficiency by accurately identifying and removing duplicate data. Using machine learning techniques, including deep learning models and autoencoders, data items that are similar or duplicate in content can be identified and processed even when there are slight differences in the data representation. This not only reduces the need for physical storage space, but also optimizes the data backup and recovery process, reducing the associated time and cost overhead.

[0095] 2. Machine learning models can learn and infer complex data patterns and similarity information from large amounts of data, accelerating data processing speed. During data migration or synchronization, duplicate data can be quickly identified, reducing the amount of data that needs to be transmitted, thereby speeding up processing and reducing network bandwidth consumption.

[0096] 3. The deduplicated data set is smaller in size, and data query and access will be faster. This is especially important for application scenarios that require frequent data reading, including online trading platforms, big data analysis, and machine learning tasks, which can improve system response speed and user satisfaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0097] Figure 1 This is a flow chart of the large-scale data storage deduplication optimization method based on the machine learning model of the present invention.

[0098] Figure 2 A module diagram for indexing similar data for the present invention.

[0099] Figure 3 A flow chart is constructed for the partitioning strategy based on data features of the present invention.

[0100] Figure 4 This is a module diagram of the dynamic partition adjustment mechanism of the present invention. DETAILED DESCRIPTION

[0101] The specific implementation modes of the present invention will be described in detail below in conjunction with the accompanying drawings.

[0102] Combined with Figure 1 ,First, a data index based on hash function is constructed and a ,fixed-size hash table is used to map data, and similar data is ,indexed by locality-sensitive hashing (LSH) technology, so that similar data items are ,more likely to be mapped to the same or similar buckets.

[0103] Secondly, design a partitioning strategy based on data characteristics, assign similar data to the same or similar partitions, reduce cross-partition comparison operations, and dynamically adjust the partitioning scheme according to data changes to adapt to changes in data distribution.

[0104] Next, the deduplication process combines the rule engine with a multi-level machine learning model. The rule engine uses predefined rules to perform preliminary deduplication screening, and the filtered data is then used by the machine learning model for more sophisticated similarity calculation and deduplication processing.

[0105] The multi-level machine learning model covers shallow and deep models, and improves the accuracy of deduplication by combining the prediction results of multiple models through ensemble learning methods. Furthermore, a multimodal similarity calculation method is designed, which combines the similarities of multiple data types such as text, images, and audio, and uses a deep learning model to learn the feature representation of the data and calculate the multimodal similarity, so that multiple types of data can be more comprehensively understood and processed.

[0106] Finally, a heuristic optimization algorithm is used to optimize the parameters of similarity calculation, and a deduplication method based on graph structure is designed. By representing the data as a graph structure and using graph matching and clustering algorithms for deduplication, this method can effectively identify and remove duplicate data, especially when dealing with complex data relationships.

[0107] Embodiment 1:

[0108] Combined with Figure 2 In the embodiment, a large user database is processed, which contains the user's name, address, email, phone number and social media activity record information, and the duplicate data in the database is cleaned up by a deduplication optimization method based on a machine learning model. A hash function with adaptive parameters is used to index the data, and a user data vector x = (x1, x2, ..., x n ), where each x i Represents a feature of user data, x1 represents the user's name, z2 represents the user's address, and so on. The following hash function is used to process the data:

[0109]

[0110] Among them, a i is the adaptive coefficient, and its value range is (0.1,1); γ i is an exponent that changes dynamically according to the data, with a value range of 0.5, 2; p is a large prime number, set to 997.

[0111] Set the user data vector x = (3, 5, 2), and the corresponding parameters are a = (0.5, 0.8, 1.2), γ = (1, 0.7, 1.5). We can calculate:

[0112] f(x)=(0.5·3 1 +0.8·5 0.7 +1.2 2 1.5 )mod 997

[0113] The calculation results are:

[0114] 3 1 =3,5 0.7 ≈3.57,2 1.5 =2.83

[0115] f(x)=(0.5·3+0.8·3.57+1.2·2.83)mod 997

[0116] f(x)=(1.5+2.856+3.396)mod 997

[0117] f(x)=7.752mod 997

[0118] The result is 7.

[0119] Next, design a partitioning strategy based on data features to assign similar data to the same or similar partitions. Use the hash value calculated by the above hash function to map user data to different buckets in the hash table. Set up the following user data vectors:

[0120] x1=(3,5,2),x2=(4,4,3),x3=(5,2,1),x4=(3,5,2)

[0121] The calculated hash values ​​are:

[0122] f(x1)=7, f(x2)=8, f(x3)=6, f(x4)=7

[0123] Assign this data to the hash table, and the corresponding buckets are:

[0124] {7:[x1,x4],8:[x2],6:[x3]}

[0125] The embodiment further calculates the size N of the hash table using the following expression:

[0126]

[0127] Where D is the data set size, S is the expected query speed, and k is the adjustment factor, which reflects the logarithmic relationship between data size and query speed. Set the data set size D = 10 6 records, the expected query speed S = 100 records / second, and the adjustment factor k = 0.5.

[0128] Substitute into the formula to calculate:

[0129]

[0130] The calculation results are:

[0131] log(10 6 )=6

[0132] 100 2 =10000

[0133]

[0134] Therefore, the size of the hash table is N=30000.

[0135] The last example introduces the local sensitive hashing (LSH) technology to process large-scale user data. The size of the data set currently processed by the company is set to D = 10 6 Records, the data is processed by designing LSH function family:

[0136]

[0137] Here, h j (x) is the sub-hash function, c j is the weight, δ j is the data sensitivity parameter, and q is a large prime number that controls the high probability aggregation of similar data.

[0138] Assume the following user data vector:

[0139] x1=(3,5,2),x2=(4,4,3),x3=(5,2,1),x4=(3,5,2)

[0140] Set the parameters of the LSH function family: Sub-hash function:

[0141]

[0142] Weight c j The value range is 0.1, 1, and the data sensitivity parameter δ j The value range is 0.5, 2, the large prime number q = 997, and the coefficient of the sub-hash function is a 1i =(0.5,0.7,0.9),a 2i =(0.3,0.6,0.8), weights c1 = 0.8, c2 = 0.6

[0143] Data sensitivity parameters: δ1 = 1.5, δ2 = 0.9

[0144] Compute the subhash value for each data vector:

[0145] For x1:

[0146] h1(x1)=0.5·3+0.7·5+0.9·2=1.5+3.5+1.8=6.8

[0147] h2(x1)=0.3·3+0.6·5+0.8·2=0.9+3+1.6=5.5

[0148] For x2:

[0149] h1(x2)=0.5·4+0.7·4+0.9·3=2+2.8+2.7=7.5

[0150] h2(x2)=0.3·4+0.6·4+0.8·3=1.2+2.4+2.4=6

[0151] For x3:

[0152] h1(x3)=0.5·5+0.7·2+0.9·1=2.5+1.4+0.9=4.8

[0153] h2(x3)=0.3·5+0.6·2+0.8·1=1.5+1.2+0.8=3.5

[0154] For x4:

[0155] h1(x4)=0.5·3+0.7·5+0.9·2=1.5+3.5+1.8=6.8

[0156] h2(x4)=0.3·3+0.6·5+0.8·2=0.9+3+1.6=5.5

[0157] Then calculate the LSH value:

[0158] g(x1)=(0.8·6.8 1.5 +0.6 5.5 0.9 )mod997

[0159] g(x2)=(0.8·7.5 1.5 +0.6 6 0.9 )mod997

[0160] g(x3)=(0.8·4.8 1.5 +0.6 3.5 0.9 )mod997

[0161] For x1:

[0162] 6.8 1.5 ≈17.68,5.5 0.9 ≈5.11

[0163] g(x1)=(0.8·17.68+0.6·5.11)mod997

[0164] g(x1)=(14.144+3.066)mod997=17.21

[0165] For x2:

[0166] 7.5 1.5 ≈20.56,6 0.9 ≈5.98

[0167] g(x2)=(0.8·20.56+0.6·5.98)mod997

[0168] g(x2)=(16.448+3.588)mod997=20.036

[0169] For x3:

[0170] 4.8 1.5 ≈10.57,3.5 0.9 ≈3.17

[0171] g(x3)=(0.8·10.57+0.6·3.17)mod997

[0172] g(x3)=(8.456+1.902)mod997=10.358

[0173] For x4:

[0174] 6.8 1.5 ≈17.68,5.5 0.9 ≈5.11

[0175] g(x4)=(0.8·17.68+0.6·5.11)mod997

[0176] g(x4)=(14.144+3.066)mod997=17.21

[0177] It can be seen that the LSH values ​​of data vectors x1 and x4 are the same, both 17.21, indicating duplicate data. The LSH values ​​of data vectors x2 and x3 are 20.036 and 10.358 respectively. Through the above steps, the LSH technology successfully indexes similar data efficiently, making similar data items clustered together with a high probability.

[0178] Embodiment 2:

[0179] Combined with Figure 3 The embodiment performs deduplication processing on a large user database and adopts a large-scale data storage deduplication optimization method based on a machine learning model. Using the machine learning model, an autoencoder is used to extract features from the input data, and a nonlinear activation function and advanced mathematical transformation are used. Through the formula:

[0180]

[0181] To learn the latent space representation of the data, where x is the input vector and w i ,u ij and α j are the learning weights and nonlinear degree parameters of the model respectively.

[0182] Assume the following user data vector:

[0183] x1=(3,5,2),x2=(4,4,3),x3=(5,2,1),x4=(3,5,2)

[0184] Set the parameters of the model as follows:

[0185] Weight w i The value range is 0.1,1;

[0186] Nonlinear degree parameter α j The value range is 0.5,2;

[0187] Learning weight u ij The value range is 0.1,1.

[0188] The specific values ​​are:

[0189] Weight: w = (0.5, 0.8, 1.0);

[0190] Nonlinear degree parameter: α = (1, 0.7, 1.5);

[0191] Learning weights:

[0192] Compute the feature representation for each data vector:

[0193]

[0194] f(x1)=0.5·tanh(0.2·3 1 +0.3 5 0.7 +0.5 2 1.5 )+0.8·tanh(0.4·3 1 +0.5 5 0.7 +0.7 2 1.5 )

[0195] +1.0·tanh(0.6·3 1 +0.8·5 0.7 +0.9 2 1.5 )

[0196] Evaluate the inner expression:

[0197] 3 1 =3,5 0.7 ≈3.57,2 1.5 ≈2.83

[0198]

[0199] Substitute it into the tanh activation function:

[0200] tanh(3.086)≈0.9958

[0201] tanh(4.966)≈0.9999

[0202] tanh(7.203)≈1.0000

[0203] The final calculation is:

[0204] f(x1)=0.5·0.9958+0.8·0.9999+1.0·1.0000

[0205] f(x1)=0.4979+0.7999+1.0000=2.2978

[0206] Performing similar calculations on other data vectors yields:

[0207] f(x2)=2.4503

[0208] f(x3)=2.1482

[0209] f(x4)=2.2978

[0210] The original data is mapped into the latent space representation through feature extraction to obtain a high-dimensional feature vector.

[0211] Then, the embodiment converts these high-dimensional feature vectors into data fingerprints, and generates fingerprints by simplifying the feature vectors through a hash function.

[0212] Set an existing eigenvector:

[0213] f(x1)=2.2978, f(x2)=2.4503, f(x3)=2.1482, f(x4)=2.2978

[0214] Next, the high-dimensional feature vector is converted into a data fingerprint using the following hash function:

[0215]

[0216] x j is a component of the eigenvector, c jis the selected coefficient, ranging from 0.1 to 1, and p is a large prime number used for modular operations, set to 997. The selected coefficient c = (0.3, 0.6, 0.9), and the large prime number p = 997.

[0217] Compute the data fingerprint for each feature vector:

[0218]

[0219] Evaluate the inner expression step by step:

[0220] sin(0.3·2.2978)≈sin(0.68934)≈0.637

[0221] sin(0.6·2.2978)≈sin(1.37868)≈0.980

[0222] sin(0.9·2.2978)≈sin(2.06802)≈0.879

[0223] Convert to a value between 0 and 1:

[0224]

[0225] Multiply by a large prime and round:

[0226]

[0227] Perform an XOR operation (⊕):

[0228] h(f(x1))=816⊕987⊕936

[0229] The calculation results are:

[0230] 816⊕987=171

[0231] 171⊕936=759

[0232] Therefore, the data fingerprint after the feature vector f(x1) is converted is 759.

[0233] Similar calculations are performed for other eigenvectors to obtain:

[0234] h(f(x2))=871

[0235] h(f(x3))=689

[0236] h(f(x4))=759

[0237] The next step of the embodiment uses the calculation of weighted Hamming distance to determine the similarity between data, and identifies and removes duplicate data accordingly. The following data fingerprint is obtained from the previous step:

[0238] h(f(x1))=759, h(f(x2))=871, h(f(x3))=689, h(f(x4))=759

[0239] Now we need to use the following weighted Hamming distance formula to perform duplicate removal comparison:

[0240]

[0241] Where x and y are two data fingerprints, γ k It is the weight coefficient based on the position, and its value range is set to 0.1, 1, and l is the length of the fingerprint. All fingerprints are set to have the same length, which is 8 bits of binary number, and the weight coefficient of all bits is the same, which is set to 0.5.

[0242] Convert the fingerprint to binary form for calculation:

[0243] Convert 759 to binary: 1011110111

[0244] Convert 871 to binary: 1101100111

[0245] Convert 689 to binary: 1010110001

[0246] Since h(f(x4)) = h(f(x1)), the binary value of x4 is the same as x1.

[0247] Compute the Hamming distance between x1 and x4:

[0248]

[0249] Since x1 and x4 are identical, all x 1k ⊕x 4k =0, therefore:

[0250] d(x1,x4)=0

[0251] This indicates that x1 and x4 are duplicate data.

[0252] Compare the binary fingerprints of x1 and x2:

[0253]

[0254] Calculate the XOR result of each bit and multiply it by the weight 0.5, the result is as follows:

[0255] d(x1,x2)=0.5·(1+0+1+0+0+0+0+0+0+0)=0.5·2=1.0

[0256] This result shows that x1 and x2 have two different bits in binary representation, and the fingerprint distance is 1.0, indicating that the two are not exactly the same data.

[0257] Embodiment 3:

[0258] Combined with Figure 4 , the embodiment implements a dynamic partition adjustment mechanism to intelligently manage and optimize the data storage structure. First, the stored data is feature extracted using an autoencoder or a deep learning network. The data fingerprint is generated through a specific nonlinear mathematical transformation formula:

[0259]

[0260] Where x represents the input data vector, β i It represents the nonlinear adjustment parameter of the feature, and its value range is set to 0.5, 2.

[0261] Assume the following user data vector:

[0262] x1=(2,3),x2=(2,2),x3=(3,5)

[0263] The nonlinear adjustment parameter is β = (1, 1.5), and the fingerprint generation formula is applied to each data vector:

[0264]

[0265] Calculate the specific value:

[0266]

[0267] Similarly, calculate the data fingerprints of x2 and x3:

[0268]

[0269] Based on the generated data fingerprint, the dynamic partition adjustment mechanism automatically allocates data to different storage partitions. The partition thresholds are set according to the value range of the fingerprint, which are (4.0-5.0) and (5.0-7.0).

[0270] x1's fingerprint of 4.924 assigns it to the (4.0-5.0) partition, x2's fingerprint of 4.205 is also assigned to the (4.0-5.0) zone, and x3's fingerprint of 6.918 is assigned to the (5.0-7.0) partition.

[0271] The embodiment then introduces a dynamic partition adjustment mechanism to optimize data storage. In order to further improve the responsiveness and efficiency of the system, the company decides to monitor the number and changes of data fingerprints in each partition in real time, and use a change rate function to evaluate the growth and diversity of data.

[0272] The system sets up a real-time monitoring mechanism to track the number of data fingerprints in each data partition. The change rate function is defined as:

[0273]

[0274] Where N is the number of data fingerprints in the partition and t is time. This function provides an instantaneous measurement of the data growth rate, that is, the ratio of the rate of change of the number of data fingerprints relative to the current number.

[0275] Assume that a data partition has 1000 data fingerprints at the beginning. After one hour, the number of data fingerprints increases to 1200. Calculate the data fingerprint change rate during this period.

[0276] The number of initial data fingerprints N initial =1000;

[0277] The number of data fingerprints N after one hour final =1200;

[0278] Growth amount ΔN=N final -N initial =1200-1000=200.

[0279] Rate of change calculation:

[0280]

[0281] This means that the data fingerprint of this partition grows by 20% every hour.

[0282] The embodiment further uses a prediction model to evaluate partition performance and determine whether partition adjustment is needed, uses a statistical probability model to predict deduplication efficiency, and adjusts the data partition strategy accordingly. The formula is as follows:

[0283]

[0284] Where y represents the deduplication efficiency, X represents the data growth rate and query frequency, and σ 2 represents the variance, and μ represents the mean of the predicted deduplication efficiency.

[0285] Variance σ 2 The value range of can be set to (0.1, 0.5), indicating the sensitivity of the model to data changes; the mean μ can be calculated based on historical data, and its value range is set to (0.7, 0.9), indicating the deduplication efficiency under normal conditions; the data growth rate of the current partition A is set to 0.2 per hour (that is, the data fingerprint increases by 20% per hour), and the query frequency is 30 times / hour; according to past data and experimental results, μ is set to 0.85, σ 2 =0.2.

[0286] Calculate the predicted probability of deduplication efficiency:

[0287]

[0288] Evaluate the deduplication efficiency when y=0.8:

[0289]

[0290] This prediction model can be used to evaluate the deduplication efficiency of different partitions under the current data growth rate and query frequency. If the predicted deduplication efficiency is lower than the threshold of 0.8, the partition is adjusted.

[0291] The last embodiment introduces a dynamic partition adjustment strategy, which relies on cluster analysis to decide whether to split the partition, ensuring that similar data remains in similar partitions, using the formula:

[0292]

[0293] where x j is the data fingerprint, ξ is the cluster center, and k is the number of clusters.

[0294] The data fingerprint set in the partition is set to {759, 871, 689, 759}; the number of clusters k is determined to be 2, which means that these data fingerprints are divided into two groups; the K-means clustering algorithm is used for clustering, and two cluster centers ξ1=750 and ξ2=880 are randomly initialized.

[0295] Calculate the distance from each data fingerprint to each cluster center and assign it to the nearest cluster center:

[0296] ||759-750|| 2 =81,||759-880|| 2 =14641

[0297] ||871-750|| 2 =14641,||871-880|| 2 =81

[0298] ||689-750|| 2 =3721,||689-880|| 2 =36481

[0299] ||759-750|| 2 =81,||759-880|| 2 =14641

[0300] x1, x3, x4 are assigned to ξ1, and x2 is assigned to ξ2.

[0301] Calculate the new cluster center and iterate until the cluster center does not change or reaches the preset number of iterations. If the amount of data in the cluster center is still too large and the deduplication efficiency is not ideal, this cluster will trigger a partition split, create a new partition, and migrate the data fingerprint belonging to the cluster to the new partition, thereby reducing the load on the original partition and optimizing the deduplication operation and query efficiency.

[0302] Embodiment 4:

[0303] The embodiment builds a multi-level machine learning model to convert these features into efficient data fingerprints. The input data first goes through a preprocessing phase, including standardization, denoising, and missing value processing to ensure that the data quality is suitable for machine learning. The function used is:

[0304]

[0305] Where x is the input data vector, w i , α, β i is the learning parameter, where w i The range can be set to (0.1,1), the range of α is (0.5,2), and the range of β i The range is (0.01,0.5).

[0306] The data vector sample to be processed is set to x = (0.5, 1.2, 0.7), and the selected parameters are:

[0307] w=(0.5,0.8,1.0)

[0308] α=(1,0.7,1.5)

[0309] β=(0.2,0.1,0.3)

[0310] Calculate the features of each dimension:

[0311]

[0312] Calculate the specific value:

[0313] 0.5·0.5·e -0.1 ≈0.5·0.5·0.905≈0.226

[0314] 0.8 1.2 0.7 ·e -0.12 ≈0.8·1.088·0.886≈0.770

[0315] 1.0 0.7 1.5 ·e -0.21 ≈1.0·0.586·0.811≈0.476

[0316] total:

[0317]

[0318] This data fingerprint 1.472 is converted from the original data vector (0.5, 1.2, 0.7) and can be used for subsequent deduplication and data management activities.

[0319] The embodiment further combines shallow and deep models to improve deduplication efficiency and data processing capabilities. The shallow model mainly focuses on extracting basic features of the data, and the function is defined as:

[0320]

[0321] Where θ is the model parameter, γ and φ j is the regularization coefficient, which is used to control the model complexity and prevent overfitting. The recommended range of parameter γ is (0.01, 0.1), j The value range of is [0,2π].

[0322] Deep models are used to process and identify complex patterns and nonlinear relationships in data. The model formula is:

[0323]

[0324] Where z is the input feature, w k and b k are weights and biases, σ is the activation function, and the sigmoid activation function is used.

[0325] set up:

[0326] Input feature x j :(1,2),(2,3),(3,5);

[0327] Target output y j : 5,8,14;

[0328] Select the linear regression model and initialize the parameters θ = (0.5, 1), γ = 0.05, φ = π / 4.

[0329] Compute one optimization iteration:

[0330]

[0331] For x1=(1,2),y1=5:

[0332] θ T x1=0.5×1+1×2=2.5

[0333] (5-2.5) 2 =6.25

[0334] A neural network with one hidden layer is used. The weights w = (0.6, 0.8) and the bias b = 0.2 are initialized.

[0335] Apply the model to input z=(1,2):

[0336] g(z)=σ(0.6×1+0.8×2+0.2)=σ(2.6)

[0337] Use Sigmoid activation function:

[0338]

[0339] The last example uses an ensemble learning method to further improve prediction accuracy and deduplication efficiency, fusing the prediction results of multiple shallow and deep models to achieve optimal data processing performance. Ensemble learning is a machine learning paradigm that aims to combine multiple models to improve prediction accuracy and stability. The ensemble technology Bagging trains each model by randomly sampling (with replacement) from the original data set, and then averages their predictions to reduce variance.

[0340] The prediction function of the ensemble is defined as:

[0341]

[0342] where h i (x) is the prediction result of the i-th model, α j is the weight factor and N is the number of models.

[0343] The following simplified data vector and target output are assumed:

[0344] Input feature x j :(1,2),(2,3),(3,5);

[0345] Target output y j :5,8,14.

[0346] Shallow model and deep model:

[0347] A shallow model (linear regression) and a deep model (neural network) have been trained and provide the following predictions:

[0348] h1(x)=4.9,7.8,13.6 (shallow model prediction)

[0349] h2(x)=5.1,8.1,14.1 (deep model prediction)

[0350] Use bagging to average the predictions of shallow and deep models:

[0351]

[0352] Through this ensemble learning approach, the company is able to obtain more accurate prediction results from multiple models and optimize the data deduplication process.

[0353] Embodiment 5:

[0354] The embodiment uses machine learning technology to process and analyze multimodal data to enhance the consistency and comparability of the data. First, a multimodal similarity calculation method is used to extract features from text data using a pre-trained natural language processing model BERT. The text feature extraction formula is defined as:

[0355]

[0356] x represents the feature vector of the input text data, is the word embedding vector, α i and β i are model parameters. Adjusting these parameters can help the model better capture the important features of text data. i The value range can be set to 0.1, 1, β i The value range is (0.5,2).

[0357] The text data sample to be processed is:

[0358] Input text: "The quick brown fox jumps over the lazy dog"

[0359] The text data preprocessing and feature extraction process is as follows:

[0360] Preprocessing: convert to lowercase, remove stop words, and perform stemming.

[0361] Feature extraction: Use the pre-trained BERT model to convert text into word embedding vectors. Set the word embedding vector x = [x1, x2, …, x n ], where each x i Represents the embedding vector of a word.

[0362] The following model parameters are set:

[0363] α i =0.8,β i =1.2 for all i.

[0364] Apply the formula to calculate the text vector representation:

[0365]

[0366] Assume that x contains three word vectors, and the values ​​of each vector after processing are 0.5, 0.3, and 0.7:

[0367] T(x)=0.8·tanh(1.2·0.5)+0.8·tanh(1.2·0.3)+0.8·tanh(1.2·0.7)

[0368] Calculate the specific value:

[0369] T(x)=0.8·tanh(0.6)+0.8·tanh(0.36)+0.8·tanh(0.84)

[0370] T(x)≈0.8·0.537+0.8·0.349+0.8·0.686≈0.4296+0.2792+0.5488≈1.2576

[0371] With this approach, the company is able to extract rich information from text data and transform it into useful, informative, high-dimensional vector representations.

[0372] The embodiment further uses a convolutional neural network (CNN) ResNet to process image data. The key features in the image are automatically learned through multi-layer filters, which are suitable for capturing local and global features of the image. The image feature extraction formula is defined as:

[0373]

[0374] Where x represents the image feature vector after CNN processing, γ j and δ j is a model parameter used to adjust the sensitivity and dimension of feature extraction. j The value range of can be set to (0.1,1),δ j The value range is (0.01,0.1).

[0375] The audio feature extraction formula is defined as:

[0376]

[0377] Where x is the feature vector of the input audio data, ∈ k and η k is a model parameter used to adjust the response speed and sensitivity of the feature extraction process. k The value range of can be set to (0.1,1),η k The value range is (0.01,0.1).

[0378] The processed data is:

[0379] Image data: a standard test image.

[0380] Audio data: A short audio clip.

[0381] Use the ResNet model to preprocess the image and extract the feature vector x=[x1,x2,…,x m ].

[0382] Set the extracted feature vectors x1=0.6, x2=0.4:

[0383] I(x)=γ1·sin(δ1·0.6)+γ2·sin(δ2·0.4)

[0384] Set γ1=0.8, δ1=0.05, γ2=0.5, δ2=0.05:

[0385] I(x)=0.8·sin(0.03)+0.5·sin(0.02)≈0.8·0.03+0.5·0.02=0.034

[0386] Audio data feature extraction example:

[0387] Use LSTM network to process audio data and extract feature vector x=[x1,x2,…,x p ].

[0388] Set the feature vector x1 = 0.5, x2 = 0.3:

[0389]

[0390] Set ∈1=0.9,η1=0.1,∈2=0.7,η2=0.1:

[0391] A(x)=0.9·e -0.025 +0.7 e -0.009 ≈0.9 0.975 + 0.7 0.991 = 0.8775 + 0.6937 = 1.5712

[0392] Through this multimodal data processing approach, the company is able to effectively extract key features from different types of data and convert them into information-rich vector representations, which can be used in subsequent similarity calculations and data deduplication processes.

[0393] Then, the embodiment standardizes and aligns the features of different modal data, and realizes feature fusion through autoencoders or variational autoencoders (VAE). First, the features extracted from different data sources are standardized to ensure that they are compared and processed on the same scale, and the features are scaled to a uniform range.

[0394] Then, an autoencoder or variational autoencoder (VAE) is used to map the processed feature vectors to a common embedding space, which is accomplished by learning to reconstruct a low-dimensional representation of the input data, while introducing probability distribution parameters in the variational autoencoder to enhance the generation ability and robustness of the model. The feature fusion function is defined as:

[0395]

[0396] where x is the input feature vector, λ i is a fusion parameter used to adjust the weights of different features in the fusion process. i The value range of can be set to 0.01, 0.1 to balance the contribution of each feature.

[0397] The data to be processed is set as follows:

[0398] Text data feature vector: [0.8, 0.6];

[0399] Image data feature vector: [0.4, 0.5, 0.7];

[0400] Audio data feature vector: [0.6, 0.5].

[0401] First, all feature vectors are normalized and resized to match the maximum length, which is three dimensions:

[0402] text: [0.8, 0.6, 0];

[0403] image: [0.4, 0.5, 0.7];

[0404] Audio: [0.6, 0.5, 0].

[0405] Set the fusion parameter λ = [0.05, 0.05, 0.05] and apply the feature fusion function:

[0406] E(x)=log(1+0.05·(0.8 2 +0.6 2 +0 2 )+0.05·(0.4 2 +0.5 2 +0.7 2 )+0.05

[0407] (0.6 2 +0.5 2 +0 2 ))

[0408] E(x)=log(1+0.05·(0.64+0.36+0)+0.05·(0.16+0.25+0.49)+0.05

[0409] (0.36+0.25+0))

[0410] E(x)=log(1+0.05·1.25+0.05·0.9+0.05·0.61)=log(1+0.0625+0.045+0.0305)

[0411] =log(1.138)

[0412] E(x)=0.1296

[0413] With this approach, the company is able to effectively fuse data features from different modalities to create a unified, informative feature representation.

[0414] Embodiment Next, a multimodal similarity calculation model is developed, which combines the features of different modalities and uses deep learning technology to learn and quantify the similarity between the features of different modalities. A deep learning model Siamese network is used to process two inputs by sharing weights and output the similarity of the two inputs.

[0415] For text, image, and audio data, neural network models are used for feature extraction respectively:

[0416] Text data: Use BERT’s pre-trained model to extract high-dimensional feature vectors;

[0417] Image data: extract features through ResNet’s CNN model;

[0418] Audio data: The LSTM RNN structure is used to extract audio feature vectors.

[0419] The similarity calculation formula is:

[0420]

[0421] Among them, α, β, γ are weight parameters that control the contribution of different modal data in the comprehensive similarity calculation. The parameter range can be set to α, β, γ ∈ [0.1, 1] to adjust the relative importance of each modality.

[0422] The specific sample data to be processed is:

[0423] Text data: two texts "Helloworld" and "Hellothere";

[0424] Image data: two similar pictures;

[0425] Audio data: Two voice recordings.

[0426] Feature extraction and similarity calculation:

[0427] Text feature extraction: Use the BERT model to obtain the vectors T(x) and T(y) of two texts;

[0428] Image feature extraction: Use the ResNet model to obtain the vectors I(x) and I(y) of the two images;

[0429] Audio feature extraction: Use the LSTM network to obtain two audio vectors A(x) and A(y).

[0430] Set α=0.3, β=0.4, γ=0.3:

[0431] S(x,y)=0.3·(T(x)·T(y))+0.4·(I(x)·I(y))+0.3·(A(x)·A(y))

[0432] Text vector product: T(x)·T(y)=0.95 (high similarity);

[0433] Image vector product: I(x)·I(y)=0.88 (high similarity);

[0434] Audio vector product: A(x)·A(y)=0.80 (medium similarity).

[0435] Comprehensive similarity:

[0436] S(x,y)=0.3·0.95+0.4·0.88+0.3·0.80=0.885

[0437] Through this method, the company calculates the similarity between data of different modalities, effectively deduplicates data and optimizes storage.

[0438] Embodiment 6:

[0439] In a large-scale data storage deduplication optimization project based on a machine learning model, a heuristic optimization algorithm is used to optimize the similarity calculation process to improve the accuracy and efficiency of data deduplication, and a genetic algorithm is used to optimize the weight parameters in the similarity calculation, and the parameter set is continuously optimized through selection, crossover and mutation operations.

[0440] Set the initial weight parameters α, β, γ in the similarity function. These parameters determine the contribution of different modal features in the comprehensive similarity calculation. The initial parameter set can be randomly generated or set based on prior knowledge.

[0441] The fitness function is used to evaluate the effect of parameter combinations:

[0442]

[0443] Where S(xi ,y i ) is the similarity calculation result, label i is the true label, indicating whether the two data are duplicate data.

[0444] Suppose there is a data set, which includes several pairs of samples and their similarity labels:

[0445] Data pair (x i ,y i ) and the corresponding label label i :

[0446] ((x1,y1),label1)=((Helloworld,Hellothere),0.9)

[0447] ((x2,y2),label2)=((Goodmorning,Goodnight),0.2)

[0448] Initial parameter set: (α, β, γ) = (0.3, 0.4, 0.3)

[0449] Text vector: T(x) = [0.8, 0.6], T(y) = [0.7, 0.5]

[0450] Image vector: I(x) = [0.6, 0.5, 0.7], I(y) = [0.4, 0.5, 0.6]

[0451] Audio vector: A(x)=[0.5,0.6], A(y)=[0.4,0.7]

[0452] Similarity calculation:

[0453]

[0454] S(x1,y1)=0.3·(0.8·0.7+0.6·0.5)+0.4·(0.6·0.4+0.5·0.5+0.7·0.6)+0.3

[0455] (0.5 0.4 + 0.6 0.7)

[0456] S(x1,y1)=0.3·(0.56+0.30)+0.4·(0.24+0.25+0.42)+0.3·(0.20+0.42)

[0457] =0.3·0.86+0.4·0.91+0.3·0.62

[0458] S(x1,y1)=0.258+0.364+0.186=0.808

[0459] Fitness function calculation:

[0460]

[0461] Selection, crossover and mutation generate new parameter sets, evaluate the fitness of the new parameter sets, and iterate until the fitness is no longer significantly improved. Through multiple iterations, the genetic algorithm determines the optimal parameter combination, (α, β, γ) = (0.4, 0.35, 0.25).

[0462] The embodiment further represents the data as a graph structure, and performs deduplication processing using graph matching and clustering algorithms.

[0463] In the graph structure G = (V, E, w), V is a node set representing data items, E is an edge set representing the relationship between data items, and w is the weight of the edge representing the similarity between data items.

[0464] The edge weight is calculated by the following function:

[0465]

[0466] where x i and x j is the feature vector of the data item, and λ is the adjustment parameter to control the influence of feature differences. The value range of parameter λ can be set to (1,2).

[0467] Data item feature vector: x1 = [0.8, 0.6], x2 = [0.7, 0.5], x3 = [0.3, 0.4]

[0468] Compute edge weights between data items:

[0469] λ=1.5

[0470] w 12 =exp(-(|0.8-0.7| 1.5 +|0.6-0.5| 1.5 ))

[0471] w 13 =exp(-(|0.8-0.3| 1.5 +|0.6-0.4| 1.5 ))

[0472] w 23 =exp(-(|0.7-0.3| 1.5 +|0.5-0.4| 1.5 ))

[0473] Specific calculation:

[0474] w12 =exp(-(0.1 1.5 +0.1 1.5 ))≈exp(-0.063)≈0.939

[0475] w 13 =exp(-(0.5 1.5 +0.2 1.5 ))≈exp(-0.25-0.089)≈exp(-0.339)≈0.712

[0476] w 23 =exp(-(0.4 1.5 +0.1 1.5 ))≈exp(-0.126-0.063)≈exp(-0.189)≈0.828

[0477] Graph matching and clustering algorithms are used to remove duplicates:

[0478] Construct graph G = (V, e, w):

[0479] Node set V = {x1,x2,x3}

[0480] Edge set E = {(x1,x2),(x1,x3),(x2,x3)}

[0481] Edge weight w = {w 12 =0.939,w 13 =0.712,w 23 =0.828}

[0482] Use the maximum matching algorithm to identify duplicate data items:

[0483] The maximum matching function is defined as:

[0484]

[0485] Where m ij It is a matching indicator function, indicating whether the edge (i, j) is in the maximum matching.

[0486] Apply the maximum matching algorithm:

[0487] E={(x1,x2),(x1,x3),(x2,x3)}

[0488] The maximum matching is (x1,x2) and (x2,x3)

[0489] Calculate the weighted sum of the maximum matching:

[0490] M(G)=0.939·1+0.828·1=1.767

[0491] With this approach, the company is able to represent data as a graph structure and effectively identify and remove duplicate data items by calculating similarities between data items and applying a maximum matching algorithm.

[0492] The last embodiment uses a spectral clustering algorithm to group graphs to further improve the accuracy and efficiency of deduplication processing. The intrinsic structure of the data is identified through the eigendecomposition of the graph Laplacian matrix, and similar data is grouped, thereby removing duplicate data more effectively.

[0493] In the graph structure G in the previous step, the Laplacian matrix L = DA, where D is the degree matrix and A is the adjacency matrix. The degree matrix D is a diagonal matrix, in which each element represents the degree of the corresponding node. The adjacency matrix A represents the connection relationship and weight between nodes. Perform eigenvalue decomposition on the graph Laplacian matrix L to obtain the eigenvalue λ j and the eigenvector φ j .

[0494] The clustering function is defined as:

[0495]

[0496] where λ j is the eigenvalue, φ j is a feature vector used to group the data.

[0497] Set the data item feature vector: x1 = [0.8, 0.6], x2 = [0.7, 0.5], x3 = [0.3, 0.4]

[0498] Edge weight calculation results:

[0499] w 12 =0.939

[0500] w 13 =0.712

[0501] w 23 =0.828

[0502] Construct the adjacency matrix and degree matrix:

[0503] Adjacency matrix A:

[0504]

[0505] Degree matrix D:

[0506]

[0507] Compute the Laplacian matrix:

[0508]

[0509] Perform eigenvalue decomposition on L and obtain the eigenvalue λ j and the eigenvector φ j :

[0510] Eigenvalues ​​λ1=0,λ2=1.0,λ3=3.0

[0511] The eigenvectors φ1, φ2, φ3 correspond to each eigenvalue.

[0512] Use the first two smallest eigenvalues ​​and the corresponding eigenvectors for clustering (set k = 2):

[0513] C(k)=λ1·φ1+λ2·φ2

[0514] Set the feature vector as follows:

[0515]

[0516] Calculate cluster centers:

[0517]

[0518] According to the calculation results, the data is divided into two clusters. Cluster 1 contains Closer data, cluster 2 contains Closer data. With this approach, the company represents data as a graph structure and groups it through a spectral clustering algorithm. This process enables better identification and removal of duplicate data, thereby optimizing data storage and processing.

[0519] The above shows and describes the basic principles, main features and advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A large-scale data storage deduplication optimization method based on a machine learning model, characterized in that The following steps are involved: S1. Build a data index based on hash function, map the data into a fixed-size hash table, and use locality sensitive hashing (LSH) technology to index similar data. S2. Design a partitioning strategy based on data features, assign similar data to the same or similar partitions, control and reduce cross-partition comparison operations, and adjust the partitioning scheme according to data changes; S3. In the process of data deduplication, the rule engine is combined with the multi-level machine learning model. First, the rule engine performs preliminary deduplication screening based on predefined rules, and the screened data is input into the machine learning model for similarity calculation and deduplication processing; S3.1, wherein the multi-level machine learning model includes a shallow model and a deep model, and the prediction results of multiple models are combined by an integrated learning method to control and improve the accuracy of deduplication; S3.2, then design a multimodal similarity calculation method, combine the similarities of multiple data types such as text, image, and audio, use a deep learning model to learn the feature representation of the data, and calculate the multimodal similarity; S3.

3. A heuristic optimization algorithm is used to optimize the parameters of similarity calculation. By designing a deduplication method based on a graph structure, the data is represented as a graph structure, and deduplication is performed through graph matching and clustering.

2. The large-scale data storage deduplication optimization method based on the machine learning model according to claim 1 is characterized in that The steps of indexing similar data include: S1. First, a hash function with adaptive parameters is used: where x=(x1,x2,…,x n ) is the data vector, a i is the adaptive coefficient, γ i is an exponent that changes dynamically according to the data, and p is a large prime number used to control the uniformity of data mapping; S2. Then calculate the size N of the hash table using the expression: Where D is the size of the data set, S is the expected query speed, and k is the adjustment factor, which reflects the logarithmic relationship between data size and query speed; S3. Introduce local sensitive hashing (LSH) technology and design the LSH function family: where h j (x) is the sub-hash function, c j is the weight, δ j is the data sensitivity parameter, and q is a large prime number that controls the high probability aggregation of similar data.

3. The large-scale data storage deduplication optimization method based on machine learning model according to claim 1 is characterized in that The partition strategy construction process based on data characteristics: First, machine learning models including autoencoders or deep neural networks are used to extract features from stored data, and the extracted high-dimensional feature vectors are converted into data fingerprints. The feature vectors are simplified through hash functions to generate fingerprints for subsequent deduplication comparison. Then, an improved clustering algorithm including spectral clustering or Gaussian mixture models is used to process data fingerprints, classify data items with high similarity into the same cluster partition, perform deduplication within the cluster partition, and control and reduce cross-partition comparison operations; Finally, a dynamic partition adjustment mechanism is integrated to adjust the partition structure in real time according to the data inflow speed and query frequency. When the partition data grows too fast, the partition splitting is controlled.

4. The large-scale data storage deduplication optimization method based on machine learning model according to claim 3 is characterized in that The means for converting the high-dimensional feature vector into a data fingerprint process include: S1. First, use an autoencoder or deep neural network to extract features from the input data, using nonlinear activation functions and advanced mathematical transformations through the formula: To learn the latent space representation of the data, where x is the input vector and w i ,u ij , and α j are the learning weights and nonlinear degree parameters of the model respectively; S2. Next, the extracted high-dimensional feature vector is passed through a hash function: Converted to data fingerprint, where x j is a component of the eigenvector, c j is the selected coefficient, and p is a large prime number used for modular operations to generate data fingerprints with high information preservation; S3. Finally, use the fingerprint to perform duplicate comparison by calculating the weighted Hamming distance between two fingerprints: Where x and y are two data fingerprints, γ k is the weight coefficient based on the position, l is the length of the fingerprint, and the d(x,y) function is used to control the identification and removal of duplicate data.

5. The large-scale data storage deduplication optimization method based on machine learning model according to claim 3 is characterized in that The dynamic partition adjustment mechanism construction includes the following steps: First, use the autoencoder or deep learning network to extract features from the stored data, using the formula: Generate data fingerprint, where x represents the input data vector, β i The nonlinear adjustment parameter representing the feature; Then monitor the number and changes of data fingerprints in each partition in real time through the change rate function: Where n is the number of data fingerprints in the partition and t is the time to evaluate the growth and diversity of the data; Then, based on the similarity of data fingerprints and real-time monitoring information, a prediction model is used: Where y represents the deduplication efficiency, X represents the data growth rate and query frequency characteristics, and predicts the partition performance and decides whether to adjust the partition; Finally, when the data in a partition grows too fast, resulting in a decrease in deduplication efficiency, cluster analysis is performed: where x j is the data fingerprint, ξ is the cluster center, k is the number of clusters, which triggers partition splitting, controls similar data to remain in similar partitions, and optimizes subsequent deduplication operations.

6. The large-scale data storage deduplication optimization method based on machine learning model according to claim 1 is characterized in that The multi-level machine learning model construction includes: S1. First, preprocess the input data, extract features using autoencoders or deep neural networks, and generate high-dimensional feature vectors through the function: Converted to data fingerprint, where x is the input data vector, w i , α, β i is the learning parameter; S2. Then train multiple shallow models and deep models. The shallow models focus on extracting basic features by optimizing the function: Optimization, where θ is the model parameter, γ and φ j is the regularization coefficient, and the deep model is calculated by the formula: Extract complex patterns and nonlinear relationships, where z is the input feature and w k and b k are weights and biases, σ is the activation function; S3, then through the integrated learning methods including Bagging, Boosting and Stacking, the prediction results of the shallow model and the deep model are integrated, using the integrated formula: Combine the results, where h i (x) is the prediction result of the i-th model, α j is the weight factor and N is the number of models.

7. The large-scale data storage deduplication optimization method based on machine learning model according to claim 1 is characterized in that The multimodal similarity calculation method comprises the following steps: S1. First, preprocess the input data and extract features. For text data, use the pre-trained natural language processing model to extract features and obtain a high-dimensional text vector representation: Where x is the input text data, α i and β i are model parameters; S2. For image data, use convolutional neural networks (CNNs), including ResNet and VGGNet, to extract features and convert images into feature vector representations: Where x is the input image data, γ j and δ j are model parameters; For audio data, use convolutional neural network or recurrent neural network RNN ​​including LSTM to process and extract audio feature vectors: Where x is the input audio data, ∈ k and η k are model parameters; S3. Then, the data of different modalities are normalized and aligned to control the feature vectors of different data types to have the same dimension, and the autoencoder or variational autoencoder VAE is used to map the features of each modality to a common embedding space through the function: To achieve feature fusion, where x is the input feature vector, λ i is the fusion parameter; S4. Finally, a multimodal similarity calculation model is designed. By combining different modal features, the similarity between data is calculated. The deep learning model including Siamese network is used to learn the similarity between different modal features, and the multimodal similarity function is defined: Among them, α, β and γ are weight parameters of different modes, which control the comprehensive similarity calculation of multimodal data.

8. The large-scale data storage deduplication optimization method based on machine learning model according to claim 1 is characterized in that The heuristic optimization algorithm optimizes the similarity calculation process and comprises the following steps: S1. First, heuristic optimization algorithms including genetic algorithms are used to optimize similarity calculation parameters, set the initial parameter set, and use the fitness function: Evaluate the effect of parameter combination, where α, β, γ are weight parameters in the similarity function, S(x i ,y i ) is the similarity calculation result, label i For the true label, the parameter set is iteratively updated to obtain the optimal parameter; S2. Then represent the data as a graph structure: G=(V,E,w) Where V is the node set, E is the edge set, and w is the weight of the edge, which represents the similarity between data items; Through the function: Calculate the edge weight, where x i and x j is the feature vector of the data item, and λ is the adjustment parameter; S3. Then use the graph matching and clustering algorithm to perform deduplication processing, identify the predicted duplicate data items through the maximum matching algorithm, and define the maximum matching function: Where m ij is the matching indicator function; S4. Finally, the spectral clustering algorithm is used to group the graphs, and the eigendecomposition is performed through the graph Laplacian matrix L = DA, where D is the degree matrix and A is the adjacency matrix, through the function: Group the data, where λ j is the eigenvalue, φ j is a feature vector to further remove duplicate data.

Citation Information

Patent Citations

  • Video duplicate checking method and device

    CN111046227A

  • MES-oriented mass data redundancy elimination method and system

    CN112162977A

  • Multi-modal power data management method and system based on deep metric learning

    CN117609278A

  • Image Construction With Multiple Clustering Realizations

    US20170084060A1

Cited By

  • Intelligent prediction method for privacy information storage and transmission security situation

    CN119377615A

  • Cloud backup platform data deduplication method based on distributed storage

    CN119938406A

  • Data deduplication method for cloud backup platform based on distributed storage

    CN119938406B

  • Deduplication optimization method and system for accumulated data

    CN120631259A