A network data deduplication method and system in a cloud environment based on an autoregressive model

Abstract

The application discloses a network data deduplication method and system in a cloud environment based on an autoregressive model, and relates to the technical field of big data processing. The method comprises the following steps: step 1, obtaining network data to be stored containing similar data in the cloud environment; step 2, establishing a cloud environment elastic space model, determining the spatial autocorrelation measurement value of the network data and performing data denoising processing; step 3, obtaining the multi-dimensional scale attribute features of the data; step 4, performing clustering processing on the network data through the multi-dimensional scale attribute features of the data; step 5, establishing a network redundant data prediction model; and step 6, inputting the denoised network data into the network redundant data prediction model and performing redundant data elimination processing. The application can not only suppress noise data in the network data, but also realize clustering of the network data in the cloud environment, improve the data deduplication efficiency, and save a large amount of storage resources and management resources of the cloud server.

Description

Technical Field

[0001] This invention relates to the field of big data processing technology, specifically to a method and system for deduplicating network data in a cloud environment based on an autoregressive model. Background Technology

[0002] With the development of cloud computing technology, network data is becoming increasingly abundant, and cloud services are becoming more and more widespread. The increase in network data volume requires high bandwidth during data transmission. A large amount of duplicate data exists within the cloud environment. The existence of this duplicate data greatly affects the efficiency of data storage and access, and causes a significant waste of resources. Therefore, implementing data deduplication in the cloud environment is crucial for reducing the storage costs of cloud service providers.

[0003] To address this issue, reference file 1 (Gao Wenjing, Xian Hequn, Cheng Runhui. Chinese Journal of Computers, 2021, 44(11):2203-2215) proposes a cloud data deduplication method based on double-layer encryption and key sharing. This method first divides the popularity of network data, and then implements double-layer encryption protection for non-popular data with high privacy based on the division results. At the same time, it uses a threshold secret sharing mechanism to store the encryption key in the key service manager to realize key sharing among network users. For data with low privacy, it uses convergent encryption to perform encryption processing. After the data is encrypted, the keys of the same data and similar data will be consistent. Therefore, the keys are classified and filtered to remove the data with similar keys, thus completing the data deduplication. Comparative document 2 (Tang Xin, Zhou Linna, Shan Weijie, et al. Journal of Communications, 2020, 41(6):98-111.) proposes a cloud data security deduplication method based on threshold re-encryption to resist side-channel attacks. This method designs a threshold encryption mechanism to convert the user-end ciphertext segmentation into key segmentation; and maps the secondary encryption of the data to the cloud for execution, reducing user overhead; then decrypts the plaintext from the encrypted ciphertext, matches the ciphertext with the plaintext, finds the redundant data, and completes the data deduplication. Comparative document 3 (Tang Xin, Zhou Linna. Computer Applications, 2020, 40(4):1085-1090.) proposes a cloud data security deduplication method based on response fuzzification to resist additional block attacks. This method first determines the minimum number of redundant blocks in the response data to realize the response fuzzification of the data; then identifies whether the data is redundant based on the minimum communication overhead; finally, based on the identification results, performs deduplication on the redundant data in the communication network data, effectively reducing the amount of network data.

[0004] The aforementioned methods are computationally intensive, and because they fail to suppress noise in the network data before deduplication, their deduplication performance is poor. Furthermore, they cannot cluster similar data blocks during deduplication, making subsequent data analysis difficult and resulting in high storage space consumption. Therefore, addressing these problems in network data deduplication to improve efficiency and overall storage system utilization is a pressing issue for those skilled in the art. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the above-mentioned technical problems and provide a method and system for deduplication of network data in a cloud environment based on an autoregressive model. This method can suppress noisy data in the network data and achieve clustering of network data in the cloud environment, thereby improving the efficiency of data deduplication and saving a lot of storage and management resources of cloud servers.

[0006] In a first aspect, the present invention provides a method for deduplicating network data in a cloud environment based on an autoregressive model, the method comprising the following steps:

[0007] Step 1: Obtain network data to be stored in the cloud environment that contains similar data;

[0008] Step 2: Establish a cloud environment elastic spatial model, determine the spatial autocorrelation metric of network data, and perform data denoising.

[0009] Step 3: Analyze the network data attribute characteristics in the cloud environment based on the denoising results, and obtain the multi-dimensional scale attribute characteristics of the data;

[0010] Step 4: Next, cluster the network data using the multi-dimensional scale attribute features of the data;

[0011] Step 5: Establish a network redundancy data prediction model based on the clustering results;

[0012] Step 6: Take the denoised network data as input, and use the network redundancy data prediction model to predict redundant data. Finally, remove the predicted redundant data to achieve deduplication of network data.

[0013] Furthermore, in step 2, the specific method for establishing the cloud environment elastic spatial model is as follows:

[0014] When establishing the elastic spatial model, the size of the cloud environment spatial model is set to [size to be specified]. The overall spatial correlation is η, which is used to establish the elastic spatial model of the cloud environment. The result is shown in the following formula:

[0015]

[0016] In the formula, Δ is the variable coefficient of the data in the cloud environment, ι is the fluctuation parameter, and r is a constant; after the cloud environment elastic space model is established, the network data stored in the cloud environment is mapped to the cloud environment elastic space model.

[0017] Based on the established cloud environment elastic spatial model, the spatial correlation between data in the model and the mean of the overall spatial correlation are calculated using radial basis functions. The results are shown in the following formula:

[0018]

[0019] In the formula, dis(i,j) is the distance between network data, ζ is the width vector of the radial basis function, Y(i) is the mean of the overall spatial correlation, B(i) is the set of neighboring data, |B(i)| is the total amount of data in the neighboring data set, and y(i,j,ζ) is the spatial correlation value of the data in the spatial model.

[0020] Furthermore, in step 2, the specific method for data denoising is as follows:

[0021] Based on the cloud environment elasticity spatial model established above, the upper and lower limits of the model are set as κ and λ, respectively. For data in the model that is greater than the upper limit, it is regarded as isolated data in the network data set, and noise is removed by the weighted denoising method of nearest neighbor data. The process is shown in the following formula:

[0022]

[0023] In the formula, x(j,t) represents the original measured value of the data in the model at time t. The result represents the noise removal process, and y(i,j,ζ) represents the weight of the noise data.

[0024] For data in the model that is below the lower limit, a weighted average denoising method is used, combining noise weights and measurements from adjacent data points, as shown in the following formula:

[0025]

[0026] In the formula, The result of denoising data below the model's lower limit.

[0027] Furthermore, the specific method for step 3 is as follows:

[0028] Let 'a' be the type of attribute feature of the network data, and 'k' be the autocorrelation statistical distribution sequence of the data. The statistical distribution of the network data sequence is expressed in the following form:

[0029]

[0030] In the formula, m represents the total number of data autocorrelation statistical distribution sequences. The statistical sequence distribution value of network data is given; based on the statistical sequence distribution value of cloud environment network data obtained above, the detection statistic of cloud environment network data is set as V(b1,b2,...,b...). n The network data is divided into blocks, and the result is shown in the following formula:

[0031]

[0032] In the formula, β is the block partitioning coefficient of the network data, d is the block density between data, q is the number of blocks of the network data, and C q This represents the result of dividing the network data into blocks, where χ is a constant.

[0033] Based on the data segmentation results, a multiple feature decomposition method is used to linearly superimpose the segmented data to obtain the multidimensional scale features of network data in the cloud environment. The process is shown in the following formula:

[0034]

[0035] In the formula, T c MT is used for fuzzy sampling prefabrication of data. c The total sampling is pre-defined, f represents the frequency characteristic of the data, and X(f) represents the adaptive weighting coefficients of the data. To obtain the multidimensional scale features of the data, l is the network data decomposition coefficient, sinc 2 The sine scaling factor is used for the data.

[0036] Because the maximum and minimum values ​​of the multidimensional scale features of the acquired data may exhibit outliers that exceed their range, the feature coefficients for different dimensions are ultimately set to x based on the extracted multidimensional scale features of the network data. t,I The z-score algorithm is used to standardize the multidimensional features of the data. The standardization process is shown in the following formula:

[0037]

[0038] In the formula, n is a constant vector, x tn,I For the actual multidimensional scale characteristics of network data, x tn To standardize the feature results, σ n The standard deviation of the multidimensional scale features of network data.

[0039] Furthermore, the specific method for step 4 is as follows:

[0040] Based on the multi-dimensional scale features of the network data obtained in step 3, multi-dimensional scale matching is performed on the network data in the cloud environment to obtain the correlation features of the data, which are described as x j =(x 1j ,x2j ,...,x mj Let m be the total amount of network data, and q(x0) be the distribution of network data state characteristics in the cloud environment. The process of mining association rules for network data in the cloud environment is as follows:

[0041]

[0042] In the formula, Q ij (k) represents the network data association rules mined, δ i (k) represents the data sampling result, δ j (k) represents the scale-adaptive sampling result, ε ij (k) is the data correlation coefficient, M i (k) represents the set of multidimensional scale features of the data;

[0043] Based on the network data association rules obtained above, a fuzzy association rule feature quantity φ(t) is established for network data in the cloud environment, and a feature distribution set S of the data is established. i And set the clustering vector basis of the data as x n This is used to obtain the cloud environment network data clustering center and realize the clustering of cloud environment network data, as shown in the following formula:

[0044]

[0045] In the formula, e(G ec G is the cluster center established. ec For the fusion clustering matrix of network data, max(G) ec ) represents the maximum value of the matrix, and γ represents the clustering coefficient.

[0046] Secondly, the present invention provides a network data deduplication system in a cloud environment based on an autoregressive model, the system comprising:

[0047] The network data acquisition module is used to acquire network data to be stored in the cloud environment that contains similar data;

[0048] The data preprocessing module is used to establish a cloud environment elastic spatial model, determine the spatial autocorrelation metric of network data, and perform data denoising.

[0049] The feature extraction module is used to analyze the attribute features of network data in the cloud environment based on the denoising results and obtain the multi-dimensional scale attribute features of the data.

[0050] The data clustering module is used to cluster network data based on the multi-dimensional scale attribute features of the data;

[0051] The model building module is used to build a network redundancy data prediction model based on the clustering results.

[0052] The target generation module takes the denoised network data as input, outputs the network redundancy data prediction model to complete the redundancy data prediction, and finally removes the predicted redundancy data to achieve deduplication of network data.

[0053] Compared with the prior art, the present invention has the following beneficial effects:

[0054] This invention discloses a method for deduplicating network data in a cloud environment based on an autoregressive model. First, it analyzes the attribute characteristics of network data in the cloud environment in detail based on the denoising results. Then, it clusters the network data using these attribute characteristics to establish a network redundancy prediction model, and predicts redundant data based on the model output. Finally, it removes the predicted redundant data, thus achieving deduplication. Before deduplication, this method not only performs timely denoising but also clusters and fuses the data based on the processing results. This is beneficial for clustering massive amounts of data and is suitable for analyzing network data in cloud environments. After deduplication, the data exhibits high space compression, saving significant storage and management resources on cloud servers.

[0055] The network data deduplication system based on an autoregressive model in the cloud environment of this invention can improve deduplication efficiency and increase the overall utilization of the storage system. Attached Figure Description

[0056] Figure 1 This is a flowchart of a network data deduplication method in a cloud environment based on an autoregressive model, according to the present invention.

[0057] Figure 2 This is a structural block diagram of a network data deduplication system in a cloud environment based on an autoregressive model, according to the present invention.

[0058] Figure 3 This is a graph showing the test results of the space compression ratio after deduplication using different methods according to an embodiment of the present invention;

[0059] Figure 4 These are actual data deduplication results of different methods tested in embodiments of the present invention. Figure (a) shows the original cloud environment data distribution, Figure (b) shows the deduplication result of the method of the present invention, Figure (c) shows the deduplication result of the method in comparison file 1, and Figure (d) shows the deduplication result of the method in comparison file 3. Detailed Implementation

[0060] The present invention will be further described below with reference to the accompanying drawings and embodiments. It should be noted that the specific embodiments of the present invention are only for the purpose of more clearly describing the technical solution and should not be construed as a limitation on the scope of protection of the present invention.

[0061] Because cloud network data contains a significant amount of noise, failure to remove this noise promptly will directly impact the deduplication effect. Therefore, before deduplication, it is necessary to establish a cloud environment elastic spatial model to determine the spatial autocorrelation metric of the network data and complete the denoising process. Then, based on the denoising results, a detailed analysis of the attribute characteristics of the network data in the cloud environment is conducted.

[0062] Firstly, this invention provides a method for deduplicating network data in a cloud environment based on an autoregressive model, such as... Figure 1 As shown, the method includes the following steps:

[0063] Step 1: Obtain network data to be stored in the cloud environment that contains similar data;

[0064] Step 2: Preprocess the data, namely, establish a cloud environment elastic spatial model, determine the spatial autocorrelation metric of the network data, and perform data denoising.

[0065] Step 3: Analyze the network data attribute characteristics in the cloud environment based on the denoising results, and obtain the multi-dimensional scale attribute characteristics of the data;

[0066] Step 4: Next, cluster the network data using the multi-dimensional scale attribute features of the data;

[0067] Step 5: Establish a network redundancy data prediction model based on the clustering results;

[0068] Step 6: Take the denoised network data as input, and use the network redundancy data prediction model to predict redundant data. Finally, remove the predicted redundant data to achieve deduplication of network data.

[0069] In step 2, since network data in the cloud environment is subject to significant fluctuations due to environmental noise, the specific method for establishing a cloud environment elastic spatial model is as follows to eliminate uncertainties during data deduplication:

[0070] When establishing the elastic spatial model, the size of the cloud environment spatial model is set to [size to be specified]. The overall spatial correlation is η, which is used to establish the elastic spatial model of the cloud environment. The result is shown in the following formula:

[0071]

[0072] In the formula, Δ is the variable coefficient of the data in the cloud environment, ι is the fluctuation parameter, and r is a constant; after the cloud environment elastic space model is established, the network data stored in the cloud environment is mapped to the cloud environment elastic space model.

[0073] Based on the established cloud environment elastic spatial model, the spatial correlation between data in the model and the mean of the overall spatial correlation are calculated using radial basis functions. The results are shown in the following formula:

[0074]

[0075] In the formula, dis(i,j) is the distance between network data, ζ is the width vector of the radial basis function, Y(i) is the mean of the overall spatial correlation, B(i) is the set of neighboring data, |B(i)| is the total amount of data in the neighboring data set, and y(i,j,ζ) is the spatial correlation value of the data in the spatial model.

[0076] In step 2, the specific method for denoising the data is as follows:

[0077] Based on the cloud environment elasticity spatial model established above, the upper and lower limits of the model are set as κ and λ, respectively. For data in the model that is greater than the upper limit, it is regarded as isolated data in the network data set, and noise is removed by the weighted denoising method of nearest neighbor data. The process is shown in the following formula:

[0078]

[0079] In the formula, x(j,t) represents the original measured value of the data in the model at time t. The result represents the noise removal process, and y(i,j,ζ) represents the weight of the noise data.

[0080] For data in the model that is below the lower limit, weighted denoising can be achieved directly using noise weights and measurements from adjacent data, as shown in the following formula:

[0081]

[0082] In the formula, The result of denoising data below the model's lower limit.

[0083] Since network data in cloud environments is typically mixed data with varying attributes, a storage structure analysis of the network data in the cloud environment is needed before deduplication to obtain multi-dimensional scale attribute characteristics, laying the foundation for data clustering during deduplication. Therefore, the specific method for step 3 is as follows:

[0084] Let 'a' be the type of attribute feature of the network data, and 'k' be the autocorrelation statistical distribution sequence of the data. The statistical distribution of the network data sequence is expressed in the following form:

[0085]

[0086] In the formula, m represents the total number of data autocorrelation statistical distribution sequences. The statistical sequence distribution value of network data is given; based on the statistical sequence distribution value of cloud environment network data obtained above, the detection statistic of cloud environment network data is set as V(b1,b2,...,b...). n The network data is divided into blocks, and the result is shown in the following formula:

[0087]

[0088] In the formula, β is the block partitioning coefficient of the network data, d is the block density between data, q is the number of blocks of the network data, and C q This represents the result of dividing the network data into blocks, where χ is a constant.

[0089] Based on the data segmentation results, a multiple feature decomposition method is used to linearly superimpose the segmented data to obtain the multidimensional scale features of network data in the cloud environment. The process is shown in the following formula:

[0090]

[0091] In the formula, T c MT is used for fuzzy sampling prefabrication of data. c The total sampling is pre-defined, f represents the frequency characteristic of the data, and X(f) represents the adaptive weighting coefficients of the data. To obtain the multidimensional scale features of the data, l is the network data decomposition coefficient, sinc 2 The sine scaling factor is used for the data.

[0092] Because the maximum and minimum values ​​of the multidimensional scale features of the acquired data may exhibit outliers that exceed their range, the feature coefficients for different dimensions are ultimately set to x based on the extracted multidimensional scale features of the network data. t,I The z-score algorithm is used to standardize the multidimensional features of the data. The standardization process is shown in the following formula:

[0093]

[0094] In the formula, n is a constant vector, x tn,I For the actual multidimensional scale characteristics of network data, x tn To standardize the feature results, σ n The standard deviation of the multidimensional scale features of network data.

[0095] Based on the multi-dimensional scale attribute characteristics of the data obtained above, clustering processing is performed on the network data in the cloud environment. Based on the clustering results, a network redundancy data prediction model is established in combination with an autoregressive model. Based on the model prediction results, network redundancy data in the cloud environment is obtained and removed to achieve accurate deduplication of network data.

[0096] The specific method for network data clustering in step 4 is as follows:

[0097] Based on the multi-dimensional scale features of the network data obtained in step 3, multi-dimensional scale matching is performed on the network data in the cloud environment to obtain the correlation features of the data, which are described as x j =(x 1j ,x 2j ,...,x mj Let m be the total amount of network data, and q(x0) be the distribution of network data state characteristics in the cloud environment. The process of mining association rules for network data in the cloud environment is as follows:

[0098]

[0099] In the formula, Q ij (k) represents the network data association rules mined, δ i (k) represents the data sampling result, δ j (k) represents the scale-adaptive sampling result, ε ij (k) is the data correlation coefficient, M i (k) represents the set of multidimensional scale features of the data;

[0100] Based on the network data association rules obtained above, a fuzzy association rule feature quantity φ(t) is established for network data in the cloud environment, and a feature distribution set S of the data is established. i And set the clustering vector basis of the data as x n This is used to obtain the cloud environment network data clustering center and realize the clustering of cloud environment network data, as shown in the following formula:

[0101]

[0102] In the formula, e(G ec G is the cluster center established. ec For the fusion clustering matrix of network data, max(G) ec ) represents the maximum value of the matrix, and γ represents the clustering coefficient.

[0103] The clustering results of the cloud environment data mentioned above are used as the basis for model building. A redundancy prediction model for cloud environment network data is established based on this model output. Redundant data in the cloud environment network data is identified based on the model output. By removing the redundant data, the deduplication of cloud environment network data is achieved.

[0104] Therefore, the specific method for constructing the network redundancy data prediction model in step 5 is as follows:

[0105] After setting the data clustering, the non-stationary time series of different categories of network data are P={p1,p2...,p...} nThe sequence was decomposed using the Mallat algorithm, and the decomposition results were reconstructed. The result is shown in the following equation:

[0106] P′=H1+H2+...+H j +O j (11)

[0107] In the formula, H1, H2, ..., H j For the single-branch reconstructed sequence at level j after sequence decomposition, O j This is the approximate reconstructed sequence at level j. After the network time series is decomposed and reconstructed, μ-step prediction is performed on the reconstructed data based on the autoregressive principle. The prediction result is shown in the following formula:

[0108]

[0109] In the formula, The prediction results are decomposed into levels for each data sequence. For O j The μ-step prediction result is given by P′(μ), where P′(μ) is the μ-step prediction result of the data sequence. Since the sequence frequency after data sequence decomposition is relatively uniform compared with the original data, it is necessary to perform smoothing processing on the prediction data and establish an autoregressive redundant data prediction model for network data. The model expression is shown in the following formula:

[0110]

[0111] In the formula, v is the order of the network redundancy data prediction model, and θ t For white noise sequences, P t-v For the dependent variable in the model, P is the maximum likelihood estimate of the model. t For redundant data prediction models, the process for obtaining the maximum likelihood estimate of the model is shown in the following formula:

[0112]

[0113] In the formula, The autocorrelation function of the data;

[0114] Based on the network redundancy data prediction model parameters determined above, block prediction is performed on the clustered data, and the prediction results are shown in the following formula:

[0115]

[0116] In the formula, Predicting results for redundant data in different data blocks.

[0117] Step 6: Finally, based on the different categories of clustered data, determine the prediction entry point, complete the prediction of redundant network data through the calculation results of the model, and finally integrate the redundant data and uniformly implement the removal process to achieve accurate deduplication of network data in the cloud environment.

[0118] Secondly, this invention provides a network data deduplication system in a cloud environment based on an autoregressive model, such as... Figure 2 As shown, the system includes:

[0119] The network data acquisition module is used to acquire network data to be stored in the cloud environment that contains similar data;

[0120] The data preprocessing module is used to establish a cloud environment elastic spatial model, determine the spatial autocorrelation metric of network data, and perform data denoising.

[0121] The feature extraction module is used to analyze the attribute features of network data in the cloud environment based on the denoising results and obtain the multi-dimensional scale attribute features of the data.

[0122] The data clustering module is used to cluster network data based on the multi-dimensional scale attribute features of the data;

[0123] The model building module is used to build a network redundancy data prediction model based on the clustering results.

[0124] The target generation module takes the denoised network data as input, outputs the network redundancy data prediction model to complete the redundancy data prediction, and finally removes the predicted redundancy data to achieve deduplication of network data.

[0125] The network data deduplication system based on an autoregressive model in a cloud environment, as described in this invention, can improve deduplication efficiency and increase the overall utilization of the storage system. By setting up a feature extraction module and a data clustering module, it can perform noise reduction processing on the data and perform clustering and fusion based on the processing results. This enables a highly efficient deduplication system for similar data, allowing cloud servers to achieve deduplication of similar data. This will further improve the efficiency of network data deduplication and save a significant amount of storage and management resources for cloud servers.

[0126] To verify the overall effectiveness of the network data deduplication method and system based on an autoregressive model in a cloud environment, this embodiment tests the method of the present invention against existing data deduplication methods.

[0127] Three deduplication methods were tested: a cloud data deduplication method based on an autoregressive model (the method of this invention), a cloud data deduplication method based on two-layer encryption and key sharing (the method in Comparative Document 1), and a cloud data security deduplication method based on response fuzzification to resist additional block attacks (the method in Comparative Document 3). During the testing, a data cloud environment was established using a Matlab computer simulation platform. The cloud environment size was set to 30×30m, the CPU size to be 55.1GHz, and the storage size to be 500GB, with a maximum capacity of 20,000 network data items. After the network cloud environment was established, 2,000 network data items and 50 redundant data items were stored in the cloud environment. The three deduplication methods described above were then used to perform data deduplication processing to verify the effectiveness of different deduplication methods.

[0128] (1) Space compression effect test

[0129] When deduplicating network data, the cloud environment saves storage space after deduplication. This can be evaluated using the space compression ratio, which is the ratio between the amount of redundant data removed and the total amount of data in the cloud environment. A higher space compression ratio indicates better space compression after deduplication. When using the method of this invention, the method in Comparative Document 1, and the method in Comparative Document 3 to perform data deduplication in the cloud environment, the space compression ratios after deduplication by different methods were tested. The test results are as follows: Figure 3 As shown.

[0130] analyze Figure 3 It can be seen that after deduplication of network data using different methods, the spatial compression ratio tested by the method of this invention is higher than that of the other two methods. Specifically, the method in Comparative Document 1 has a large error in classifying and filtering keys, resulting in a lower spatial compression ratio after data deduplication compared to the method of this invention. The method in Comparative Document 3 fails to consider the impact of environmental noise on the data during data response fuzzification, leading to a lower spatial compression ratio after data deduplication. In contrast, the method of this invention not only performs timely noise reduction on the data before deduplication but also performs cluster analysis based on the processing results, thus achieving a higher spatial compression ratio after data deduplication.

[0131] (2) Deduplication effect test

[0132] When using the method of this invention, the method of Comparative Document 1, and the method of Comparative Document 3 to perform deduplication of cloud environment data, the actual deduplication effect of the three methods after deduplication is tested, and the test results are shown in Table 4.

[0133] Table 1. Test results of deduplication effect of different methods

[0134]

[0135]

[0136] Based on the above test results, the three data deduplication methods were applied to network data in a cloud environment to verify their practical application effectiveness. The test results are as follows: Figure 4 As shown.

[0137] exist Figure 4 In this analysis, data with the same shape but different fill content are considered network data with consistent attributes, while data with the same shape and fill content are considered consistent data (i.e., redundant data exists). Figure 2 It can be seen that the method of the present invention can not only effectively remove redundant data in network data when deduplicating data, but also cluster data during data deduplication, so that data with the same attributes are clustered together. In contrast, the other two methods cannot cluster deduplicated data into clusters, and there is still some redundant data after data deduplication, so the deduplication effect is not ideal.

[0138] The above description is a detailed description of the preferred embodiments of the present invention. However, the embodiments are not intended to limit the scope of the patent application of the present invention. All equivalent changes or modifications made under the technical spirit of the present invention should fall within the patent scope covered by the present invention.

Claims

1. A method for network data deduplication in a cloud environment based on an autoregressive model, characterized in that, It comprises the following steps: Step 1, obtaining network data to be stored containing similar data in the cloud environment; Step 2, establishing a cloud environment elastic space model, determining the spatial autocorrelation measure value of the network data and performing data denoising processing; In step 2, the specific method for establishing a cloud environment elastic space model is: In the establishment of the elastic space model, the size of the space model of the cloud environment is set as , and the overall correlation of the space is . Thus, the establishment of the elastic space model of the cloud environment is completed, and the result is as follows: （1） In the formula, is a variable coefficient of data in a cloud environment, is a fluctuation parameter, is a constant; after the cloud environment elasticity space model is established, the network data stored in the cloud environment is mapped into the cloud environment elasticity space model; Based on the established cloud environment elastic space model, the spatial correlation between the data in the model and the spatial overall correlation average are calculated using a radial basis function, and the results are as follows: （2） wherein is a distance between network data, is a width vector of the radial basis function, is a spatial overall correlation average, is a neighboring data set of the data, is a total amount of data in the neighboring data set, is a spatial correlation value of the data in the spatial model; In step 2, the specific method for data denoising processing is: Based on the cloud environment elasticity space model established above, the upper and lower limits of the model are set as follows: , For data in the model that exceeds the upper limit, it is treated as isolated data in the network dataset, and noise is removed using a weighted denoising method based on nearest neighbor data. The process is shown in the following formula: （3） In the formula, is the original measurement value of the data in the time model, is the noise removal result, is the noise data weight; For data in the model lower than the lower limit value, the weighted average denoising of data is realized through the noise weight and the adjacent data measurement value result, and the results are as follows: （4） In the formula, Data denoising results below the lower limit of the model; Step 3, analyzing the network data attribute characteristics in the cloud environment according to the denoising result, and obtaining the multi-dimensional scale attribute characteristics of the data; Step 4, further clustering the network data through the multi-dimensional scale attribute characteristics of the data; Step 5, obtaining the multi-dimensional scale attribute characteristics of the data, implementing clustering processing on the network data in the cloud environment, and based on the clustering result of the data, combining the autoregressive model, establishing a network redundant data prediction model according to the clustering processing result; Step 6, taking the network data after denoising processing as input, determining the prediction entrance according to different categories of clustered data, completing the redundant data prediction of network data through the calculation result of the model, to determine the redundant data in the cloud environment network data, integrating the redundant data, and uniformly implementing the elimination processing, to remove the redundant data in the cloud environment network data. 2.The method of claim 1, wherein, The specific method of step 3 is: The attribute characteristic category of the network data is set as , the autocorrelation statistical distribution sequence of the data is , and the network data statistical sequence distribution expression form is as shown in the following formula: （5） In the formula, is the total number of data autocorrelation statistical distribution sequences, is the network data statistical sequence distribution value; based on the cloud environment network data statistical sequence distribution value obtained above, the detection statistical quantity of the cloud environment network data is set to The network data is subjected to block processing, and the block result is as follows: （6） wherein is a block coefficient of network data, is a block density between data, is a block quantity of network data, is a network data block result, is a constant; According to the data block result, a multiple feature decomposition method is used to implement linear superposition on the block data, so as to obtain the multi-dimensional scale characteristics of the network data in the cloud environment, and the process is as follows: （7） In the formula, is a fuzzy sampling pre-configuration of data, is a total sampling pre-configuration, is a frequency feature of data, is an adaptive weighting coefficient of data, is a multi-dimensional scale feature of acquired data, is a network data decomposition coefficient, is a data sinusoidal scale coefficient; Since the maximum and minimum values of the multi-dimensional scale features of the acquired data may be out of the range of values, the different dimension feature coefficients are set based on the extracted network data multi-dimensional scale features , the z-score algorithm is used to implement standardization processing on the multi-dimensional features, and the standardization processing procedure is shown in the following formula: （8） wherein, is a constant vector, is the actual multidimensional scaling feature of the network data, is the standardized feature result, is the standard deviation of the network data multidimensional scaling feature. 3.The method of claim 1, wherein, The specific method of step 4 is: Based on the network data multi-dimensional scale features acquired in step 3, the network data in the cloud environment is subjected to multi-dimensional scale matching to acquire the correlation features of the data, which are described as wherein the total amount of network data is denoted as Therefore, the network data state feature distribution in the cloud environment is set as The correlation rule mining of the network data in the cloud environment is completed, and the process is shown in the following formula: （9） In the formula, is a mined network data association rule, is a data sampling result, is a scale adaptive sampling result, is a data association coefficient, is a multi-dimensional scale feature set of data; Based on the network data association rules obtained above, the fuzzy association rule characteristic quantity of network data in the cloud environment is established , the feature distribution set of data is established , and the clustering vector base of data is set as , so as to obtain the clustering center of cloud environment network data, realize the clustering of cloud environment network data, and the process is shown in the following formula: （10） In the formula, is the established cluster center, is the fusion cluster matrix of network data, is the maximum value of the matrix, is the clustering coefficient.

4. An autoregressive model based network data deduplication system in cloud environment corresponding to the autoregressive model based network data deduplication method of any one of claims 1-3, characterized in that, It comprises: A network data acquisition module for acquiring network data to be stored containing similar data in the cloud environment; A data preprocessing module for establishing a cloud environment elastic space model, determining the spatial autocorrelation measure value of the network data and performing data denoising processing; A feature extraction module for analyzing the network data attribute characteristics in the cloud environment according to the denoising result, and obtaining the multi-dimensional scale attribute characteristics of the data; A data clustering module for clustering the network data through the multi-dimensional scale attribute characteristics of the data; A model construction module for establishing a network redundant data prediction model according to the clustering processing result; A target generation module for taking the network data after denoising processing as input, completing the redundant data prediction based on the network redundant data prediction model, and finally performing the elimination processing on the predicted redundant data, to realize the deduplication of network data.

Images