Data storage optimization method and system based on matrix relevance reduction

Abstract

The invention discloses a data storage optimization method and system based on matrix relevance reduction, and the method comprises the steps: employing the significant relevance between matrixes, taking a matrix relevance coefficient as a judgment basis, and determining whether interpolation processing is suitable or not; a matrix which does not need to be interpolated is effectively identified by setting a threshold value of a correlation coefficient; solving by adopting different interpolation methods to obtain the number of matrixes which do not meet constraint conditions after interpolation; and the effect of the matrix interpolation method is evaluated, and the interpolation model is ensured to effectively simplify the complexity of matrix operation, so that the optimal optimization strategy of data storage is realized. By the adoption of the method, the high-frequency collection data storage space can be optimized, a proper data storage optimization mode is provided, and the method has the advantages of being easy to implement and efficient.

Description

Technical Field

[0001] The present invention belongs to the field of large-scale data storage optimization, and in particular relates to a data storage optimization method and system based on matrix correlation reduction. Background Art

[0002] As a basic tool for data representation and calculation, matrices are not only commonly used in applied mathematics such as statistical analysis, but also widely used in physics, computer science and other fields. Usually, signals in computer vision, wireless communications, sonar, phased array radar, radio astronomy and other fields can be presented by matrices, and there is a certain correlation in certain dimensions. Therefore, such situations can be represented mathematically by matrix groups.

[0003] With the popularization of the Internet and the widespread use of intelligent devices, the amount of global data has exploded. It is predicted that in the next few years, global data will increase several times, and this growth has brought tremendous pressure to data storage technology. The hardware cost and operating cost of data storage are also rising. Enterprises need to control costs and improve storage efficiency while ensuring data storage needs. Many studies are based on matrix decomposition technology, which improves storage and computing efficiency by distributing large matrices to multiple nodes for storage and parallel computing. In Europe and the United States, especially in the fields of artificial intelligence and data science, research on matrix operations and storage optimization is relatively mature, and many efficient algorithms and frameworks have emerged. Domestic research institutions and universities are gradually increasing their research on matrix-based data storage optimization, especially the development and application of big data and cloud computing technologies.

[0004] Using conventional algorithm processing will generate exponentially more computational workloads and increase storage requirements, which poses a huge challenge to the implementation cost and power consumption of processing devices or algorithms. With the improvement of environmental awareness, the sustainable development of data storage has also become an important issue. Reducing the energy consumption of data storage devices is an important direction for the development of data storage technology. Therefore, it is of great significance and value to fully explore the correlation between matrices to ultimately achieve low-complexity computing and storage. Therefore, research on reducing computational complexity based on matrix correlation to obtain better storage methods is the current mainstream direction. Summary of the invention

[0005] In view of the problems existing in the prior art, the present invention provides a data storage optimization method and system based on reducing matrix correlation. By mining the correlation between matrices and designing an effective optimization algorithm, the computational complexity and storage requirements can be reduced, and the efficiency and quality of data processing and storage can be improved.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions: a data storage optimization method based on matrix correlation reduction, comprising the following steps:

[0007] S1. Calculate the correlation coefficient between the separated matrices for different matrix types, and then eliminate the matrices that cannot be interpolated according to the correlation coefficient between the separated matrices, and retain the matrices that can be interpolated;

[0008] S2. Determine the correlation coefficient threshold value based on the threshold selection method of statistical significance test, and use the correlation coefficient threshold value to eliminate matrices that do not need to be interpolated, and retain matrices that need to be interpolated;

[0009] S3, for the matrix that needs to be interpolated, use the two-matrix interpolation method and the three-matrix interpolation method to perform interpolation operations on the matrix;

[0010] S4. By comparing the correlation coefficients between matrices and the number of matrices obtained by corresponding matrix interpolation that do not meet the constraints, the effect of the matrix interpolation method can be checked to further reduce the computational complexity and achieve the best optimization strategy for data storage.

[0011] Furthermore, in the aforementioned step S1, the matrix includes a real matrix and a complex matrix;

[0012] The correlation coefficient is calculated as follows:

[0013] Real matrix A=[a ij ] M×N and real matrix B = [b ij ] M×N The correlation coefficient between them is:

[0014]

[0015] Where M is the number of matrix rows, N is the number of matrix columns,

[0016] Complex matrix A = A R +A I i and the complex matrix B = B R +B I i(A R ,A I ,B R ,B I ) is:

[0017]

[0018] When the correlation coefficient is closer to 1, the correlation between the two matrices is stronger.

[0019] Furthermore, the aforementioned step S1 also includes minimizing the computational complexity of all matrices by using constraints during the computation process; the constraints are:

[0020] ρ min (V)≥0.99 (3)

[0021] Among them, ρ min is the minimum precision and V is a matrix.

[0022] Furthermore, in the aforementioned step S1, matrices that cannot be interpolated are eliminated according to the correlation coefficients between the separated matrices, and matrices that can be interpolated are retained, as follows:

[0023]

[0024] Among them, YES means interpolation is possible, NO means interpolation is not possible, s is the threshold value, the threshold value r s Used to make Satisfy the constraints.

[0025] Furthermore, the aforementioned step S2 includes the following sub-steps:

[0026] S2.1. Set the significance level α, which represents the probability of incorrectly rejecting the null hypothesis when the null hypothesis is true.

[0027] S2.2. Perform statistical tests on the data: Use the t-test method to process the data, check whether the data meets the assumptions, and calculate the corresponding p-value, where the p-value represents the probability of observing the current sample data or extreme data when the null hypothesis is true;

[0028] S2.3. Compare the p value in step S2.2 with the significance level. If p < α, the result is considered statistically significant. If p ≥ α, the result is considered not statistically significant and the null hypothesis is accepted or no conclusion can be drawn.

[0029] S2.4. Determine the threshold: The threshold is determined by the significance level. When p < α, the corresponding result or parameter is considered significant, and the result corresponding to the significance level α is the threshold.

[0030] Furthermore, in the aforementioned step S3, the two-matrix interpolation method is specifically: V j,k-1 With V j,k+1 Take the weighted average to calculate Right now:

[0031]

[0032] Among them, 0≤α≤1, the calculation amount of this interpolation algorithm is NL additions or subtractions.

[0033] Furthermore, in the aforementioned step S3, the three-matrix interpolation method specifically comprises: using three matrices to interpolate two matrices, that is, using V j,k , V j,k+2 and V j,k+4 calculate and It is expressed as:

[0034]

[0035] in, is the weighting coefficient.

[0036] Furthermore, the aforementioned data storage optimization method based on reducing matrix correlation uses the basis function of the Lagrange interpolation method to determine the weighting coefficient, and V j,k , V j,k+1 , V j,k+2 , V j,k+3 , V j,k+4 The corresponding one-dimensional coordinates are x0=0, x1=1, x2=2, x3=3, x4=4. The following interpolation basis function is constructed with the help of points x0, x2, x4:

[0037]

[0038] Substituting x1 and x3 into the above formula, we get the weighted coefficient:

[0039]

[0040] The computational effort of this interpolation algorithm is 2NL additions or subtractions.

[0041] Another aspect of the present invention highlights an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any one of the methods of the present invention when executing the computer program.

[0042] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of any one of the methods of the present invention.

[0043] Compared with the prior art, the beneficial technical effects of the present invention using the above technical solution are as follows:

[0044] (1) By analyzing the correlation between matrices, defining the correlation coefficient of the matrices, and setting the correlation coefficient threshold, we can find matrices that do not require interpolation and select a suitable algorithm to solve them, further reducing the complexity of matrix calculations.

[0045] (2) The optimization scheme proposed based on the interpolation algorithm can select a more standardized and scientific data storage method, facilitate the management of platform storage resources, and provide a reference for subsequent storage selection.

[0046] (3) Different from traditional optimization models that require data to be processed one by one, the data storage optimization method based on matrix correlation can reduce computational complexity, reduce input costs and system energy consumption, and help achieve green computing. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 It is a schematic diagram of the flow chart of the data storage optimization method of the present invention.

[0048] Figure 2 Flowchart of the matrix interpolation model.

[0049] Figure 3 is the matrix V j,k-1 and V j,k+1 The correlation coefficient of the two matrices and the matrix obtained by interpolating the two matrices Comparison chart that does not meet the constraints.

[0050] In the figure, (a) is Data1, a comparison chart of the first row of the matrix, (b) is Data2, a comparison chart of the second row of the matrix, (c) is Data3, a comparison chart of the first row of the matrix, (d) is Data3, a comparison chart of the second row of the matrix, (e) is Data5, a comparison chart of the second row of the matrix, and (f) is Data6, a comparison chart of the first row of the matrix. DETAILED DESCRIPTION

[0051] In order to better understand the technical content of the present invention, specific embodiments are given and described as follows in conjunction with the accompanying drawings.

[0052] Various aspects of the invention are described herein with reference to the accompanying drawings, in which many illustrative embodiments are shown. The embodiments of the invention are not limited to those described in the accompanying drawings. It should be understood that the invention is implemented by any of the various concepts and embodiments described above, as well as the concepts and embodiments described in detail below, because the concepts and embodiments disclosed in the invention are not limited to any implementation. In addition, some aspects disclosed in the invention may be used alone or in any appropriate combination with other aspects disclosed in the invention.

[0053] refer to Figure 1 The present invention proposes a data storage optimization method based on matrix correlation reduction, comprising the following steps:

[0054] S1. Calculate the correlation coefficient between the separated matrices for different matrix types, and then eliminate the matrices that cannot be interpolated according to the correlation coefficient between the separated matrices, and retain the matrices that can be interpolated;

[0055] S2. Determine the correlation coefficient threshold value based on the threshold selection method of statistical significance test, and use the correlation coefficient threshold value to eliminate matrices that do not need to be interpolated, and retain matrices that need to be interpolated;

[0056] S3. For matrices that need to be interpolated, two-matrix interpolation method and three-matrix interpolation method are used to perform interpolation operations on the matrices respectively; S4. By comparing the correlation coefficients between matrices and the number of matrices that do not meet the constraints in the corresponding matrix interpolation, the effect of the matrix interpolation method is checked to further reduce the computational complexity and achieve the best optimization strategy for data storage.

[0057] There is a strong correlation between the matrices. When the correlation coefficient is closer to 1, the correlation between the two matrices is stronger. The correlation coefficient can be divided into the following two cases: Specifically, in S1, the matrix includes real matrix and complex matrix;

[0058] The correlation coefficient is calculated as follows:

[0059] Real matrix A=[a ij ] M×N and real matrix B = [b ij ] M×N The correlation coefficient between them is:

[0060]

[0061] Where M is the number of matrix rows, N is the number of matrix columns,

[0062] Complex matrix A = A R +A I i and the complex matrix B = B R +B I i(A R ,A I ,B R ,B I ) is:

[0063]

[0064] Step S1 also includes minimizing the computational complexity of all matrices by using constraints during the computation process; the constraints are:

[0065] ρ min (V)≥0.99 (3)

[0066] Among them, ρ min is the minimum precision and V is a matrix.

[0067] As a preferred embodiment of the present invention, in step S1, matrices that cannot be interpolated are eliminated according to the correlation coefficient between the separated matrices, and matrices that can be interpolated are retained, as follows:

[0068]

[0069] Among them, YES means interpolation is possible, NO means interpolation is not possible, s is the threshold value, the threshold value r s Used to make Satisfy the constraints.

[0070] As a preferred embodiment of the present invention, in step S2, determining the correlation coefficient threshold can effectively identify matrices that do not need to be interpolated; specifically, the correlation coefficient threshold of the matrix is ​​a concept widely used in the fields of data analysis, statistics, and complex network construction. By setting the correlation coefficient threshold, a matrix that does not need to be interpolated can be found.

[0071] In matrix analysis, the correlation coefficient is used to measure the strength and direction of the linear relationship between two or more variables. The range of the correlation coefficient is usually between -1 and 1, where: values ​​close to 1 indicate strong positive correlation; values ​​close to -1 indicate strong negative correlation; values ​​close to 0 indicate almost no linear relationship. The purpose of setting the correlation coefficient threshold is to focus only on variable pairs with significant correlation in further data analysis or network construction, thereby simplifying the problem, reducing the amount of calculation, and improving the accuracy and efficiency of the analysis.

[0072] Step S2 includes the following sub-steps:

[0073] S2.1. Set the significance level α, which represents the probability of incorrectly rejecting the null hypothesis when the null hypothesis is true.

[0074] S2.2. Perform statistical tests on the data: Use the t-test method to process the data, check whether the data meets the assumptions, and calculate the corresponding p-value, where the p-value represents the probability of observing the current sample data or extreme data when the null hypothesis is true;

[0075] S2.3. Compare the p value in step S2.2 with the significance level. If p < α, the result is considered statistically significant. If p ≥ α, the result is considered not statistically significant and the null hypothesis is accepted or no conclusion can be drawn.

[0076] S2.4. Determine the threshold: The threshold is determined by the significance level. When p < α, the corresponding result or parameter is considered significant, and the result corresponding to the significance level α is the threshold.

[0077] like Figure 2As shown, as a preferred embodiment of the present invention, step S3 uses different interpolation methods to solve to obtain the number of matrices that do not meet the constraint conditions after interpolation.

[0078] Specifically, by using two-matrix and three-matrix interpolation methods, the number of matrices that do not meet the constraints can be obtained. The calculation formula is as follows:

[0079] Two-matrix interpolation method: V j,k-1 With V j,k+1 Take the weighted average to calculate Right now:

[0080]

[0081] Among them, 0≤α≤1, generally speaking, the weight coefficient α=1 / 2, and the calculation amount of this interpolation algorithm is NL additions or subtractions.

[0082] Three-matrix interpolation method: Use three matrices to interpolate two matrices, that is, use V j,k , V j,k+2 and V j,k+4 calculate and It is expressed as:

[0083]

[0084] in, is the weighting coefficient. For simplicity, the weighting coefficient is determined by using the basis function of the Lagrange interpolation method. Assume that V j,k , V j,k+1 , V j,k+2 , V j,k+3 , V j,k+4 The corresponding one-dimensional coordinates are x0=0, x1=1, x2=2, x3=3, x4=4. With the help of points x0, x2, x4, the following interpolation basis function can be constructed:

[0085]

[0086] Substituting x1 and x3 into the above formula, we can get the weighted coefficient:

[0087]

[0088] From the above description, it can be seen that the computational complexity of this interpolation algorithm is 2NL additions or subtractions.

[0089] In addition, in order to further reduce the computational complexity, V j,k-1 With V j,k+2 Calculate the weighted average and The specific calculation formula is as follows:

[0090]

[0091] Among them, 0≤α1≤α2≤1, generally speaking, α1=1 / 3, α2=1 / 3. Similarly, two matrices can also be used to interpolate three matrices or four matrices. However, as the number of matrices increases, the effect of the intermediate interpolation matrix will become worse. It can be seen that the use of interpolation can reduce the number of matrices. The computational complexity of .

[0092] Step S4 is specifically: by comparing the correlation coefficients between the matrices and the number of matrices that do not meet the constraints in the interpolation of the two matrices, the effect of the matrix interpolation method is checked, the complexity of matrix calculation is further reduced, and the current lack of optimization of high-frequency acquisition data storage space is compensated. Save the storage resource space of the big data platform, thereby improving data storage efficiency and reducing investment costs and system energy consumption.

[0093] In a specific embodiment, the obtained data Data1-Data6 are solved by two matrix interpolation method, where each set of data has 4 rows and 384 columns. The threshold is determined by the statistical significance test method. It can be seen that when the correlation coefficient threshold r is set s = 0.8, the interpolation matrix To further verify the reliability of the above method, the two matrices V under different thresholds are calculated respectively. j,k-1 and V j,k+1 Interpolation The number of constraints that do not meet the constraints is shown in Table 1. It can be seen that when the correlation coefficient threshold r s When =0.8, the interpolation effect is best.

[0094] Fixed correlation coefficient threshold r s = 0.8, giving the matrix V j,k-1 and V j,k+1 The correlation coefficient and the matrix obtained by interpolating these two matrices In the comparison chart that does not meet the constraints, only some data are selected for illustration, such as Figure 3 As shown. Figure 3 It can be seen from (a), (b), (c), (d), (e), and (f) that the matrix that does not meet the constraints The corresponding interval matrix V j,k-1 and V j,k+1 The correlation coefficient between them is obviously smaller. Furthermore, Table 2 gives the correlation coefficient threshold r s When 0.8 is taken, the matrix V in each set of data j,k-1 and V j,k+1 The average value of the correlation coefficient and the matrix calculated using the two-matrix interpolation method The number of matrices that meet the constraints in , from which we can find that the larger the average correlation coefficient, the more matrices that can be interpolated; the smaller the average correlation coefficient, the fewer matrices that can be interpolated.

[0095] Table 1: Two matrices V when the threshold takes different values j,k-1 and V j,k+1 Interpolation The number of constraints that are not satisfied

[0096]

[0097] Table 2: When the threshold r s When the value is 0.8, the two matrices V j,k-1 and V j,k+1 Interpolation The number of constraints that are satisfied

[0098]

[0099]

[0100] From the above description, it can be seen that the present invention adopts a data storage optimization method based on reducing matrix correlation, which reduces the computational complexity, helps to formulate a reasonable storage format, saves a large amount of data storage resources, and improves the resource utilization of the big data platform.

[0101] Another aspect of the present invention highlights an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any one of the methods of the present invention when executing the computer program.

[0102] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of any one of the methods of the present invention.

[0103] Although the present invention has been described above with preferred embodiments, it is not intended to limit the present invention. A person skilled in the art of the present invention may make various modifications and improvements without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the present invention shall be determined by the claims.

Claims

1. A data storage optimization method based on matrix correlation reduction, characterized in that: The following steps are involved: S1. Calculate the correlation coefficient between the separated matrices for different matrix types, and then eliminate the matrices that cannot be interpolated according to the correlation coefficient between the separated matrices, and retain the matrices that can be interpolated; S2. Determine the correlation coefficient threshold value based on the threshold selection method of statistical significance test, and use the correlation coefficient threshold value to eliminate matrices that do not need to be interpolated, and retain matrices that need to be interpolated; S3, for the matrix that needs to be interpolated, use the two-matrix interpolation method and the three-matrix interpolation method to perform interpolation operations on the matrix; S4. By comparing the correlation coefficients between matrices and the number of matrices obtained by corresponding matrix interpolation that do not meet the constraints, the effect of the matrix interpolation method can be checked to further reduce the computational complexity and achieve the best optimization strategy for data storage.

2. A data storage optimization method based on matrix correlation reduction according to claim 1, characterized in that: In step S1, the matrix includes a real matrix and a complex matrix; The correlation coefficient is calculated as follows: Real matrix A = [a ij ] M×N and real matrix B = [b ij ] M×N The correlation coefficient between them is: Where M is the number of matrix rows, N is the number of matrix columns, Complex matrix A = A R +A I i and the complex matrix B = B R +B I i(A R ,A I ,B R ,B I ) is: When the correlation coefficient is closer to 1, the correlation between the two matrices is stronger.

3. The data storage optimization method based on matrix correlation reduction according to claim 1 is characterized in that: Step S1 also includes minimizing the computational complexity of all matrices by using constraints during the computation process; the constraints are: r min (V)≥0.99 (3) Among them, ρ min is the minimum precision and V is a matrix.

4. The data storage optimization method based on matrix correlation reduction according to claim 1 is characterized in that: In step S1, matrices that cannot be interpolated are eliminated according to the correlation coefficients between the separated matrices, and matrices that can be interpolated are retained, as follows: Among them, YES means interpolation is possible, NO means interpolation is not possible, s is the threshold value, the threshold value r s Used to make Satisfy the constraints.

5. The data storage optimization method based on matrix correlation reduction according to claim 1 is characterized in that: Step S2 includes the following sub-steps: S2.

1. Set the significance level α, which represents the probability of incorrectly rejecting the null hypothesis when the null hypothesis is true. S2.

2. Perform statistical tests on the data: Use the t-test method to process the data, check whether the data meets the assumptions, and calculate the corresponding p-value, where the p-value represents the probability of observing the current sample data or extreme data when the null hypothesis is true; S2.

3. Compare the p value in step S2.2 with the significance level. If p < α, the result is considered statistically significant. If p ≥ α, the result is considered not statistically significant and the null hypothesis is accepted or no conclusion can be drawn. S2.

4. Determine the threshold: The threshold is determined by the significance level. When p < α, the corresponding result or parameter is considered significant, and the result corresponding to the significance level α is the threshold.

6. The data storage optimization method based on matrix correlation reduction according to claim 4 is characterized in that: In step S3, the two-matrix interpolation method is specifically: V j,k-1 With V j,k+1 Take the weighted average to calculate Right now: Among them, 0≤α≤1, the calculation amount of this interpolation algorithm is NL additions or subtractions.

7. The data storage optimization method based on matrix correlation reduction according to claim 4 is characterized in that: In step S3, the three-matrix interpolation method is specifically: using three matrices to interpolate two matrices, that is, using V j,k , V j,k+2 and V j,k+4 calculate and It is expressed as: in, is the weighting coefficient.

8. The data storage optimization method based on matrix correlation reduction according to claim 7 is characterized in that: The weighting coefficients are determined using the basis functions of the Lagrange interpolation method. Let V j,k , V j,k+1 , V j,k+2 , V j,k+3 , V j,k+4 The corresponding one-dimensional coordinates are x0=0, x1=1, x2=2, x3=3, x4=4. The following interpolation basis function is constructed with the help of points x0, x2, x4: Substituting x1 and x3 into the above formula, we get the weighted coefficient: The computational effort of this interpolation algorithm is 2NL additions or subtractions.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images