Big data prediction method based on submatrix decomposition

Abstract

The invention discloses a big data prediction method based on sub-matrix factorization, which comprises the following steps: constructing a data matrix, selecting sub-matrixes, determining the size and the quantity of the sub-matrixes, constructing a linear equation set, solving a weight matrix, and performing prediction output. According to the method, the sub-matrixes are used for constructing corresponding linear equations, so that a final prediction result is obtained, the weights of the equations are averaged, the obtained model can adapt to the requirement of big data processing, the prediction accuracy and stability can be remarkably improved, and the prediction efficiency is improved. The dimensionality of the original data is reduced by using the linear independent sub-matrix, and the calculation amount is reduced; by selecting a full-rank sub-matrix, the prediction accuracy of the model in different sub-spaces can be ensured; the method is suitable for high-dimensional data processing in various fields, and can be used in combination with other machine learning algorithms to further improve the performance.

Description

Technical Field

[0001] The present invention belongs to the technical field of big data prediction, and in particular relates to a big data prediction method based on submatrix decomposition. Background Art

[0002] With the advent of the big data era, the analysis and processing of high-dimensional data has become one of the core issues of modern data science. Big data prediction is the most core application of big data. The advantage of big data prediction is that it transforms a very difficult prediction problem into a relatively simple description problem. From the perspective of prediction, the results obtained by big data prediction not only obtain simple and objective conclusions for dealing with real business, but can also be used to help enterprises make business decisions. The collected information can also be planned to guide the development of greater consumer power.

[0003] In the existing technology, traditional linear regression methods can work effectively when processing low-dimensional data, but when faced with large-scale, high-dimensional data, they usually face problems of high computational complexity and excessive data dimensionality, resulting in poor prediction results of the model or difficulty in achieving real-time calculation. Therefore, there is an urgent need for an efficient prediction method that can significantly reduce computational complexity while maintaining prediction accuracy. Summary of the invention

[0004] To solve the above problems, the present invention proposes a big data prediction method based on submatrix decomposition. This method simplifies the computational complexity of large-scale data and improves the prediction accuracy by decomposing a high-dimensional matrix into several linearly independent submatrices.

[0005] To achieve the above object, the present invention provides the following technical solution: a big data prediction method based on submatrix decomposition, including the following method: Step 1: Construction of data matrix: Given a set of data matrix pairs (X i , Y i ), where X i is m i ×n matrix, Y i is m i ×1 output vector that converts all X i Merge into a large matrix X and convert the corresponding Y i Merge into vector Y; Step 2: Submatrix selection: Select non-overlapping n×n submatrices from the large matrix X. These submatrices need to satisfy the full rank condition and be full rank matrices. Step 3: Determine the size and number of sub-matrices: Determine the size and number of each sub-matrix based on the big data resources and prediction task requirements; Step 4: Construct a linear system of equations: For each selected submatrix, construct the corresponding linear system of equations: k W k = k ,in k and k are the submatrices and the corresponding target values, respectively, W k is the weight matrix to be solved; Step 5: Solve the weight matrix: average the weight matrices of all sub-matrices to obtain the final prediction weight matrix W; Step 6. Prediction output: Use matrix multiplication to multiply the new input matrix X by the weight matrix W to obtain the predicted output Y=XW.

[0006] Preferably, the full rank matrix refers to a matrix whose rank is equal to its order, and whose determinant is not equal to 0. For an n-order square matrix, its rank r(X) = n, then X is called a full rank matrix, and the large matrix X is a full rank matrix.

[0007] Preferably, the weight matrix is ​​a mathematical method used in the fields of statistics and machine learning to evaluate the degree of data correlation. The weight matrix uses different indicators to more accurately measure the relationship between data and produce favorable inferences.

[0008] Preferably, the weight matrix mainly describes the weight relationship between objects.

[0009] Preferably, for the existence of the big data, K is a relatively large number.

[0010] Preferably, the predicted output Y uses a matrix decomposition algorithm, which is a commonly used data analysis technique that decomposes a large matrix into smaller sub-matrices to simplify the difficulty of data processing and analysis.

[0011] Preferably, the matrix decomposition algorithm is suitable for high-dimensional data processing in various fields.

[0012] Preferably, the size and number of the sub-matrices in step three need to be optimized according to the performance of the computing device and the task requirements.

[0013] Technical effects and advantages of the present invention: 1. The present invention decomposes the original data matrix into a set of full-rank sub-matrices, and uses these sub-matrices to construct corresponding linear equations to obtain the final prediction result. By averaging the weights of these equations, the obtained model can not only meet the needs of big data processing, but also significantly improve the accuracy and stability of the prediction.

[0014] 2. The present invention has high computational efficiency: the use of linearly independent sub-matrices reduces the dimension of the original data and reduces the amount of computation.

[0015] 3. The present invention has high prediction accuracy: by selecting a full-rank submatrix, the prediction accuracy of the model in different subspaces can be ensured.

[0016] 4. The present invention is highly flexible: the method is applicable to high-dimensional data processing in various fields and can be used in combination with other machine learning algorithms to further improve performance.

[0017] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures pointed out in the description and claims. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in combination with the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0019] The present invention provides a big data prediction method based on submatrix decomposition, including the following method: Step 1: Construction of data matrix: Given a set of data matrix pairs (X i , Y i ), where X i is m i ×n matrix, Y i is m i ×1 output vector that converts all X i Merge into a large matrix X and convert the corresponding Y i Merge into vector Y; Step 2: Submatrix selection: Select non-overlapping n×n submatrices from the large matrix X. These submatrices need to satisfy the full rank condition and be full rank matrices. Step 3: Determine the size and number of sub-matrices: Determine the size and number of each sub-matrix based on the big data resources and prediction task requirements; Step 4: Construct a linear system of equations: For each selected submatrix, construct the corresponding linear system of equations: k W k = k ,in k and kare the submatrices and the corresponding target values, respectively, W k is the weight matrix to be solved; Step 5: Solve the weight matrix: average the weight matrices of all sub-matrices to obtain the final prediction weight matrix W; Step 6. Prediction output: Use matrix multiplication to multiply the new input matrix X by the weight matrix W to obtain the predicted output Y=XW.

[0020] As a specific implementation of the present invention, the full rank matrix refers to a matrix whose rank is equal to its order, and the determinant is not equal to 0. For an n-order square matrix, its rank r(X) = n, then X is called a full rank matrix, and the large matrix X is a full rank matrix.

[0021] As a specific implementation of the present invention, the weight matrix is ​​a mathematical method used in the fields of statistics and machine learning to evaluate the degree of data correlation. The weight matrix uses different indicators to more accurately measure the relationship between data and produce favorable inferences.

[0022] As a specific implementation of the present invention, the weight matrix mainly describes the weight relationship between objects.

[0023] As a specific implementation of the present invention, the big data exists, and K is a relatively large number.

[0024] As a specific implementation of the present invention, the predicted output Y adopts a matrix decomposition algorithm. The matrix decomposition algorithm is a commonly used data analysis technology that decomposes a large matrix into smaller sub-matrices to simplify the difficulty of data processing and analysis.

[0025] As a specific implementation of the present invention, the matrix decomposition algorithm is applicable to high-dimensional data processing in various fields.

[0026] As a specific implementation of the present invention, the size and number of the sub-matrices in step three need to be optimized according to the performance of the computing device and the task requirements.

[0027] How it works: Consider a pair of matrices (X i ; Y i ), i=1, 2…, L; where X i is an m i ×n matrix, and Y i is the corresponding m i ×1 output vector, and satisfies ; Assume that there is some definite relationship between these matrix pairs; First, all given X i Merge into a high matrix X , The superscript "t" indicates the matrix transpose. And merge Y accordingly i Get Y , By merging the matrices, a high-dimensional matrix X and a vector Y are constructed, which represent the feature matrix and the target vector respectively; So the dimension of X is (m1+…+ m L )×n, and the dimension of Y is (m1+…+ m L )×1; Assume that the linear independence of these n features means that the rank of the long submatrix of X is n; Since X is a high-dimensional matrix, some submatrices are selected to simplify the calculation. A sufficiently large non-overlapping n×n submatrix is ​​selected from X. These submatrices need to satisfy the full rank condition. In order to make a reasonable selection of these submatrices, a metric is introduced, namely the linear independence of these matrices in the original data set. By selecting linearly independent submatrices, the optimal subspace can be found in the feature space for prediction.

[0028] Using the corresponding Y parts, we form a new set of matrix pairs , Due to the existence of big data, K is a large number. For each k, since the matrix is full rank, and has a solution W k , so that , By selecting enough sub-matrices, the corresponding linear equations can be constructed; These equations combine the submatrices with the corresponding weight matrices W by matrix multiplication. k Combined, the target output Y is finally obtained k , since each W k are all obtained by solving the full-rank submatrix, so for all W k Find the average and get the final prediction weight matrix W.

[0029] , After obtaining the weight matrix, for any dimension of X, we can get the prediction result through simple matrix multiplication: , When a new X is given, the corresponding output Y can be predicted; by decomposing the high-dimensional matrix into several linearly independent sub-matrices, the computational complexity of large-scale data is simplified and the prediction accuracy is improved, which also significantly improves the prediction accuracy.

[0030] This submatrix decomposition-based method has broad application prospects in big data; it can be used for high-dimensional prediction problems in various fields such as financial data analysis, image recognition, and bioinformatics; in addition, due to the flexibility and scalability of this method, it can also be combined with other machine learning algorithms, such as random forests or deep learning models, to further improve the prediction performance.

[0031] The original data matrix is ​​decomposed into a set of full-rank sub-matrices, and these sub-matrices are used to construct the corresponding linear equations to obtain the final prediction result. By averaging the weights of these equations, the obtained model can not only meet the needs of big data processing, but also significantly improve the accuracy and stability of the prediction.

[0032] High computational efficiency: The use of linearly independent sub-matrices reduces the dimension of the original data and reduces the amount of computation.

[0033] High prediction accuracy: By selecting a full-rank submatrix, the prediction accuracy of the model in different subspaces can be ensured.

[0034] Strong flexibility: The method is applicable to high-dimensional data processing in various fields and can be combined with other machine learning algorithms to further improve performance.

[0035] Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent substitutions for some of the technical features therein; and these modifications or substitutions do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A big data prediction method based on submatrix decomposition, characterized in that: The method is as follows: Step 1: Construction of data matrix: Given a set of data matrix pairs (X i , Y i ), where X i is m i ×n matrix and m i >>n,Y i is m i ×1 output vector that converts all X i Merge into a large matrix X and convert the corresponding Y i Merge into vector Y; Step 2: Submatrix selection: Select non-overlapping n×n submatrices from the large matrix X. These submatrices need to satisfy the full rank condition and be full rank matrices. Step 3: Determine the size and number of sub-matrices: Determine the size and number of each sub-matrix based on the big data resources and prediction task requirements; Step 4: Construct a linear system of equations: For each selected submatrix, construct the corresponding linear system of equations: k W k = k ,in k and k are the submatrices and the corresponding target values, respectively, W k is the weight matrix to be solved; Step 5: Solve the weight matrix: average the weight matrices of all sub-matrices to obtain the final prediction weight matrix W; Step 6. Prediction output: Use matrix multiplication to multiply the new input matrix X by the weight matrix W to obtain the predicted output Y=XW.

2. The big data prediction method based on submatrix decomposition according to claim 1, characterized in that: The full-rank matrix refers to a matrix whose rank is equal to its order, and whose determinant is not equal to 0. For an n-order square matrix, its rank r(X) = n, then X is called a full-rank matrix, and the large matrix X is a full-rank matrix.

3. The big data prediction method based on submatrix decomposition according to claim 1, characterized in that: The weight matrix is ​​a mathematical method used in statistics and machine learning to evaluate the degree of data correlation. The weight matrix uses different indicators to more accurately measure the relationship between data and generate favorable inferences.

4. The big data prediction method based on submatrix decomposition according to claim 1, characterized in that: The weight matrix mainly describes the weight relationship between objects.

5. The big data prediction method based on submatrix decomposition according to claim 1, characterized in that: With the existence of the big data, K is a relatively large number.

6. The big data prediction method based on submatrix decomposition according to claim 1, characterized in that: The predicted output Y uses a matrix decomposition algorithm, which is a commonly used data analysis technique that decomposes a large matrix into smaller sub-matrices to simplify the difficulty of data processing and analysis.

7. The big data prediction method based on submatrix decomposition according to claim 6, characterized in that: The matrix decomposition algorithm is suitable for high-dimensional data processing in various fields.

8. The big data prediction method based on submatrix decomposition according to claim 1, characterized in that: The size and number of sub-matrices in step three need to be optimized according to the performance of the computing device and the task requirements.