Data prediction method based on real-time adaptive signal decomposition

Abstract

The invention discloses a data prediction method based on real-time adaptive signal decomposition. The method comprises the steps of adaptive signal dynamic adjustment, original data matrix decomposition, linear equation set construction, linear equation set solving, prediction result obtaining and sub-matrix decomposition method combination model. According to the method, the signal parameters are dynamically adjusted according to the input signal change through the parameter adjusting model, the purpose of adjusting the parameters is to enable the signal decomposition algorithm to better adapt to the signal change, the adaptive signal decomposition can adaptively adjust the decomposition parameters according to the dynamic change of the signal under the real-time requirement of signal processing, and the real-time performance of signal processing is improved. Accurate decomposition of the signal is realized; the dimensionality of the original data is reduced by using the linear independent sub-matrix, and the data volume is reduced; the prediction accuracy of the model in different subspaces is ensured by selecting the full-rank sub-matrixes, noise in original information is effectively removed by selecting the full-rank sub-matrixes, and the prediction performance of the model is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of big data prediction, and in particular to a data prediction method based on real-time adaptive signal decomposition. Background Art

[0002] With the rapid development of the Internet, information data has shown explosive growth. The amount of data continues to rise. Traditional data processing technology can no longer meet the demand. The concept of big data has emerged. With the advent of the big data era, the analysis and processing of high-dimensional data has become one of the core issues of modern data science. Traditional linear regression methods can work effectively when processing low-dimensional data, but when faced with large-scale, high-dimensional data, they usually face problems of high computational complexity and excessive data dimension, resulting in poor prediction results of the model or difficulty in achieving real-time calculation. Therefore, there is an urgent need for an efficient prediction method that can significantly reduce computational complexity while maintaining prediction accuracy. Summary of the invention

[0003] The object of the present invention is to provide a data prediction method based on real-time adaptive signal decomposition to solve the problems raised in the above background technology.

[0004] To achieve the above object, the present invention provides the following technical solution: a data prediction method based on real-time adaptive signal decomposition, comprising the following steps:

[0005] S1, adaptive signal dynamic adjustment, preprocessing the captured signal, checking the missing values, abnormal values, and repeated values ​​in the signal data, and performing appropriate preprocessing, inputting the preprocessed signal into the pre-trained parameter adjustment model, according to the changes of the input signal, the parameter adjustment model dynamically adjusts the signal parameters and outputs the adjusted parameters;

[0006] S2, decomposition of the original data matrix, inputting the real-time signal after parameter adjustment into the signal matrix decomposition algorithm, decomposing the original data matrix containing a large number of observations and variables into a set of full-rank sub-matrices, achieving real-time adaptive signal decomposition;

[0007] S3, construct a linear system of equations, based on a set of full-rank sub-matrices decomposed, given a set of data matrix pairs (X i ,Y i ), i = 1, 2, ..., L, where X i is m i ×n matrix, Y i is m i ×1 output vector, representing the target value or response variable of the ith data point, and satisfying

[0008] All given X iMerge together to form a high matrix X, and the corresponding Y i Merge into vector Y,

[0009] X=(X1,…,X L ) t

[0010] Y=(Y1,…,Y L ) t

[0011] Where t represents the matrix transpose;

[0012] So the dimension of X is (m1+…+m L )×n, and the dimension of Y is (m1+…+m L )×1, the linear independence of n features means that the rank of the longest submatrix of X is n;

[0013] We select sufficiently large non-overlapping n×n submatrices from X to make them full rank, and use the corresponding Y parts to form a new set of matrix pairs

[0014]

[0015] Due to the existence of big data, K is a large number. For each R, since the matrix is full rank, and has a solution W R , construct the corresponding linear equations:

[0016]

[0017] in, and are the submatrices and the corresponding target values, respectively, W R is the weight matrix to be solved;

[0018] S4. Solve the linear equations, average the weight matrices of all sub-matrices, and obtain the final prediction weight matrix W.

[0019]

[0020] By averaging the weights of the equation group, the prediction matrix obtained can not only meet the needs of big data processing, but also significantly improve the accuracy and stability of the prediction;

[0021] S5. Get the prediction result. After getting the prediction matrix, use matrix multiplication to multiply the new input matrix X by the weight matrix W to get the prediction result output as:

[0022] Y=XW

[0023] S6. Submatrix decomposition method combined with model: Based on the flexibility and scalability of the submatrix decomposition method, it is combined with the deep learning model to further improve the prediction performance. The specific method is as follows:

[0024] In the training process of deep learning models, the original data is reduced in dimension and denoised through sub-matrix decomposition, thereby improving the quality and availability of the data;

[0025] Submatrix decomposition is an effective feature extraction method. By decomposing the original data matrix of the training set, a series of representative feature vectors are obtained. These feature vectors are used as inputs of the deep learning model to help the model better identify and understand the data.

[0026] During the training process of deep learning models, submatrix decomposition provides useful information to the optimization algorithm, such as gradient and Hessian matrix, which helps the optimization algorithm converge to the optimal solution faster, thereby improving the training efficiency and performance of the deep learning model.

[0027] Furthermore, in S1, a machine learning algorithm is used to train a parameter adjustment model, and the specific steps are as follows:

[0028] Data preprocessing: Extract useful features from the input signal. The features are numerical, categorical or time series data. Use standardization, normalization and encoding techniques to preprocess the features. The preprocessed features will be used as the input of the model to normalize or standardize the feature data.

[0029] Data segmentation: The preprocessed data set is divided into training set, validation set and test set. The training set is used to train the model, the validation set is used to adjust the model parameters (such as hyperparameters), and the test set is used to evaluate the final performance of the model. The test set should be kept independent of the training set and validation set to ensure the fairness and reliability of the evaluation results.

[0030] Model training: Use training set data to train the classification model. During the training process, the model will learn the mapping relationship between input features and target parameters, and has high adaptability and accuracy;

[0031] Model validation: Use validation set data to evaluate the performance of the model and select appropriate evaluation indicators to quantify the performance of the model, such as mean square error, mean absolute error, and accuracy;

[0032] Model optimization: Tune the model according to the verification results, and use L1 and L2 regularization methods to prevent model overfitting;

[0033] Model testing: Use the test set data to evaluate the final performance of the model, calculate the evaluation indicators on the test set to evaluate the generalization ability of the model, analyze the errors of the model on the test set, understand the weaknesses of the model and make improvements.

[0034] Compared with the prior art, the beneficial effects of the present invention are as follows: the signal parameters are dynamically adjusted according to the input signal changes through the trained parameter adjustment model, the purpose of adjusting the parameters is to enable the signal decomposition algorithm to better adapt to the signal changes and realize real-time adaptive signal decomposition, and the adaptive signal decomposition can adaptively adjust the decomposition parameters according to the dynamic changes of the signal under the real-time requirements of signal processing to realize accurate decomposition of the signal, thereby improving the accuracy and efficiency of decomposition; the dimension of the original data is reduced by using linearly independent sub-matrices, so that the amount of data to be processed in subsequent calculations and analyses will be greatly reduced, and the data after dimensionality reduction has a lower dimension and a simpler structure, This makes the calculation process more efficient, and the reduced-dimensional data takes up less storage space and has a faster transmission speed. By selecting full-rank submatrices, the prediction accuracy of the model in different subspaces is ensured. The linear independence of the full-rank submatrix makes the prediction results of the model in different subspaces more stable. Even when the data changes slightly, the model can maintain good prediction performance. By selecting full-rank submatrices, the noise in the original information is effectively removed, thereby improving the prediction performance of the model. The flexibility and scalability of the submatrix decomposition-based method are suitable for high-dimensional data processing in various fields, and can be used in combination with deep learning models to further improve the prediction performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 is a flow chart of the present invention;

[0036] Figure 2 is a flow chart of the method of the present invention;

[0037] Figure 3 The parameter adjustment model training flow chart of the present invention;

[0038] Figure 4 This is a flow chart of the sub-matrix decomposition method combined with the model of the present invention. DETAILED DESCRIPTION

[0039] The technical solution of the present invention is described in detail below through specific embodiments.

[0040] See also Figure 1-4 The present invention provides a data prediction method based on real-time adaptive signal decomposition, comprising the following steps:

[0041] S1. Adaptive signal dynamic adjustment: pre-process the captured signal, check the missing values, abnormal values, and repeated values ​​in the signal data, and perform appropriate processing such as filtering and denoising to improve the signal quality, ensure that the format and type of the data are consistent with the model requirements, and input the pre-processed signal into the pre-trained parameter adjustment model. According to the changes in the input signal, the parameter adjustment model dynamically adjusts the signal parameters and outputs the adjusted parameters. The purpose of adjusting the parameters is to enable the signal decomposition algorithm to better adapt to the changes in the signal, thereby improving the accuracy and efficiency of the decomposition;

[0042] Among them, the parameter adjustment model is trained using a machine learning algorithm. The specific steps are as follows:

[0043] Data preprocessing: Extract useful features from the input signal. The features are numerical, categorical or time series data. Use standardization, normalization and encoding techniques to preprocess the features. The preprocessed features will be used as the input of the model. The feature data will be normalized or standardized to ensure the stability and efficiency of the model training.

[0044] Data segmentation: The preprocessed dataset is divided into a training set, a validation set, and a test set. The training set accounts for 70%, the validation set accounts for 15%, and the test set accounts for 15%. The training set is used to train the model, the validation set is used to adjust the model parameters (such as hyperparameters), and the test set is used to evaluate the final performance of the model. The test set should be independent of the training set and the validation set to ensure the fairness and reliability of the evaluation results.

[0045] Model training: Use training set data to train the classification model. During the training process, the model will learn the mapping relationship between input features and target parameters, and has high adaptability and accuracy;

[0046] Model validation: Use validation set data to evaluate the performance of the model and select appropriate evaluation indicators to quantify the performance of the model, such as mean square error, mean absolute error, and accuracy. If the model performs poorly on the validation set, adjust the model's hyperparameters through grid search, random search, or Bayesian optimization methods to improve the model's performance.

[0047] Model optimization: Tune the model according to the verification results, and use L1 and L2 regularization methods to prevent model overfitting;

[0048] Model testing: Use the test set data to evaluate the final performance of the model, calculate the evaluation indicators on the test set to evaluate the generalization ability of the model, analyze the errors of the model on the test set, understand the weaknesses of the model and make improvements;

[0049] S2, decomposition of the original data matrix, inputting the signal after real-time parameter adjustment into the signal matrix decomposition algorithm, decomposing the original data matrix containing a large number of observations and variables into a set of full-rank sub-matrices, the rank of which is equal to the number of rows or columns, thereby retaining most of the information in the original data and realizing real-time adaptive signal decomposition;

[0050] S3, construct a linear system of equations, based on a set of full-rank sub-matrices decomposed, given a set of data matrix pairs (X i ,Y i ), i = 1, 2, ..., L, where X i is m i ×n matrix, Y i is m i ×1 output vector, representing the target value or response variable of the ith data point, and satisfying

[0051] X i and Y i There is a definite relationship between them. For a new matrix X of dimension m×n, the task is to determine the corresponding prediction result Y, whose dimension is m×1; i The n columns are called features, and they represent all given X i Merge together to form a high matrix X, and the corresponding Y i Merge into vector Y,

[0052] X=(X1,…,X L ) t

[0053] Y=(Y1,…,Y L ) t

[0054] Where t represents the matrix transpose;

[0055] So the dimension of X is (m1+…+m L )×n, and the dimension of Y is (m1+…+m L )×1, the linear independence of n features means that the rank of the longest submatrix of X is n;

[0056] We select sufficiently large non-overlapping n×n submatrices from X to make them full rank, and use the corresponding Y parts to form a new set of matrix pairs

[0057]

[0058] Due to the existence of big data, K is a large number. For each R, since the matrix is full rank, and has a solution W R , construct the corresponding linear equations:

[0059]

[0060] in, and are the submatrices and the corresponding target values, respectively, W R is the weight matrix to be solved;

[0061] S4. Solve the linear equations, average the weight matrices of all sub-matrices, and obtain the final prediction weight matrix W.

[0062]

[0063] By averaging the weights of the equation group, the resulting model can not only meet the needs of big data processing, but also significantly improve the accuracy and stability of predictions;

[0064] S5. Get the prediction result. After getting the prediction matrix, use matrix multiplication to multiply the new input matrix X by the weight matrix W to get the prediction result output as:

[0065] Y=XW

[0066] The final prediction result is obtained by decomposing the original data matrix into a set of full-rank sub-matrices and using these sub-matrices to construct the corresponding linear equations.

[0067] S6. Submatrix decomposition method combined with model: Based on the flexibility and scalability of the submatrix decomposition method, it is combined with the deep learning model to further improve the prediction performance. The specific method is as follows:

[0068] In the training process of deep learning models, sub-matrix decomposition is used to reduce the dimension and remove noise from the original data, thereby improving the quality and availability of the data, which helps the deep learning model to better learn the characteristics and laws of the data;

[0069] The processed data set is divided into training set, validation set and test set. The training set is used to train the model, the validation set is used to adjust the model's hyperparameters and perform model selection, and the test set is used to evaluate the final performance of the model.

[0070] As an effective feature extraction method, submatrix decomposition decomposes the original data matrix of the training set to obtain a series of representative feature vectors. These feature vectors are used as inputs of deep learning models to help the models better identify and understand data.

[0071] After data preparation is completed, it is necessary to build a suitable deep learning model, select a deep learning framework, and define the network structure;

[0072] Choose a deep learning framework. The PyTorch framework provides a wealth of tools for building and training neural networks.

[0073] Define the network structure. According to the characteristics of the task and data, define the number of layers of the neural network, the type of each layer (such as fully connected layer, convolutional layer, recurrent layer) and the number of nodes. At the same time, select the appropriate activation function (such as ReLU, Sigmoid) and output layer (such as softmax layer in classification tasks);

[0074] After building the model, you need to compile the model and specify the loss function, optimizer, and evaluation metrics;

[0075] Loss function: used to measure the difference between the model's prediction results and the actual labels. Select the appropriate loss function according to the task type, such as the cross entropy loss function in classification tasks and the mean square error loss function in regression tasks.

[0076] Optimizer: used to adjust the weights of the neural network to minimize the loss function. Optimizers include SGD and Adam. Selecting a suitable optimizer can improve the training efficiency and performance of the model.

[0077] Evaluation indicators: used to evaluate the performance of the model. Appropriate evaluation indicators are selected according to the task type, such as accuracy, precision, and recall in classification tasks, and mean square error and R-square value in regression tasks;

[0078] The deep learning model learns by inputting feature vectors and adjusting the internal weights;

[0079] Divide the training data into multiple small batches, each batch contains a certain number of samples. During the training process, the model processes the data batch by batch and updates the weights.

[0080] The entire training process is iterative, usually requiring multiple passes through the training set. In each iteration, the model processes all batches of data and updates the weights.

[0081] When the performance on the validation set no longer improves, stop training and save the current model to avoid overfitting of the model;

[0082] During the training process of deep learning models, submatrix decomposition provides useful information to the optimization algorithm, such as gradient and Hessian matrix, which helps the optimization algorithm converge to the optimal solution faster, thereby improving the training efficiency and performance of deep learning models.

[0083] After training is completed, a test dataset is needed to evaluate the performance of the model;

[0084] Use the test data set to calculate the model's accuracy, precision, and recall metrics to evaluate the model's performance;

[0085] Adjust the model’s hyperparameters (such as learning rate, batch size, number of network layers) based on the evaluation results to improve the model’s performance;

[0086] The trained model is used to make predictions on new data, input new data into the model, and obtain the model's prediction results.

[0087] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can replace or change the technical solution and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A data prediction method based on real-time adaptive signal decomposition, characterized in that: The following steps are involved: S1, adaptive signal dynamic adjustment, preprocessing the captured signal, checking the missing values, abnormal values, and repeated values ​​in the signal data, and performing appropriate preprocessing, inputting the preprocessed signal into the pre-trained parameter adjustment model, according to the changes of the input signal, the parameter adjustment model dynamically adjusts the signal parameters and outputs the adjusted parameters; S2, decomposition of the original data matrix, inputting the real-time signal after parameter adjustment into the signal matrix decomposition algorithm, decomposing the original data matrix containing a large number of observations and variables into a set of full-rank sub-matrices, achieving real-time adaptive signal decomposition; S3, construct a linear system of equations, based on a set of full-rank sub-matrices decomposed, given a set of data matrix pairs (X i ,Y i ), i = 1, 2, ..., L, where X i is m i ×n matrix, Y i is m i ×1 output vector, representing the target value or response variable of the ith data point, and satisfying All given X i Merge together to form a high matrix X, and the corresponding Y i Merge into vector Y, X=(X1,…,X L ) t Y=(Y1,…,Y L ) t Where t represents the matrix transpose; So the dimension of X is (m1+…+m L )×n, and the dimension of Y is (m1+…+m L )×1, the linear independence of n features means that the rank of the longest submatrix of X is n; We select sufficiently large non-overlapping n×n submatrices from X to make them full rank, and use the corresponding Y parts to form a new set of matrix pairs Due to the existence of big data, K is a large number. For each R, since the matrix is full rank, and has a solution W R , construct the corresponding linear equations: in, and are the submatrices and the corresponding target values, respectively, W R is the weight matrix to be solved; S4. Solve the linear equations, average the weight matrices of all sub-matrices, and obtain the final prediction weight matrix W. By averaging the weights of the equation group, the prediction matrix obtained can not only meet the needs of big data processing, but also significantly improve the accuracy and stability of the prediction; S5. Get the prediction result. After getting the prediction matrix, use matrix multiplication to multiply the new input matrix X by the weight matrix W to get the prediction result output as: Y=XW S6. The sub-matrix decomposition method is combined with a model. Based on the flexibility and scalability of the sub-matrix decomposition method, it is combined with a deep learning model to predict the data.

2. A data prediction method based on real-time adaptive signal decomposition according to claim 1, characterized in that: In S1, a machine learning algorithm is used to train a parameter adjustment model, and the specific steps are as follows: Data preprocessing: Extract useful features from the input signal. The features are numerical, categorical or time series data. Use standardization, normalization and encoding techniques to preprocess the features. The preprocessed features will be used as the input of the model to normalize or standardize the feature data. Data segmentation: The preprocessed data set is divided into training set, validation set and test set. The training set is used to train the model, the validation set is used to adjust the model parameters (such as hyperparameters), and the test set is used to evaluate the final performance of the model. The test set should be kept independent of the training set and validation set to ensure the fairness and reliability of the evaluation results. Model training: Use training set data to train the classification model. During the training process, the model will learn the mapping relationship between input features and target parameters, and has high adaptability and accuracy; Model validation: Use validation set data to evaluate the performance of the model and select appropriate evaluation indicators to quantify the performance of the model, such as mean square error, mean absolute error, and accuracy; Model optimization: Tune the model according to the verification results, and use L1 and L2 regularization methods to prevent the model from overfitting; Model testing: Use the test set data to evaluate the final performance of the model, calculate the evaluation indicators on the test set to evaluate the generalization ability of the model, analyze the errors of the model on the test set, understand the weaknesses of the model and make improvements.

Images