A data prediction method based on real-time adaptive signal decomposition

Abstract

The application discloses a data prediction method based on real-time adaptive signal decomposition, which comprises adaptive signal dynamic adjustment, original data matrix decomposition, linear equation set construction, linear equation set solution, prediction result obtaining and sub-matrix decomposition method combined model. The application dynamically adjusts signal parameters of input signal change through a parameter adjustment model, and the purpose of adjusting parameters is to make the signal decomposition algorithm better adapt to signal change. The adaptive signal decomposition can adjust decomposition parameters according to dynamic change of the signal under the real-time requirement of signal processing, and realizes accurate decomposition of the signal. The linear independent sub-matrix is used to reduce the dimension of original data and reduce data volume. By selecting a full-rank sub-matrix, the prediction accuracy of the model in different subspaces is ensured, and by selecting the full-rank sub-matrix, noise in original information is effectively removed, and the prediction performance of the model is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of big data prediction, and particularly relates to a data prediction method based on real-time adaptive signal decomposition. BACKGROUND

[0002] With the rapid development of the Internet, information data presents explosive growth, and the amount of data is increasing. The traditional data processing technology cannot meet the demand, and the concept of big data emerges as the times require. In the era of big data, the analysis and processing of high-dimensional data become one of the core problems of modern data science. The traditional linear regression method can effectively work when processing low-dimensional data, but when facing large-scale and high-dimensional data, it usually faces the problems of high computational complexity and high data dimension, which leads to poor prediction effect of the model or difficulty in realizing real-time calculation. Therefore, there is an urgent need for an efficient prediction method which can significantly reduce the computational complexity while maintaining the prediction accuracy. SUMMARY

[0003] The present application aims to provide a data prediction method based on real-time adaptive signal decomposition to solve the problems in the background art.

[0004] To achieve the above-mentioned purpose, the present application provides the following technical scheme: a data prediction method based on real-time adaptive signal decomposition, comprising the following steps:

[0005] S1, adaptive signal dynamic adjustment, preprocessing the captured signal, checking the missing values, abnormal values and repeated values in the signal data, and appropriately preprocessing the signal, inputting the preprocessed signal into a pre-trained parameter adjustment model, dynamically adjusting the signal parameters according to the change of the input signal, and outputting the adjusted parameters;

[0006] S2, original data matrix decomposition, inputting the real-time signal after parameter adjustment into a signal matrix decomposition algorithm, decomposing the original data matrix containing a large number of observation values and variables into a group of full-rank submatrices to achieve real-time adaptive signal decomposition;

[0007] S3, constructing a linear equation set, based on the group of full-rank submatrices decomposed, a group of data matrices (X i ,Y i ), i=1,2,…,L, wherein, X i is an m i ×n matrix, Y i is an m i ×1 output vector, representing the target value or response variable of the i-th data point, and satisfying

[0008] inputting all given X iMerge together to form a high matrix X, and the corresponding Y i Merge into a vector Y,

[0009] X = (X1, …, X L ) t

[0010] Y = (Y1, …, Y L ) t

[0011] Where t represents matrix transposition;

[0012] So the dimension of X is (m1+…+m L )×n, and the dimension of Y is (m1+…+m L )×1, the linear independence of n features means that the rank of the long submatrix of X is n;

[0013] A sufficient number of non-overlapping n×n submatrices are selected from X, whose rank is full rank, and a new set of matrix pairs is formed using the corresponding Y part

[0014]

[0015] Due to the existence of big data, K is a large number, and for each R, due to the full rank of the matrix There is a solution W R , and the corresponding linear equation system is constructed:

[0016]

[0017] Where and are the submatrix and the corresponding target value respectively, and W R is the weight matrix to be solved;

[0018] S4, solve the linear equation system, and average the weight matrix of all submatrices to obtain the final prediction weight matrix W,

[0019]

[0020] The prediction matrix obtained by averaging the weights of the equation system not only meets the demand of big data processing, but also significantly improves the accuracy and stability of prediction;

[0021] S5, get the prediction result, after getting the prediction matrix, multiply the new input matrix X with the weight matrix W by using matrix multiplication, and get the prediction result output as:

[0022] Y = XW

[0023] S6, the sub-matrix decomposition method combines the model, based on the flexibility and scalability of the sub-matrix decomposition method, combined with deep learning model, further improve the prediction performance, the specific method is:

[0024] In the training process of deep learning model, through sub-matrix decomposition, the original data is processed by dimensionality reduction and denoising, so as to improve the quality and availability of data;

[0025] And as an effective feature extraction method, sub-matrix decomposition can obtain a series of representative feature vectors by decomposing the original data matrix of the training set, which can help the model better identify and understand the data as the input of the deep learning model;

[0026] In the training process of deep learning model, sub-matrix decomposition provides useful information for optimization algorithm, such as gradient and Hessian matrix, which helps optimization algorithm converge to optimal solution faster, thus improving the training efficiency and performance of deep learning model.

[0027] Further, in the S1, a parameter adjustment model is trained using machine learning algorithm, and the specific steps are as follows:

[0028] Data preprocessing: useful features are extracted from input signals, and features are numerical, categorical or time series data. Standardization, normalization and encoding techniques are used for feature preprocessing. The preprocessed features will be used as the input of the model, and the feature data will be normalized or standardized;

[0029] Data segmentation: the preprocessed data set is divided into training set, validation set and test set. The training set is used to train the model, the validation set is used to adjust the model parameters (such as hyperparameters), and the test set is used to evaluate the final performance of the model. The test set should be independent of the training set and validation set to ensure the fairness and reliability of the evaluation results;

[0030] Model training: use training set data to train classification model. In the training process, the model will learn the mapping relationship between input features and target parameters, which has high adaptability and accuracy;

[0031] Model verification: use validation set data to evaluate the performance of the model, select appropriate evaluation indicators to quantify the performance of the model, such as mean square error, mean absolute error and accuracy;

[0032] Model optimization: according to the verification result, the model is optimized, and L1, L2 regularization method is used to prevent model overfitting;

[0033] Model testing: use test set data to evaluate the final performance of the model, calculate the evaluation indicators on the test set to evaluate the generalization ability of the model, analyze the errors of the model on the test set, understand the weaknesses of the model and improve it.

[0034] Compared with the prior art, the beneficial effects of the present application are: the trained parameter adjustment model dynamically adjusts the signal parameters of the input signal changes, and the purpose of adjusting the parameters is to enable the signal decomposition algorithm to better adapt to the changes of the signal, realize real-time adaptive signal decomposition, and adaptively adjust the decomposition parameters according to the dynamic changes of the signal under the real-time requirement of signal processing, realize accurate decomposition of the signal, and thus improve the accuracy and efficiency of the decomposition; the use of linear independent sub-matrices reduces the dimensionality of the original data, so that the amount of data to be processed in subsequent calculation and analysis will be greatly reduced, and the reduced data has lower dimensionality and simpler structure, which makes the calculation process more efficient, and the reduced data occupies less storage space and has faster transmission speed; by selecting a full rank sub-matrix, the prediction accuracy of the model in different subspaces is ensured, and the linear independence of the full rank sub-matrix makes the prediction results of the model in different subspaces more stable, that is, even in the case of slight changes in data, the model can maintain good prediction performance, and by selecting a full rank sub-matrix, the noise in the original information is effectively removed, and the prediction performance of the model is improved; the flexibility and scalability of the sub-matrix decomposition method are suitable for high-dimensional data processing in various fields, and are combined with a deep learning model to further improve the prediction performance. BRIEF DESCRIPTION OF DRAWINGS

[0035] Figure 1 The flowchart of the present application;

[0036] Figure 2 The method flowchart of the present application;

[0037] Figure 3 The parameter adjustment model training flowchart of the present application;

[0038] Figure 4 The sub-matrix decomposition method combined model flowchart of the present application. DETAILED DESCRIPTION

[0039] In the following, the technical solutions of the present application will be described in detail through specific embodiments.

[0040] Please refer to Figures 1-4 The present application provides a data prediction method based on real-time adaptive signal decomposition, comprising the following steps:

[0041] S1, adaptive signal dynamic adjustment, preprocessing the captured signal, checking missing values, outliers, and repeated values in the signal data, and performing appropriate processing such as filtering and denoising to improve signal quality, ensuring that the format and type of data are consistent with the model requirements, inputting the preprocessed signal into the pre-trained parameter adjustment model, dynamically adjusting the signal parameters according to the input signal changes, and outputting the adjusted parameters, the purpose of adjusting the parameters is to make the signal decomposition algorithm better adapt to the changes of the signal, so as to improve the accuracy and efficiency of the decomposition;

[0042] The parameter adjustment model is trained using a machine learning algorithm, and the specific steps are as follows:

[0043] Data preprocessing: extract useful features from the input signal, the features are numerical, categorical or time series data, use standardization, normalization and encoding techniques to preprocess the features, the preprocessed features will be used as the input of the model, normalize or standardize the feature data to ensure the stability and efficiency of the model training;

[0044] Data segmentation: divide the preprocessed data set into training set, validation set and test set, the training set accounts for 70%, the validation set accounts for 15%, and the test set accounts for 15%, the training set is used to train the model, the validation set is used to adjust the model parameters (such as hyperparameters), and the test set is used to evaluate the final performance of the model, the test set should be independent of the training set and the validation set to ensure the fairness and reliability of the evaluation results;

[0045] Model training: use the training set data to train the classification model, in the training process, the model will learn the mapping relationship between the input features and the target parameters, and has high adaptability and accuracy;

[0046] Model validation: use the validation set data to evaluate the performance of the model, select appropriate evaluation indicators to quantify the performance of the model, such as mean square error, mean absolute error, and accuracy, if the model performs poorly on the validation set, adjust the hyperparameters of the model through grid search, random search or Bayesian optimization method to improve the performance of the model;

[0047] Model optimization: according to the validation results, the model is optimized, L1 and L2 regularization methods are used to prevent model overfitting;

[0048] Model testing: use the test set data to evaluate the final performance of the model, calculate the evaluation indicators on the test set to evaluate the generalization ability of the model, analyze the errors of the model on the test set, understand the weaknesses of the model and improve it;

[0049] S2, original data matrix decomposition, the signal input to the signal matrix decomposition algorithm after adjusting the real-time parameter, the original data matrix containing a large number of observation values and variables is decomposed into a group of full rank submatrix, the rank of the submatrix is the number of rows or columns, so as to retain most of the information in the original data, realize real-time adaptive signal decomposition;

[0050] S3, construct linear equations, based on the decomposed group of full rank submatrix, give a group of data matrix pair (X i ,Y i ), i=1, 2, …, L, wherein, X i is m i ×n matrix, Y i is m i ×1 output vector, indicating the target value or response variable of the i-th data point, and satisfies

[0051] There is a certain relationship between X i and Y i , for a new dimension m×n matrix X, the task is to determine the corresponding prediction result Y, which is m×1; X i n column is called a feature, all given X i is combined together to form a high matrix X, and the corresponding Y i is combined into a vector Y,

[0052] X=(X1,…,X L ) t

[0053] Y=(Y1,…,Y L ) t

[0054] Where t represents matrix transposition;

[0055] Then the dimension of X is (m1+…+m L )×n, and the dimension of Y is (m1+…+m L )×1, the linear independence of n features means that the rank of the long submatrix of X is n;

[0056] A large enough non-overlapping n×n submatrix is selected from X, so that its rank is full rank, and a new group of matrix pairs

[0057]

[0058] Because of the existence of big data, K is a large number, for each R, because the matrix is full rank, there is a solution W R , construct the corresponding linear equations:

[0059]

[0060] wherein, and are sub-matrices and corresponding target values, respectively, W R is the weight matrix to be solved;

[0061] S4, solving the linear equations, averaging the weight matrix of all sub-matrices to obtain the final prediction weight matrix W,

[0062]

[0063] The model obtained by averaging the weights of the equation set not only can adapt to the demand of big data processing, but also can significantly improve the accuracy and stability of prediction;

[0064] S5, obtaining the prediction result, after obtaining the prediction matrix, multiplying the new input matrix X with the weight matrix W by using matrix multiplication to obtain the prediction result output as:

[0065] Y = XW

[0066] By decomposing the original data matrix into a set of full rank sub-matrices, and using these sub-matrices to construct the corresponding linear equation set, the final prediction result is obtained;

[0067] S6, the sub-matrix decomposition method combines the model, based on the flexibility and scalability of the sub-matrix decomposition method, combined with the deep learning model, further improves the prediction performance, the specific method is:

[0068] In the training process of the deep learning model, through sub-matrix decomposition, the original data is processed by dimensionality reduction and denoising, so as to improve the quality and availability of data, which helps the deep learning model to better learn the characteristics and rules of data;

[0069] The processed data set is divided into training set, validation set and test set, the training set is used to train the model, the validation set is used to adjust the hyperparameters of the model and select the model, and the test set is used to evaluate the final performance of the model;

[0070] As an effective feature extraction method, sub-matrix decomposition decomposes the original data matrix of the training set to obtain a series of representative feature vectors, which are used as the input of the deep learning model to help the model better identify and understand data;

[0071] After the data preparation is completed, a suitable deep learning model needs to be constructed, the deep learning framework needs to be selected, and the network structure needs to be defined;

[0072] Select a deep learning framework, choose PyToRch framework provides rich neural network construction and training tools;

[0073] Define network structure, according to the task and data characteristics, define the number of layers of neural network, the type of each layer (such as full connection layer, convolutional layer, cycle layer) and node number, at the same time, choose appropriate activation function (such as ReLU, Sigmoid) and output layer (such as softmax layer in classification task);

[0074] After building the model, you need to compile the model, specify the loss function, optimizer and evaluation index;

[0075] Loss function: used to measure the difference between the predicted results of the model and the actual labels, select appropriate loss function according to task type, such as cross-entropy loss function in classification task, mean square error loss function in regression task.

[0076] Optimizer: used to adjust the weights of neural network to minimize the loss function, optimizer includes SGD, Adam, select appropriate optimizer to improve the training efficiency and performance of the model.

[0077] Evaluation index: used to evaluate the performance of the model, select appropriate evaluation index according to task type, such as accuracy, precision, recall in classification task, mean square error, R square value in regression task;

[0078] Deep learning model learns through input feature vector and adjusts internal weight;

[0079] Divide the training data into multiple small batches, each batch contains a certain number of samples, in the training process, the model will process the data batch by batch and update the weight;

[0080] The whole training process is iterative, usually needs to traverse the training set multiple times, in each iteration, the model will process all batches of data and update the weight;

[0081] When the performance on the validation set no longer improves, stop training and save the current model to avoid overfitting of the model;

[0082] In the training process of deep learning model, submatrix decomposition provides useful information for optimization algorithm, such as gradient, Hessian matrix, helps optimization algorithm to converge to optimal solution faster, so as to improve the training efficiency and performance of deep learning model;

[0083] After training, you need to use test data set to evaluate the performance of the model;

[0084] Use test data set to calculate the accuracy, precision, recall index of the model to evaluate the performance of the model;

[0085] Adjust the hyperparameters (such as learning rate, batch size, network layer) of the model according to the evaluation results to improve the performance of the model;

[0086] The trained model is used for predicting new data, inputting the new data into the model, and obtaining the prediction result of the model.

[0087] The above is only a preferred specific embodiment of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art within the technical range disclosed by the present application, according to the technical solution and the inventive concept of the present application, replaces or changes, should be covered within the protection scope of the present application.

Claims

1. A data prediction method based on real-time adaptive signal decomposition, characterized by: The following steps are involved: S1. Adaptive signal dynamic adjustment: preprocess the captured signal, check for missing values, outliers, and duplicate values ​​in the signal data, and perform appropriate preprocessing. The preprocessed signal is input into the pre-trained parameter adjustment model. According to the changes in the input signal, the parameter adjustment model dynamically adjusts the signal parameters and outputs the adjusted parameters. S2, raw data matrix decomposition, inputs the real-time signal after parameter adjustment into the signal matrix decomposition algorithm, decomposes the raw data matrix containing a large number of observations and variables into a set of full-rank sub-matrices, achieving real-time adaptive signal decomposition; S3, construct a linear equation system based on a set of full-rank submatrices of the decomposition, given a set of data matrix pairs (X i ,Y i ), i = 1, 2, ..., L, where X i is m i ×n matrix, Y i is m i ×1 output vector, representing the target value or response variable of the i-th data point, and satisfying All given X i Merge together to form a high matrix X, and the corresponding Y i Merge into vector Y, X=(X1,…,X L ) t Y=(Y1,…,Y L ) t Where t represents the matrix transpose; So the dimension of X is (m1+…+m L )×n, and the dimension of Y is (m1+…+m L )×1, the linear independence of n features means that the rank of the longest submatrix of X is n; We select sufficiently large non-overlapping n×n submatrices from X to make them full rank, and use the corresponding parts of Y to form a new set of matrix pairs Due to the existence of big data, K is a large number. For each R, since the matrix is full rank, there is a solution W R , construct the corresponding linear equations: in, and are the submatrices and the corresponding target values, W R is the weight matrix to be solved; S4. Solve the linear equations, average the weight matrices of all sub-matrices, and obtain the final prediction weight matrix W. By averaging the weights of the equation group, the resulting prediction matrix can not only meet the needs of big data processing, but also significantly improve the accuracy and stability of the prediction; S5. Get the prediction result. After getting the prediction matrix, use matrix multiplication to multiply the new input matrix X with the weight matrix W to get the prediction result output as: Y=XW S6. Sub-matrix decomposition method combined with model, based on the flexibility and scalability of the sub-matrix decomposition method, combined with the deep learning model to predict the data.

2. A data prediction method based on real-time adaptive signal decomposition according to claim 1, characterized in that: In S1, a machine learning algorithm is used to train a parameter adjustment model. The specific steps are as follows: Data preprocessing: Extract useful features from the input signal. The features are numerical, categorical, or time series data. Use standardization, normalization, and encoding techniques to preprocess the features. The preprocessed features will serve as the input of the model, and the feature data will be normalized or standardized. Data segmentation: Divide the preprocessed dataset into a training set, a validation set, and a test set. The training set is used to train the model, the validation set is used to adjust model parameters (such as hyperparameters), and the test set is used to evaluate the final performance of the model. The test set should be kept independent of the training and validation sets to ensure the fairness and reliability of the evaluation results. Model training: Use the training set data to train the classification model. During the training process, the model will learn the mapping relationship between input features and target parameters, with high adaptability and accuracy; Model validation: Use validation set data to evaluate the performance of the model and select appropriate evaluation metrics to quantify the performance of the model, such as mean squared error, mean absolute error, and accuracy; Model optimization: Tune the model based on the validation results and use L1 and L2 regularization methods to prevent overfitting of the model; Model testing: Use the test set data to evaluate the final performance of the model, calculate the evaluation indicators on the test set to evaluate the generalization ability of the model, analyze the errors of the model on the test set, understand the weaknesses of the model and make improvements.

Images