Combined prediction method based on SVR and CNN-LSTM-ATTENTION model

Abstract

The invention discloses a combined prediction method based on an SVR model and a CNN-LSTM-ATTENTION model, and aims to solve the problems that a current single model is difficult to consider local features and long-term dependency relationships of data at the same time, and an overfitting risk may exist on small sample data. According to the technical scheme, a combined prediction system based on SVR and CNN-LSTM-ATTENTION models is constructed, a data preprocessing module receives N pieces of historical data from an input module and carries out data preprocessing on the N pieces of historical data, and a support vector regression module inputs training set and test set data to obtain first training set and test set prediction values; the CNN-LSTM-ATTENTION module inputs data of the training set and the test set to obtain predicted values of the second training set and the test set; the combined prediction module obtains a combined prediction value; and the output module transmits the test set combination prediction value to the display module for display. Compared with a single model, the method has the advantages that the precision of system prediction data is improved, and application requirements are well met.

Description

Technical Field

[0001] The present invention belongs to the field of data prediction, and specifically relates to a combined prediction method based on SVR (Support Vector Regression) and CNN-LSTM-ATTENTION model. Background Art

[0002] With the rapid development of information technology, data prediction plays an increasingly important role in all walks of life. Especially in the fields of finance, market analysis, supply chain management, etc., accurate data prediction can help companies make more informed decisions and improve operational efficiency and market competitiveness. However, traditional prediction methods often rely on simple statistical models or expert experience, which are often unable to cope with complex, nonlinear, and high-dimensional data. In recent years, with the continuous advancement of machine learning technology, especially the development of deep learning technology, it has brought revolutionary changes to the field of data prediction.

[0003] Deep learning, especially convolutional neural networks (CNN), long short-term memory networks (LSTM) and attention mechanisms, has achieved remarkable results in many fields such as image recognition, natural language processing, and speech recognition. These technologies can automatically learn complex feature representations from large amounts of data, thereby improving the accuracy of predictions. Support Vector Machine (SVM), as a powerful classification and regression tool, is widely used in various prediction tasks due to its superior performance in high-dimensional space.

[0004] Although a single model has achieved certain results in specific tasks, it is often difficult for a single model to take into account both the local characteristics and long-term dependencies of the data, and there may be a risk of overfitting on small sample data. Therefore, combining the advantages of multiple models to build a combined prediction model to improve the accuracy and generalization of predictions has become a research hotspot in the field of data prediction.

[0005] Based on this, the present invention proposes a combined prediction method based on SVR and CNN-LSTM-ATTENTION model, which aims to improve the accuracy and generalization ability of data prediction by integrating SVR's ability to handle small sample nonlinear relationships with CNN-LSTM-ATTENTION's advantage in capturing time series features. Through the present invention, a new technical means can be provided for the field of data prediction, helping enterprises and research institutions to better cope with complex and changing data environments and achieve more accurate predictions. Summary of the invention

[0006] The technical problem to be solved by the present invention is to provide a combined prediction method based on SVR and CNN-LSTM-ATTENTION model. This method integrates the ability of SVR to process nonlinear relationships of small samples and the advantages of CNN-LSTM-ATTENTION in capturing spatiotemporal features and attention mechanism through optimization method to perform combined prediction and obtain prediction results.

[0007] In order to solve the above technical problems, the technical solution of the present invention is: using a combined prediction method, combining the ideas of SVR and CNN-LSTM-ATTENTION, and constructing a combined prediction calculator to combine the prediction results of the two models, integrating the ability of SVR to handle small sample nonlinear relationships with CNN-LSTM-ATTENTION to capture spatiotemporal features, reducing the risk of overfitting, and improving the accuracy and generalization ability of the prediction model. In the data preprocessing stage, since the proper processing of missing points will directly affect the model prediction effect, the present invention fills in the missing values. Finally, the present invention allows prediction within the effective window, which effectively improves the accuracy while ensuring the generalization ability of the system.

[0008] The technical solution of the present invention is: The first step is to build a combined prediction system based on SVR and CNN-LSTM-ATTENTION model. The method is: The combined prediction system based on SVR and CNN-LSTM-ATTENTION model consists of a database, an input module, a data preprocessing module, a support vector regression module, a CNN-LSTM-ATTENTION module, a combined prediction module, an output module, and a display module.

[0009] The database is connected to the input module. The database stores N historical data. Each data is represented by a triple (Time, Label, Value). Time represents time, which is arranged in ascending order. Label represents the label. Value is the actual observation value, which represents the value of the corresponding label at the corresponding time. If Value = NaN (i.e., null value), it means that the point is a missing point.

[0010] The input module is connected to the database and the data preprocessing module. The input module extracts N historical data from the database and sends the N historical data to the data preprocessing module. The data preprocessing module is connected to the input module, the support vector regression module, and the CNN-LSTM-ATTENTION module. The data preprocessing module includes a data filling module and a data standardization module. The data filling module of the data preprocessing module receives N historical data from the input module and detects the N historical data. If there are missing values, the data filling module fills the missing values ​​and sends the N historical data after filling the missing values ​​to the data standardization module of the data preprocessing module. The data standardization module of the data preprocessing module standardizes the N historical data after filling the missing values ​​to obtain the standardized N historical data, and then sends the standardized N historical data to the support vector regression module and the CNN-LSTM-ATTENTION module respectively.

[0011] The support vector regression module is connected to the combined prediction module and the data standardization module of the data preprocessing module. The support vector regression module receives the standardized N historical data from the data standardization module of the data preprocessing module, and divides the data into the training set and the test set of the SVR model. Then, the standardized N historical data are mapped to the high-dimensional feature space through the kernel function, and the originally linearly inseparable data are transformed into linearly separable data. The constructed optimization problem is solved to determine the support vector. The support vector regression module uses the five-fold cross-validation method to solve the optimal parameters of the SVR model, and the SVR model is trained using the training set to obtain a trained SVR model. The support vector regression module predicts the training set data and the test set data with the trained SVR model to obtain the first set of training set prediction values ​​and the first set of test set prediction values, and then sends the first set of training set prediction values ​​and the first set of test set prediction values ​​to the combined prediction module.

[0012] The CNN-LSTM-ATTENTION module is combined with the prediction module and the data standardization module of the data preprocessing module. The CNN-LSTM-ATTENTION module receives the standardized N historical data from the data standardization module of the data preprocessing module, and divides the data into the training set and test set of the CNN-LSTM-ATTENTION model. The CNN-LSTM-ATTENTION model consists of an input layer, a hidden layer, and a fully connected layer, where the hidden layer consists of a convolutional layer, a pooling layer, an LSTM layer, and a self-attention mechanism layer. The input layer of the CNN-LSTM-ATTENTION model receives the training set; the CNN layer convolutional neural network extracts data spatial features from the training set, the pooling layer downsamples the extracted data spatial features, and then outputs the sampled data spatial features to the LSTM layer; the LSTM layer recurrent neural network receives the data spatial features sampled by the CNN layer convolutional neural network, the LSTM layer recurrent neural network captures the dependency of the data time series features from the training set, and finally outputs the training set data spatiotemporal features (data time series features and data spatial features) to the ATTENTION self-attention mechanism layer; the ATTENTION self-attention mechanism layer calculates the score result and inputs the score result to the fully connected layer of the CNN-LSTM-ATTENTION module; the fully connected layer integrates the data spatiotemporal features extracted by the hidden layer to obtain a trained CNN-LSTM-ATTENTION model; the CNN-LSTM-ATTENTION model inputs the training set and test set data respectively, outputs the second training set prediction value y_train_pred2 and the second test set prediction value y_test_pred2 respectively, and then sends the second group of training set prediction values ​​and the second group of test set prediction values ​​to the combined prediction module.

[0013] The combined prediction module is connected to the support vector regression module and the CNN-LSTM-ATTENTION module. The combined prediction module obtains the first set of training set prediction values ​​and the first set of test set prediction values ​​from the support vector regression module, and obtains the second set of training set prediction values ​​and the second set of test set prediction values ​​from the CNN-LSTM-ATTENTION module; then, the optimal weight is solved based on the error between the first set of training set prediction values, the second set of training set prediction values, and the true observation values. and ; Finally, according to the weight and The first group of test set prediction values ​​and the second group of test set prediction values ​​are calculated by combining the combined weights to obtain the test set combined prediction values.

[0014] The output module is connected to the combination prediction module. The output module obtains the test set combination prediction value from the combination prediction module and transmits the test set combination prediction value to the display module.

[0015] The display module is connected to the output module. The display module receives the test set combination prediction value from the output module and displays the test set combination prediction value.

[0016] In the second step, the input module extracts N historical data from the database and sends them to the data preprocessing module; In the third step, the data preprocessing module receives N historical data from the input module and performs data preprocessing on the N historical data. The method is: 3.1 The data filling module of the data preprocessing module receives N historical data from the input module. The N historical data are represented by a triple (Time, Label, Value). Time represents time and is arranged in ascending order of time; Label represents the label; Value represents the value of the corresponding label at the corresponding time; if Value NaN, NaN is a null value, which means that the point is not missing. If Value=NaN, it means that the point is missing. 3.2 The data filling module of the data preprocessing module determines whether the VALUE values ​​in the N data are continuous or the boundary data are missing. If yes, go to step 3.3; if not, go to step 3.4.

[0017] 3.3 If the VALUE values ​​in N data are missing continuously or the boundary data are missing, the data preprocessing module uses the polynomial fitting method to fill the missing values. The method is: The fitting data selected for the corresponding variables are the adjacent Time. .in, , , , , is a constant, x is a variable, and the degree of the polynomial is the x The highest power of n express.

[0018] The present invention adopts the least square method to fit the polynomial, such as formula 1 As shown. Where m is the number of data points used, It is i The observed value of the dependent variable for each data point, is the predicted value of the dependent variable for the ith data point. Use formula 1 to calculate the missing value and go to step 3.4.

[0019] 3.4 The data filling module of the data preprocessing module determines whether the middle value of the VALUE value in the N data is missing. If so, go to step 3.5. If it is filled, go to step 3.6.

[0020] 3.5 If the middle value of N data is missing, the data preprocessing module uses the data filling module Lagrange interpolation method to fill the missing data. The method is: When there are n+1 data points and each variable x is different, the Lagrangian basis function is defined as in Formula 2 As shown, is the i-th Lagrangian basis function, n is the number of interpolation points, and are the coordinates of the interpolation points, where is the index of the current basis function, is the index of other basis functions. The Lagrange interpolation polynomial is expressed as a linear combination of all basis functions and corresponding function values ​​as shown in Formula 3 As shown, The function value of the ith data point.

[0021] Use Formula 2 and Formula 3 to calculate the missing values ​​for filling, send the N historical data after filling the missing values ​​to the data standardization module of the data preprocessing module, and go to step 3.6.

[0022] 3.6 The data standardization module of the data preprocessing module performs min-max normalization on the N historical data after filling in the missing values ​​to obtain the standardized N historical data, namely the standardized value Value_norm, and transmits the standardized value Value_norm to the support vector regression module, and goes to the fourth step. At the same time, the standardized value Value_norm is transmitted to the CNN-LSTM-ATTENTION module, and goes to the fifth step.

[0023] In the fourth step, the support vector regression module builds an SVR model, then inputs the training set to train the SVR model to obtain a trained SVR model, and finally inputs the training set and test set data to the trained SVR model to obtain the first training set prediction value and the first test set prediction value. The method is: 4.1 The support vector regression module receives the standardized N historical data from the data standardization module of the data preprocessing module; 4.1.1 According to the triples of the data, the standardized value Value_norm is divided into the training set of the SVR model and the test set of the SVR model.

[0024] 4.2 Support Vector Regression Module The SVR model is constructed by: 4.2.1 Support vector regression module creates support vector regression objects, high-dimensional regression models such as formula As shown, is the weight vector, T refers to the transpose, It is a kernel function that maps the input space to a high-dimensional feature space. The kernel function used in the present invention is a Gaussian kernel function. is the bias term, m is the number of support vectors, and is the Lagrange multiplier, is the support vector, It is a kernel function used to calculate the inner product of two data points in high-dimensional space.

[0025] 4.2.2 The optimization goal of SVR is to minimize the complexity of the model while ensuring that the error is within Points outside the error band are appropriately penalized, and C controls the degree of penalty for slack variables. The optimal parameters are selected using a five-fold cross-validation method, which is: 4.2.2.1 The dataset is divided into five equal parts, four for training sets and one for testing sets 4.2.2.2 Set three to be selected value, The value is 0.1, The value is 0.2, The value is 0.3. Four C values ​​to be selected are set. The value is 10, The value is 100, The value is 1000, The value is 10000. Coefficients of the Gaussian kernel function Denoted as g, five g values ​​to be selected are set, among which The value is 0.01, The value is 0.001, The value is 0.0001, , , 'scale' is 1 / (n_features * var), where n_features is the number of features and var is the variance of the features. 'auto' is 1 / n_features.

[0026] The SVR model was trained using different parameter combinations, one of the subsets was selected as the test set, the other four subsets were selected as the training set, the model was trained and evaluated on the test set. This process was repeated five times to ensure that each subset was used as a test set once.

[0027] 4.2.2.3 After the five cross-validations are completed, the average prediction error index is calculated as the evaluation criterion for the model performance and the optimal parameter combination is selected.

[0028] 4.2.3 SVR model Select the optimal parameters, input the training set to train the SVR model, and obtain the trained SVR model.

[0029] 4.3 The support vector regression module inputs the training set data and the test set data into the trained SVR model to predict and obtain the first set of training set prediction values ​​y_train_pred1 and the first set of test set prediction values ​​y_test_pred1.

[0030] 4.4 The support vector regression module sends the first set of training set prediction values ​​y_train_pred1 and the first set of test set prediction values ​​y_test_pred1 to the combined prediction module and goes to step 6.1.

[0031] In the fifth step, the CNN-LSTM-ATTENTION module constructs a CNN-LSTM-ATTENTION neural network model, and then inputs the training set to train the CNN-LSTM-ATTENTION neural network to obtain a trained CNN-LSTM-ATTENTION neural network model. The training set and test set data are input to the trained CNN-LSTM-ATTENTION neural network model to obtain the second training set prediction value and the second test set prediction value. The method is: 5.1 The CNN-LSTM-ATTENTION module receives the standardized N historical data from the data standardization module of the data preprocessing module, sets the time step according to the triple Time, and divides the standardized N historical data into the training set and test set of the CNN-LSTM-ATTENTION model.

[0032] 5.2 CNN-LSTM-ATTENTION module Construct the CNN-LSTM-ATTENTION network model by: 5.2.1 CNN-LSTM-ATTENTION module The CNN-LSTM-ATTENTION network model is constructed. The first layer is the input layer, and the number of neurons is the number of features n_feature, that is, the data dimension. The second layer is the hidden layer. The first layer of the hidden layer is a one-dimensional convolution layer to extract spatial data features; the second layer of the hidden layer is a pooling layer to downsample the features, and the third layer of the hidden layer is an LSTM layer for extracting time series features; the fourth layer of the hidden layer is a self-attention mechanism layer. The present invention uses a variant of the additive attention mechanism and introduces a nonlinear activation function (tanh) after the dot product operation to help the model capture and focus on important information more flexibly when processing the input sequence. The third layer is the output layer, that is, the fully connected layer. The fully connected layer has two layers, the first layer has 32 neurons, and the second layer has 1 neuron. The loss function used in the present invention is huber to enhance the robustness of the system. Epoch is set to 150, batch_size is set to 8, and the learning rate is 0.04.

[0033] 5.2 CNN-LSTM-ATTENTION module Input the training set to the CNN-LSTM-ATTENTION model for training to obtain the trained CNN-LSTM-ATTENTION model. The method is: 5.2.1 The CNN neural network model uses a one-dimensional convolution layer to perform convolution operations on the time series data of the training set to extract the spatial features of the training set data, and downsamples the convolved features through the pooling layer to reduce the data dimension and extract key information.

[0034] 5.2.2 The CNN neural network model outputs the spatial features of the training set data to the LSTM recurrent neural network; 5.2.3 LSTM recurrent neural network obtains the spatial features of training set data from CNN convolutional neural network; 5.2.4 LSTM Recurrent Neural Network The gating mechanism of LSTM (including input gate, forget gate and output gate) is used to capture the long-term dependencies of the time series of the training set data. LSTM can remember long-term information while forgetting unimportant historical information, thus better understanding the temporal dynamic characteristics of the data.

[0035] 5.2.5 The LSTM recurrent neural network outputs the spatiotemporal features (time series features and spatial features) of the training set data to the ATTENTION self-attention mechanism.

[0036] 5.2.6 ATTENTION The self-attention mechanism calculates the importance weights, performs weighted summation on the output of LSTM, and obtains the calculated score result.

[0037] 5.2.7 ATTENTION The scoring result of the self-attention mechanism is output to the fully connected layer.

[0038] 5.2.8 Perform back propagation and adjust weight parameters.

[0039] 5.2.9 Iterate all epochs and get the trained model after completion.

[0040] 5.3 CNN-LSTM-ATTENTION module inputs the training set data and the test set data into the trained CNN-LSTM-ATTENTION model respectively, and predicts the second training set prediction value y_train_pred2 and the second test set prediction value y_test_pred2, the method is: 5.3.1 The input layer receives the training set and the test set; 5.3.2 The CNN neural network model uses a one-dimensional convolution layer to perform convolution operations on the time series data of the training set and the test set to extract the spatial features of the training set and the test set data. The convolutional features are downsampled through the pooling layer to reduce the data dimension and extract key information.

[0041] 5.3.3 The CNN neural network model outputs the spatial features of the training set and test set data to the LSTM recurrent neural network; 5.3.4 LSTM Recurrent Neural Network The gating mechanism of LSTM (including input gate, forget gate and output gate) is used to capture the long-term dependencies of the time series of training set and test set data.

[0042] 5.3.5 The LSTM recurrent neural network outputs the spatiotemporal features (time series features and spatial features) of the training set and test set data to the ATTENTION self-attention mechanism.

[0043] 5.3.6 ATTENTION The self-attention mechanism calculates the importance weights, performs weighted summation on the output of LSTM, and obtains the calculated score result.

[0044] 5.3.7 ATTENTION The scoring result of the self-attention mechanism is output to the fully connected layer.

[0045] 5.3.8 The fully connected layer integrates the features extracted by the hidden layer, outputs the second training set prediction value y_train_pred2 and the second test set prediction value y_test_pred2, and goes to step 6.1.

[0046] In the sixth step, the combined prediction module obtains the combined prediction value by: 6.1 The combined prediction module obtains the first set of training set prediction values ​​y_train_pred1 from the support vector regression module, and also obtains the second set of training set prediction values ​​y_train_pred2 from the CNN-LSTM-ATTENTION module.

[0047] 6.2 The combined prediction module solves the optimal weight according to the error of the first set of training set prediction values, the second set of training set prediction values, and the true observation value Value and , the method is: 6.2.1 Weighted prediction value of combined prediction module The actual observed value The absolute error between them is minimized, and a set of optimal weights is found and .

[0048] 6.2.2 Combination prediction module defines absolute loss function , as shown in the formula As shown. Among them, n refers to the number of observations, , and They are observations, the i-th prediction of the SVR model, and the i-th prediction of the CNN-LSTM-ATTENTION model.

[0049] 6.3 Combining prediction modules according to weights and And the first set of test set predictions And the second set of test set prediction values Calculate the combined forecast value , the method is: Final prediction value

[0050] In the seventh step, the output module obtains the test set combined prediction value from the combined prediction module and transmits the test set combined prediction value to the display module.

[0051] In the eighth step, the display module receives the test set combination prediction value from the output module and displays the test set combination prediction value.

[0052] The following technical effects can be achieved by adopting the present invention: 1. In the second step of the present invention, a data preprocessing module is constructed. The Lagrange interpolation module and the polynomial fitting module improve the validity of the data. The data standardization module eliminates the dimensions between the data, accelerates the prediction convergence speed, and reduces the use requirements of this method.

[0053] 2. In the fourth, fifth and sixth steps of the present invention, a support vector regression module, a CNN-LSTM-ATTENTION module and a prediction result combination calculator are designed. The combined prediction model of SVR and CNN-LSTM-ATTENTION integrates the advantages of SVR being good at processing nonlinear data and CNN-LSTM-ATTENTION being good at capturing temporal relationships. Compared with a single model, the generalization ability and robustness of the system are greatly improved, and data can be predicted with higher accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 It is the overall flow chart of the present invention.

[0055] Figure 2 A logical structure diagram of a combined prediction system based on SVR and CNN-LSTM-ATTENTION model constructed in the first step of the present invention.

[0056] Figure 3 This is a structural diagram of the CNN-LSTM-ATTENTION network model constructed in step 5.2 of the present invention.

[0057] Figure 4 This is a comparison chart of the prediction effects of the combined prediction system model of the present invention and the traditional single model. DETAILED DESCRIPTION

[0058] Figure 1 It is the overall flow chart of the present invention. Figure 1 As shown, the present invention comprises the following steps: The first step is to build a combined prediction system based on SVR and CNN-LSTM-ATTENTION model. Figure 2 As shown in the figure, the combined prediction system based on SVR and CNN-LSTM-ATTENTION model consists of a database, an input module, a data preprocessing module, a support vector regression module, a CNN-LSTM-ATTENTION module, a combined prediction module, an output module, and a display module.

[0059] The database is connected to the input module. The database stores grain production data. Each data is represented by a triple (Time, Label, Value). Time represents time, which is arranged in ascending order. Label represents the label. Value is the actual observation value, which represents the value of the corresponding label at the corresponding time. If Value = NaN (i.e., null value), it means that the point is a missing point.

[0060] The input module is connected to the database and the data preprocessing module. The input module extracts the grain production data of Hebei Province from 1990 to 2021 from the database and sends the grain production data to the data preprocessing module. The data preprocessing module is connected to the input module, the support vector regression module, and the CNN-LSTM-ATTENTION module. The data preprocessing module includes a data filling module and a data standardization module. The data filling module of the data preprocessing module receives the grain production data from the input module and detects the grain production data. If there are missing values, the data filling module fills the missing values ​​and sends the grain production data after filling the missing values ​​to the data standardization module of the data preprocessing module. The data standardization module of the data preprocessing module standardizes the grain production data after filling the missing values ​​to obtain the standardized grain production data, and then sends the standardized grain production data to the support vector regression module and the CNN-LSTM-ATTENTION module respectively.

[0061] The support vector regression module is connected to the combined prediction module and the data standardization module of the data preprocessing module. The support vector regression module receives the standardized grain production data from the data standardization module of the data preprocessing module and divides it into a training set and a test set. Then, the standardized grain production data is mapped to a high-dimensional feature space through a kernel function, and the originally linearly inseparable data is transformed into a linearly separable one. The constructed optimization problem is solved to determine the support vector. The support vector regression module uses a five-fold cross-validation method to solve the optimal parameters of the SVR model to be selected, and the SVR model is trained using the training set to obtain a trained SVR model. The support vector regression module predicts the training set data and the test set data with the trained SVR model to obtain the first set of training set prediction values ​​y_train_pred1 and the first set of test set prediction values ​​y_test_pred1, and then sends the first set of training set prediction values ​​and the first set of test set prediction values ​​to the combined prediction module.

[0062] The CNN-LSTM-ATTENTION module is connected to the combined prediction module, the data standardization module of the data preprocessing module, and the particle swarm module. The CNN-LSTM-ATTENTION module receives the standardized grain yield data from the data standardization module of the data preprocessing module and divides it into training set and test set.

[0063] The CNN-LSTM-ATTENTION model consists of an input layer, a hidden layer, and a fully connected layer, where the hidden layer consists of a convolutional layer, a pooling layer, an LSTM layer, and a self-attention mechanism layer. The input layer of the CNN-LSTM-ATTENTION model receives the training set, the CNN convolutional neural network extracts data spatial features from the training set, the pooling layer downsamples the extracted data spatial features, and then outputs the sampled data spatial features to the LSTM; the LSTM recurrent neural network receives the data spatial features sampled by the CNN convolutional neural network, the LSTM recurrent neural network captures the dependencies of the data time series features from the training set, and finally outputs the training set data spatiotemporal features (data time series features and data spatial features) to the ATTENTION self-attention mechanism; the ATTENTION self-attention mechanism calculates the score result and inputs the score result to the fully connected layer of the CNN-LSTM-ATTENTION module; the fully connected layer integrates the data spatiotemporal features extracted by the hidden layer to obtain a trained CNN-LSTM-ATTENTION model; the CNN-LSTM-ATTENTION model inputs the training set and test set data respectively, outputs the second training set prediction value y_train_pred2 and the second test set prediction value y_test_pred2 respectively, and then sends the second group of training set prediction values ​​and the second group of test set prediction values ​​to the combined prediction module.

[0064] The combined prediction module is connected to the support vector regression module and the CNN-LSTM-ATTENTION module. The combined prediction module obtains the first set of training set prediction values ​​and the first set of test set prediction values ​​from the support vector regression module, and obtains the second set of training set prediction values ​​and the second set of test set prediction values ​​from the CNN-LSTM-ATTENTION module; then, the optimal weight is solved based on the error between the first set of training set prediction values, the second set of training set prediction values, and the true observation values. and ; Finally, according to the weight and The predicted values ​​of the first test set and the predicted values ​​of the second test set are calculated by combining the combined weights to obtain a combined predicted value.

[0065] The output module is connected to the combined prediction module, obtains the combined prediction value from the combined prediction module, and transmits the combined prediction value to the display module.

[0066] The display module is connected to the output module. The display module receives the combined prediction value of the output module and displays the combined prediction value of the test set.

[0067] In the second step, the input module extracts grain production data from the database and sends it to the data preprocessing module; In the third step, the data preprocessing module receives the grain production data from the input module and performs data preprocessing on the grain production data. The method is: 3.1 The data filling module of the data preprocessing module receives the grain production data from the input module. The grain production data is represented by a triple (Time, Label, Value), where Time represents time and is arranged in ascending order; Label represents the label; Value represents the value of the corresponding label at the corresponding time; if Value NaN, NaN is a null value, which means that the point is not missing. If Value=NaN, it means that the point is missing. 3.2 The data filling module of the data preprocessing module determines whether the VALUE value in the grain production data is continuous or the boundary data is missing. If yes, go to step 3.3; if not, go to step 3.4.

[0068] 3.3 If the VALUE values ​​in the grain production data are missing continuously or the boundary data are missing, the data preprocessing module uses the polynomial fitting method to fill the missing values. The method is: The fitting data selected for the corresponding variables are the adjacent Time. .in, , , , , is a constant, x is a variable, and the degree of the polynomial is the x The highest power of n express.

[0069] The present invention adopts the least square method to fit the polynomial, such as formula 1 As shown. Where m is the number of data points used, It is i The observed value of the dependent variable for each data point, is the predicted value of the dependent variable for the ith data point. Use formula 1 to calculate the missing value and go to step 3.4.

[0070] 3.4 The data filling module of the data preprocessing module determines whether the intermediate value of the VALUE value in the grain production data is missing. If so, go to step 3.5. If it is filled, go to step 3.6.

[0071] 3.5 If the intermediate value of grain production data is missing, the data preprocessing module uses the data filling module Lagrange interpolation method to fill the missing data. The method is: When there are n+1 data points and each variable x is different, the Lagrangian basis function is defined as in Formula 2 As shown, is the i-th Lagrangian basis function, n is the number of interpolation points, and are the coordinates of the interpolation points, where is the index of the current basis function, is the index of other basis functions. The Lagrange interpolation polynomial is expressed as a linear combination of all basis functions and corresponding function values ​​as shown in Formula 3 As shown, The function value of the ith data point.

[0072] Use Formula 2 and Formula 3 to calculate the missing values ​​for filling, send the grain yield data after filling the missing values ​​to the data standardization module of the data preprocessing module, and go to step 3.6.

[0073] 3.6 The data standardization module of the data preprocessing module performs min-max normalization on the grain yield data after filling in the missing values ​​to obtain the standardized grain yield data, namely the standardized value Value_norm, and transmits the standardized value Value_norm to the support vector regression module, and goes to the fourth step. At the same time, the standardized value Value_norm is transmitted to the CNN-LSTM-ATTENTION module, and goes to the fifth step.

[0074] In the fourth step, the support vector regression module builds an SVR model, then inputs the training set to train the SVR model to obtain a trained SVR model, and finally inputs the training set and test set data to the trained SVR model to obtain the first training set prediction value and the first test set prediction value. The method is: 4.1 The support vector regression module receives the standardized grain yield data from the data standardization module of the data preprocessing module; 4.1.1 According to the triples of the data, the standardized value Value_norm is divided into the training set of the SVR model and the test set of the SVR model.

[0075] 4.2 Support Vector Regression Module The SVR model is constructed by: 4.2.1 Support vector regression module creates support vector regression objects, high-dimensional regression models such as formula As shown, is the weight vector, T refers to the transpose, It is a kernel function that maps the input space to a high-dimensional feature space. The kernel function used in the present invention is a Gaussian kernel function. is the bias term, m is the number of support vectors, and is the Lagrange multiplier, is the support vector, It is a kernel function used to calculate the inner product of two data points in high-dimensional space.

[0076] 4.2.2 The optimization goal of SVR is to minimize the complexity of the model while ensuring that the error is within Points outside the error band are appropriately penalized, and C controls the degree of penalty for slack variables. The optimal parameters are selected using a five-fold cross-validation method, which is: 4.2.2.1 The dataset is divided into five equal parts, four for training sets and one for testing sets.

[0077] 4.2.2.2 Set three to be selected value, The value is 0.1, The value is 0.2, The value is 0.3. Four C values ​​to be selected are set. The value is 10, The value is 100, The value is 1000, The value is 10000. Coefficients of the Gaussian kernel function Denoted as g, five g values ​​to be selected are set, among which The value is 0.01, The value is 0.001, The value is 0.0001, , , 'scale' is 1 / (n_features * var), where n_features is the number of features and var is the variance of the features. 'auto' is 1 / n_features.

[0078] The SVR model was trained using different parameter combinations, one of the subsets was selected as the test set, the other four subsets were selected as the training set, the model was trained and evaluated on the test set. This process was repeated five times to ensure that each subset was used as a test set once.

[0079] 4.2.2.3 After the five cross-validations are completed, the average prediction error index is calculated as the evaluation criterion for the model performance. The optimal parameter combination is selected.

[0080] 4.2.3 SVR model Select the optimal parameters, input the training set to train the SVR model, and obtain the trained SVR model.

[0081] 4.3 The support vector regression module inputs the training set data and the test set data into the trained SVR model to predict and obtain the first set of training set prediction values ​​y_train_pred1 and the first set of test set prediction values ​​y_test_pred1.

[0082] 4.4 The support vector regression module sends the first set of training set prediction values ​​and the first set of test set prediction values ​​to the combined prediction module and goes to step 6.1.

[0083] In the fifth step, the CNN-LSTM-ATTENTION module constructs a CNN-LSTM-ATTENTION neural network model, and then inputs the training set to train the CNN-LSTM-ATTENTION neural network to obtain a trained CNN-LSTM-ATTENTION neural network model. The training set and test set data are input to obtain the second training set prediction value and the second test set prediction value. The method is: 5.1 The CNN-LSTM-ATTENTION module receives the standardized grain yield data from the data standardization module of the data preprocessing module, sets the time step according to the triple Time, and divides the standardized grain yield data into the training set and test set of the CNN-LSTM-ATTENTION model.

[0084] Figure 3 This is a structural diagram of the CNN-LSTM-ATTENTION network model constructed in step 5.2 of the present invention. For example, step 5.2 is: 5.2 CNN-LSTM-ATTENTION module Construct the CNN-LSTM-ATTENTION network model by: 5.2.1 CNN-LSTM-ATTENTION module Construct the CNN-LSTM-ATTENTION network model, such as Figure 3 As shown, the first layer is the input layer, and the number of neurons is the number of features n_feature, that is, the data dimension. The second layer is the hidden layer, and the first layer of the hidden layer is a one-dimensional convolution layer to extract spatial data features; the second layer of the hidden layer is a pooling layer to downsample the features, and the third layer of the hidden layer is an LSTM layer for extracting time series features; the fourth layer of the hidden layer is a self-attention mechanism layer. The present invention uses a variant of the additive attention mechanism, and introduces a nonlinear activation function (tanh) after the dot product operation to help the model capture and focus on important information more flexibly when processing the input sequence. The third layer is the output layer, that is, the fully connected layer. The fully connected layer has two layers, the first layer has 32 neurons, and the second layer has 1 neuron. The loss function used in the present invention is huber to enhance the robustness of the system. Epoch is set to 150, batch_size is set to 8, and the learning rate is 0.04.

[0085] 5.2 CNN-LSTM-ATTENTION module Input the training set to the CNN-LSTM-ATTENTION model for training to obtain the trained CNN-LSTM-ATTENTION model. The method is: 5.2.1 The CNN neural network model uses a one-dimensional convolution layer to perform convolution operations on the time series data of the training set to extract the spatial features of the training set data. The convolutional features are downsampled through the pooling layer to reduce the data dimension and extract key information.

[0086] 5.2.2 The CNN neural network model outputs the spatial features of the training set data to the LSTM recurrent neural network; 5.2.3 LSTM recurrent neural network obtains the spatial features of training set data from CNN convolutional neural network; 5.2.4 LSTM Recurrent Neural Network The gating mechanism of LSTM (including input gate, forget gate and output gate) is used to capture the long-term dependencies of the time series of training data. LSTM can remember long-term information while forgetting unimportant historical information, so as to better understand the temporal dynamic characteristics of data.

[0087] 5.2.5 The LSTM recurrent neural network outputs the spatiotemporal features (time series features and spatial features) of the training set data to the ATTENTION self-attention mechanism.

[0088] 5.2.6 ATTENTION The self-attention mechanism calculates the importance weights, performs weighted summation on the output of LSTM, and obtains the calculated score result.

[0089] 5.2.7 ATTENTION The score result of the self-attention mechanism is output to the fully connected layer. 5.2.8 Perform back propagation and adjust weight parameters.

[0090] 5.2.9 Iterate all epochs and get the trained model after completion.

[0091] 5.3 CNN-LSTM-ATTENTION module inputs the training set data and the test set data into the trained CNN-LSTM-ATTENTION model respectively, and predicts the second training set prediction value y_train_pred2 and the second test set prediction value y_test_pred2, the method is: 5.3.1 The input layer receives the training set and the test set; 5.3.2 The CNN neural network model uses a one-dimensional convolution layer to perform convolution operations on the time series data of the training set and the test set to extract the spatial features of the training set and the test set data. The convolutional features are downsampled through the pooling layer to reduce the data dimension and extract key information.

[0092] 5.3.3 The CNN neural network model outputs the spatial features of the training set and test set data to the LSTM recurrent neural network; 5.3.4 LSTM Recurrent Neural Network The gating mechanism of LSTM (including input gate, forget gate and output gate) is used to capture the long-term dependencies of the time series of training set and test set data.

[0093] 5.3.5 The LSTM recurrent neural network outputs the spatiotemporal features (time series features and spatial features) of the training set and test set data to the ATTENTION self-attention mechanism.

[0094] 5.3.6 ATTENTION The self-attention mechanism calculates the importance weights, performs weighted summation on the output of LSTM, and obtains the calculated score result.

[0095] 5.3.7 ATTENTION The scoring result of the self-attention mechanism is output to the fully connected layer.

[0096] 5.3.8 The fully connected layer integrates the features extracted by the hidden layer, outputs the second training set prediction value y_train_pred2 and the second test set prediction value y_test_pred2, and goes to step 6.1.

[0097] In the sixth step, the combined prediction module obtains the combined prediction value by: 6.1 The combined prediction module obtains the first set of training set prediction values ​​y_train_pred1 from the support vector regression module, and also obtains the second set of training set prediction values ​​y_train_pred2 from the CNN-LSTM-ATTENTION module.

[0098] 6.2 The combined prediction module solves the optimal weight according to the error of the first set of training set prediction values, the second set of training set prediction values, and the true observation value Value and , the method is: 6.2.1 Weighted prediction value of combined prediction module The actual observed value The absolute error between them is minimized, and a set of optimal weights is found and .

[0099] 6.2.2 Combination prediction module defines absolute loss function , as shown in the formula As shown. Among them, n refers to the number of observations, , and They are observations, the i-th prediction of the SVR model, and the i-th prediction of the CNN-LSTM-ATTENTION model.

[0100] 6.3 Combining prediction modules according to weights and And the first set of test set predictions And the second set of test set prediction values Calculate the combined forecast value , the method is: Final prediction value

[0101] In the seventh step, the output module obtains the combined prediction value from the combined prediction module and transmits the combined prediction value to the display module.

[0102] In the eighth step, the display module receives the combined prediction value of the output module and displays the combined prediction value of the test set.

[0103] Figure 4 The following is a comparison chart of the prediction effects of three prediction models. The present invention is compared with a single method using the SVR model and the CNN-LSTM-ATTENTION model to verify that the present invention has higher precision prediction data. The actual observation values ​​of the test set are 2017: 3829.25, 2018: 3700.86, 2019: 3739.24, 2020: 3795.89, 2021: 3825.09. The predicted values ​​of the SVR method are 2017: 3631.21, 2018: 3737.93, 2019: 3733.86, 2020: 3751.11, 2021: 3766.43. The prediction values ​​of the CNN-LSTM-ATTENTION method are 3577.56 in 2017, 3597.85 in 2018, 3672.49 in 2019, 3780.17 in 2020, and 3877.53 in 2021. The combined prediction values ​​are 3615.59 in 2017, 3679.52 in 2018, 3715.63 in 2019, 3778.10 in 2020, and 3834.09 in 2021. The mean absolute percentage error (MAPE) of the combined prediction is only 1.498%, which is 17% and 42% higher than that of the single SVR model and CNN-LSTM-ATTENTION model, respectively.

[0104] The above is a detailed introduction to the combined prediction method based on SVR and CNN-LSTM-ATTENTION model provided by the present invention. This article explains the principles and implementation methods of the present invention, and the above description is used to help understand the core idea of ​​the present invention. It should be pointed out that for ordinary researchers in this technical field, without departing from the principles of the present invention, several improvements and modifications can be made to the present invention, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.

Claims

1. A combined prediction method based on SVR and CNN-LSTM-ATTENTION model, characterized in that The following steps are involved: The technical solution of the present invention is: The first step is to build a combined prediction system based on SVR and CNN-LSTM-ATTENTION model. The method is: The combined prediction system based on SVR and CNN-LSTM-ATTENTION model consists of database, input module, data preprocessing module, support vector regression module, CNN-LSTM-ATTENTION module, combined prediction module, output module and display module; The database is connected to the input module. The database stores N historical data. Each data is represented by a triple (Time, Label, Value). Time represents time, which is arranged in ascending order of time. Label represents label. Value is the real observation value, which represents the value of the corresponding label at the corresponding time. If Value = NaN, NaN is a null value, which means that the point is a missing point. The input module is connected to the database and the data preprocessing module; the input module extracts N pieces of historical data from the database and sends the N pieces of historical data to the data preprocessing module; The data preprocessing module is connected with the input module, the support vector regression module, and the CNN-LSTM-ATTENTION module; the data preprocessing module includes a data filling module and a data standardization module; The data filling module of the data preprocessing module receives N pieces of historical data from the input module, and detects the N pieces of historical data. If there are missing values, the data filling module fills in the missing values, and sends the N pieces of historical data after the missing values ​​are filled to the data standardization module of the data preprocessing module; The data standardization module of the data preprocessing module standardizes the N historical data after filling the missing values ​​to obtain the standardized N historical data, and then sends the standardized N historical data to the support vector regression module and the CNN-LSTM-ATTENTION module respectively; The support vector regression module is connected to the combined prediction module and the data standardization module of the data preprocessing module; the support vector regression module receives the standardized N historical data from the data standardization module of the data preprocessing module, and divides the data into a training set and a test set of the SVR model; then the standardized N historical data are mapped to a high-dimensional feature space through a kernel function, and the originally linearly inseparable data are transformed into linearly separable data, and the constructed optimization problem is solved to determine the support vector. The support vector regression module uses a five-fold cross-validation method to solve the optimal parameters to be selected for the SVR model, and the SVR model is trained using the training set to obtain a trained SVR model; the support vector regression module predicts the training set data and the test set data using the trained SVR model to obtain a first set of training set prediction values ​​and a first set of test set prediction values, and then sends the first set of training set prediction values ​​and the first set of test set prediction values ​​to the combined prediction module; The CNN-LSTM-ATTENTION module is combined with the prediction module and the data standardization module of the data preprocessing module; the CNN-LSTM-ATTENTION module receives the standardized N historical data from the data standardization module of the data preprocessing module, and divides the data into the training set and the test set of the CNN-LSTM-ATTENTION model; the CNN-LSTM-ATTENTION model is composed of an input layer, a hidden layer, and a fully connected layer, wherein the hidden layer is composed of a convolutional layer, a pooling layer, an LSTM layer, and a self-attention mechanism layer; the input layer of the CNN-LSTM-ATTENTION model receives the training set; the CNN layer convolutional neural network extracts data spatial features from the training set, the pooling layer downsamples the extracted data spatial features, and then outputs the sampled data spatial features to the LSTM layer; the LSTM layer recurrent neural network receives the data spatial features sampled by the CNN layer convolutional neural network, and the LSTM layer recurrent neural network captures the dependency of the data time series features from the training set, and finally outputs the training set data spatiotemporal features, i.e., the data time series features and the data spatial features, to the ATTENTION self-attention mechanism layer; The ATTENTION self-attention mechanism layer calculates the score result and inputs the score result to the fully connected layer of the CNN-LSTM-ATTENTION module; The fully connected layer integrates the spatiotemporal features of the data extracted by the hidden layer to obtain a trained CNN-LSTM-ATTENTION model; the CNN-LSTM-ATTENTION model inputs the training set and test set data respectively, outputs the second training set prediction value y_train_pred2 and the second test set prediction value y_test_pred2 respectively, and then sends the second set of training set prediction values ​​and the second set of test set prediction values ​​to the combined prediction module; The combined prediction module is connected to the support vector regression module and the CNN-LSTM-ATTENTION module; the combined prediction module obtains the first set of training set prediction values ​​and the first set of test set prediction values ​​from the support vector regression module, and obtains the second set of training set prediction values ​​and the second set of test set prediction values ​​from the CNN-LSTM-ATTENTION module; then, the optimal weight is solved according to the error between the first set of training set prediction values, the second set of training set prediction values, and the true observation values. and ; Finally, according to the weight and and the first group of test set prediction values ​​and the second group of test set prediction values ​​are calculated by combining the combination weights to obtain the test set combination prediction values; The output module is connected to the combination prediction module; the output module obtains the test set combination prediction value from the combination prediction module, and transmits the test set combination prediction value to the display module; The display module is connected to the output module; the display module receives the test set combination prediction value of the output module and displays the test set combination prediction value; In the second step, the input module extracts N historical data from the database and sends them to the data preprocessing module; In the third step, the data preprocessing module receives N historical data from the input module and performs data preprocessing on the N historical data. The method is: 3.1 The data filling module of the data preprocessing module receives N historical data from the input module. The N historical data are represented by a triple (Time, Label, Value). Time represents time and is arranged in ascending order of time; Label represents the label; Value represents the value of the corresponding label at the corresponding time; if Value NaN, NaN is a null value, which means that the point is not missing. If Value=NaN, it means that the point is missing. 3.2 The data filling module of the data preprocessing module determines whether the VALUE values ​​in the N data are continuous or the boundary data are missing. If yes, go to step 3.3; if not, go to step 3.4; 3.3 If the VALUE values ​​in N data are missing continuously or the boundary data are missing, the data preprocessing module uses the polynomial fitting method to fill the missing values. The method is: The fitting data selected for the corresponding variables are the adjacent Time. ;in, , , , , is a constant, x is a variable, and the degree of the polynomial is the x The highest power of n express; The present invention adopts the least square method to fit the polynomial, such as formula 1 As shown; where m is the number of data points used, It is i The observed value of the dependent variable for each data point, is the predicted value of the dependent variable for the ith data point; use formula 1 to calculate the missing value for filling, and go to step 3.4; 3.4 The data filling module of the data preprocessing module determines whether the middle value of the VALUE value in the N data is missing. If so, go to step 3.

5. If it is complete, go to step 3.

6. 3.5 If the middle value of N data is missing, the data preprocessing module uses the data filling module Lagrange interpolation method to fill the missing data. The method is: When there are n+1 data points and each variable x is different, the Lagrangian basis function is defined as in Formula 2 As shown, is the i-th Lagrangian basis function, n is the number of interpolation points, and are the coordinates of the interpolation points, where is the index of the current basis function, is the index of other basis functions; the Lagrange interpolation polynomial is expressed as a linear combination of all basis functions and corresponding function values ​​as shown in Formula 3 As shown, The function value of the i-th data point; Use formula 2 and formula 3 to calculate the missing values ​​for filling, send the N pieces of historical data after filling the missing values ​​to the data standardization module of the data preprocessing module, and go to step 3.6; 3.6 The data standardization module of the data preprocessing module performs min-max normalization on the N historical data after filling the missing values ​​to obtain the standardized N historical data, that is, the standardized value Value_norm, and transmits the standardized value Value_norm to the support vector regression module, and goes to the fourth step. At the same time, the standardized value Value_norm is transmitted to the CNN-LSTM-ATTENTION module, and goes to the fifth step; In the fourth step, the support vector regression module builds an SVR model, then inputs the training set to train the SVR model to obtain a trained SVR model, and finally inputs the training set and test set data to the trained SVR model to obtain the first training set prediction value and the first test set prediction value. The method is: 4.1 The support vector regression module receives the standardized N historical data from the data standardization module of the data preprocessing module; 4.1.1 Divide the standardized value Value_norm into the training set of the SVR model and the test set of the SVR model according to the triples of the data; 4.2 Support Vector Regression Module The SVR model is constructed by: 4.2.1 Support vector regression module creates support vector regression objects, high-dimensional regression models such as formula As shown, is the weight vector, T refers to the transpose, It is a kernel function that maps the input space to a high-dimensional feature space. The kernel function used in the present invention is a Gaussian kernel function. is the bias term, m is the number of support vectors, and is the Lagrange multiplier, is the support vector, is a kernel function used to calculate the inner product of two data points in a high-dimensional space; 4.2.2 The optimization goal of SVR is to minimize the complexity of the model while ensuring that the error is within Points outside the error band are appropriately penalized, and C controls the degree of penalty for slack variables. The optimal parameters are selected using a five-fold cross-validation method, which is: 4.2.2.1 The dataset is divided into five equal parts, four for training sets and one for testing sets; 4.2.2.2 Set three to be selected value, The value is 0.1, The value is 0.2, The value is 0.3; four C values ​​to be selected are set. The value is 10, The value is 100, The value is 1000, The value is 10000; the coefficient of the Gaussian kernel function Denoted as g, five g values ​​to be selected are set, among which The value is 0.01, The value is 0.001, The value is 0.0001, , , scale is 1 / (n_features * var), where n_features is the number of features and var is the variance of the feature; auto is 1 / n_features; Train the SVR model using different parameter combinations, select one of the subsets as the test set and the other four subsets as the training set, train the model and evaluate it on the test set; repeat this process five times to ensure that each subset is used as the test set once; 4.2.2.3 After the five cross-validations are completed, the average prediction error index is calculated as the evaluation criterion for model performance and the optimal parameter combination is selected; 4.2.3 SVR model Select the optimal parameters, input the training set to train the SVR model, and obtain the trained SVR model; 4.3 The support vector regression module inputs the training set data and the test set data into the trained SVR model to predict and obtain the first set of training set prediction values ​​y_train_pred1 and the first set of test set prediction values ​​y_test_pred1; 4.4 The support vector regression module sends the first set of training set prediction values ​​y_train_pred1 and the first set of test set prediction values ​​y_test_pred1 to the combined prediction module, and then goes to step 6.1; In the fifth step, the CNN-LSTM-ATTENTION module constructs a CNN-LSTM-ATTENTION neural network model, and then inputs the training set to train the CNN-LSTM-ATTENTION neural network to obtain a trained CNN-LSTM-ATTENTION neural network model. The training set and test set data are input to obtain the second training set prediction value and the second test set prediction value. The method is: 5.1 The CNN-LSTM-ATTENTION module receives the standardized N historical data from the data standardization module of the data preprocessing module, sets the time step according to the triple Time, and divides the standardized N historical data into the training set and test set of the CNN-LSTM-ATTENTION model; 5.2 CNN-LSTM-ATTENTION module Construct the CNN-LSTM-ATTENTION network model by: 5.2.1 CNN-LSTM-ATTENTION module builds the CNN-LSTM-ATTENTION network model; 5.2 CNN-LSTM-ATTENTION module Input the training set to the CNN-LSTM-ATTENTION model for training to obtain the trained CNN-LSTM-ATTENTION model. The method is: 5.2.1 The CNN neural network model uses a one-dimensional convolution layer to perform convolution operations on the time series data of the training set to extract the spatial features of the training set data. The convolutional features are downsampled through the pooling layer to reduce the data dimension and extract key information. 5.2.2 The CNN neural network model outputs the spatial features of the training set data to the LSTM recurrent neural network; 5.2.3 LSTM recurrent neural network obtains the spatial features of training set data from CNN convolutional neural network; 5.2.4 LSTM recurrent neural network uses the LSTM gating mechanism (including input gate, forget gate and output gate) to capture the long-term dependencies of the time series of the training set data; LSTM can remember long-term information while forgetting unimportant historical information, thereby better understanding the temporal dynamic characteristics of the data; 5.2.5 The LSTM recurrent neural network outputs the spatiotemporal features (time series features and spatial features) of the training set data to the ATTENTION self-attention mechanism; 5.2.6 ATTENTION The self-attention mechanism calculates the importance weight, performs weighted summation on the output of LSTM, and obtains the calculated score result; 5.2.7 ATTENTION The score result of the self-attention mechanism is output to the fully connected layer; 5.2.8 Perform back propagation and adjust weight parameters; 5.2.9 Iterate all epochs and get the trained model after completion; 5.3 CNN-LSTM-ATTENTION module inputs the training set data and the test set data into the trained CNN-LSTM-ATTENTION model respectively, and predicts the second training set prediction value y_train_pred2 and the second test set prediction value y_test_pred2, the method is: 5.3.1 The input layer receives the training set and the test set; 5.3.2 The CNN neural network model uses a one-dimensional convolution layer to perform convolution operations on the time series data of the training set and the test set to extract the spatial features of the training set and the test set data. The convolution features are downsampled through the pooling layer to reduce the data dimension and extract key information. 5.3.3 The CNN neural network model outputs the spatial features of the training set and test set data to the LSTM recurrent neural network; 5.3.4 LSTM recurrent neural network uses the gating mechanism of LSTM to capture the long-term dependencies of the time series of training set and test set data; 5.3.5 The LSTM recurrent neural network outputs the spatiotemporal features of the training set and test set data, namely the time series features and spatial features, to the ATTENTION self-attention mechanism; 5.3.6 ATTENTION The self-attention mechanism calculates the importance weight, performs weighted summation on the output of LSTM, and obtains the calculated score result; 5.3.7 ATTENTION The score result of the self-attention mechanism is output to the fully connected layer; 5.3.8 The fully connected layer integrates the features extracted by the hidden layer, outputs the second training set prediction value y_train_pred2 and the second test set prediction value y_test_pred2, and goes to step 6.1; In the sixth step, the combined prediction module obtains the combined prediction value by: 6.1 The combined prediction module obtains the first set of training set prediction values ​​y_train_pred1 from the support vector regression module, and also obtains the second set of training set prediction values ​​y_train_pred2 from the CNN-LSTM-ATTENTION module; 6.2 The combined prediction module solves the optimal weight according to the error between the predicted value of the first training set, the predicted value of the second training set, and the true observed value Value and , the method is: 6.2.1 Weighted prediction value of combined prediction module The actual observed value The absolute error between them is minimized, and a set of optimal weights is found and ; 6.2.2 Combination prediction module defines absolute loss function , as shown in the formula As shown; among them, n refers to the number of observations, , and They are observations, the i-th prediction of the SVR model, and the i-th prediction of the CNN-LSTM-ATTENTION model; 6.3 Combining prediction modules according to weights and And the first set of test set predictions And the second set of test set prediction values Calculate the combined forecast value , the method is: Final prediction value ; Step 7: The output module obtains the test set combination prediction value from the combination prediction module, and transmits the test set combination prediction value to the display module; In the eighth step, the display module receives the test set combination prediction value from the output module and displays the test set combination prediction value.

2. The combined prediction method based on SVR and CNN-LSTM-ATTENTION model as claimed in claim 1, characterized in that The first layer of the CNN-LSTM-ATTENTION network model is the input layer, and the number of neurons is the number of features n_feature, that is, the data dimension; the second layer is the hidden layer, and the first layer of the hidden layer is a one-dimensional convolution layer to extract spatial data features; the second layer of the hidden layer is a pooling layer to downsample the features, and the third layer of the hidden layer is an LSTM layer for extracting time series features; The fourth hidden layer is the self-attention mechanism layer, and the third layer is the output layer, that is, the fully connected layer. The fully connected layer consists of two layers, the first layer has 32 neurons, and the second layer has 1 neuron.

Images