Time series data completion method based on echo state network optimization

Abstract

The application discloses a time series data completion method based on echo state network optimization, which comprises the following steps: initializing an echo state network; running the echo state network by using the first 1000 points of input data to update the state vector of the storage layer and obtain a group of training signals of the storage layer output; adopting a linear regression method to optimize the output matrix between the storage layer and the output layer; predicting the prediction value of the 1000th-1200th point of the input data, wherein the 1000th-1100th point is missing; regarding the last 100 prediction values as the non-missing part, calculating the mean square error between the prediction value of the 1100th-1200th point and the actual data, adjusting the parameters of the echo state network based on the mean square error; and generating the prediction value of the 1000th-1100th point through the adjusted echo state network. According to the application, the existing time series data can be effectively utilized, and the missing data information can be significantly restored.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of reservoir computing (RC), specifically relates to a time series data completion method based on echo state network optimization. This method initializes the echo state network and uses optimization techniques to efficiently and accurately complete the missing data trajectory, especially suitable for time series prediction tasks. BACKGROUND

[0002] Echo state network (ESN) is a kind of recurrent neural network (RNN), which is widely used in chaotic time series prediction, weather prediction, power demand prediction and traffic flow prediction due to its simple training process and superior performance. ESN uses a large-scale hidden layer reservoir constructed by random method, and only needs to use simple linear regression to complete the training, without the need of back propagation algorithm. However, the performance of echo state network is highly dependent on the structure of its reservoir.

[0003] Traditional ESN optimization methods mainly focus on reducing training error and improving generalization ability, but there are challenges in dealing with missing data. Data missing will affect the training and prediction accuracy of the model, especially in time series prediction tasks. Most of the existing methods do not effectively solve the problem of how to deal with data missing in complex dynamic environment. SUMMARY

[0004] The present application provides a time series data completion method based on echo state network optimization, which initializes and trains the echo state network by using the existing complete data segment to predict and complete the missing data points, improves the integrity and reliability of the data, and the missing part of the data set used in the present application is from the 1000th to the 1100th.

[0005] A time series data completion method based on echo state network optimization, comprising:

[0006] Step 1: initialize the echo state network, including initializing the state feedback weight and the connection weight from the input layer to the storage layer, and the size of the input layer, storage layer and output layer;

[0007] Step 2: run the echo state network using the first 1000 points of the input data to update the state vector of the storage layer, and get a set of training signals of the storage layer output ;

[0008] Step 3: record the expected output of the network as When the output layer activation function takes a unit activation function, a linear regression method is used to optimize the output matrix between the storage layer and the output layer, and the objective function of the linear regression method is:

[0009] (3)

[0010] The first part of the objective function measures the error between the network prediction output and the expected output, and the regularization term is used to control the size of the weight matrix, is the output matrix between the storage layer and the output layer;

[0011] The output matrix between the storage layer and the output layer is obtained by solving formula (3):

[0012] (4)

[0013] is a regularization coefficient that controls the weight of the regularization term to prevent model overfitting, I is an identity matrix with a dimension of mxm, where m is the number of neurons in the storage layer, and the diagonal elements of the identity matrix are 1 and the rest are 0;

[0014] Step 4: After training the output matrix according to formula (4), the prediction values of the 1000-1200 points of the input data are predicted, wherein the 1000-1100 points are missing;

[0015] Step 5: The last 100 prediction values of the 200 prediction values obtained in step 4 are regarded as the non-missing part, the mean square error between the prediction values of the 1100-1200 points and the actual data is calculated, and the parameters of the echo state network are adjusted based on the mean square error to minimize the mean square error, to improve the accuracy of the completed data, wherein the parameters of the echo state network include the storage layer size and the regularization coefficient;

[0016] Step 6: Generate the prediction values of the 1000-1100 points through the adjusted echo state network, and take the prediction values as the final experimental data.

[0017] The time series data completion method based on the echo state network provided by the application has the following beneficial effects:

[0018] Efficiency: The reserve pool characteristics of ESN can efficiently initialize and adjust the network state, and have strong adaptability.

[0019] Accuracy: Since the missing part is completed, the reliability of the data cannot be judged after obtaining the data, the method optimizes the parameters by evaluating the mean square error of the non-missing part, which significantly improves the accuracy of prediction and completion.

[0020] Practicality: suitable for various practical application scenarios such as signal processing, time series prediction, financial data analysis, etc., significantly improving the integrity and reliability of data.

[0021] The method of the present application not only can effectively complete the missing data, but also can improve the overall performance of the prediction model by optimizing the network parameters, providing a powerful tool for time series data analysis. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 A structural schematic diagram of an ESN according to an embodiment of the present application is shown;

[0023] Figure 2 A comparison of the completed data and the real data of the UAV trajectory is shown;

[0024] Figure 3 A comparison of the completed data and the real data of the chaotic sequence is shown;

[0025] Figure 4 A comparison of the MSE before and after the parameter optimization of the UAV trajectory sequence is shown;

[0026] Figure 5 A comparison of the MSE before and after the parameter optimization of the chaotic sequence is shown. DETAILED DESCRIPTION

[0027] Exemplary embodiments of the present disclosure will be described in greater detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms, and the present disclosure should not be limited by the embodiments set forth herein. Rather, these embodiments are provided so that the present disclosure can be more thoroughly understood, and the scope of the present disclosure can be accurately conveyed to those skilled in the art.

[0028] In order to achieve the purpose of the present application, a brief introduction to ESN is given, and some parameters in the network are specified. Figure 1 A structural schematic diagram of an ESN according to an embodiment of the present application is shown. As shown in Figure 1 ESN is composed of three parts, namely input layer (Input Layer), storage layer (Reservoir), and output layer (Readout), which are used to indicates the input signal at time t, is a real set, and the dimension of the input signal is n x 1 or the connection weight from the input layer to the storage layer , where m is the number of neurons in the storage layer, and n is the dimension of the input signal, which is completed by random initialization. The storage layer accepts two-way input: one from the input layer , and the other is the output from the previous state of the storage layer , is the state vector of the storage layer at time t-1, with dimension m x 1. The state feedback weight and are determined by the initial state and do not need to be trained, is a large sparse matrix, and the non-zero elements in the matrix indicate the activated neurons in the storage layer:

[0029] (1)

[0030] wherein, is an activation function, represents a leak rate or update speed in the storage layer state update process, and controls the influence degree of the previous state x(t-1) in the storage layer state x(t), is the connection weight from the input layer to the storage layer, represents the state feedback weight matrix of the storage layer, with dimension m x m, represents the state of the storage layer, with dimension m x 1, and the output equation of the ESN is:

[0031] (2)

[0032] wherein, is an activation function of the output layer, is an output matrix between the storage layer and the output layer.

[0033] For a continuous set of training samples with length , the training storage layer signals ] can be obtained by sequentially inputting the training samples into the ESN according to formula (1).

[0034] According to an embodiment of the present application, a time series data completion method based on echo state network optimization is provided, comprising the steps of:

[0035] Step 1: initializing the echo state network, including initializing the state feedback weight and the connection weight from the input layer to the storage layer , and the sizes of the input layer, the storage layer and the output layer.

[0036] In an embodiment, step 1 can include: determining the input signal dimension and the output signal dimension (i.e. initializing the sizes of the input layer and the output layer), and initializing the state feedback weight of the storage layer.

[0037] Initialization of the storage layer state feedback weight includes:

[0038] (1) Initialize the input weight coefficient matrix , The elements in the matrix can be randomly generated according to a certain probability distribution, for example, using a uniform distribution between 0 and 1.

[0039] ,

[0040] (2) Determine the storage layer update factor , and initialize the state feedback weight . Like , the elements in the matrix can also be randomly generated. It should be noted that must satisfy the necessary conditions of the echo state network echo state property (see Gallicchio, 2017, Echo State Property of Deep Reservoir Computing Networks).

[0041] (3)

[0042] where represents the spectral radius of the matrix, which is numerically equal to the maximum value of the absolute value of the eigenvalue of the matrix to be solved, a represents the leakage rate or update speed, which controls the proportion of new and old states in the storage layer state update process, and the value range is usually between 0 and 1, I: identity matrix (Identity matrix), with a dimension of m x m, where m is the number of neurons in the storage layer. The diagonal elements of the identity matrix are 1, and the other elements are 0. : Storage layer state feedback weight matrix, with a dimension of m x m. It is a large sparse matrix, and the non-zero elements in it represent the connection strength between the activated neurons in the storage layer.

[0043] Step 2: Run the echo state network using the first 1000 points (20 seconds) of the input data to update the state vector of the storage layer, obtaining a set of training signals of the storage layer output. The state vector represents the activation value of all neurons in the storage layer.

[0044] Step 3: Let the expected output of the network be When the output layer activation function takes the unit activation function, the linear regression method is used to optimize the output matrix between the storage layer and the output layer, and the objective function of the linear regression method is:

[0045] (4) ​​​

[0046] The first part of the objective function measures the error between the network's predicted output and the expected output. The unit activation function indicates that the output layer of the network is linear, which allows us to use a linear regression method to train the weight matrix of the output layer The regularization term is used to control the size of the weight matrix, prevent overfitting, and maintain the generalization ability of the model.

[0047] In one embodiment, the linear regression method can be ridge regression.

[0048] The objective function aims to optimize the weights of the output layer through the linear regression method so that the network's predicted output is as close as possible to the expected output Y, while controlling the complexity of the model through regularization.

[0049] The output matrix between the storage layer and the output layer is obtained by solving equation (4), as shown in the following equation (5):

[0050] (5)

[0051] is the regularization coefficient, which controls the weight of the regularization term and prevents the model from overfitting.

[0052] Step 4: After training the output matrix according to equation (4), predict the 1000-1200 points of the input data, where the 1000-1100 points (2 seconds in total) are missing.

[0053] Specifically, step 4 can include:

[0054] Take the 1000th value as the input value, get the new storage layer output according to equation (1), and get the network output at the next time according to equation (2).

[0055] Take the obtained network output at the next time as the input value at the next time , and get the network output at the next time by applying equations (1) and (2).

[0056] Repeat the above until 200 predicted values are obtained.

[0057] Step 5: Consider the last 100 predicted values ​​out of the 200 predicted values ​​obtained in Step 4 as the non-missing portion. Calculate the mean squared error (MSE) between the predicted values ​​for points 1100-1200 and the actual data. Adjust the parameters of the echo state network based on the MSE to minimize the MSE and improve the accuracy of the completed data. The parameters of the echo state network include the storage layer size and the regularization coefficient.

[0058] Use gradient descent or other optimization algorithms, such as grid optimization, to adjust the parameters of the echo state network to minimize the mean square error between the predicted values ​​and the actual data for the 1100-1200th data points.

[0059] Step 6: Using the adjusted echo state network, generate predicted values ​​for points 1000-1100, and use these predicted values ​​as the final experimental data. These predicted values ​​for points 1000-1100 are used to complete the missing data.

[0060] The initialization and data prediction processes of the echo state network described above can be executed automatically in a computer system.

[0061] The above methods may also include a result verification step:

[0062] To verify whether parameter optimization improves the performance, we will compare the mean squared error between the 1000th to 1100th data points and the real dataset without parameter optimization.

[0063] The mean squared error of the network training is:

[0064] (6)

[0065] In the formula, Let K and t represent the t-th real data point and the t-th predicted data point, respectively. K is the number of samples, and t represents the index of each data point.

[0066] The selected task scenario is a 31st-order MackeyGlass chaotic time series prediction dataset combined with real-world drone data collection. The storage layer size before parameter optimization is 1000, and the regularization coefficient is [value missing]. The grid optimization range is for storage layer sizes of 500-1500, with intervals of 100, and a regularization coefficient of [value missing]. .

[0067] Figure 2 and Figure 3 The figures show a comparison between real data collected by a drone and the completed data derived from a chaotic time series after parameter optimization, as well as the real data. The horizontal axis represents time points, and the vertical axis represents the pitch angle and the chaotic sequence values. As can be seen from the figures, the completed data from this invention closely resembles the real data.

[0068] Figure 4 The mean square error of the UAV trajectory sequence after parameter optimization and the mean square error before parameter optimization are compared, and repeated experiments are performed on data of different time periods. It can be seen from the figure that after optimization, the order of magnitude of the mean square error reaches , which is greatly improved compared to before parameter optimization.

[0069] Figure 5 The mean square error of the chaotic sequence before and after parameter optimization is shown, which also improves by two orders of magnitude, and the completion effect is significantly improved.

[0070] In the specification provided herein, a large number of specific details are described. However, it can be understood that the embodiments of the present application can be practiced without these specific details. In some examples, well-known methods, structures and techniques are not shown in detail in order not to obscure the understanding of the present specification.

[0071] Although the present application is described in terms of a limited number of embodiments, those skilled in the art, with the benefit of the description above, will appreciate that other embodiments can be conceived within the scope of the application described herein. In addition, it should be noted that the language used in the present specification is mainly selected for readability and instructional purposes, and is not selected to explain or limit the subject matter of the present application.

Claims

1. A time series data completion method based on echo state network optimization, used in the field of signal processing, characterized in that, The method comprises the steps of: Step 1: initializing the echo state network, including initializing state feedback weights and connection weights from the input layer to the storage layer, and the sizes of the input layer, the storage layer and the output layer; Step 2: Run the echo state network using the first 1000 points of the input data to update the state vector of the reservoir layer, resulting in a set of reservoir layer outputs, the training signal ; Step 3: Let the desired output of the network be When the output layer activation function takes the unit activation function, the linear regression method is used to optimize the output matrix between the storage layer and the output layer, and the objective function of the linear regression method is: （4） a first part of the objective function measures an error between a network predicted output and an expected output, a regularization term for controlling a size of the weight matrix, is an output matrix between the storage layer and the output layer; The output matrix between the storage layer and the output layer is obtained by formula (4): （5） is a regularization coefficient that controls the weight of the regularization term to prevent overfitting of the model, I is an identity matrix with a dimension of mxm, where m is the number of neurons in the storage layer, the diagonal elements of the identity matrix are 1, and the remaining elements are 0; Step 4: after training the output matrix according to formula (4), the prediction values of the 1000th-1200th points of the input data are predicted, wherein the 1000th-1100th points are missing; Step 5: the last 100 prediction values of the 200 prediction values obtained in step 4 are regarded as the non-missing part, the mean square error between the prediction values of the 1100th-1200th points and the actual data is calculated, the parameters of the echo state network are adjusted based on the mean square error to minimize the mean square error, so as to improve the accuracy of the completed data, wherein the parameters of the echo state network include the size of the storage layer and the regularization coefficient; Step 6: the prediction values of the 1000th-1100th points are generated by the adjusted echo state network, and the prediction values are taken as the final experimental data.

2. The time series data completion method based on echo state network optimization according to claim 1, characterized in that, Step 1 comprises: determining the input signal dimension and the output signal dimension, initializing the storage layer weight coefficient, and the initialization of the storage layer weight coefficient comprises: (1) initialize the connection weights from the input layer to the storage layer , The middle elements are randomly generated according to a specific probability distribution; (2) Determine the storage layer update factor , and initialize the state feedback weight , The necessary conditions for satisfying the echo state network echo state properties are as follows: （3） where, denotes the spectral radius of a matrix, which is numerically equal to the maximum value of the absolute value of the eigenvalues of the matrix to be solved, a denotes the leakage rate or the update speed, which controls the proportion of the new and old states in the state updating process of the storage layer, and its value range is between 0 and 1, I: unit matrix, the dimension is m x m, where m is the number of neurons in the storage layer, the diagonal elements of the unit matrix are 1, and the rest of the elements are 0, : the state feedback matrix of the storage layer, the dimension is m x m, which is a large sparse matrix, and the non-zero elements in it represent the connection strength between the activated neurons in the storage layer.

3. The time series data completion method based on echo state network optimization according to claim 2, characterized in that, Using to uniform distribution between the initialization of the input layer to the storage layer connection weights : 。 4. The method of claim 1, wherein, Step 4 comprises: The 1000th value is taken as the input value, and a new storage layer output is obtained according to formula (1) , and a network output at the next moment is obtained according to formula (2) : （1） wherein, is an activation function, denotes the leakage rate or update speed in the storage layer state update process, is the input of the network at time t-1; （2） wherein, is an output layer activation function, is an output matrix between the storage layer and the output layer; the network output at the next time point is obtained by applying formula (1) and formula (2) as the input value at the next time point , the network output at the next time point is obtained by applying formula (1) and formula (2) ; The above is repeated until 200 prediction values are obtained.

5. The method of claim 1, wherein, In step 5, the gradient descent method or the grid optimization is used to adjust the parameters of the echo state network to minimize the mean square error between the prediction values of the 1100th-1200th data points and the actual data.

6. The method of claim 1 to 5, wherein, The initialization of the echo state network and the data prediction process are automatically executed in a computer system.

Images