Edge time series prediction method based on improved knowledge distillation ESN

By improving the edge-end time series prediction method of knowledge distillation ESN, the reserve pool structure is optimized, and the problem of limited computing power of edge-end equipment is solved, the prediction performance is improved and the calculation overhead is reduced, and it is suitable for time series prediction at the edge-end.

CN114742306BActive Publication Date: 2024-11-08NANJING UNIV OF POSTS & TELECOMM
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210414935.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-04-20
Publication Date
2024-11-08
Estimated Expiration
2042-04-20

AI Technical Summary

Technical Problem

The computing and storage capabilities of edge devices are limited, which limits the long-term memory capability and computing overhead of the Echo State Network (ESN) in time series prediction, resulting in insufficient prediction performance.

Method used

Adopting the edge-end time series prediction method of improved knowledge distillation ESN, by constructing teacher network, teaching assistant network and student network, optimizing the reserve pool structure, improving long-term memory ability and reducing computing overhead. The specific steps include training the model on the server side and at the edge side. Make predictions.

Benefits of technology

It improves the accuracy of time series prediction, reduces the computational overhead in the prediction process, and makes ESN more suitable for deployment at the edge for time series prediction.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114742306B_ABST
    Figure CN114742306B_ABST
Patent Text Reader

Abstract

The edge-end time series prediction method based on improved knowledge distillation ESN first combines the echo state network ESN with knowledge distillation KD to construct a time series prediction model based on improved knowledge distillation ESN; then a training algorithm is proposed to train the model on the server side, optimize the reserve pool structure, improve its long-term memory capacity and reduce its computational overhead; finally, the trained time series prediction model based on improved knowledge distillation ESN is used on the edge to perform time series prediction. This method can improve the long-term memory capacity of the reserve pool and reduce its computational overhead, making ESN more suitable for deployment on the edge to complete time series prediction tasks.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of computer application technology, and in particular relates to an edge time series prediction method based on improved knowledge distillation ESN. Background Art

[0002] With the advent of the Internet of Everything era and the popularization of wireless network devices, the number of devices at the edge of the network and the data generated are growing rapidly, increasing the workload on the server side. Edge intelligence can allocate computing tasks to the edge side close to the data source, effectively reducing the workload on the server side while increasing the data processing speed. However, devices on the edge side generally have limited computing power and storage capacity. This limits the application of edge intelligence in specific production practices.

[0003] Time series prediction, by modeling the time series, the time series values ​​at future moments can be predicted. Time series prediction has always been one of the hot topics in academia and industry, especially in industry, where it has more practical research value, such as temperature prediction. Time series prediction at the edge can effectively improve the real-time performance of the prediction. As a type of neural network model, the Echo State Network (ESN) has high prediction performance and low computational overhead, and has achieved good results in time series prediction. However, while ESN has strong short-term memory capabilities, its long-term memory capabilities are slightly insufficient, which limits the further improvement of its prediction performance. On the other hand, due to the limitations of the computing power and storage capacity of edge devices, the computational overhead of ESN needs to be further reduced. Summary of the invention

[0004] In view of the problems raised in the above background technology, the present invention proposes an edge-end time series prediction method based on improved knowledge distillation ESN. The method can optimize the reserve pool structure of ESN, improve its long-term memory capacity and reduce computational overhead. The present invention can improve the accuracy of time series prediction, reduce the computational overhead in the prediction process, and is more suitable for deployment at the edge for time series prediction.

[0005] The edge time series prediction method based on improved knowledge distillation ESN includes the following steps:

[0006] Step 1: Construct a time series prediction model based on improved knowledge distillation ESN, including teacher network, teaching assistant network and student network;

[0007] Step 2: Collect time series data at the edge and use the collected time series data on the server to train the time series prediction model based on improved knowledge distillation ESN constructed in step 1; it includes three parts: teacher network training, assistant network training and student network training; with the help of the assistant network, the student network learns the long-term memory ability of the teacher network and optimizes the reserve pool structure, thereby obtaining the corresponding reserve pool connection matrix W;

[0008] Step 3: Use the time series prediction model based on improved knowledge distillation ESN trained in step 2 on the edge to predict the future time series based on the collected time series data to be predicted.

[0009] Furthermore, in step 1, the specific steps of constructing a time series prediction model based on improved knowledge distillation ESN are as follows:

[0010] Step 1-1: Use LSTM as the teacher network and set its parameters;

[0011] Step 1-2: Use ESN as the student network and set its parameters;

[0012] Step 1-3: Use the dual-ring ESN as the teaching assistant network and set its parameters;

[0013] Step 1-4: Use the teacher network, student network and teaching assistant network in the above steps to construct a time series prediction model based on improved knowledge distillation ESN; the teacher network inputs the training set and outputs distilled knowledge, the teaching assistant network inputs the training set and distilled knowledge, and outputs teaching assistant knowledge; the student network inputs the training set, distilled knowledge and teaching assistant knowledge to complete the training of the reserve pool structure; set the teaching assistant distillation learning rate α, β, the student distillation learning rate μ, v, o, and the iterative pruning times it;

[0014] Step 1-5: Determine the input and output of the model; the input vector of the model is the time series historical data u(t) = (tr(t-P+1), tr(t-P+2),…, tr(t)), P is the sliding window size, tr(t) is the time series data at time t; the output vector is the predicted data y(t) = tr(t+1).

[0015] Furthermore, in step 1-1, LSTM is responsible for teaching long-term memory ability, including input layer, hidden layer, and output layer, where the hidden layer adopts a two-layer structure; the input layer and hidden layer, the hidden layer and the output layer are fully connected; the number of neurons in the input layer and output layer of LSTM is set to K and L respectively; the number of neurons in the first hidden layer is set to M1, and the number of neurons in the second hidden layer is set to M2; the linear rectifier function relu(·) is used as the hidden layer neuron excitation function, and the linear function liner(·) is used as the output layer neuron excitation function; the learning rate is set to

[0016] Furthermore, in step 1-2, ESN is responsible for time series prediction, including input layer, reserve pool, and output layer; the input layer and hidden layer, as well as the output layer and hidden layer are fully connected, and the neurons in the reserve pool are sparsely connected; the number of neurons in the input layer and output layer of ESN is set to K and L respectively; the number of neurons in the reserve pool is set to N; and the input connection matrix W of size K×N is constructed in , the internal weights are randomly initialized; construct an initial reservoir connection matrix W of size N×N, and the internal weights are randomly initialized; use the hyperbolic tangent function tanh(·) as the internal activation function f of the reservoir in , using the identity function identity(·) as the output activation function f out ; Set the learning rate to

[0017] Furthermore, in steps 1-3, the dual-loop ESN is responsible for supervising the distillation learning direction, including the input layer, dual-loop reservoir, and output layer; constructing an input connection matrix W of size K×N in-dlrs , the internal weights are randomly initialized; construct a double-ring reserve pool connection matrix W of size N×N dlrs , the internal weights are initialized to zero; the double-loop reserve pool is set as a circular feedback reserve pool, the neurons in the reserve pool are connected in a circular manner, and there is a feedback connection between two adjacent neurons; the double-loop interval is set to d, and there is a feedback connection between neurons at intervals of d positions; the absolute value of the non-zero element r is set as the feedback connection weight, that is, the connection matrix W dlrs There is w i,i+1 = r, w i+1,i = r, where i = 1, 2, ... N-1, N is the number of neurons in the reserve pool. When i = N, set w i,1 =r,w 1,i = r; with w i,i+d =r,w i+d,i = r, where d is the double ring interval; when When setting w i,1 =r,w1,i = r; use the hyperbolic tangent function tanh(·) as the internal excitation function of the reservoir f in , using the identity function identity(·) as the output activation function f out ; Set the learning rate to

[0018] Furthermore, in step 2, the specific steps of training are as follows:

[0019] Step 2-1: Collect time series data and construct a training set;

[0020] Within the specified sampling time, the time series data set T is collected at the edge. r (p) = {tr(1), tr(2), ..., tr(t), ..., tr(p)} and transmitted to the server, where tr(t) represents the time series data at time t, and p is the total number of collected data;

[0021] On the server side, according to T r (p) to construct the training sample set:

[0022] U={(u train (t),y train (t+1)),P≤t≤m-1}

[0023] Among them, u train (t)=(tr(t-P+1),tr(t-P+2),…,tr(t)),y train (t+1)=tr(t+1), P is the sliding window size;

[0024] The training set U is divided into three parts, including teacher training set U1, teaching assistant training set U2, and student training set U3;

[0025] Step 2-2: On the server side, according to U1, train the teacher network and output the distilled knowledge T1;

[0026] Step 2-3: On the server side, according to U2, train the teaching assistant network and output the teaching assistant knowledge A and the new teacher distilled knowledge T2;

[0027] Step 2-4: On the server side, according to U3, train the student network and output the learned reserve pool connection matrix W;

[0028] Step 2-5: On the server side, prune the neurons in the student network reserve pool;

[0029] According to U3, redundant knowledge is pruned from the student network classic ESN to reduce the impact of redundant learning; neuron i in the reserve pool is tried to be pruned; the objective function of classic ESN training is used as the evaluation index; its objective function is as follows:

[0030]

[0031] Among them, y(t+1) is the true value, is the predicted value of the classic ESN, and m is the size of the training set;

[0032] If the objective function value decreases after pruning neuron i, that is, the network prediction performance is improved, then neuron i is pruned and the connections between the remaining neurons and neuron i are removed; specifically, let w i,j =w j,i =0; where w i,j represents the connection weight from neuron i to neuron j, 0≤j≤N-1, N is the number of neurons in the reserve pool; if the objective function value increases, no pruning is performed;

[0033] Step 2-6: Repeat the contents described in step 2-5 until the preset number of pruning iterations is reached or the training objective function value does not change, and obtain the pruned reserve pool connection matrix W;

[0034] Step 2-7: On the server side, train the student network with the new reservoir structure;

[0035] According to U3, the optimized student network is trained by ridge regression to obtain the output connection matrix W out .

[0036] Furthermore, in step 2-2, according to U1, the teacher network LSTM is trained by the adaptive moment estimation gradient descent method to obtain the coefficient weight matrix; the objective function of LSTM training is as follows:

[0037]

[0038] Among them, y(t+1) is the true value, is the predicted value of LSTM, and m is the size of the training set;

[0039] For the trained teacher network LSTM, input u in U2 train (t), output LSTM distilled knowledge T1;

[0040]

[0041] Furthermore, in step 2-3, according to U2, the teaching assistant network double-loop ESN is trained by ridge regression; the objective function of the double-loop ESN training is as follows:

[0042]

[0043] in, is the predicted value of the double-ring ESN, m is the size of the training set, It is the target prediction value in the training process, which is obtained by fusing the true value with the distilled knowledge T1. The calculation formula is as follows:

[0044]

[0045] Among them, α and β are the assistant distillation learning rates, and α+β=1;

[0046] For the trained assistant network dual-loop ESN, input u in U3 train (t), output the teaching assistant knowledge A of the dual-loop ESN; for the trained teacher network LSTM, input u in U3 train (t), output LSTM distilled knowledge T2;

[0047]

[0048]

[0049] in, Input u for the dual-ring ESN pair of the teaching assistant network train The predicted value of (t), For the teacher network LSTM to input u train The predicted value of (t).

[0050] Furthermore, in step 2-4, according to U3, ESN is trained by ridge regression to obtain the output connection matrix W out ; The objective function of training is as follows:

[0051]

[0052] Among them, y(t+1) is the true value, is the predicted value of the classic ESN, and m is the size of the training set;

[0053] With the help of distilled knowledge T2 and teaching assistant knowledge A, the student network classic ESN's reserve pool learns long-term memory ability; for classic ESN, a training set is constructed based on U3 in, is the true value y train (t+1), teacher network LSTM prediction value The prediction value of double loop ESN of teaching assistant network The weighted average of is calculated as follows:

[0054]

[0055] Where μ, ν, and o are the student distillation learning rates, and μ+ν+o=1;

[0056] according to Calculate the new reserve pool state vector x(t) using the following formula:

[0057]

[0058] Among them, f arcout is the inverse function of the output activation function, W out The output connection matrix obtained from ridge regression training;

[0059] According to the new reservoir state vector x(t), calculate the new reservoir connection matrix W:

[0060] W=[f arc (x(t))-W in u train (t)] / x(t-1)

[0061] Among them, u train (t) is the input vector at time t, x(t-1) is the state vector of the reservoir at time t-1, and f arc is the inverse function of the reserve pool incentive function;

[0062] W is scaled as follows to maintain the echo characteristics of the reservoir;

[0063]

[0064] Among them, sc is the scaling factor, λ max (W) is the maximum eigenvalue of W.

[0065] Furthermore, in step 3, the trained time series prediction model based on the improved knowledge distillation ESN is deployed on the edge; at the edge, the time series data u to be predicted is collected and obtained. pred (t') = {tr(t'-P+1), tr(t'-P+2), ..., tr(t')}; Use the trained time series prediction model based on improved knowledge distillation ESN at the edge to predict future time series data and output the time series data x at the next moment pred (t'+1); the specific steps are as follows:

[0066] Step 3-1: Student network input u pred (t'), update the reserve pool state vector x pred (t'):

[0067] x pred (t') = fin (W in u pred (t')+Wx pred (t'-1));

[0068] Among them, f in is the internal incentive function of the reserve pool, W in is the input connection matrix, W is the reserve pool connection matrix trained in step 2, x pred (t'-1) is the state vector of the reserve pool at the previous moment, u pred (t') and x pred (t') respectively represent the input vector and the state vector of the reservoir at this moment;

[0069] Step 3-2: According to the state vector of the reserve pool, the student network outputs the time series data y at the next moment pred (t'+1):

[0070] y pred (t'+1)=f out (W out (u pred (t'),x pred (t'))).

[0071] Among them, f out is the output activation function, W out is the output connection matrix obtained by training in step 2.

[0072] The beneficial effects achieved by the present invention are as follows: First, ESN and KD are combined at the same time to construct a time series prediction model based on the improved knowledge distillation ESN. Then, time series data is continuously collected at the edge, a training algorithm is proposed, and training is performed on the server to optimize the reserve pool structure. Finally, the trained time series prediction model based on the improved knowledge distillation ESN is used at the edge to perform time series prediction. This method can optimize the reserve pool structure, improve its long-term memory capacity, and reduce its computational overhead, making ESN more suitable for deployment at the edge to complete time series prediction tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] Figure 1 1 is a schematic diagram of the steps of the prediction method described in an embodiment of the present invention.

[0074] Figure 2 It is a schematic diagram of the time series prediction model described in an embodiment of the present invention. DETAILED DESCRIPTION

[0075] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings.

[0076] The edge time series prediction method based on improved knowledge distillation (KD) echo state network (ESN) is referenced Figure 1 , the specific steps are as follows:

[0077] Step 1: Construct a time series prediction model based on improved knowledge distillation ESN, such as Figure 2 The specific steps are as follows:

[0078] Step 1-1: Use LSTM as the teacher network and set its parameters. LSTM is responsible for teaching long-term memory ability. It consists of an input layer, a hidden layer, and an output layer. The hidden layer adopts a two-layer structure. The input layer and the hidden layer, the hidden layer and the output layer are fully connected. The number of neurons in the input layer and the output layer of LSTM is set to K and L respectively. The number of neurons in the first hidden layer is set to M1, and the number of neurons in the second hidden layer is set to M2. The linear rectifier function relu(·) is used as the hidden layer neuron excitation function, and the linear function liner(·) is used as the output layer neuron excitation function. The learning rate is set to

[0079] Step 1-2: Use the classic ESN as the student network and set its parameters. The classic ESN is responsible for time series prediction and consists of an input layer, a reserve pool, and an output layer. The input layer and the hidden layer, as well as the output layer and the hidden layer are fully connected, and the neurons in the reserve pool are sparsely connected. Set the number of neurons in the input layer and output layer of the classic ESN to K and L, respectively, which is the same number of neurons used by the LSTM in step 1-1. Set the number of neurons in the reserve pool to N. Construct an input connection matrix W of size K×N. in , the internal weights are randomly initialized; construct an initial reservoir connection matrix W of size N×N, and the internal weights are randomly initialized; use the hyperbolic tangent function tanh(·) as the internal activation function f of the reservoir in , using the identity function identity(·) as the output activation function f out ; Set the learning rate to

[0080] Step 1-3: Use the dual-loop ESN as the teaching assistant network and set its parameters. The dual-loop ESN is responsible for supervising the distillation learning direction and consists of an input layer, a dual-loop reserve pool, and an output layer; construct an input connection matrix W of size K×N in-dlrs , the internal weights are randomly initialized; construct a double-ring reserve pool connection matrix W of size N×N dlrs, the internal weights are initialized to zero; the double-loop reserve pool is set as a circular feedback reserve pool, the neurons in the reserve pool are connected in a circular manner, and there is a feedback connection between two adjacent neurons; the double-loop interval is set to d, and there is a feedback connection between neurons at intervals of d positions; the absolute value of the non-zero element r is set as the feedback connection weight, that is, the connection matrix W dlrs There is w i,i+1 = r, w i+1,i = r, where i = 1, 2, ... N-1, N is the number of neurons in the reserve pool. When i = N, set w i,1 =r,w 1,i = r; with w i,i+d =r,w i+d,i = r, where d is the double ring interval; when When setting w i,1 =r,w 1,i = r; use the hyperbolic tangent function tanh(·) as the internal excitation function of the reservoir f in , using the identity function identity(·) as the output activation function f out ; Set the learning rate to

[0081] Step 1-4: Use the teacher network, student network and teaching assistant network in the above steps to construct a time series prediction model based on improved knowledge distillation ESN, refer to Figure 2 ; Among them, the teacher network inputs the training set and outputs distilled knowledge, the teaching assistant network inputs the training set and distilled knowledge, and outputs teaching assistant knowledge; the student network inputs the training set, distilled knowledge and teaching assistant knowledge to complete the training of the reserve pool structure; set the teaching assistant distillation learning rate α, β, the student distillation learning rate μ, v, o, and the number of iterative pruning it.

[0082] Step 1-5: Determine the input and output of the model. The input vector of the model is the time series historical data u(t) = (tr(t-P+1), tr(t-P+2), ..., tr(t)), P is the sliding window size, tr(t) is the time series data at time t; the output vector is the predicted data y(t) = tr(t+1).

[0083] Step 2: Collect time series data at the edge, transmit it to the server, and use the collected time series data on the server to train the time series prediction model based on the improved knowledge distillation ESN, which includes three parts: teacher network training, assistant network training, and student network training. With the help of the assistant network, the student network learns the long-term memory ability of the teacher network and optimizes the reserve pool structure, thereby obtaining the corresponding reserve pool connection matrix W; the specific steps for training the time series prediction model based on the improved knowledge distillation ESN are as follows:

[0084] Step 2-1: Collect time series data and construct a training set.

[0085] Within the specified sampling time, the time series data set T is collected at the edge. r (p) = {tr(1), tr(2), ..., tr(t), ..., tr(p)} and transmitted to the server, where tr(t) represents the time series data at time t and p is the total number of collected data.

[0086] On the server side, according to T r (p) to construct the training sample set:

[0087] U={(u train (t),y train (t+1)),K≤t≤m-1}

[0088] Among them, u train (t)=(tr(t-P+1),tr(t-P+2),…,tr(t)),y train (t+1)=tr(t+1), P is the sliding window size.

[0089] The training set U is divided into three parts, including teacher training set U1, teaching assistant training set U2, and student training set U3.

[0090] Step 2-2: On the server side, according to U1, train the teacher network and output the distilled knowledge T1.

[0091] According to U1, the teacher network LSTM is trained by the adaptive moment estimation gradient descent method to obtain the coefficient weight matrix. The objective function of LSTM training is as follows:

[0092]

[0093] Among them, y(t+1) is the true value, is the predicted value of LSTM, and m is the size of the training set.

[0094] For the trained teacher network LSTM, input u in U2 train (t), output the distilled knowledge T1 of LSTM.

[0095]

[0096] Step 2-3: On the server side, according to U2, train the teaching assistant network and output the teaching assistant knowledge A and the new teacher distilled knowledge T2.

[0097] According to U2, the teaching assistant network double-loop ESN is trained by ridge regression. The objective function of double-loop ESN training is as follows:

[0098]

[0099] in, is the predicted value of the double-ring ESN, m is the size of the training set, It is the target prediction value in the training process, which is obtained by fusing the true value with the distilled knowledge T1. The calculation formula is as follows:

[0100]

[0101] Among them, α and β are the assistant distillation learning rates, and α+β=1.

[0102] For the trained assistant network dual-loop ESN, input u in U3 train (t), output the teaching assistant knowledge A of the dual-loop ESN. For the trained teacher network LSTM, input u in U3 train (t), output the distilled knowledge T2 of LSTM.

[0103]

[0104]

[0105] in, Input u for the dual-ring ESN pair of the teaching assistant network train The predicted value of (t), For the teacher network LSTM to input u train The predicted value of (t).

[0106] Step 2-4: On the server side, according to U3, train the student network and output the learned reserve pool connection matrix W.

[0107] According to U3, the classic ESN is trained by ridge regression to obtain the output connection matrix W out The objective function of the training is as follows:

[0108]

[0109] Among them, y(t+1) is the true value, is the predicted value of the classic ESN, and m is the size of the training set.

[0110] With the help of distilled knowledge T2 and teaching assistant knowledge A, the student network classic ESN's reserve pool learns long-term memory ability. For classic ESN, the training set is constructed according to U3 in, is the true value y train (t+1), teacher network LSTM prediction value The prediction value of double loop ESN of teaching assistant network The weighted average of is calculated as follows:

[0111]

[0112] Among them, μ, v, o are the student distillation learning rates, and μ+v+o=1.

[0113] according to Calculate the new reserve pool state vector x(t) using the following formula:

[0114]

[0115] Among them, f arcout is the inverse function of the output activation function, W out is the output connection matrix obtained from ridge regression training.

[0116] According to the new reservoir state vector x(t), calculate the new reservoir connection matrix W:

[0117] W=[f arc (x(t))-W in u train (t)] / x(t-1)

[0118] Among them, u train (t) is the input vector at time t, x(t-1) is the state vector of the reservoir at time t-1, and f arc It is the inverse function of the reserve pool incentive function.

[0119] W is scaled as follows to maintain the echo characteristics of the reservoir.

[0120]

[0121] Among them, sc is the scaling factor, λ max (W) is the maximum eigenvalue of W.

[0122] Step 2-5: On the server side, prune the neurons in the student network reserve pool.

[0123] According to U3, redundant knowledge is pruned from the student network classic ESN to reduce the impact of redundant learning. Try to prune the neuron i in the reserve pool. The objective function of classic ESN training is used as the evaluation indicator. Its objective function is as follows:

[0124]

[0125] Among them, y(t+1) is the true value, is the predicted value of the classic ESN, and m is the size of the training set.

[0126] If the objective function value decreases after pruning neuron i, that is, the network prediction performance improves, then neuron i is pruned and the connections between the remaining neurons and neuron i are removed. Specifically, let w i,j =w j,i = 0. Among them, w i,j Represents the connection weight from neuron i to neuron j, 0≤j≤N-1, N is the number of neurons in the reserve pool. If the objective function value increases, no pruning is done.

[0127] Step 2-6: Repeat the contents described in step 2-5 until the preset number of pruning iterations is reached or the training objective function value does not change, and obtain the pruned reserve pool connection matrix W.

[0128] Step 2-7: On the server side, train the student network with the new reservoir structure.

[0129] According to U3, the optimized student network is trained by ridge regression similar to the above steps to obtain the output connection matrix W out .

[0130] Step 3: Use the time series prediction model based on improved knowledge distillation ESN trained in step 2 to predict the future time series at the edge. At the edge, collect and obtain new time series data u pred (t') = {tr(t'-P+1), tr(t'-P+2), ..., tr(t')}; Use the trained time series prediction model based on improved knowledge distillation ESN at the edge to predict future time series data and output the time series data y at the next moment pred (t'+1). The specific steps are as follows:

[0131] Step 3-1: Student network input u pred (t'), update the reserve pool state vector x pred (t'):

[0132] x pred (t') = f in (W in u pred (t')+Wx pred (t'-1));

[0133] Among them, f in is the internal incentive function of the reserve pool, W in is the input connection matrix, W is the reserve pool connection matrix trained in step 2, x pred (t'-1) is the state vector of the reserve pool at the previous moment, u pred (t') and x pred (t') represent the input vector and the state vector of the storage pool at this moment respectively.

[0134] Step 3-2: According to the state vector of the reserve pool, the student network outputs the time series data y at the next moment pred (t'+1):

[0135] y pred (t'+1)=f out (W out (u pred (t'),x pred (t'))).

[0136] Among them, f out is the output activation function, W out is the output connection matrix obtained by training in step 2.

[0137] The above method was simulated using the Intel Berkeley Research Lab data (IBRL) time series data set. Normalized Root Mean Square Error (NRMSE) was used as the prediction performance evaluation index. The smaller the NRMSE, the stronger the prediction performance. Multiply-Accumulate Operation (MACC) was used as the computational overhead evaluation index. One computation of a neuron during the forward propagation process is considered to be 1 MACC. The smaller the MACC, the lower the computational overhead.

[0138] In order to verify the effectiveness of optimizing the reserve pool structure based on improved knowledge distillation, ESN-IKD is compared with the student network classic ESN, the assistant network double-loop ESN, and the teacher network LSTM. Table 1 shows the average NRMSE of the above four models in 20 experiments on the dataset.

[0139] Table 1 NRMSE of classic ESN, dual-loop ESN, LSTM, and ESN-IKD

[0140] Model Classic ESN Double Ring ESN LSTM ESN-IKD NRMSE 0.0193 0.0146 0.0036 0.0021

[0141] ESN-IKD achieved the lowest prediction error on the dataset and enhanced the long-term memory capacity of the reserve pool.

[0142] In order to verify the advantage of ESN-IKD in terms of computational overhead, the MACC of classic ESN and ESN-IKD is statistically analyzed as shown in Table 2. LSTM has a complex network structure and uses two hidden layers, with more neurons, so the MACC of LSTM far exceeds that of other models. The dual-loop ESN does not prune neurons, and its MACC is consistent with that of the classic ESN. Therefore, the MACC of LSTM and dual-loop ESN is not repeated in the table.

[0143] Table 2 MACC of classic ESN and ESN-IKD

[0144] Classic ESN 2754 ESN-IKD 2350

[0145] As can be seen from Table 2, ESN-IKD has a lower MACC, which reduces the computational overhead. Therefore, ESN-IKD is more suitable for deployment on edge devices with limited computing power.

[0146] The above description is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiment. Any equivalent modifications or changes made by ordinary technicians in this field based on the contents disclosed by the present invention should be included in the protection scope recorded in the claims.

Claims

1. The edge time series prediction method based on improved knowledge distillation ESN is characterized by: The method comprises the following steps: Step 1: Construct a time series prediction model based on improved knowledge distillation ESN, including teacher network, teaching assistant network and student network; In step 1, the specific steps of constructing a time series prediction model based on improved knowledge distillation ESN are as follows: Step 1-1: Use LSTM as the teacher network and set its parameters; Step 1-2: Use ESN as the student network and set its parameters; Step 1-3: Use the dual-ring ESN as the assistant network and set its parameters; Step 1-4: Use the teacher network, student network and teaching assistant network in the above steps to construct a time series prediction model based on improved knowledge distillation ESN; the teacher network inputs the training set and outputs distilled knowledge, the teaching assistant network inputs the training set and distilled knowledge, and outputs teaching assistant knowledge; the student network inputs the training set, distilled knowledge and teaching assistant knowledge to complete the training of the reserve pool structure; set the teaching assistant distillation learning rate α, β, the student distillation learning rate μ, v, o, and the iterative pruning times it; Step 1-5: Determine the input and output of the model; the input vector of the model is the time series historical data u(t) = (tr(t-P+1), tr(t-P+2), ..., tr(t)), P is the sliding window size, tr(t) is the time series data at time t; the output vector is the predicted data y(t) = tr(t+1); Step 2: Collect time series data at the edge and use the collected time series data on the server to train the time series prediction model based on improved knowledge distillation ESN constructed in step 1; it includes three parts: teacher network training, assistant network training and student network training; with the help of the assistant network, the student network learns the long-term memory ability of the teacher network and optimizes the reserve pool structure, thereby obtaining the corresponding reserve pool connection matrix W; In step 2, the specific steps of training are as follows: Step 2-1: Collect time series data and construct a training set; Within the specified sampling time, the time series data set T is collected at the edge. r (p) = {tr(1), tr(2), ..., tr(t), ..., tr(p)} and transmitted to the server, where tr(t) represents the time series data at time t, and p is the total number of collected data; On the server side, according to T r (p) to construct the training sample set: U={(u train (t),y train (t+1)),P≤t≤m-1} Among them, u train (t)=(tr(t-P+1), tr(t-P+2),…, tr(t)), y train (t+1)=tr(t+1), P is the sliding window size; The training set U is divided into three parts, including teacher training set U1, teaching assistant training set U2, and student training set U3; Step 2-2: On the server side, according to U1, train the teacher network and output the distilled knowledge T1; Step 2-3: On the server side, according to U2, train the teaching assistant network and output the teaching assistant knowledge A and the new teacher distilled knowledge T2; Step 2-4: On the server side, according to U3, train the student network and output the learned reserve pool connection matrix W; Step 2-5: On the server side, prune the neurons in the student network reserve pool; According to U3, redundant knowledge is pruned from the student network classic ESN to reduce the impact of redundant learning; neuron i in the reserve pool is tried to be pruned; the objective function of classic ESN training is used as the evaluation index; its objective function is as follows: Among them, y(t+1) is the true value, is the predicted value of the classic ESN, and m is the size of the training set; If the objective function value decreases after pruning neuron i, that is, the network prediction performance improves, then prune neuron i and remove the connections between the remaining neurons and neuron i; specifically, let w i,j =w j,i =0; where w i,j represents the connection weight from neuron i to neuron j, 0≤j≤N-1, N is the number of neurons in the reserve pool; if the objective function value increases, no pruning is performed; Step 2-6: Repeat steps 2-5 until the preset number of pruning iterations is reached or the training objective function value does not change, and obtain the pruned reserve pool connection matrix W; Step 2-7: On the server side, train the student network with the new reservoir structure; According to U3, the optimized student network is trained by ridge regression to obtain the output connection matrix W out ; Step 3: Use the time series prediction model based on improved knowledge distillation ESN trained in step 2 on the edge to predict the future time series based on the collected time series data to be predicted; In step 3, the trained time series prediction model based on improved knowledge distillation ESN is deployed on the edge; at the edge, the time series data u to be predicted is collected and obtained. pred (t′) = {tr(t′-P+1), tr(t′-P+2), …, tr(t′)}; Use the trained time series prediction model based on improved knowledge distillation ESN at the edge to predict future time series data and output the time series data y at the next moment pred (t′+1); the specific steps are as follows: Step 3-1: Student network input u pred (t′), update the reserve pool state vector x pred (t′): x pred (t′)=f in (W in u pred (t′)+Wx pred (t′-1)); Among them, f in is the internal incentive function of the reserve pool, W in is the input connection matrix, W is the reserve pool connection matrix trained in step 2, x pred (t′-1) is the state vector of the reserve pool at the previous moment, u pred (t′) and x pred (t′) represents the input vector and the state vector of the reservoir at this moment respectively; Step 3-2: According to the state vector of the reserve pool, the student network outputs the time series data y at the next moment pred (t′+1): y pred (t′+1)=f out (W out (u pred (t′),x pred (t′))) Among them, f out is the output activation function, W out is the output connection matrix obtained by training in step 2.

2. The edge time series prediction method based on improved knowledge distillation ESN according to claim 1 is characterized in that: In step 1-1, LSTM is responsible for teaching long-term memory ability, including input layer, hidden layer, and output layer, where the hidden layer adopts a two-layer structure; the input layer and hidden layer, the hidden layer and the output layer are fully connected; the number of neurons in the input layer and output layer of LSTM is set to K and L respectively; the number of neurons in the first hidden layer is set to M1, and the number of neurons in the second hidden layer is set to M2; the linear rectifier function relu(·) is used as the hidden layer neuron excitation function, and the linear function liner(·) is used as the output layer neuron excitation function; the learning rate is set to 3. The edge time series prediction method based on improved knowledge distillation ESN according to claim 1 is characterized in that: In step 1-2, ESN is responsible for time series prediction, including input layer, reserve pool, and output layer; the input layer and hidden layer, as well as the output layer and hidden layer are fully connected, and the neurons in the reserve pool are sparsely connected; the number of neurons in the input layer and output layer of ESN is set to K and L respectively; the number of neurons in the reserve pool is set to N; and the input connection matrix W of size K×N is constructed. in , the internal weights are randomly initialized; construct an initial reservoir connection matrix W of size N×N, and the internal weights are randomly initialized; use the hyperbolic tangent function tanh(·) as the internal activation function f of the reservoir in , using the identity function identity(·) as the output activation function f out ; Set the learning rate to 4. The edge time series prediction method based on improved knowledge distillation ESN according to claim 1 is characterized in that: In steps 1-3, the dual-loop ESN is responsible for supervising the distillation learning direction, including the input layer, dual-loop reservoir, and output layer; constructing an input connection matrix W of size K×N in-dlrs , the internal weights are randomly initialized; construct a double-ring reserve pool connection matrix W of size N×N dlrs , the internal weights are initialized to zero; the double-loop reserve pool is set as a circular feedback reserve pool, the neurons in the reserve pool are connected in a circular manner, and there is a feedback connection between two adjacent neurons; the double-loop interval is set to d, and there is a feedback connection between neurons at intervals of d positions; the absolute value of the non-zero element r is set as the feedback connection weight, that is, the connection matrix W dlrs There is w i,i+1 = r, w i+1,i = r, where i = 1, 2, ... N-1, N is the number of neurons in the reserve pool. When i = N, set w i,1 = r, w 1,i = r; there is w i,i+d = r, w i+d,i = r, where d is the double ring interval; when When setting w i,1 = r, w 1,i = r; use the hyperbolic tangent function tanh(·) as the internal excitation function of the reservoir f in , using the identity function identity(·) as the output activation function f out ; Set the learning rate to 5. The edge time series prediction method based on improved knowledge distillation ESN according to claim 1 is characterized in that: In step 2-2, according to U1, the teacher network LSTM is trained by the adaptive moment estimation gradient descent method to obtain the coefficient weight matrix; the objective function of LSTM training is as follows: Among them, y(t+1) is the true value, is the predicted value of LSTM, and m is the size of the training set; For the trained teacher network LSTM, input u in U2 train (t), output LSTM distilled knowledge T1; 6. The edge time series prediction method based on improved knowledge distillation ESN according to claim 1 is characterized in that: In step 2-3, according to U2, the teaching assistant network double-loop ESN is trained by ridge regression; the objective function of the double-loop ESN training is as follows: in, is the predicted value of the double-ring ESN, m is the size of the training set, It is the target prediction value in the training process, which is obtained by fusing the true value with the distilled knowledge T1. The calculation formula is as follows: Among them, α and β are the assistant distillation learning rates, and α+β=1; For the trained assistant network dual-loop ESN, input u in U3 train (t), output the teaching assistant knowledge A of the dual-loop ESN; for the trained teacher network LSTM, input u in U3 train (t), output LSTM distilled knowledge T2; in, Input u for the dual-ring ESN pair of the teaching assistant network train The predicted value of (t), For the teacher network LSTM to input u train The predicted value of (t).

7. The edge time series prediction method based on improved knowledge distillation ESN according to claim 1 is characterized in that: In step 2-4, according to U3, ESN is trained by ridge regression to obtain the output connection matrix W out ; The objective function of training is as follows: Among them, y(t+1) is the true value, is the predicted value of the classic ESN, and m is the size of the training set; With the help of distilled knowledge T2 and teaching assistant knowledge A, the student network classic ESN's reserve pool is used to learn long-term memory ability; for classic ESN, a training set is constructed based on U3 in, is the true value y train (t+1), teacher network LSTM prediction value The prediction value of double loop ESN of teaching assistant network The weighted average of is calculated as follows: Where μ, v, o are the student distillation learning rates, and μ+v+o=1; according to Calculate the new reserve pool state vector x(t) using the following formula: Among them, f arcout is the inverse function of the output activation function, W out The output connection matrix obtained from ridge regression training; According to the new reservoir state vector x(t), calculate the new reservoir connection matrix W: W=[f arc (x(t))-W in u train (t)] / x(t-1) Among them, u train (t) is the input vector at time t, x(t-1) is the state vector of the reservoir at time t-1, and f arc is the inverse function of the reserve pool incentive function; W is scaled as follows to maintain the echo characteristics of the reservoir; Among them, sc is the scaling factor, λ max (W) is the maximum eigenvalue of W.