LSTM network model for enhancing short-term memory

Abstract

The invention provides an LSTM network model for enhancing short-term memory, and belongs to the technical field of network models.The LSTM network model comprises a data input module, a data calculation unit, a feature extraction module and an output module, the data input module is connected with the data input end of the data calculation unit, and the data input module is used for inputting external processing data; the output end of the data calculation unit is connected with the feature extraction module and the output module, the output end of the feature extraction module is connected with the input end of the data calculation unit, the data calculation unit is used for memorizing and forgetting information, and the feature extraction module extracts feature values from vectors output by the data calculation unit and outputs the feature values to the output module. And then the data is circulated back to the data calculation unit, so that the processed data can be memorized again, and the data processing efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of network models, and in particular to an LSTM network model for enhancing short-term memory. Background Art

[0002] The computing unit of the LSTM model is more complex than that of the RNN. The LSTM model is a recursive neural network that is particularly suitable for processing and predicting long-term dependency problems in time series data. However, the existing LSTM model requires more computing resources and parameter adjustment experience, and is prone to gradient vanishing or gradient explosion problems, which affects the training effect. For long sequence data, the computational complexity of LSTM is relatively high. Excessive reliance on initialization weights and parameters, different initializations may affect the final results. Therefore, it is necessary to design an LSTM network model that enhances short-term memory so that the model can converge quickly and improve data processing efficiency. Summary of the invention

[0003] The purpose of the present invention is to provide an LSTM network model with enhanced short-term memory, so as to solve the technical problems that the existing LSTM network model is prone to gradient disappearance and gradient explosion.

[0004] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0005] The invention discloses an LSTM network model for enhancing short-term memory, comprising a data input module, a data calculation unit, a feature extraction module and an output module. The data input module is connected to the data input end of the data calculation unit, the data input module is used to input external processing data, the output end of the data calculation unit is respectively connected to the feature extraction module and the output module, the output end of the feature extraction module is connected to the input end of the data calculation unit, the data calculation unit is used to realize the memory and forgetting of information, the feature extraction module extracts feature values ​​from the vector output by the data calculation unit, and then circulates back to the data calculation unit to realize the re-memory of the processed data, and at the same time improve the data processing efficiency.

[0006] Furthermore, the data calculation unit includes an input gate, a forget gate, an output gate and a temporary memory unit. In the calculation of the input gate, the forget gate, the output gate and the temporary memory unit, the characteristic value of the hidden state at time t-2 is added to improve data processing efficiency.

[0007] Furthermore, the feature extraction module is used to extract features from the output feature vector, thereby reducing the amount of data after feature extraction, and at the same time processing the feature values ​​of t-2 data at processing time t, doubling the processing data and improving efficiency.

[0008] Furthermore, the calculation process of the data calculation unit is:

[0009]

[0010]

[0011]

[0012]

[0013]

[0014] h t =o t *tanh(c t ) (6)

[0015] Among them, x t represents the input at time t, h t Represents the hidden state value at time t, and W has different subscripts.

[0016] The weights of different input values ​​are represented by b, and different subscripts represent different input biases. represents the cell state of the current input memory information, σ represents the gated activation function sigmoid, h' t-2 Represents the feature of the hidden unit value at time t-2, W h-2 , b h-2 They represent the parameters and bias of the hidden unit value at time t-2 respectively.

[0017] Furthermore, the verification process of the network model is:

[0018] 1. Data preparation and processing. According to the experimental plan, artificial data sets and natural data sets are processed. The artificial data sets are processed into experimental data by adding Gaussian noise and smoothing to simulate periodic and non-periodic sequence data, and the sequence data sets are processed into training data sets and test data sets. The natural data sets use wind power generation, solar power generation, and power load data, and the sequence data sets are processed into experimental data and divided into training data sets and test data sets.

[0019] 2. Model training: import the time series prediction training data set into the model, find suitable hyperparameters for model training, and export the training results;

[0020] 3. Model testing: test the model according to the training results, and calculate the test results based on the RMSE and R2 indicators;

[0021] 4. Model evaluation: save the model test results, and compare and evaluate the model convergence and prediction performance under different conditions.

[0022] The present invention has the following beneficial effects due to the adoption of the above technical solution:

[0023] The model of the present invention effectively handles the learning problem of long sequence dependencies, prevents gradient vanishing, has a faster convergence speed, and has stronger feature extraction and characterization capabilities. It is widely used in various time series tasks such as video, image, speech, finance, traffic, disease, climate, energy, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is the network structure diagram of the LSTM network model of the present invention;

[0025] Figure 2 It is a structural diagram of the calculation unit of the LSTM network model of the present invention. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and preferred embodiments. However, it should be noted that many details listed in the specification are only for the purpose of enabling the reader to have a thorough understanding of one or more aspects of the present invention, and these aspects of the present invention can be implemented even without these specific details.

[0027] like Figure 1-2 As shown, an LSTM network model for enhancing short-term memory includes a data input module, a data calculation unit, a feature extraction module and an output module. The data input module is connected to the data input end of the data calculation unit, the data input module is used to input external processing data, the output end of the data calculation unit is respectively connected to the feature extraction module and the output module, the output end of the feature extraction module is connected to the input end of the data calculation unit, the data calculation unit is used to realize the memory and forgetting of information, the feature extraction module extracts the eigenvalue of the vector output by the data calculation unit, and then circulates it back to the data calculation unit to realize the re-memory of the processed data and improve the data processing efficiency at the same time.

[0028] In an embodiment of the present invention, the data calculation unit includes an input gate, a forget gate, an output gate and a temporary memory unit. In the calculation of the input gate, the forget gate, the output gate and the temporary memory unit, the characteristic value of the hidden state at time t-2 is added to improve data processing efficiency.

[0029] In the embodiment of the present invention, the feature extraction module is used to extract features from the output feature vector, reduce the amount of data after feature extraction, and simultaneously process the feature values ​​of t-2 data at processing time t, doubling the processed data and improving efficiency.

[0030] In the embodiment of the present invention, Figure 2 As shown, the calculation process of the data calculation unit is:

[0031]

[0032]

[0033]

[0034]

[0035]

[0036] h t =o t *tanh(c t ) (6)

[0037] Among them, x t represents the input at time t, h t represents the hidden state value at time t, W with different subscripts represents the weights of different input values, and b with different subscripts represents the bias of different inputs. represents the cell state of the current input memory information, σ represents the gated activation function sigmoid, h' t-2 Represents the feature of the hidden unit value at time t-2, W h-2 , b h-2 They represent the parameters and bias of the hidden unit value at time t-2 respectively.

[0038] In the embodiment of the present invention, the verification process of the network model is as follows:

[0039] 1. Data preparation and processing. According to the experimental plan, artificial data sets and natural data sets are processed. The artificial data sets are processed into experimental data by adding Gaussian noise and smoothing to simulate periodic and non-periodic sequence data, and the sequence data sets are processed into training data sets and test data sets. The natural data sets use wind power generation, solar power generation, and power load data, and the sequence data sets are processed into experimental data and divided into training data sets and test data sets.

[0040] 2. Model training: import the time series prediction training data set into the model, find suitable hyperparameters for model training, and export the training results;

[0041] 3. Model testing: test the model according to the training results, and calculate the test results based on the RMSE and R2 indicators;

[0042] 4. Model evaluation: save the model test results, and compare and evaluate the model convergence and prediction performance under different conditions.

[0043] This application is widely used in various time series tasks such as video, image, speech, finance, traffic, disease, climate, energy, etc. The model of this application effectively handles the learning problem of long sequence dependence, prevents the gradient disappearance problem, has a faster convergence speed, and has stronger feature extraction and representation capabilities.

[0044] The data set processed by this application model is as follows:

[0045] 1. Artificially constructed periodic dataset based on Sinn function;

[0046] 2. Artificially constructed non-periodic data sets based on polynomial functions;

[0047] 3. Kuwait region hourly wind power generation dataset in 2019;

[0048] 4. Hourly electricity load dataset for France, Norway and Serbia in 2021.

[0049] Table 1 shows the data results.

[0050]

[0051] From the above data, we can see that the model of this application achieves a faster convergence speed and has stronger feature extraction and characterization capabilities.

[0052] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. An LSTM network model for enhancing short-term memory, characterized in that: It includes a data input module, a data calculation unit, a feature extraction module and an output module. The data input module is connected to the data input end of the data calculation unit. The data input module is used to input external processing data. The output end of the data calculation unit is respectively connected to the feature extraction module and the output module. The output end of the feature extraction module is connected to the input end of the data calculation unit. The data calculation unit is used to realize the memory and forgetting of information. The feature extraction module extracts the feature value of the vector output by the data calculation unit, and then circulates it back to the data calculation unit to realize the re-memory of the processed data and improve the data processing efficiency.

2. The LSTM network model for enhancing short-term memory according to claim 1, characterized in that: The data calculation unit includes an input gate, a forget gate, an output gate and a temporary memory unit. In the calculation of the input gate, the forget gate, the output gate and the temporary memory unit, the characteristic value of the hidden state at time t-2 is added to improve the data processing efficiency.

3. According to claim 1, a LSTM network model for enhancing short-term memory, characterized in that: The feature extraction module is used to extract features from the output feature vector. After feature extraction, the amount of data is reduced. At the same time, the feature values ​​of t-2 data are processed together when processing time t, which doubles the processing data and improves efficiency.

4. The LSTM network model for enhancing short-term memory according to claim 1, characterized in that: The calculation process of the data calculation unit is: h t =o t *tanh(c t ) (6) Among them, x t represents the input at time t, h t represents the hidden state value at time t, W with different subscripts represents the weights of different input values, and b with different subscripts represents the bias of different inputs. represents the cell state of the current input memory information, σ represents the gated activation function sigmoid, h' t-2 Represents the feature of the hidden unit value at time t-2, W h-2 、b h-2 They represent the parameters and bias of the hidden unit value at time t-2 respectively.

5. The LSTM network model for enhancing short-term memory according to claim 1, characterized in that: The verification process of the network model is:

1. Data preparation and processing. According to the experimental plan, artificial data sets and natural data sets are processed. The artificial data sets are processed into experimental data by adding Gaussian noise and smoothing to simulate periodic and non-periodic sequence data, and the sequence data sets are processed into training data sets and test data sets. The natural data sets use wind power generation, solar power generation, and power load data, and the sequence data sets are processed into experimental data and divided into training data sets and test data sets.

2. Model training: import the time series prediction training data set into the model, find suitable hyperparameters for model training, and export the training results; 3. Model testing: test the model according to the training results, and calculate the test results based on the RMSE and R2 indicators; 4. Model evaluation: save the model test results, and compare and evaluate the model convergence and prediction performance under different conditions.

Images