MSDBA-LSTM pollutant concentration prediction method based on self-attention

Abstract

The present invention discloses a MSDBA-LSTM pollutant concentration prediction method based on self-attention, comprising the steps of obtaining initial monitoring data, constructing an MSDBA algorithm, extracting features from the initial monitoring data, outputting the features as LSTM memory information, inputting the LSTM memory information into a self-attention mechanism, obtaining an attention value, constructing an MSDBA-LSTM pollutant concentration prediction model based on the self-attention mechanism, enhancing the global search and local development capabilities of a BA algorithm by simulating a prey defense mechanism, introducing an LSTM network based on a self-attention mechanism, dynamically allocating weights to hidden states of the LSTM network, strengthening the influence of key features, accurately modeling the nonlinear time series relationship between pollutant concentration monitoring and influencing factors, improving prediction accuracy, combining the MSDBA algorithm with an LSTM network based on a self-attention mechanism, constructing an MSDBA-LSTM pollutant concentration prediction model based on self-attention, automatically optimizing hyperparameters of the LSTM network, reducing manual intervention, improving prediction accuracy and efficiency, and reducing uncertainty in model construction.

Description

Technical Field

[0001] The present invention relates to the field of concentration detection technology, and in particular to a MSDBA-LSTM pollutant concentration prediction method based on self-attention. Background Art

[0002] The rapid development of the economy has increased the demand for energy, and the extensive development of the energy industry has caused a series of environmental air pollution problems. The concentration of various pollutant gases emitted during industrial production and coal-fired power generation has increased, and a large amount of pollutants such as sulfur dioxide (SO2), nitrogen oxides (NOx) and particulate matter have reduced air quality and harmed human health. Therefore, studying the composition and content of various gaseous pollutants emitted by industry and predicting future concentration changes are important for the prevention and monitoring of atmospheric pollutants.

[0003] The content of atmospheric pollutants changes continuously and dynamically over time, which is a typical time series data. How to build a time series data prediction model, analyze historical data, capture the characteristics of dynamic changes, find the correlation of data in time series, and predict data at future times is crucial to preventing air pollution. The concentration of nitrogen oxide emissions monitored online changes continuously and dynamically in the time dimension. The time series prediction model needs to consider the trend changes of time series data and the correlation of adjacent time observations. The existing machine learning-based methods do not consider the time dependence of historical data and sequences when learning data features, and it is difficult to handle multivariate, nonlinear, and irregular time series data, resulting in inaccurate prediction results. Summary of the invention

[0004] The technical problem solved by the present invention is that the machine learning-based methods in the prior art do not consider the time dependence of historical data and sequences when learning data features, and are difficult to process multivariate, nonlinear, and irregular time series data, resulting in inaccurate prediction results.

[0005] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0006] The MSDBA-LSTM pollutant concentration prediction method based on self-attention includes the following steps:

[0007] Step S1, obtaining initial monitoring data;

[0008] Step S2, constructing the MSDBA algorithm;

[0009] Step S3, extracting features from the initial monitoring data, performing nonlinear transformation on the initial monitoring data according to the internal activation function of the LSTM model, learning the relationship between different features, and outputting them as LSTM memory information;

[0010] Step S4, input the LSTM memory information into the self-attention mechanism, build an LSTM network based on the self-attention mechanism, calculate the Query value, Key value and Value value of the LSTM memory information respectively, calculate the similarity and attention weight of the Query value and the Key value, sum them up to get the attention value, map the attention value to the predicted value and output it;

[0011] Step S5, construct an MSDBA-LSTM pollutant concentration prediction model based on the self-attention mechanism.

[0012] Preferably, the step S1 specifically includes:

[0013] The initial monitoring data include atmospheric pollutant nitrogen oxide concentration monitoring value data, flue gas temperature data, flue gas humidity and flue gas static pressure data, and the atmospheric pollutant nitrogen oxide concentration monitoring value data, flue gas temperature data, flue gas humidity and flue gas static pressure data are merged with time data.

[0014] Preferably, the step S2 specifically includes:

[0015] Defense strategies are introduced into the global exploration phase and the local development phase of the MSDBA algorithm. The defense strategies include multi-strategy defense in the global exploration phase and multi-strategy defense in the local development phase.

[0016] Preferably, the multi-strategy defense in the global exploration phase includes:

[0017] The mathematical expressions for the speed and position update of the MSDBA algorithm in the global search phase are:

[0018]

[0019] in, For The frequency value randomly selected in the interval, , is a random variable It follows a uniform distribution on the interval (0,1). and are the position and velocity of the i-th bat in the t-th iteration, is the position of the i-th bat in the t-1th iteration, is the optimal solution currently found.

[0020] The multi-strategy defense in the global exploration phase includes the first defense strategy and the second defense strategy. and The size of determines whether to execute the first defense strategy or the second defense strategy:

[0021] when When , the first defense strategy is executed, and the mathematical expression of position update is:

[0022]

[0023] in, is a random number that follows a normal distribution, is a random number in the interval (0,1), N is the population size, r is a random number in the interval [1,N], is an individual randomly selected from the population, is the position generated between the current individual and an individual randomly selected from the population.

[0024] At this time, the position of the i-th individual at the next moment With the current time position It is related to the distance between the current optimal solution and the predator. When , the individual's position at the next moment will be close to the current optimal solution. When the individual's position at the next moment It will be far away from the current optimal solution.

[0025] when When , the second defense strategy is executed, and the mathematical expression of position update is:

[0026]

[0027] in, is a random number in the interval (0,1), p and q are random numbers in the interval [1,N], and are the positions of the i-th bat in the p-th iteration and the q-th iteration, respectively. For multidimensional vector A vector with values ​​of 0 or 1 in each dimension.

[0028] like If all elements are 0, the second defense strategy is successful and the predator no longer moves toward the prey. If the element in contains 1, the predator moves towards the prey. , predators will approach bat populations if

[0029] , predators will stay away from bat populations.

[0030] Preferably, the multi-strategy defense in the local development stage is:

[0031] like , , then the bat optimization algorithm performs the local development phase around the optimal solution neighborhood, and the mathematical expression of position update is:

[0032]

[0033] in, is a multidimensional random vector, the values ​​of each dimension of the vector belong to the interval -1 to 1, is the average loudness of all individuals at the tth iteration.

[0034] The local development phase includes multiple strategies including the third defense strategy and the fourth defense strategy, which compares random numbers and The size of the defense strategy determines whether to implement the third or fourth defense strategy.

[0035] when When , the third defense strategy is executed, and the mathematical expression of position update is:

[0036]

[0037] in, is a random number between [1,N], For the i The bat in the The position in the iteration, is a random number in the interval (0,1). is the parameter vector used to control the search direction, As a defense factor, is the odor diffusion factor, For the The individual in The fitness of the next moment, is a random number in the interval (0,1). is a vector whose element value is 0 or 1. is the maximum number of iterations, exp{} is the base e raised to the power of {}, rand is a random number, the random number is ≤0.5, , random number>0.5, .

[0038] This function is the x position update function, and the variables involved are , and The three factors adjust the position, thus acting together to update the x position, mainly based on , and at the same time simulate the prey releasing gas and making corresponding defensive actions, changing the predator's search direction, thereby changing the positions of the predator and prey.

[0039] when When the element is 0, the predator stops moving. When the element contains 1, the predator keeps its distance from the prey. The larger it is, the closer the individual tends to be to the optimal solution.

[0040] when When , the fourth defense strategy is executed, and the mathematical expression of position update is:

[0041]

[0042] in, is a random number in the interval (0,1). and are random numbers in the interval (0,1), The inelastic collision force generated when the prey attacks the predator with its body. is a random vector whose element values ​​are in the interval [0,1], For the quality of the predator.

[0043] Preferably, the LSTM model in step S3 includes a forget gate, an input gate and an output gate:

[0044] The forget gate is used to determine whether the state information in the memory unit is retained. The state information in the memory is passed to the current memory unit through the forget gate. , the mathematical expression is:

[0045]

[0046] in, is the forget gate at the current time t, is the sigmoid activation function, with a value in the (0,1) interval. and are the weight matrix and bias term of the hidden layer of the forget gate, is the hidden unit state at the previous moment, the hidden unit state at the previous moment and current time input information They are passed together as input information to the forget gate hidden layer.

[0047] The input gate is used to update the memory unit state and determine the updated input information at the current moment to be stored in the memory unit. , the mathematical expression is:

[0048]

[0049] in, is the input gate at the current time t, is the sigmoid activation function, with a value in the (0,1) interval. and are the weight matrix and bias term of the input gate hidden layer respectively.

[0050] The output gate is used to determine the memory unit at the current moment. Can the state information in be updated to the hidden unit? , which is calculated as follows:

[0051]

[0052] in, is the output gate at the current time t, is the sigmoid activation function, with a value in the (0,1) interval. and are the weight matrix and bias term of the output gate hidden layer respectively.

[0053] Use the tanh function to generate the memory unit at the current moment , the input gate With the current memory unit Multiply and store the last moment memory unit With the forget gate Multiply them to determine what information is updated at the current moment, and the memory unit state is updated as follows:

[0054]

[0055] Use the tanh function to store the current moment memory unit Normalization, and output gate Multiply, the output hidden unit state for:

[0056] .

[0057] Preferably, step S4 includes the following sub-steps:

[0058] Step S401, the calculation process of the self-attention mechanism includes:

[0059] Calculate the similarity between the Query value and the Key value. The mathematical expression is:

[0060]

[0061] in, is the feature vector of the input data, , For the origin Query value, For the origin Key value, Calculate similarity for the dot product, for The similarity with other feature vectors, , is the learnable parameter matrix.

[0062] Calculate the attention weight, the mathematical expression is:

[0063]

[0064] in is the normalization function, N is the number of input feature vectors, is the attention weight;

[0065] Calculate the attention value, the mathematical expression is:

[0066]

[0067] in is the learnable parameter matrix, For the The Value value corresponding to the input vector.

[0068] Step S302, map the attention value to a predicted value and output it.

[0069] Preferably, step S5 includes the following sub-steps:

[0070] Step S501, all individuals are randomly initialized by the MSDBA algorithm, and each individual is initialized with a corresponding hyperparameter combination of the LSTM network based on the self-attention mechanism, including the number of hidden layers, the number of neurons in the hidden layer, the learning rate, the number of iterations, and the batch size.

[0071] Step S502, the LSTM network based on the self-attention mechanism constructs a corresponding network structure according to the hyperparameter combination of the individual MSDBA algorithm, and each individual of the MSDBA algorithm corresponds to an LSTM network model based on the self-attention mechanism.

[0072] In step S503, each LSTM network model based on the self-attention mechanism is trained on the training data set, and after the training is completed, prediction is performed on the test data set, and the mean square error is calculated.

[0073] Step S504: The MSDBA algorithm uses the mean square error as fitness, calculates the fitness values ​​of all individuals, and updates the current optimal individual.

[0074] Step S505, the MSDBA algorithm executes the global exploration or local development phase, updates the frequency, speed and position of the individual, and returns to step S503 for a new round of iteration.

[0075] Step S506, after the MSDBA algorithm reaches the maximum number of iterations, the global optimal solution at this time represents the optimal hyperparameter configuration of the LSTM network based on the self-attention mechanism. The initial monitoring data is predicted and output using the optimal hyperparameter configuration.

[0076] The beneficial effects of the present invention are as follows: an MSDBA algorithm is proposed, which enhances the global search and local development capabilities of the BA algorithm by simulating the prey defense mechanism, solves the problems of slow convergence, low precision, and easy local extreme value, improves the diversity of the solution space, introduces an LSTM network based on a self-attention mechanism, dynamically allocates weights to the hidden states of the LSTM network, strengthens the influence of key features, ignores useless information, accurately models the nonlinear time series relationship between pollutant concentration monitoring and influencing factors, improves prediction accuracy, combines the MSDBA algorithm with the LSTM network based on the self-attention mechanism, constructs a MSDBA-LSTM pollutant concentration prediction model based on self-attention, automatically optimizes the hyperparameters of the LSTM network, reduces manual intervention, improves prediction accuracy and efficiency, and reduces the uncertainty of model construction. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] Figure 1 A flowchart of the steps of a self-attention-based MSDBA-LSTM pollutant concentration prediction method provided by one embodiment of the present invention;

[0078] Figure 2 A self-attention mechanism structure of a self-attention-based MSDBA-LSTM pollutant concentration prediction method provided by an embodiment of the present invention;

[0079] Figure 3 An LSTM network model structure based on self-attention for a MSDBA-LSTM pollutant concentration prediction method based on self-attention provided by an embodiment of the present invention;

[0080] Figure 4 A flow chart of a MSDBA-LSTM pollutant concentration prediction model based on self-attention for a MSDBA-LSTM pollutant concentration prediction method based on self-attention provided in one embodiment of the present invention. DETAILED DESCRIPTION

[0081] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the drawings. It is obvious that the described embodiments are only part of the embodiments of the present invention, but not all of the embodiments.

[0082] Embodiment 1, refer to Figure 1 , provides a MSDBA-LSTM pollutant concentration prediction method based on self-attention, and the MSDBA-LSTM pollutant concentration prediction method based on self-attention includes the following steps:

[0083] Step S1, obtaining initial monitoring data.

[0084] Step S2, constructing the MSDBA algorithm.

[0085] Step S3, extract features from the initial monitoring data, perform nonlinear transformation on the initial monitoring data according to the internal activation function of the LSTM model, learn the relationship between different features, and output them as LSTM memory information.

[0086] Step S4, input the LSTM memory information into the self-attention mechanism, build an LSTM network based on the self-attention mechanism, calculate the Query value, Key value and Value value of the LSTM memory information respectively, and calculate the similarity and attention weight of the Query value and the Key value, sum them up to get the attention value, map the attention value to the predicted value and output it.

[0087] Step S5, construct an MSDBA-LSTM pollutant concentration prediction model based on the self-attention mechanism.

[0088] By combining the MSDBA algorithm with the LSTM network based on the self-attention mechanism, a self-attention-based MSDBA-LSTM pollutant concentration prediction model is constructed to automatically optimize the hyperparameters of the LSTM network, reduce manual intervention, improve prediction accuracy and efficiency, and reduce the uncertainty of model construction.

[0089] Step S1 specifically includes:

[0090] The initial monitoring data include atmospheric pollutant nitrogen oxide concentration monitoring value data, flue gas temperature data, flue gas humidity and flue gas static pressure data, and the atmospheric pollutant nitrogen oxide concentration monitoring value data, flue gas temperature data, flue gas humidity and flue gas static pressure data are merged with time data.

[0091] The effect of LSTM in predicting pollutant concentration monitoring values ​​depends on the configuration of hyperparameter values, including the number of hidden layers, the number of hidden layer neurons, the learning rate, the number of iterations, the batch size, etc. Setting appropriate hyperparameter values ​​is crucial to improving the LSTM prediction effect. Otherwise, it may lead to training timeout and decreased accuracy. In order to achieve accurate prediction of nitrogen oxide concentration monitoring values, it is necessary to find the optimal combination of hyperparameters.

[0092] First, a Bat Algorithm based on Multi-stage Defense Strategy (MSDBA) is proposed to quickly and accurately find the optimal solution within the scope of the objective function. Then, an LSTM network based on the self-attention mechanism (Self-Attention LSTM) is proposed to learn the internal associations of the input data and assign different weights to reduce the prediction error. Finally, a MSDBA-LSTM pollutant concentration prediction model based on self-attention (MSDBA-Self Attention-LSTM) is constructed. The iterative optimization capability of MSDBA is used to solve the Self Attention-LSTM hyperparameters, reduce the impact of manual parameter adjustment on the model, and improve the accuracy and efficiency of the model.

[0093] As a meta-heuristic algorithm based on population, the BA algorithm solves the optimal value of the objective function by simulating the echolocation behavior of bats in nature. In the process of optimization, it relies on a unique search mechanism to continuously change its own state, so as to better adapt to changes in the external environment and finally find the optimal solution. However, the global search of the population in the BA algorithm relies on random walks, and the speed and displacement cannot be corrected according to the environmental state. The local development step of the population gradually decreases according to the loudness, and it is difficult to jump out of the local optimal area. Therefore, the MSDBA algorithm is proposed. MSDBA introduces four defense strategies into the global exploration and local development stages of the BA algorithm, simulating the physical behavior of prey to avoid predators through vision, sound, smell and physical attacks, and prompting individual bats to search the solution space on a larger scale.

[0094] Step S2 specifically includes:

[0095] Defense strategies are introduced into the global exploration phase and the local development phase of the MSDBA algorithm. The defense strategies include multi-strategy defense in the global exploration phase and multi-strategy defense in the local development phase.

[0096] The multi-strategy defense in the global exploration phase includes:

[0097] The mathematical expressions for the speed and position update of the MSDBA algorithm in the global search phase are:

[0098]

[0099] in, For The frequency value randomly selected in the interval, is a random variable It follows a uniform distribution on the interval (0,1). and are the position and velocity of the i-th bat in the t-th iteration, is the position of the i-th bat in the t-1th iteration, is the optimal solution currently found.

[0100] It can be seen from the above formula that the update of individual frequency is highly random. This random selection method cannot provide guidance for the adjustment of speed. In addition, the speed increment is limited to the communication between individuals and the optimal individual. There is a lack of learning about the movement path of the population, which increases the degree of deviation in the search direction. Combined with the frequency of random values, the global search is blind. Two defense strategies are set for the prey, namely the first defense strategy and the second defense strategy. The defense strategy to be executed is determined by comparing the sum of the two random numbers.

[0101] When the prey becomes aware of the bat, the bat has two options. The first option is that the bat moves towards the prey, the distance between the two decreases, and the bat gradually explores the vicinity of the prey to speed up convergence. On the contrary, in the second option, the bat moves away from the prey, the distance between the two increases, and the bat searches a more distant space to obtain more candidate solutions.

[0102] The multi-strategy defense in the global exploration phase includes the first defense strategy and the second defense strategy. and The size of determines whether to execute the first defense strategy or the second defense strategy:

[0103] when When , the first defense strategy is executed, and the mathematical expression of position update is:

[0104]

[0105] in, is a random number that follows a normal distribution, is a random number in the interval (0,1), N is the population size, r is a random number in the interval [1,N], is an individual randomly selected from the population, is the position generated between the current individual and an individual randomly selected from the population.

[0106] At this time, the position of the i-th individual at the next moment With the current time position It is related to the distance between the current optimal solution and the predator. When the individual's position at the next moment will be close to the current optimal solution. When the individual's position at the next moment It will be far away from the current optimal solution, which will prompt the BA algorithm to avoid falling into the local optimum with a certain probability.

[0107] when When , the second defense strategy is executed, and the mathematical expression of position update is:

[0108]

[0109] in, is a random number in the interval (0,1), p and q are random numbers in the interval [1,N], and are the positions of the i-th bat in the p-th iteration and the q-th iteration, respectively. For multidimensional vector A vector with values ​​of 0 or 1 in each dimension.

[0110] like If all elements are 0, the second defense strategy is successful and the predator no longer moves toward the prey. If the element in contains 1, the predator moves towards the prey. , predators will approach bat populations if

[0111] , predators will stay away from the bat population to prevent the population from being too concentrated near the current optimal solution, thus avoiding falling into the local optimum.

[0112] The multi-strategy defense in the local development stage is:

[0113] like , , then the bat optimization algorithm performs the local development phase around the optimal solution neighborhood, and the mathematical expression of position update is:

[0114]

[0115] in, is a multidimensional random vector, the values ​​of each dimension of the vector belong to the interval -1 to 1, is the average loudness of all individuals at the tth iteration.

[0116] It can be seen from the above formula that the development step of an individual around the optimal solution is mainly determined by loudness, which decreases rapidly with the progress of iterations. The change of step length lacks flexibility, which easily causes the population to gather around the local extreme value and makes it difficult to break through the constraints of the local extreme value. Two defense strategies are set for the prey, namely the third defense strategy and the fourth defense strategy. The defense strategy to be executed is decided by comparing the size of the sum of two random numbers.

[0117] The local development phase includes multiple strategies including the third defense strategy and the fourth defense strategy, which compares random numbers and The size of the defense strategy determines whether to implement the third or fourth defense strategy.

[0118] when When , the third defense strategy is executed, and the mathematical expression of position update is:

[0119]

[0120] in, is a random number between [1,N], For the The bat in the The position in the iteration, is a random number in the interval (0,1). is the parameter vector used to control the search direction, As a defense factor, is the odor diffusion factor, For the The individual in The fitness of the next moment, is a random number in the interval (0,1). is a vector whose element value is 0 or 1. is the maximum number of iterations, exp{} is the base e raised to the power of {}, rand is a random number, the random number is ≤0.5, , random number>0.5, .

[0121] when When the element is 0, the predator stops moving. When the element contains 1, the predator keeps its distance from the prey. The larger it is, the closer the individual tends to be to the optimal solution.

[0122] when When , the fourth defense strategy is executed, and the mathematical expression of position update is:

[0123]

[0124] in, is a random number in the interval (0,1). and are random numbers in the interval (0,1), The inelastic collision force generated when the prey attacks the predator with its body. is a random vector whose element values ​​are in the interval [0,1], For the quality of the predator.

[0125] The physical attack behavior of prey on predator is simulated. When the predator is very close, the prey will take physical attack. During the physical attack, the prey and predator merge strongly, resulting in inelastic collision, which helps to generate more widely distributed candidate solutions in the search space, thereby conducting a wider search in the space adjacent to the current optimal solution.

[0126] LSTM is a variant of RNN (Recurrent Neural Network, RNN) that can learn long-term dependent information. The generation of atmospheric pollutants, nitrogen oxides, is the result of the coupling of multiple factors including time. The concentration of nitrogen oxide emissions at each moment will be affected to varying degrees by the parameters of the previous moments, constituting a time series prediction problem, which creates the prerequisite for using LSTM as a prediction model.

[0127] Through three control gates, the LSTM neural network can selectively forget and retain information in the neuron state, realize the input and outflow of memory information and control data, and allow internal information to selectively enter the next moment.

[0128] The LSTM model in step S3 includes a forget gate, an input gate, and an output gate:

[0129] The forget gate is used to determine whether the state information in the memory unit is retained. At the current moment, the memory unit at the previous moment The state information in the memory is passed to the current memory unit through the forget gate. , the mathematical expression is:

[0130]

[0131] in, is the forget gate at the current time t, is the sigmoid activation function, with a value in the (0,1) interval. and are the weight matrix and bias term of the hidden layer of the forget gate, is the hidden unit state at the previous moment, the hidden unit state at the previous moment and current time input information They are passed together as input information to the forget gate hidden layer.

[0132] The input gate is used to update the state of the memory unit and determine the updated input information at the current moment to be stored in the memory unit. , the mathematical expression is:

[0133]

[0134] in, is the input gate at the current time t, is the sigmoid activation function, with a value in the (0,1) interval. and are the weight matrix and bias term of the input gate hidden layer respectively.

[0135] The output gate is used to determine the memory cell at the current moment Can the state information in be updated to the hidden unit? , which is calculated as follows:

[0136]

[0137] in, is the output gate at the current time t, is the sigmoid activation function, with a value in the (0,1) interval. and are the weight matrix and bias term of the output gate hidden layer respectively.

[0138] Use the tanh function to generate the memory unit at the current moment , the input gate With the current memory unit Multiply and store the last moment memory unit With the forget gate Multiply them to determine what information is updated at the current moment, and the memory unit state is updated as follows:

[0139]

[0140] Use the tanh function to store the current moment memory unit Normalization, and output gate Multiply, the output hidden unit state for:

[0141] .

[0142] Therefore, the LSTM neural network controls the abandonment and retention of memory unit state information through the forget gate, input gate and output gate, generates long-term memory and short-term memory, can remember longer historical information, and effectively fit the changing trends of long-term and short-term time series.

[0143] When using LSTM neural network to predict the concentration of atmospheric pollutants, the prediction weight of data at any position in the historical sequence for future data is the same, ignoring the specific impact of relevant influencing factors on the different components of the time series, making it difficult to explain the correlation between the model input and output. Therefore, based on the excellent information selection and time series fitting capabilities of LSTM, this paper introduces the self-attention mechanism and proposes the Self-Attention LSTM neural network. Different weights are given to the output influencing factors of LSTM prediction of atmospheric pollutant concentrations, highlighting more critical influencing factors, which is conducive to the model making more accurate judgments.

[0144] The attention mechanism refers to the ability of the brain to selectively focus on certain important information and ignore other information when processing information. In neural networks, the attention mechanism gives different weights to the input data of the model, which is used to increase the model's attention to certain important data in the input data, i.e., the weight, and can more accurately predict the output results. By weighting the input data, the model can pay more attention to information that is highly important to the current output. Introducing the attention mechanism into the LSTM model can strengthen the weight of the more important features for predicting the concentration of atmospheric pollutants and ignore the impact of useless information.

[0145] There are many types and variations of attention mechanisms, among which the self-attention mechanism is one of them. The structure is as follows Figure 2 shown.

[0146] Step S4 includes the following sub-steps:

[0147] Step S401, the calculation process of the self-attention mechanism includes:

[0148] Calculate the similarity between the Query value and the Key value. The mathematical expression is:

[0149]

[0150] in, is the feature vector of the input data, , For the origin Query value, For the origin Key value, Calculate similarity for the dot product, for The similarity with other feature vectors, , is the learnable parameter matrix.

[0151] The self-attention mechanism pays more attention to the correlation between input features. The core is to assign weights and calculate attention weights. The mathematical expression is:

[0152]

[0153] in is the normalization function, N is the number of input feature vectors, is the attention weight;

[0154] Calculate the attention value, the mathematical expression is:

[0155]

[0156] in is the learnable parameter matrix, For the The Value value corresponding to the input vector.

[0157] Step S302, map the attention value to a predicted value and output it.

[0158] Therefore, the constructed LSTM network model based on self-attention includes input layer, LSTM layer, self-attention layer, fully connected layer and output layer. The structure is as follows: Figure 3 As shown:

[0159] (1) Input layer: The atmospheric pollutant nitrogen oxide concentration monitoring data, flue gas temperature data, flue gas humidity, flue gas static pressure data and time data are combined as the input of the network model. The input data of the network model of the present invention are: the nitrogen oxide monitoring data of the enterprise at 24 moments within one day before the forecast day (monitoring every hour), and the flue gas humidity, flue gas temperature and flue gas static pressure on the forecast day and the day before.

[0160] (2) LSTM layer: The LSTM network is used to extract features of the input nitrogen oxide monitoring value, time, humidity, temperature, and static pressure. A nonlinear transformation is performed through the internal activation function to learn the relationship between different features and output h.

[0161] (3) Self-attention layer: The input of the self-attention layer is the output h of the LSTM layer. The query value, key value, and value value of h are calculated respectively. Then, the similarity and attention weight are calculated, and the sum is obtained to obtain the attention value, and the output is b.

[0162] (4) Fully connected layer: The fully connected layer is used to map the output results of the self-attention layer to predicted values, thereby strengthening the learning of features by the LSTM and self-attention mechanisms.

[0163] Output layer: The model outputs the monitoring value of nitrogen oxide concentration of atmospheric pollutants through the output layer. The output data is the nitrogen oxide concentration detection value of the enterprise at 24 times on the day to be predicted.

[0164] Finally, the MSDBA-Self Attention-LSTM pollutant concentration prediction model is constructed based on the MSDBA algorithm and the Self-Attention LSTM network. MSDBA is used to find the optimal number of hidden layers, number of hidden layer neurons, learning rate, number of iterations, batch size and other hyperparameters of the Self-Attention LSTM network model. The specific process of constructing the MSDBA-SelfAttention-LSTM model is as follows: Figure 4 shown.

[0165] Step S5 includes the following sub-steps:

[0166] Step S501, all individuals are randomly initialized by the MSDBA algorithm, and each individual is initialized with a corresponding hyperparameter combination of the LSTM network based on the self-attention mechanism, including the number of hidden layers, the number of neurons in the hidden layer, the learning rate, the number of iterations, and the batch size.

[0167] Step S502, the LSTM network based on the self-attention mechanism constructs a corresponding network structure according to the hyperparameter combination of the individual MSDBA algorithm, and each individual of the MSDBA algorithm corresponds to an LSTM network model based on the self-attention mechanism.

[0168] In step S503, each LSTM network model based on the self-attention mechanism is trained on the training data set, and after the training is completed, prediction is performed on the test data set, and the mean square error is calculated.

[0169] Step S504: The MSDBA algorithm uses the mean square error as fitness, calculates the fitness values ​​of all individuals, and updates the current optimal individual.

[0170] Step S505, the MSDBA algorithm executes the global exploration or local development phase, updates the frequency, speed and position of the individual, and returns to step S503 for a new round of iteration.

[0171] Step S506, after the MSDBA algorithm reaches the maximum number of iterations, the global optimal solution at this time represents the optimal hyperparameter configuration of the LSTM network based on the self-attention mechanism. The initial monitoring data is predicted and output using the optimal hyperparameter configuration.

[0172] In order to solve the problems of slow convergence speed, low optimization accuracy and easy falling into local extreme value of BA algorithm, a BA algorithm based on multi-stage defense strategy, namely MSDBA algorithm, is proposed. Four defense strategies are introduced in the global exploration and local development stages to simulate the physical behavior of prey avoiding predators by using vision, sound, smell and physical attack, and adaptively change the search position according to the distance between the individual and the optimal solution, so as to generate candidate solutions with wider distribution in the search space and avoid premature convergence.

[0173] Aiming at the problem that LSTM network cannot assign corresponding weights to key information according to the importance of each data to the prediction results, an LSTM network model based on self-attention mechanism (Self Attention-LSTM) is proposed to realize the modeling of each component of the pollutant concentration monitoring sequence and the corresponding influencing factors. Different weights are assigned to the hidden state of the LSTM layer output to strengthen the influence of key features, especially to strengthen the weight of important features and ignore the influence of useless information. The nonlinear time series relationship between each component of pollutant concentration monitoring and each influencing factor is learned, and the attention to the information with high importance of the current output is strengthened, so as to improve the ability of the prediction model to process data.

[0174] Since the determination of LSTM hyperparameters requires repeated manual experiments and has strong randomness and uncertainty, a pollutant concentration prediction model based on MSDBA-Self Attention-LSTM is proposed. The selection of hyperparameters is modeled as a multivariable combination optimization problem. MSDBA is used to solve the optimal number of hidden layers, number of hidden layer neurons, learning rate, number of iterations and batch size hyperparameters of the Self Attention-LSTM network model. According to the prediction effect of Self Attention-LSTM on pollutant concentration, the optimal parameter combination is iteratively solved. Only a small range of parameter values ​​needs to be set in the early stage to train the model by itself, which reduces the uncertainty of manual selection and improves the prediction accuracy of the model.

[0175] It should be understood by those skilled in the art that the embodiments of the present invention may be provided as methods, systems or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware aspects. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media containing computer-usable program codes. Among them, the storage medium may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (Static Random Access Memory, referred to as SRAM), electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, referred to as EEPROM), erasable programmable read-only memory (Erasable Programmable Read Only Memory, referred to as EPROM), programmable read-only memory (Programmable Read-Only Memory, referred to as PROM), read-only memory (Read-Only Memory, referred to as ROM), magnetic memory, flash memory, magnetic disk or optical disk. These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0176] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. The MSDBA-LSTM pollutant concentration prediction method based on self-attention is characterized by: The steps include: Step S1, obtaining initial monitoring data; Step S2, constructing the MSDBA algorithm; Step S3, extracting features from the initial monitoring data, performing nonlinear transformation on the initial monitoring data according to the internal activation function of the LSTM model, learning the relationship between different features, and outputting them as LSTM memory information; Step S4, input the LSTM memory information into the self-attention mechanism, build an LSTM network based on the self-attention mechanism, calculate the Query value, Key value and Value value of the LSTM memory information respectively, calculate the similarity and attention weight of the Query value and the Key value, sum them up to get the attention value, map the attention value to the predicted value and output it; Step S5, constructing a MSDBA-LSTM pollutant concentration prediction model based on the self-attention mechanism; The step S1 specifically includes: The initial monitoring data includes atmospheric pollutant nitrogen oxide concentration monitoring value data, flue gas temperature data, flue gas humidity and flue gas static pressure data, and the atmospheric pollutant nitrogen oxide concentration monitoring value data, flue gas temperature data, flue gas humidity and flue gas static pressure data are combined with the time data; The step S2 specifically includes: Introducing defense strategies into the global exploration phase and the local development phase of the BA algorithm, wherein the defense strategies include multi-strategy defense in the global exploration phase and multi-strategy defense in the local development phase; The multi-strategy defense in the global exploration phase includes: The mathematical expressions for the speed and position update of the MSDBA algorithm in the global search phase are: in, and are the minimum and maximum frequencies respectively, For The frequency value randomly selected in the interval, is a random variable It follows a uniform distribution on the interval (0,1). and are the position and velocity of the i-th bat in the t-th iteration, is the position of the i-th bat in the t-1th iteration, is the optimal solution currently searched; The multi-strategy defense in the global exploration phase includes the first defense strategy and the second defense strategy. and The size of determines whether to execute the first defense strategy or the second defense strategy: when When , the first defense strategy is executed, and the mathematical expression of position update is: in, is a random number that follows a normal distribution, is a random number in the interval (0,1), N is the population size, is a random number between [1,N], is an individual randomly selected from the population, is the position generated between the current individual and an individual randomly selected from the population; At this time, the position of the i-th individual at the next moment With the current time position It is related to the distance between the current optimal solution and the predator. When the individual's position at the next moment will be close to the current optimal solution. When the individual's position at the next moment Will be far away from the current optimal solution; when When , the second defense strategy is executed, and the mathematical expression of position update is: in, is a random number in the interval (0,1), p and q are random numbers in the interval [1,N], and are the positions of the i-th bat in the p-th iteration and the q-th iteration, respectively. For multidimensional vector A vector whose values ​​in each dimension are 0 or 1; like If all elements are 0, the second defense strategy is successful and the predator no longer moves toward the prey. If the element in contains 1, the predator moves towards the prey. , predators will approach bat populations if , predators will stay away from bat populations; The multi-strategy defense in the local development stage is: like , , then the bat optimization algorithm performs the local development phase around the optimal solution neighborhood, and the mathematical expression of position update is: in, is a multidimensional random vector, the values ​​of each dimension of the vector belong to the interval -1 to 1, is the average loudness of all individuals at the tth iteration; The local development phase includes multiple strategies including the third defense strategy and the fourth defense strategy, which compares random numbers and The size of the decision to implement the third defense strategy or the fourth defense strategy; when When , the third defense strategy is executed, and the mathematical expression of position update is: in, is a random number between [1,N], For the The bat in the The position in the iteration, is a random number in the interval (0,1). is the parameter vector used to control the search direction, As a defense factor, is the odor diffusion factor, For the The individual in The fitness of the next moment, is a random number in the interval (0,1). is a vector whose element value is 0 or 1. is the maximum number of iterations, exp{} is the base e raised to the power of {}, rand is a random number, the random number is ≤0.5, , random number>0.5, ; when When the element is 0, the predator stops moving. When the element contains 1, the predator keeps its distance from the prey. The larger it is, the more the individual tends to move closer to the optimal solution; when When , the fourth defense strategy is executed, and the mathematical expression of position update is: in, is a random number in the interval (0,1). and are random numbers in the interval (0,1), The inelastic collision force generated when the prey attacks the predator with its body. is a random vector whose element values ​​are in the interval [0,1], for the quality of the predator; The step S5 comprises the following sub-steps: Step S501, all individuals are randomly initialized by the MSDBA algorithm, and each individual is initialized with a corresponding hyperparameter combination of the LSTM network based on the self-attention mechanism, including the number of hidden layers, the number of neurons in the hidden layer, the learning rate, the number of iterations, and the batch size; Step S502, the LSTM network based on the self-attention mechanism constructs a corresponding network structure according to the hyperparameter combination of the individual MSDBA algorithm, and each individual of the MSDBA algorithm corresponds to an LSTM network model based on the self-attention mechanism; Step S503, each LSTM network model based on the self-attention mechanism is trained on the training data set, and after the training is completed, prediction is performed on the test data set, and the mean square error is calculated; Step S504, the MSDBA algorithm uses the mean square error as fitness, calculates the fitness values ​​of all individuals, and updates the current optimal individual; Step S505, the MSDBA algorithm executes the global exploration or local development phase, updates the frequency, speed and position of the individual, and returns to step S503 for a new round of iteration; Step S506, after the MSDBA algorithm reaches the maximum number of iterations, the global optimal solution at this time represents the optimal hyperparameter configuration of the LSTM network based on the self-attention mechanism. The initial monitoring data is predicted and output using the optimal hyperparameter configuration.

2. The MSDBA-LSTM pollutant concentration prediction method based on self-attention as claimed in claim 1, characterized in that: The LSTM model in step S3 includes a forget gate, an input gate, and an output gate: The forget gate is used to determine whether the state information in the memory unit is retained. The state information in the memory is passed to the current memory unit through the forget gate. , the mathematical expression is: in, is the forget gate at the current time t, is the sigmoid activation function, with a value in the (0,1) interval. and are the weight matrix and bias term of the hidden layer of the forget gate, is the hidden unit state at the previous moment, the hidden unit state at the previous moment and current time input information They are passed together as input information to the hidden layer of the forget gate; The input gate is used to update the memory unit state and determine the updated input information at the current moment to be stored in the memory unit. , the mathematical expression is: in, is the input gate at the current time t, is the sigmoid activation function, with a value in the (0,1) interval. and are the weight matrix and bias term of the hidden layer of the input gate respectively; The output gate is used to determine the memory unit at the current moment. Can the state information in be updated to the hidden unit? , which is calculated as follows: in, is the output gate at the current time t, is the sigmoid activation function, with a value in the (0,1) interval. and are the weight matrix and bias term of the output gate hidden layer respectively; Use the tanh function to generate the memory unit at the current moment , the input gate With the current memory unit Multiply and store the last moment memory unit With the forget gate Multiply them to determine what information is updated at the current moment, and the memory unit state is updated as follows: Use the tanh function to store the current moment memory unit Normalization, and output gate Multiply, the output hidden unit state for: 。 3. The MSDBA-LSTM pollutant concentration prediction method based on self-attention as claimed in claim 2 is characterized in that: The step S4 includes the following sub-steps: Step S401, the calculation process of the self-attention mechanism includes: Calculate the similarity between the Query value and the Key value. The mathematical expression is: in, is the feature vector of the input data, , For the origin Query value, For the origin Key value, Calculate similarity for the dot product, for The similarity with other feature vectors, , is the learnable parameter matrix; Calculate the attention weight, the mathematical expression is: in is the normalization function, N is the number of input feature vectors, is the attention weight; Calculate the attention value, the mathematical expression is: in is the learnable parameter matrix, For the The Value corresponding to the input vector; Step S302, map the attention value to a predicted value and output it.

Images