An air quality PM2.5 prediction method based on an ordered neuron long short-term memory network

Abstract

The application discloses an air quality PM2.5 prediction method based on an ordered neuron long short-term memory network. The method uses the continuous time data of the air quality parameters of the air quality detection station and the meteorological parameters which have an influence on PM2.5 to construct a complete air quality data set which can be used for prediction. Then, the proportion of the preprocessed data set is divided to construct a training set and a test set. The ON-LSTM model is parameter optimized by using the training set data, including the number of neurons, the random inactivation rate, the initial learning rate, the iteration number, the batch size and the like. Then, the parameter optimized ON-LSTM model which can be used for predicting the PM2.5 concentration is constructed, the hierarchical parameters of the ON-LSTM are determined according to the model training performance, and the hierarchical updating capability is realized. The optimization algorithm is determined, and the activation function is selected. Finally, the concentration prediction value of PM2.5 is obtained based on the trained model, the model precision is improved, and the generalization capability of the model under the premise that the pollutants and the meteorological parameters have a potential influence on PM2.5 is enhanced.

Description

Technical Field

[0001] This invention relates to the field of time-series prediction of PM2.5 air quality, and particularly to a prediction technique for PM2.5 concentration data exhibiting nonlinearity, uncertainty, and long-term dependence. The invention is a specific application of a method based on the efficient extraction of neural hierarchical structures from long short-term memory networks in the field of PM2.5 air quality prediction. This invention improves the accuracy of air quality prediction while preserving the temporal correlations hidden in the time series data. Background Technology

[0002] Fine particulate matter (PM2.5) refers to particulate matter in ambient air with an aerodynamic equivalent diameter of 2.5 micrometers or less. It is an important component of air pollution and one of the main bottlenecks in improving air quality.

[0003] Currently, PM2.5 prediction models mainly include numerical models and statistical models. Numerical models, based on a thorough understanding of atmospheric physical and chemical processes, utilize meteorological principles and mathematical methods to simulate air quality on a large scale, both horizontally and vertically, reproducing the processes of pollutant emissions (anthropogenic and natural sources), reactions (gas, liquid, and solid-phase chemical reactions), diffusion (horizontal and vertical diffusion), transport (horizontal advection and vertical convection), and removal mechanisms (dry and wet deposition). However, numerical models have stringent requirements for meteorological and pollutant data, and require users to be familiar with the model architecture, program code, and basic atmospheric physical and chemical principles. Therefore, their application in PM2.5 prediction is limited. Statistical models, on the other hand, use mathematical statistics methods to analyze historical observation data of air quality and meteorological parameters, establish statistical relationships between air pollutants and influencing factors, obtain fitted equations or statistical models, and then extrapolate to obtain future air quality forecasts. Currently, the mainstream statistical models include regression equation methods and neural network methods. The distribution of PM2.5 concentration is influenced by multiple factors and exhibits strong nonlinear characteristics. Neural networks, as a nonlinear science, are suitable for studying objects with multifactorial, uncertain, random, and nonlinear characteristics. However, current methods mainly utilize latent temporal patterns from different perspectives, but neglect the temporal dependence of the prediction task on the target and the unpredicted sequence, resulting in less than ideal model prediction results. Summary of the Invention

[0004] To address the inaccuracy of PM2.5 predictions due to the nonlinearity and time dependence of PM2.5 data, this invention provides an ordered neuron long short-term memory network (ON-LSTM) for PM2.5 air quality prediction. By using ON-LSTM to rank neurons and represent the information hierarchy by their position, this further resolves the long-term dependency issue in PM2.5 air quality prediction. Compared to other neural network models, our method is unique in that it uses a hierarchical embedding structure to achieve long-term retention of important historical information, improving the dynamic consistency between the predicted sequence and the original true sequence.

[0005] The present invention adopts the following technical solution and implementation steps:

[0006] S1: Obtain continuous time data of air quality parameters and meteorological parameters affecting PM2.5 from multiple air quality monitoring stations in a certain region, and construct the original air quality dataset.

[0007] S2: Preprocess the constructed initial air quality dataset, correct outliers, fill data gaps using interpolation, and construct a complete air quality dataset that can be used for prediction.

[0008] S3: Based on the preprocessed dataset, divide it proportionally to construct training and test sets. Use the training set data to fine-tune the parameters of the ON-LSTM model, including the number of neurons, random inactivation rate, initial learning rate, number of iterations, and batch size.

[0009] S4: Construct a parameter-tuned ON-LSTM model that can be used to predict PM2.5 concentration. Determine the chunk size of the ON-LSTM model based on its training performance, determine the optimization algorithm, select the activation function, and ensure the dynamic consistency between the training sequence and the original real sequence.

[0010] S5: Based on the trained ON-LSTM prediction model, predict the PM2.5 concentration on the test set. Then, evaluate the model's performance based on the predicted values ​​and the actual values.

[0011] Specifically, influenced by seasonal pollution source emissions, precipitation, wind speed, and atmospheric diffusion conditions, PM2.5 concentrations tend to be higher in autumn and winter. Furthermore, the additional emissions during the winter heating season in North China exacerbate PM2.5 pollution, making PM2.5 concentration prediction for autumn and winter necessary for environmental assessment. Therefore, this experiment selected air pollutant concentration datasets from November to March to predict PM2.5 concentrations at multiple stations in the selected region. The dataset includes the target variable, hourly PM2.5 concentration data, as well as other relevant variables such as PM10, PM2.5, SO2, NO2, CO, and meteorological parameters including dew point, temperature, pressure, wind direction, wind speed (m / s), snowfall, and precipitation.

[0012] Furthermore, outliers and missing values ​​are first examined, then outliers are deleted, and the missing values ​​are filled in the dataset using median interpolation.

[0013] Preferably, 80% of the dataset is divided into the training set and the remaining 20% ​​into the test set. The number of neurons is 64, the random inactivation rate is 0.1, the learning rate is 0.001, the step size is 200, the number of iterations is 200, and the batch size is 128.

[0014] Furthermore, the model comprises a basic LSTM framework, a novel ON-LSTM framework with a new cell state update strategy, an encoding / decoding input / output strategy, and fully connected layers. The gate structure within each cell unit consists of three parts: a forget gate, an input gate, and an output gate. These gates are responsible for updating the cell state and hidden layer state. The model employs a novel cell state update strategy, dividing the hierarchy into historical and current levels. Based on the different sizes of these levels, historical and current information are preserved, removed, and fused. The disordered neurons in common neural networks are sorted to fully utilize the ordered information of neurons; this information hierarchy forms an ordered hierarchical neuron structure. The encoding / decoding input / output structure enables input and output of arbitrary step sizes, improving the model's flexibility and practical applicability. The fully connected layers are responsible for the final output of the prediction model of this invention.

[0015] Preferably, the loss function in the prediction model is the standard normalized MSE, the neuron parameter chunk_size is selected as 4, the activation function is the RelU function, and the Adam function is used for learning during the training process of S4.

[0016] Compared with the prior art, the beneficial effects of the present invention are:

[0017] (1) A novel cell state update strategy was integrated with the gate structure of a long short-term memory network to determine the criteria for selecting historical and current information during model training.

[0018] (2) The ingenious and unique ON-LSTM structure design can extract the hierarchical structure of the input sequence, solving the problem that traditional LSTM models have difficulty learning the correlation between samples that are far apart in the sequence. This improves the accuracy of the model when predicting data with long-term dependency characteristics. Attached Figure Description

[0019] Figure 1 This is a diagram showing the overall structure of the prediction model proposed by the method of this invention.

[0020] Figure 2 This is a diagram of the ON-LSTM network structure proposed in the method of this invention;

[0021] Figure 3 A flowchart illustrating the hierarchical cell state update process proposed in this invention.

[0022] Figure 4 This is a schematic diagram comparing the prediction results of the method of the present invention. Detailed Implementation

[0023] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] This invention proposes a novel prediction framework, ON-LSTM, for multi-step prediction of multivariate time series data, addressing the aforementioned problems. The core idea of ​​ON-LSTM is to rank neurons, using their position to represent the information hierarchy. This hierarchical structure, embedded through information grading, enables long-term retention of important historical information and further solves the problem of long-term dependencies. It focuses on the stochastic dependencies hidden in time series, improving the dynamic consistency between the predicted sequence and the original true sequence. It particularly emphasizes the temporal dependence of the prediction task on the target and non-predicted sequences. Finally, by constructing a fully connected layer network, it outputs future PM2.5 concentration values, thereby achieving accurate prediction of PM2.5 concentration.

[0025] As in this invention Figure 1The overall flowchart for predicting PM2.5 concentration is shown, mainly consisting of the following steps: S1: Obtain continuous time data of air quality parameters and meteorological parameters affecting PM2.5 from multiple air quality monitoring stations in a certain area to construct an initial air quality dataset. S2: Preprocess the constructed initial air quality dataset, correct outliers, and fill data gaps using interpolation to construct a complete air quality dataset suitable for prediction. S3: Divide the preprocessed dataset proportionally to construct training and test sets. Optimize the parameters of the ON-LSTM model using the training set data, including the number of neurons, random inactivation rate, initial learning rate, number of iterations, and batch size. S4: Construct a parameter-optimized ON-LSTM model suitable for predicting PM2.5 concentration. Determine the ON-LSTM chunk size based on model training performance, determine the optimization algorithm, select the activation function, and ensure dynamic consistency between the training sequence and the original real sequence. S5: Predict PM2.5 concentration values ​​on the test set based on the trained ON-LSTM prediction model. Evaluate the model performance based on the predicted values ​​and the actual values.

[0026] S1: Affected by seasonal pollution source emissions, precipitation, wind speed, and atmospheric diffusion conditions, PM2.5 concentrations in Beijing tend to be higher in autumn and winter. From November to March, PM2.5 pollution worsens due to additional emissions during the winter heating season in North China. This case study selected Beijing for PM2.5 concentration prediction. Hourly sampling data of atmospheric pollutants PM10, PM2.5, SO2, NO2, CO, and meteorological parameters dew point, temperature, pressure, wind direction, wind speed (m / s), snowfall, and rainfall were collected from ten environmental monitoring stations in Beijing: Dongsi in Dongcheng District, Wanshouxigong in Xicheng District, Olympic Sports Center in Chaoyang District, Wanliu in Haidian District, Gucheng in Shijingshan District, and Changping District.

[0027] Table 1. Variables used in PM2.5 concentration prediction in this invention

[0028]

[0029] S2: Due to uncertainties in data acquisition equipment, the collected data will always contain some missing and outlier values, which will affect the accuracy of the model. Our invention proposes a bit-level interpolation method for data preprocessing. First, outliers are removed, and then all existing missing values ​​are identified. These values ​​need to be processed. Our method uses the sliding window principle combined with contextual data at the missing points to fill in the median of the missing data. This process continues from the first missing data sampling point to the last. Then, we construct an M*N (M>0, N>0) dimensional initial dataset based on the data collected in S1, where M is the sum of the predictor variable and the target variable (in this example, M is 12), and N is the total number of data samples acquired. The input data can be represented by the following formula:

[0030]

[0031] The input at time t is:

[0032] S3: Divide the dataset established in S2 into a training set and a test set, with the training set comprising 80% and the remainder as the test set. The former is used to build the model, while the latter is used to evaluate the model's generalization ability when predicting unknown samples. When the model's generalization ability does not meet the requirements (this experiment uses R...), 2 If the accuracy is below 80%, we need to adjust the network's hyperparameters. The specific steps are as follows:

[0033] STEP 1: Denote the neuron input as Z, with each dimension being Z0. i The corresponding weights v can enhance or weaken the input signal. i This achieves a nonlinear mapping to [0,1] or [−1,1]. The specific number of neurons will be adjusted based on the actual feedback from the model. To avoid making the overall model too complex, the number of neurons should be less than 100, using 2n as an example. n The model exhibits a consistent variation in number of neurons, and the number of neurons is selected based on the highest model accuracy. In this example, the number of neurons is determined to be 64. Increasing the number of neurons generally improves the model's accuracy, but if the number is increased too much, the model may overfit.

[0034] STEP 2: Dropout is a regularization method for neural network models. During training, it enhances the model's generalization ability by randomly ignoring some neurons. In deep network training, a portion of neurons are randomly "temporarily dropped" with a certain probability. For example, if dropout is 0.1, then 10% of the nodes are dropped. Specifically, dropout is applied to each mini-batch of training data. Due to its mechanism of randomly dropping some neurons, it's equivalent to training a neural network with a different structure in each iteration. In this example, to prevent overfitting but also to avoid dropping too many neurons, dropout is set to 0.1.

[0035] STEP 3: Select a learning rate scheme. As an important hyperparameter in supervised learning and deep learning, the learning rate determines whether and when the objective function converges to a local minimum. A suitable learning rate enables the objective function to converge to a local minimum within a reasonable time. To avoid the model's loss function changing too slowly and getting stuck in a local minimum, or the loss function failing to converge, we have developed a fractional incremental learning rate mitigation scheme, which adjusts the initial learning rate... Set to 0.001, the learning rate decreases according to the following formula:

[0036]

[0037] Where k is a hyperparameter used to control the rate of decrease in the learning rate. Each transformation is above 90% of the previous one, therefore k is set to 0.1 in this example. t is the number of training epochs, set to 200 to match the epoch count.

[0038] STEP 4: Determine an appropriate number of epochs. One epoch is defined as when a complete dataset passes through the neural network once and returns correctly. The main purpose of setting epochs is to divide the entire training process into several segments, allowing for better observation and adjustment of the model's training. By iterating and comparing network accuracy by increasing the epoch by 10 each time, the improvement in network accuracy becomes less significant after reaching 200 epochs. To avoid excessively long network training times, 200 epochs were set, which also ensures the model's prediction accuracy.

[0039] STEP 5: Determine an appropriate batch size. Before using batch size, this means that during network training, all data is fed into the network at once, and then their gradients are calculated and backpropagated. Set the batch size to 2. m(m>0), increment the batch size until the model accuracy no longer changes significantly. To ensure model accuracy while improving training speed, the batch size is set to 128.

[0040] S4: The main neural network of the prediction model constructed in this invention is ON-LSTM, such as... Figure 2 The text describes a Long Short-Term Memory (LSTM) network, a type of time-recurrent network commonly used in deep learning for time series prediction. The core concept of LSTM is the cell state. Through training, the gate structure controls which information to retain and which to forget. This selected information is added to the cell state through computation. As the cell state is cyclically passed between time steps, information from earlier time steps can also be used in later time steps, thus achieving long-term memory. The corresponding cell update process formula is as follows:

[0041]

[0042] in For the Gate of Oblivion It is the sigmoid activation function. for The weight matrix, The input at time t, for Previous moment The weight, For the memory at time t-1, for The bias term.

[0043]

[0044] in For input gate, It is the sigmoid activation function. for The weight matrix, The input at time t, for Previous moment The weight, for The bias term.

[0045]

[0046] in For output gate, for The weight matrix, for Previous moment The weight, for The bias term.

[0047]

[0048] It is the hyperbolic tangent function. In cellular state, This is a matrix element mapping multiplication operation.

[0049]

[0050] This represents the current cell state. for The weight matrix, for Previous moment The weight, for The bias term.

[0051]

[0052]

[0053] It is an accumulation function. This is a newly defined forget gate, where softmax is a normalized exponential function. for The weight matrix, for Previous moment The weight, for The bias term.

[0054]

[0055] For the new input gate, for The weight matrix, for Previous moment The weight, for The bias term.

[0056]

[0057] for Calculated value.

[0058]

[0059] This is the updated cell state (memory unit), and also the input for subsequent cell units. The gate structure inside the cell unit is responsible for updating the cell state and the hidden layer state. The gate structure consists of three parts: the forget gate... Input gate and output gate The cell state at the previous moment. The information is forgotten through the forget gate and added through the input gate. The output gate is responsible for controlling the output of the hidden layer at the current moment. ON-LSTM has a completely new... The update strategy uses a and A fully connected layer is used to predict two vectors and perform a softmax operation to obtain... and Approximation, using Represents the lowest level to which historical information reaches. The highest level of information currently being accessed. Figure 3 A partitioned diagram describing the ON-LSTM cell state update process is presented. Based on... and The hierarchical relationship determines which neurons are retained, removed, or eliminated. After building the core ON-LSTM network, we need some optimization methods to ensure dynamic consistency between the training sequence and the original real sequence. The specific steps are as follows:

[0060] STEP 1: Determining the chunk_size parameter related to ordered neurons. In this example, the ON-LSTM has 64 neurons, which means the described layer is very large. However, in reality, when a sequence has layers, the number of layers is not large. To resolve this contradiction, we use chunk_size to represent the layer and divide the layer into 4. In this example, we choose chunk_size as 4. This avoids the complexity caused by dividing the layer, reduces the computational load of the model, and improves the efficiency of the model when used for PM2.5 prediction.

[0061] STEP 2: Select an optimization algorithm. Currently, mainstream optimization algorithms include gradient descent and Nesterov gradient acceleration. This invention uses the Adam algorithm to address the large-scale and non-stationary characteristics of PM2.5 data. The specific steps are as follows:

[0062] (1) Initialize and determine the parameters (step size), (the exponential decay rate is estimated by moment, between [0, 1)) and the stochastic objective function.

[0063] (2) Initialize the parameter vector, first-order moment vector, second-order moment vector, and time step;

[0064] (3) Loop: When the parameters do not converge, iteratively update each part. That is, increment the time step by 1, update the gradient of the objective function with respect to the parameters at that time step, update the first moment estimate and the second moment estimate of the bias, then calculate the first moment estimate and the second moment estimate of the bias correction, and then use the values ​​calculated above to update the initial parameters of the model;

[0065] (4) When the parameters converge, return the model parameters.

[0066] STEP 3: Determine the activation function. The neurons in the hidden and output layers convert the weighted sum of their inputs into the neuron's output value using the activation function. To accommodate the nonlinearity of PM2.5 data and improve the network's robustness, this invention employs the ReLU activation function, the formula of which is as follows:

[0067]

[0068] S5: Based on the trained ON-LSTM prediction model, the concentration prediction value of PM2.5 is obtained by predicting the test set. Figure 4 A comparison chart of the generated predicted values ​​and the actual values ​​is provided. Then, the model's performance is evaluated based on the predicted and actual values. The evaluation formula used in this invention is as follows:

[0069]

[0070]

[0071]

[0072] Where MAE is the mean absolute error and RMSE is the root mean square error. The coefficient of determination. and These represent the predicted value and the actual value, respectively. Lower MAE and RMSE values ​​indicate better model performance. The larger the value, the better. Table 2 shows the evaluation examples of PM2.5 air quality prediction using ten stations selected in this invention. The ordered neuronal long short-term memory network (ON-LSTM) adopted in this invention has shown very good accuracy for stations in different regions. The hierarchical update mechanism of the model ensures its generalization ability.

[0073] Table 2: ON-LSTM Prediction Performance Evaluation for Ten Sites

[0074]

[0075] The above-disclosed embodiments are merely a few specific examples of the present invention. However, the embodiments of the present invention are not limited thereto. Any modifications, substitutions, and improvements made within the spirit and principles of the above embodiments should be included within the protection scope of the technical solution.

Claims

1. A method for predicting PM2.5 air quality based on ordered neural long short-term memory networks, characterized in that, Includes the following steps: S1: Obtain continuous time data of air quality parameters and meteorological parameters affecting PM2.5 from multiple air quality monitoring stations in a certain area, and construct the original air quality dataset; S2: Preprocess the constructed initial air quality dataset, correct outliers, fill data gaps using interpolation, and construct a complete air quality dataset Z that can be used for prediction; where M is the sum of the predicted variable and the target variable, and N is the total number of data samples obtained. ； S3: Based on the preprocessed dataset, divide it proportionally to construct training and test sets; use the training set data to fine-tune the parameters of the ON-LSTM model, including the number of neurons, random inactivation rate, and initial learning rate. Number of iterations, batch size; Where k is a hyperparameter used to control the rate of decrease in the learning rate. Each transformation is above 90% of the previous one, k is set to 0.1; t is the number of training epochs, set to 200 to match the epoch count. S4: Construct a parameter-tuned ON-LSTM model that can be used to predict PM2.5 concentration. Determine the chunk size of the ON-LSTM model based on its training performance, identify the optimization algorithm, and select the activation function. This ensures dynamic consistency between the training sequence and the original real sequence; the novel cell state update strategy for PM2.5 prediction is formulated as follows: ； ； ； ； ； It is an accumulation function. This is a newly defined forget gate, where softmax is a normalized exponential function. for The weight matrix, for Previous moment The weight, for The bias term; For the new input gate, for The weight matrix, for Previous moment The weight, for The bias term; for Operational value; It is the updated cell state, and also the input for subsequent cell units; S5: Based on the trained ON-LSTM prediction model, predict the concentration of PM2.5 on the test set; and evaluate the model performance based on the predicted values ​​and the actual values.

2. The PM2.5 air quality prediction method based on ordered neural long short-term memory networks as described in claim 1, characterized in that, A multidimensional initial dataset combining time and space was constructed, with the time dimension set to 8670 and the spatial dimension set to 12, which is the sum of PM2.5 and other pollutants and meteorological parameters that affect it.

3. The PM2.5 air quality prediction method based on ordered neural long short-term memory networks as described in claim 1, characterized in that, Outliers are assigned null values, the number of missing values ​​is detected, and then all missing values ​​are filled using median interpolation.

4. The PM2.5 air quality prediction method based on ordered neural long short-term memory networks as described in claim 1, characterized in that, The initial learning rate was set to 0.001, with a reduction of 0.

95.

5. The PM2.5 air quality prediction method based on ordered neural long short-term memory networks as described in claim 1, characterized in that, The model uses the number of neurons as a factor of chunk_size to divide the number of layers, reducing the computational cost; the optimization algorithm chosen is Adam, and the activation function is selected... .

6. The PM2.5 air quality prediction method based on ordered neural long short-term memory networks as described in claim 1, characterized in that, By using a recursive prediction method to treat the predicted values ​​as feature values, a fully connected layer that works in tandem with the output steps is constructed to enable the model to predict PM2.5 at any step.

Images