Air quality prediction method and system based on improved Transform

Abstract

The invention discloses an improved Transform-based air quality prediction method and system, and relates to the technical field of artificial intelligence and environmental science, and the method comprises the steps: collecting air pollution data; preprocessing the air pollution data to obtain a multivariable time sequence; feature extraction is carried out on the preprocessed data, and initialization processing is carried out; an air quality prediction model based on the improved Transform network is constructed; and inputting the initialized data into the air quality prediction model to obtain an air quality prediction result. According to the method, on the basis of keeping the original parallel computing and multi-modal adaptation advantages of Transform, the time complexity is effectively reduced, the multi-modal data utilization is optimized, the global representation learning ability and generalization performance of the model are remarkably improved, and the air quality is predicted more accurately.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and environmental science technology, and more specifically to an air quality prediction method and system based on an improved Transformer. Background Technology

[0002] Time series analysis, as an important interdisciplinary research direction in environmental science and artificial intelligence, has demonstrated increasingly prominent value in the field of air quality prediction. In recent years, with the rapid evolution of deep learning technology and continuous innovation in spatiotemporal modeling methods, this field has achieved several key breakthroughs in models and data. These technological advancements have significantly improved prediction accuracy and timeliness, continuously expanding application scenarios: supporting precise pollution source analysis and early warning decisions in urban environmental governance; providing dynamic assessment and optimization schemes for industrial emission monitoring; and enabling intelligent route planning based on real-time prediction in traffic control. Simultaneously, with the development of edge computing and IoT technologies, time series analysis methods are gradually evolving towards distributed and real-time approaches, providing core technological support for building an integrated "prediction-early warning-control" environmental governance system.

[0003] In the context of the era of intelligentization and informatization, the development of environmental management has placed higher demands on air quality forecasting, especially in terms of forecast accuracy and multimodal data integration. Current time series forecasting methods are mainly divided into two categories: recurrent neural network (RNN) methods and spatiotemporal modeling methods, each with different characteristics and applicable scenarios. Firstly, RNN methods, by memorizing and updating time step information, can effectively capture the long-term and short-term dependencies in air quality data. These methods are particularly suitable for modeling the dynamic changes of pollutants over time. However, RNNs suffer from low computational efficiency and gradient vanishing issues in practical applications, especially when processing long-term series. In contrast, spatiotemporal modeling methods (represented by Transformers) exhibit stronger advantages. These methods, by introducing spatial relationship modeling, achieve joint capture of temporal and spatial features. Their outstanding parallel computing capabilities enable them to better adapt to multimodal data and efficiently capture spatial dependencies, making them more suitable for complex spatiotemporal interaction scenarios. However, it is worth noting that existing spatiotemporal modeling methods still face several challenges. Taking the classic Transformer as an example, its computational complexity is high and its data requirements are large. When processing variable-length sequences, it is necessary to pad or truncate the sequence to a fixed length, which may introduce noise or lead to information loss. In addition, the Transformer's method of embedding all multivariate features of each timestamp into a single token may lose the correlation between variables, thereby affecting the accuracy of prediction.

[0004] Therefore, how to provide an air quality prediction method and system based on an improved Transformer, while maintaining the advantages of the original Transformer in parallel computing and multimodal adaptation, effectively reducing time complexity, optimizing the utilization of multimodal data, and significantly improving the model's global representation learning ability and generalization performance, so as to more accurately predict air quality, is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] In view of this, the present invention provides an air quality prediction method and system based on an improved Transformer. To reduce time complexity, better utilize the correlation of multimodal data, and enhance the global modeling capability of the model, the present invention proposes an improved Transformer architecture. By embedding each independent time series as a variable token, the attention mechanism can more accurately capture the correlation between multiple variables. At the same time, the information coverage of the model is broadened through sequence-level tokenization design to enhance the global modeling capability. While maintaining the original advantages of Transformer in parallel computing and multimodal adaptation, the present invention effectively reduces time complexity, optimizes the utilization of multimodal data, and significantly improves the model's global representation learning ability and generalization performance, making it more suitable for the requirements of time series prediction tasks and more accurately predicting air quality.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: an air quality prediction method based on an improved Transformer, comprising:

[0007] S100, collecting air pollution data;

[0008] S200. Preprocess the air pollution data to obtain a multivariate time series;

[0009] S300. Extract features from the preprocessed data and perform initialization processing;

[0010] S400. Construct an air quality prediction model based on an improved Transformer network; input the initialized data into the air quality prediction model to obtain the air quality prediction result;

[0011] The air quality prediction model embeds each variable as an independent token; it captures the dependencies between variables through a multi-head attention mechanism and generates prediction results by combining residual connections and normalization operations.

[0012] Preferably, the air pollution data includes: time, relative altitude, longitude, latitude, temperature, humidity, air pressure, SO2, CO, NO2, O3+NO2, PM2.5. 1.0 PM2.5 PM 10 .

[0013] Preferably, the air pollution data is preprocessed to obtain a multivariate time series, including:

[0014] The air pollution data was cleaned to obtain a multivariate time series. The formula is as follows:

[0015] X :,T =clean(x) :,T );

[0016] Where T represents the time step, N represents the number of variables, and clean() represents the cleaning operation. It represents the set of real numbers.

[0017] Preferably, feature extraction and initialization processing are performed on the preprocessed data, including:

[0018] The preprocessed data is mapped to a high-dimensional feature space to obtain spatial features;

[0019] The spatial features are initialized linearly along the time dimension.

[0020] Preferably, the air quality prediction model includes an embedding layer, a multi-head attention layer, a first normalization layer, a feedforward network layer, a second normalization layer, and a prediction layer connected in sequence.

[0021] Preferably, in step S401, the initialized data is mapped to obtain the Q, K, V vectors required for the multi-head attention layer, and the attention score matrix is ​​calculated.

[0022] S402. Based on the V vector and the attention score matrix, a weighted sum is performed to obtain the context feature representation, which is used to capture the complex relationships between data.

[0023] S403. The first normalization layer is used to normalize the data at each time step in the context feature representation; the processed data is then input into the feedforward network layer to mine the complex features of the time series and obtain the corresponding complex representation.

[0024] S404. The data processed by the first normalization layer and the complex representation output by the feedforward network layer are combined using residual connections, and then the data features are optimized by the second normalization layer.

[0025] S405, based on S300 and S401-S404, are stacked to form a deep network, resulting in the final air quality prediction model.

[0026] Preferably, an air quality prediction system based on an improved Transformer includes: a data acquisition module for collecting air pollution data;

[0027] The data preprocessing module allows users to preprocess the air pollution data to obtain a multivariate time series.

[0028] The feature extraction and initialization module is used to extract features from the preprocessed data and perform initialization processing.

[0029] An air quality prediction module is used to construct an air quality prediction model based on an improved Transformer network; the initialized data is input into the air quality prediction model to obtain the air quality prediction results.

[0030] As can be seen from the above technical solution, compared with the prior art, this invention discloses an air quality prediction method and system based on an improved Transformer, including: collecting air pollution data; preprocessing the air pollution data to obtain a multivariate time series; extracting features from the preprocessed data and performing initialization processing; constructing an air quality prediction model based on an improved Transformer network; and inputting the initialized data into the air quality prediction model to obtain the air quality prediction result. Compared with the prior art, this invention has the following beneficial effects:

[0031] (1) This invention proposes to embed each independent time series (variable) as a variable token, and use an attention mechanism to capture the correlation between multiple variables (e.g., how temperature affects humidity), which can aggregate these features and reduce the loss of correlation between variables.

[0032] (2) This invention uses only encoders for modeling, focuses more on variable-specific learning, can process diverse time series data more efficiently, and can better generalize unseen data and more accurately predict air quality by utilizing global representation and multivariate correlation. Attached Figure Description

[0033] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0034] Figure 1 This is a flowchart of a time-series heterogeneous mode air quality prediction method based on Transformer.

[0035] Figure 2 The network structure diagram of the improved Transformer air quality prediction model provided as an example of the present invention.

[0036] Figure 3 This is a schematic diagram showing the comparison results of the root mean square error between the present invention and existing methods. Detailed Implementation

[0037] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0038] This invention discloses an air quality prediction method based on an improved Transformer, such as... Figure 1 As shown, the collected data (meteorological data and pollution data) are used directly, then network features are extracted, and finally, a network model is used for prediction. Specifically, this includes:

[0039] S100, collecting air pollution data;

[0040] Specifically, in this embodiment of the invention, air pollution data is collected using an unmanned data collection device: collection time, relative altitude, longitude, latitude, temperature (°C), humidity (%), air pressure (hPa), and SO2 (μg / m³). 3 CO (mg / m³) 3 NO2 (μg / m 3 O3+NO2(μg / m 3 PM 1.0 (μg / m 3 PM 2.5 (μg / m 3 PM 10 (μg / m 3 ).

[0041] Table 1 shows the sample data collected in the embodiments of the present invention, which consists of 10 data entries.

[0042] Table 1 Sample Data

[0043]

[0044]

[0045] S200. Preprocess the air pollution data to obtain a multivariate time series, specifically including:

[0046] The air pollution data was cleaned to obtain a multivariate time series. As shown in formula (1):

[0047] X :,T =clean(x) :,T (1)

[0048] Where T represents the time step, N represents the number of variables, and clean() represents the cleaning operation. It represents the set of real numbers.

[0049] S300. Perform feature extraction and initialization on the preprocessed data, specifically including:

[0050] The preprocessed data is mapped to a high-dimensional feature space to obtain spatial features;

[0051] The spatial features are initialized linearly along the time dimension.

[0052] In one specific embodiment of the present invention, the original time series X is... :,T By mapping to a high-dimensional feature space using formula (2), spatial features are obtained.

[0053]

[0054] in, This represents the representation at time step t in layer 0, where Embedding() is the feature encoder; it converts spatial features... Obtained by linear initialization along the time dimension

[0055]

[0056] in, Let μ represent the representation of the t-th time step after linear initialization at level 0, where μ and σ represent the mean and standard deviation calculated along the time dimension, respectively.

[0057] S400. Construct an air quality prediction model based on an improved Transformer network; input the initialized data into the air quality prediction model to obtain the air quality prediction result;

[0058] The air quality prediction model inputs each variable (i.e., each variable in Table 1 above) into the embedding layer according to the variable type, embedding them as independent tokens; it captures the dependencies between variables through a multi-head attention mechanism, and generates prediction results by combining residual connections and normalization operations.

[0059] Specifically, such as Figure 2 As shown, the air quality prediction model includes an embedding layer, a multi-head attention layer, a first normalization layer, a feedforward network layer, a second normalization layer, and a prediction layer connected in sequence.

[0060] Specifically, S401, the initialized data is mapped to obtain the Q, K, V vectors required for the multi-head attention layer, and the attention score matrix is ​​calculated;

[0061] Specifically, will The Q, K, V vectors required for the attention mechanism are obtained by mapping using formula (4).

[0062]

[0063] Where Q represents the query vector, K represents the key vector, V represents the value vector, and W represents the value vector. Q W k W V These are learnable linear transformation parameters;

[0064] Input the obtained Q and K into formula (5) to obtain the attention score matrix.

[0065]

[0066] Where d is the embedding dimension, softmax() is the normalization operation, and K is the embedding dimension. T This represents the transpose operation of K.

[0067] S402. Based on the V vector and the attention score matrix, a weighted sum is performed to obtain the context feature representation, which is used to capture the complex relationships between data.

[0068] Specifically, the V and A obtained from S401 are weighted and summed using formula (6) to obtain the context feature representation Z. t+1 ,

[0069] Z t+1 =AV (6)

[0070] S403. The first normalization layer is used to normalize the data at each time step in the context feature representation; the processed data is then input into the feedforward network layer to mine the complex features of the time series and obtain the corresponding complex representation.

[0071] Specifically, the context feature is represented as Z. t+1 The data at each time step is normalized using formula (7) to obtain the results.

[0072]

[0073] Mean() represents taking the average vector, and Var() represents taking the variance;

[0074] Normalized data By applying a feedforward neural network using formula (8), a complex representation Z describing the time series is obtained. t+2 ,

[0075]

[0076] Where W1, b1, W2, b2 are the parameters of the feedforward network, Dropout() is used for regularization, and ReLU() is the activation function.

[0077] S404. The data processed by the first normalization layer and the complex representation output by the feedforward network layer are combined using residual connections, and then the data features are optimized by the second normalization layer.

[0078] Specifically, the S403 obtained and Z t+2 Z is obtained by using residual connections and performing hierarchical normalization. t+3 :

[0079]

[0080] LayerNorm() is the normalization function.

[0081] S405, based on S300 and S401-S404, are stacked to form a deep network, resulting in the final air quality prediction model.

[0082] Specifically, the structures of S300 and S401-S404 are stacked L times according to formula (10) to form a deep network, thus forming the final representation Z;

[0083] Z = TransformerBlocks L (X :,T (10)

[0084] Among them, TransformerBlocks L () Stack L operations;

[0085] The final representation Z is mapped to the prediction target space using formula (11) to obtain the prediction output result.

[0086] Projection() represents the prediction operation.

[0087] This invention first acquires multimodal data (such as meteorological information and pollutant concentrations) using drones and air sampling equipment. Then, the time series data is mapped to a high-dimensional feature space. An improved Transformer model is used to embed each variable as an independent token. Finally, a multi-head attention mechanism is employed to capture the dependencies between variables, and residual connections and normalization operations are combined to improve model stability. The final prediction result is generated through a projection layer. This embodiment features lower computational complexity and stronger global modeling capabilities, making it suitable for environmental monitoring and smart city construction, providing an effective solution for high-precision air quality prediction.

[0088] like Figure 3 The figure shows a comparison of the root mean square error (RMSE) between the embodiments of the present invention and existing methods. It illustrates the comparison of the RMSE loss between the embodiments of the present invention and the Autoformer and Informer methods. Compared to the latter two, the method of the present invention can effectively mine deeper representational features in time series data, exhibiting lower RMSE loss in prediction and demonstrating significant performance advantages.

[0089] In one specific embodiment of the present invention, an air quality prediction system based on an improved Transformer includes: a data acquisition module for acquiring air pollution data;

[0090] The data preprocessing module allows users to preprocess the air pollution data to obtain a multivariate time series.

[0091] The feature extraction and initialization module is used to extract features from the preprocessed data and perform initialization processing.

[0092] An air quality prediction module is used to construct an air quality prediction model based on an improved Transformer network; the initialized data is input into the air quality prediction model to obtain the air quality prediction results.

[0093] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0094] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An air quality prediction method based on an improved Transformer, characterized in that, include: S100, collecting air pollution data; S200. Preprocess the air pollution data to obtain a multivariate time series; S300. Extract features from the preprocessed data and perform initialization processing; S400. Construct an air quality prediction model based on an improved Transformer network; input the initialized data into the air quality prediction model to obtain the air quality prediction result; The air quality prediction model embeds each variable as an independent token; it captures the dependencies between variables through a multi-head attention mechanism and generates prediction results by combining residual connections and normalization operations.

2. The air quality prediction method based on an improved Transformer according to claim 1, characterized in that, The air pollution data includes: time, relative altitude, longitude, latitude, temperature, humidity, air pressure, SO2, CO, NO2, O3+NO2, PM2.

5. 1.0 PM 2.5 PM 10 .

3. The air quality prediction method based on an improved Transformer according to claim 1, characterized in that, The air pollution data is preprocessed to obtain a multivariate time series, including: The air pollution data was cleaned to obtain a multivariate time series. The formula is as follows: X :,T =clean(x :,T ); Where T represents the time step, N represents the number of variables, and clean() represents the cleaning operation. It represents the set of real numbers.

4. The air quality prediction method based on an improved Transformer according to claim 1, characterized in that, Feature extraction and initialization processing are performed on the preprocessed data, including: The preprocessed data is mapped to a high-dimensional feature space to obtain spatial features; The spatial features are initialized linearly along the time dimension.

5. The air quality prediction method based on an improved Transformer according to claim 1, characterized in that, The air quality prediction model comprises an embedding layer, a multi-head attention layer, a first normalization layer, a feedforward network layer, a second normalization layer, and a prediction layer connected in sequence.

6. The air quality prediction method based on an improved Transformer according to claim 5, characterized in that, S401. Map the initialized data to obtain the Q, K, V vectors required for the multi-head attention layer, and calculate the attention score matrix. S402. Based on the V vector and the attention score matrix, a weighted sum is performed to obtain the context feature representation, which is used to capture the complex relationships between data. S403. The first normalization layer is used to normalize the data at each time step in the context feature representation; the processed data is then input into the feedforward network layer to mine the complex features of the time series and obtain the corresponding complex representation. S404. The data processed by the first normalization layer and the complex representation output by the feedforward network layer are combined using residual connections, and then the data features are optimized by the second normalization layer. S405, based on S300 and S401-S404, are stacked to form a deep network, resulting in the final air quality prediction model.

7. An air quality prediction system based on an improved Transformer, employing the air quality prediction method based on an improved Transformer as described in any one of claims 1-6, characterized in that, include: The data acquisition module is used to collect air pollution data; The data preprocessing module allows users to preprocess the air pollution data to obtain a multivariate time series. The feature extraction and initialization module is used to extract features from the preprocessed data and perform initialization processing. An air quality prediction module is used to construct an air quality prediction model based on an improved Transformer network; the initialized data is input into the air quality prediction model to obtain the air quality prediction results.

Images