An air quality forecasting method fusing monitoring image data

Abstract

The application discloses an air quality forecasting method fusing monitoring image data, comprising the following steps: constructing an LSTM single-channel deep learning network for extracting features of historical air quality data, meteorological data and time data; constructing a CNN-LSTM single-channel deep learning network for extracting features of monitoring images; integrating the two single-channel networks to construct a double-channel integrated deep learning model for air quality forecasting; collecting monitoring images, air quality and meteorological data and other multi-source data to construct an air quality multi-source data set; using a sliding window method and a 'T1:T2' mode to generate sequence samples from the constructed air quality multi-source data set; training the double-channel integrated deep learning model by using a sequence sample training set to obtain a regression model with multi-step and multi-variable output forecasting capability; and inputting to-be-detected sequence sample data into the regression model to output forecasting results of multiple air quality indexes at multiple future time points. The application combines two single-channel neural networks to construct a relationship model between air pollutant concentration and monitoring images, meteorological data and time, realizes time sequence forecasting of air quality, and integrates outdoor monitoring image data to improve the forecasting precision of air quality.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of air monitoring, and relates to an air quality forecasting method, in particular to an air quality forecasting method fusing monitoring image data. BACKGROUND

[0002] Air pollution has many harmful effects on human beings and the environment. Therefore, it is necessary to make an estimate of the air quality status and trend in the future period of time, and provide pollution warning information when there is a possibility of serious pollution or the pollution concentration exceeds a certain limit value, so that the relevant environmental protection agencies can take corresponding preventive measures in advance to prevent air pollution hazards from occurring. Forecasting the concentration of various pollutants in the air in advance is the basis for strengthening air pollution prevention and control and achieving comprehensive environmental management, and has important practical significance for people's daily health and government environmental protection.

[0003] Air quality changes have the characteristics of coexistence and superposition of multiple pollutants, mutual influence of multiple media, and coupling of multiple processes. A single data source often cannot fully reflect the complex relationship between them. While multi-source data can provide information from different dimensions and angles to make up for the shortcomings of a single data source. By fusing and comprehensively analyzing multi-source data, new data relationships and mutual influences can be explored, and more comprehensive and rich information can be obtained, thereby improving the accuracy and reliability of the forecast. According to the summary and analysis of existing air quality forecasting research, it can be found that the most commonly used multi-source data for single-site air quality forecasting includes air pollutant data, meteorological data, and time data, because air quality changes are closely related to atmospheric pollutants, meteorological conditions, and human activities, which interact in a complex way in urban environments. Therefore, it is scientific and reasonable to use these factors as forecasting factors to forecast air quality. Therefore, in the present application, the commonly used data of air pollutant data, meteorological data, and time data will also be used.

[0004] Furthermore, with the widespread use of imaging devices such as surveillance cameras, portable cameras, and smartphones, and the rapid development of computer vision technology for image analysis, using real-scene images for urban air quality perception has become a promising and popular research direction. Currently, numerous studies have explored the relationship between air quality and outdoor real-scene images, utilizing images to achieve qualitative and quantitative monitoring of urban ambient air quality. Unlike air quality monitoring stations, tens of thousands of fixed imaging devices (e.g., surveillance and traffic cameras) are widely deployed in various cities, facilitating timely collection of image or video information. Moreover, this data continuously records the dynamic changes in air quality, containing rich spatiotemporal information. Therefore, these surveillance video or image data can be combined to deeply mine their inherent information, improving the accuracy and precision of air quality forecasts. To this end, this invention proposes an air quality forecasting method that integrates multi-source heterogeneous data such as surveillance image data, air pollution data, meteorological data, and temporal data. Summary of the Invention

[0005] Purpose of the invention: To improve the accuracy of air quality forecasting, this invention provides an air quality forecasting method that integrates monitoring image data. This method fully considers the correlation between air quality and imaging, the temporal variation characteristics of air pollution, the influence of meteorological factors, and the interaction between pollutants. It combines two single-channel deep learning models to learn the spatial characteristics and temporal correlations of homogeneous and heterogeneous data respectively, and constructs a regression model of multi-source data spatiotemporal characteristics and air quality, thereby achieving high-precision, all-weather air quality forecasting.

[0006] Technical Solution: To achieve the above objectives, this invention provides an air quality forecasting method that integrates monitoring image data, comprising the following steps:

[0007] S1: Construct a Long-Short Term Memory (LSTM) single-channel deep learning network to learn deep features of data with the same structure, such as air pollutant concentrations, meteorological elements, and time information at multiple historical moments.

[0008] S2: Construct a CNN-LSTM single-channel deep learning network for feature extraction of monitoring images. Connect the Convolutional Neural Network (CNN) and LSTM. The CNN serves as the image spatial feature extractor, and the LSTM serves as the image temporal feature extractor. Together, they extract the spatiotemporal information contained in the monitoring image data from multiple historical moments.

[0009] S3: Fuse the constructed LSTM and CNN-LSTM two single-channel network models, establish the mapping relationship between the fused features and the future multiple time points of multiple air quality indicators, and form a double-channel integrated deep learning model with multiple-step and multi-variable output;

[0010] S4: Collect long-term continuous monitoring camera video images, continuously collected air pollutant concentrations and meteorological data, and based on the time stamp of the data, combine the data records at the same time one by one to construct an air quality multi-source data set;

[0011] S5: Use the sliding window method and "T1:T2" mode to generate sequence samples from the constructed air quality multi-source data set, and divide the training set;

[0012] S6: Train the double-channel integrated deep learning model using the sequence sample training set to obtain an air quality regression model with prediction capability;

[0013] S7: Input the image sequence to be detected into the regression model obtained in step S6, and output the predicted future multiple time points of multiple air pollutant concentration values.

[0014] Further, the specific process of constructing the LSTM single-channel deep learning network in step S1 is as follows:

[0015] First, a homogeneous data set composed of T1 time step historical air pollutant concentration data, meteorological element data and time information is taken as input. Then, it is transmitted to the LSTM layer to learn and extract the time sequence features of the multi-source homogeneous data, and finally the feature vector F lstm .

[0016] Further, the specific process of constructing the CNN-LSTM single-channel deep learning network in step S2 is as follows:

[0017] First, a sequence with dimensions of (T1, H, W, C) composed of multiple consecutive monitoring images is taken as input, T1 is the length of the image sequence, which is also the time dimension of the historical data to be learned, (H, W, C) is the dimension of a single monitoring image, H, W and C are the height, width and color channel number of the input image. Then, remove the fully connected layer and output layer in the complete CNN framework, only keep the feature extractor part in front for spatial feature learning of the image. Subsequently, seamlessly connect the LSTM network layer, take the spatial features of the image sequence output by the CNN feature extractor as the input of the LSTM layer, store and write them into the memory unit of the LSTM, learn the time characteristics between them, and thus obtain the spatio-temporal feature vector F cnn-lstm .

[0018] Further, the network structure of the dual-channel integrated deep learning model in step S3 is as follows:

[0019] The F lstm and F cnn-lstm characteristic vectors output by the two single-channel networks are fused, and a full connection layer and an output layer are added after the fusion layer. The fused features are input into the full connection layer to realize mapping with the output target. The number of neurons in the output layer is consistent with the time length to be predicted. Moreover, the output layer of the model is provided with a sigmoid activation function to control the output value in the range of [0, 1].

[0020] Further, in the data set construction process in step S4, the data needs to be preprocessed, and the specific process is as follows:

[0021] Firstly, all the monitoring images in the data set are compressed to the input size required by the deep learning network. Then, the image data and air quality data are normalized respectively to accelerate the convergence speed of model training. For image data, the pixel value of the input image is divided by the upper limit value of the pixel, which is scaled from 0-255 to 0-1. For air pollutant concentration data and meteorological data, the Min-Max Normalization method is used to map the air pollutant concentration and meteorological element indicators to 0-1.

[0022] Further, the generation method of the sequence sample in step S5 is as follows:

[0023] The sliding window method is used to generate sequence samples from the entire data. The time length of the historical data and the time length of the target to be predicted are set in the "T1:T2" mode, that is, the data collected for T1+T2 consecutive hours is taken as a time sequence sample. In the time sequence, the first T1 hours of monitoring image data, air quality data, meteorological data and time information are used as historical data for prediction, and the last T2 hours of air pollutant concentration values are used as the target to be estimated.

[0024] Further, the model training process in step S6 needs to initialize the model hyperparameters in advance, including:

[0025] Loss function, gradient descent optimizer in the back propagation process, learning rate for parameter update, training round number, batch size.

[0026] Beneficial effects: compared with the prior art, the application increases the utilization of monitoring image data, uses a double-channel integrated deep learning model to extract and learn the spatiotemporal characteristics of historical air quality, weather, time and image and other multi-source heterogeneous data, constructs a relationship model of spatiotemporal characteristics and air quality, and can realize the prediction of future air quality. Since the time sequence variation characteristics of air pollutants, the influence of meteorological factors, the correlation between pollution factors and the characteristics of image representation of air quality are considered, the method effectively improves the prediction accuracy of all-weather air quality. BRIEF DESCRIPTION OF DRAWINGS

[0027] Figure 1 is a flowchart of the method of the application;

[0028] Figure 2 is a network structure diagram of the double-channel integrated deep learning model;

[0029] Figure 3 is a scatter plot of the prediction results of the method of the application on the test data set;

[0030] Figure 4 is an example diagram of the prediction results of the method of the application on the test image;

[0031] Figure 5 is a comparison result of the method of the application and the traditional prediction method. DETAILED DESCRIPTION

[0032] The application will be further illustrated below in combination with the drawings and specific embodiments, and it should be understood that these embodiments are only used to illustrate the application and not to limit the scope of the application, and after reading the application, those skilled in the art can make various equivalent modifications of the application, which all fall within the scope defined by the claims attached hereto.

[0033] The application provides an air quality prediction method fusing monitoring image data, as shown in Figure 1 In this embodiment, the method is applied as an example, i.e. using historical 12 hours (T1) of multi-source heterogeneous data to predict the concentrations of six air pollutants PM 2.5 , PM 10 , SO2, NO2, O3 and CO in the next 12 hours (T2), which specifically comprises the following steps:

[0034] Step 1: build an LSTM single-channel deep learning network based on Python programming language and TensorFlow open source framework, the LSTM single-channel network takes 12 historical time steps of six air pollutant concentration data, weather data and time data and other multi-source homogeneous data as input, transmits to the LSTM network layer to learn the time characteristics of the data, the LSTM layer contains 200 hidden layer neurons, and outputs a 200-dimensional feature vector F after the LSTM layerlstm .

[0035] Step 2: Take VGG16 as the CNN extractor, build a VGG16-LSTM single-channel deep learning network, and convert the 12-hour continuous monitoring image sequence into a multi-dimensional array form of (12, 224, 224, 3), and input it into the VGG16-LSTM feature extractor for learning. After the image sequence is extracted by the VGG16 structure, a feature vector with a dimension of (12, 512) is output, and then transmitted to the memory unit of the LSTM layer for extracting and learning the time correlation between the spatial feature sequence. In order to match the feature output of VGG16, the number of hidden neurons in the LSTM layer is set to 512. Finally, a 512-dimensional spatiotemporal feature F vgg16-lstm is obtained, waiting for the next step of integrated learning.

[0036] Step 3: In order to fully tap the spatiotemporal information of multi-source heterogeneous data, the method of the application associates and merges the time sequence features of the isomorphic data extracted by the LSTM single-channel network with the spatiotemporal features of the image sequence extracted by the VGG16-LSTM single-channel network, realizes "feature level" fusion, and the fused features are represented as F c After feature fusion, two fully connected layers are connected to form a mapping relationship from input to output. The first fully connected layer contains 200 neuron nodes, which further map the fused features F c into high-level abstract features. The second fully connected layer is equivalent to the output layer of the model, and the number of neurons is 12, corresponding to the time length to be predicted, and each time point will output PM 2.5 , PM 10 , SO2, NO2, O3 and CO six variables. Therefore, the whole dual-channel integrated air quality prediction deep learning model is as shown in Figure 2 .

[0037] Step 4: After collecting 8499 monitoring camera images taken every hour from 00:00 on January 1, 2021 to 23:00 on December 31, 2021 in Shanghai Lujiazui, obtaining PM 2.5 , PM 10 , SO2, NO2, O3 and CO six air pollutant concentration data corresponding to the time of each image from the historical data published by China Environmental Monitoring Center, collecting and calculating 7 kinds of meteorological elements such as precipitation, pressure, temperature, evapotranspiration, relative humidity, wind speed and wind direction corresponding to the time of each image from the fifth generation meteorological reanalysis data ERA5, and combining the month and time information of each image, an air quality multi-source data set is constructed. During the data set construction process, the data needs to be preprocessed, and the specific process is as follows:

[0038] Step 2-1: compress all monitoring images in the dataset to the input size required by the VGG16 network, i.e. 224x224; divide the pixel value of each image by 255 to scale it from 0-255 pixel value to 0-1;

[0039] Step 2-2: normalize the air quality data and meteorological data using the min-max normalization method, and map the numerical values of PM 2.5 , PM 10 , SO2, NO2, O3 and CO concentrations, and precipitation, pressure, temperature, evapotranspiration, relative humidity, wind speed and wind direction, seven meteorological elements, to 0-1.

[0040] Step 5: use the sliding window method and "T1:T2" mode to generate training sequence samples for the constructed air quality multi-source dataset, i.e. for the entire time series data, take the first 1 to 12 hours of continuous images, six air pollutant concentrations, seven meteorological elements, and time information as the first historical sequence, and take the PM 2.5 , PM 10 , SO2, NO2, O3 and CO six air pollution concentration values from the 13th to 24th hour as the first target label; i.e. take the second to 13th hour of continuous images, six air pollutant concentrations, seven meteorological elements, and time information as the second historical sequence, and take the PM 2.5 , PM 10 , SO2, NO2, O3 and CO six air pollution concentration values from the 14th to 25th hour as the second target label; continue to generate new samples in this way until the pollutant data at the last time of the entire time series data is included in the last target label. According to this sequence generation method, a total of 8365 sequence samples are generated from the self-collected air quality image dataset, and then all sequence samples are divided into training set and test set in a 1:1 ratio for training and testing.

[0041] Step 6: set the loss function of the model as mean error MSE; set the gradient descent optimization algorithm as Adam optimizer, and set the learning rate as 0.00001; set the number of training rounds as 100, and set the batch size as 8. Then, use the training sample set to train the dual-channel integrated deep learning model, and finally obtain an air quality regression prediction model with prediction ability;

[0042] Step 7: input each historical sequence in the test set into the regression model obtained in step 6, and output the corresponding PM 2.5 , PM 10 , SO2, NO2, O3 and CO concentrations for the next 12 hours.

[0043] In this embodiment, the coefficient of determination (R2 The performance of the proposed method on the test set is evaluated using the root mean square error (RMSE) and the coefficient of determination (R²). A good model performance should have a high R². 2 And a low RMSE. 2 The formula for calculating RMSE is as follows:

[0044]

[0045] Ultimately, the air quality forecasting method fused with monitoring image data proposed in this invention achieved good performance on the test set, such as... Figure 3 As shown, all air pollutants achieved high R values. 2 And lower RMSE: PM 2.5 R 2 The RMSE values ​​were 0.95 and 4.79 μg / m, respectively. 3 PM 10 R 2 The RMSE and RMSE were 0.90 and 11.51 μg / m, respectively. 3 SO2's R 2 The RMSE and RMSE were 0.78 and 1.17 μg / m, respectively. 3 NO2's R 2 The RMSE values ​​were 0.95 and 5.17 μg / m², respectively. 3 O3's R 2 The RMSE and RMSE were 0.96 and 7.57 μg / m, respectively. 3 CO's R 2 The RMSE and RMSE were 0.92 and 0.07 μg / m, respectively. 3 .

[0046] Meanwhile, an example of the prediction results of the method of the present invention on the test image is provided ( Figure 4 ) and the comparison results of the method of the present invention and the traditional method on the test set ( Figure 5 ).

[0047] It is evident that the evaluation results of the method of the present invention are better than those of traditional machine learning forecasting methods, and the method of the present invention effectively improves the forecasting accuracy of air quality.

Claims

1. An air quality forecasting method that fuses monitoring image data, characterized by, The method comprises the following steps: S1: constructing a long short-term memory (LSTM) single-channel deep learning network for learning deep features of historical air pollutant concentration, meteorological elements and time information data at multiple time points; S2: constructing a CNN-LSTM single-channel deep learning network for monitoring image feature extraction, connecting a convolutional neural network (CNN) and an LSTM, the CNN serving as an image spatial feature extractor and the LSTM serving as an image temporal feature extractor, to jointly extract spatiotemporal information contained in monitoring image data at multiple time points; The specific process of constructing the CNN-LSTM single-channel deep learning network in step S2 is as follows: First, a plurality of continuously captured monitoring images are taken as input in the form of a sequence with dimensions (T1, H, W, C), T1 being the length of the image sequence and also the time dimension of the historical data to be learned, (H, W, C) being the dimensions of a single monitoring image, H, W and C being the height, width and color channel number of the input image; then, the full connection layer and the output layer in the complete CNN framework are removed, and only the feature extractor part in front is kept for spatial feature learning of the image; Subsequently, a seamless connection LSTM network layer is used to take the spatial features of the image sequence output by the CNN feature extractor as the input of the LSTM layer, store and write the memory cells of the LSTM, learn the temporal characteristics between them, and thus obtain the spatiotemporal feature vector of the monitoring image ; S3: performing feature fusion on the two single-channel network models of LSTM and CNN-LSTM, establishing a mapping relationship between the fused features and multiple air quality indexes at future time points, and forming a double-channel integrated deep learning model with multiple-step and multiple-variable outputs; The network structure of the double-channel integrated deep learning model in step S3 is as follows: The two single-channel networks respectively output and Two feature vectors are fused, a full connection layer and an output layer are added after the fusion layer, the fused features are input into the full connection layer to realize mapping with the output target, and the number of neurons of the output layer is consistent with the time length to be predicted; wherein, the output layer of the model is provided with a sigmoid activation function to control the output value in the range of [0, 1]; S4: collecting long-term continuously captured monitoring camera video images, continuously collected air pollutant concentration and meteorological data, and combining data records at the same time point one by one based on the time stamp of the data to construct an air quality multi-source data set; S5: using a sliding window method and a "T1:T2" mode to generate sequence samples from the constructed air quality multi-source data set, and dividing out a training set; S6: training the double-channel integrated deep learning model using the sequence sample training set to obtain an air quality regression model with prediction capability; S7: inputting a to-be-detected image sequence into the regression model obtained in step S6 to output predicted air pollutant concentration values at multiple future time points.

2. The method of claim 1, wherein the method of air quality forecasting by fusion of monitoring image data is characterized by, The specific process of constructing the LSTM single-channel deep learning network in step S1 is as follows: First, the isomorphic data set composed of T1 time step historical air pollutant concentration data, meteorological element data and time information is taken as input; then, it is transmitted to the LSTM layer to learn and extract the time sequence characteristics of multi-source isomorphic data, and finally the feature vector is output .

3. The method of claim 1, wherein the method further comprises: In the data set construction process in step S4, the data needs to be preprocessed, and the specific process is as follows: First, all monitoring images in the data set are compressed to the input size required by the deep learning network; then, the image data and air quality data are normalized respectively to speed up the convergence speed of model training; for the image data, the pixel value of the input image is divided by the upper limit value of the pixel, which is scaled from 0-255 to 0-1; and for the air pollutant concentration data and meteorological data, the minimum-maximum normalization method is used to map the air pollutant concentration and meteorological element indexes to 0-1.

4. The method of claim 1, wherein the method of air quality forecasting by fusion of monitoring image data is characterized by, The sequence sample generation method in step S5 is as follows: The sliding window method is used to generate sequence samples from the entire data. The time length of the historical data and the time length of the target to be predicted are set in the "T1: T2" mode, that is, the data collected for T1+T2 consecutive hours is taken as a time sequence sample. In the time sequence, the monitoring image data, air quality data, meteorological data and time information of the previous T1 hours are used as historical data for prediction, and the air pollutant concentration value of the last T2 hours is taken as the target to be estimated.

5. The method of claim 1, wherein the method of air quality forecasting by fusion of monitoring image data is characterized by, The model training process in the step S6 needs to initialize the model hyperparameters in advance, specifically including: Loss function, gradient descent optimizer in the back propagation process, learning rate for parameter update, training round number, batch size.

Images