Ambient air quality detection method based on monitoring camera image
By combining CNN and LSTM models, the spatiotemporal characteristics of the monitoring camera images are extracted and the air quality regression model is constructed, which solves the problem of night air quality estimation and achieves high-precision air quality monitoring all-weather.
Patent Information
- Application Number
- CN202411092259.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-09
- Publication Date
- 2024-12-13
Smart Images
Figure CN119126261A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of air monitoring, and in particular to an environmental air quality detection method based on monitoring camera images. Background Art
[0002] Air pollution usually refers to the phenomenon that harmful substances in the atmosphere reach or exceed a certain limit, causing harm to humans, animals, plants and the environment. At present, ground monitoring of ambient air quality and satellite remote sensing inversion are the most commonly used measurement methods with high accuracy and stability. However, the installation and maintenance costs of ground monitoring station instruments are high, and the stations cannot be widely deployed, resulting in limited monitoring areas; satellite remote sensing inversion has a time lag and is easily affected by weather such as clouds, rain, snow and bright surfaces, resulting in serious data loss. Therefore, there is an urgent need for a high-precision, low-cost air quality detection method to achieve high temporal and spatial resolution air quality monitoring. With the popularization of imaging devices such as portable cameras, smart phones and surveillance cameras, we are prompted to explore and develop new methods for air quality monitoring from the perspective of opportunity perception. At present, tens of thousands of surveillance cameras and traffic cameras have been deployed in major cities, which can collect video and image data intensively and conveniently, which provides the possibility for refined monitoring of regional ambient air quality.
[0003] At present, many studies have used image analysis technology, combined with computer vision algorithms such as machine learning and deep learning, to explore the correlation between ambient air pollution and the quality of real-life or street view images, and achieved qualitative and quantitative estimation of PM2.5, PM10 or AQI. However, most of these methods only estimate ambient air quality through a single image. Moreover, some image features, such as transmittance and color, can be easily and effectively calculated and extracted from daytime images, but are difficult to achieve for low-intensity nighttime images. This is because there is little ambient light at night, and the bright light in the scene is mainly composed of light sources such as street lamps and light transmitted through windows in illuminated rooms, which makes the estimation of nighttime air quality still a challenge. Summary of the invention
[0004] The purpose of the present invention is to improve the level of nighttime air quality estimation and realize high-precision, all-weather, multi-indicator air quality monitoring, and to provide an ambient air quality detection method based on surveillance camera images.
[0005] The purpose of the present invention can be achieved by the following technical solutions:
[0006] A method for detecting ambient air quality based on monitoring camera images, the method comprising the following steps:
[0007] S1. Using the feature extractor part of the CNN convolutional neural network as an image spatial feature extractor, using the LSTM network layer in the LSTM long short-term memory network as an image temporal feature extractor, connecting the image spatial feature extractor and the image temporal feature extractor, and adding a fully connected layer to obtain a hybrid deep learning network;
[0008] S2, obtain long-term continuous surveillance camera video images and the air quality data corresponding to the image shooting time, and construct an air quality time series image dataset;
[0009] S3, using the sliding window method to construct sequence samples from the air quality time series image dataset, using the "T:1" mode to set the length of the sequence samples and the prediction target, and dividing the generated sequence samples into a training set and a test set;
[0010] S4. Train the hybrid deep learning network based on the training set and verify it based on the test set to obtain the multi-index regression model of air quality;
[0011] S5. Acquire actual air quality time series image data, input the actual air quality time series image data into the air quality multi-index regression model, and obtain the predicted PM2.5 concentration, PM10 concentration and AQI three air quality indicators.
[0012] Furthermore, the specific steps of S3 are:
[0013] Take the images of continuous T hours in the air quality time series image dataset as a sequence sample, and the air quality index value of the last hour in the sequence sample as the prediction target. For the air quality time series image dataset containing n images, take the first to Tth continuous images [I1,I2,...I T-1 ,I T ] as the first sequence sample, with the Tth image I T The corresponding air quality index value is taken as the first target label, and the second to T+1th continuous images [I2,I3,...I T ,I T+1 ] as the second sequence sample, with the T+1th image I T+1 The corresponding air quality index value is used as the second target label, and so on, until the last image I in the air quality time series image dataset n Included in the last sequence sample, all sequence samples are integrated, and the sequence samples are divided into a training set and a test set.
[0014] Furthermore, the dimension of the sequence samples is (T, H, W, C), where T is the length of the image sequence, which is also the added time dimension, and (H, W, C) is the dimension of a single image, the image size is H×W, and H, W, and C are the height, width, and number of color channels of the input image, respectively.
[0015] Furthermore, the specific steps of S1 are:
[0016] Set the input of the hybrid deep learning network, then remove the fully connected layer and output layer of the complete CNN model, retain the feature extractor part of the CNN model as the image spatial feature extractor, use the spatial features of the image sequence output by the image spatial feature extractor as the input of the LSTM network layer in the LSTM long short-term memory network, and use the LSTM network layer as the image temporal feature extractor. Connect the image spatial feature extractor and the image temporal feature extractor, then add two fully connected layers after the image temporal feature extractor, and use the output of the image temporal feature extractor as the input of the fully connected layer to obtain a hybrid deep learning network.
[0017] Furthermore, among the two fully connected layers after the image temporal feature extractor, the first fully connected layer directly connected to the image temporal feature extractor is used to decode the output of the image temporal feature extractor.
[0018] Furthermore, the second fully connected layer serves as the output layer of the deep learning network.
[0019] Furthermore, the specific steps of S2 are:
[0020] Acquire long-term continuous surveillance camera video images and the air quality data at the time when the images were taken, compress and normalize the surveillance camera video images, normalize the air quality data at the same time, and construct an air quality time series image dataset based on the normalized images and air quality data.
[0021] Furthermore, the compression and normalization of the surveillance camera video image is specifically as follows:
[0022] The surveillance camera video image is compressed to the input size required by the image space feature extractor, and the pixel value of each image is divided by the upper limit of the pixel value to scale the pixel value to between 0 and 1.
[0023] Furthermore, the normalizing of the air quality data is specifically: mapping the air quality data to between 0 and 1 based on a minimum-maximum normalization method.
[0024] Furthermore, the hybrid deep learning network is trained based on the training set as follows:
[0025] The training set is input into the hybrid deep learning network, and the mean square error loss function is calculated based on the output of the hybrid deep learning network. Based on the loss function and the Adam optimizer, the gradient descent optimization algorithm is used to optimize the parameters of the hybrid deep learning network, and GPU acceleration is used to obtain the air quality multi-indicator regression model.
[0026] Compared with the prior art, the present invention has the following beneficial effects:
[0027] The present invention combines two deep learning models, CNN and LSTM, to extract and learn the spatiotemporal features of image sequences, and construct a relationship model between the spatiotemporal features of images and multiple air quality indicators, which can achieve regression and prediction of air quality. Since the time dependence of air quality changes is taken into account, this method improves the current difficulty in estimating air quality from nighttime images and effectively improves the estimation accuracy of air quality throughout the day. BRIEF DESCRIPTION OF THE DRAWINGS
[0028] Figure 1 is a flow chart of the present invention;
[0029] Figure 2 A network structure diagram of the regression model of the present invention;
[0030] Figure 3 A scatter plot of the estimation results of the regression model of the present invention on the test data set;
[0031] Figure 4 This is an example diagram of the estimation results of the model of the present invention for a test image. DETAILED DESCRIPTION
[0032] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.
[0033] The present invention proposes an environmental air quality detection method based on surveillance camera images, which combines CNN and LSTM two deep learning models to learn the spatial features and temporal associations of image sequences respectively, and constructs a regression model of image spatiotemporal features and air quality, thereby improving the level of nighttime air quality estimation and achieving high-precision, all-weather, multi-indicator air quality monitoring. The flowchart of the method of the present invention is as follows Figure 1 As shown, the method comprises the following steps:
[0034] S1. Using the feature extractor part of the CNN convolutional neural network as an image spatial feature extractor, using the LSTM network layer in the LSTM long short-term memory network as an image temporal feature extractor, connecting the image spatial feature extractor and the image temporal feature extractor, and adding a fully connected layer to obtain a hybrid deep learning network;
[0035] S2, obtain long-term continuous surveillance camera video images and the air quality data corresponding to the image shooting time, and construct an air quality time series image dataset;
[0036] S3, using the sliding window method to construct sequence samples from the air quality time series image dataset, using the "T:1" mode to set the length of the sequence samples and the prediction target, and dividing the generated sequence samples into a training set and a test set;
[0037] S4. Train the hybrid deep learning network based on the training set and verify it based on the test set to obtain the multi-index regression model of air quality;
[0038] S5. Acquire actual air quality time series image data, input the actual air quality time series image data into the air quality multi-index regression model, and obtain the predicted PM2.5 concentration, PM10 concentration and AQI three air quality indicators.
[0039] The network structure of the CNN-LSTM hybrid deep learning model in S1 of the present invention is:
[0040] The method proposed in the present invention will improve the accuracy of air quality estimation by learning the temporal features between continuously shot images. Therefore, the input of the network is no longer a single image, but a sequence of multiple continuously shot images with dimensions (T, H, W, C) as input. T is the length of the image sequence, which is also the added time dimension. (H, W, C) is the dimension of a single image. The image size is H×W, and H, W, C are the height, width and number of color channels of the input image, respectively. Then, the fully connected layer and output layer in the complete CNN framework are removed, and only the previous feature extractor part is retained for image spatial feature learning. In order to reduce the number of model parameters and avoid overfitting, a global average pooling layer is added at its tail end to reduce the dimension of the feature map and reduce the number of parameters. Subsequently, the CNN feature extractor and the LSTM network layer are seamlessly connected, and the spatial features of the image sequence output by the CNN feature extractor are used as the input of the LSTM layer. These spatial features are stored and written into the memory unit of the LSTM, and the temporal characteristics between them are read and transmitted through the gate structure of the LSTM, thereby obtaining the spatiotemporal features of the image. Finally, a fully connected layer and an output layer are added after the LSTM layer, and the output of the LSTM layer is used as the input of the fully connected layer to achieve mapping of features and target labels. The CNN-LSTM model designed by the present invention can output multiple air quality indicators at the same time, so the number of neurons in the output fully connected layer is consistent with the number of output target values. In order to accelerate the convergence of model training, all input data are generally normalized before model training. Therefore, the method of the present invention sets a sigmoid activation function in the output layer of the CNN-LSTM model to limit the output to the range of [0,1].
[0041] In the process of constructing the dataset in S2, data preprocessing is required. The specific process is as follows:
[0042] First, all monitoring images in the data set are compressed to the input size required by the CNN network, that is, H×W×C. Then, the image data and air quality data are normalized to speed up the convergence of model training. For image data, the pixel value of each image is divided by the pixel upper limit of 255, and it is scaled from 0-255 to 0-1; for air quality data, the minimum-maximum normalization method is used to map the air quality index value to 0-1. The formula is as follows:
[0043]
[0044] For each type of air quality indicator, x represents the original data, x max and x min They represent the maximum and minimum values of the original data respectively, and x′ represents the normalized result.
[0045] The method for generating sequence samples in S3 is:
[0046] In order to obtain as many image sequence experimental sample data as possible, the study used the sliding window method to generate sequence samples from the entire data, and used the "T:1" mode to set the length of the sequence sample and the prediction target, that is, the images of consecutive T hours were used as a time series sample, and the air quality index value of the last hour in the time series was used as the target to be estimated. The specific steps are: for the entire time series data containing n images, take the first to Tth continuous images [I1,I2,...I T-1 ,I T ] as the first sequence sample, with the Tth image I T The corresponding index value is taken as the first target label; take the second to T+1th continuous images [I2,I3,...I T ,I T+1 ] as the second sequence sample, with the T+1th image I T+1 The corresponding indicator value is used as the second label; and so on, new samples are generated until the last image I of the entire time series data n Contained in a sequence sample.
[0047] The model parameters of the model training process in S4 are specifically:
[0048] During the model training process, there are some important hyperparameters that will affect the model training effect, so these hyperparameters need to be initialized before training the model. First, because CNN-LSTM is a regression model, the loss function in the model training process is set to the mean square error (MSE) to measure the accuracy of the model training. Then the Adam optimizer is set as the gradient descent optimization algorithm in the back propagation process. Secondly, the step size when updating the parameters is the learning rate. The learning rate is very important for the training effect of the model. It can affect the convergence speed, stability and final generalization performance of the model. Generally speaking, the larger the learning rate, the faster the model converges, but it is also prone to oscillation and instability; the smaller the learning rate, the slower the convergence speed, but it is more stable, which can help the model jump out of the local optimal solution and obtain better generalization performance.
[0049] Compared with the prior art, the present invention combines two deep learning models, CNN and LSTM, to extract and learn the spatiotemporal features of image sequences, and construct a relationship model between the spatiotemporal features of images and multiple air quality indicators, which can achieve regression and prediction of air quality. Since the time dependence of air quality changes is taken into account, this method improves the current difficulty in estimating air quality from nighttime images and effectively improves the estimation accuracy of air quality throughout the day.
[0050] The following uses the VGG16 deep learning model as an example of a CNN model, and takes the three air quality indicators of PM2.5 concentration, PM10 concentration and air quality index (AQI) as prediction targets as an example. When the steps of the present invention are specifically executed, the following steps may be included:
[0051] Step 1: Build a VGG16-LSTM hybrid deep learning network for air quality estimation based on Python programming language and TensorFlow open source framework (such as Figure 2 As shown in the figure), a relationship model between the image sequence and the three air quality indicators of PM2.5 concentration, PM10 concentration and air quality index AQI is established. The specific construction process is as follows:
[0052] Step 1-1: The model input is a sequence of multiple continuously shot surveillance images with a dimension of (7, 224, 224, 3). The length of the image sequence T is 7 and the image size is 224×224×3.
[0053] Step 1-2: Remove the fully connected layers and output layers in the complete VGG16 network framework, and only keep the feature extractor part in front for spatial feature learning of the image. In order to reduce the number of model parameters and avoid overfitting, a global average pooling layer is added at the end to reduce the dimension of the feature map and reduce the number of parameters. The dimension of the output feature after VGG16 feature extraction is (7,512);
[0054] Step 1-3: Seamlessly connect the VGG16 feature extractor and the LSTM network layer, and use the spatial features of the image sequence output by the VGG16 feature extractor as the input of the LSTM layer. In order to match the feature output of VGG16, the number of hidden neurons in the LSTM layer is set to 512. These spatial features are stored and written into the memory cells of the LSTM, and the temporal characteristics between them are read and transmitted through the gate structure of the LSTM, thereby obtaining the spatiotemporal features of the image.
[0055] Step 1-4: Add two fully connected layers after the LSTM layer, and the output of the LSTM layer is used as the input of the fully connected layer. The first fully connected layer contains 32 neuron nodes, which are used to decode the output of the LSTM layer. The second fully connected layer is equivalent to the output layer of the model, which realizes the mapping of features and target labels. The VGG16-LSTM model designed in the example of the present invention can simultaneously output the three air quality indicators of PM2.5 concentration, PM10 concentration and AQI at each estimated moment, so the number of neurons in the last fully connected layer is 3. In addition, a sigmoid activation function is set in the output layer of the VGG16-LSTM model to limit the output to the range of [0,1].
[0056] Step 2: 2691 real-life images taken by surveillance cameras every hour from 19:00 on July 27, 2021 to 15:00 on December 31, 2021 were collected, and the three air quality indicators of PM2.5 concentration, PM10 concentration and AQI at the corresponding time of each image were obtained from the historical data published by the Environmental Monitoring Center to construct an air quality image dataset. The data needs to be preprocessed during the dataset construction process. The specific process is as follows:
[0057] Step 2-1: Compress all surveillance images in the dataset to the input size required by the VGG16 network, i.e. 224×224;
[0058] Step 2-2: Normalize the image data to speed up the convergence of model training. Divide the pixel value of each image by the upper pixel value of 255, and scale it from 0-255 to 0-1;
[0059] Step 2-3: Normalize the air quality data to speed up the convergence of model training. Use the Min-Max Normalization method to map PM2.5 concentration, PM10 concentration, and AQI value to between 0 and 1. The formula is as follows:
[0060]
[0061] For each type of air quality indicator, x represents the original data, x max and x min They represent the maximum and minimum values of the original data respectively, and x′ represents the normalized result.
[0062] Step 3: Using the sliding window method and the "T:1" mode, generate training sequence samples for the constructed air quality time series image dataset, that is, for the entire time series data containing 2691 images, take the 1st to 7th continuous images as the first sequence sample, and use the PM2.5 concentration, PM10 concentration and AQI value corresponding to the 7th image as the first target label; take the 2nd to 8th continuous images as the second sequence sample, and use the PM2.5 concentration, PM10 concentration and AQI value corresponding to the 8th image as the second label; and so on, continue to generate new samples until the 2691th image of the entire time series data is included in the last sequence sample. According to the sequence generation method, a total of 2259 sequence samples are generated for the self-collected air quality image dataset, and then all sequence samples are divided into training sets and test sets in a 1:1 ratio for training and testing, respectively.
[0063] Step 4: Set the model's loss function to MSE; set the gradient descent optimization algorithm to Adam optimizer, and the learning rate to 0.00001; set the number of training rounds to 100, and the batch size to 8. Then, the VGG16-LSTM hybrid deep learning model is trained using the training sample set and accelerated using GPU, and finally an air quality regression model with estimation capability is obtained;
[0064] Step 5: Input the image sequence in the test set into the regression model obtained in step 4, and output the predicted PM2.5 concentration, PM10 concentration and AQI three air quality indicators.
[0065] In this example, the coefficient of determination (R 2 ) and root mean square error (RMSE) were used to evaluate the performance of the method on the test set. A good model performance should have a high coefficient of determination R 2 and low RMSE. 2 And the RMSE calculation formula is as follows:
[0066]
[0067] Finally, the VGG16-LSTM instance model of the ambient air quality detection method based on surveillance camera images proposed in the present invention achieved good performance on the test set, and the estimated R of PM2.5 2 The mean square error was 0.89, and the RMSE was 6.51 μg / m 3 , PM10 estimation R 2 The mean square error was 0.87, and the RMSE was 10.89 μg / m 3 , AQI estimation R 2 is 0.85, RMSE is 9.03, and the overall performance is Figure 3 Compared with the independent VGG16 model without LSTM model, the VGG16-LSTM model effectively improves the air quality estimation level during the day and night. The results are shown in Table 1.
[0068] Table 1 Comparison between the VGG16-LSTM instance model and the independent VGG16 model without LSTM model
[0069]
[0070] At the same time, the estimation results of the VGG16-LSTM instance model for the test image are as follows Figure 4 As shown in Figure 2, the comparison results of the VGG16-LSTM instance model and other existing methods on the public PM2.5 image dataset are shown in Table 2.
[0071] Table 2 Comparison results of VGG16-LSTM instance model and other existing methods on the public PM2.5 image dataset
[0072]
[0073] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.
Claims
1. A method for detecting ambient air quality based on surveillance camera images, It is characterized in that The method comprises the following steps: S1. Using the feature extractor part of the CNN convolutional neural network as an image spatial feature extractor, using the LSTM network layer in the LSTM long short-term memory network as an image temporal feature extractor, connecting the image spatial feature extractor and the image temporal feature extractor, and adding a fully connected layer to obtain a hybrid deep learning network; S2, obtain long-term continuous surveillance camera video images and the air quality data corresponding to the image shooting time, and construct an air quality time series image dataset; S3, using the sliding window method to construct sequence samples from the air quality time series image dataset, using the "T:1" mode to set the length of the sequence samples and the prediction target, and dividing the generated sequence samples into a training set and a test set; S4. Train the hybrid deep learning network based on the training set and verify it based on the test set to obtain the multi-index regression model of air quality; S5. Acquire actual air quality time series image data, input the actual air quality time series image data into the air quality multi-index regression model, and obtain the predicted PM2.5 concentration, PM10 concentration and AQI three air quality indicators.
2. The method for detecting ambient air quality based on surveillance camera images according to claim 1, It is characterized in that The specific steps of S3 are: Take the images of continuous T hours in the air quality time series image dataset as a sequence sample, and the air quality index value of the last hour in the sequence sample as the prediction target. For the air quality time series image dataset containing n images, take the first to Tth continuous images [I 1 ,I 2 ,...I T-1 ,I T ] as the first sequence sample, with the Tth image I T The corresponding air quality index value is taken as the first target label, and the second to T+1th continuous images [I 2 ,I 3 ,...I T ,I T+1 ] as the second sequence sample, with the T+1th image I T+1 The corresponding air quality index value is used as the second target label, and so on, until the last image I in the air quality time series image dataset n Included in the last sequence sample, all sequence samples are integrated, and the sequence samples are divided into a training set and a test set.
3. The method for detecting ambient air quality based on surveillance camera images according to claim 2, It is characterized in that The dimension of the sequence samples is (T, H, W, C), where T is the length of the image sequence, which is also the added time dimension, and (H, W, C) is the dimension of a single image. The image size is H×W, and H, W, and C are the height, width, and number of color channels of the input image, respectively.
4. The method for detecting ambient air quality based on surveillance camera images according to claim 1, It is characterized in that The specific steps of S1 are: Set the input of the hybrid deep learning network, then remove the fully connected layer and output layer of the complete CNN model, retain the feature extractor part of the CNN model as the image spatial feature extractor, use the spatial features of the image sequence output by the image spatial feature extractor as the input of the LSTM network layer in the LSTM long short-term memory network, and use the LSTM network layer as the image temporal feature extractor. Connect the image spatial feature extractor and the image temporal feature extractor, then add two fully connected layers after the image temporal feature extractor, and use the output of the image temporal feature extractor as the input of the fully connected layer to obtain a hybrid deep learning network.
5. The method for detecting ambient air quality based on surveillance camera images according to claim 4, It is characterized in that Of the two fully connected layers after the image temporal feature extractor, the first fully connected layer directly connected to the image temporal feature extractor is used to decode the output of the image temporal feature extractor.
6. The method for detecting ambient air quality based on surveillance camera images according to claim 5, It is characterized in that The second fully connected layer serves as the output layer of the deep learning network.
7. The method for detecting ambient air quality based on surveillance camera images according to claim 1, It is characterized in that The specific steps of S2 are: Acquire long-term continuous surveillance camera video images and the air quality data at the time when the images were taken, compress and normalize the surveillance camera video images, normalize the air quality data at the same time, and construct an air quality time series image dataset based on the normalized images and air quality data.
8. The method for detecting ambient air quality based on surveillance camera images according to claim 7, It is characterized in that The specific steps of compressing and normalizing the surveillance camera video image are as follows: The surveillance camera video image is compressed to the input size required by the image space feature extractor, and the pixel value of each image is divided by the upper limit of the pixel value to scale the pixel value to between 0 and 1.
9. The method for detecting ambient air quality based on surveillance camera images according to claim 8, It is characterized in that The normalization of the air quality data specifically includes: mapping the air quality data to between 0 and 1 based on a minimum-maximum normalization method.
10. The method for detecting ambient air quality based on surveillance camera images according to claim 1, It is characterized in that The hybrid deep learning network training based on the training set is specifically as follows: The training set is input into the hybrid deep learning network, and the mean square error loss function is calculated based on the output of the hybrid deep learning network. Based on the loss function and the Adam optimizer, the gradient descent optimization algorithm is used to optimize the parameters of the hybrid deep learning network, and GPU acceleration is used to obtain the air quality multi-indicator regression model.