A method for predicting air pollutant concentrations based on street view videos
By segmenting street view videos into frames, extracting frequency histogram features from the HSV channel, and selecting high-frequency hues as input, a deep learning model is established. This solves the problem of insufficient utilization of street view image information in existing technologies and achieves efficient pollutant concentration prediction.
Patent Information
- Application Number
- CN202311655901.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-12-05
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2043-12-05
AI Technical Summary
Existing deep learning-based methods for predicting air pollutant concentrations cannot effectively utilize street view image information, have high computational costs and require large amounts of data, resulting in poor prediction performance.
By segmenting street view videos into frames, extracting frequency histogram features from the HSV channels, selecting high-frequency hues as input, and establishing recurrent neural networks, long short-term memory networks, or gated recurrent unit models to predict pollutant concentrations.
It reduced computational costs, improved the accuracy and effectiveness of pollutant concentration prediction, and achieved an effective link between pollutant concentration and image information.
Smart Images

Figure CN118115745B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to environmental pollutant prediction technology, and more particularly to a method for predicting pollutant concentrations based on street view video, belonging to the interdisciplinary field of image processing and environmental pollutant prediction. Background Technology
[0002] Pollutant prediction is an important branch of environmental science, closely related to people's daily lives. Air pollution analysis and prediction are complex and dynamic; accurate prediction requires processing large amounts of related environmental data and information. The rise of artificial intelligence has provided technical means to process large amounts of information, and the development of deep learning methods has enabled pollutant concentration prediction to achieve better results. Currently, deep learning-based methods for predicting air pollutant concentrations can be divided into two categories: prediction based on environmental indicators and prediction based on street view images.
[0003] In the method for predicting air pollutant concentration based on environmental indicators, a series of environmental values and pollutant concentrations are monitored to establish the correlation between indicators such as latitude and longitude, wind direction, wind speed, temperature, and altitude and pollutant concentrations. The features of environmental indicators are extracted through deep learning to predict pollutant concentrations.
[0004] In predictive pollutant analysis based on street view image data, computer technology is used to process and analyze street view images, enabling operations such as image enhancement, restoration, transformation, and image segmentation. Deep learning methods are employed to analyze features in environmental images, revealing the distribution of vegetation, roads, vehicles, sky, buildings, etc., and establishing a correlation between these features and the concentration of air pollutants.
[0005] Most existing deep learning-based pollutant prediction algorithms use environmental indicators or input street view images themselves into the deep learning algorithm for analysis. However, using environmental indicators cannot establish a one-to-one correspondence between street view information closely related to air quality, such as greenery, roads, vehicles, sky, and buildings, and pollutant values in that location. For example, traditional machine learning methods such as decision trees, support vector machines, and random forests, as well as deep learning methods such as recurrent neural networks, long short-term memory networks, and gated recurrent units, all perform well in predicting pollutant concentrations, but they can only use environmental indicators for prediction without establishing a connection with image information. Currently, most common street view image pollutant concentration prediction methods first segment different objects in each image to calculate the number of pixels occupied by each object, using pixel indicators as input variables for pollutant prediction. However, this method requires manual image segmentation as input, which is complex. Therefore, methods that input street view images into the model for prediction have high requirements for training data, require large amounts of data, and have high computational costs. Summary of the Invention
[0006] To overcome the shortcomings of the prior art, this invention provides a method for predicting air pollutant concentration based on street view video. By converting street view video into images frame by frame, extracting the frequency histogram features of the images using a feature extraction method, and then predicting pollutant concentration based on a deep learning model, the computational cost can be greatly reduced and the effect of pollutant concentration prediction can be improved.
[0007] The technical solution provided by this invention is:
[0008] A method for predicting pollutant concentration based on street view video selects frequently occurring hues from the image features, constructs a deep learning-based neural network model for pollutant concentration prediction, and uses the frequency values of the hues as input to perform deep learning-based pollutant concentration prediction; the method includes the following steps:
[0009] Step 1: Data Collection
[0010] By deploying data collection vehicles, street view video data is collected, along with corresponding pollutant data from the same time period and location.
[0011] Step 2: Video data frame segmentation
[0012] For a video clip, it needs to be segmented into a sequence of images. To match the segmented images obtained from the original video with their corresponding pollutant values, starting from the time of video recording and pollutant detection, one segmented image needs to be obtained at each monitoring interval, so that it corresponds to the time frequency of the monitored pollutant values. This results in a series of images and their corresponding pollutant values.
[0013] Step 3: Extract the image frequency histogram features to obtain the frequency distribution of each image in the three HSV channels;
[0014] First, the pixel values of the three channels of the HSV color space are extracted from each image, resulting in three pixel value matrices, each the same size as the original image. In the Hue channel, all pixels contain 180 hues; the frequency distribution of each hue needs to be statistically analyzed to determine which hues are key hues. Frequency histograms are plotted to analyze the frequency distribution of each image across the three HSV channels.
[0015] Step 4: Perform feature selection on each image to obtain the frequency values of the hues, which will be used as input variables for the prediction model;
[0016] Because an image contains a large number of pixels and a wealth of information, we select key pixel information for analysis. The color distribution of an image is determined by the hue channel, and the frequency of each color represents the area occupied by objects of that color in the image. Since pollutant concentration is determined by various objects in a street scene, we can select the hues that appear most frequently in the image as features to input into the model.
[0017] Feature selection methods include the following process:
[0018] 41) For the n time-series images obtained by video frame division, obtain the hue frequency distribution of channel H of each image; let Hk represent the frequency distribution of the hue channel H of the k-th image, including the number of times each hue of channel H appears in the image;
[0019] 42) Calculate the maximum frequency of each hue in each image based on the hue frequency distribution of each image;
[0020] 43) Represent the hues corresponding to the highest frequency of hues in n images as a set, denoted as S;
[0021] 44) Count the number of times each hue value appears in S, denoted as {Hi—H.inumber}, and sort the hues by the number of times. Here, Hi represents one of the hues i in all n images, and H.inumber represents the number of times the hue appears.
[0022] 45) Set adjacent color tones; the color selection method is expressed as:
[0023] Hi←max{H.inumber}①
[0024] Hj←max{H.jnumber\H.inumber}and |ij|<10 ②
[0025] i←j③
[0026] Where Hi represents the first selected hue, Hj represents the second and subsequent selected hues, max{H.inumber} represents the maximum number of occurrences of the maximum hue, max{H.jnumber\H.inumber} represents the maximum number of occurrences of the hue after removing the selected hues, |ij| represents the position difference between two adjacent hues, and ← represents the assignment operation.
[0027] Step 5: Establish three deep learning neural network models respectively, using the frequency value of hue as the input of the model, and perform pollutant concentration prediction based on deep learning to obtain the pollutant concentration.
[0028] After feature selection, the frequency values of six hues are obtained for each image, which are the independent variables, and the pollutant values in the corresponding images are the dependent variables. Time series models of Recurrent Neural Network (RNN), Long Short-Term Memory (LSTM), and Gate Recurrent Unit (GRU) are established respectively to predict pollutant concentrations from a time perspective.
[0029] Compared with the prior art, the beneficial effects of the present invention are:
[0030] This invention discloses a method for predicting pollutants using street view video. It involves acquiring street view video and monitoring air quality data in real time, segmenting the video into frames corresponding to air quality data. The HSV three channels of each image are extracted, and the highest frequency hue in the Hue channel of each image is counted. The frequency of occurrence of the highest frequency hue is then counted in the time-series images (street view video frames). High-frequency hues with frequent occurrences are selected and used as input features, i.e., the frequency values of these hues are used as independent variables to predict pollutant concentrations. This method predicts pollutant values by using the hue of street view images, achieving high-fitness prediction results and high prediction accuracy. This invention can directly extract image information without using environmental indicators to predict pollutant concentrations. It reduces model computational costs while also establishing a correlation between pollutant concentration and images. Attached Figure Description
[0031] Figure 1 The flowchart of the method for obtaining image tone frequency values by feature selection provided by the present invention is shown.
[0032] Figure 2 A comparison chart of PM2.5 prediction results for an air pollutant prediction method based on street view video provided by this invention. Detailed Implementation
[0033] The present invention will be further described below with reference to the accompanying drawings and embodiments, but the scope of the invention is not limited in any way.
[0034] The pollutant concentration prediction method based on street view video provided by this invention includes five steps: data collection, video data framing, extraction of image frequency histogram features, feature selection, and pollutant concentration prediction based on deep learning. The specific operations are as follows.
[0035] Step 1: Data Collection
[0036] The dataset was collected using a mobile data collection vehicle in street view. The vehicle was equipped with: 1) a location recording device, which records real-time latitude and longitude information along the collected street view route; 2) a real-time video recording device, which records street view video in real time; and 3) a pollutant concentration monitor, which simultaneously collects real-time pollutant concentration data at the corresponding location.
[0037] Step 2: Video data frame segmentation
[0038] For a captured video segment, it needs to be divided into frames to form a sequence of images. The original video has a frame rate of 60 frames per second, and pollutant values are monitored and recorded every second. One frame is selected from the 60 frames per second of video as a representative image, and its correspondence with the monitored pollutant value at that moment is obtained, resulting in a series of images and their corresponding pollutant values.
[0039] Step 3: Extract image frequency histogram features; Figure 1 The diagram shows the flow of the image frequency extraction method.
[0040] Each pixel of an image is represented by pixel values from three channels: red (R), green (G), and blue (B). This can be converted to HSV representation using a formula, which means that each image is represented by pixel values from three channels: hue (H), saturation (S), and value (V). The conversion formula is shown in equation (1).
[0041] V←max(R,G,B)
[0042]
[0043]
[0044] H < 0, H ← H + 360
[0045] The analysis focuses on the pixel values of the Hue channel. Each image's Hue channel contains 180 hues, and the frequency distribution of each hue needs to be statistically analyzed. This can be achieved by calculating the histogram distribution of each hue, thus identifying the key hues. Figure 1 The frequency histogram shows the frequency distribution of each image across the three HSV channels. The blue line with multiple distinct peaks represents the hue frequency distribution, used for key hue feature selection in subsequent steps.
[0046] In a frequency histogram, the complete color space is divided into several smaller color spaces, i.e., the bins (group intervals) of the histogram. The number of bins and the color quantization method determine the final algorithm performance and efficiency. Generally, the more bins, the stronger the histogram's ability to distinguish colors. However, a larger number of bins not only increases the computational burden but also hinders indexing in large image databases. Moreover, for some applications, using a very fine color space partitioning method may not necessarily improve retrieval results, especially for applications that cannot tolerate the omission of relevant images. An effective way to reduce the number of histogram bins is to select only those bins with larger values (i.e., more pixels) to construct image features, because these bins representing the main colors can express the colors of most pixels in the image. In a specific implementation of this invention, feature selection is performed on 180 hues of the Hue channel.
[0047] Step 4: Perform feature selection on the image, selecting high-frequency tones from the image hue;
[0048] Since an image contains 180 hues in the Hue channel, the computational cost is substantial. Therefore, feature selection is performed, extracting only the most important information (i.e., the hues with a high proportion of color in the image) for analysis. The color distribution of an image is determined by the hues in the Hue channel; the frequency of each hue represents the number of pixels occupied by objects of that color in the image. The pollutant prediction result is determined by the proportion of various objects in the image. This can be addressed by selecting frequently occurring elements as model input, rather than using the entire image. Figure 1 The independent variable, frequency, is shown, and a histogram of the hue channels of the image in the example is plotted, with the six selected high-frequency hues outlined. This feature selection process is detailed below.
[0049] For n images in a time series (i.e., n images obtained from video frame division), let Hk represent the frequency distribution of the hue channel in the k-th image. This is a set containing the number of times each hue appears in an image. And let Hki represent the number of times hue i appears in the k-th image. The relationship between the two is expressed as Hk = {Hk1, Hk2, ..., Hki, ..., Hk180}, where i ∈ [0, 180] represents 180 hues. This records the distribution of 180 hues in an image, with the hue distribution order roughly transitioning from red to purple. The frequency distribution of the hue in each image can be statistically obtained using this method.
[0050] Based on the hue frequency distribution, the maximum hue frequency in each image k is calculated and denoted as Hkmax, which is expressed as Hkmax=max{Hk1,Hk2,…,Hki,…,Hk180}, which is to take the maximum value among all hue frequencies in the k-th image.
[0051] In all n images, each image will have a corresponding maximum hue value Hkmax. The hues corresponding to the maximum frequency of the n images are represented as a set, i.e., S = {H1max, H2max, ..., Himax, ..., Hnmax}. Then, the frequency of each hue value in S is counted, and the hues are sorted from most frequent to least frequent, denoted as {Hi - H.inumber}, where Hi represents one of the hues i in all n images, and H.inumber represents the frequency of that hue, as shown in the first and second rows of Table 1.
[0052] Table 1. Statistical results of maximum frequency hue for each image.
[0053]
[0054] Table 1 shows the statistical count of the highest frequency hue in all n images for the first 10 images. Hue 108 is the highest frequency hue in all images 1762 times, and the frequency of other hues decreases sequentially.
[0055] The more times a hue appears, the more information it conveys in the image set, and therefore the more important its frequency value becomes. However, the choice of hue is not strictly based on the number of times it appears, because some frequently occurring hues have very similar colors, and their hue meanings are particularly similar. Therefore, for these adjacent hues, the hue corresponding to the highest frequency value can be selected. Here, multiple adjacent hues (10 in this specific implementation) are set as adjacent hues. The selection of hues is shown in formula (2).
[0056] Hi←max{H.inumber},①
[0057] Hj←max{H.jnumber\H.inumber}and|ij|<10,② (2)
[0058] i←j,③
[0059] Where Hi represents the frequency value of the first selected hue, Hj represents the frequency value of the second and subsequent selected hues, max{H.inumber} represents the maximum number of times a hue appears, max{H.jnumber\H.inumber} represents the maximum number of times a hue appears after removing the selected hues, |ij| represents the positional difference between two adjacent hues, and ← represents assignment.
[0060] In step ①, the first hue is directly selected as the hue i that appears most frequently. Then, the selection of each hue (step ②) is to select the hue j that appears most frequently after removing the previously selected hues. Then, in step ③, the selected hues are assigned values to complete a complete hue selection. Repeat steps ② and ③ until the selection is completed.
[0061] In practice, the hue difference between hue i and hue j is set to be less than 10. The purpose is to avoid hue information redundancy caused by similar hues while selecting the maximum value. This hue selection method is continued until the set number of hues to be selected is reached or the number of times the selected hue appears is less than the set threshold number. Then the hue selection is stopped and the selected hues are used as independent variables to input into the model.
[0062] The selected hue results are shown in bold in Table 1. This result is obtained by separating the hue channel from the histogram of each image and using the frequency values of several high-frequency hues as input variables for the model. The frequency value refers to the number of pixels corresponding to that hue in an image. The number of times the maximum hue value appears in the frequency histogram of all images was counted, and several high-frequency hues (108, 30, 0, 21, 150, 90) were selected from 180 hues. The selection criteria were: the hue appeared frequently; if similar hue values existed (adjacent hues), they were only selected once. This completed the data extraction, obtaining the frequency values of six hues for each image in a time series.
[0063] Step 5: Predict pollutant concentrations based on deep learning and verify the effectiveness of the extracted frequencies;
[0064] The frequency values of six selected hues in each image were used as independent variables, and the corresponding pollutant values in the image were used as dependent variables. Recurrent Neural Network (RNN), Long Short-Term Memory (LSTM), and Gate Recurrent Unit (GRU) time-series models were established to predict pollutant concentrations over time. The first 70% of the time-series images (i.e., images obtained from video frame segmentation) were used as the training set for model training, and the last 30% of the time-series images (i.e., images obtained from video frame segmentation) were used as the test set for model testing. The process was as follows: the training and test sets were split in a 7:3 ratio. The first 70% of the data was input into the model, and the model was trained by setting a loss function until the loss function result showed no significant change, indicating that the data on the training set achieved the best performance. Training was then stopped, and the trained model was obtained. The last 30% of the data was then input into the trained model to predict the data in the test set, and the prediction results were obtained. Evaluation metrics were set to verify the prediction effect of the results.
[0065] Example:
[0066] A) Dataset: The dataset uses street view video data of a certain city. The dataset was obtained by a data collection vehicle. The collection route was the city street view route. The content is street view video and corresponding pollutant detection data for the same period. The duration is about 90 minutes, the video frame rate is 60 frames / second, and the pollutant data is detected once per second. The selected pollutant is PM2.5.
[0067] The street view video data was divided into frames every second to obtain image data. The image data contains a total of 5011 street view images, each with a pixel size of 1920*1080.
[0068] Before building the model, the data needs to be preprocessed, including deleting abnormal data and deleting missing data that exists during the collection process.
[0069] B) To verify the effectiveness of the frequencies extracted in step 4, three prediction methods were selected for comparison and verification: RNN, LSTM, and GRU were used to predict pollutants.
[0070] RNNs can process sequential data and are widely used in natural language processing, speech recognition, image processing, and other fields. However, RNN models suffer from the vanishing gradient problem when the sequence is too long, causing the parameters to only capture local relationships and fail to learn long-term associations, resulting in a weak ability to remember words at the beginning of the sequence.
[0071] LSTM (Long Short-Term Memory) networks are a special type of RNN (Recurrent Learning Network) with the ability to remember long short-term information. LSTM was proposed to address the problem of long-term dependencies. It can solve the gradient vanishing and gradient exploding problems existing in traditional RNNs, thus better handling long sequence data. However, LSTM has a large number of parameters. The GRU (Gross Root Response) model, to some extent, ensures that fewer parameters are used to alleviate the gradient vanishing problem.
[0072] GRU is a highly effective variant of LSTM networks. It has a simpler structure than LSTM and performs well, making it a very popular network choice. As a variant of LSTM, GRU can also address the long dependency problem in RNN networks. GRU has fewer parameters, resulting in faster training and reduced risk of overfitting.
[0073] C) Evaluation Indicators: Five evaluation indicators are used to evaluate and analyze the prediction results, including: mean-square error (MSE), root mean square error (RMSE), mean absolute error (MAE), mean absolute percentage error (MAPE), and goodness of fit R². 2 Its formula is shown in equation (3).
[0074]
[0075]
[0076]
[0077]
[0078]
[0079] D) Settings Details:
[0080] In this experiment, the first 70% of the data was used as the training set for model training, and the last 30% of the data was used as the test set for model testing. The training set consisted of 3502 images, and the test set consisted of 1504 images.
[0081] The parameters of the neural network are optimized using the mean squared error minimization method. The model has two hidden layers with dimensions of 128 and 32, respectively. The window_size is set to 5, the batch_size is set to 4, and the epochs are set to 30.
[0082] E) Experimental Results
[0083] Table 2 compares the prediction and evaluation results using three models.
[0084]
[0085] The method of this invention uses three network models, RNN, LSTM, and GRU, to predict PM2.5 pollutant concentration. The prediction results of the test set are shown in Table 2. As can be seen from the results in the table, the method proposed in this patent achieves good results in predicting pollutant concentration by selecting color tone. Moreover, the evaluation results of the three methods are similar, indicating that the method of extracting image color tone for predicting pollutant concentration has practical application value and can greatly reduce the model calculation cost and improve the prediction effect.
[0086] It should be noted that the purpose of disclosing the embodiments is to help further understand the present invention. However, those skilled in the art will understand that various substitutions and modifications are possible without departing from the scope of the present invention and the appended claims. Therefore, the present invention should not be limited to the content disclosed in the embodiments, and the scope of protection of the present invention is defined by the scope of the claims.
Claims
1. A method for predicting concentration of pollutants based on street view videos, characterized in that, By performing image feature selection on street view videos, high-frequency hue features are selected. A deep learning-based neural network model for predicting pollutant concentration is constructed, using hue frequency values as input to the prediction model. This enables deep learning-based prediction of pollutant concentration in street view videos, including the following steps: Step 1: Acquire street view video data and corresponding pollutant data for the same time period and location; Step 2: Divide the video data into frames and match the resulting frame images with their corresponding pollutant values; Step 3: Convert each image into pixel values represented by three channels: hue (H), saturation (S), and brightness (V); extract the image frequency histogram features to obtain the frequency distribution of each image in the three HSV channels; Step 4: Perform feature selection on each image obtained from video frame segmentation to obtain the frequency values of the hue, which will be used as input variables for the prediction model. The feature selection method includes the following process: 41) For the n time-series images obtained by video frame division, obtain the hue frequency distribution of channel H of each image; let Hk represent the frequency distribution of the hue channel H of the k-th image, including the number of times each hue of channel H appears in the image; 42) Calculate the maximum frequency of each hue in each image based on the hue frequency distribution of each image; 43) Represent the hues corresponding to the highest frequency of hues in n images as a set, denoted as S; 44) Count the number of times each hue value appears in S, denoted as {Hi—H.inumber}, and sort the hues by the number of times. Here, Hi represents one of the hues i in all n images, and H.inumber represents the number of times the hue appears. 45) Set adjacent color tones; the color selection method is expressed as: Hi←max{H.inumber}① Hj ← max{H.jnumber\H.inumber} and |ij| < 10 ② i←j③ Where Hi represents the first selected hue, Hj represents the second and subsequent selected hues, max{H.inumber} represents the maximum number of occurrences of the maximum hue, max{H.jnumber\H.inumber} represents the maximum number of occurrences of the hue after removing the selected hues, |ij| represents the position difference between two adjacent hues, and ← represents the assignment operation. Repeat steps ② and ③ until the hue selection is complete, obtaining the frequency values of the highest-frequency hues in each image of a time series; Step 5: Establish three deep learning neural network models, including recurrent neural network (RNN), long short-term memory network (LSTM), and gated recurrent unit (GRU) time series model; use the frequency value of hue as the input of the model, and perform pollutant concentration prediction based on deep learning from the time dimension to obtain the pollutant concentration.
2. The method of claim 1, wherein the street view video-based pollutant concentration prediction method is characterized by, By plotting the frequency histogram of the image hue, the frequency distribution of each image in the three HSV channels is analyzed. 3.The method of claim 1, wherein the method further comprises: determining a pollution concentration of the area based on the street view video. Step 1 involves collecting data by pushing a data collection vehicle; the vehicle's equipment includes: location recording equipment, real-time video recording equipment, and pollutant concentration monitoring instrument. 4.The method of claim 1, wherein the method further comprises: determining a pollution concentration of the area based on the street view video. Step 2 specifically involves: acquiring a video frame rate of 60 frames per second and dividing it into time-series images; monitoring and recording pollutant values every second; selecting one frame from the 60 frames per second of video as a representative image and matching it with the monitored pollutant value at that moment, thus obtaining the image series and the pollutant values corresponding to the images.
5. The pollutant concentration prediction method based on street view video as described in claim 1, characterized in that, In step 4, a total of 180 hues are recorded in the image, and feature selection is performed on the 180 hues in the H channel.
6. The pollutant concentration prediction method based on street view video as described in claim 5, characterized in that, For n time-series images of a video frame, the frequency distribution of the hue channel H of the k-th image is obtained, denoted as Hk, which includes the number of times each hue of channel H appears in the image, Hki; the maximum frequency of the hue in the k-th image is denoted as Hkmax, and is expressed as Hkmax=max{Hk1,Hk2,…,Hki,…,Hk180}; the set S={H1max,H2max,…,Himax,…,Hnmax}.
7. The pollutant concentration prediction method based on street view video as described in claim 1, characterized in that, After feature selection in step 4, the frequency values of 6 hues are obtained for each image, which are used as independent variables for input model.
