A vision-based method and system for nighttime PM2.5 monitoring.

Abstract

This invention discloses a vision-based method and system for nighttime PM2.5 monitoring, applied in the field of nighttime atmospheric environment monitoring technology. The method includes the following steps: S1 Acquires original images using a camera device and performs effective frame detection to obtain valid images; S2 Extracts glow feature maps from the target scene using image processing algorithms, while simultaneously acquiring PM2.5 environmental readings in the real atmospheric environment of the target scene using self-deployed sensors or environmental monitoring stations; S3 Adjusts the glow feature map to a fixed size, uses the glow feature map as input to a neural network, and uses the PM2.5 environmental readings in the real atmospheric environment as labels to perform end-to-end neural network learning; S4 Trains the neural network based on its input, and infers the PM2.5 value using the trained neural network. This invention extracts glow feature maps based on two attributes in the image: brightness and intensity attenuation, and fits the PM2.5 concentration in the atmospheric environment using a neural network.

Description

Technical Field

[0001] This invention relates to the field of nighttime atmospheric environment monitoring technology, and more specifically to a vision-based nighttime PM2.5 monitoring method and system. Background Technology

[0002] Currently, the most common monitoring method on the market is based on sensor networks, which achieves wide-area atmospheric environmental monitoring by deploying sensors at multiple nodes within the detection area. For example, the design scheme with patent application number CN202021353526.5 utilizes a central wireless communication module to wirelessly network with the wireless communication modules of each node, facilitating the uploading of data collected by each sensor node device to the central control device, thereby achieving data acquisition. This type of solution requires bearing the costs of equipment purchase, power supply, communication, and maintenance at multiple node locations, which to some extent limits its practical application scope.

[0003] On the other hand, methods for monitoring nighttime PM2.5 concentration based on remote sensing data of nighttime light pollution have been proposed. These methods monitor the nighttime atmospheric environment by analyzing the relationship between nighttime light radiation received by remote sensing sensors and PM2.5 concentration. For example, the design scheme in patent application number CN202110949698.1 extracts data on DNB radiance, cloud cover, lunar phase angle, satellite zenith angle, and satellite transit time received by the sensor, and analyzes the PM2.5 concentration at the time of satellite transit based on radiative transfer theory. However, due to the limitations of remote sensing data, such schemes have poor implementation capabilities and are not yet satisfactory in terms of spatial granularity and monitoring accuracy.

[0004] With the maturity of computer vision technology, the most common vision-based methods in academia currently mainly process images acquired during the day, that is, to assess ambient air quality through prior features or other elements of daytime images. However, there is still a lack of methods for PM2.5 detection in nighttime scenes using visual approaches.

[0005] Therefore, proposing a vision-based method and system for nighttime PM2.5 monitoring to address the difficulties of existing technologies is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0006] In view of this, the present invention provides a vision-based method and system for nighttime PM2.5 monitoring to solve the problems existing in the prior art.

[0007] To achieve the above objectives, the present invention provides the following technical solution:

[0008] A vision-based method for nighttime PM2.5 monitoring includes the following steps:

[0009] S1. Acquire the original image using a camera device. raw Valid frame detection is performed to obtain valid images;

[0010] S2. Based on the valid image obtained in S1, extract the glow feature map from the target scene using an image processing algorithm. glow Meanwhile, it obtains PM2.5 environmental readings in the real atmospheric environment of the target scene through self-deployed sensors or environmental monitoring stations;

[0011] S3. Extract the glow feature map from the target scene in S2. glow The size was adjusted to a fixed value, and the glow feature map was used as the input to the neural network, while the PM2.5 environmental readings in the real atmospheric environment were used as labels to perform end-to-end neural network learning.

[0012] S4. Train the neural network based on the input from S3, and infer the PM2.5 value through the trained neural network.

[0013] Optionally, the specific method for valid frame detection in S1 is as follows: In order to verify whether the original image acquired by the camera device is valid, the original image needs to be compared with the previous valid frame. Taking into account the differences in the image caused by the camera device's swing and the changes in the target scene, the probability of a valid frame is calculated. Based on the calculated probability of a valid frame, it is determined whether to discard the original image or retain the original image as a valid image and proceed with the subsequent steps.

[0014] Optionally, the steps for obtaining a valid image in S1 are as follows:

[0015] S11. Based on the original image Image raw and effective value image valid Calculate the image difference diagram:

[0016]

[0017] Where x is the index of the image pixel, Ω(x) is the patch centered at x, and x' is the index of the pixel in the patch centered at x;

[0018] S12. Based on the image difference diagram diff Calculate the probability of valid frames:

[0019]

[0020]

[0021] Where N is the number of pixels in the image, ε is the set threshold, the f function is used to calculate the probability of a valid frame, and x is the index of the image pixel;

[0022] S13. Compare the obtained effective frame probability with the preset threshold. If it is lower than the threshold, proceed to the next step; otherwise, discard the original image and re-execute step S11.

[0023] Optionally, the specific method for extracting the glow feature map in S2 is as follows:

[0024] The brightness information and brightness decay rate of each pixel position are obtained from the effective image, resulting in two feature maps: the brightness feature map and the intensity decay feature map. The value of each pixel in the feature map represents the probability of whether the corresponding pixel belongs to the light source region and the intensity decay rate of the patch centered on that pixel. Based on the brightness feature map and the intensity decay feature map, and combined with the original image after the dilation operation, convolution operation is performed on pixel position to obtain the glow feature map.

[0025] Optionally, the specific steps for extracting the image glow feature map in S2 are as follows:

[0026] S21. Calculate the brightness feature map:

[0027]

[0028] Where γ is the gamma correction coefficient. It is the input image. It is the input image The specified channels, {R,G,B} are the different channels of the image in RGB color mode, and x is the index of the image pixel;

[0029] S22. Calculate the intensity attenuation characteristic map:

[0030]

[0031] Attenuation(x)=GuideFilter(t(x),image raw )

[0032] Among them, A c (y) represents atmospheric brightness, Attenuation(x) is the intensity attenuation characteristic map, and GuideFilter is the guided filter. It is the input image. It is the input image The specified channels, {R,G,B} are the different channels of the image in RGB color mode, and x is the index of the image pixel;

[0033] S23. Divide the original image into three channels, namely RGB, i.e., three feature maps, and perform expansion operations on each to increase the exposure effect;

[0034] Glow c=dilate(x)*Attenuation(x)*Bright(x)

[0035]

[0036] Among them, Glow c These are different channels of the glow feature map, dilate(x) is the image dilation function, attenuation(x) is the intensity attenuation feature map, bright(x) is the brightness feature map, and x' is the pixel index in the patch. It is the maximum pixel value of a specific channel in the tile centered at x;

[0037] S24. Multiply the specific channels after dilation in S23 pixel by pixel with the brightness feature map and the intensity attenuation feature map;

[0038] S25. Merge the obtained channels into a new RGB image, i.e., a glow feature map.

[0039] Optionally, the specific steps for end-to-end neural network learning in S3 are as follows:

[0040] S31. Downsample the original image to make its resolution 108×192;

[0041] S32. An end-to-end neural network has the following setup: The neural network contains 3 consecutive convolution-pooling-activation blocks, the convolution kernel size is 3×3, the pooling kernel size is 2×2, and the activation function is the Leaky-ReLU function.

[0042] Optionally, in S4, the PM2.5 value is inferred using the trained neural network as follows:

[0043] S41. Acquire the original image using a camera device. raw Based on S2 and S3, glow feature maps are extracted and images are downsampled.

[0044] S42. Use the acquired image as input to the trained neural network, and the output of the network is the inferred value of PM2.5 in the corresponding nighttime environment.

[0045] A vision-based nighttime PM2.5 monitoring system, employing any of the aforementioned vision-based nighttime PM2.5 monitoring methods, includes a module for acquiring valid images, a glow feature map extraction module, a neural network learning module, and a PM2.5 monitoring module connected in sequence.

[0046] Image Acquisition Module: Acquires raw images from the camera device. raw Valid frame detection is performed to obtain valid images;

[0047] Glow Feature Map Extraction Module: Based on the valid images obtained from the valid image acquisition module, this module uses image processing algorithms to extract glow feature maps from the target scene. glow Meanwhile, it obtains PM2.5 environmental readings in the real atmospheric environment of the target scene through self-deployed sensors or environmental monitoring stations;

[0048] Neural network learning module: extracts glow feature maps from the target scene from the glow feature map extraction module. glow The size was adjusted to a fixed value, and the glow feature map was used as the input to the neural network, while the PM2.5 environmental readings in the real atmospheric environment were used as labels to perform end-to-end neural network learning.

[0049] PM2.5 monitoring module: Trains the neural network based on the input of the neural network learning module, and infers the PM2.5 value through the trained neural network.

[0050] As can be seen from the above technical solution, compared with the prior art, the present invention discloses a vision-based method and system for nighttime PM2.5 monitoring, the beneficial effects of which are:

[0051] 1) This invention can use widely deployed surveillance cameras to acquire images and estimate the atmospheric PM2.5 concentration in a target scene;

[0052] 2) This invention is universal; images can be acquired using ordinary surveillance cameras or other cameras (such as mobile phone cameras), and does not depend on a specific model of device.

[0053] 3) This invention proposes a vision-based method for monitoring PM2.5 at night. Based on the two attributes of brightness and intensity attenuation in the image, the glow feature map of artificial light source in the original image is extracted, and the concentration of PM2.5 in the atmosphere is fitted by a neural network. Attached Figure Description

[0054] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0055] Figure 1 A flowchart of a vision-based nighttime PM2.5 monitoring method provided by the present invention;

[0056] Figure 2A structural diagram of a vision-based nighttime PM2.5 monitoring system provided by the present invention;

[0057] Figure 3 The glow feature map provided by the present invention includes 3a as the original image, 3b as the brightness feature map, 3c as the intensity attenuation feature map, and 3d as the glow feature map.

[0058] Figure 4 This is a schematic diagram of the neural network structure provided by the present invention. Detailed Implementation

[0059] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0060] See Figure 1 As shown, this invention discloses a vision-based method for nighttime PM2.5 monitoring, comprising the following steps:

[0061] S1. Acquire the original image using a camera device. raw Valid frame detection is performed to obtain valid images;

[0062] S2. Based on the valid image obtained in S1, extract the glow feature map from the target scene using an image processing algorithm. glow Meanwhile, it obtains PM2.5 environmental readings in the real atmospheric environment of the target scene through self-deployed sensors or environmental monitoring stations;

[0063] S3. Extract the glow feature map from the target scene in S2. glow The size was adjusted to a fixed value, and the glow feature map was used as the input to the neural network, while the PM2.5 environmental readings in the real atmospheric environment were used as labels to perform end-to-end neural network learning.

[0064] S4. Train the neural network based on the input from S3, and infer the PM2.5 value through the trained neural network.

[0065] Furthermore, the specific method for valid frame detection in S1 is as follows: In order to verify whether the original image acquired by the camera device is valid, the original image needs to be compared with the previous valid frame. Taking into account the differences in the image caused by the camera device's swing and the changes in the target scene, the probability of a valid frame is calculated. Based on the calculated probability of a valid frame, it is determined whether to discard the original image or retain the original image as a valid image and proceed with subsequent steps.

[0066] Specifically, if the probability of a valid frame is less than the threshold, the original image is an invalid frame; otherwise, it is a valid frame. Valid frames are retained and subsequent operations are performed.

[0067] Furthermore, the specific steps for obtaining a valid image in S1 are as follows:

[0068] S11. Based on the original image Image raw and effective value image valid Calculate the image difference diagram:

[0069]

[0070] Where x is the index of the image pixel, Ω(x) is the patch centered at x, and x' is the index of the pixel in the patch centered at x;

[0071] S12. Based on the image difference diagram diff Calculate the probability of valid frames:

[0072]

[0073]

[0074] Where N is the number of pixels in the image, ε is the set threshold, the f function is used to calculate the probability of a valid frame, and x is the index of the image pixel;

[0075] S13. Compare the obtained effective frame probability with the preset threshold. If it is lower than the threshold, proceed to the next step; otherwise, discard the original image and re-execute step S11.

[0076] Furthermore, the specific method for extracting the glow feature map in S2 is as follows:

[0077] The brightness information and brightness decay rate of each pixel position are obtained from the effective image, resulting in two feature maps: the brightness feature map and the intensity decay feature map. The value of each pixel in the feature map represents the probability of whether the corresponding pixel belongs to the light source region and the intensity decay rate of the patch centered on that pixel. Based on the brightness feature map and the intensity decay feature map, and combined with the original image after the dilation operation, convolution operation is performed on pixel position to obtain the glow feature map.

[0078] Further, see Figure 3 As shown, 3a is the original image, 3b is the brightness feature map, 3c is the intensity attenuation feature map, and 3d is the glow feature map. The specific steps for extracting the glow feature map in S2 are as follows:

[0079] S21. Calculate the brightness feature map:

[0080]

[0081] Where γ is the gamma correction coefficient. It is the input image. It is the input image The specified channels, {R,G,B} are the different channels of the image in RGB color mode, and x is the index of the image pixel;

[0082] Specifically, by combining the two features of pixel location intensity (maximum value of RGB three channels) and color shift (maximum difference of RGB three channels), it can be regarded as obtaining the probability value of whether the corresponding pixel location belongs to the light source on a pixel-by-pixel basis, and setting the maximum threshold to 1.

[0083] S22. Calculate the intensity attenuation characteristic map:

[0084]

[0085] Attenuation(x)=GuideFilter(t(x),image raw )

[0086] Among them, A c (y) represents atmospheric brightness, Attenuation(x) is the intensity attenuation characteristic map, and GuideFilter is the guided filter. It is the input image. It is the input image The specified channels, {R,G,B} are the different channels of the image in RGB color mode, and x is the index of the image pixel;

[0087] Specifically, we select the average value of the brightest 0.1% pixels in the dark channel to represent atmospheric brightness. We obtain a transmittance feature map through the dark channel features as an approximation of the glow intensity attenuation feature. A GuideFilter, as a soft extinction technique, is used to reduce artifacts caused by block operations and refine the transmittance map to obtain the intensity attenuation feature map.

[0088] S23. Divide the original image into three channels, namely RGB, i.e., three feature maps, and perform expansion operations on each to increase the exposure effect;

[0089] Glow c =dilate(x)*Attenuation(x)*Bright(x)

[0090]

[0091] Among them, Glow cThese are different channels of the glow feature map, dilate(x) is the image dilation function, attenuation(x) is the intensity attenuation feature map, bright(x) is the brightness feature map, and x' is the pixel index in the patch. It is the maximum pixel value of a specific channel in the tile centered at x;

[0092] S24. Multiply the specific channels after dilation in S23 pixel by pixel with the brightness feature map and the intensity attenuation feature map;

[0093] S25. Merge the obtained channels into a new RGB image, i.e., a glow feature map.

[0094] Further, see Figure 4 As shown, the specific steps for end-to-end neural network learning in S3 are as follows:

[0095] S31. Downsample the original image to make its resolution 108×192;

[0096] S32. An end-to-end neural network has the following setup: The neural network contains 3 consecutive convolution-pooling-activation blocks, the convolution kernel size is 3×3, the pooling kernel size is 2×2, and the activation function is the Leaky-ReLU function.

[0097] Specifically, neurons are flattened and connected to fully connected layers, using a single neuron as the output / evaluation value; the loss function is mean squared error, the optimizer is the Adam optimizer, and the learning rate decay strategy is a multi-step learning rate decay strategy; when the loss value tends to plateau, the model is maintained.

[0098] Furthermore, in S4, the PM2.5 value is inferred using the trained neural network as follows:

[0099] S41. Acquire the original image using a camera device. raw Based on S2 and S3, glow feature maps are extracted and images are downsampled.

[0100] S42. Use the acquired image as input to the trained neural network, and the output of the network is the inferred value of PM2.5 in the corresponding nighttime environment.

[0101] and Figure 1 Corresponding to the method described above, the present invention also provides a vision-based nighttime PM2.5 monitoring system for monitoring... Figure 1 For a detailed implementation of the method, please refer to the structural diagram. Figure 2 As shown, it includes a module for acquiring valid images, a glow feature map extraction module, a neural network learning module, and a PM2.5 monitoring module connected in sequence.

[0102] Image Acquisition Module: Acquires raw images from the camera device. raw Valid frame detection is performed to obtain valid images;

[0103] Glow Feature Map Extraction Module: Based on the valid images obtained from the valid image acquisition module, this module uses image processing algorithms to extract glow feature maps from the target scene. glow Meanwhile, it obtains PM2.5 environmental readings in the real atmospheric environment of the target scene through self-deployed sensors or environmental monitoring stations;

[0104] Neural network learning module: extracts glow feature maps from the target scene from the glow feature map extraction module. glow The size was adjusted to a fixed value, and the glow feature map was used as the input to the neural network, while the PM2.5 environmental readings in the real atmospheric environment were used as labels to perform end-to-end neural network learning.

[0105] PM2.5 monitoring module: Trains the neural network based on the input of the neural network learning module, and infers the PM2.5 value through the trained neural network.

[0106] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0107] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1.A visual-based night PM2.5 monitoring method, characterized in that, The method comprises the following steps: S1. Obtain an original image through a camera device , perform effective frame detection to obtain an effective image; S2. Extract the luminescence feature map in the target scene from the effective image obtained in S1 through an image processing algorithm At the same time, obtain the PM2.5 environmental reading in the real atmospheric environment of the target scene through the self-deployed sensor or environmental monitoring station. S3. Extract the luminescence feature map in the target scene in S2 Adjust to fixed size, and take the luminescence feature map as the input of the neural network, and take the PM2.5 environmental reading in the real atmospheric environment as the label to perform end-to-end neural network learning; S4. Training according to the input of the neural network in S3, and inferring the value of PM2.5 through the trained neural network; The specific method for extracting the glow feature map in S2 is as follows: The luminance information of each pixel position and the luminance decay rate of each pixel position are obtained from the effective image to obtain two feature maps, i.e. a luminance feature map and an intensity decay feature map. The values of the pixel points in the feature maps represent the probability that the corresponding pixels belong to the light source area and the intensity decay speed of the block centered on the pixel, respectively. On the basis of the luminance feature map and the intensity decay feature map, convolution operation is performed on each pixel position in combination with the original image after the dilation operation to obtain the glow feature map. 2.The visual-based night PM2.5 monitoring method of claim 1, wherein, The specific method for detecting the effective frame in S1 is as follows: In order to verify whether the original image obtained by the camera device is effective, the original image needs to be compared with the previous effective frame. The picture difference caused by the swing of the camera device and the change of the target scene is comprehensively considered to calculate the effective frame probability. According to the calculated effective frame probability, it is determined whether the original image is discarded or reserved as an effective image and subjected to subsequent steps. 3.The visual-based night PM2.5 monitoring method of claim 1, wherein, The step of obtaining the effective image in S1 is specifically as follows: S11. Calculate a picture difference map from the original image and the effective value image S12. Calculate a picture difference map from the original image in It is the index of the image pixels. Therefore The central tile, Therefore The index of the pixels in the central tile; S12. Calculate the effective frame probability from the picture difference map , the effective frame probability: wherein, is the number of pixels in the image, is a set threshold value, the function is used to calculate the probability of valid frames, is the index of the image pixel; S13. Comparing the obtained effective frame probability with a pre-set threshold value, if the effective frame probability is lower than the threshold value, subsequent steps are performed, otherwise the original image is discarded and step S11 is re-executed. 4.The visual-based night PM2.5 monitoring method of claim 1, wherein, The step of extracting the image glow feature map in S2 is specifically as follows: S21. Calculating the luminance feature map: wherein, is a gamma correction coefficient, is an input image, is an input image of a specified channel, is different channels of the image in color mode, is an index of an image pixel; S22. Calculating the intensity decay feature map: ) wherein, ) is the atmospheric luminance, is the intensity decay signature, is the steering filter, is the input image, is the input image of a specified channel, is different channels of the image in color mode, is the index of the image pixel; S23. Dividing the original image into three channels, i.e. three feature maps, and performing dilation operation on each channel to increase the exposure effect; wherein, is a different channel of the glow feature map, is an image dilation processing function, is an intensity decay feature map, is a brightness feature map, is an image pixel index in the tile, is a maximum value of pixel values of a specific channel in the tile centered at . S24. Multiplying each pixel of the specific channel after the dilation operation in S23 with the luminance feature map and the intensity decay feature map, respectively; S25. Merging the obtained channels into a new RGB picture, i.e. the glow feature map. 5.The visual-based night PM2.5 monitoring method of claim 1, wherein, The specific steps of the end-to-end neural network learning in S3 are as follows: S31. Reducing the resolution of the original image to 108x192; S32. The end-to-end neural network comprises the following settings: the neural network comprises three consecutive convolution-pooling-activation blocks, the size of the convolution kernel is 3x3, the size of the pooling kernel is 2x2, and the activation function is Leaky-ReLU function. 6.The visual-based night PM2.5 monitoring method of claim 1, wherein, The specific method for inferring the value of PM2.5 through the trained neural network in S4 is as follows: S41. Obtain the original image through the camera device According to S2 and S3, the extraction of the glow feature map and the down-sampling operation of the image are performed; S42. Taking the obtained image as the input of the trained neural network, and the output of the network is the inferred value of PM2.5 in the night environment. 7.A visual-based night PM2.5 monitoring system, characterized in that The application of the visual-based night PM2.5 monitoring method according to any one of claims 1-6 comprises the sequentially connected effective image acquisition module, the glow feature map extraction module, the neural network learning module, and the PM2.5 monitoring module. An effective image acquisition module: acquiring an original image through a camera device , performing effective frame detection to obtain an effective image; The glow feature extraction module extracts the glow feature map in the target scene according to the effective image obtained by the effective image acquisition module through an image processing algorithm Meanwhile, PM2.5 environment readings in the real atmospheric environment of the target scene are obtained through self-deployed sensors or environmental monitoring stations. The neural network learning module learns the glow feature map in the target scene extracted by the glow feature map extraction module The neural network learning module learns the glow feature map in the target scene extracted by the glow feature map extraction module The neural network learning module learns the glow feature map in the target scene extracted by the glow feature map extraction module The PM2.5 monitoring module: training according to the input of the neural network in the neural network learning module, and inferring the value of PM2.5 through the trained neural network.

Images