A vehicle black smoke detection method based on frequency domain feature enhancement

Abstract

The application discloses a vehicle black smoke detection method based on frequency domain feature enhancement. The method first extracts the basic features of the black smoke detection picture by using an RGB feature extraction network E; then converts the black smoke detection picture to a YCbCr color space and uses a block two-dimensional discrete cosine transform DCT-2D method to preliminarily extract frequency domain information, and then sends the frequency domain information into a frequency domain feature extraction network M, uses a self-attention mechanism to enhance the frequency domain part conducive to distinguishing the black smoke target, and uses a convolutional neural network to further extract high receptive field and high semantic frequency domain features; finally, a feature aggregation network is used to fuse the basic features and the frequency domain features as enhanced features, which are used for subsequent black smoke classification and positioning, and judgment of whether the whole picture contains black smoke. The method can be embedded in a vehicle black smoke detection method based on a single frame image or continuous multiple frames of video and weaken road surface, shadow and other easily confused targets with black smoke to improve the black smoke detection performance.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and pattern recognition technology, and specifically relates to a method for detecting vehicle black smoke based on frequency domain feature enhancement. Background Technology

[0002] In recent years, with the gradual improvement of people's living standards, the number of cars has been increasing, and cars are playing an increasingly important role in the travel and transportation industry. However, with the increase in the number of vehicles, substandard exhaust emissions, especially heavy black smoke, have become a major cause of environmental pollution, posing a threat to human health. With the widespread application of traffic monitoring equipment and the rapid development of artificial intelligence and computer vision technologies, it has become possible to use artificial intelligence technology to semi-automatically or fully automatically analyze and process a series of images and videos captured by monitoring equipment, including vehicle black smoke detection. Using artificial intelligence methods to intelligently analyze and process vehicle black smoke detection tasks can, on the one hand, reduce the burden on manpower, freeing up personnel to perform more complex tasks; and on the other hand, in some scenarios, it can achieve more stable and accurate results than manual observation.

[0003] Traditional methods for detecting vehicle black smoke typically rely on background subtraction or pre-extracting candidate regions for potential black smoke based on color, shape, or other logical parameters. The presence of black smoke in these candidate regions is then determined using manually designed features or deep learning methods. This approach is inconsistent in complex outdoor environments and prone to false positives or false negatives. In recent years, with advancements in target detection technology, some methods based on single-image or multi-frame video capture black smoke. These methods utilize collected vehicle road driving data to extract appearance or spatiotemporal features based on RGB color space, enabling more effective and accurate learning to capture the appearance or motion characteristics of black smoke for detection. While these methods improve the detection rate and accuracy of black smoke targets to some extent, relying solely on RGB color space features can lead to false positives in low-quality areas or false negatives in less obvious areas when dealing with interference from vehicle or building shadows, insufficient lighting, or poor road conditions that appear darker than black smoke. Summary of the Invention

[0004] To address the issue of numerous false positives and false negatives in existing vehicle black smoke detection algorithms when faced with interference from targets that resemble black smoke, this invention provides a vehicle black smoke detection method based on frequency domain feature enhancement. This method can fully extract the frequency domain features of black smoke targets in images and explore the differences in frequency domain features with other easily confused targets, thereby enabling the identification and localization of black smoke targets in any image. To our knowledge, this is the first time that a deep learning method has been used in vehicle black smoke detection tasks to fuse frequency domain features with RGB basic features to improve the accuracy of vehicle black smoke detection in real-world scenarios.

[0005] To achieve the above objectives, the technical solution adopted by the vehicle black smoke detection method based on frequency domain feature enhancement in this invention is as follows:

[0006] A method for detecting black smoke from vehicles based on frequency domain feature enhancement includes the following steps:

[0007] (1) The vehicle driving video collected on the road is split into a series of single-frame images, and black smoke is marked with black smoke bounding boxes in the images;

[0008] (2) If a single-frame image object detection network is used, a target image I is randomly selected from the image set and input into the RGB feature extraction network E to extract the basic features F. rgb If a multi-frame video object detection network is used, then a target image I is randomly selected from the image set along with several adjacent images I. ref Input RGB feature extraction network E to extract basic features F rgb ;

[0009] (3) Convert the target image I selected in step (2) from the RGB color space to the YCbCr color space, and use block-based two-dimensional discrete cosine transform (DCT-2D) to obtain the frequency domain information F. f Then the frequency domain information F f The input is fed into a frequency domain feature extraction network M, where self-attention methods in both the frequency and spatial dimensions are used to enhance features that help distinguish black smoke. Furthermore, a convolutional neural network is used to extract frequency domain features F with high receptive fields and high semantic meaning. freq ;

[0010] (4) The basic feature F obtained in step (2) rgb The frequency domain feature F obtained in step (3) freq Input feature aggregation network A, first concatenate the two types of features through channels, then enhance them using spatial attention and channel attention mechanisms, and finally fuse the enhanced features using a projection module to obtain the final fused feature F. agg ;

[0011] (5) Fuse features F agg Input the black smoke detector H to obtain the predicted bounding box B and class confidence C of the black smoke; calculate the loss based on the bounding box and the true class value to optimize the entire network; if the maximum value of the black smoke class confidence C output by the optimized network exceeds a set threshold C... th If the image contains black smoke, the bounding box coordinates and category confidence score of the black smoke will be output; otherwise, the image will be considered to contain no black smoke.

[0012] Furthermore, the RGB feature extraction network E in step (2) can be any RGB feature extraction network;

[0013] If the network is a single-frame image object detection backbone network I Then the basic feature F rgb The expression is as follows:

[0014] F rgb =Backbone I (I);

[0015] If the network is a multi-frame video object detection backbone network V Then the basic feature F rgb The expression is as follows:

[0016] F rgb =Backbone V (I,I ref );

[0017] Furthermore, in step (3), the selected target image I is converted from the RGB color space to the YCbCr color space, and the frequency domain information F is obtained using block-based two-dimensional discrete cosine transform (DCT-2D). f The expression is as follows:

[0018] F f =DCT(YCbCr(I));

[0019] The self-attention method of the frequency domain feature extraction network M is based on the multi-head self-attention (MHSA) mechanism, and the frequency domain information F f Enhanced frequency domain features are obtained through self-attention mechanisms in both the frequency and spatial dimensions. The expression is as follows:

[0020]

[0021] Among them, MHSA spat (·) and MHSA freq (·) represent multi-head self-attention modules in the spatial and frequency dimensions, respectively. The spatial multi-head self-attention module has only one layer, while the frequency multi-head self-attention module has three layers, corresponding to attention layers for low frequency, high frequency, and full frequency range, respectively. High receptive field and high semantic frequency domain features F are extracted from the frequency domain feature extraction network M. freq The network is a multilayer perceptron (MLP), and the final frequency domain features F freq The expression is as follows:

[0022]

[0023] MLP(·) is a mapping that expands the receptive field for any current implementation based on convolutional networks.

[0024] Further, step (4) specifically involves: using the feature aggregation network A to aggregate the basic features F rgb and frequency domain features F f The splicing feature F is obtained by splicing along the channel dimension. cat Then, spatial attention and channel attention mechanisms are used to enhance the features, and channel transformation is performed on the enhanced features to obtain the feature F that is consistent with the basic feature F. rgb Fusion features F with the same channel dimension agg Among them, the splicing feature F cat and fusion feature F agg The expression is as follows:

[0025] F cat =Concate(F rgb ,F freq );

[0026] F agg =Proj(Attn) chan (Attn spat (F cat )));

[0027] Here, Concate(·) represents the channel concatenation operation, Proj(·) is a convolution module that implements channel number transformation, and Attn chan (·) and Attn spat (·) represents any attention module that can currently assign different weights to each channel and position of the feature map.

[0028] Furthermore, in step (5), the entire network is optimized by calculating the loss based on the bounding boxes and the true values ​​of the categories. The calculated loss includes the bounding box regression loss. and category confidence loss Based on the classification features, the probability P of the entire image containing black smoke targets is further predicted, and the classification loss L is calculated. CLS This will optimize the entire network.

[0029] The beneficial effects of this invention are as follows: Based on the basic features of RGB images, this invention fully extracts the frequency domain features of black smoke. This allows the model to effectively distinguish black smoke targets when the basic RGB image features alone are insufficient. By utilizing the feature differences between black smoke and other easily confused targets in the frequency domain, the model can further identify and locate the black smoke target. This is also the first time that deep learning methods have been used in vehicle black smoke detection tasks to supplement and enhance the basic RGB features with frequency domain features, improving the accuracy of the model in detecting and identifying black smoke. In real-world scenarios, this invention enables the vehicle black smoke detection method to more accurately identify and locate black smoke under different lighting conditions and road conditions. This invention is expected to enhance the robustness of the vehicle black smoke detection method to environmental changes to a certain extent, and improve the accuracy of black smoke detection in real-world scenarios. Attached Figure Description

[0030] Figure 1 This is a flowchart of the vehicle black smoke detection method based on frequency domain feature enhancement in this invention;

[0031] Figure 2 This is a schematic diagram of the model structure of the vehicle black smoke detection method based on frequency domain feature enhancement in this invention;

[0032] Figure 3 This is a schematic diagram of the multi-head self-attention mechanism in the frequency domain feature extraction network of this invention. Detailed Implementation

[0033] To make the technical solution of the present invention clearer, the vehicle black smoke detection method based on frequency domain feature enhancement of the present invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0034] The video captured by roadside surveillance equipment is split frame by frame into a series of images. Images containing black smoke targets are labeled with black smoke bounding boxes. Then, a target image I (which may not contain black smoke targets) is randomly selected and input into the vehicle black smoke detection network based on frequency domain feature enhancement disclosed in this invention to identify and locate the black smoke targets. Figure 1 As shown, proceed with the following steps:

[0035] Step 1: If the RGB feature extraction network E uses a single-frame image object detection backbone network... I Then the target image I is input into the network to extract basic features F. rgb Its expression is as follows:

[0036] F rgb =Backbone I (I);

[0037] If the RGB feature extraction network E adopts a multi-frame video object detection backbone network (Backbone)V Then select several frames of image I adjacent to image I from the image set. ref Together with the target image I, it serves as input to the network to extract basic features F. rgb Its expression is as follows:

[0038] F rgb =Backbone V (I,I ref );

[0039] like Figure 2 As shown, in this embodiment of the invention, the RGB feature extraction network E is a feature extraction backbone network based on the YOLOv5 model that can process multiple frames of video, and can fully extract the spatiotemporal features of the black smoke target.

[0040] Step 2: Convert image I from RGB color space to YCbCr color space and use block-based two-dimensional discrete cosine transform to initially extract frequency domain information F. f Its expression is as follows:

[0041] F f =DCT(YCbCr(I));

[0042] In this embodiment of the invention, the block size of the block-based two-dimensional discrete cosine transform used is 8×8;

[0043] Step 3: F f Information is categorized by frequency level into low-frequency information. and high-frequency information Then low-frequency information and high-frequency information Input to the low-frequency self-attention module MHSA respectively low and high-frequency self-attention module MHSA high The enhanced low-frequency and high-frequency information is obtained, and then the two are concatenated and input into the full-frequency self-attention module MHSA. all Obtain the frequency dimension-enhanced feature F′ f Low-frequency information High-frequency information and frequency dimension enhanced features F′ f The expression is as follows:

[0044]

[0045]

[0046]

[0047] Split(·) represents the operation of splitting information into low-frequency and high-frequency information according to frequency, MHSAlow (·), MHSA high (·) and MHSA all (·) represent multi-head self-attention modules for low frequency, high frequency, and full frequency respectively, and Concate(·) represents channel splicing operation (e.g., Figure 3 (as shown);

[0048] Step 4: Enhance the frequency dimension of the feature F′ f Enter MHSA spat Further obtain features with enhanced spatial dimensions Its expression is as follows:

[0049]

[0050] Among them, MHSA spat (·) represents a multi-head self-attention module in the spatial dimension;

[0051] Step 5: Features that enhance spatial dimensions The input is further processed by a multilayer perceptron (MLP) to extract frequency domain features F with high receptive field and high semantic depth. freq Its expression is as follows:

[0052]

[0053] MLP(·) can be a mapping that expands the receptive field of any current convolutional network implementation.

[0054] Step 6: Transfer the basic features F rgb and frequency domain features F freq The data is concatenated along the channel dimension, and the input spatial attention module Attn is used. spat Obtain basic features F rgb and frequency domain features F freq The spatial attention weights are multiplied by the corresponding features to obtain the spatially enhanced fusion features. Its expression is as follows:

[0055]

[0056]

[0057] in, They represent the basic features F respectively rgb and frequency domain features F freq Spatial attention weights, This indicates the Hadamard product operation;

[0058] Step 7: Merge features Input Channel Attention Module (Attn) chanObtain the attention weights of each channel and... Multiplication further yields channel-dimensional enhanced fusion features. Its expression is as follows:

[0059]

[0060]

[0061] Among them, W chan This indicates the attention weight of each channel;

[0062] Step 8: Merge features The input feature projection module Proj performs channel number transformation to achieve fusion, obtaining the result that is consistent with the basic feature F. rgb Fusion features F with the same number of channels agg Its expression is as follows:

[0063]

[0064] Among them, Proj(·) is a convolution module that implements channel number transformation;

[0065] Step 9: Fuse the features F agg As the final feature input to image I, the black smoke detection head H obtains the predicted bounding box B and class confidence C of the black smoke. In this embodiment of the invention, the black smoke detection head H is a one-stage target detection network based on anchor boxes. During the training phase, the probability P of the entire image containing the black smoke target is further predicted based on the classification features, and the bounding box regression loss is calculated. Category confidence loss And classification loss L CLS Optimize the entire network;

[0066] Step 10: During the testing phase, if the maximum value of the confidence score C for the black smoke category exceeds the set threshold C... th If the image contains black smoke, the bounding box coordinates and category confidence score of the black smoke are output; otherwise, the image is considered to contain no black smoke. In this embodiment of the invention, the threshold is set to C. th =0.75.

[0067] The frequency domain feature-enhanced vehicle black smoke detection method disclosed in this invention was applied to a self-built multi-scene video vehicle black smoke detection task. The experimental test data obtained are shown in Table 1.

[0068] Table 1

[0069]

[0070] As can be seen from the experimental results in Table 1, the bounding boxes of black smoke targets in the four scenarios are predicted relatively accurately, with mAP all above 50%. In particular, in the self-built black smoke scenarios 3 and 4, the mAP exceeds 90%. This indicates that the technical solution proposed in this invention can reduce the interference of targets such as road surfaces and shadows, and effectively extract the features of black smoke.

[0071] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for detecting vehicle black smoke based on frequency domain feature enhancement, characterized in that, Specifically, the steps include the following: (1) The vehicle driving video collected on the road will be split into a series of single-frame images, and the images with black smoke will be marked with black smoke bounding boxes; (2) If a single-frame image object detection network is used, a target image is randomly selected from the image set. Input RGB feature extraction network Extracting basic features ; If a multi-frame video object detection network is used, a target image is randomly selected from the image set. With several adjacent images Input RGB feature extraction network Extracting basic features ; (3) Select the target image in step (2) Convert from RGB color space to YCbCr color space, and obtain frequency domain information using block-based 2D Discrete Cosine Transform (DCT-2D). ; Then frequency domain information Input to frequency domain feature extraction network The method employs self-attention techniques in both frequency and spatial dimensions to enhance features that help distinguish black smoke, and further uses convolutional neural networks to extract frequency domain features with high receptive fields and high semantic depth. Specifically, the selected target image Convert from RGB color space to YCbCr color space, and obtain frequency domain information using block-based 2D Discrete Cosine Transform (DCT-2D). The expression is as follows: ； Will Information is categorized by frequency level into low-frequency information. and high-frequency information Then low-frequency information and high-frequency information Input the low-frequency self-attention module respectively and high-frequency self-attention module The enhanced low-frequency and high-frequency information is obtained, and then the two are concatenated and input into a self-attention module covering the entire frequency range. Obtain features with enhanced frequency dimension Low-frequency information High-frequency information and frequency dimension enhanced features The expression is as follows: ; ; ; in, This represents the operation of splitting information into low-frequency and high-frequency information based on frequency. , and Multi-head self-attention modules representing low frequency, high frequency, and full frequency respectively. This represents a channel splicing operation; Features that enhance the frequency dimension enter Further obtain features with enhanced spatial dimensions Its expression is as follows: ; in, Multi-head self-attention modules representing spatial dimensions; The spatial dimension multi-head self-attention module has only one layer, while the frequency dimension multi-head self-attention module has three layers, corresponding to attention layers for low frequency, high frequency, and full frequency range, respectively. In the frequency domain feature extraction network... Extracting high receptive field and high semantic frequency domain features The network is a multilayer perceptron (MLP), and the final frequency domain features The expression is as follows: ; in, It is a mapping that expands the receptive field for any current implementation based on convolutional networks; (4) The basic features obtained in step (2) and the frequency domain features obtained in step (3) Input Feature Aggregation Network First, the two types of features are concatenated using channels, and then enhanced using spatial attention and channel attention mechanisms. Finally, a projection module is used to fuse the enhanced features to obtain the final fused features. ; (5) Incorporate fusion features Input black smoke detector Obtain the bounding box prediction results for black smoke. and category confidence ; Calculate the loss based on the bounding box and the true value of the category to optimize the entire network; If the confidence level of the black smoke category output by the optimized network is... The maximum value exceeds the set threshold. If the image contains black smoke, the bounding box coordinates and category confidence score of the black smoke will be output; otherwise, the image will be considered to contain no black smoke.

2. The vehicle black smoke detection method based on frequency domain feature enhancement according to claim 1, characterized in that, The RGB feature extraction network in step (2) For any RGB feature extraction network; If this network is a single-frame image object detection backbone network Then the basic features The expression is as follows: ； If the network is a multi-frame video object detection backbone network Then the basic features The expression is as follows: 。 3. The vehicle black smoke detection method based on frequency domain feature enhancement according to claim 1, characterized in that, Step (4) specifically involves: utilizing the feature aggregation network Basic features and frequency domain features The splicing feature is obtained by splicing along the channel dimension. Then, spatial attention and channel attention mechanisms are used to enhance the features, and channel transformation is performed on the enhanced features to obtain the same features as the basic features. Fusion features with the same channel dimension splicing features and fusion features The expression is as follows: ； ； in, This represents a channel splicing operation. It is a convolution module that implements channel number transformation. and It is currently the only attention module capable of assigning different weights to each channel and position of the feature map.

4. The vehicle black smoke detection method based on frequency domain feature enhancement according to claim 1, characterized in that, In step (5), the entire network is optimized by calculating the loss based on the bounding box and the class ground truth. The calculated loss includes the bounding box regression loss. and category confidence loss Based on the classification features, it further predicts the probability that the entire image contains a black smoke target. And calculate this classification loss. This will optimize the entire network.

Images