A fast segmentation method for thin smoke based on multiple attention mechanism

Abstract

The present invention discloses a method for fast segmentation of thin smoke based on multiple attention mechanisms. The key of the present invention is to use multiple attention mechanisms (time domain, channel, and spatial attention mechanisms) to filter out useless interference information multiple times, and to use the sensitivity of the frame difference method to smoke and the subsequent lightweighting of the entire network to improve the algorithm speed, so as to capture the newly appeared translucent smoke. Experimental verification shows that the use of this algorithm on the smoke data set of the National Laboratory of Fire Science of the University of Science and Technology of China can filter out a large amount of interference while accurately segmenting early thin smoke, and the processing delay of each frame is less than 50ms. This breaks through the problem that the existing thin smoke monitoring algorithm cannot take into account both accuracy and interference screening capabilities as well as low processing delay, and realizes the fast and accurate segmentation of early thin smoke.

Description

Technical Field

[0001] The invention belongs to the field of image processing and relates to a thin smoke fast segmentation method based on a multiple attention mechanism. Background Art

[0002] Fire is a catastrophic problem that every country in the world needs to face. It is sudden and destructive, and can pose a great threat to human property and ecological resources. If the fire can be discovered in time, it can not only greatly reduce casualties, but also save a lot of ecological and social resources.

[0003] By observing the occurrence of fire, we can know that there will be a process of smoke diffusion before the fire occurs: from thin to thick, from translucent to milky white or black. If the newly appeared thin and translucent smoke can be captured, the early warning of the fire can be realized with the greatest efficiency. The specific characteristics of thin smoke are: 1) thin smoke has no fixed shape, and its edge gradually blends into the background as the smoke moves, so it does not form a strong contrast with the background, and the edge feature is weak; 2) thin smoke has no fixed smoke color, and due to its internal translucent characteristics, it will introduce a lot of background interference information, and the internal features are weak; 3) thin smoke is greatly affected by the ambient airflow and changes in real time. At present, many methods have been produced in the research of smoke monitoring, such as: optical flow method based on deep learning, single and double stage detection method, dual stream method, etc. These methods have played a huge role in monitoring some more obvious smoke. However, in detecting the newly appeared thin smoke, the effect is far from the ideal goal. The main problems include: 1) The method based on static image features is less sensitive to thin smoke and often misses detection; 2) The method based on dynamic features is more sensitive to moving thin smoke, but it will bring a lot of non-smoke dynamic information, that is, false alarms; 3) The method based on a simple combination of static and dynamic features has improved the effect of detecting thin smoke, but due to the lack of specificity for smoke, there are still some missed detections and false alarms, and the detection efficiency is low and cannot keep up with the real-time changes of smoke.

[0004] In summary, the accuracy of the current method for thin smoke and a large amount of interference are a pair of difficult-to-solve contradictions. Therefore, in the fire warning task, we increasingly hope to eliminate most of the interference while being able to quickly and accurately achieve early thin smoke monitoring. Summary of the invention

[0005] In view of the shortcomings of the prior art, the present invention provides a method for fast segmentation of thin smoke based on a multi-attention mechanism, which attempts to break through the limitations of the existing thin smoke detection methods. The method is divided into three parts:

[0006] 1) A dynamic monitoring module with a temporal attention mechanism is added: it consists of a frame difference method, a speed filter (the results of frame difference between frames 1, 2, 3, and 4 in the four-frame image are frame-differenced again, and then this part is removed from the frame difference results of frames 1 and 2 to filter out targets that move too fast), and a size filter (to filter out targets that are too small).

[0007] 2) Static monitoring module with added channel and spatial attention mechanism: It is composed of the improved YOLOv5 (the backbone part is replaced with the mobilenet v2 lightweight feature extraction network, and the channel feature and spatial feature attention mechanism are added).

[0008] 3) Result generation module: Generates the final thin smoke segmentation image result.

[0009] This method is different from some previous smoke monitoring methods. It proposes multiple attention mechanisms for thin smoke from various aspects. Through these mechanisms, some useless interference information can be continuously eliminated during the smoke monitoring process, and only the smoke itself can be focused on. The speed is improved through a lightweight network, thereby achieving efficient monitoring of thin smoke.

[0010] Step (1): Use the dynamic monitoring module’s temporal attention mechanism to filter out interference

[0011] After removing the static information according to the frame difference method, the frame difference images of all dynamic targets are obtained; secondly, the dynamic target frame difference images are speed filtered to remove the targets that move too fast; then the frame difference images are mean filtered to eliminate noise, and binarized to initially obtain the dynamic target mask binary image. Next, the bounding rectangles of all dynamic targets in the binary mask image are taken, and the bounding rectangles of non-smoke targets that are smaller than the set threshold are removed, that is, size filtering; finally, according to the position information of the remaining bounding rectangle frame, all interference content that does not belong to the bounding box range in the first frame of the original image is set to zero; and the time domain preliminary screening image is obtained; the above series of operations are the time domain attention mechanism in dynamic monitoring, and the purpose is to obtain a picture that has been preliminarily screened in the time domain and sent to the subsequent steps. Since no network reasoning is involved in the middle and the frame difference method is simple to calculate, the calculation delay of the whole process is relatively small.

[0012] Step (2), using the static monitoring module channel and spatial attention mechanism to filter out interference and obtain specific information of thin smoke;

[0013] The improved YOLOv5 extracts features from the time domain initial screening image through mobilenet v2 to obtain the original feature map. After generating the feature map, the attention weights are inferred along the channel and space dimensions using the added channel and space attention mechanisms, and then multiplied with the original feature map to adjust the features, thereby filtering out unimportant information again. Finally, the adjusted feature map is input into the neck and head parts of YOLOv5 to obtain the target position and category prediction information. According to the output of the thin smoke position result, the target is framed in the original image and the early smoke part in the binary mask image in step (1) is filtered out;

[0014] The improved YOLOv5 replaces the backbone part with the mobilenet v2 lightweight feature extraction network, and adds channel feature and spatial feature attention mechanisms;

[0015] Step (3), generate thin smoke segmentation image

[0016] The early smoke mask image filtered by multiple attention mechanisms is superimposed with the original image to generate the final early smoke segmentation result.

[0017] Preferably, the frame difference method is specifically as follows: cyclically read in 4 frames of original images of early thin smoke and grayscale them respectively; perform frame difference again on the results of frame 1, 2, 3, and 4 of the 4 grayscale images after frame difference, and then remove this part from the frame difference results of frame 1 and 2.

[0018] The beneficial effects of the present invention are as follows: The key to the present invention is to use multiple attention mechanisms to filter out useless interference information multiple times, and to use the sensitivity of the frame difference method to smoke and the lightweight of the entire network, thereby achieving fast and accurate segmentation of early thin smoke. Experimental verification shows that the use of this method on the smoke dataset of the National Laboratory of Fire Science of the University of Science and Technology of China can filter out a large amount of interference while accurately segmenting early thin smoke, and the processing delay of each frame is less than 50ms, breaking through the problem that the existing thin smoke monitoring method cannot simultaneously take into account accuracy, interference screening ability, and low processing delay. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 is a flow chart of the present invention;

[0020] Figure 2 It is the thin smoke segmentation diagram of the present invention. DETAILED DESCRIPTION

[0021] The present invention is further analyzed below in conjunction with specific implementation.

[0022] This experiment uses the dry leaf smoldering video of the National Laboratory of Fire Science at the University of Science and Technology of China as the experimental object. Figure 1 As shown in FIG. 1 , the specific steps for segmenting the thin smoke generated in the early stage of smoldering of dry leaves are as follows:

[0023] Step (1): Use the dynamic monitoring module’s temporal attention mechanism to filter out interference

[0024] The four original images of the dry leaf smoldering video are read in a loop, and the 1st and 2nd frames are grayed out respectively. The frame difference images of all dynamic targets are obtained after the static information is filtered out. Secondly, the speed filter is performed to filter out the targets with too fast movement speed (clutter in the image). Then the frame difference image is mean filtered to eliminate noise, and the dynamic target mask binary image is initially obtained by binarization. Next, the bounding rectangles of all dynamic targets in the binary mask image are taken, and the bounding rectangles of non-smoke targets less than a certain value are filtered out, that is, size filtering. Finally, according to the position information of the remaining bounding rectangle frame, all the interference content in the first frame of the original image that does not belong to the bounding box range is set to zero. The above series of operations is the time domain attention mechanism in dynamic monitoring, and the purpose is to obtain a dry leaf smoldering image that has been initially screened in the time domain and sent to the subsequent steps. Since no network reasoning is involved in the middle and the frame difference method is simple to calculate, the calculation delay of the whole process is small.

[0025] Step (2): Use the static monitoring module channel and spatial attention mechanism to filter out interference and obtain detailed information about thin smoke.

[0026] The improved YOLOv5 extracts features from the time-domain preliminary screening image of the dry leaf smoldering through mobilenet v2. After generating the feature map, the attention weights are inferred along the channel and space dimensions using the added channel and spatial attention mechanisms, and then multiplied with the original feature map to adjust the features, thereby filtering out unimportant information (such as leaves moving in the wind in the background). Finally, the adjusted feature map is input into the neck and head parts of YOLOv5 to obtain the prediction information of the thin smoke generated in the early stage of the dry leaf smoldering. According to the output position result, the target is framed in the first frame of the original image of the dry leaf smoldering video and the early smoke part in the mask image in step (1) is filtered out.

[0027] Step (3): Generate a thin smoke segmentation image

[0028] The early smoke mask image filtered by multiple attention mechanisms is superimposed with the original image to generate the final early smoke segmentation result.

[0029] By analyzing the final result video, we found that in the entire process from the appearance to the end of the thin smoke, this method can not only filter out a lot of interference such as leaves and clutter, but also accurately segment the thin smoke, and the processing delay of each frame is less than 50ms. This breaks through the problem that the existing thin smoke monitoring method cannot take into account accuracy, interference screening ability and low processing delay at the same time. Figure 2 As shown, a frame in the result video is extracted as the result graph.

[0030] The above embodiments are not limitations of the present invention, and the present invention is not limited to the above embodiments. As long as the requirements of the present invention are met, they belong to the protection scope of the present invention.

Claims

1. A fast thin smoke segmentation method based on multiple attention mechanisms, characterized in that: The method specifically comprises the following steps: Step (1), using the dynamic monitoring module’s time-domain attention mechanism to filter out interference; After filtering out the static information according to the frame difference method, the frame difference images of all dynamic targets are obtained; secondly, the dynamic target frame difference images are subjected to speed filtering to filter out targets that move too fast; then the frame difference images are subjected to mean filtering to eliminate noise, and binarization is performed to initially obtain a dynamic target mask binary image; then, the bounding rectangles of all dynamic targets in the binary mask image are taken, and the bounding rectangles of non-smoke targets that are smaller than the set threshold are filtered out, i.e., size filtering; finally, according to the remaining bounding rectangle frame position information, all interference contents in the first frame of the original image that do not belong to the bounding box range are set to zero; and the time domain preliminary screening image is obtained; Step (2), using the static monitoring module channel and spatial attention mechanism to filter out interference and obtain specific information of thin smoke; The improved YOLOv5 extracts features from the time domain initial screening image through mobilenet v2 to obtain the original feature map. After generating the feature map, the attention weights are inferred along the channel and space dimensions respectively using the added channel and space attention mechanisms, and then multiplied with the original feature map to adjust the features, thereby filtering out unimportant information again; finally, the adjusted feature map is input into the neck and head parts of YOLOv5 to obtain the target position and category prediction information. According to the output of the thin smoke position result, the target is framed in the original image and the early smoke part in the binary mask image in step (1) is filtered out; The improved YOLOv5 replaces the backbone part with the mobilenet v2 lightweight feature extraction network, and adds channel feature and spatial feature attention mechanisms; Step (3): Generate a thin smoke segmentation image The early smoke binary mask image filtered by multiple attention mechanisms is superimposed with the original image to generate the final early smoke segmentation result.

2. According to claim 1, a method for fast segmentation of thin smoke based on a multiple attention mechanism is characterized in that: The frame difference method is specifically as follows: cyclically read in 4 frames of original images of early thin smoke and grayscale them respectively; perform frame difference again on the results of frame 1, 2, 3, and 4 of the 4 grayscale images after frame difference, and then remove this part from the frame difference results of frame 1 and 2.

Images