A real-time smoke detection method based on video

Abstract

The present invention relates to a real-time smoke detection method based on video, comprising the following steps: video input; image preprocessing; multi-task learning network processing images, including: (1) real-time smoke segmentation network: adopting a BiSeNet-based dual-path image segmentation network, including a spatial path and a context path, and introducing a pyramid pooling module in the spatial path and a lightweight channel attention module in the context path; performing feature fusion on the feature maps output by the two paths and outputting a segmentation result; (2) lightweight smoke detection network: adopting an FCOS paradigm network based on an anchor-free strategy, wherein the backbone network is set to be MobileNetV3 with a fully connected layer removed, and connecting a feature pyramid module to fuse low-level texture information with high-level semantic information to generate a more expressive feature map; inputting the labeled smoke image into the lightweight smoke detection network; and smoke discrimination.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and deep learning, and specifically to a real-time smoke detection method based on video, which is essentially a problem of target detection and image segmentation. Background Art

[0002] Fire is a very common disaster that poses a significant threat to people's lives and property. The initial stages of a fire are often accompanied by the appearance of smoke, making accurate smoke identification crucial for fire prevention. Current image smoke detection tasks require low-latency, high-precision smoke detection and alarming on monitoring platforms, a challenging task. Furthermore, smoke is always translucent, resulting in a highly complex blend of the background and itself. Sparse or fine smoke is not visually distinct, and its boundaries are often blurred. These inherent characteristics of smoke pose significant challenges to smoke detection.

[0003] In recent years, with the rapid development of deep learning technology, numerous image segmentation and object detection methods have emerged, playing an important role in many fields. However, most of these methods cannot meet real-time requirements, and their recognition performance in specific scenarios such as smoke cannot meet social production needs. Therefore, how to improve the accuracy of smoke detection algorithms while meeting real-time requirements is an urgent problem to be solved. Summary of the Invention

[0004] The present invention provides a video-based real-time smoke detection method with high detection accuracy and meeting real-time requirements. The technical solution is as follows:

[0005] A video-based real-time smoke detection method includes the following steps:

[0006] Step 1, video input: read the image from the camera sub-stream or video.

[0007] Step 2: Image preprocessing: perform image scaling and image denoising preprocessing operations on the read image;

[0008] Step 3: Multi-task learning network processing images, including two parts:

[0009] (1) Real-time smoke segmentation network

[0010] Adopt a dual-path image segmentation network based on BiSeNet, including a spatial path and a context path, and introduce a pyramid pooling module in the spatial path and a lightweight channel attention module in the context path;

[0011] The pyramid pooling module adopts a four-branch parallel structure. Each branch contains a global average pooling-convolution-batch normalization-ReLU activation function path. The sizes of the pooling templates from top to bottom are 1×1, 2×2, 3×3, and 6×6, respectively, used to generate feature maps of different resolutions. Feature maps of different resolutions are spliced ​​into a single feature map by channel.

[0012] The lightweight channel attention module is used to extract the dependencies between channels. The method is as follows: ① Perform a global average pooling operation on the input image; ② Perform a 1D convolution operation with a convolution kernel size of k, and obtain the weight ω of each channel through the Sigmoid activation function, where the k value is selected by the formula We get C, which represents the number of feature channels, γ = 2, and b = 1. ③ Perform matrix multiplication on the weight ω and the input image to output the feature map.

[0013] The feature maps output by the two paths are fused to output the segmentation result;

[0014] (2) Lightweight smoke detection network

[0015] The FCOS paradigm network based on the anchor-free strategy was used. Its backbone network was set as MobileNetV3 with the fully connected layers removed. After the side output of MobileNetV3, a feature pyramid module was connected. This feature pyramid module consists of convolution operations with a convolution kernel of 1, which is used to fuse low-level texture information with high-level semantic information to generate more expressive feature maps. The labeled smoke images were input into the lightweight smoke detection network, and the network was trained to obtain smoke detection results.

[0016] Step 4, smoke identification: Combine the smoke segmentation result obtained in step 3 with the smoke detection result to achieve smoke identification.

[0017] Combine the smoke segmentation result obtained in step 3 with the smoke detection result, and set the minimum bounding rectangle of the smoke segmentation result as M seg , the rectangular detection box of the smoke detection result is M det , the final detection box is composed of M seg ∩M det To determine and obtain more accurate detection results to determine whether smoke is generated.

[0018] The smoke detection method proposed in the present invention has the characteristics of high speed, good real-time performance, high precision, etc., and can be applied to actual production and life. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 is a flow chart of the method of the present invention;

[0020] Figure 2 It is a screenshot from the video data;

[0021] Figure 3 is a smoke segmentation result diagram of the method of the present invention;

[0022] Figure 4 is a smoke detection result diagram of the method of the present invention;

[0023] Figure 5 It is the alarm picture of the method of the present invention; DETAILED DESCRIPTION

[0024] This embodiment provides a real-time smoke detection method based on video. The specific steps are as follows: Figure 1 As shown, the following steps are included:

[0025] (1) Read images from the camera sub-stream or video to complete data collection.

[0026] (2) Image scaling and image denoising operations are performed on the read image. Assuming that the input image is I and the output image is O, the image preprocessing process can be expressed by the following calculation formula:

[0027] O(i,j)=median[I Bili (i-1,j-1),I Bili (i,j-1),I Bili (i+1,j-1),I Bili (i-1,j),I Bili (i,j),I Bili (i+1,j),I Bili (i-1,j+1),I Bili (i,j+1),I Bili (i+1,j+1)]

[0028] In the above formula, I Bili "i" represents image scaling after bilinear interpolation, "i" and "j" represent the pixel coordinates, and "median" indicates the use of a median filter. The process involves selecting a 3×3 window, sorting the pixels within the window by value, and generating a monotonically increasing (or decreasing) two-dimensional data sequence. All pixel values ​​within the window are then replaced with the median pixel. Finally, the window is slid until all pixels in the image have been processed. This preprocessed image reduces noise, effectively improving final detection accuracy.

[0029] (3) A multi-task learning network is used to process and analyze the image. First, the input image is converted into a 4-dimensional tensor tensor = [N, C, H, W], where N represents the number of images processed at one time, which is set to 1 here; C represents the number of image channels, which is 3 for RGB channels; H and W represent the height and width of the image, respectively. After bilinear interpolation, the image is scaled to a uniform scale of 1024 × 1024, resulting in a 4-dimensional tensor tensor = [1, 3, 1024, 1024]. The image is then input into the real-time smoke segmentation network and the lightweight smoke detection network simultaneously in the form of a tensor.

[0030] For the smoke segmentation network, a BiSeNet-based dual-path image segmentation network is adopted, which includes a spatial path and a context path. A pyramid pooling module is introduced in the spatial path, and a lightweight channel attention module is introduced in the context path.

[0031] The pyramid pooling module adopts a four-branch parallel structure. Each branch contains a global average pooling-convolution-batch normalization-ReLU activation function path. The sizes of the pooling templates from top to bottom are 1×1, 2×2, 3×3, and 6×6, respectively. The purpose is to generate feature maps of different resolutions. Finally, these multi-scale feature maps are spliced ​​into one feature map by channel. The pyramid pooling module can effectively enhance the multi-scale feature extraction capability of the segmentation network.

[0032] The lightweight channel attention module can extract the dependencies between channels. The specific operations are as follows: ① Perform global average pooling on the original input image; ② Perform a 1D convolution operation with a convolution kernel size of k, and obtain the weight ω of each channel through the Sigmoid activation function, where the k value is selected by the formula It is obtained that C represents the number of feature channels, γ = 2, b = 1; ③ Perform matrix multiplication operation on the weight ω and the original input image to obtain the final output feature map. The use of one-dimensional convolution can efficiently realize the interaction of local channels and reduce the parameters of the segmentation network.

[0033] Finally, the feature maps output by the two paths are fused through the feature fusion module to output the segmentation result. The obtained smoke segmentation map has rich spatial detail information and the segmentation result is relatively complete.

[0034] For the smoke detection network, an FCOS paradigm network based on the anchor-free strategy is adopted. The anchor-free strategy can effectively reduce network parameters. At the same time, its backbone network is set to MobileNetV3 with the fully connected layer removed. This network uses depthwise separable convolution and a linear bottleneck to residual structure, which greatly improves the detection speed. At the same time, to improve detection accuracy, a feature pyramid module is connected after the side output of MobileNetV3. This module consists of convolution operations with a convolution kernel of 1, which can effectively integrate low-level texture information with high-level semantic information to generate more expressive feature maps, thereby improving detection accuracy.

[0035] The two networks were trained with a total of 8,500 images, of which 6,600 were used for training and 1,900 for testing. All images collected were 24-bit RGB images with resolutions ranging from 256×256 to 1920×1080, and were arranged in a unified naming format.

[0036] The training of the smoke detection network uses the fusion of Focal loss function and IOU loss function to calculate the error. The specific formula can be expressed as:

[0037]

[0038] In the above formula, L cls Represents the Focal loss function, L reg Represents the IOU loss function, N pos Represents the number of positive samples, λ=1 is L reg The balance weight, is an indicator function, if C * 1 if >0, 0 otherwise.

[0039] The smoke segmentation network training uses the cross entropy and Dice fusion loss function to calculate the error. Assuming that the image output by the network is Y and the corresponding binary label image is L, the cross entropy loss function can be expressed as:

[0040]

[0041] In the above formula, y i and l i Denotes the i-th pixel in the output image Y and the label image L, respectively, and n represents the number of all pixels. The Dice loss function can be expressed as:

[0042]

[0043] Therefore, the overall loss function can be expressed as:

[0044] loss final =losscross-entropy +loss Dice

[0045] During the training process, an end-to-end training method was adopted, using the PyTorch framework for training, and the SGD optimizer was used. The minimum data batch was set to 2, the number of training rounds was 40, the initial learning rate was 0.0001, and the learning rate decayed to the previous 0.1 every 10 rounds.

[0046] (4) Combine the smoke segmentation result obtained in step (3) with the smoke detection result. Assume that the minimum bounding rectangle of the smoke segmentation result is M seg , the rectangular detection box of the smoke detection result is M det , then the final detection box size is M seg ∩M det To determine and obtain more accurate detection results, to judge whether smoke is generated, and if so, output alarm information and location information for subsequent expert system analysis. Figure 5 This is a saved alarm picture. It can be seen that the present invention can meet the requirements of real-time smoke detection.

Claims

1. A video-based real-time smoke detection method comprising the following steps: Step 1, video input: read the image from the camera sub-stream or video; Step 2: Image preprocessing: perform image scaling and image denoising preprocessing operations on the read image; Step 3: Multi-task learning network processes images, which includes two parts: (1) Real-time smoke segmentation network Adopt a dual-path image segmentation network based on BiSeNet, including a spatial path and a context path, and introduce a pyramid pooling module in the spatial path and a lightweight channel attention module in the context path; The pyramid pooling module adopts a four-branch parallel structure. Each branch contains a global average pooling-convolution-batch normalization-ReLU activation function path. The sizes of the pooling templates from top to bottom are 1×1, 2×2, 3×3, and 6×6, respectively, used to generate feature maps of different resolutions. Feature maps of different resolutions are spliced ​​into a single feature map by channel. The lightweight channel attention module is used to extract the dependencies between channels. The method is as follows: ① Perform a global average pooling operation on the input image; ② Perform a 1D convolution operation with a convolution kernel size of k, and obtain the weight ω of each channel through the Sigmoid activation function, where the k value is selected by the formula We get C, which represents the number of feature channels, γ = 2, and b = 1. ③ Perform matrix multiplication on the weight ω and the input image to output the feature map. The feature maps output by the two paths are fused to output the segmentation result; (2) Lightweight smoke detection network The FCOS paradigm network based on the anchor-free strategy is adopted. Its backbone network is set as MobileNetV3 with the fully connected layer removed. After the side output of MobileNetV3, a feature pyramid module is connected. This feature pyramid module consists of convolution operations with a convolution kernel of 1, which is used to fuse low-level texture information with high-level semantic information to generate more expressive feature maps. Input the labeled smoke image into the lightweight smoke detection network, train the network, and obtain the smoke detection results; Step 4, smoke identification: Combine the smoke segmentation result obtained in step 3 with the smoke detection result, and set the minimum enclosing rectangle of the smoke segmentation result as M seg , the rectangular detection box of the smoke detection result is M det , the final detection box is composed of M seg ∩M det To determine and obtain more accurate detection results to determine whether smoke is generated.

Images