Early fire detection method based on shallow-layer guided deep feature fusion

Abstract

A method for early fire detection based on shallow-layer guided deep feature fusion includes the following steps: 1. Using scaling and pooling operations to reduce the size of the input feature map; 2. Using an SGD module to assign different weights to each shallow feature channel to guide the subsequent fusion of deep channel features; 3. Using a backbone network to extract input features and fuse the channels in the deep features; 4. Processing the fused deep features to further extract fused features; 5. Using a detection head to process the output features to obtain early fire detection results for the video frame. This method can effectively detect early fires with low latency and high accuracy.

Description

Technical Field

[0001] The present invention relates to the technical field of fire detection in the field of computer vision, and proposes an early fire detection method based on shallow-layer guided deep-layer feature fusion. Background Art

[0002] Fire is a common disaster that not only causes huge losses but also endangers life and property. If fires can be detected in their early stages, prompting alarms and firefighting measures, the losses caused by fires can be greatly reduced.

[0003] Traditional fire detection methods primarily include temperature sensing, smoke sensing, and gas sensing, which detect changes in the environment to determine if a fire has occurred. However, these traditional fire detection methods have high environmental requirements, significantly reducing sensitivity in open environments and requiring a long time from ignition to alarm. By the time an alarm is triggered, the fire has often spread and is difficult to control. Early-stage fire detection methods based on computer vision can effectively overcome the environmental dependence of traditional fire detection methods and have attracted increasing research attention. While their detection accuracy continues to improve and detection time is gradually decreasing, they still fall short of meeting the needs of practical applications. Key challenges include: 1) Some methods directly use existing network models without tailoring and optimizing them for detecting a single type of fire. 2) Some methods ignore the size of flames at different stages and are more effective at recognizing large flames. However, early-stage fires are smaller and more difficult to recognize than those in the developing stages. 3) Some methods fail to balance the balance between network model size and recognition accuracy: On the one hand, in pursuit of higher accuracy, large network models with a large number of parameters are designed, resulting in significant time consumption. On the other hand, some methods excessively pursue lightweight network models, sacrificing detection accuracy.

[0004] For example, Chinese patent document CN111341059A discloses a method and apparatus for early wildfire smoke detection based on depth separation and target perception. A sample image and a target image are fed into a target perception deep network to generate a first and second depth feature matrix. The separated feature matrices are then convolved to obtain the maximum value of the smoke response matrix, which is then compared with a threshold to determine the smoke location.

[0005] For example, Chinese patent document CN115424116A discloses a fire detection method using a lightweight convolutional neural network, which obtains an image of the ultra-high voltage converter transformer area and inputs it into a pre-trained fire recognition network model to obtain fire detection results.

[0006] However, these technical documents all have the above challenges. The research and development team of this application is committed to solving these problems in the existing technology by integrating shallow and deep network features. Summary of the Invention

[0007] In view of the technical problems existing in the prior art, the present invention aims to propose a lightweight early fire detection method - a shallow-guided deep feature fusion early fire detection method. According to the characteristics of early fires, the existing model is targetedly improved, and the shallow and deep features of the network are integrated. The accuracy of the model is guaranteed while reducing the scale. At the same time, multi-threading technology is used to effectively improve the execution efficiency and stability of the algorithm.

[0008] Specifically, the present invention aims to provide an early fire detection method based on shallow-layer guided deep-layer feature fusion, which comprises the following steps:

[0009] Step A: Use scaling and pooling operations to reduce the size of the input feature map;

[0010] Step B: Use the SGD module to assign different weights to each shallow feature channel to provide guidance for the subsequent deep channel feature fusion;

[0011] Step C: Use the backbone network to extract input features and fuse the channels in the deep features;

[0012] Step D: Process the fused deep features and further extract the fused features;

[0013] Step E: Use the detection head to process the output features to obtain the early fire detection results of the video frame.

[0014] Furthermore, step A specifically includes the following steps:

[0015] Step A01: Perform adaptive image scaling on the input image to unify the input image to 640×640 size;

[0016] Step A02: Use a 2×2 convolution kernel to perform a maximum pooling operation with a stride of 2 on the image processed in step A01 to reduce the image size to 320×320.

[0017] Wherein step B specifically comprises the following steps:

[0018] Step B01: Split the input image in RGB color space.

[0019] [R,G,B]=split(frame)

[0020] Where frame is the feature map obtained after processing in step 1), split(·) is the channel splitting function in OpenCV, which splits the image in the RGB color space into R, G, and B channels respectively, and obtains the features of each channel [R, G, B].

[0021] Step B03: compress the Y, Cb, and Cr channel features obtained in step B02.

[0022]

[0023] where X c =[X Y ,X Cb ,X Cr ], represents the features of Y, Cb, and Cr channels, and H and W represent the height and width of the feature map of each channel respectively.

[0024] Step B04: For Y obtained in step B03 c Perform activation processing to obtain the weights of each channel in the shallow features,

[0025] W=σ(FC(Relu(FC(Y c ))))

[0026] Where σ represents the sigmoid activation function, FC represents the fully connected layer, Relu represents the Relu activation function, and W represents the weight of each channel obtained after processing.

[0027] Step B05: Use the weight W obtained in step B04 to update the X representing the Y, Cb, and Cr channel features obtained in step B02.

[0028]

[0029] in represents channel-by-channel multiplication, Represents the fused shallow features obtained after weight update of the input image.

[0030] Furthermore, step C specifically includes the following steps:

[0031] Step C01: For the input features Perform channel splitting and divide the features into two channels on average.

[0032]

[0033] Where i = 1, 2. Avg_split means splitting the input feature into two parts evenly.

[0034] Step C02: For the split features obtained in step C01 Perform a 1×1 convolution operation:

[0035]

[0036] Where i = 1, 2. Conv2D 1×1 Represents a 1×1 convolution operation.

[0037] Step C03: For the Perform channel scattering operation.

[0038]

[0039] Where i = 1, 2. Shuffle represents a channel scattering operation, which scatters the features of each channel.

[0040] Step C04: For the Perform a 3×3 convolution operation:

[0041]

[0042] Where i = 1, 2. Conv2D 3×3 Represents a 3×3 convolution operation.

[0043] Step C05: For the Perform a 1×1 convolution operation:

[0044]

[0045] Where i = 1, 2. Conv2D 1×1 Represents a 1×1 convolution operation.

[0046] Step C06: For the result obtained in step C05 Perform concat splicing operation:

[0047]

[0048] Step C07: For the Perform channel scattering operation:

[0049]

[0050] Step C08: For the The features entered in step C01 Perform addition operation:

[0051]

[0052] Step C09: The result obtained in step C08 As input, repeat the operations in step C01 to step C08 6 times. The result of the i-th operation is

[0053] Step C10: The deep features obtained in step C09 after six repeated operations are fused Perform convolution operation:

[0054]

[0055] Step C11: For the Perform upsampling operation:

[0056]

[0057] Further, step D specifically includes the following steps:

[0058] Step D01: For the and the feature obtained by repeating the operation for the third time in step C09 Perform concat splicing operation:

[0059]

[0060] Step D02: For the The image is processed using the C3 module in YOLO v5. C3 is a built-in module in the YOLOv5 network structure, consisting of convolutional layers, activation functions, and residual connections. It is used to extract low-level and high-level features from images.

[0061]

[0062] Step D03: For the Perform convolution operation:

[0063]

[0064] Step D04: For the Perform upsampling operation:

[0065]

[0066] Step D05: For the The same as the first operation in step C09 Perform splicing operations:

[0067]

[0068] Step D06: Perform C3 operation on the spliced ​​features obtained in step D05:

[0069]

[0070] Furthermore, step E specifically includes the following steps:

[0071] E01. Further process the deep features obtained in step D Send it to YOLO Head to get the final recognition result.

[0072] According to the present invention, since the size of the feature image is reduced by scaling and pooling processing, the time loss of the error back propagation of the convolution kernel can be reduced, thereby avoiding huge time consumption.

[0073] According to the present invention, by making targeted improvements to existing models based on the characteristics of early-stage fires, the network's shallow and deep features are integrated, minimizing the size while maintaining model accuracy. Furthermore, the use of multi-threading technology effectively improves the algorithm's execution efficiency and stability. Consequently, this method can effectively detect early-stage fires with low latency and high accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] Figure 1 1 is a structural diagram showing modules used in an early fire detection method based on shallow-layer guided deep-layer feature fusion according to a specific embodiment of the present invention.

[0075] Figure 2 It is a simplified diagram showing the operation of the SGD module (Shallow Guide Deep module) used in the early fire detection method based on shallow-guided deep feature fusion involved in a specific embodiment of the present invention.

[0076] Figure 3 3 is a flow chart showing an early fire detection method based on shallow-layer guided deep-layer feature fusion according to a specific embodiment of the present invention. DETAILED DESCRIPTION

[0077] The present invention will be further described in detail below in combination with specific embodiments with reference to the accompanying drawings. Those skilled in the art will understand that the description is exemplary and the present invention is not limited to the specific embodiments.

[0078] Figure 1-3 A flowchart of an early fire detection method based on shallow-layer guided deep-layer feature fusion and a structural diagram of the modules used therein are shown according to a specific embodiment of the present invention.

[0079] like Figure 1As shown in the figure, after the video frames are max-pooled, they are fused by the ShallowGuide Deep module, then shuffled by n 2Stream Shuffle Modules. After convolution and upsampling, they enter the splicing operation, pass through the YOLOv5 network C3 module, and finally output to the YOLO detection head to obtain the detection results. This will be described in detail later.

[0080] like Figure 2 As shown, in one embodiment of the present invention, in the shallow-layer guided deep fusion module, video frame features are input, RGB features are converted to YCbCr, and Y channel features, Cb channel features, and Cr channel features are obtained. This data is then compressed, and the one-dimensional vector Y is activated using an activation function. The weights of each channel are obtained and then updated to obtain shallow features. This will be described in detail later.

[0081] like Figure 3 , which shows a flow chart of an early fire detection method based on shallow-layer guided deep-layer feature fusion according to a specific embodiment of the present invention.

[0082] In one embodiment of the present invention, the early fire detection method based on shallow-layer guided deep feature fusion includes four processes: reducing the feature map size (step 1), shallow feature fusion (step 2), deep feature fusion (step 3), and detecting output features (step 4).

[0083] After inputting the video frame, scaling and pooling operations are used to reduce the size of the input feature map.

[0084] Regarding step 1, the following steps are specifically included:

[0085] Step 101: Perform adaptive image scaling on the input image to unify the input image to a size of 640×640;

[0086] Step 102: Use a 2×2 convolution kernel to perform a maximum pooling operation with a stride of 2 on the processed image to reduce the image size to 320×320.

[0087] Regarding step 2), the scaled image features obtained in step 1) are fed into the SGD module for shallow feature fusion. Specifically, the following steps are included:

[0088] Step 201: Split the input image in RGB color space.

[0089] [R,G,B]=split(frame)

[0090] Where frame is the feature map obtained after processing in step 1), split(·) is the channel splitting function in OpenCV, which splits the image in the RGB color space into R, G, and B channels respectively.

[0091] Step 202: Convert the features of the R, G, and B channels into features in the YCbCr color space.

[0092]

[0093] Wherein, R, G, and B are channel features obtained after splitting in step 201, cuda() indicates the use of matrix multiplication operations on the GPU to increase the calculation speed, and Y, Cb, and Cr represent Y, Cb, and Cr channel features converted from R, G, and B channels.

[0094] Step 203: Compress the Y, Cb, and Cr channel features obtained in step 202.

[0095]

[0096] where X c =[X Y ,X Cb ,X Cr ], represents the features of Y, Cb, and Cr channels, and H and W represent the height and width of the feature map of each channel respectively.

[0097] Step 204: For Y obtained in step 203 c Perform activation processing to obtain the weights of each channel in the shallow features,

[0098] W=σ(FC(Relu(FC(Y c ))))

[0099] Where σ represents the sigmoid activation function, FC represents the fully connected layer, Relu represents the Relu activation function, and W represents the weight of each channel obtained after processing.

[0100] Step 205: Use the weight W obtained in step 204 to update the X representing the Y, Cb, and Cr channel features obtained in step 202.

[0101]

[0102] in represents channel-by-channel multiplication, Represents the fused shallow features obtained after weight update of the input image.

[0103] For step 3), for the fused shallow features obtained in step 2) Multiple 2Stream-ShuffleNet modules are used for deep feature fusion.

[0104] Step 301: For the input features Perform channel splitting and divide the features into two channels on average.

[0105]

[0106] Where i = 1, 2. Avg_split means splitting the input feature into two parts evenly.

[0107] Step 302: For the split features obtained in step 301 Perform a 1×1 convolution operation.

[0108]

[0109] Where i = 1, 2. Conv2D 1×1 Represents a 1×1 convolution operation.

[0110] Step 303: For the Perform channel scattering operation.

[0111]

[0112] Where i = 1, 2. Shuffle means the channel scattering operation.

[0113] Step 304: For the Perform a 3×3 convolution operation

[0114]

[0115] Where i = 1, 2. Conv2D 3×3 Represents a 3×3 convolution operation.

[0116] Step 305: For the Perform a 1×1 convolution operation.

[0117]

[0118] Where i = 1, 2. Conv2D 1×1 Represents a 1×1 convolution operation.

[0119] Step 306: For the result obtained in step 305 Perform concat splicing operation

[0120]

[0121] Step 307: For the Perform channel scattering operation

[0122]

[0123] Step 308: For the The features input in step 301 Perform addition operation

[0124]

[0125] Step 309: The result of step 308 As input, repeat the operations in step 301 to step 308 6 times. The result of the i-th operation is

[0126] Step 310: The deep features obtained in step 309 after six repeated operations are fused Perform convolution operation

[0127]

[0128] Step 311: For the Perform upsampling operation

[0129]

[0130] With respect to step 4), the fused deep features obtained in step 3) are processed to further extract the fused features.

[0131] Step 401: For the and the feature obtained by the third repetition of step 309 Perform concat operation.

[0132]

[0133] Step 402: For the Processing is performed using the C3 module in YOLO v5.

[0134]

[0135] Step 403: For the Perform convolution operation.

[0136]

[0137] Step 404: For the Perform upsampling operation.

[0138]

[0139] Step 405: For the The first operation obtained in step 309 Perform splicing operations.

[0140]

[0141] Step 406: Perform C3 operation on the spliced ​​features obtained in step 405.

[0142]

[0143] Through the above four steps, the depth features obtained in step 4) are further processed Send it to YOLO Head (detection head) to get the final recognition result.

[0144] The present invention has been described in detail above based on specific implementation methods. Those skilled in the art will understand that these are merely exemplary descriptions and that various changes and modifications may be made. As long as they do not depart from the purpose and spirit of the present invention, these changes and modifications should fall within the scope of the present invention. The scope of protection of the present invention is defined by the appended claims.

Claims

1. An early fire detection method based on shallow-layer guided deep-layer feature fusion, characterized in that: The following steps are involved: Step A: Use scaling and pooling operations to reduce the size of the input feature map; Step B: Use the SGD module to assign different weights to each shallow feature channel to provide guidance for the subsequent deep channel feature fusion; Step C: Use the backbone network to extract input features and fuse the channels in the deep features; Step D: Process the fused deep features and further extract the fused features; Step E: Process the output features using the detection head to obtain early fire detection results for the video frame, wherein step C specifically includes the following steps: Step C01: For the input features Perform channel splitting and divide the features into two channels on average; Where i=1, 2, Avg_split means splitting the input feature into two parts evenly; Step C02: For the split features obtained in step C01 Perform a 1×1 convolution operation; Where i = 1, 2, Conv2D 1×1 Represents a 1×1 convolution operation; Step C03: For the Perform channel scattering operation; Where i=1, 2, Shuffle represents the channel scattering operation, that is, the features of each channel are scattering; Step C04: For the Perform a 3×3 convolution operation: Where i = 1, 2, Conv2D 3×3 Represents a 3×3 convolution operation; Step C05: For the Perform a 1×1 convolution operation: Where i = 1, 2, Conv2D 1×1 Represents a 1×1 convolution operation; Step C06: For the result obtained in step C05 Perform concat splicing operation: Step C07: For the Perform channel scattering operation: Step C08: For the The features entered in step C01 Perform addition operation: Step C09: The result obtained in step C08 As input, repeat the operations in step C01 to step C08 6 times, where the result of the i-th operation is Step C10: The deep features obtained in step C09 after six repeated operations are fused Perform convolution operation: Step C11: For the Perform upsampling operation:

2. The method according to claim 1, wherein step A specifically comprises the following steps: Step A01: Perform adaptive image scaling on the input image to unify the input image to 640×640 size; Step A02: Use a 2×2 convolution kernel to perform a maximum pooling operation with a stride of 2 on the image processed in step A01 to reduce the image size to 320×320.

3. The method according to claim 1, wherein step B specifically comprises the following steps: Step B01: Split the input image in RGB color space. [R,G,B]=split(frame) Where frame is the feature map obtained after step 1), split(·) is the channel splitting function in OpenCV, which splits the image in the RGB color space into R, G, and B channels respectively; Step B02: Convert the features of the R, G, and B channels into features in the YCbCr color space; Where R, G, and B are the channel features obtained after splitting in step B01, cuda() indicates the use of matrix multiplication on the GPU to improve the calculation speed, and Y, Cb, and Cr represent the Y, Cb, and Cr channel features converted from the R, G, and B channels; Step B03: compress the Y, Cb, and Cr channel features obtained in step B02. where X c =[X Y ,X Cb ,X Cr ], represents the features of Y, Cb, and Cr channels, H and W represent the height and width of each channel feature map respectively, Y c =[Y Y ,Y Cb ,Y Cr ] is a one-dimensional vector, representing the calculated mean of Y, Cb, and Cr channels; Step B04: For Y obtained in step B03 c Perform activation processing to obtain the weights of each channel in the shallow features, W=σ(FC(Relu(FC(Y c )))) Where σ represents the sigmoid activation function, FC represents the fully connected layer, Relu represents the Relu activation function, and W represents the weight of each channel obtained after processing; Step B05: Use the weight W obtained in step B04 to update the X representing the Y, Cb, and Cr channel features obtained in step B02. in represents channel-by-channel multiplication, Represents the fused shallow features obtained after weight update of the input image.

4. The method according to claim 1, wherein step D specifically comprises the following steps: Step D01: For the and the feature obtained by repeating the operation for the third time in step C09 Perform concat splicing operation: Step D02: For the Processing is done using the C3 module in YOLO v5: Step D03: For the Perform convolution operation: Step D04: For the Perform upsampling operation: Step D05: For the The same as the first operation in step C09 Perform splicing operations: Step D06: Perform C3 operation on the concatenated features obtained in step D05 to obtain deep features:

5. The method according to claim 4, wherein step E specifically comprises the following steps: E01. Further process the depth features obtained in step D Send it to YOLO Head to get the final recognition result.

Images