Flame and smoke detection method based on multi-branch reparameterized heterogeneous kernel convolution

Abstract

The application discloses a flame and smoke detection method based on multi-branch reparameterization heterogeneous kernel convolution, which significantly improves the detection accuracy and real-time performance of fire targets in complex environments through innovative network architecture design. The method makes the following key improvements based on the YOLOv8 framework: first, a multi-scale feature fusion enhancement network is constructed, a double-path feature pyramid structure of shallow auxiliary fusion (SAF) and deep auxiliary fusion (DAF) is adopted, and the dynamic feature interaction mechanism is used to effectively retain the detail texture features of flame and smoke; second, the reparameterization heterogeneous convolution module (RepHCM) is innovatively designed, the multi-branch depth separable convolution parallel feature extraction in the training stage and the parameter reorganization in the inference stage are used to improve the calculation efficiency while maintaining the large receptive field advantage; third, the global heterogeneous kernel selection mechanism is deployed in the main stem and neck network, and different size convolution kernels are dynamically applied to strengthen the feature extraction capability of multi-scale fire targets; fourth, in view of the problem of small target recognition difficulty, a high-resolution detection head branch (160x160) is constructed to improve the detection effect of small-scale flame and smoke.

Description

Technical Field

[0001] This invention relates to the fields of image processing and computer vision technology, and in particular to a deep learning method suitable for flame and smoke detection in complex scenarios, belonging to the fields of intelligent video surveillance and smart fire protection technology. Background Technology

[0002] With the rapid advancement of urbanization, high-density building clusters are increasing, and the safety risks posed by frequent fires are becoming increasingly prominent. Traditional fire detection systems mainly rely on smoke sensors or temperature sensors. While these systems can detect fires to a certain extent, they generally suffer from problems such as response delays, complex deployment, and high false alarm and false negative rates, making it difficult to meet the efficiency and reliability requirements of modern fire early warning systems. In recent years, vision-based flame and smoke detection methods have gradually emerged, especially with the development of deep learning models, which has significantly improved the accuracy and real-time performance of target detection. However, due to the variable morphology, blurred boundaries, and varying scales of flames and smoke, existing models still have shortcomings in feature fusion, small target detection, and adaptability to complex backgrounds. Therefore, a new method with high detection accuracy, robustness, and high deployment efficiency is urgently needed. Summary of the Invention

[0003] This invention aims to provide a flame and smoke detection method based on multi-branch reparameterized heterogeneous kernel convolution, so as to improve the accuracy, robustness and real-time performance of target detection, and is particularly suitable for the detection of flame and smoke targets in fire scenarios.

[0004] This invention proposes an improved neural network structure, specifically including the following technical features:

[0005] a. Multi-branch assisted fusion feature pyramid network (FPN);

[0006] b. Reparameterized heterogeneous convolutional module RepHCM;

[0007] c. Global heterogeneous kernel selection mechanism;

[0008] d. Add a new detection head branch;

[0009] e. The overall network architecture is an improved YOLO series model (RHK-YOLO).

[0010] This invention aims to solve the following technical problems:

[0011] a. Information loss problem in the multi-scale feature fusion process;

[0012] b. Problems such as insufficient receptive field and poor adaptability to complex target shapes in the network structure;

[0013] c. Problems with low accuracy in small target recognition and inaccurate positioning;

[0014] d. While improving detection accuracy, maintain the model's lightweight nature and real-time performance.

[0015] To address the above problems, this invention proposes a flame and smoke detection method based on multi-branch reparameterized heterogeneous kernel convolution, the core technical solution of which includes:

[0016] A multi-branch assisted fusion FPN structure dynamically interacts with low-level high-resolution features and high-level strong semantic features; high-frequency details such as flame edge gradients and smoke textures are preserved through channel concatenation; and dynamic weight adjustment and efficient fusion of feature layers are achieved.

[0017] A reparameterized heterogeneous convolutional module. It utilizes a parallel multi-branch structure to extract multi-scale features during training; introduces cascaded information flow to achieve full information propagation; and merges parameter structures during the inference phase to improve speed.

[0018] A global heterogeneous kernel selection mechanism is implemented. Convolutional kernels of different sizes (3×3 to 9×9) are used at key layers of the network to enhance the network's ability to simultaneously perceive small-scale and large-scale targets, thereby ensuring detection stability and robustness.

[0019] A new 160×160 high-resolution detection head branch has been added. A dual-stream prediction architecture has been adopted to enhance the detection capability of small target flames and thin smoke; achieving coverage and fine prediction of multi-scale targets.

[0020] The beneficial effects of this invention are as follows:

[0021] Significantly improves the accuracy of flame and smoke detection (mAP@0.5 increased to 0.68);

[0022] Improve small target detection capabilities and robustness in complex scenarios;

[0023] Maintain low computational cost and support high frame rate real-time inference deployment;

[0024] The network structure is flexible, with good engineering portability and practicality. It can be integrated into edge devices such as smart cameras and drone inspection systems, and is suitable for a variety of smart fire protection scenarios. Attached Figure Description

[0025] Figure 1 Overall structural framework diagram of the flame detection network of this invention

[0026] Figure 2 Shallow Assisted Fusion (SAF) Module Structure Diagram

[0027] Figure 3 Deep Assisted Fusion (DAF) Module Structure Diagram

[0028] Figure 4 Reparameterized Heterogeneous Convolutional Module (RepHCM) architecture diagram

[0029] Figure 5 Schematic diagram of global heterogeneous kernel mechanism

[0030] Figure 6 Flame detection network scene recognition effect diagram; Detailed Implementation Plan

[0031] To facilitate understanding and implementation of the present invention by those skilled in the art, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0032] This invention provides a flame and smoke detection method based on multi-branch reparameterized heterogeneous kernel convolution. This method is based on an improved YOLOv8 architecture design, details of which can be found in [link to documentation]. Figure 1 It mainly includes the following modules: backbone network, neck network, and detection head. In each module, a multi-branch feature fusion mechanism, a reparameterized convolution module, and a global heterogeneous kernel selection mechanism are introduced to achieve high-precision detection of multi-scale flame and smoke targets in complex fire scenarios.

[0033] The overall network architecture includes an input layer, a backbone network, a neck network, and a multi-scale detection head. Among them: (1) the backbone network is responsible for extracting basic features of different scales of the feature map; (2) the neck network adopts a multi-branch assisted fusion FPN structure to realize context-aware fusion of deep multi-scale features and enhance feature expression capabilities; (3) a 160×160 high-resolution detection branch is added to the three basic detection heads, and the four detection branches are specifically for handling targets of different sizes.

[0034] Input image Basic semantic features are extracted from the backbone network to generate multi-scale feature layers B2, B3, B4, and B5, with spatial dimensions as follows:

[0035] The spatial dimensions representing the features extracted by the second layer of the network;

[0036] The spatial dimensions representing the features extracted by the third layer of the network;

[0037] The spatial dimensions representing the features extracted by the fourth layer of the network;

[0038] The spatial dimensions representing the features extracted by the fifth layer of the network;

[0039] This invention innovatively introduces a reparameterized heterogeneous convolution module (RepHCM) into the shallow convolutions of the improved YOLOv8 backbone structure, as detailed below. Figure 4 It consists of multiple parallel branches, each branch consisting of multiple 1×1 convolutions, depthwise separable convolutions of different sizes, activation functions and normalization layers, and has the ability to adjust the receptive field and adaptively capture features.

[0040] The input feature map is split into N sub-streams using a Split operation. Each sub-stream is processed by M convolutional units before outputting its result. Finally, these sub-streams are concatenated and merged to form a highly expressive feature map. This structure enables multi-scale joint learning during the training phase, and the parallel structures are reparameterized and merged during the inference phase without affecting the model's inference speed.

[0041] Its training phase structure is as follows:

[0042] For the input feature map X∈R H×W×C Using n heterogeneous convolutional kernels with sizes k1, k2, ..., k n Construct parallel paths to obtain intermediate features

[0043]

[0044] The final output feature Y is expressed as:

[0045]

[0046] During the inference phase, each convolutional path is merged into an equivalent convolutional kernel, reducing computational burden and ensuring inference efficiency.

[0047] In the neck network, an improved FPN structure is used to achieve multi-scale feature fusion. This module includes a shallow auxiliary fusion module (SAF) and a deep auxiliary fusion module (DAF).

[0048] The SAF module combines low-level spatial details with high-level semantic information through upsampling, downsampling, and channel concatenation operations, effectively preserving the edges and textures of small targets. This provides rich clues for detecting small targets and accurately locating their edges. See details... Figure 2 Its expression is as follows:

[0049] F n =concat(δ(GAP(B)) n-1 )),B n ,δ(Conv(U(B n+1 ))),U(F n+1 ))

[0050] The DAF module guides the integration of deep features with multi-scale semantic information through multi-directional connections and residual paths, enhancing its ability to represent large targets and complex backgrounds. This helps the network better understand large targets and their overall structure. See details. Figure 3 Its expression is as follows:

[0051] P n =concat(δ(Down(F)) n-1 )),δ(GAP(P n-1 )),F n ,δ(Conv(U(F n+1 ))))

[0052] The above modules enable the effective expression and fusion of different features such as the core region of the flame, the high-temperature profile, and the edge of the smoke diffusion in a multi-scale space.

[0053] The reparameterized heterogeneous convolutional fusion module runs through the backbone and neck network structure, see details. Figure 5 Its core idea is to use a parallel multi-core path structure to obtain rich feature representation capabilities during the training phase, and then fuse it into a single-core operation during the inference phase. The specific deployment method is as follows:

[0054] By employing multi-scale DWConv combinations, flexible inter-channel information flow is formed within the network;

[0055] The outputs of each branch are concatenated through channels and then uniformly mapped to the channel dimension using a 1×1 convolution.

[0056] It supports sharing of kernel weights across different levels and uses an attention mechanism for weighted selection.

[0057] The introduction of this module significantly improves the ability to perceive the boundaries and describe the shape of flame and smoke targets, especially for targets with blurred outlines and obvious scale changes.

[0058] To expand the effective receptive field and adapt to the needs of target detection at different scales, the method introduces a global heterogeneous kernel selection mechanism into the entire network structure, and the specific deployment method is as follows:

[0059] RepHCM was constructed by sequentially using 3×3, 5×5, 7×7, and 9×9 convolutional kernels at each stage of the backbone network;

[0060] In the feature map fusion stage of FPN, 3×3, 5×5, 7×7 and 9×9 convolutional kernels are used;

[0061] The output features of each heterogeneous kernel are spliced ​​and uniformly processed to achieve adaptive modeling of multi-scale scenes.

[0062] Feature map X per layer i After convolution kernel ki The processed output is:

[0063]

[0064] After being concatenated during the feature fusion stage, the output features are unified in channel dimension through a 1×1 convolution, and the contribution weights of each convolution kernel size are dynamically adjusted through an attention mechanism.

[0065]

[0066] The above mechanism, combined with an attention strategy, adjusts the weights of kernel outputs of different sizes in feature fusion, thereby improving the robustness and detection stability of the system in scenarios such as sparse smoke, localized flames, and dense backgrounds.

[0067] To enhance the detection capabilities for small-scale flames and thin smoke, this invention adds a high-resolution detection head (160×160) to the existing three YOLO detection heads (80×80, 40×40, and 20×20). This detection head employs a dual-stream prediction architecture, specifically designed as follows:

[0068] Input feature map F 160 Channel reduction is performed using 1×1 convolution:

[0069] F cls =Conv 1×1 (F 160 )

[0070] F reg =Conv 1×1 (F 160 )

[0071] Where F cls F is used to output the class probability in the classification branch. reg Used for bounding box regression output location information. The prediction results of this detection head, along with the output of the original detection head, are processed together with NMS in the post-processing stage to eliminate redundant detection boxes and ensure the accuracy of the detection results.

[0072] This system uses the PyTorch deep learning framework during training, with an input image size of 640×640 and 300 training epochs. Supervised learning is performed using a self-built flame-smoke dataset. For training results and examples, please refer to [link to example training performance]. Figure 6 The loss function includes: classification loss L cls Location regression loss L box Confidence loss L obj The overall loss function is:

[0073] L=λ1L cls +λ2L box +λ3Lobj

[0074] λ1, λ2, and λ3 are adjustable weight parameters that are configured according to the task weights.

[0075] The specific implementation steps are as follows:

[0076] This invention uses a self-built flame-smoke dataset for supervised learning, and collects image data from multi-source fire scenarios (including indoor fires, forest fires, industrial fires, etc.) to ensure data diversity;

[0077] The image is rotated (±30°), scaled (0.8-1.2 times), horizontally flipped, and its color perturbed (brightness, contrast, saturation adjusted) to improve the model's generalization ability.

[0078] The network structure of this invention is used to train the dataset, and pre-trained weights are used to initialize the model parameters to accelerate convergence; the AdamW optimizer is used in combination with mixed precision training; the loss function includes classification loss, confidence loss and bounding box regression loss.

[0079] The trained model is deployed to embedded devices or monitoring platforms using the TensorRT acceleration engine to achieve real-time fire monitoring. This invention's method can be widely applied to various scenarios such as intelligent video surveillance, smart cities, forest fire prevention, and industrial safety, exhibiting good detection accuracy, robustness, and real-time performance.

Claims

1. A flame and smoke detection method based on multi-branch reparameterized heterogeneous kernel convolution, characterized in that, Includes the following steps: S1. Construct a neural network structure containing a multi-branch assisted fusion (FPN); wherein the multi-branch assisted fusion (FPN) includes a shallow assisted fusion (SAF) module and a deep assisted fusion (DAF) module, and the output expression of the SAF module is: Where δ represents the SiLU function, Conv represents a 1×1 convolution controlling the number of channels, U(•) represents an upsampling operation, and GAP represents global average pooling; the SAF module fuses shallow and deep features through upsampling and downsampling operations, preserving local details of the flame target; the output expression of the DAF module is: Where Down represents a 3×3 downsampling convolution with batch normalization layer. The DAF module achieves multi-scale feature fusion and enhancement by concatenating detailed information from shallow downsampling, contextual information from upper-layer global pooling, original information from the current layer, and semantic information from deep upsampling, thereby enhancing the detection capability of large-scale flame targets; S2. Introduce a reparameterized heterogeneous convolution module RepHCM in the backbone network and neck network; wherein, the specific process of the reparameterized heterogeneous convolution module RepHCM is as follows: the input features are generated into multiple information streams through 1×1 convolution and Split operation; each information stream extracts features sequentially through concatenated convolutional blocks and retains the output of each branch; the outputs of each branch are concatenated and integrated through 1×1 convolution to obtain the final feature representation; S3. Employ a global heterogeneous kernel selection mechanism to dynamically adjust the convolution kernel size; S4. Construct a high-resolution detection head at the output layer; S5. Perform feature extraction, fusion, and prediction on the input image to detect flame and smoke targets.

2. The method according to claim 1, characterized in that, The convolutional block described in step S2 consists of three 1×1 convolutional layers and depthwise separable convolutional (DWConv) layers arranged alternately, and multi-branch parallel convolution is used during training and merged into a single convolution during inference.

3. The method according to claim 1, characterized in that, In step S3, the global heterogeneous kernel selection mechanism uses 3×3, 5×5, 7×7, and 9×9 convolutional kernel combinations in the backbone network and neck network respectively to adapt to the feature fusion requirements at different resolutions.

4. The method according to claim 1, characterized in that, In step S4, the structure of the high-resolution detection head is consistent with that of the original detection head of the YOLO network. The input is a feature map of size 160×160 from the output of the multi-branch FPN. After dimensionality reduction by 1×1 convolution, it is fed into the classification branch and the bounding box regression branch respectively. The classification branch and the regression branch output the target's class probability and bounding box information.

5. The method according to claim 1, characterized in that, In step S5, feature extraction, fusion, and prediction are performed on the input image, and the detection of flame and smoke targets is achieved by using the target category probability and bounding box information.

Images