Lightweight adaptive fire smoke flame detection method

Abstract

The application discloses a kind of lightweight adaptive fire smoke flame detection methods, comprising the following steps: (1) based on known fire image video data, construct fire image dataset, (2) construct the lightweight adaptive fire smoke flame detection neural network combined with CNN of Transform, including several depth separable convolution modules, deformable adaptive convolution and Transform combination module and deformable adaptive convolution detection module, utilize fire image dataset to train neural network, obtain lightweight adaptive fire smoke flame detection model, (3) apply lightweight adaptive fire smoke flame detection model, when fire occurs, the target object position in fire smoke and flame is identified.Compared with mainstream fire detection model, the application achieves good results in training, verification and testing, and the model is relatively lightweight, achieving a balance between detection speed and accuracy.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence detection, specifically relating to a lightweight adaptive fire smoke and flame detection method. Background Technology

[0002] Real-time monitoring of fire-prone areas and timely alarms are crucial for protecting people's lives and property and ensuring industrial safety. With rapid socio-economic development, the emergence of new business models such as new energy vehicles, live streaming, escape rooms, and postpartum care centers has made fire situations increasingly complex, frequent, and damaging. Traditional fire monitoring technologies primarily rely on contact-based fire detectors, such as carbon monoxide, smoke, and carbon dioxide sensors. However, these detectors are difficult to apply to large indoor spaces where open flames are strictly prohibited, as well as open outdoor spaces. Compared to contact-based detectors, non-contact video fire monitoring, with the rapid development of deep learning, offers significant advantages in monitoring fire-prone areas in large indoor spaces and open outdoor spaces, greatly reducing losses caused by delayed alarms in these areas.

[0003] The Transformer, following the emergence of CNNs, is a new and high-performing neural network architecture. It exhibits strong robustness and generalization ability in handling perturbations and occlusions in fire images. However, Transformer models are typically large and not lightweight, making them difficult to deploy on edge devices. Furthermore, the lack of inductive bias in CNNs means that Transformers require large amounts of data to drive them. CNNs are not perfect either; the spatially acquired features are localized, which limits their performance. In contrast, Transformers can acquire global information through self-attention mechanisms. Therefore, it is worthwhile to explore how to combine Transformers and CNNs to design a lightweight fire detection method. Summary of the Invention

[0004] To address the problems in existing technologies, this invention proposes a lightweight adaptive fire smoke and flame detection method that not only supports all-weather fire smoke and flame detection, but can also be deployed and applied on smart devices at the fire edge.

[0005] To address the above technical problems, this invention provides the following technical solution: a lightweight adaptive fire smoke and flame detection method, comprising the following steps:

[0006] S1. Based on known fire image and video data, construct a fire image dataset, including several natural light fire images, infrared light fire images, and various preset attribute labels of the target objects in the images.

[0007] S2. Construct a lightweight adaptive fire smoke and flame detection neural network combining Transformer and CNN, including a backbone network and a detection network connected to the backbone network. The backbone network includes: a convolutional module and several depthwise separable convolutional modules connected to it, and a deformable adaptive convolutional and Transformer combined module. The detection network includes several deformable adaptive convolutional detection modules. The deformable adaptive convolutional and Transformer combined module is connected to the detection network as the output of the backbone network. The deformable adaptive convolutional detection module is the output of the lightweight adaptive fire smoke and flame detection neural network.

[0008] S3. Using the fire image dataset from step S1, train the lightweight adaptive fire smoke and flame detection neural network combining Transformer and CNN constructed in step S2 to obtain a lightweight adaptive fire smoke and flame detection model. Specifically, take several natural light fire images and infrared fire images from the fire image dataset as input, and the corresponding locations of smoke and flame targets in the fire as output, and train the neural network. During the training process, the features of the input images are dynamically extracted, and multiple prediction boxes are given for the smoke and flame regions in the fire. The best target boxes that meet the preset confidence level of the fire smoke and flame prediction boxes are selected by the scoring and non-maximum suppression methods to obtain the optimal locations of smoke and flame targets in the fire.

[0009] S4. Apply a lightweight adaptive fire smoke and flame detection model. When a fire occurs, input real-time acquired fire images to identify the location of target objects in the fire smoke and flames.

[0010] Furthermore, the aforementioned step S1 includes the following sub-steps:

[0011] S101. The known fire images and videos are processed into images by slicing them into frames, and then the images are rotated, mirrored and affineed with a predetermined probability to form a fire image dataset.

[0012] S102. Use the LabelImg rectangular box sample annotation tool to annotate the images in the fire image dataset to obtain the center point, width, height and category name of the target objects in the images.

[0013] Further, in the aforementioned step S2, the backbone network consists of: a first 3×3 convolution, a first depthwise separable convolution module, a second depthwise separable convolution module, a third depthwise separable convolution module, a fourth depthwise separable convolution module, a fifth depthwise separable convolution module, a first deformable adaptive convolution combined with Transformer module, a sixth depthwise separable convolution module, a second deformable adaptive convolution combined with Transformer module, a seventh depthwise separable convolution module, and a third deformable adaptive convolution combined with Transformer module, connected sequentially.

[0014] The detection network consists of a first deformable adaptive convolution detection module, a second deformable adaptive convolution detection module, and a third deformable adaptive convolution detection module connected in sequence.

[0015] The first deformable adaptive convolution combined with Transformer module serves as the first output of the backbone network and is connected to the first deformable adaptive convolution detection module; the second deformable adaptive convolution combined with Transformer module serves as the second output of the backbone network and is connected to the second deformable adaptive convolution detection module; and the third deformable adaptive convolution combined with Transformer module serves as the third output of the backbone network and is connected to the third deformable adaptive convolution detection module.

[0016] Furthermore, in the aforementioned step S2, the depthwise separable convolution module includes a first 1×1 convolution, a layer-by-layer convolution, a point-by-point convolution, and a second 1×1 convolution connected in sequence. The first 1×1 convolution serves as the input of the depthwise separable convolution module, and the second 1×1 convolution serves as the output of the depthwise separable convolution module.

[0017] 5. A lightweight adaptive fire smoke and flame detection method according to claim 3, characterized in that, in step S2, the deformable adaptive convolution and Transformer combined module includes a first deformable adaptive convolution, a third 1×1 convolution, an unfolding operation module, a Transformer module, a folding operation module, a fourth 1×1 convolution, and a third deformable adaptive convolution connected in sequence; it also includes a second deformable adaptive convolution, the second deformable adaptive convolution and the first deformable adaptive convolution simultaneously serving as the input end of the deformable adaptive convolution and Transformer combined module; the output end of the second deformable adaptive convolution is connected to the input end of the third deformable adaptive convolution, the third deformable adaptive convolution serving as the output end of the deformable adaptive convolution and Transformer combined module.

[0018] Furthermore, in the aforementioned step S2, the deformable adaptive convolution detection module includes: a bounding box detection module and a classification loss detection module;

[0019] The bounding box detection module includes: a fourth deformable adaptive convolution, a second 3×3 convolution, a third 3×3 convolution, a fifth 1×1 convolution, and a bounding box regression loss detection unit connected in sequence;

[0020] The classification loss detection module includes: a fifth deformable adaptive convolution, a fourth 3×3 convolution, a fifth 3×3 convolution, a sixth 1×1 convolution, and a classification loss detection unit connected in sequence;

[0021] The fourth and fifth deformable adaptive convolutions serve as inputs to the deformable adaptive convolution detection module; the bounding box regression loss detection unit and the classification loss detection unit serve as outputs to the deformable adaptive convolution detection module.

[0022] Furthermore, in step S3 above, when training the lightweight adaptive fire smoke and flame detection neural network combining Transformer and CNN constructed in step S2, downsampling is performed using the depthwise separable convolution module in the network, and convolution calculations are performed on the irregularly changing smoke and flame features under fire conditions using the deformable adaptive convolution and Transformer combination module. Specifically:

[0023]

[0024] Where y(·) refers to the output result on the feature map after the convolution operation; P out P refers to the position of the output feature after convolution operation. n P refers to the location of the pixel in the receptive field sampled by the K×K convolution. n ={P ij}(0≤i,j≤K-1;0≤n≤K 2 -1; i,j,n∈N); ω(·) represents the value learned by the network in the convolution; x(·) represents the value after gridding on the input feature map; △P n For point P on the feature map n Position offset; Δγ n This represents the weighting coefficient at each sampling point.

[0025] Furthermore, in the aforementioned lightweight adaptive fire smoke and flame detection method, the unfolding module, Transformer module, and folding module, which are sequentially connected in the deformable adaptive convolution and Transformer combination module, perform unfolding-Transformer-folding operations on the input of the third 1×1 convolution, as shown in the following equation:

[0026] X G (p) = Transformer(X) U(p)),1 <p<P (2)

[0027] Where p represents the position of each pixel in a Patch(P) during the unfolding operation, and the width and height of a P are w and h, respectively; X U (·) indicates the expansion operation. The feature map represents the height and width of the input, P = w·h. X is the number of P; G (·) represents a folding operation.

[0028] Furthermore, in the aforementioned lightweight adaptive fire smoke and flame detection method, the bounding box regression loss detection unit utilizes CIoU loss and DFL loss, while the classification loss detection unit utilizes BCE loss; the CIoU loss formula is as follows:

[0029]

[0030] in:

[0031]

[0032] B、B t These are the predicted bounding box and the ground truth bounding box, b and b', respectively. t ρ is the center point position of the predicted box and the ground truth box, respectively; c is the diagonal distance of the smallest bounding box formed by the predicted box and the ground truth box; ρ is the Euclidean distance between the center points of the predicted box and the ground truth box; α and v are the aspect ratios, respectively; w and h are the width and height of the predicted box, respectively. t h t These are the width and height of the actual bounding box, respectively;

[0033] The DFL loss formula is as follows:

[0034] DFL(S i ,S i+1 )=-((y i+1 -y)log(S i )+(yy i )log(S i+1 (5)

[0035] DFL optimizes the left (y) element that is closest to the label y in a manner similar to cross-entropy. i )right(y i+1 The probabilities of the two positions are used to determine the distribution of the neighborhood where the network can quickly focus on the target position. S is the output distribution.

[0036] The BCE loss formula is as follows:

[0037]

[0038] Where LN represents the number of samples, PT Value T represents the weight value. Value P represents the label value. Value This represents the model's predicted value.

[0039] Furthermore, in step S3 above, when training the lightweight adaptive fire smoke and flame detection neural network combining Transformer and CNN, an improved dynamic adaptive activation function is used to alleviate gradient vanishing in the model, accelerate the convergence speed of network model training, and improve network performance, as shown in the following formula:

[0040]

[0041] Where x represents the input and β is a learnable, dynamically adaptive parameter.

[0042] Compared with the prior art, the beneficial technical effects of the present invention using the above technical solution are as follows:

[0043] Compared with existing technologies, the lightweight adaptive fire smoke and flame detection method of this invention has achieved good results in training, validation, and testing compared with mainstream fire detection methods, and the model is relatively lightweight, achieving a balance between detection speed and accuracy.

[0044] ① The backbone network is constructed using deformable adaptive convolution and Transformer. Compared with ordinary convolution, deformable adaptive convolution can better adapt to the irregular changes of smoke and flames in fire scenarios, while the use of Transformer can better focus on global information and improve the network's detection accuracy of fire smoke and flames;

[0045] ② An improved detection head using deformable adaptive convolution further enhances the network model's adaptability to detecting irregularly changing fire smoke and flames;

[0046] ③ An improved dynamic adaptive activation function was used, which can better alleviate gradient vanishing in the model, further accelerate the convergence speed of network model training, and improve network performance.

[0047] The trained model can be deployed to edge intelligent devices to perform more efficient and accurate fire detection in fire-prone areas where open flames are strictly prohibited, reducing losses caused by delayed alarms and giving us a better grasp of the initiative in fire alarms. Attached Figure Description

[0048] Figure 1 This is a flowchart of the present invention.

[0049] Figure 2The diagrams show ordinary convolution and deformable adaptive convolution combined with Transformer. In the diagram, (I) shows the convolution calculation of the input feature map by ordinary convolution, and (II) shows the convolution calculation of the input feature map by combining deformable adaptive convolution with Transformer.

[0050] Figure 3 This is a diagram illustrating the unfolding-folding operation.

[0051] Figure 4 This is a structural diagram of a depthwise separable convolutional module.

[0052] Figure 5 This is a structural diagram of a module combining deformable adaptive convolution and Transformer.

[0053] Figure 6 This is a structural diagram of the deformable adaptive convolution detection module.

[0054] Figure 7 This is a diagram of the overall structure of a lightweight adaptive fire smoke and flame detection network that combines Transformer and CNN.

[0055] Figure 8 Fire detection network deployment and fire detection flowchart. Detailed Implementation

[0056] To better understand the technical content of the present invention, specific embodiments are described below in conjunction with the accompanying drawings.

[0057] In this invention, various aspects of the invention are described with reference to the accompanying drawings, in which numerous illustrative embodiments are shown. Embodiments of the invention are not limited to those depicted in the drawings. It should be understood that the invention is implemented through any of the various concepts and embodiments described above, as well as the concepts and embodiments described in detail below, because the concepts and embodiments disclosed herein are not limited to any particular implementation. Furthermore, some aspects of the invention disclosed may be used alone or in any suitable combination with other aspects of the invention disclosed.

[0058] refer to Figure 1 This invention provides a lightweight adaptive fire smoke and flame detection method, comprising the following steps:

[0059] S1. Based on known fire image and video data, construct a fire image dataset, including several natural light fire images, infrared light fire images, and pre-defined attribute labels for the objects in the images; including the following sub-steps:

[0060] S101. Real-world fire scene natural light video data acquired using both ordinary cameras and infrared structured light cameras is processed into image format by frame segmentation. The acquired video data is then processed into fire images by frame segmentation at 0.1s. The fire images are then subjected to rotation-mirror-affine transformation operations with a 50% probability: counterclockwise rotation of 20°, 30°, 40°, 45°, 50°, 60°, 70°, and 90°, horizontal flipping, vertical flipping, horizontal and vertical flipping, and affine transformation to obtain the final fire image dataset.

[0061] S102. Use the rectangular image annotation tool LabelImg to annotate the fire smoke and flame data processed in (1), obtain the center point, width, height and category name of the target objects in the image, and save it in txt file format. Divide it into training set, validation set and test set in a ratio of 7:2:1.

[0062] S2, Reference Figure 7 A lightweight adaptive fire smoke and flame detection neural network combining Transformer and CNN is constructed, including a backbone network and a detection network connected to the backbone network. The backbone network includes: a first 3×3 convolution, and several depthwise separable convolutional modules connected to it, and a deformable adaptive convolutional and Transformer combined module. The detection network includes several deformable adaptive convolutional detection modules. The deformable adaptive convolutional and Transformer combined module is connected to the detection network as the output of the backbone network. The deformable adaptive convolutional detection module is the output of the lightweight adaptive fire smoke and flame detection neural network.

[0063] The backbone network consists of the following modules connected in sequence: a convolutional module, a first depthwise separable convolutional module, a second depthwise separable convolutional module, a third depthwise separable convolutional module, a fourth depthwise separable convolutional module, a fifth depthwise separable convolutional module, a first deformable adaptive convolutional module combined with Transformer, a sixth depthwise separable convolutional module, a second deformable adaptive convolutional module combined with Transformer, a seventh depthwise separable convolutional module, and a third deformable adaptive convolutional module combined with Transformer.

[0064] The detection network consists of a first deformable adaptive convolution detection module, a second deformable adaptive convolution detection module, and a third deformable adaptive convolution detection module connected in sequence.

[0065] The first deformable adaptive convolution combined with Transformer module serves as the first output of the backbone network and is connected to the first deformable adaptive convolution detection module; the second deformable adaptive convolution combined with Transformer module serves as the second output of the backbone network and is connected to the second deformable adaptive convolution detection module; and the third deformable adaptive convolution combined with Transformer module serves as the third output of the backbone network and is connected to the third deformable adaptive convolution detection module.

[0066] Combination Figure 4 The depthwise separable convolution module includes a first 1×1 convolution, a layer-by-layer convolution, a point-by-point convolution, and a second 1×1 convolution connected in sequence. The first 1×1 convolution serves as the input of the depthwise separable convolution module, and the second 1×1 convolution serves as the output of the depthwise separable convolution module.

[0067] refer to Figure 5 The deformable adaptive convolution combined with Transformer module includes a first deformable adaptive convolution, a third 1×1 convolution, an unfolding operation module, a Transformer module, a folding operation module, a fourth 1×1 convolution, and a third deformable adaptive convolution connected in sequence. It also includes a second deformable adaptive convolution, which, along with the first, serves as the input to the module. The output of the second deformable adaptive convolution is connected to the input of the third deformable adaptive convolution, which serves as the output of the module. An adaptive activation function, AdaptGELU, is used in the Transformer to accelerate network convergence and improve detection accuracy.

[0068] refer to Figure 6 The deformable adaptive convolutional detection module includes: a bounding box detection module and a classification loss detection module; the deformable adaptive convolutional detection module can better adapt to the irregular changes of smoke and flames in fire scenarios, wherein the bounding box detection module includes: a fourth deformable adaptive convolution, a second 3×3 convolution, a third 3×3 convolution, a fifth 1×1 convolution, and a bounding box regression loss detection unit connected in sequence.

[0069] The classification loss detection module includes: a fifth deformable adaptive convolution, a fourth 3×3 convolution, a fifth 3×3 convolution, a sixth 1×1 convolution, and a classification loss detection unit connected in sequence;

[0070] The fourth and fifth deformable adaptive convolutions serve as inputs to the deformable adaptive convolution detection module; the bounding box regression loss detection unit and the classification loss detection unit serve as outputs to the deformable adaptive convolution detection module.

[0071] S3. Using the fire image dataset from step S1, train the lightweight adaptive fire smoke and flame detection neural network combining Transformer and CNN constructed in step S2 to obtain a lightweight adaptive fire smoke and flame detection model. Specifically, take several natural light fire images and infrared fire images from the fire image dataset as input, and the corresponding locations of smoke and flame targets in the fire as output, and train the neural network. During training, features of the input images are dynamically extracted, and multiple prediction boxes are given for the smoke and flame regions in the fire. The best target boxes that meet the pre-set confidence level for fire smoke and flame prediction boxes are selected by using a scoring and non-maximum suppression method to obtain the optimal locations of smoke and flame targets in the fire. No pre-trained weights are loaded before network training. Input RGB images with a size of (640, 640), SGD optimizer, initial learning rate of 0.02, weight decay of 0.0005, SGD momentum of 0.937, 8 images per batch, maximum number of images loaded of 4, and a total of 50 training iterations. During the training process, the model is validated using the validation set in dataset (2) for each training round. Finally, the minimum loss model parameters obtained from the training and validation are used as the prediction weights of the final fire detection model.

[0072] like Figure 2 As shown, Figure 2 Image (I) shows the convolution calculation of the input feature map using ordinary convolution, and image (II) shows the convolution calculation of the input feature map using a combination of deformable adaptive convolution and Transformer. The combination of deformable adaptive convolution and Transformer is used to perform convolution calculations on the irregularly changing smoke and flame features in a fire scenario, specifically as follows:

[0073]

[0074] Where y(·) refers to the output result on the feature map after the convolution operation; P out P refers to the position of the output feature after convolution operation. n P refers to the location of the pixel in the receptive field sampled by the K×K convolution. n ={P ij}(0≤i,j≤K-1;0≤n≤K 2 -1; i,j,n∈N); ω(•) represents the value learned by the network in the convolution; x(·) represents the value after gridding on the input feature map; △P n For point P on the feature map n Position offset; Δγ n This represents the weighting coefficient at each sampling point.

[0075] like Figure 3 As shown, the unfolding module, Transformer module, and folding module, which are connected sequentially in the deformable adaptive convolution and Transformer combination module, perform unfolding-Transformer-folding operations on the input of the third 1×1 convolution, as follows:

[0076] X G (p) = Transformer(X) U (p)),1 <p<P (2)

[0077] Where p represents Figure 3 The leftmost and top-left corner of the Patch(P) is outlined with a dark black box, showing the position of each pixel. The width and height of a P are w and h, respectively; X U (·) indicates the expansion operation. The feature map represents the height (H) and width (W) of the input, P = w·h. X is the number of P; G (·) represents a folding operation.

[0078] The deformable convolutional detection module in the detection network is decoupled. The bounding box regression loss detection unit utilizes CIoU loss and DFL loss, while the classification loss detection unit utilizes BCE loss. The CIoU loss formula is as follows:

[0079]

[0080] in:

[0081]

[0082] B、B t These are the predicted bounding box and the ground truth bounding box, b and b', respectively. t ρ is the center point position of the predicted box and the ground truth box, respectively; c is the diagonal distance of the smallest bounding box formed by the predicted box and the ground truth box; ρ is the Euclidean distance between the center points of the predicted box and the ground truth box; α and v are the aspect ratios, respectively; w and h are the width and height of the predicted box, respectively. t h t These are the width and height of the actual bounding box, respectively;

[0083] The DFL loss formula is as follows:

[0084] DFL(S i ,S i+1 )=-((y i+1 -y)log(S i )+(yy i )log(S i+1(5)

[0085] DFL optimizes the left (y) element that is closest to the label y in a manner similar to cross-entropy. i )right(y i+1 The probabilities of the two positions are used to determine the distribution of the neighborhood where the network can quickly focus on the target position. S is the output distribution.

[0086] The BCE loss formula is as follows:

[0087]

[0088] Where LN represents the number of samples, PT Value T represents the weight value. Value P represents the label value. Value This represents the model's predicted value.

[0089] When training a lightweight adaptive fire smoke and flame detection neural network combining Transformer and CNN, an improved dynamic adaptive activation function is used to alleviate gradient vanishing in the model, accelerate the convergence speed of network model training, and improve network performance. The specific formula is as follows:

[0090]

[0091] Where x represents the input and β is a learnable, dynamically adaptive parameter.

[0092] A lightweight adaptive fire smoke and flame detection model combining Transformer and CNN, trained using this architecture, is deployed to an edge-based intelligent fire monitoring embedded device. The deployment process is as follows: The lightweight adaptive fire smoke and flame detection network model, trained using Transformer and CNN in the PyTorch framework, is exported to ONNX format. The ONNX format model is then input into TensorRT with specified optimization parameters and INT8 quantization to obtain the TensorRT Engine. Finally, the TensorRT Engine is used for inference and detection of smoke and flames, obtaining bounding boxes, categories, and confidence scores for smoke and flames at the time of a fire. Figure 8 As shown.

[0093] S4. A lightweight adaptive fire smoke and flame detection model is applied. When a fire occurs, the trained Transformer and CNN-based lightweight adaptive fire smoke and flame detection model is deployed to an embedded edge intelligent device for fire smoke and flame detection. This edge intelligent device performs real-time fire detection in fire-prone areas where open flames are strictly prohibited, inputting real-time acquired fire images and identifying the locations of target objects in the fire smoke and flames. The edge intelligent embedded fire monitoring device, with the deployed Transformer and CNN-based lightweight adaptive fire smoke and flame detection model, performs fire monitoring in large indoor spaces or open outdoor spaces where open flames are strictly prohibited. When a fire occurs, the fire monitoring video captured under monitoring is processed frame by frame into 640×640 images, which are then input into the fire detection model. By using the 3×3 convolution of the fire detection model and modules 1 and 2 at the bottom layer of the network, three effective fire feature layers of 64×64, 80×80, and 96×96 are obtained. These three effective feature layers are then input into module 3 to detect fire smoke and flames in fire scenarios. The bounding boxes, categories, and confidence scores of fire smoke and flames are visualized. The embedded fire monitoring device issues an alarm, reducing losses caused by delayed fire alarms and allowing for better control over fire alarms.

[0094] While the present invention has been described above with reference to preferred embodiments, it is not intended to limit the invention. Those skilled in the art can make various modifications and refinements without departing from the spirit and scope of the invention. Therefore, the scope of protection of the present invention shall be determined by the claims.

Claims

1. A lightweight adaptive fire smoke and flame detection method, characterized in that, Includes the following steps: S1. Based on known fire image and video data, construct a fire image dataset, including several natural light fire images, infrared light fire images, and various preset attribute labels of the target objects in the images. S2. Construct a lightweight adaptive fire smoke and flame detection neural network combining Transformer and CNN, including a backbone network and a detection network connected to the backbone network. The backbone network includes: a convolutional module and several depthwise separable convolutional modules connected to it, and a deformable adaptive convolutional and Transformer combined module. The detection network includes several deformable adaptive convolutional detection modules. The deformable adaptive convolutional and Transformer combined module is connected to the detection network as the output of the backbone network. The deformable adaptive convolutional detection module is the output of the lightweight adaptive fire smoke and flame detection neural network. S3. Using the fire image dataset from step S1, train the lightweight adaptive fire smoke and flame detection neural network combining Transformer and CNN constructed in step S2 to obtain a lightweight adaptive fire smoke and flame detection model. Specifically, take several natural light fire images and infrared fire images from the fire image dataset as input, and the corresponding locations of smoke and flame targets in the fire as output, and train the neural network. During the training process, the features of the input images are dynamically extracted, and multiple prediction boxes are given for the smoke and flame regions in the fire. The best target boxes that meet the preset confidence level of the fire smoke and flame prediction boxes are selected by the scoring and non-maximum suppression methods to obtain the optimal locations of smoke and flame targets in the fire. S4. Apply a lightweight adaptive fire smoke and flame detection model. When a fire occurs, input real-time acquired fire images to identify the location of target objects in the fire smoke and flames.

2. The lightweight adaptive fire smoke and flame detection method according to claim 1, characterized in that, Step S1 includes the following sub-steps: S101. The known fire images and videos are processed into images by slicing them into frames, and then the images are rotated, mirrored and affineed with a predetermined probability to form a fire image dataset. S102. Use the LabelImg rectangular box sample annotation tool to annotate the images in the fire image dataset to obtain the center point, width, height and category name of the target objects in the images.

3. The lightweight adaptive fire smoke and flame detection method according to claim 1, characterized in that, In step S2, the backbone network consists of: a first 3×3 convolution, a first depthwise separable convolution module, a second depthwise separable convolution module, a third depthwise separable convolution module, a fourth depthwise separable convolution module, a fifth depthwise separable convolution module, a first deformable adaptive convolution combined with Transformer module, a sixth depthwise separable convolution module, a second deformable adaptive convolution combined with Transformer module, a seventh depthwise separable convolution module, and a third deformable adaptive convolution combined with Transformer module, connected sequentially. The detection network consists of a first deformable adaptive convolution detection module, a second deformable adaptive convolution detection module, and a third deformable adaptive convolution detection module connected in sequence. The first deformable adaptive convolution combined with Transformer module serves as the first output of the backbone network and is connected to the first deformable adaptive convolution detection module; the second deformable adaptive convolution combined with Transformer module serves as the second output of the backbone network and is connected to the second deformable adaptive convolution detection module; and the third deformable adaptive convolution combined with Transformer module serves as the third output of the backbone network and is connected to the third deformable adaptive convolution detection module.

4. The lightweight adaptive fire smoke and flame detection method according to claim 3, characterized in that, In step S2, the depthwise separable convolution module includes a first 1×1 convolution, a layer-by-layer convolution, a point-by-point convolution, and a second 1×1 convolution connected in sequence. The first 1×1 convolution serves as the input of the depthwise separable convolution module, and the second 1×1 convolution serves as the output of the depthwise separable convolution module.

5. A lightweight adaptive fire smoke and flame detection method according to claim 3, characterized in that, In step S2, the deformable adaptive convolution and Transformer combined module includes a first deformable adaptive convolution, a third 1×1 convolution, an unfolding operation module, a Transformer module, a folding operation module, a fourth 1×1 convolution, and a third deformable adaptive convolution connected in sequence; it also includes a second deformable adaptive convolution, which and the first deformable adaptive convolution serve as inputs to the deformable adaptive convolution and Transformer combined module; the output of the second deformable adaptive convolution is connected to the input of the third deformable adaptive convolution, which serves as the output of the deformable adaptive convolution and Transformer combined module.

6. The lightweight adaptive fire smoke and flame detection method according to claim 3, characterized in that, In step S2, the deformable adaptive convolution detection module includes: a bounding box detection module and a classification loss detection module; The bounding box detection module includes: a fourth deformable adaptive convolution, a second 3×3 convolution, a third 3×3 convolution, a fifth 1×1 convolution, and a bounding box regression loss detection unit connected in sequence; The classification loss detection module includes: a fifth deformable adaptive convolution, a fourth 3×3 convolution, a fifth 3×3 convolution, a sixth 1×1 convolution, and a classification loss detection unit connected in sequence; The fourth and fifth deformable adaptive convolutions serve as inputs to the deformable adaptive convolution detection module; the bounding box regression loss detection unit and the classification loss detection unit serve as outputs to the deformable adaptive convolution detection module.

7. The lightweight adaptive fire smoke and flame detection method according to claim 1, characterized in that, In step S3, when training the lightweight adaptive fire smoke and flame detection neural network combining Transformer and CNN constructed in step S2, downsampling is performed using the depthwise separable convolution module in the network, and convolution calculations are performed on the irregularly changing smoke and flame features under fire conditions using the deformable adaptive convolution and Transformer combination module. Specifically: Where y(·) refers to the output result on the feature map after the convolution operation; P out P refers to the position of the output feature after convolution operation. n P refers to the location of the pixel in the receptive field sampled by the K×K convolution. n ={P ij }(0≤i,j≤K-1;0≤n≤K 2 -1; i,j,n∈N); ω(·) represents the value learned by the network in the convolution; x(·) represents the value after gridding on the input feature map; △P n For point P on the feature map n Position offset; Δγ n This represents the weighting coefficient at each sampling point.

8. The lightweight adaptive fire smoke and flame detection method according to claim 5, characterized in that, The deformable adaptive convolution combined with Transformer module, which is sequentially connected to the unfolding module, Transformer module, and folding module, performs unfold-Transformer-folding operations on the input of the third 1×1 convolution, as shown in the following equation: X G (p)=Transformer(X U (p)),1<p<P (2) Where p is the position of each pixel in a patch(P) of the unfolding operation, and the width and height of a P are w and h, respectively; X U (·) indicates the expansion operation. The feature map represents the height and width of the input, P = w·h. X is the number of P; G (·) represents a folding operation.

9. A lightweight adaptive fire smoke and flame detection method according to claim 6, characterized in that, The bounding box regression detection unit utilizes CIoU loss and DFL loss, while the classification loss detection unit utilizes BCE loss. The CIoU loss formula is as follows: in: B、B t These are the predicted bounding box and the ground truth bounding box, b and b', respectively. t ρ is the center point position of the predicted box and the ground truth box, respectively; c is the diagonal distance of the smallest bounding box formed by the predicted box and the ground truth box; ρ is the Euclidean distance between the center points of the predicted box and the ground truth box; α and ν are the aspect ratios, respectively; w and h are the width and height of the predicted box, respectively. t h t These are the width and height of the actual bounding box, respectively; The DFL loss formula is as follows: DFL(S i ,S i+1 )=-((y i+1 -y)log(S i )+(y-y i )log(S i+1 )) (5) DFL optimizes the left (y) element that is closest to the label y in a manner similar to cross-entropy. i )right(y i+1 The probabilities of the two positions are used to determine the distribution of the neighborhood where the network can quickly focus on the target position. S is the output distribution. The BCE loss formula is as follows: Where LN represents the number of samples, PT Value T represents the weight value. Value P represents the label value. Value This represents the model's predicted value.

10. A lightweight adaptive fire smoke and flame detection method according to claim 1, characterized in that, In step S3, when training the lightweight adaptive fire smoke and flame detection neural network combining Transformer and CNN, an improved dynamic adaptive activation function is used to alleviate gradient vanishing in the model, accelerate the convergence speed of network model training, and improve network performance. The specific formula is as follows: Where x represents the input and β is a learnable, dynamically adaptive parameter.

Images