A fire smoke flame image recognition method

Abstract

This invention discloses a method for recognizing fire smoke and flame images. The method involves creating a dataset of fire smoke and flame images; improving the YOLOv5s model to obtain a fire recognition model for recognizing fire smoke and flame images; performing data augmentation on the fire smoke and flame images; inputting the data-augmented fire smoke and flame images into the fire recognition model for training; and finally, inputting the fire smoke and flame images into the trained fire recognition model to obtain the recognition results. This invention achieves accurate and efficient recognition of early-stage fire smoke and flame images during a fire through a fire recognition model.

Description

Technical Field

[0001] This invention relates to the field of fire monitoring technology, and in particular to a method for recognizing fire smoke and flame images. Background Technology

[0002] Traditional fire monitoring technologies mainly rely on contact fire detectors, which are commonly found in various public settings. However, they are limited by their indoor detection range and susceptibility to failure over time, making them unsuitable for effective fire monitoring in large indoor spaces or open outdoor spaces.

[0003] Compared to contact-type fire detectors, non-contact video fire detection technology offers advantages such as requiring no additional hardware, providing intuitive and comprehensive fire alarm information, rapid response, and a large detection range. It is suitable for fire monitoring in large indoor spaces, open outdoor spaces, and forests, and is already becoming increasingly common in high-rise buildings, industrial plants, and forest areas.

[0004] Convolutional Neural Network (CNN) models have been widely applied in the field of intelligent recognition and have achieved good results. In pursuit of higher accuracy, CNN models are becoming increasingly deep and complex. However, in some real-world applications, such as mobile or embedded devices, such large and complex models are difficult to apply. CNN models are too large and currently face the problem of insufficient memory on edge devices, while fire smoke and flame monitoring scenarios require low latency and high accuracy. Therefore, designing a small and efficient fire recognition model using fire smoke and flame images captured by surveillance cameras for efficient and accurate fire identification is particularly important. Summary of the Invention

[0005] This invention provides a method for fire smoke and flame image recognition, which reduces the redundancy of fire recognition models and enables accurate and efficient recognition of fire smoke and flame images in the early stages of a fire.

[0006] To achieve the above effects, the technical solution of the present invention is as follows:

[0007] A method for recognizing fire smoke and flame images includes the following steps:

[0008] Step S1: Create a dataset of fire smoke and flame images;

[0009] Step S2: Improve the YOLOv5s model to obtain a fire recognition model for recognizing fire smoke and flame images;

[0010] Step S3: Perform data augmentation on the fire smoke and flame images, and input the data-augmented fire smoke and flame images into the fire recognition model for training to obtain the trained fire recognition model.

[0011] Step S4: Input the fire smoke and flame image into the trained fire recognition model to obtain the recognition result of the fire smoke and flame image;

[0012] Step S2 includes: S21: Structural improvement of the YOLOv5s model; The fire detection model consists of three parts: Backbone network, Neck network, and Prediction; The Backbone network uses depthwise separable convolution;

[0013] S22: Add an attention mechanism module that focuses on the number of channels at the bottom layer of the Backbone network;

[0014] S23: Introduce a parameterless attention mechanism module into the Neck network of the fire identification model.

[0015] It should be noted that the fire identification model specifically includes: a 640*640 image tensor input layer, a Backbone network, a Neck network, and Prediction. The input layer incorporates Mosaic data augmentation. The Backbone network is the core structure of YOLOv5, and the Neck network includes a parameter-free attention mechanism module. In Prediction, the loss function for the bounding box uses CIOU. loss Loss function; In the post-processing of target detection, the fire identification model uses non-maximum suppression (NMS) to filter multiple target anchor boxes.

[0016] In the above scheme, the fire smoke and flame image dataset in step 1 includes a daytime fire smoke and flame image dataset and a nighttime infrared fire flame dataset. The recognition of fire smoke and flame images is applicable regardless of daytime or nighttime. Based on the recognition results, it can be determined whether a fire has occurred. Furthermore, it allows for flexible deployment of intelligent fire edge monitoring equipment, is simple to operate, has low latency, and provides accurate and efficient recognition.

[0017] Furthermore, step S1 specifically involves:

[0018] S11: Segment the fire video into frames to obtain fire smoke and flame images, construct smoke-like and fire-like samples, complete the annotation of the fire scene, and form a fire smoke and flame image dataset.

[0019] S12: Divide the fire smoke and flame image dataset into training set, validation set and test set according to the proportions;

[0020] S13: Use the image bounding box annotation tool LabelImg to annotate the smoke and flames in the fire smoke and flame images, obtain the coordinates, width and height of the target objects and the name of the target objects, and save them in the form of XML files to create the labels corresponding to the images in the fire smoke and flame image dataset.

[0021] Understandably, the collected data comes from real-world fire smoke and flame scenarios, where fire videos can be captured using ordinary or infrared surveillance cameras. For targets of varying sizes at different distances, a sample balance is constructed within the fire smoke and flame image dataset for different target objects.

[0022] Furthermore, in step S21, the Backbone network includes a first convolutional layer, two first modules 1, two second modules 1, a third module 1, a fourth module 1, a fifth module 1, six sixth modules 1, and a module 2 connected in sequence.

[0023] Among them, the convolution kernels of the two first modules 1 and the two second modules 1 are all 3x3; the convolution kernels of the third module 1 and the fourth module 1 are all 3x3; and the convolution kernels of the fifth module 1 and the six sixth modules 1 are all 5x5.

[0024] The Neck network comprises, in sequence, a second convolutional layer, a first upsampling layer, a first fusion layer, three first modules 3, a third convolutional layer, a second upsampling layer, a second fusion layer, three second modules 3, a first module 4, a seventh module 1, a third fusion layer, three third modules 3, a second module 4, an eighth module 1, a fourth fusion layer, three fourth modules 3, and a third module 4;

[0025] In particular, the convolution kernels of Module 1 and Module 1 are both 5x5;

[0026] The output of the third module 1 is connected to the input of the second fusion layer, the output of the fifth module 1 is connected to the input of the first fusion layer, the output of the third convolutional layer is connected to the input of the third fusion layer, and the output of the second convolutional layer is connected to the input of the fourth fusion layer.

[0027] The fire smoke and flame image is passed through the first convolutional layer, two first modules 1, two second modules 1, and the third module 1 to obtain the first feature map. The first feature map is passed through the fourth module 1 and the fifth module 1 to obtain the second feature map. The second feature map is passed through six sixth modules 1 and 2 to obtain the third feature map.

[0028] After feature fusion in module 2, the third feature map is convolved by the second convolutional layer and upsampled by the first upsampling layer to obtain an upsampled feature map. The upsampled feature map is then fused with the second feature map to obtain a first feature fusion map. The first feature fusion map is then fused with the first feature map after passing through three first modules 3, the third convolutional layer, and the second upsampling layer to obtain a second feature fusion map. The second feature fusion map is then fused with the first feature map after passing through three second modules 3 and the first module 4 to obtain the first output feature map.

[0029] The second feature fusion map is then passed through three second modules 3, one first module 4, and one seventh module 1 in sequence, and then fused with the feature map of the first feature fusion map through three first modules 3 and the third convolutional layer in sequence to obtain the third feature fusion map; the third feature fusion map is then passed through three third modules 3 and one second module 4 in sequence to obtain the second output feature map.

[0030] The third feature fusion map passes through three third modules 3, second module 4, and eighth module 1 in sequence, and then performs feature fusion operation with the third feature map that has passed through the second convolutional layer to obtain the fourth feature fusion map; the fourth feature fusion map passes through three fourth modules 3 and third module 4 to obtain the third output feature map.

[0031] Understandably, the fire detection model consists of three parts: a backbone network, a neck network, and a prediction network, with the backbone network serving as the main network. Depthwise separable convolutions can significantly reduce the number of parameters in the fire detection model. To ensure the accuracy, real-time performance, and deployment requirements of fire detection, the backbone network design avoids using residual connections or other methods that increase model parameters; instead, a small number of attention modules are added to the lower layers of the backbone network. Referring to the energy function of parameterless attention mechanisms in neuroscience, the ability to identify fire smoke and flames can be further improved from a three-dimensional perspective.

[0032] Furthermore, the two first modules 1, the two second modules 1, the third module 1, the fourth module 1, the fifth module 1, the six sixth modules 1, the seventh module 1, and the eighth module 1 all include sequentially connected layer-by-layer convolutional layers, point-by-point convolutional layers, global average pooling layers, first fully connected layers, and second fully connected layers; wherein, the layer-by-layer convolutional layers and point-by-point convolutional layers constitute depthwise separable convolutions, the layer-by-layer convolutional layers, point-by-point convolutional layers, and the first fully connected layer use the SiLU activation function, and the second fully connected layer uses the h-sigmoid activation function;

[0033] In step S21, the depth-separable convolution is divided into layer-by-layer convolutional layers and point-by-point convolutional layers;

[0034] The attention mechanism module that focuses on the number of channels in step S22 consists of a global average pooling layer, a first fully connected layer, and a second fully connected layer.

[0035] Module 2: Includes a convolutional layer with multiple residuals connected in sequence, a first max pooling layer, a second max pooling layer, a third max pooling layer, and a fusion layer;

[0036] The three first modules 3, the three second modules 3, the three third modules 3, and the three third modules 3 each consist of 7 or 5 sequentially connected convolutional layers and one fusion layer;

[0037] First Module 4, Second Module 4, and Third Module 4: These are the parameterless attention mechanism modules in step S23. Each parameterless attention mechanism module consists of a sequentially connected Sigmoid activation function and an energy function.

[0038] Furthermore, the SiLU activation function formula is as follows:

[0039] SiLU(x) = x * Sigmoid(x)

[0040] The Sigmoid activation function is expressed as:

[0041]

[0042] The SiLU activation function is not monotonically increasing, and its key characteristics include self-stability, no upper bound but lower bound, smoothness, and non-monotonicity, which give it an advantage in deep models.

[0043] Furthermore, the energy function mentioned in step S23 is calculated using the following formula:

[0044]

[0045] Among them, E t Let be the energy function. N = H × W, where N is the number of neurons in a single channel, t is the target neuron, and O is the number of neurons in the channel. i For other neurons, the coefficient γ = 10 -4 The importance of each neuron is calculated get.

[0046] Furthermore, in step S3, data augmentation is performed on the fire smoke and flame image dataset. Specifically, multiple fire smoke and flame images in the dataset are randomly scaled, randomly cropped, and randomly arranged to be stitched together to achieve Mosaic data augmentation.

[0047] Understandably, Mosaic data augmentation can enrich the background information of the detected objects, regularize the data, and generate more data from limited data, thereby increasing the diversity and quantity of fire samples and improving the robustness of fire identification models.

[0048] Furthermore, in step S3, the data-augmented fire smoke and flame image is input into the fire recognition model for training, specifically as follows:

[0049] S31: The training set is input into the fire recognition model, and features are extracted according to the preset parameters and the preset number of training iterations. The fire smoke and flame image is divided into several feature points.

[0050] S32: The feature points are offset and multiple prediction boxes are given based on the feature points falling into the suspected fire area; the prediction boxes that meet the confidence level are selected by sorting the scores of the prediction boxes, and non-maximum suppression is used to prevent the accumulation of prediction boxes of the same type and select the prediction box with the highest score, thereby obtaining the optimal fire smoke and flame position.

[0051] S33: Re-apply gray bars and zero-fill to the fire smoke and flame images in the training set to ensure that the dimensions of the fire smoke and flame images are the same. Using the predicted bounding boxes generated by the fire recognition model during training, a CIOU-based method is employed. loss Non-maximum suppression of the loss function is used to obtain the fire smoke and flame prediction box with the highest confidence.

[0052] S34: The model parameters with the minimum loss on the validation set during the entire iterative training process of the fire identification model are used as the prediction parameters of the fire identification model to obtain the trained fire identification model.

[0053] In the above scheme, step S33 involves re-applying gray bars and zero-filling to the fire smoke and flame images in the training set during model training, and then changing the size of the fire smoke and flame images. This can prevent the fire smoke and flame images from becoming distorted during the process of changing their size.

[0054] Furthermore, in step S33, CIOU loss The formula for calculating the loss function is:

[0055] CIOU loss =1-CIoU(2)

[0056] in:

[0057]

[0058]

[0059]

[0060]

[0061] Among them, B, B gt These are the predicted bounding box and the ground truth bounding box, b and b', respectively. gt ρ represents the center point positions of the predicted bounding box and the ground truth bounding box, respectively; α and ν represent the aspect ratios of the predicted and ground truth bounding boxes, respectively; and w and h represent the width and height of the predicted bounding box, respectively. gt h gt These are the width and height of the actual bounding box, respectively.

[0062] In the above scheme, CIoU considers the overlap area between the predicted box and the ground truth box, the distance between the center points, the aspect ratio, and adds loss and aspect ratio loss to the detection box, making the predicted box more consistent with the ground truth box.

[0063] Further, step S4 specifically involves: converting the trained fire identification model to obtain the ONNX model with the required accuracy; converting the ONNX model into a suitable edge intelligent model; deploying the edge intelligent model to an edge intelligent device; processing the real-time video captured by the edge intelligent device into fire smoke and flame images and inputting them into the fire identification model; completing efficient identification and real-time detection of fire smoke and flames at the edge of the edge intelligent device; and decoding the output of the fire identification model to obtain the category, confidence score, and location of the fire smoke and flames.

[0064] In the above scheme, the backbone network of the fire identification model uses depthwise separable convolution with embedded attention instead of residuals, pooling, or other parameter-increasing operations. This reduces the parameters of the fire identification network and the inference time while ensuring the accuracy of fire identification. The edge intelligent device can be a monitoring device or other data acquisition device.

[0065] Edge intelligence models can also monitor and forecast fires in target areas, and present real-time information about the fire and its geographical location through data visualization.

[0066] Compared with the prior art, the beneficial effects of the technical solution of the present invention are:

[0067] This invention improves upon the YOLOv5s model, achieving excellent training results and demonstrating advantages in both accuracy and speed of fire smoke and flame detection. It also reduces redundancy in the fire detection model, achieving a balance between detection speed and accuracy.

[0068] The Backbone network of the fire identification model uses depthwise separable convolutions, which reduces the parameters and inference time of the fire identification network while ensuring the accuracy of fire identification.

[0069] A parameter-free attention mechanism module is introduced into the Neck network of the fire identification model, which improves the fire identification network's ability to identify small target fire smoke and flames without increasing the parameters.

[0070] The SiLU activation function, which has advantages in deep models and has self-stabilizing properties, was used to further improve the fire identification accuracy of the fire identification network.

[0071] The fire identification model trained by this invention can be ported to edge intelligent devices, enabling the edge intelligent devices to perform at their best within their performance limits, and achieving more accurate and real-time fire identification by the fire identification model. Attached Figure Description

[0072] The accompanying drawings are for illustrative purposes only and should not be construed as limiting the invention. To better illustrate this embodiment, some components in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0073] Figure 1 This is a schematic diagram of the method flow provided in an embodiment of the present invention;

[0074] Figure 2 This is a schematic diagram of dataset partitioning provided in an embodiment of the present invention;

[0075] Figure 3 This is a schematic diagram of module 1 provided in an embodiment of the present invention;

[0076] Figure 4 This is a schematic diagram of module 2 provided in an embodiment of the present invention;

[0077] Figure 5 This is a schematic diagram of module 3 provided in an embodiment of the present invention;

[0078] Figure 6 A schematic diagram of module 4 provided in an embodiment of the present invention;

[0079] Figure 7 An improved overall structure diagram of the YOLOv5s network provided in this embodiment of the invention;

[0080] Figure 8 A flowchart illustrating the deployment of a fire identification network as provided in this embodiment of the invention;

[0081] Figure 9 This is an overall design drawing of a fire identification system provided in an embodiment of the present invention;

[0082] Figure 10 This is a fire detection effect diagram provided for an embodiment of the present invention. Detailed Implementation

[0083] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0084] Before introducing the embodiments of this application, the relevant terms used in this application are defined as follows:

[0085] Concat: to merge;

[0086] Upsampl: Upsampling;

[0087] SGD: Gradient Descent Method;

[0088] MSRA: Multiscale Attention Module;

[0089] ONNX: Open Neural Network Exchange;

[0090] Example 1

[0091] For easier understanding, please refer to Figure 1 An embodiment of a fire smoke and flame image recognition method provided by the present invention includes the following steps:

[0092] Step S1: Create a dataset of fire smoke and flame images;

[0093] Step S2: Improve the YOLOv5s model to obtain a fire recognition model for recognizing fire smoke and flame images;

[0094] Step S3: Perform data augmentation on the fire smoke and flame image dataset, and input the data augmented dataset training set into the fire recognition model to obtain the trained fire recognition model;

[0095] Step S4: Input the dataset into the trained fire recognition model to obtain the recognition results of the fire smoke and flame images, and determine whether a fire has occurred;

[0096] Step S2 specifically involves: S21: Improving the structure of the YOLOv5s model based on the actual fire scenario and detection requirements; the fire recognition model consists of three parts: Backbone network, Neck network, and Prediction; the Backbone network uses depthwise separable convolutions;

[0097] S22: Add an attention mechanism module that focuses on the number of channels at the bottom layer of the Backbone network;

[0098] S23: Introduce a parameterless attention mechanism module into the Neck network of the fire identification model; the parameterless attention mechanism module consists of activation functions and energy functions connected in sequence.

[0099] The fire identification model trained by this invention can be ported to edge intelligent devices, enabling these devices to perform at their best within their performance limits. This allows the fire identification model to identify fires more accurately and in real time, providing an effective solution for smart fire protection.

[0100] Example 2

[0101] Specifically, based on Example 1, the solution will be described in conjunction with specific embodiments to further demonstrate its technical effects. Specifically:

[0102] The fire smoke and flame image recognition method based on improved YOLOv5s edge intelligence provided in this embodiment includes the following steps:

[0103] Step S1: Create a fire smoke and flame image dataset; collect video data of real fire scenes under sunlight and infrared at night, obtain fire smoke and flame images by slicing frames every 1 second, filter the fire smoke and flame images, and select the images containing fire smoke and flame targets.

[0104] Step S2: Improve the YOLOv5s model to obtain a fire recognition model for recognizing fire smoke and flame images;

[0105] Step S21: Improve the structure of the YOLOv5s model; the fire detection model consists of three parts: Backbone network, Neck network, and Prediction; the Backbone network of the fire detection model uses depthwise separable convolution.

[0106] S22: Add an attention mechanism module that focuses on the number of channels at the bottom layer of the Backbone network;

[0107] S23: Introduce a parameterless attention mechanism module into the Neck network of the fire identification model; the parameterless attention mechanism module consists of activation functions and energy functions connected in sequence.

[0108] Specifically, the Backbone network includes a first convolutional layer, two first modules 1, two second modules 1, a third module 1, a fourth module 1, a fifth module 1, and six sixth modules 1 connected in sequence; wherein, the convolutional kernels of the two first modules 1 and the two second modules 1 are all 3x3, and the stride of the six sixth modules 1 is 2; the convolutional kernels of the third module 1 and the fourth module 1 are all 3x3; the convolutional kernels of the fifth module 1 and the six sixth modules 1 are all 5x5, and the stride of the six sixth modules 1 is 6;

[0109] The Neck network comprises, in sequence, a second convolutional layer, a first upsampling layer, a first fusion layer, three first modules 3, a third convolutional layer, a second upsampling layer, a second fusion layer, three second modules 3, a first module 4, a seventh module 1, a third fusion layer, three third modules 3, a second module 4, an eighth module 1, a fourth fusion layer, three fourth modules 3, and a third module 4; wherein the stride of the three first modules 3, the three second modules 3, the three third modules 3, and the three third modules 3 is 3; the convolutional kernels of the seventh module 1 and the eighth module 1 are both 5x5;

[0110] The output of the third module 1 is connected to the input of the second fusion layer, the output of the fifth module 1 is connected to the input of the first fusion layer, the output of the third convolutional layer is connected to the input of the third fusion layer, and the output of the second convolutional layer is connected to the input of the fourth fusion layer.

[0111] Specifically, the two first modules 1, the two second modules 1, the third module 1, the fourth module 1, the fifth module 1, the six sixth modules 1, the seventh module 1, and the eighth module 1 all include sequentially connected layer-by-layer convolutional layers, point-by-point convolutional layers, global average pooling layers, first fully connected layers, and second fully connected layers. The layer-by-layer convolutional layers, point-by-point convolutional layers, and the first fully connected layer use the SiLU activation function, while the second fully connected layer uses the h-sigmoid activation function. The depthwise separable convolutional layer is divided into layer-by-layer convolutional layers and point-by-point convolutional layers. The global average pooling layer, the first fully connected layer, and the second fully connected layer constitute an attention mechanism module that focuses on the number of channels.

[0112] Module 2: Includes a convolutional layer with multiple residuals connected in sequence, a first max pooling layer, a second max pooling layer, a third max pooling layer, and a fusion layer;

[0113] The three first modules 3, the three second modules 3, the three third modules 3, and the three third modules 3 each consist of 7 or 5 sequentially connected convolutional layers and one fusion layer;

[0114] Module 1, Module 2, and Module 3: This is a parameter-free attention mechanism module that focuses on three-dimensional space. It consists of a sigmoid activation function and an energy function connected in sequence.

[0115] Training process:

[0116] Step S3: Use the visual image bounding box annotation tool LabelImg to annotate the smoke and flames in daytime and nighttime fires, obtain the coordinates, width, height, and name of the target objects, and save them in XML format to create the labels corresponding to the fire smoke and flame images in the fire smoke and flame image dataset, and construct smoke-like and fire-like samples; the daytime fire smoke and flame image dataset is used as the training set, the nighttime fire smoke and flame dataset is used as the validation set, and the smoke-like and fire-like samples are used as the test set; divide the training set, validation set, and test set of the fire smoke and flame dataset in a ratio of 8:1:1;

[0117] S31: The training set is input into the fire recognition model, and features are extracted according to the preset parameters and the preset number of training iterations. The fire smoke and flame image is divided into several feature points.

[0118] S32: The feature points are offset and multiple prediction boxes are given based on the feature points falling into the suspected fire area; the prediction boxes that meet the confidence level are selected by sorting the scores of the prediction boxes, and non-maximum suppression is used to prevent the accumulation of prediction boxes of the same type and select the prediction box with the highest score, thereby obtaining the optimal fire smoke and flame position.

[0119] Fires occur in complex environments, and the surrounding weather and lighting conditions can change. In such cases, Mosaic data augmentation is performed on daytime and nighttime fire smoke and flame datasets. Fire smoke and flame images are stitched together by random scaling, cropping, and arrangement, which greatly enriches the background information of the detected objects.

[0120] Based on the characteristics of day and night fire smoke and flame datasets, for anchor-based YOLOv5s models, the fire recognition model does not need to perform separate additional operations on the preset anchor boxes of object bounding boxes. Instead, it uses adaptive anchor boxes, which adaptively calculate the best anchor box values ​​in different training sets during each training session to obtain the best anchor boxes.

[0121] S33: Re-apply gray bars and zero-fill to the fire smoke and flame images in the training set to ensure that the dimensions of the fire smoke and flame images are the same. Using the predicted bounding boxes generated by the fire recognition model during training, a CIOU-based method is employed. loss Non-maximum suppression of the loss function is used to obtain the fire smoke and flame prediction box with the highest confidence.

[0122] The daytime and nighttime fire smoke and flame datasets were collected and used as the training set for the fire recognition model. Before training, the model was initialized using MSRA without loading any pre-trained weights. During training, the input image tensor was (640, 640, 3), and the model was trained using the SGD optimizer and a cosine annealing learning rate (cos-lr). The initial learning rate was 0.01, the weight decay was 0.0005, the SGD momentum was 0.9, the training batch size was 8, and the total number of training iterations was 300. During the last 100 iterations, the training parameters were saved every 10 iterations.

[0123] S34: After training, the model parameters with the minimum loss on the validation set during the entire iterative training of the fire identification model are used as the prediction parameters of the fire identification model, thus obtaining the trained fire identification model.

[0124] Step S4: As Figure 8 As shown, the trained fire identification model is converted to the required accuracy ONNX model. The batch input is set to 32, and the accuracy is set to FP16. The ONNX model is then converted into a suitable edge intelligence model and deployed to an edge intelligence device. The real-time video captured by the edge intelligence device is processed into fire smoke and flame images and input into the fire identification model. The edge intelligence device performs efficient identification and real-time detection of fire smoke and flames at its edge. The edge intelligence model decodes the output of the fire identification model to obtain the category, confidence score, and location of the fire smoke and flames.

[0125] Fire smoke and flames are distinguished by their categories. The confidence score represents the accuracy of smoke or flame identification. The location of fire smoke and flames can be directly seen from the image.

[0126] Edge intelligent devices are installed in fire-prone areas for real-time fire monitoring. The captured video is processed into frame images, which serve as fire smoke and flame images. These images are then re-painted with gray bars and zero-filled to a size of 460×460. If the length of the fire smoke and flame image is less than 460 pixels, gray bars are added to the length and zero-filling to the width, ensuring that the image is the same in both length and width and thus does not distort the fire smoke and flame image. Finally, the fire smoke and flame images with gray bars and zero-filling are sent to the fire recognition model at the edge intelligent terminal for detection.

[0127] like Figure 7As shown, the fire smoke and flame image is input into the fire recognition model. The fire smoke and flame image is used for feature extraction through the Backbone network: after passing through the first convolutional layer, two first modules 1, two second modules 1, and the third module 1, a 64×64 feature map is obtained. The 64×64 feature map is then passed through the fourth module 1 and the fifth module 1 to obtain a 128×128 feature map. The 128×128 feature map is then passed through six sixth modules 1 and 2 to obtain a 256×256 feature map.

[0128] At this time, as Figure 3 As shown, the fire smoke and flame image is processed through layer-by-layer convolution, point-by-point convolution, global average pooling layer, first fully connected layer, and second fully connected layer. In this embodiment, only the first and second fully connected layers are optional to achieve a balance between the efficiency and accuracy of the fire recognition model, thus better extracting the features of fire smoke and flame. Here, the layer-by-layer convolution, point-by-point convolution, and first fully connected layer use the SiLU activation function, the second fully connected layer uses the h-sigmoid activation function, and the optional attention mechanism module focuses on the number of channels.

[0129] A 64×64 feature map is used to detect large targets in a fire, a 128×128 feature map is used to detect medium-sized targets, and a 256×256 feature map is used to detect small targets. The 64×64 feature map is fused using module 2. The three feature maps are then input into the Neck network for further enhanced feature extraction. A parameterless attention mechanism module is used to enhance feature extraction from fire smoke and flame images. For example... Figure 6 As shown, the parameterless attention mechanism module (i.e., module 4) consists of activation functions and energy functions connected in sequence; as Figure 4 As shown, the SPPF fast feature pyramid includes a convolutional layer with multiple residuals connected in sequence, a first max pooling layer, a second max pooling layer, a third max pooling layer, and a fusion layer. The pooling kernels of the first max pooling layer, the second max pooling layer, and the third max pooling layer are 5×5, 9×9, and 13×13, respectively.

[0130] The Neck network consists of a 256×256 feature map that undergoes feature fusion in module 2, followed by convolution in the second convolutional layer and upsampling in the first upsampling layer to obtain an upsampled feature map. This upsampled feature map is then fused with a 128×128 feature map to obtain a first feature fusion map. The first feature fusion map then passes through three first modules (3), a third convolutional layer, and a second upsampling layer sequentially before being fused with a 64×64 feature map to obtain a second feature fusion map. Finally, the second feature fusion map passes through three second modules (3) and a first module (4) sequentially to obtain a 64×64 output feature map.

[0131] The second feature fusion map is then passed through three second modules 3, one first module 4, and seventh module 1 in sequence, and then fused with the feature map of the first feature fusion map through three first modules 3 and the third convolutional layer in sequence to obtain the third feature fusion map; the third feature fusion map is then passed through three third modules 3 and two second modules 4 in sequence to obtain a 128×128 output feature map.

[0132] The third feature fusion map passes through three third modules 3, second module 4, and eighth module 1 in sequence, and then performs feature fusion operation with the 256×256 feature map that has passed through the second convolutional layer to obtain the fourth feature fusion map; the fourth feature fusion map passes through three fourth modules 3 and third module 4 to obtain the 256×256 output feature map;

[0133] The 64×64, 128×128, and 256×256 output feature maps are input into Prediction for target classification and detection; in case of a fire, CIOU based on the introduction of a penalty coefficient is used. loss Non-maximum suppression of the loss function and a confidence threshold of 0.5 are used to obtain the fire smoke and flame prediction box with the highest confidence, thus achieving accurate identification of fire smoke and flame.

[0134] Example 3

[0135] Specifically, based on Example 2, the solution will be described in conjunction with specific embodiments to further demonstrate its technical effects. Specifically:

[0136] The activation function used in step S23 is the SiLU activation function, as shown in the following formula:

[0137] SiLU(x) = x * Sigmoid(x)

[0138] The sigmoid activation function is expressed as:

[0139]

[0140] The h-sigmoid activation function is a piecewise approximation of the sigmoid activation function, and its formula is:

[0141]

[0142] Where x represents the input unit.

[0143] The energy function mentioned in step S23 is calculated using the following formula:

[0144]

[0145] in, N = H × W, where N is the number of neurons in a single channel, t is the target neuron, and O is the number of neurons in the channel. i For other neurons, the coefficient γ = 10 -4 The importance of each neuron is calculated get.

[0146] This invention combines real-world daytime and nighttime fire smoke and flame scenarios with the need for deployment on edge intelligent devices. When a fire occurs in a fire monitoring area, the edge intelligent devices deploying the fire identification model are linked through switches, cloud computing servers, and the fire monitoring system platform. Figure 9 As shown, data visualization is used to present real-time information about the occurrence of a fire, fire alarm information, and the geographical location of the fire, providing an effective solution for smart fire protection.

[0147] Table 1 Comparison of recognition performance of different models

[0148]

[0149] Here, Model refers to different models, Params refers to the memory size occupied by different models, and mAP refers to the average accuracy of fire smoke and flame recognition in fire smoke and flame images.

[0150] like Figure 10 As shown, the fire identification model deployed according to this invention identifies fire smoke and flame images. The accuracy for identifying flames in the fire smoke and flame images is 0.89, and the accuracy for identifying smoke in the fire smoke and flame images is 0.70. Compared with other models, the fire identification model of this invention has a small memory footprint and good recognition performance.

[0151] It should be noted that the parameterless attention mechanism module focuses on three-dimensional space.

[0152] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art can make other variations or modifications based on the above description. It is neither necessary nor possible to exhaustively describe all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A method for recognizing fire smoke and flame images, characterized in that, Includes the following steps: Step S1: Create a dataset of fire smoke and flame images; Step S2: Improve the YOLOv5s model to obtain a fire recognition model for recognizing fire smoke and flame images; Step S3: Perform data augmentation on the fire smoke and flame images, and input the data-augmented fire smoke and flame images into the fire recognition model for training to obtain the trained fire recognition model. Step S4: Input the fire smoke and flame image into the trained fire recognition model to obtain the recognition result of the fire smoke and flame image; Step S2 includes: S21: Structural improvement of the YOLOv5s model; The fire detection model consists of three parts: Backbone network, Neck network, and Prediction; The Backbone network uses depthwise separable convolution; S22: Add an attention mechanism module that focuses on the number of channels at the bottom layer of the Backbone network; S23: Introduce a parameterless attention mechanism module into the Neck network of the fire identification model.

2. The fire smoke and flame image recognition method according to claim 1, characterized in that, Step S1 specifically involves: S11: Segment the fire video into frames to obtain fire smoke and flame images, construct fire smoke and flame scene annotations, and form a fire smoke and flame image dataset; S12: Divide the fire smoke and flame image dataset into training set, validation set and test set according to the proportions; S13: Use the image bounding box annotation tool LabelImg to annotate the smoke and flames in the fire smoke and flame images, obtain the coordinates, width and height of the target objects and the name of the target objects, and save them in the form of XML files to create the labels corresponding to the images in the fire smoke and flame image dataset.

3. The fire smoke and flame image recognition method according to claim 1, characterized in that, In step S21, the Backbone network includes a first convolutional layer, two first modules 1, two second modules 1, a third module 1, a fourth module 1, a fifth module 1, six sixth modules 1, and a module 2 connected in sequence. Among them, the convolution kernels of the two first modules 1 and the two second modules 1 are all 3x3; the convolution kernels of the third module 1 and the fourth module 1 are all 3x3; and the convolution kernels of the fifth module 1 and the six sixth modules 1 are all 5x5. The Neck network comprises, in sequence, a second convolutional layer, a first upsampling layer, a first fusion layer, three first modules 3, a third convolutional layer, a second upsampling layer, a second fusion layer, three second modules 3, a first module 4, a seventh module 1, a third fusion layer, three third modules 3, a second module 4, an eighth module 1, a fourth fusion layer, three fourth modules 3, and a third module 4; In particular, the convolution kernels of Module 1 and Module 1 are both 5x5; The output of the third module 1 is connected to the input of the second fusion layer, the output of the fifth module 1 is connected to the input of the first fusion layer, the output of the third convolutional layer is connected to the input of the third fusion layer, and the output of the second convolutional layer is connected to the input of the fourth fusion layer. The fire smoke and flame image is passed through the first convolutional layer, two first modules 1, two second modules 1, and the third module 1 to obtain the first feature map. The first feature map is passed through the fourth module 1 and the fifth module 1 to obtain the second feature map. The second feature map is passed through six sixth modules 1 and 2 to obtain the third feature map. After feature fusion in module 2, the third feature map is convolved by the second convolutional layer and upsampled by the first upsampling layer to obtain an upsampled feature map. The upsampled feature map is then fused with the second feature map to obtain a first feature fusion map. The first feature fusion map is then fused with the first feature map after passing through three first modules 3, the third convolutional layer, and the second upsampling layer to obtain a second feature fusion map. The second feature fusion map is then fused with the first feature map after passing through three second modules 3 and the first module 4 to obtain the first output feature map. The second feature fusion map is then passed through three second modules 3, one first module 4, and one seventh module 1 in sequence, and then fused with the feature map of the first feature fusion map through three first modules 3 and the third convolutional layer in sequence to obtain the third feature fusion map; the third feature fusion map is then passed through three third modules 3 and one second module 4 in sequence to obtain the second output feature map. The third feature fusion map passes through three third modules 3, second module 4, and eighth module 1 in sequence, and then performs feature fusion operation with the third feature map that has passed through the second convolutional layer to obtain the fourth feature fusion map; the fourth feature fusion map passes through three fourth modules 3 and third module 4 to obtain the third output feature map.

4. The fire smoke and flame image recognition method according to claim 3, characterized in that, Two first modules 1, two second modules 1, a third module 1, a fourth module 1, a fifth module 1, six sixth modules 1, a seventh module 1, and an eighth module 1: each includes sequentially connected layer-by-layer convolutional layers, point-by-point convolutional layers, global average pooling layers, a first fully connected layer, and a second fully connected layer; wherein, the layer-by-layer convolutional layers and point-by-point convolutional layers constitute depthwise separable convolutions, the layer-by-layer convolutional layers, point-by-point convolutional layers, and the first fully connected layer use the SiLU activation function, and the second fully connected layer uses the h-sigmoid activation function; In step S21, the depth-separable convolution is divided into layer-by-layer convolutional layers and point-by-point convolutional layers; The attention mechanism module that focuses on the number of channels in step S22 consists of a global average pooling layer, a first fully connected layer, and a second fully connected layer. Module 2: Includes a convolutional layer with multiple residuals connected in sequence, a first max pooling layer, a second max pooling layer, a third max pooling layer, and a fusion layer; The three first modules 3, the three second modules 3, the three third modules 3, and the three third modules 3 each consist of 7 or 5 sequentially connected convolutional layers and one fusion layer; First Module 4, Second Module 4, and Third Module 4: These are the parameterless attention mechanism modules in step S23. Each parameterless attention mechanism module consists of a sequentially connected Sigmoid activation function and an energy function.

5. The fire smoke and flame image recognition method according to claim 4, characterized in that, The SiLU activation function formula is as follows: The Sigmoid activation function is expressed as: 。 6. The fire smoke and flame image recognition method according to claim 5, characterized in that, The energy function mentioned in step S23 is calculated using the following formula: Among them, E t Let be the energy function. , , The number of neurons in a single channel. For the target neuron, For other neurons, the coefficient The importance of each neuron is calculated get.

7. The fire smoke and flame image recognition method according to claim 6, characterized in that, In step S3, data augmentation is performed on the fire smoke and flame image dataset. Specifically, multiple fire smoke and flame images in the dataset are randomly scaled, cropped, and arranged to be stitched together to achieve Mosaic data augmentation.

8. The fire smoke and flame image recognition method according to claim 7, characterized in that, In step S3, the data-augmented fire smoke and flame image is input into the fire recognition model for training, specifically as follows: S31: The training set is input into the fire recognition model, and features are extracted according to the preset parameters and the preset number of training iterations. The fire smoke and flame image is divided into several feature points. S32: The feature points are offset and multiple prediction boxes are given based on the feature points falling into the suspected fire area; the prediction boxes that meet the confidence level are selected by sorting the scores of the prediction boxes, and non-maximum suppression is used to prevent the accumulation of prediction boxes of the same type and select the prediction box with the highest score, thereby obtaining the optimal fire smoke and flame position. S33: The fire smoke and flame images in the training set are re-painted with gray bars and zero-filled to ensure that the dimensions of the fire smoke and flame images are the same. Based on the predicted bounding boxes generated by the fire recognition model during training, a method using... Non-maximum suppression of the loss function is used to obtain the fire smoke and flame prediction box with the highest confidence. S34: The model parameters with the minimum loss on the validation set during the entire iterative training process of the fire identification model are used as the prediction parameters of the fire identification model to obtain the trained fire identification model.

9. The fire smoke and flame image recognition method according to claim 8, characterized in that, In step S33 The formula for calculating the loss function is: in: Among them, B, These are the predicted bounding box and the ground truth bounding box, b, These are the center points of the predicted bounding box and the ground truth bounding box, respectively. The distance between the center points of the predicted bounding box and the ground truth bounding box is the Euclidean distance. , ... , These are the width and height of the actual bounding box, respectively.

10. The fire smoke and flame image recognition method according to claim 1, characterized in that, Step S4 specifically involves: converting the trained fire identification model to obtain the ONNX model with the required accuracy; converting the ONNX model into a suitable edge intelligence model; deploying the edge intelligence model to an edge intelligence device; processing the real-time video captured by the edge intelligence device into fire smoke and flame images and inputting them into the fire identification model; performing efficient identification and real-time detection of fire smoke and flames at the edge of the edge intelligence device; and decoding the output of the fire identification model to obtain the category, confidence score, and location of the fire smoke and flames.

Images