A flame and smoke detection method combined with efficient sampling enhancement

Abstract

This invention relates to a flame and smoke detection method that incorporates efficient sampling and enhancement, belonging to the field of computer vision target detection technology. The method comprises steps S1: collecting flame and smoke images and manually annotating them to obtain a smoke and flame dataset containing bounding boxes; S2: preprocessing and data enhancement of the images in the smoke and flame dataset obtained in S1; S3: designing an efficient sampling and enhancement module; S4: building a network model that includes the efficient sampling and enhancement module; and S5: training the network model built in S4 and outputting detection results. This method can address the issue of low flame and smoke detection and recognition accuracy, providing new improvement ideas for subsequent fire detection, fire analysis, and other related engineering applications.

Description

Technical Field

[0001] The present invention relates to a flame and smoke detection method combined with efficient sampling enhancement, belonging to the technical field of computer vision target detection. Background Art

[0002] Object detection algorithms from computer vision can identify and locate targets, thereby locating flames and assisting firefighting equipment in extinguishing them. Furthermore, with the continuous advancement of deep learning research, deep learning-based object detection methods have achieved high recognition accuracy. However, due to the variable and irregular shapes of flames and smoke, current mainstream detection algorithms cannot achieve high accuracy for these two types of targets.

[0003] Chinese patent CN111091072A discloses a flame and smoke detection method based on YOLOv3. This method primarily uses the deep learning-based YOLOv3 algorithm to detect flames and smoke. By establishing a custom dataset, the YOLOv3 algorithm is used to train a flame detection model and a smoke detection model. In an actual system, the results of these two models are combined to determine whether a fire has occurred. However, this method only uses an object detection algorithm to detect flames and smoke, and does not address the issue of low flame and smoke detection accuracy.

[0004] Therefore, it is of great significance to rationally design an enhancement method to improve the detection accuracy of flames and smoke based on the variable and irregular shapes of these two types of targets. Summary of the Invention

[0005] The purpose of the present invention is to provide a flame and smoke detection method combined with efficient sampling enhancement to solve the problem of low accuracy in flame and smoke detection and recognition, and to lay the foundation for subsequent fire detection and fire analysis.

[0006] In order to achieve the above object, the technical solution adopted by the present invention is:

[0007] A flame and smoke detection method combined with efficient sampling enhancement comprises the following steps:

[0008] S1: Collect flame and smoke images and manually annotate them to obtain a smoke and flame dataset containing bounding boxes;

[0009] S2: Preprocess and enhance the images in the smoke and flame dataset obtained in S1;

[0010] S3: Design an efficient sampling enhancement module;

[0011] S4: Build a network model that includes an efficient sampling enhancement module;

[0012] S5: Train the network model built in S4 and output the detection results.

[0013] A further improvement of the technical solution of the present invention is that the specific operation of step S1 is:

[0014] Web crawler tools were used to collect images of flame and smoke scenes from search engine galleries, and some video images of flame scenes were captured. Unqualified image data were cleaned and then merged into a total flame and smoke image set. Image annotation tools were used to annotate the bounding boxes of smoke and flame in the images to obtain the smoke and flame dataset.

[0015] A further improvement of the technical solution of the present invention is that the specific operation of step S2 is:

[0016] After loading the images and their label data in the smoke and flame dataset obtained by S1, the images are scaled to a certain pixel size, the hue, saturation, and brightness of the images are dynamically changed within a range with a certain probability, and horizontally flipped with a certain probability, and then enhanced using the mosaic data augmentation method.

[0017] A further improvement of the technical solution of the present invention is that the specific operation of step S3 is:

[0018] In the neural network, after the input image is calculated through the intermediate network layer, the intermediate feature map output is obtained. For the obtained feature map, the deformable convolution obtains the offset of each position of the convolution kernel through learning, and then performs a weighted summation operation, which can be expressed as follows:

[0019]

[0020] Where p0 is a certain position on the feature map, Ω is the set of eight-neighborhood displacement offsets of the pixel, x(.) is the pixel vector at a certain position in the feature map, y(.) is the output after convolution of a certain position in the input feature map, w(.) is the weight vector of the convolution kernel at a certain position, and Δp is the offset of the new sampling point relative to the original position in the convolution kernel;

[0021] The feature map output by the previous layer of the network is first subjected to a 1x1 convolution operation to obtain a new feature map FMi1. At this time, the number of channels is reduced to half of the original. Then, FMi1 is subjected to a 3×3 deformable convolution operation to obtain a feature map FMi2. FMi1 and FMi2 are then connected in the channel direction to obtain the output FMi of the i-th layer network. The above operation steps are combined in one module to obtain an efficient sampling enhancement module.

[0022] The further improvement of the technical solution of the present invention is that: the backbone network of the network model adopts the CSPNet structure, and an efficient sampling enhancement module designed by S3 is added after the last layer and the penultimate layer of CSPNet; the feature combination layer of the network model adopts the PAN structure, which integrates the context information of different layers of the network and enhances the extraction of overall features; the detection head of the network model adopts the Yolo detection head to output the predicted category information and the position information of the target box.

[0023] A further improvement of the technical solution of the present invention is that: step S5 trains the network model based on the Pytorch framework, and the loss functions used in the training process include classification loss cls_loss, confidence loss obj_loss and positioning loss box_loss;

[0024] The expression of the classification loss cls_loss is:

[0025]

[0026] Among them, c represents the number of categories, l i Indicates whether category i is the true target category, the true value is 1, otherwise it is 0, P cli Represents the predicted value for this class;

[0027] The expression of the confidence loss obj_loss is:

[0028] obj_loss=(T conf -P conf ) 2

[0029] Among them, T conf True value, 1 when it contains the target, otherwise 0, P conf is the confidence of the network prediction;

[0030] The expression of the positioning loss box_loss is:

[0031] box_loss = (1-CIoU)

[0032] Among them, CIoU represents the intersection-over-union ratio of the enhanced prediction box and the true value box;

[0033] The number of rounds used for network model training is 300; the initial learning rate used is 0.01; the learning rate decay strategy uses the cosine annealing strategy; the network momentum is set to 0.937; the warm-up strategy is used, and the number of warm-up rounds is 3; the batch size used for training is 8;

[0034] The output of the network model needs to be post-processed, that is, non-maximum suppression, to obtain the final prediction output.

[0035] Due to the adoption of the above technical solution, the technical effects achieved by the present invention are as follows:

[0036] The present invention solves the problem of low accuracy in flame and smoke detection and recognition. Specifically, the proposed efficient sampling enhancement module solves the problem that ordinary networks have difficulty extracting the features of targets with large shape variations, such as flames and smoke, thereby greatly improving the overall detection success rate of the network and providing new improvement ideas for subsequent fire detection, fire analysis and other related engineering applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is a schematic diagram of the network structure of the present invention;

[0038] Figure 2 is a schematic diagram of an efficient sampling enhancement module of the present invention;

[0039] Figure 3 This is a heat map comparison between this method and the original method. DETAILED DESCRIPTION

[0040] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0041] A flame and smoke detection method combined with efficient sampling enhancement comprises the following steps:

[0042] S1: Collect flame and smoke images and manually annotate them to obtain a smoke and flame dataset containing bounding boxes

[0043] Web crawler tools were used to collect images of flame and smoke scenes from search engine galleries, and some video images of flame scenes were captured. Unqualified image data were cleaned and then merged into a total flame and smoke image set. Image annotation tools were used to annotate the bounding boxes of smoke and flames in the images, resulting in a smoke and flame dataset: a total of 5,000 images, divided into two categories, with 4,653 flame instances and 3,747 smoke instances.

[0044] S2: Preprocess and enhance the images in the smoke and flame dataset obtained in S1

[0045] After loading the images and their label data from the smoke and flame dataset obtained in S1, the images are scaled to a pixel size of 640*640, and the hue, saturation, and brightness of the images are dynamically changed within a range with a probability of 0.5. The images are horizontally flipped with a probability of 0.5, and then enhanced using the mosaic data augmentation method.

[0046] S3: Designing an efficient sampling enhancement module

[0047] In a neural network, the input image is processed through the intermediate network layer to obtain the intermediate feature map output. For the obtained feature map, the deformable convolution learns the offset of each position of the convolution kernel and then performs a weighted summation operation, which can be expressed as:

[0048]

[0049] Among them, p0 is a certain position on the feature map, Ω is the set of eight-neighborhood displacement offsets of the pixel, x(.) is the pixel vector at a certain position in the feature map, y(.) is the output after convolution of a certain position in the input feature map, w(.) is the weight vector of the convolution kernel at a certain position, and Δp is the offset of the new sampling point relative to the original position in the convolution kernel.

[0050] In order to reduce the large amount of calculation and parameters brought by the use of deformable convolution, the feature map output by the previous layer of the network is first subjected to a 1x1 convolution operation to obtain a new feature map FMi1. At this time, the number of channels is reduced to half of the original. Then, FMi1 is subjected to a 3×3 deformable convolution operation to obtain a feature map FMi2. FMi1 and FMi2 are then connected in the channel direction to obtain the output FMi of the i-th layer network. The above operation steps are combined in one module to obtain an efficient sampling enhancement module (ESEM), as shown in the following figure. Figure 2 shown.

[0051] S4: Build a network model with an efficient sampling enhancement module

[0052] like Figure 1 As shown in the figure, the backbone network of the network model adopts the CSPNet structure. The efficient sampling enhancement module (ESEM) designed by S3 is added after the last layer and the penultimate layer of CSPNet to obtain higher efficiency. The feature combination layer of the network model adopts the PAN structure to fuse the context information of different layers of the network and enhance the extraction of overall features. The detection head of the network model adopts the Yolo detection head to output the predicted category information and the position information of the target box.

[0053] S5: Train the network model built in S4 and output the detection results

[0054] The network model is trained based on the Pytorch framework. The loss functions used in the training process include classification loss cls_loss, confidence loss obj_loss, and positioning loss box_loss.

[0055] The expression of the classification loss cls_loss is:

[0056]

[0057] Among them, c represents the number of categories, li Indicates whether category i is the true target category, the true value is 1, otherwise it is 0, P cli Represents the predicted value for this class;

[0058] The expression of the confidence loss obj_loss is:

[0059] obj_loss=(T conf -P conf ) 2

[0060] Among them, T conf True value, 1 when it contains the target, otherwise 0, P conf is the confidence of the network prediction;

[0061] The expression of the positioning loss box_loss is:

[0062] box_loss = (1-CIoU)

[0063] Among them, CIoU represents the intersection-over-union ratio of the enhanced prediction box and the true value box;

[0064] The number of rounds used for network model training is 300; the initial learning rate used is 0.01; the learning rate decay strategy uses the cosine annealing strategy; the network momentum is set to 0.937; the warm-up strategy is used, and the number of warm-up rounds is 3; the batch size used for training is 8;

[0065] The output of the network model needs to be post-processed, that is, non-maximum suppression, to obtain the final prediction output.

[0066] In order to prove the effectiveness of the present invention, a comparative experiment was conducted between the present invention and the current mainstream target detection algorithm on the dataset prepared by the present invention. The experimental results are shown in the following table:

[0067] algorithm mAP (%) Parameter quantity Computational capacity (GFLOPs) Faster R-CNN 69.6 31.75M 88.9 SSD 70.8 23M 13.8 FCOS 72.9 45M 65.6 CenterNet 66.7 19M 25.7 The present invention 74.2 8.1M 17.6

[0068] At the same time, the improved network model of the present invention is compared with the thermal map of the network layer before improvement (network without ESEM), as shown in FIG. Figure 3 As shown, the comparison results intuitively demonstrate the superiority of the present invention in learning flame and smoke characteristics.

Claims

1. A flame and smoke detection method combined with efficient sampling enhancement, characterized in that: The following steps are involved: S1: Collect flame and smoke images and manually annotate them to obtain a smoke and flame dataset containing bounding boxes; S2: Preprocess and enhance the images in the smoke and flame dataset obtained in S1; S3: Design an efficient sampling enhancement module; the specific operations of step S3 are: In the neural network, after the input image is calculated through the intermediate network layer, the intermediate feature map output is obtained. For the obtained feature map, the deformable convolution obtains the offset of each position of the convolution kernel through learning, and then performs a weighted summation operation, which can be expressed as follows: Where p0 is a certain position on the feature map, Ω is the set of eight-neighborhood displacement offsets of the pixel, x(.) is the pixel vector at a certain position in the feature map, y(.) is the output after convolution of a certain position in the input feature map, w(.) is the weight vector of the convolution kernel at a certain position, and Δp is the offset of the new sampling point relative to the original position in the convolution kernel; The feature map output by the previous layer of network is first subjected to a 1x1 convolution operation to obtain a new feature map FMi1. At this time, the number of channels is reduced to half of the original. Then, FMi1 is subjected to a 3×3 deformable convolution operation to obtain a feature map FMi2. FMi1 and FMi2 are then connected in the channel direction to obtain the output FMi of the i-th layer network. The above steps are combined in one module to obtain an efficient sampling enhancement module. S4: Build a network model that includes an efficient sampling enhancement module. The backbone network of the network model adopts the CSPNet structure, and the efficient sampling enhancement module designed in S3 is added after the last layer and the penultimate layer of CSPNet. The feature combination layer of the network model adopts the PAN structure to integrate the context information of different layers of the network and enhance the extraction of overall features. The detection head of the network model adopts the Yolo detection head to output the predicted category information and the location information of the target box. S5: Train the network model built in S4 and output the detection results.

2. The flame and smoke detection method combined with efficient sampling enhancement according to claim 1, characterized in that: The specific operations of step S1 are: Web crawler tools were used to collect images of flame and smoke scenes from search engine galleries, and some video images of flame scenes were captured. Unqualified image data were cleaned and then merged into a total flame and smoke image set. Image annotation tools were used to annotate the bounding boxes of smoke and flame in the images to obtain the smoke and flame dataset.

3. The flame and smoke detection method combined with efficient sampling enhancement according to claim 1, characterized in that: The specific operations of step S2 are: After loading the images and their label data in the smoke and flame dataset obtained by S1, the images are scaled to a certain pixel size, the hue, saturation, and brightness of the images are dynamically changed within a range with a certain probability, and horizontally flipped with a certain probability, and then enhanced using the mosaic data augmentation method.

4. The flame and smoke detection method combined with efficient sampling enhancement according to claim 1, characterized in that: Step S5 trains the network model based on the Pytorch framework, and the loss functions used in the training process include classification loss cls_loss, confidence loss obj_loss, and positioning loss box_loss; The expression of the classification loss cls_loss is: Among them, c represents the number of categories, l i Indicates whether category i is the true target category, the true value is 1, otherwise it is 0, P cli Represents the predicted value for this class; The expression of the confidence loss obj_loss is: obj_loss=(T conf -P conf ) 2 Among them, T conf True value, 1 when it contains the target, otherwise 0, P conf is the confidence of the network prediction; The expression of the positioning loss box_loss is: box_loss = (1-CIoU) Among them, CIoU represents the intersection-over-union ratio of the enhanced prediction box and the true value box; The number of rounds used for network model training is 300; the initial learning rate used is 0.01; the learning rate decay strategy uses the cosine annealing strategy; the network momentum is set to 0.937; the warm-up strategy is used, and the number of warm-up rounds is 3; the batch size used for training is 8; The output of the network model needs to be post-processed, that is, non-maximum suppression, to obtain the final prediction output.

Images