A multi-stage weakly supervised flame position detection method based on CAM

Abstract

The present invention discloses a multi-stage weakly supervised flame position detection method based on CAM, which belongs to the field of image target detection. The present invention generates a CAM heat map through a multi-layer feature fusion structure, and through the joint flame classification training of the top-level features of the backbone network and the multi-layer fusion features, the model can accurately extract the CAM heat map for target positioning, thereby achieving flame target positioning accuracy close to that under full supervision conditions without using weak supervision of position labels. This method can complete the inference of flame position by relying on category label images when only flame images and non-flame images are collected, avoiding dependence on position labels and greatly reducing the data acquisition cost of deep learning target detection methods in the field of flame detection.

Description

Technical Field

[0001] The present invention is designed in the field of flame detection based on deep neural networks, specifically a multi-stage weakly supervised flame position detection method based on CAM. Background Art

[0002] Flame detection is a crucial technology in the security field. Common flame detection methods include smoke detectors and infrared sensors. While these methods offer some degree of accuracy, they are limited by their sensing range, making them impractical for large-scale, cost-effective deployment. For example, physical methods are difficult to apply in large, open areas like forest fire prevention and large factories.

[0003] Deep learning neural network technology has achieved significant success in the field of image object detection. Algorithms such as FastRCNN and YOLO have demonstrated impressive accuracy. However, because deep learning methods rely heavily on repeated training with large amounts of data, their application to flame detection is limited. The cost of acquiring training data remains a bottleneck hindering the application of deep learning technologies, particularly those related to object detection, in flame detection.

[0004] After obtaining raw image data, detection requires two major steps: object classification and location. Both steps require accurate annotations for deep learning neural network training. While category annotation requires manual selection of categories, location annotation, which requires manual box drawing, is more tedious and time-consuming, accounting for the vast majority of data generation costs. This, in turn, limits the widespread application of deep learning object detection methods in flame detection scenarios. Summary of the Invention

[0005] To address the shortcomings of the aforementioned prior art, the present invention aims to provide a multi-stage weakly supervised flame position detection method based on CAM. This method can infer flame position by collecting only flame and non-flame images and relying on class-labeled images, thus avoiding reliance on position labels and significantly reducing the data acquisition cost of deep learning target detection methods in the flame detection field.

[0006] The technical solutions specifically adopted in the present invention are as follows:

[0007] A multi-stage weakly supervised flame position detection method based on CAM comprises the following steps:

[0008] S1. Obtain a training set, wherein the training set contains two types of image data: one containing flames and the other not containing flames, and pre-label each image to determine whether it contains flames;

[0009] S2. Using Resnet as the backbone network, a multi-layer feature fusion structure is added to the backbone network to form a CAM heat map, and the CAM heat map is input into the classification head to perform a binary classification of whether the image contains flames, thereby forming a classification model; using the training set, the classification model is trained for flame classification using the top-level features and the multi-layer fusion features;

[0010] S3. Generate a CAM heat map corresponding to each image in the training set using the trained classification model. Use the natural breakpoint method to determine the background threshold for the thermal value of each point in the CAM heat map, thereby obtaining the flame target range. Then, obtain the flame position box corresponding to each image in the training set by taking the circumscribed rectangle.

[0011] S4. Using the obtained flame position frame as the label, the retinanet fully supervised object detection network is iteratively trained using the training set until convergence. The trained retinanet fully supervised object detection network is directly used for flame position detection in the image.

[0012] Preferably, in said S1, when the image data in the training set are annotated, only whether flames are contained is annotated, without annotating the specific position of the flames.

[0013] Preferably, the specific steps of S2 are as follows:

[0014] S201, using Resnet as the backbone network, and adding a multi-layer feature fusion structure based on the last three layers of Resnet, performing global average pooling (GAP) on the last three layers of feature maps output by Resnet, and then uniformly adjusting them to 128-dimensional vectors through 1×1 convolution, and multiplying the corresponding positions of the three 128-dimensional vectors to obtain a fusion vector; the fusion vector is adjusted to the same dimension size as the last three layers of feature maps output by Resnet through 1×1 convolution, and then multiplied with the corresponding feature maps, and then the three feature maps obtained by multiplication are uniformly adjusted to 128 dimensions and the same size through 1×1 convolution or a combination of 1×1 convolution and nearest neighbor upsampling, and finally the adjusted three feature maps are multiplied at corresponding positions to obtain a fusion feature map; the fusion feature map is subjected to 1×1 convolution to obtain a CAM heat map;

[0015] S202, connect the classification head at the end of the backbone network and the multi-layer feature fusion structure for joint training, wherein the fused feature map is globally average pooled and passed through a 1×1 convolution layer to obtain a first category vector. At the same time, the feature map output by the last layer of Resnet is globally average pooled and passed through the same 1×1 convolution layer to obtain a second category vector. According to the annotation labels in the training set, the classification losses loss1 and loss2 are calculated for the first category vector and the second category vector respectively, and the two parts of the loss are weighted as the total loss Loss to perform network parameter back propagation optimization, and the weighted weights of the two parts of the loss are dynamically adjusted with the training rounds, as follows:

[0016]

[0017] Where e represents the current training round, and E represents the upper limit of the dynamically adjusted round.

[0018] Preferably, the upper limit E of the dynamic adjustment rounds is set to 20 rounds.

[0019] Preferably, the last three layers of feature maps output by the Resnet are the three layers of feature maps output by the last three stages of the Resnet.

[0020] Preferably, the specific steps of S3 are as follows:

[0021] S301. Each image in the training set is input into the trained classification model. A CAM heat map is generated by the multi-layer feature fusion structure. The natural breakpoint method is used to perform one-dimensional clustering of the heat values ​​with a number of 2 categories, corresponding to the target and background respectively. The heat map is then converted into a binary map according to the breakpoint value to obtain the target area.

[0022] S302 , taking a circumscribed rectangle of the target area in the binarized image corresponding to each image in the training set, obtaining a target position frame, and recording it as a flame target position label corresponding to each image.

[0023] Preferably, in S4, the retinanet target detection network is trained according to the training set with flame target position labels obtained in S3, and the training process is no less than 30 epochs, and the learning rate is reduced in each round until the model converges. The retinanet target detection network finally obtained is directly used for flame position detection of unlabeled images.

[0024] Compared with the prior art, the present invention has the following beneficial effects:

[0025] 1. The present invention generates CAM heat maps through a multi-layer feature fusion structure, and through the joint flame classification training of the top-level features of the backbone network and the multi-layer fusion features, the model can accurately extract the CAM heat map for target positioning, thereby achieving flame target positioning accuracy close to that under full supervision conditions without using weak supervision of position labels.

[0026] 2. The present invention greatly reduces the data acquisition cost during flame detection and reduces the difficulty of applying deep learning methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 Schematic diagram of the steps of the flame position detection method of the present invention.

[0028] Figure 2 This is a structural diagram of a classification model according to an embodiment of the present invention.

[0029] Figure 3 This is a flame target detection effect diagram of an embodiment of the present invention. DETAILED DESCRIPTION

[0030] In order to make the above-mentioned objects, features and advantages of the present invention more clearly understood, the specific embodiments of the present invention are described in detail below with reference to the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways than those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below. The technical features in the various embodiments of the present invention can be combined accordingly without conflicting with each other.

[0031] In the description of the present invention, it should be understood that the terms "first" and "second" are used solely for descriptive purposes and are not to be construed as indicating or implying relative importance or implicitly specifying the number of technical features being described. Therefore, features defined as "first" or "second" may explicitly or implicitly include at least one of such features.

[0032] like Figure 1 As shown, in a preferred embodiment of the present invention, a multi-stage weakly supervised flame position detection method based on CAM is provided, which includes the following steps:

[0033] S1. Obtain a training set, which contains two types of image data: one containing flames and the other not containing flames, and pre-label each image to determine whether it contains flames.

[0034] It should be noted that, in the embodiment of the present invention, when the image data in the training set are annotated, only whether flames are contained is annotated, without annotating the specific location of the flames, thereby greatly reducing the time and cost of annotation.

[0035] In step S2, using ResNet as the backbone network, a multi-layer feature fusion structure is added to the backbone network to form a CAM heat map. The CAM heat map is then input into the classification head for binary classification of whether the image contains flames, thereby forming a classification model. The classification model is trained on the training set obtained in step S1 to jointly classify flames using top-level features and multi-layer fusion features.

[0036] In an embodiment of the present invention, the specific sub-steps of step S2 are as follows:

[0037] S201. Resnet is used as the backbone network, and a multi-layer feature fusion structure is added on the basis of the last three layers of Resnet. Global average pooling (GAP) is performed on the last three layers of feature maps output by Resnet, and then they are uniformly adjusted to 128-dimensional vectors through 1×1 convolution. The three 128-dimensional vectors are multiplied at corresponding positions to obtain a fusion vector; the fusion vector is adjusted to the same dimension size as the last three layers of feature maps output by Resnet through 1×1 convolution and then multiplied with the corresponding feature maps. The three feature maps obtained by the multiplication are then uniformly adjusted to 128 dimensions and the same size through 1×1 convolution or a combination of 1×1 convolution and nearest neighbor upsampling. Finally, the adjusted three feature maps are multiplied at corresponding positions to obtain a fusion feature map; the fusion feature map is subjected to 1×1 convolution to obtain a CAM heat map.

[0038] It should be noted that, in the embodiment of the present invention, the last three layers of feature maps output by the above Resnet are the three layers of feature maps output by the last three stages of Resnet. The specific classification model structure is as follows Figure 2As shown in the figure, the last three feature maps of the ResNet output are recorded as the first feature map of 512×56×56, the second feature map of 1024×28×28, and the third feature map of 2048×14×14. After GAP and 1×1 convolution, the first, second, and third feature maps are uniformly resized into three 128×1×1 vectors. These three vectors are multiplied at corresponding positions to form a 128×1×1 fused vector. The fused vector is then resized into a first intermediate vector of 512×1×1, a second intermediate vector of 1024×1×1, and a third intermediate vector of 2048×1×1 through 1×1 convolution. The first, second, and third intermediate vectors are then multiplied at corresponding positions with the first, second, and third feature maps to form a fourth feature map of 512×56×56, a fifth feature map of 1024×28×28, and a sixth feature map of 2048×14×14. The fourth, fifth, and sixth feature maps are then unified into three feature maps of 128×56×56 through 1×1 convolution and nearest neighbor upsampling (Upsample) (of which the 512×56×56 fourth feature map only requires 1×1 convolution and no upsampling is required). These three 128×56×56 feature maps are multiplied at corresponding positions to obtain a 128×56×56 fused feature map. The fused feature map is subjected to 1×1 convolution to obtain a 2×56×56 CAM heat map.

[0039] S202. Connect the classification head at the end of the backbone network and the multi-layer feature fusion structure for joint training. On the one hand, global average pooling is performed on the fused feature map, and the first category vector is obtained through a 1×1 convolution layer. On the other hand, global average pooling is performed on the feature map output by the last layer of Resnet, and the second category vector is obtained through the same 1×1 convolution layer. Thus, based on the labeled labels in the training set, the classification loss loss1 and loss2 are calculated for the first category vector and the second category vector respectively. The two parts of the loss are weighted as the total loss Loss for network parameter backpropagation optimization, and the weighted weights of the two parts of the loss are dynamically adjusted with the training rounds, as follows:

[0040]

[0041] Where e represents the current training round, and E represents the upper limit of the dynamically adjusted round.

[0042] In an embodiment of the present invention, the upper limit E of the dynamic adjustment rounds is set to 20 rounds.

[0043] S3. Use the trained classification model to generate the CAM thermal map corresponding to each image in the training set, and use the natural breakpoint method to determine the background threshold for the thermal value of each point in the CAM thermal map, so as to obtain the flame target range. Then, the flame position box corresponding to each image in the training set is obtained by taking the circumscribed rectangle.

[0044] In an embodiment of the present invention, the specific sub-steps of step S3 are as follows:

[0045] S301. Each image in the training set is input into the trained classification model. A CAM heat map is generated by a multi-layer feature fusion structure. The natural breakpoint method is used to perform one-dimensional clustering of the heat values ​​with a category number of 2, corresponding to the target and background respectively. The heat map is then converted into a binary map according to the breakpoint value to obtain the target area.

[0046] S302 , taking a circumscribed rectangle of the target area in the binarized image corresponding to each image in the training set, obtaining a target position frame, and recording it as a flame target position label corresponding to each image.

[0047] S4. Using the obtained flame position frame as the label, the retinanet fully supervised object detection network is iteratively trained using the training set until convergence. The trained retinanet fully supervised object detection network is directly used for flame position detection in the image.

[0048] In an embodiment of the present invention, the specific implementation steps of the above step S4 are as follows:

[0049] Based on the training set with flame target position labels obtained in S3, the RetinaNet target detection network is trained with full supervision, and the training process is no less than 30 epochs. The learning rate is reduced in each round until the model converges. The final RetinaNet target detection network is directly used for flame position detection in unlabeled images.

[0050] After verification on the collected flame detection dataset, Figure 3 The detection results in some example images show that it can accurately detect the flame area in different scenes. In addition, the positioning accuracy of the RetinaNet target detection network obtained by the method of the present invention on the entire data set is shown in Table 1 below. Correct positioning is defined as the IOU between the prediction box and the target box being greater than 0.5, and the positioning accuracy is the proportion of correctly positioned images in the test set to the total proportion of the test set. For comparison, this embodiment also uses data with manually labeled flame target position labels to train the RetinaNet target detection network (referred to as fully supervised RetinaNet in Table 1) to compare and demonstrate the effect of the present invention (referred to as "the method of the present invention" in Table 1).

[0051] Table 1 The positioning accuracy of the present invention on the data set

[0052] type Full supervision retinanet Method of the present invention Positioning accuracy 0.7248 0.6412 AP 0.6746 0.5811

[0053] As shown in Table 1, the RetinaNet target detection network, obtained through multi-stage weakly supervised training, achieves positioning accuracy close to that of a fully supervised network. This demonstrates that the method of the present invention can infer flame positions by collecting only flame and non-flame images and relying on class-labeled images, thus avoiding reliance on position labels and significantly reducing the data acquisition cost for deep learning target detection methods in the flame detection field.

[0054] The embodiment described above is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Persons skilled in the art may make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, any technical solution obtained by equivalent substitution or equivalent transformation falls within the scope of protection of the present invention.

Claims

1. A multi-stage weakly supervised flame position detection method based on CAM, characterized in that: The following steps are involved: S1. Obtain a training set, wherein the training set contains two types of image data: one containing flames and the other not containing flames, and pre-label each image to determine whether it contains flames; S2. Using Resnet as the backbone network, a multi-layer feature fusion structure is added to the backbone network to form a CAM heat map, and the CAM heat map is input into the classification head to perform a binary classification of whether the image contains flames, thereby forming a classification model; using the training set, the classification model is trained for flame classification using the top-level features and the multi-layer fusion features; S3. Generate a CAM heat map corresponding to each image in the training set using the trained classification model. Use the natural breakpoint method to determine the background threshold for the thermal value of each point in the CAM heat map, thereby obtaining the flame target range. Then, obtain the flame position box corresponding to each image in the training set by taking the circumscribed rectangle. S4. Using the obtained flame position frame as the label, the retinanet fully supervised object detection network is iteratively trained using the training set until convergence. The trained retinanet fully supervised object detection network is directly used for flame position detection in the image.

2. The multi-stage weakly supervised flame position detection method based on CAM according to claim 1 is characterized in that: In S1, when the image data in the training set are annotated, only whether the flame is contained is annotated, without annotating the specific location of the flame.

3. The multi-stage weakly supervised flame position detection method based on CAM according to claim 1 is characterized in that: The specific steps of S2 are as follows: S201, using Resnet as the backbone network, and adding a multi-layer feature fusion structure based on the last three layers of Resnet, performing global average pooling on the last three layers of feature maps output by Resnet, and then uniformly adjusting them to 128-dimensional vectors through 1×1 convolution, and multiplying the corresponding positions of the three 128-dimensional vectors to obtain a fusion vector; the fusion vector is adjusted to the same dimension size as the last three layers of feature maps output by Resnet through 1×1 convolution, and then multiplied with the corresponding feature maps, and then the three feature maps obtained by multiplication are uniformly adjusted to 128 dimensions and the same size through 1×1 convolution or a combination of 1×1 convolution and nearest neighbor upsampling, and finally the adjusted three feature maps are multiplied at corresponding positions to obtain a fusion feature map; the fusion feature map is subjected to 1×1 convolution to obtain a CAM heat map; S202, connect the classification head at the end of the backbone network and the multi-layer feature fusion structure for joint training, wherein the fused feature map is globally average pooled and passed through a 1×1 convolution layer to obtain a first category vector. At the same time, the feature map output by the last layer of Resnet is globally average pooled and passed through the same 1×1 convolution layer to obtain a second category vector. According to the annotation labels in the training set, the classification losses loss1 and loss2 are calculated for the first category vector and the second category vector respectively, and the two parts of the loss are weighted as the total loss Loss to perform network parameter back propagation optimization, and the weighted weights of the two parts of the loss are dynamically adjusted with the training rounds, as follows: Where e represents the current training round, and E represents the upper limit of the dynamically adjusted round.

4. The multi-stage weakly supervised flame position detection method based on CAM according to claim 3 is characterized in that: The upper limit E of the dynamic adjustment rounds is set to 20 rounds.

5. The multi-stage weakly supervised flame position detection method based on CAM according to claim 1 is characterized in that: The last three layers of feature maps output by the Resnet are the three layers of feature maps output by the last three stages of the Resnet.

6. The multi-stage weakly supervised flame position detection method based on CAM according to claim 1 is characterized in that: The specific steps of S3 are as follows: S301. Each image in the training set is input into the trained classification model. A CAM heat map is generated by the multi-layer feature fusion structure. The natural breakpoint method is used to perform one-dimensional clustering of the heat values ​​with a number of 2 categories, corresponding to the target and background respectively. The heat map is then converted into a binary map according to the breakpoint value to obtain the target area. S302 , taking a circumscribed rectangle of the target area in the binarized image corresponding to each image in the training set, obtaining a target position frame, and recording it as a flame target position label corresponding to each image.

7. The multi-stage weakly supervised flame position detection method based on CAM according to claim 1 is characterized in that: In S4, the retinanet target detection network is trained based on the training set with flame target position labels obtained in S3, and the training process is no less than 30 epochs, and the learning rate is reduced in each round until the model converges. The retinanet target detection network finally obtained is directly used to detect the flame position of the unlabeled image.

Images