Flame smoke visual monitoring network for multiple scenes

Abstract

The invention discloses a multi-scene-oriented flame smoke visual monitoring network, and relates to the technical field of neural networks, a pyramid pooling module SPP is introduced into a Backbone output end, the receptive field of the network is expanded by means of pooling kernels of different sizes, and the utilization of context feature information is enhanced. A lightweight feature fusion module C2fFR is designed, the C2fFR is used to reduce model parameter quantity and calculation quantity, meanwhile, the attention of the model to smoke flame is improved, useless information such as background is inhibited, and the feature extraction capability is improved. An improved K-means clustering algorithm is adopted to optimize a prior frame to better adapt to a complex scene, a Complete-loU loss function is adopted, and the function considers the correlation between the center of an object and the width and height when calculating a coordinate error. Through the improvement, the monitoring network is obviously improved in the aspects of fire detection accuracy and robustness.

Description

Technical Field

[0001] The present application relates to the field of neural network technology, and in particular to a flame and smoke visual monitoring network for multiple scenarios. Background Art

[0002] Fire, as a common disaster, poses a serious threat to social economy and human life. With the deepening of socialization, the damage caused by fire is becoming more and more serious. Therefore, the development of efficient fire detection technology is of great practical significance. Compared with traditional sensors and other detection methods, image-based fire detection technology has higher detection accuracy and a wider monitoring range.

[0003] Chinese patent CN2020104630443 discloses a fire smoke detection method and device, which determines the final detection result of the flame smoke according to the dynamic detection result and static detection result of the flame smoke in each frame of the detection video. Chinese patent CN2022115907336 discloses a fire smoke detection method based on ODConvBS-YOLOv5s, annotates the acquired smoke image through labelimgs to make a data set, and improves YOLOv5 for smoke and flame detection. Chinese patent CN2022114223796 discloses a flame early smoke detection method based on improved DeeplabV3+, constructs a specific data set and uses the DeeplabV3+ model to identify smoke, combines Unet to optimize DeeplabV3+ and improves the accuracy of smoke recognition. The study found that the existing flame smoke detection method and device are still empirical in threshold selection, which limits the robustness and generalization ability of the algorithm to a certain extent.

[0004] However, traditional image detection methods rely on manual feature selection, resulting in poor robustness. Deep learning technology has attracted widespread attention due to its ability to automatically learn features, but in practical applications, existing fire detection still faces problems such as low small target detection rate, insufficient detection accuracy in complex scenes, and detection delays due to defects such as weak small-scale feature extraction capabilities of classical detection algorithms, structural limitations of algorithm models, and high complexity of the scenes in which the detection objects are located. Summary of the invention

[0005] In order to solve the above technical problems, this application proposes the following technical solutions:

[0006] In the first aspect, an embodiment of the present application provides a flame and smoke visual monitoring network for multiple scenarios, including: a backbone network Backbone, a neck network Neck and a head network Head, wherein the Backbone adopts a C2fFR module as a basic building block, extracts features by adopting a series of convolution and deconvolution layers, and also uses residual connections and bottleneck structures to reduce the size of the network and improve performance; the Neck uses multi-scale feature fusion technology to fuse feature maps from different stages of the Backbone to enhance feature representation capabilities; the Head includes a priori box generated by a K-means clustering algorithm and a classification module, the priori box is used to generate detection results, and the classification module uses global average pooling to classify each feature map.

[0007] In a possible implementation, the Backbone is provided with convolution layers Conv of different sizes, a feature fusion module C2fFR, a spatial pyramid pooling module SPPF and a pyramid pooling module SPP; the high-frequency wavelet information of the original image is introduced into the receiving end of the backbone network, and is output to Neck through a series of convolution and pooling operations of the backbone. C2fFR is the main module of the backbone, and each branch is parallel to a sequence of concat modules of Neck. At the same time, the feature map is introduced at the tail output end of the Backbone and transmitted to SPP.

[0008] In a possible implementation, the tail output end of the Backbone introduces a feature map and passes it to the SPP for three convolution operations as shown in the following formula:

[0009] (f*k)(i,j)=Σ m ∑ n f(i+m,j+n)·k(m,n)

[0010] Among them, f represents the input feature map, k represents the convolution kernel, (i, j) represents the coordinates of the output feature map, and (m, n) represents the coordinates of the convolution kernel;

[0011] In order to improve the detection accuracy of small targets, dynamic snake convolution is introduced in the convolution operation:

[0012] K={(x-1,y-1),(x-1,y),K,(x+1,y+1)}

[0013] Among them, (x, y) is the center coordinate of the convolution kernel, and (x+m, y+n) is the offset relative to the center;

[0014] In order to make the convolution kernel more flexible to focus on the geometric features of the target image, an offset Δ is introduced:

[0015] Δ=Ki ±c=(x i ±c,y i ±c)

[0016] Where c represents the horizontal distance from the center grid, and Δ is the offset;

[0017] Since the offset Δ is usually a fraction, the convolution kernel needs to be adjusted through bilinear interpolation. The formula is as follows:

[0018] K=ΣK′B(K′,K)·K′

[0019] Wherein, B(K′,K) represents the weight function of bilinear interpolation.

[0020] In one possible implementation, the attention mechanism LSKA is used to improve the accuracy of small target detection, and the long-term adaptability and dependency are obtained through the decomposition of large kernel convolution operations, while reducing the computational complexity. The structure of LSKA is as follows:

[0021] LSKA=DepthwiseConv(PointwiseConv(DepthwiseConv(InputSize)))

[0022] Among them, DepthwiseConv represents depth-separable convolution, and PointwiseConv represents 1x1 convolution;

[0023]

[0024] Among them, InputSize is the size (width or height) of the input feature map; KernelSize is the size of the convolution; Padding is the number of zeros added to the edge of the input feature map; Stride is the stride of the convolution operation.

[0025] In a possible implementation, the OutputSize sizes are 512×1×1, 1024×3×3, and 512×1×1, respectively. The above results are subjected to the maximum pooling operation maxpool using pooling kernels of four sizes: 1×1, 2×2, 3×3, and 5×5; 1×1 is the direct connection from input to output as the first output, 2×2 represents the pooling kernel of the second branch, 3×3 and 5×5 are the pooling kernels of the third branch and the fourth branch, and finally the obtained feature maps are stacked and Concat is performed to aggregate the features extracted at different scales. The size of the maximum pooling kernel is equal to the size of the feature map, so as to ensure that partial features and global feature information of the image are combined.

[0026] In one possible implementation, C2fFR in Backbone is lightweight. The backbone structure of C2fFR, FRepBlock, consists of PRConv, two 1x1 convolutions and BN layers, and ReLU activation function, which is an inverted residual structure. PRConv divides the obtained number of channels C into partial channels C, performs multi-branch parallel convolution operations on partial input channels C, and splices them together after a 1x1 convolution, a 3x3 convolution, and an identity mapping. The feature extraction capability of the network is improved through multi-branch parallel convolution operations. During inference, the structure is re-parameterized into a single-path structure to speed up the inference speed, while the remaining channels are not convolved. The high similarity between feature channels is used to retain the feature information of the previous layer, and then the two channels are spliced ​​and output. Finally, the number of channels in the output feature map is the same as the number of input channels.

[0027] In one possible implementation, PRConv reduces computational cost by performing partial channel convolution. The FLOPs of PRConv is:

[0028]

[0029] Where h, w, k, and c represent the height, width, convolution kernel size, and number of convolution channels of the feature map, respectively;

[0030] When the number of channels for partial convolution is c p When it is 1 / 4, When , the ratio of FLOPs of PRConv to FLOPs of regular convolution is:

[0031]

[0032] In one possible implementation, the improved intersection over union (IoU) is used to improve the K-means clustering algorithm and the improved K-means clustering algorithm is used to iteratively generate anchor boxes of various sizes:

[0033] d b,c =1-IoU b,c

[0034] Among them, d b,c Indicates the generated new distance, b represents the real box, and c represents the cluster center. It can be seen that the larger the IoU value, the smaller the d b,c The smaller;

[0035] The DIoU loss function is used to improve the IoU by considering the overlapping area, center point distance and scale of the predicted box and the real box, so as to ensure the accuracy of the overlap between the predicted box and the real box in target detection:

[0036]

[0037] Among them, ρ 2 (b,b gt ) is the predicted box center b and the real box center b gt The square of the Euclidean distance between them, c is the diagonal length of the minimum closure area that can contain both the predicted box and the true box.

[0038] In a possible implementation, a K-means clustering algorithm is used to generate a priori boxes, including:

[0039] At the beginning of clustering, k real bounding boxes are randomly selected as the initial anchor boxes, and the IoU of all bounding boxes and anchor boxes is obtained. The larger the IoU, the smaller the error.

[0040] Assign each bounding box to the anchor box with the smallest error in turn;

[0041] Finally, the width and height values ​​of all bounding boxes in each anchor box are calculated as the new anchor box.

[0042] Repeat the iterative update until the anchor boxes to which all real boxes belong are the same as those assigned last time, stop updating, and save the final anchor box as the prior box after clustering.

[0043] In one possible implementation, in order to improve the convergence speed of the model, the WIoU loss function formula used is as follows:

[0044]

[0045] L IOU =1-IOU

[0046] Among them, (x, y) and (x gt ,y gt ) are the coordinates W of the center points of the anchor box and the target box respectively. g and H g is the size of the minimum bounding box, β is defined as the outlier degree, and the quality of the anchor box is described by introducing the outlier degree parameter, which is negatively correlated with the quality of the anchor box;

[0047] The outlier degree is calculated as follows:

[0048]

[0049] in, is the gradient gain of the monotonic focusing coefficient, dynamically adjusts the highest gradient gain, and * indicates that the changes will be continuously calculated according to the situation of each target detection during the training process. is the sliding average of momentum m, and introduce This means that the maximum gradient gain can be dynamically adjusted according to the training process. The calculation formula of momentum m is as follows:

[0050]

[0051] Among them, t is the value of epoch, n is the value of batches, and the significance of introducing momentum m is that after t rounds of training, WIoU assigns small gradient gains to low-quality anchor boxes to reduce harmful gradients.

[0052] In the embodiment of the present application, the pyramid pooling module SPP is introduced at the output end of Backbone, and the receptive field of the network is expanded with the help of 4 pooling kernels of different sizes, thereby enhancing the utilization of contextual feature information. A lightweight feature fusion module C2fFR is designed. C2fFR is used to reduce the number of model parameters and the amount of calculation while increasing the model's attention to smoke and flames, suppressing useless information such as background, and improving feature extraction capabilities. An improved K-means clustering algorithm is used to optimize the prior frame to better adapt to complex scenes, and a Complete-loU loss function is used, which takes into account the correlation between the center of the object and the width and height when calculating the coordinate error. Through these improvements, the accuracy and robustness of the monitoring network in fire detection have been significantly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 A schematic diagram of a flame and smoke visual monitoring network for multiple scenarios provided in an embodiment of the present application;

[0054] Figure 2 A schematic diagram of the Backbone network structure provided in the embodiment of the present application;

[0055] Figure 3 A schematic diagram of the Neck network structure provided in an embodiment of the present application;

[0056] Figure 4 A schematic diagram of the Head network structure provided in an embodiment of the present application;

[0057] Figure 5 A schematic diagram of the YOLOv8+SPP network structure provided in an embodiment of the present application;

[0058] Figure 6 A comparison diagram of the effects provided by the embodiments of the present application;

[0059] Figure 7 A schematic diagram of a lightweight C2fFR structure provided in an embodiment of the present application;

[0060] Figure 8 A schematic diagram of grid prediction provided in an embodiment of the present application;

[0061] Fig. 9 A relationship diagram between the number of prior boxes K and AgIoU provided in an embodiment of the present application. DETAILED DESCRIPTION

[0062] The present solution is described below in conjunction with the accompanying drawings and specific implementation methods.

[0063] join Figure 1 The flame and smoke visual monitoring network for multiple scenarios provided in this embodiment includes: a backbone network Backbone, a neck network Neck and a head network Head. Figure 2-Figure 4 The enlarged diagrams of the network structures of Backbone, Neck and Head are shown in sequence. In this embodiment, the Backbone uses the C2f module as the basic building block, extracts features by using a series of convolution and deconvolution layers, and also uses residual connections and bottleneck structures to reduce the size of the network and improve performance; the Neck uses multi-scale feature fusion technology to fuse feature maps from different stages of the Backbone to enhance feature representation capabilities; the Head includes a priori box generated by the K-means clustering algorithm and a classification module, the priori box is used to generate detection results, and the classification module uses global average pooling to classify each feature map.

[0064] The Backbone is provided with convolution layers Conv of different sizes, feature fusion module C2fFR, spatial pyramid pooling module SPPF and pyramid pooling module SPP. The high-frequency wavelet information of the original image is introduced into the receiving end of the backbone network, and is output to Neck through a series of convolution and pooling operations of the backbone. C2fFR is the main module of the backbone, and each branch is parallel to a sequence of concat modules of Neck. At the same time, the feature map is introduced at the tail output end of the Backbone and transmitted to SPP. Neck fuses the feature maps corresponding to C2fFR from different stages of the backbone to further enhance the feature representation capability.

[0065] The improvement of the main architecture of the network in this embodiment is divided into two parts: adding a spatial pyramid pooling module SPP, adding a lightweight feature fusion module C2fFR, and improving the loss function.

[0066] After YOLOv8 downsamples the input image 5 times, it does not provide any deep features required for small target detection. Therefore, YOLOv8 has poor detection effect on small targets. In view of the above disadvantages, for flame and smoke detection in the field of fire detection, early detection of fire is crucial to reduce the damage caused by fire. The core principle of fire prevention is to "extinguish the fire before it spreads into a large-scale fire." Therefore, the detection of small targets is very important in fire detection.

[0067] Therefore, this embodiment improves the original CSPDarknet53 structure, adds an SPP module after its head, and continues to process the 20×20 feature map with the best information. By using the feature information of the previous and next networks, the robustness of the model to scale changes is enhanced, and finally the purpose of enhancing the receptive field of the entire network and reducing overfitting is achieved, so that the network can increase the detection effect of small targets. Figure 2 shown.

[0068] Depend on Figure 5 and Figure 6 By comparison, we can see that the 20×20 feature map with the best information is selected for further processing, and the 20×20 feature map is subjected to three convolution operations as shown in the following formula:

[0069] (f*k)(i,j)=Σ m ∑ n f(i+m,j+n)·k(m,n)

[0070] Among them, f represents the input feature map, k represents the convolution kernel, (i, j) represents the coordinates of the output feature map, and (m, n) represents the coordinates of the convolution kernel.

[0071] In order to improve the detection accuracy of small targets, dynamic snake convolution is introduced in the convolution operation:

[0072] K={(x-1,y-1),(x-1,y),K,(x+1,y+1)}

[0073] Among them, (x, y) is the center coordinate of the convolution kernel, and (x+m, y+n) is the offset relative to the center.

[0074] In order to make the convolution kernel more flexible to focus on the geometric features of the target image, an offset Δ is introduced. The offset is calculated as follows:

[0075] Δ=K i ±c=(x i ±c,y i ±c)

[0076] Where c is the horizontal distance from the center grid and Δ is the offset.

[0077] Since the offset Δ is usually a fraction, the convolution kernel needs to be adjusted through bilinear interpolation. The formula is as follows:

[0078] K=∑K′B(K′,K)·K′

[0079] Wherein, B(K′,K) represents the weight function of bilinear interpolation.

[0080] The attention mechanism LSKA is used to improve the accuracy of small target detection, and the large kernel convolution operation is decomposed to obtain long-term adaptability and dependency, while reducing the computational complexity. The structure of LSKA is as follows:

[0081] LSKA=DepthwiseConv(PointwiseConv(DepthwiseConv(InputSize)))

[0082] Among them, DepthwiseConv represents depth-wise separable convolution, and PointwiseConv represents 1x1 convolution.

[0083]

[0084] Among them, InputSize is the size (width or height) of the input feature map; KernelSize is the size of the convolution; Padding is the number of zeros added to the edge of the input feature map; Stride is the stride of the convolution operation.

[0085] Through the above targeted convolution operations, the OutputSize obtained are 512×1×1, 1024×3×3, and 512×1×1 respectively. The above results are subjected to the maximum pooling operation (maxpool) using pooling kernels of four sizes: 1×1, 2×2, 3×3, and 5×5. Among them, 1×1 directly connects the input to the output as the first output. 2×2 represents the pooling kernel of the second branch, and similarly 3×3 and 5×5 are used as the pooling kernels of the third and fourth branches. Finally, the obtained feature maps are stacked (Concat) to aggregate the features extracted at different scales. The size of the maximum pooling kernel is equal to the size of the feature map to ensure that partial features and global feature information of the image are combined.

[0086] As can be seen from the above effect diagram, by adding the compensatory pyramid pooling module ASPP at the output of the Backbone (backbone network), it can be seen that after retaining the resolution of the original input feature map, Figure 4 Compared with the first two columns, the third column shows improved results in fire monitoring in complex scenes and detection of smaller-scale flames.

[0087] In view of the problem that the existing feature fusion module is easily disturbed by the background during feature extraction, a lightweight feature fusion module C2fFR is specially designed, and the structure is shown in the figure. C2fFR is a module redesigned based on FasterBlock. Its core is to enhance the expression of target features while ensuring lightweight and high-speed feature extraction processing, so that the model pays more attention to foreground targets and suppresses background interference.

[0088] See also Figure 7 The main structure of C2fFR, FRepBlock, consists of PRConv (Partial Rep Conv), two 1x1 convolutions and BN layers, and ReLU activation function. It is an inverted residual structure. PRConv divides the obtained number of channels C into partial channels C, and performs multi-branch parallel convolution operations on partial input channels C, respectively. After a 1x1 convolution, a 3x3 convolution and an identity mapping, they are spliced ​​together. The feature extraction ability of the network is improved through multi-branch parallel convolution operations. During inference, the structure is re-parameterized into a single-channel structure to speed up the inference speed, while the remaining channels are not convolved. The high similarity between feature channels is used to retain the feature information of the previous layer. Then the two channels are spliced ​​and output. The number of channels of the output feature map is the same as the number of input channels. Since the feature maps have high similarity between different channels, C2fFR can effectively utilize the redundancy of feature maps, optimize the computational cost, and improve the ability to extract spatial features through multi-branch parallel convolution.

[0089] PRConv can effectively reduce the computational cost by performing partial channel convolution. In terms of computational cost, the FLOPS of conventional convolution is:

[0090] FLOPs Conv =h×w×k 2 ×c 2

[0091] The FLOPs of PRConv are:

[0092]

[0093] Where h, w, k, and c represent the height, width, convolution kernel size, and number of convolution channels of the feature map, respectively.

[0094] When the number of channels for partial convolution is c p When it is 1 / 4, When , the ratio of FLOPs of PRConv to FLOPs of regular convolution is:

[0095]

[0096] When the conventional convolution kernel is 3, the FLOPS of PRConv is only 5 / 72 of that of conventional convolution. The above data proves that C2fFR can effectively reduce the number of model parameters, making the model lighter and improving the detection speed. At the same time, the multi-branch parallel convolution operation improves the network's extraction of smoke and flame features.

[0097] This embodiment uses the WIoU loss function instead of the CIoU loss function, which can better solve the problem of sample imbalance and improve the convergence speed of the model. This method adopts a dynamic non-monotonic focused gradient gain allocation strategy to reduce the competitive advantage of high-quality anchor frames and mitigate the adverse gradient effects caused by low-quality samples, so that the model has higher accuracy and builds a distance attention mechanism based on the distance metric.

[0098] The WIoU loss function formula is as follows:

[0099]

[0100] L IOU =1-IOU

[0101] Among them, (x, y) and (x gt ,y gt ) are the coordinates W of the center points of the anchor box and the target box respectively. g and H g is the size of the minimum bounding box, β is defined as the outlier degree, and the quality of the anchor box is described by introducing the outlier degree parameter, which is negatively correlated with the quality of the anchor box. The outlier degree is calculated as follows:

[0102]

[0103] in, is the gradient gain of the monotonic focusing coefficient, dynamically adjusts the maximum gradient gain, and L IoU The definition is the same, and the * here means that changes will be calculated continuously according to the situation of each target detection during training. is the sliding average of momentum m, and introduce This means that the maximum gradient gain can be dynamically adjusted according to the training process. The calculation formula of momentum m is as follows:

[0104]

[0105] Among them, t is the value of epoch, n is the value of batches, and the significance of introducing momentum m is that after t rounds of training, WIoU will allocate small gradient gains to low-quality anchor frames to reduce harmful gradients. This study will use the WIoU loss function instead of the CIoU loss function, which can better solve the problem of sample imbalance and improve the convergence speed of the model. This method adopts a dynamic non-monotonic focused gradient gain allocation strategy to reduce high-quality anchor frames.

[0106] The anchor-free detection head used in the original YOLOv8 and classic flame and smoke detection has certain accuracy defects in small target detection compared to anchor-based detection. To address this problem, the present invention presets anchor frames to adaptively match targets, and uses lightweight preset prior frames with the anchor-free detection head to greatly improve the efficiency of network positioning of target objects.

[0107] By adding the K-means clustering algorithm, the input image is initially divided into multiple non-overlapping small grids, and three feature maps with different resolutions are generated through the backbone network. The pixels of each feature map correspond to a specific area in the image. If the center point of the target object falls at the center of a grid, then the grid is responsible for detecting the object. The specific effect is as follows Figure 8 shown.

[0108] The initial prior frame is generated by the K-means clustering algorithm. A set of prior frames with different sizes and aspect ratios are preset for the center points of each feature area. There are 20 data sets, and there are 9 preset prior frames (anchors). The smallest sizes are (10, 13) and (16, 30), which are used to detect small targets. The prior frame sizes with the largest receptive field are (156, 198) and (373, 326). The target categories of this study are flames and smoke. In order to achieve fast and accurate flame and smoke detection, the true boxes (TrueBox) labeled with flames and smoke are analyzed, and the improved re-clustering method is used to better match the prior frames of this study’s data set, and then processed by the anchor-free detection head.

[0109] In the actual application of flame and smoke detection, clustering is not used for point clustering but for border clustering, which will produce errors. The larger the size of the entire frame, the greater the deviation, which leads to inaccurate or undetectable detection of small targets. Therefore, this study decided to use the improved intersection over union (IoU) to improve the K-means clustering algorithm and iteratively generate 9 anchor frames of different sizes with the help of the improved K-means clustering algorithm, as shown in the following formula:

[0110] d b,c =1-IoUb,c

[0111] Among them, d b,c Indicates the generated new distance, b represents the real box, and c represents the cluster center. It can be seen that the larger the IoU value, the smaller the d b,c The smaller.

[0112] The DIoU loss function is used to improve the intersection over union (IoU) by considering the overlap area, center point distance, and scale of the predicted box and the true box. This ensures the accuracy of the overlap between the predicted box and the true box in object detection:

[0113]

[0114] Among them, ρ 2 (b,b gt ) is the predicted box center b and the real box center b gt The square of the Euclidean distance between them, c is the diagonal length of the minimum closure area that can contain both the predicted box and the true box.

[0115] The specific steps of the clustering algorithm are as follows: At the beginning of clustering, k real boxes (bounding boxes) are randomly selected as the initial anchor box (anchor box), and the IoU of all bounding boxes and anchor boxes is obtained. The larger the IoU, the smaller the error. Assign each bounding box to the anchor box with the smallest error in turn. Finally, calculate the width and height median size of all bounding boxes in each anchor box as the new anchor box. Repeat the iterative update until the anchor box to which all real boxes belong is the same as the last assignment, and stop updating. Save the final anchor box as the prior box after clustering. The relevant processing formulas for the above steps are shown below:

[0116]

[0117] Among them, x i is a data point, c j is the cluster center, n is the dimension of the data, and d represents the distance.

[0118] The K-means algorithm is used to minimize the sum of squared errors (SSE), where the sum of squared errors is calculated as follows:

[0119]

[0120] Where k is the number of clusters, S i is the sample set in the i-th cluster, μ iis the cluster center of the ith cluster, and x is the sample point.

[0121] Calculate the average IoU (AgIoU) between the bounding box and the k anchor boxes in the sample, and the relationship results are as follows Fig. 9 Show.

[0122] When the k value increases, the AvgIoU value increases rapidly first and then increases slowly. The larger the AvgIoU is, the more consistent the obtained anchors are with the size of the real box. Considering the calculation accuracy and speed of the detection model, the optimal number of clusters k=9 is finally selected. The improved cluster center coordinates and prior box size are shown in Table 1:

[0123] Table 1 Relationship between cluster center coordinates and prior box size

[0124]

[0125] This embodiment evenly distributes anchor boxes of 9 different sizes to feature maps of 3 different scales. After 32 times downsampling, a 13×13 feature map with a larger receptive field is obtained. This feature map is used to detect larger flame and smoke targets, and anchor boxes of sizes 141×237, 236×150 and 312×283 are assigned to it. For the 26×26 feature map, its receptive field is moderate, so anchor boxes of sizes 62×57, 74×50 and 127×91 are assigned. Finally, for the 52×52 feature map with the smallest receptive field, we assign anchor boxes of sizes 13×23, 24×44 and 35×99 to detect smaller flame and smoke targets. This allocation strategy can effectively detect targets of different sizes while ensuring the accuracy of small-scale target detection to the greatest extent.

[0126] In the embodiments of the present application, "at least one" refers to one or more, and "plurality" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent the existence of A alone, the existence of A and B at the same time, and the existence of B alone. Among them, A and B can be singular or plural. The character " / " generally indicates that the previous and subsequent associated objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b and c can be represented by: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0127] The above is only a specific implementation of the present application. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. The protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A flame and smoke visual monitoring network for multiple scenarios, characterized in that: include: The backbone network Backbone, the neck network Neck and the head network Head, the Backbone adopts the C2fFR module as the basic building block, extracts features by adopting a series of convolution and deconvolution layers, and also uses residual connections and bottleneck structures to reduce the size of the network and improve performance; the Neck uses multi-scale feature fusion technology to fuse feature maps from different stages of the Backbone to enhance feature representation capabilities; the Head includes a priori box generated by the K-means clustering algorithm and a classification module, the priori box is used to generate detection results, and the classification module uses global average pooling to classify each feature map.

2. The multi-scenario flame and smoke visual monitoring network according to claim 1 is characterized in that: The Backbone is provided with convolution layers Conv of different sizes, feature fusion module C2fFR, spatial pyramid pooling module SPPF and pyramid pooling module SPP; the high-frequency wavelet information of the original image is introduced into the receiving end of the backbone network, and is output to Neck through a series of convolution and pooling operations of the backbone. C2fFR is the main module of the backbone, and each branch is parallel to a sequence of concat modules of Neck. At the same time, the feature map is introduced into the tail output end of the Backbone and transmitted to SPP.

3. The multi-scenario flame and smoke visual monitoring network according to claim 1 is characterized in that: The tail output end of the Backbone introduces the feature map and passes it to the SPP for three convolution operations as shown in the following formula: (f*k)(i,j)=∑ m Σ n f(i+m,j+n)·k(m,n) Among them, f represents the input feature map, k represents the convolution kernel, (i, j) represents the coordinates of the output feature map, and (m, n) represents the coordinates of the convolution kernel; In order to improve the detection accuracy of small targets, dynamic snake convolution is introduced in the convolution operation: K={(x-1,y-1),(x-1,y),K,(x+1,y+1)} Among them, (x, y) is the center coordinate of the convolution kernel, and (x+m, y+n) is the offset relative to the center; In order to make the convolution kernel more flexible to focus on the geometric features of the target image, an offset Δ is introduced: Δ=K i ±c=(x i ±c,y i ±c) Where c represents the horizontal distance from the center grid, and Δ is the offset; Since the offset Δ is usually a fraction, the convolution kernel needs to be adjusted through bilinear interpolation. The formula is as follows: K=ΣK′B(K′,K)·K′ Wherein, B(K′,K) represents the weight function of bilinear interpolation.

4. The multi-scenario flame and smoke visual monitoring network according to claim 3 is characterized in that: The attention mechanism LSKA is used to improve the accuracy of small target detection. The large kernel convolution operation is decomposed to obtain long-term adaptability and dependency, while reducing the computational complexity. The structure of LSKA is as follows: LSKA=DepthwiseConv(PointwiseConv(DepthwiseConv(InputSize))) Among them, DepthwiseConv represents depth-separable convolution, and PointwiseConv represents 1x1 convolution; Among them, InputSize is the size (width or height) of the input feature map; KernelSize is the size of the convolution; Padding is the number of zeros added to the edge of the input feature map; Stride is the stride of the convolution operation.

5. The multi-scenario flame and smoke visual monitoring network according to claim 4 is characterized in that: The output sizes are 512×1×1, 1024×3×3, and 512×1×1 respectively. The above results are subjected to the maximum pooling operation maxpool using pooling kernels of four sizes: 1×1, 2×2, 3×3, and 5×5. Among them, 1×1 means directly connecting the input to the output as the first output, 2×2 represents the pooling kernel of the second branch, 3×3 and 5×5 are used as the pooling kernels of the third and fourth branches, and finally the obtained feature maps are stacked and Concat is performed to aggregate the features extracted at different scales. The size of the maximum pooling kernel is equal to the size of the feature map, so as to ensure that the partial features and global feature information of the image are combined.

6. The multi-scenario flame and smoke visual monitoring network according to claim 2 is characterized in that: For the lightweight C2fFR in Backbone, the backbone structure FRepBlock of C2fFR consists of PRConv, two 1x1 convolutions and BN layers, and ReLU activation function, which is an inverted residual structure; PRConv divides the obtained number of channels C into partial channels C, and performs multi-branch parallel convolution operations on partial input channels C, which are spliced ​​together after a 1x1 convolution, a 3x3 convolution, and an identity mapping. The feature extraction capability of the network is improved through multi-branch parallel convolution operations. During inference, the structure is re-parameterized into a single-path structure to speed up the inference speed, while the remaining channels are not convolved. The high similarity between feature channels is used to retain the feature information of the previous layer, and then the two channels are spliced ​​and output. Finally, the number of channels in the output feature map is the same as the number of input channels.

7. The multi-scenario flame and smoke visual monitoring network according to claim 6 is characterized in that: PRConv reduces computational cost by performing partial channel convolution. The FLOPs of PRConv are: Where h, w, k, and c represent the height, width, convolution kernel size, and number of convolution channels of the feature map, respectively; When the number of channels for partial convolution is c p When it is 1 / 4, When , the ratio of FLOPs of PRConv to FLOPs of regular convolution is:

8. The multi-scenario flame and smoke visual monitoring network according to claim 1 is characterized in that: The improved intersection over union (IoU) is used to improve the K-means clustering algorithm and the improved K-means clustering algorithm is used to iteratively generate anchor boxes of various sizes: d b,c =1-IoU b,c Among them, d b,c Indicates the generated new distance, b represents the real box, and c represents the cluster center. It can be seen that the larger the IoU value, the smaller the d b,c The smaller; The DIoU loss function is used to improve the IoU by considering the overlapping area, center point distance and scale of the predicted box and the real box, so as to ensure the accuracy of the overlap between the predicted box and the real box in target detection: Among them, ρ 2 (b,b gt ) is the predicted box center b and the real box center b gt The square of the Euclidean distance between them, c is the diagonal length of the minimum closure area that can contain both the predicted box and the true box.

9. The multi-scenario flame and smoke visual monitoring network according to claim 8 is characterized in that: The K-means clustering algorithm is used to generate the prior frame, including: At the beginning of clustering, k real bounding boxes are randomly selected as the initial anchor boxes, and the IoU of all bounding boxes and anchor boxes is obtained. The larger the IoU, the smaller the error. Assign each boundingbox to the anchorbox with the smallest error in turn; Finally, the width and height values ​​of all boundingboxes in each anchorbox are calculated as the new anchorbox. Repeat the iterative update until the anchor boxes of all real boxes are the same as those assigned last time, stop updating, and save the final anchor box as the prior box after clustering.

10. The multi-scenario flame and smoke visual monitoring network according to claim 9 is characterized in that: In order to improve the convergence speed of the model, the WIoU loss function formula used is as follows: L IOU =1-IOU Among them, (x, y) and (x gt ,y gt ) are the coordinates W of the center points of the anchor box and the target box respectively. g and H g is the size of the minimum bounding box, β is defined as the outlier degree, and the quality of the anchor box is described by introducing the outlier degree parameter, which is negatively correlated with the quality of the anchor box; The outlier degree is calculated as follows: in, is the gradient gain of the monotonic focusing coefficient, dynamically adjusts the highest gradient gain, and * indicates that the changes will be continuously calculated according to the situation of each target detection during the training process. is the sliding average of momentum m, and introduce This means that the maximum gradient gain can be dynamically adjusted according to the training process. The calculation formula of momentum m is as follows: Among them, t is the value of epoch, n is the value of batches, and the significance of introducing momentum m is that after t rounds of training, WIoU assigns small gradient gains to low-quality anchor boxes to reduce harmful gradients.

Images