Structured lightweight fire smoke detection method for petrochemical plant area

Abstract

The invention discloses a structured lightweight fire smoke detection method for a petrochemical plant area. The method comprises the following steps: acquiring image data; the image data are input into an IFS-DETR model, a detection result is obtained, the IFS-DETR model is obtained through training of a training set, and the training set comprises preprocessed industrial fire smoke images and corresponding bounding boxes and category labels; and alarm information is sent based on the detection result, so that the structured lightweight fire smoke detection for the petrochemical plant area is realized. According to the invention, the complexity of the network is reduced, the detection effect of fire smoke is improved, and the real-time detection performance of the network is improved; by introducing the feature fusion network based on the alignment mechanism, the detection performance of the network on small targets and salient targets is effectively improved, and the network precision is improved.

Description

Technical Field

[0001] The invention belongs to the technical field of fire smoke detection, and in particular relates to a structured lightweight fire smoke detection method for petrochemical plants. Background Art

[0003] Traditional detection methods are mainly based on various sensors. However, due to the limited detection range of sensors, there are large transmission delays and high false alarm rates in complex industrial production environments. Therefore, in large outdoor spaces or places with large noise interference, sensor-based fire detection methods have gradually been replaced by other methods. In recent years, with the upgrade of industrial monitoring hardware equipment, researchers have applied image-based fire detection methods in industrial production. Image-based fire detection methods can cover a larger range and are more suitable for early fire detection in open environments such as outdoor environments. Therefore, there is an urgent need for a structured lightweight fire smoke detection method for petrochemical plants to address the shortcomings of existing technologies. Summary of the invention

[0004] The purpose of the present invention is to propose a structured lightweight fire smoke detection method for petrochemical plants to solve the problems existing in the above-mentioned prior art.

[0005] To achieve the above object, the present invention provides a structured lightweight fire smoke detection method for a petrochemical plant, comprising the following steps:

[0006] Get image data;

[0007] Inputting the image data into an IFS-DETR model to obtain a detection result, wherein the IFS-DETR model is trained by a training set, and the training set includes pre-processed industrial fire smoke images and corresponding bounding boxes and category annotations;

[0008] An alarm message is sent based on the detection result to realize structured lightweight fire smoke detection for petrochemical plants.

[0009] Optionally, the IFS-DETR model consists of a backbone feature extraction network LeanNet, an encoder, a feature fusion network AFFNet, a decoder and a prediction head;

[0010] The backbone feature extraction network LeanNet is used to extract features from the image data to obtain a plurality of image features, wherein the plurality of image features include a first image feature, a second image feature, a third image feature and a fourth image feature;

[0011] The encoder is configured to obtain an output image feature of the encoder based on the fourth image feature and the position embedding;

[0012] The feature fusion network AFFNet is used to fuse the second image feature, the third image feature and the output image feature of the encoder to form a Memory;

[0013] The decoder is used to obtain output image features of the encoder based on the memory, content query and position query;

[0014] The prediction head is used to classify the output image features of the encoder and predict the bounding box of the object to obtain the detection result.

[0015] Optionally, the backbone feature extraction network LeanNet includes several feature extraction stages, each of which is composed of a convolutional layer and several DS-C blocks, and each of which is composed of a depth-separable convolutional layer, a standard convolutional layer, a batch normalization layer and an activation layer;

[0016] The convolution layer is used to reduce the dimension of the image data; the DS-C block is used to extract features from the image data after the dimension reduction to obtain a number of image features;

[0017] The DS-C block is:

[0018] d1=Conv(DSConv(d))

[0019] d2=BN(ReLU(d1))

[0020] d3=Conv(d2)

[0021] d out =d⊕d3

[0022] Among them, d is the input feature of the DS-C block, d1 is the feature after depth-separable convolution and standard convolution, d2 is the feature after ReLU activation and batch normalization, d3 is the feature after convolution operation, and d out is the output feature of the DS-C block, Conv(·) is the standard convolution operation, DSConv(·) is the depth-wise separable convolution operation, BN(·) is the batch normalization operation, ReLU(·) is the activation function, and ⊕ is the concatenation operation.

[0023] Optionally, the encoder is composed of a multi-head self-attention network and a feedforward neural network, and each of the multi-head self-attention network and the feedforward neural network is followed by a residual link;

[0024] Among them, the multi-head self-attention network is:

[0025]

[0026] Among them, MultiHeadAtten(·) is the multi-head self-attention operation, z q is query, q is z q The index of ρ is the input feature, m is the number of heads of multi-head attention, M is the total number of heads of multi-head attention, and W m , W' m is the learnable weight, k is the index of the sampling key, K is the total number of sampling keys, A mqk represents the mth head attention weight, ρ k is the feature represented by key.

[0027] Optionally, before fusing the second image feature, the third image feature, and the output image feature of the encoder, the method further includes:

[0028] The output image features of the encoder are mapped, and the position information is supplemented using sinusoidal position coding, wherein the sinusoidal position coding is:

[0029]

[0030] Among them, PE(·) is the encoding result, pos represents the position of the feature, i is the value of the i-th dimension in the position encoding matrix, and d m is the dimension of the model embedding vector.

[0031] Optionally, the feature fusion network AFFNet is composed of an information alignment module, a cross-layer information fusion module and an information refinement module;

[0032] The information alignment module is used to unify the scales of the second image feature, the third image feature and the output image feature of the encoder to obtain aligned feature information;

[0033] The information alignment module is:

[0034] d 11 =Avgpool(S2)

[0035] d 22 =S3

[0036] d 33 =Interpolation(F1)

[0037] D=d 11 ⊕d 22 ⊕d 33

[0038] Among them, d 11 is the feature output by the high-dimensional feature S2 after dimensionality reduction by average pooling, d 22 For feature S3, d33 is the feature output by the low-dimensional feature F1 after bilinear interpolation, S2 is the feature layer output by the second stage of the backbone network, S3 is the feature layer output by the third stage of the backbone network, Avgpool(·) is the average pooling operation, Interpolation(·) is the bilinear interpolation operation, F1 is the output feature of the encoder, and ⊕ is the concatenation operation;

[0039] The cross-layer information fusion module is used to fuse the aligned feature information to obtain fused feature information;

[0040] The cross-layer information fusion module is:

[0041] L1=Conv 1×1 (D)

[0042] L2=EFCA(L1)

[0043] L=Conv 3×3 (L2)

[0044] Among them, L1 is the feature after the 1×1 convolution kernel operation, L2 is the feature after the EFCA attention operation, L is the feature after the 3×3 convolution kernel operation, Conv 1×1 (·) is a convolution operation with a convolution kernel of 1×1, D is the input feature, EFCA(·) is the efficient frequency channel attention calculation, Conv 3×3 (·) is the convolution operation with a convolution kernel of 3×3;

[0045] The information refinement module is used to refine the fused feature information and obtain the Memory in combination with the output image features of the encoder;

[0046] The information refinement module is:

[0047] E1=RepC3[Conv 1×1 (f1)]

[0048] E2=RepC3[Conv 1×1 [Interpolation(f2)]]

[0049] Among them, E1 is the first set of output features, E2 is the second set of output features, RepC3[·] is the reparameterization module, f1 is the first set of input features, Interpolation(·) is the bilinear interpolation operation, and f2 is the second set of input features.

[0050] Optionally, the cross-layer information fusion module consists of a first convolutional layer, an efficient frequency channel attention network, and a second convolutional layer;

[0051] The first convolutional layer is used to perform preliminary extraction on the aligned feature information to obtain preliminary extracted feature information;

[0052] The efficient frequency channel attention adopts an efficient frequency channel attention network based on 2D DCT to perform inter-channel information interaction on the initially extracted feature information and obtain feature information after information interaction;

[0053] Wherein, the 2D DCT and its inverse are:

[0054]

[0055] In the formula, is the basis of 2D DCT, h is the height of 2D DCT basis, w is the width of 2D DCT basis, H is the height of 2D DCT spectrum, Q is the width of 2D DCT spectrum, i is the number of rows of real matrix, j is the number of columns of real matrix, is the 2D DCT spectrum, 2D means two-dimensional, is the input;

[0056] The second convolutional layer is used to extract the feature information after the information interaction to obtain the fused feature information.

[0057] Optionally, obtaining output image features of the encoder based on the Memory, content query, and position query includes:

[0058] Introducing the IoU-Aware query into the IFS-DETR model, and performing an intersection-over-union query on the memory to obtain the content query;

[0059] Performing comparative denoising training on the Memory to obtain the location query, wherein the comparative denoising training includes matching and denoising;

[0060] The memory, the content query and the position query are input into the decoder to obtain output image features of the encoder.

[0061] Optionally, the decoder is composed of a deformable attention network, a multi-scale variable attention network and a feedforward neural network, and the deformable attention network, the multi-scale variable attention network and the feedforward neural network are each followed by a residual link;

[0062] The deformable attention network is:

[0063]

[0064] Among them, DeformAtten(·) is the deformable multi-head self-attention operation, j is the index of the sampling key, K is the total number of sampling keys, m is the number of heads of multi-head attention, and W m 、V' m is the learnable weight, A mqk represents the mth head attention weight, Δp mqk represents the sampling offset of the kth sampling point in the mth attention head, z q For query, p q is a two-dimensional reference point, ρ is the input feature, and M is the total number of deformable multi-head self-attention heads;

[0065] The multi-scale variable attention network is:

[0066]

[0067] Among them, MSDeformAtten(·) is a multi-scale deformable multi-head self-attention operation. is the normalized coordinate of the reference point of the query element, ρ l is the multi-scale input feature, L is the number of layers of input features, φ l is the normalization operation, l is the index of the feature map level, A mlqk is the attention weight, Δp mlqk Represents the sampling offset of the kth sampling point of the mth attention head on the first feature layer.

[0068] Optionally, obtaining the preprocessed industrial fire smoke image includes:

[0069] Acquire industrial fire smoke images;

[0070] Cleaning the industrial fire smoke image;

[0071] The cleaned industrial fire smoke image is annotated with a bounding box and a category, and double-checked to obtain the pre-processed industrial fire smoke image.

[0072] The present invention has the following beneficial effects:

[0073] The present invention has made targeted improvement measures on the basis of the DETR network. Firstly, by adopting a more efficient feature extraction network and combining it with the optimized Transformer Encoder-Decoder architecture, the complexity of the network is reduced and the overall real-time performance of the network is improved; secondly, by introducing a feature fusion network based on an alignment mechanism, the network detection performance for small targets and significant targets is effectively improved, and the network accuracy is improved; at the same time, considering the physical properties of the main targets (smoke / flame) for fire detection, a new loss function is introduced to obtain a more accurate boundary loss; finally, the network acceleration proposed by the present invention is deployed in edge computing devices, and is successfully applied in industrial intelligent monitoring equipment and wheeled fire inspection robots to meet the real needs of industrial firefighting. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0075] Figure 1 This is a flow chart of a structured lightweight fire smoke detection method for a petrochemical plant area according to an embodiment of the present invention;

[0076] Figure 2 A structural diagram of the IFS-DETR model proposed in an embodiment of the present invention;

[0077] Figure 3 A structural diagram of LeanNet proposed in an embodiment of the present invention;

[0078] Figure 4 The structure diagram of the encoder-decoder of the IFS-DETR proposed in the embodiment of the present invention, wherein (a) is the encoder structure and (b) is the decoder structure;

[0079] Figure 5 A flowchart of contrast denoising training proposed in an embodiment of the present invention;

[0080] Figure 6 A structural diagram of the EFCA proposed in an embodiment of the present invention;

[0081] Figure 7 AFFNet is a structural diagram of the AFFNet proposed in the embodiment of the present invention;

[0082] Figure 8 This is a schematic diagram of related information of IoU proposed in an embodiment of the present invention;

[0083] Fig. 9 A schematic diagram of image samples in a data set proposed in an embodiment of the present invention;

[0084] Fig.10 This is a comparison curve chart of mAP50 under different loss functions proposed in the embodiments of the present invention. DETAILED DESCRIPTION

[0085] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0086] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0087] The DETR series of algorithms can complete various target detection tasks well by combining CNN with Transformer. However, the real-time performance of DETR models is still behind that of mainstream detectors. Therefore, a simple but efficient backbone network is designed based on deep separable convolution. The simple design idea enables the detector to have better real-time performance. At the same time, in order to overcome the shortcomings of DETR detectors in the field of small target detection, a feature fusion network based on alignment mechanism is introduced into the network, and a channel attention mechanism is embedded in it. In the face of the problem that DETR models are difficult to train efficiently, the intersection over union (IOU) query mechanism and contrast denoising training based on aspect ratio are introduced into the detector, which effectively improves the training strategy of DETR models. In the face of the deployment difficulties caused by the high model complexity of DETR models, the number of layers of encoder and decoder is optimized to achieve the best balance between model accuracy and speed. Finally, Inner-IoU and SIoU are combined as part of the positioning loss function to accelerate the convergence of the model.

[0088] like Figure 1 As shown, this embodiment provides a structured lightweight fire smoke detection method for petrochemical plants, including the following steps: acquiring image data; inputting the image data into an IFS-DETR model to obtain detection results, wherein the IFS-DETR model is trained by a training set, and the training set includes preprocessed industrial fire smoke images and corresponding bounding boxes and category annotations; sending an alarm message based on the detection results to realize structured lightweight fire smoke detection for petrochemical plants.

[0089] Specifically, Figure 1As shown in the figure, taking the unmanned inspection robot as an application case, the IFS-DETR model is deployed in an embedded device. The high-definition camera on the top of the embedded device captures the external image in real time, identifies the characteristics of flames and smoke, and predicts the location of the fire. When there are signs of fire (smoke / flame) on the scene, the model quickly responds to detect the suspicious target area and transmits the corresponding detection results and time nodes to the fire control computer to complete the alarm. The specific operation process is as follows:

[0090] (1) The inspection robot moves along a predetermined route and uses a high-definition camera on top to capture image data in real time;

[0091] (2) The image data is then transmitted to the Jetson OrinNano device and detected in real time using the IFS-DETR model;

[0092] (3) Once fire or smoke features are detected, the edge device will immediately send an alarm message containing suspicious information, location, time and fire image to the fire control computer via WIFI to complete the alarm, and simultaneously record it on the vehicle touch screen for subsequent inspection.

[0093] Furthermore, the IFS-DETR model consists of a backbone feature extraction network LeanNet, an encoder, a feature fusion network AFFNet, a decoder, and a prediction head;

[0094] A backbone feature extraction network LeanNet, used to extract features from image data to obtain a plurality of image features, wherein the plurality of image features include a first image feature, a second image feature, a third image feature, and a fourth image feature;

[0095] An encoder, configured to obtain an output image feature of the encoder based on the fourth image feature and the position embedding;

[0096] Feature fusion network AFFNet, used to fuse the second image features, the third image features and the output image features of the encoder to form a memory;

[0097] Decoder, used to obtain the output image features of the decoder based on memory, content query and location query;

[0098] The prediction head is used to classify the output image features of the decoder and predict the bounding box of the object to obtain the detection results.

[0099] Specifically, Figure 2 As shown in the figure, the IFS-DETR model consists of five parts: the backbone feature extraction network LeanNet, Encoder and Decoder, the feature fusion network AFFNet, and the prediction head.

[0100] The prediction head is the last part of the model, which is used to classify the decoder output and predict the bounding box of the object. It was also found that in the early stage of a fire, both the flame and smoke samples always have different shapes and small targets. Therefore, the loss function was improved, which effectively improved the convergence speed of the model during training.

[0101] The feedforward network is used as the prediction head of the model to output the location information and category of the box. Specifically, the final prediction can be regarded as calculated through a 1×1 convolution, which contains a linear projection layer, a ReLU activation function, and a three-layer perceptron with a hidden dimension d. FFN predicts the center coordinates, height, and width of the box, and uses Softmax to complete the prediction of the category label. However, due to the particularity of the set prediction, the number of bounding boxes predicted by the model is usually much larger than the actual number of objects in the image, so an additional category φ is used to indicate that no object is detected.

[0102] In the early stages of a fire, both flames and smoke are always presented at a smaller scale, and the two types of samples have large differences in color, shape, and texture. Therefore, in the process of network training, a loss function that can adapt to tasks with strong sample diversity is needed to more effectively learn the optimal weights of the model on this task.

[0103] IoU has been widely used in the calculation of positioning loss for target detection. It represents the overlap between the “prediction box” and the “real box”. When the two completely overlap, IoU = 1. Figure 8 DETR is consistent with many mainstream algorithms and uses GIoU combined with smooth L1 to calculate the positioning loss. The loss formula is as follows:

[0104]

[0105] Among them, IoU is the intersection-over-union ratio between the predicted box and the real box. is the GIoU loss function, A is the area of ​​the predicted box, B is the area of ​​the real box, and D is the area of ​​the minimum circumscribed rectangle of the two. GIoU is the lower bound of IoU. When the predicted box and the real box overlap infinitely, the two are equivalent. When the two have no intersection and are infinitely far away, the minimum value -1 is taken. The value range of GIoU is bounded and symmetrical, and there will be no drastic fluctuations during training. However, when the predicted box and the real box have overlapping areas in the same direction but at different distances, they cannot respond quickly to the loss function, causing the network to not converge quickly in the early training.

[0106] In order to overcome the problem that DETR-like networks are difficult to converge quickly, Inner-SIoU is introduced into the positioning loss. Gevorgyan et al. first introduced the concept of angle loss in IoU, which takes the angle between the predicted box and the true box as the cost term. By calculating the angle between the predicted box and the true box, the predicted box is quickly aligned with the nearest coordinate axis, and then the regression task is completed. Specifically, SIoU consists of 4 cost functions. The first is the angle cost, which can be expressed by the following formula:

[0107]

[0108] Among them, Λ is the angle cost function, x is the sine value of the angle between the center point of the predicted box and the center point of the real box, and c h is the height difference between the center of the predicted box and the center of the real box, σ is the distance between the center of the predicted box and the center of the real box, α is the angle between the center of the predicted box and the center of the real box, is the coordinate of the center point of the prediction box, is the coordinate of the center point of the real box. The angle cost brings the predicted box to the nearest coordinate axis and considers whether to approach the x-axis or y-axis first according to the change in angle. When α=45°, Λ=1. When the center point of the box is aligned on the x-axis or y-axis, Λ=0. After considering the angle cost, the distance cost can be defined by the following formula:

[0109]

[0110] γ=2-Λ

[0111]

[0112] Among them, Δ is the distance cost function, t is the x-axis or y-axis, e is a natural constant, γ is a constant, ∈ t As part of the consideration, d w and d h Represent the width and height of the minimum bounding rectangle respectively. Finally, the shape cost is used to describe the size difference between the predicted box and the real box, and the importance of the shape cost is set by the value of θ (take θ = 2), which is defined as follows:

[0113]

[0114] Where Ω is the shape cost function, t is the width or height, ω w is the width penalty, ω h is the height cost loss, w A and h A Represents the width and height of the prediction box, w B and h BRepresents the width and height of the real box. Taking into account all the above costs, the SIoU loss function is defined as follows:

[0115]

[0116] in, is the SIoU loss function.

[0117] SIoU reduces the degrees of freedom of bounding box regression by adding geometric constraints, thereby accelerating the convergence of the model, but it does not consider the rationality of the IoU loss itself, which largely determines the quality of the detection results. To make up for this deficiency, Inner-IoU is combined with SIoU and calculated using auxiliary bounding boxes. Figure 8 As shown, A represents the prediction box and is the coordinate of its center point, B represents the real box and are the coordinates of its center point. A and h A Represents the width and height of the prediction box, w B and h B Represents the width and height of the real box, and Represents the width and height of the Inner prediction box, and Represents the width and height of the Inner real box. The Inner-IoU can be calculated using the following formula:

[0118]

[0119] inter=[min(A r ,B r )-max(A l ,B l )]×[min(A b ,B b )-max(A t ,B t )]

[0120] union=ξ 2 (w B ×h B )+ξ 2 (w A ×h A )-inter

[0121]

[0122] Among them, A l , A r , A t , Ab , B l , B r , B t , B b , inter, and union are all function calculations, IoU inner is the Inner joint intersection-over-union ratio, ξ is the scaling factor, usually ξ∈[0.5,1.5]. Inner-IoU makes up for the weak generalization and slow convergence of the existing IoU loss in detection tasks. In summary, Inner-IoU is combined with SIoU as an important component of positioning loss, which is defined as follows:

[0123]

[0124] in, is the Inner-SIoU loss function.

[0125] At the same time, considering that the initial value is large in the early stage of training, the loss gradient is too large, which may lead to the model training being not stable enough. When the training gradually stabilizes in the later stage, the loss gradient should be small enough to make the model converge to the global or local minimum. Introducing smooth L1 as another part into the positioning loss can prevent the gradient value from being too large when the gap between the real box and the predicted box is too large, and ensure that the gradient value can be small enough when the gap between the real box and the predicted box is small. The definition of smooth L1 loss is as follows, where g is the coordinate deviation between the real box and the predicted box:

[0126]

[0127] in, is the smooth L1 loss function.

[0128] Finally, the localization loss of the model Mainly by and It consists of two parts, which are defined as follows:

[0129]

[0130] Among them, β and is the weight coefficient of the loss term, where β=2.

[0131] Furthermore, the backbone feature extraction network LeanNet contains several feature extraction stages, each of which consists of a convolutional layer and several DS-C blocks, and each DS-C block consists of a depthwise separable convolutional layer, a standard convolutional layer, a batch normalization layer, and an activation layer;

[0132] Among them, the convolution layer is used to reduce the dimension of the image data; the DS-C block is used to extract features from the image data after dimension reduction to obtain several image features;

[0133] The DS-C blocks are:

[0134] d1=Conv(DSConv(d))

[0135] d2=BN(ReLU(d1))

[0136] d3=Conv(d2)

[0137] d out =d⊕d3

[0138] Among them, d is the input feature of the DS-C block, d1 is the feature after depth-separable convolution and standard convolution, d2 is the feature after ReLU activation and batch normalization, d3 is the feature after convolution operation, and d out is the output feature of the DS-C block, Conv(·) is the standard convolution operation, DSConv(·) is the depth-wise separable convolution operation, BN(·) is the batch normalization operation, ReLU(·) is the activation function, and ⊕ is the concatenation operation.

[0139] Specifically, traditional DETR detectors use the ResNet50 backbone network to initially extract image features. ResNet introduces residual links into the network design, allowing the network to be designed deeper in scale, which has promoted the progress of deep learning. However, for target detection algorithms that require better real-time performance, an efficient and lightweight feature extraction network is the key factor in improving detection speed, which requires a more lightweight design solution.

[0140] The present invention designs a lightweight and efficient feature extraction network LeanNet based on deep separable convolution. Similar to other mainstream backbone networks, LeanNet is also composed of many repeatable modules, but tries to keep these modules without overly complex operations to ensure that the network has lower complexity. Overall, LeanNet uses four stages to complete the extraction of image features, and different numbers of DS-C blocks are set in different stages to adapt to the calculation amount of different depths.

[0141] like Figure 3 As shown in the figure, in each stage, convolution is first used to reduce the dimension of the feature map. After this step, the scale of the feature map is reduced from d before input in ∈R C×H×W When changing to output Then, feature extraction is completed through the depth-separable convolution, standard convolution, batch normalization layer and activation layer of the DS-C block. At the same time, a residual link is set inside each DS-C block to reuse the input features, which can make up for the loss of inter-channel correlation information caused by the depth separation operation. Among them, the DS-C block can be expressed by the above formula.

[0142] Furthermore, the encoder consists of a multi-head self-attention network and a feedforward neural network, and each of the multi-head self-attention network and the feedforward neural network is followed by a residual link;

[0143] Among them, the multi-head self-attention network is:

[0144]

[0145] Among them, MultiHeadAtten(·) is the multi-head self-attention operation, z q is query, q is z q The index of ρ is the input feature, m is the number of heads of multi-head attention, M is the total number of heads of multi-head attention, and W m , W' m is the learnable weight, k is the index of the sampling key, K is the total number of sampling keys, A mqk represents the mth head attention weight, ρ k is the feature represented by key.

[0146] Specifically, Figure 4 As shown in (a), the encoder structure consists of a multi-head self-attention network and a feed-forward neural network (FFN). For a given query element and a set of key elements, the multi-head self-attention network adaptively aggregates the key content according to the attention weights. These weights can measure the compatibility of the two and finally complete the global information modeling. The calculation method of the multi-head self-attention network is as follows:

[0147]

[0148] Where q∈Ω q Index a with representation feature z q ∈R C query, k∈Ω k Index a with representation feature ρ k ∈R C The keys of , C is the feature dimension, Ω q and Ω k Specify the set of Query and Keys respectively, m is the number of heads of multi-head attention, represents the mth head attention weight,k∈K A mqk = 1 for normalization, are all learnable weights.

[0149] Furthermore, before fusing the second image feature, the third image feature and the output image feature of the encoder, the following steps are further included:

[0150] The output image features of the encoder are mapped and the position information is supplemented using sinusoidal position coding, where the sinusoidal position coding is:

[0151]

[0152] Among them, PE(·) is the encoding result, pos represents the position of the feature, i is the value of the i-th dimension in the position encoding matrix, and d m is the dimension of the model embedding vector.

[0153] Specifically, in DETR networks, the Encoder usually occupies the largest amount of network computation. Taking Deformable-DETR as an example, its Encoder's Gflop accounts for 49% of the total Gflop of the network, but only contributes 11% of AP. The main reason for this phenomenon is that despite the use of a more efficient attention calculation scheme, the overly long input sequence still brings a lot of computational overhead, which seriously affects the detection speed of the detector. In order to address this problem, an independent feature fusion network is used for multi-scale feature interaction. Specifically, the image features output by CNN are combined into a single feature fusion network. Mapped to And use sinusoidal position encoding to supplement the position information.

[0154] Furthermore, the feature fusion network AFFNet consists of an information alignment module, a cross-layer information fusion module, and an information refinement module;

[0155] An information alignment module, used to unify the scales of the second image feature, the third image feature, and the output image feature of the encoder to obtain aligned feature information;

[0156] The information alignment module is:

[0157] d 11 =Avgpool(S2)

[0158] d 22 =S3

[0159] d 33 =Interpolation(F1)

[0160] D=d 11 ⊕d 22 ⊕d 33

[0161] Among them, d 11 is the feature output by the high-dimensional feature S2 after dimensionality reduction by average pooling, d 22 For feature S3, d 33 is the feature output by the low-dimensional feature F1 after bilinear interpolation, S2 is the feature layer output by the second stage of the backbone network, S3 is the feature layer output by the third stage of the backbone network, Avgpool(·) is the average pooling operation, Interplation(·) is the bilinear interpolation operation, F1 is the output feature of the encoder, and ⊕ is the concatenation operation;

[0162] A cross-layer information fusion module is used to fuse the aligned feature information to obtain fused feature information;

[0163] The cross-layer information fusion module is:

[0164] L1=Conv 1×1 (D)

[0165] L2=EFCA(L1)

[0166] L=Conv 3×3 (L2)

[0167] Among them, L1 is the feature after the 1×1 convolution kernel operation, L2 is the feature after the EFCA attention operation, L is the feature after the 3×3 convolution kernel operation, Conv 1×1 (·) is a convolution operation with a convolution kernel of 1×1, D is the input feature, EFCA(·) is the efficient frequency channel attention calculation, Conv 3×3 (·) is the convolution operation with a convolution kernel of 3×3;

[0168] The information refinement module is used to refine the fused feature information and obtain the memory by combining the output image features of the encoder;

[0169] The information refinement modules are:

[0170] E1=RepC3[Conv 1×1 (f1)]

[0171] E2=RepC3[Conv 1×1 [Interpolation(f2)]]

[0172] Among them, E1 is the first set of output features, E2 is the second set of output features, RepC3[·] is the reparameterization module, f1 is the first set of input features, Interpolation(·) is the bilinear interpolation operation, and f2 is the second set of input features.

[0173] Specifically, the multi-head self-attention mechanism performs attention operations between features of the same scale, which improves the network's ability to express features. However, it does not consider the correlation between features of different scales, resulting in a waste of useful information. Therefore, a feature fusion network based on an alignment mechanism is introduced into the model to effectively aggregate multi-scale information on different channels.

[0174] In order to make up for the lack of cross-scale interaction capabilities of DETR networks, a feature fusion network based on alignment mechanism is designed. The network is mainly composed of information alignment module (IAM), cross-layer information fusion module (CIFM), and information refinement module (IRM). Aligning feature information to the same scale avoids the loss of beneficial information caused by cross-layer recursive calculations in the FPN network, and enhances the information fusion capability of the network without significantly increasing the delay. Figure 7 Specifically, in order to better integrate global features with cross-scale features, we select and The three-layer features are used as the input of AFFNet, and the information is unified into the same scale by the alignment module; then, the aligned feature information is passed through CIFM to realize the information interaction of features at different levels; finally, the fused feature information is further refined by IRM.

[0175] In IAM, average pooling is used to downsample high-dimensional feature information, and bilinear interpolation is used to upsample low-dimensional feature information. Finally, feature information of different scales is unified into the feature size of stage 3, i.e. Moderate feature maps retain more contextual information, which helps the network locate small objects more accurately. At the same time, they do not generate high computational delays during calculation, which helps to better balance the accuracy and speed of the network. Feature maps of uniform size are connected into a whole X and then sent to CIFM for cross-layer information fusion.

[0176] In CIFM, EFCA is used to guide the network to pay more attention to useful information and suppress irrelevant information. This solution improves the accuracy of the network. At the same time, adaptive local cross-channel interaction also ensures that EFCA is lightweight and efficient, and does not add too much reasoning delay to the network.

[0177] In order to make use of the fused information in the subsequent network, an efficient information refinement module IRM based on the reparameterized convolution block is designed. IRM mainly consists of two paths. At the beginning of the path, 1×1 convolution and upsampling are set to standardize the feature map scale, and the reparameterized convolution block (RepC3) is used to further refine the feature texture; finally, the feature maps of different scales are connected with the output of the encoder after Falten to form a memory, so as to make up for the shortcomings of the DETR model in multi-scale fusion ability.

[0178] Furthermore, the cross-layer information fusion module consists of the first convolutional layer, the efficient frequency channel attention network, and the second convolutional layer;

[0179] The first convolutional layer is used to perform preliminary extraction of the aligned feature information to obtain the preliminary extracted feature information;

[0180] Efficient frequency channel attention uses an efficient frequency channel attention network based on 2D DCT to perform inter-channel information interaction on the initially extracted feature information and obtain feature information after information interaction;

[0181] Among them, 2D DCT and its inverse are:

[0182]

[0183] In the formula, is the basis of 2D DCT, h is the height of 2D DCT basis, w is the width of 2D DCT basis, H is the height of 2D DCT spectrum, W is the width of 2D DCT spectrum, i is the number of rows of real matrix, j is the number of columns of real matrix, is the 2D DCT spectrum, 2D means two-dimensional, is the input;

[0184] The second convolutional layer is used to extract feature information after information interaction and obtain fused feature information.

[0185] Specifically, CIFM is mainly composed of convolution and efficient frequency channel attention (EFCA). Generally speaking, 1×1 convolution is used to extract information, and the extracted information is interacted between channels through the attention network. Then, 3×3 convolution is used to output the feature information L fused by CIFM.

[0186] Traditional channel attention methods are dedicated to constructing various weight functions to evaluate the importance of channels. This weight function requires a scalar for each channel to be calculated. In order to save computational overhead, global average pooling (GAP) has become the standard choice for feature compression, but a potential problem is that the rich input information cannot be effectively captured by the average value alone. At the same time, considering that both flames and smoke have rich color forms and boundary textures, using multiple frequency components of 2D DCT instead of GAP operation can better adapt to task requirements. Therefore, an efficient frequency channel attention network (EFCANet) based on the 2D DCT compression scheme is designed. 2D DCT and its inverse can be expressed by the following formula:

[0187]

[0188] Among them, i∈{0,1,…,H-1}, j∈{0,1,…,W-1}, is the basis of 2D DCT, freq 2D ∈R H×W is the 2DDCT spectrum, x 2D ∈R H×W When h=w=0 in the above formula, 2D DCT is equivalent to GAP.

[0189] like Figure 6 As shown, before size compression, the input feature x∈R C×H×W In the channel direction, it is divided into {x 1 ,x 2 ,…,x n}, each part has the size x i ∈R C'×H×W ,in For any x i , a corresponding 2D DCT frequency component is assigned to it. The information compression process can be expressed by the following formula:

[0190]

[0191] f=Concat([f 1 ,f 2 ,…f n ])

[0192] Among them, u i 、v i is x i The corresponding 2D frequency component index, f i ∈R C' is the output component after information compression, f∈R C is the multispectral vector after information compression, is a two-dimensional discrete cosine transform, and Concat(·) is a concatenation operation.

[0193] After compressing the information in the channel, the input feature is x∈R C×H×W Change to f∈R C×1×1 Wang et al. experimentally confirmed that dimensionality reduction will have side effects on channel attention prediction, and capturing the dependencies between all channels is inefficient and unnecessary. Therefore, the information interaction method of ECANet is borrowed, that is, cross-channel information interaction is achieved through 1D same convolution of scale k, where K is the coverage of local cross-channel interaction. Obviously, by mapping ψ, high-dimensional channels have a longer range of interactions. For a given number of channels C, the coverage k of cross-channel interaction can be determined by the following formula:

[0194]

[0195] Among them, |*| odd Indicates the nearest odd integer operation.

[0196] Subsequently, the feature weights are activated by Sigmoid and multiplied by the original features channel by channel to obtain the output feature f'∈R of the attention mechanism C×H×W , the process can be expressed by the following formula:

[0197] f′=Sigmoid[Conv1D k (f)]

[0198]

[0199] Among them, Sigmoid[·] is the activation function, Conv1D k (·) is a one-dimensional convolution, is the output feature of EFCA.

[0200] Furthermore, based on memory, content query and location query, the output image features of the encoder are obtained, including:

[0201] Introduce IoU-Aware query into the IFS-DETR model, and perform intersection-over-union query on memory to obtain content query;

[0202] Perform comparative denoising training on Memory to obtain location query, where comparative denoising training includes matching and denoising;

[0203] Memory, content query and location query are input into the decoder to obtain the output image features of the encoder.

[0204] Specifically, query initialization is optimized, that is, the problem that the query is difficult to be optimized by the decoder is solved from the perspective of query initialization. The object query in the traditional DETR is a set of learnable embeddings optimized by the decoder. Since these embeddings have no clear physical meaning, they are difficult to interpret and optimize during training. At present, the improvement measures for object query initialization mainly include two schemes: content query and location query. Both use the classification score to select the top-K tokens from the encoder to initialize the query. However, due to the distribution difference between the classification score and the location confidence, some boxes with high classification scores but low location confidence are selected, while boxes with low classification scores but high location confidence are discarded, which will damage the performance of the model. In order to improve the training quality, IoU-Aware query is introduced into the model. This method constrains the model to produce high classification scores for features with high location confidence and low classification scores for those features with low location confidence during training. After adding IoU-Aware query selection, the prediction box corresponding to the top-K encoder features selected by the model according to the classification score has both high classification scores and low location confidence, which can be expressed as:

[0205]

[0206] in, is the IoU query constraint function, represents the predicted value, y = {b, c} represents the true value, c is the category, and b is the bounding box.

[0207] Adding denoising training of positive and negative samples solves the problem of unstable Hungarian matching. DETR uses the Hungarian matching algorithm to achieve the optimal matching of predicted targets and real targets. However, due to the discreteness of Hungarian matching and the randomness of model training, the matching of Query to GT becomes a dynamic and unstable process. This causes the same Query to match different GTs in different training cycles, making the optimization of the model ambiguous. Therefore, the GT noise-added data is used as the input of the decoder, and then the decoder is allowed to learn the denoising process so that the predicted output results are close to the real GT.

[0208] like Figure 5As shown in the figure, denoising training can be divided into two parts: denoising and matching. Since Hungarian matching is not required, the Denoising Part can be regarded as a shortcut to help the Decoder optimize parameters efficiently. When adding noise, each GT will generate 5 groups of noise samples to improve training efficiency and robustness. Since the GT after adding noise is similar to the correct answer, in order to ensure that there is no information leakage, the Attention Mask is used to specify the rules for calculating Attention between each Query and key: (1) Queries in the Matching Part cannot be matched with those in the Denoising Part; (2) Queries in each group of the Denoising Part cannot interact with each other.

[0209] In order to better adapt to the physical characteristics of flames and smoke, bounding box visualization was performed on multiple fire smoke scene datasets. It was found that the GT of flame and smoke samples is often presented in a geometric shape with one side long and the other side short. In order to make the added positive samples within a reasonable scale, a coordinate denoising method based on aspect ratio was proposed. In the data preprocessing stage, the coordinates of GT are converted from (x, y, w, h) to (x min ,y min ,x max ,y max ), and based on the sine of the width and height GT is divided into two categories for noise addition. The coordinates after noise addition (x' min ,y' min ,x' max ,y' max ) can be calculated using the following formula:

[0210]

[0211] Among them, μ1, μ2, μ3, μ4 are perturbation coefficients sampled from a uniform distribution. The perturbation coefficients of Positive samples range from {μ1, μ2, μ3, μ4}∈(0, 1), and the perturbation coefficients of Negative samples range from {μ1, μ2, μ3, μ4}∈(1, 2). When tanα≤1, λ=2, v=4. When tanα>1, λ=4, v=2. When adding label noise, the GT label is randomly converted to other categories, and the category value is encoded as an Embedding vector. Finally, since the denoising process is only performed in the model training phase, in order to ensure the compatibility of the network, the noise query and IoU-Aware query are concat and input into the Decoder together, but the two are independent of each other and are not coupled.

[0212] Furthermore, the decoder is composed of a deformable attention network, a multi-scale variability attention network and a feedforward neural network, and each of the deformable attention network, the multi-scale variability attention network and the feedforward neural network has a residual link after it;

[0213] The deformable attention network is:

[0214]

[0215] Among them, DeformAtten(·) is the deformable multi-head self-attention operation, k is the index of the sampling key, K is the total number of sampling keys, m is the number of heads of multi-head attention, and W m , W' m is the learnable weight, A mqk represents the mth head attention weight, Δp mqk represents the sampling offset of the kth sampling point in the mth attention head, z q is query, p q is a two-dimensional reference point, ρ is the input feature, and M is the total number of deformable multi-head self-attention heads;

[0216] The multi-scale variable attention network is:

[0217]

[0218] Among them, MSDeformAtten(·) is a multi-scale deformable multi-head self-attention operation. is the normalized coordinate of the reference point of the query element, ρ l is the multi-scale input feature, L is the number of layers of input features, φ l is the normalization operation, l is the index of the feature map level, A mlqk is the attention weight, Δp mlak Represents the sampling offset of the kth sampling point of the mth attention head on the first feature layer.

[0219] Specifically, during the model training phase, the input of the Decoder consists of Memory, Class query (i.e., content query) and location query. In order to improve the real-time performance of the detector, a multi-scale deformable attention is introduced as a calculation scheme for the cross attention part. Compared with the traditional multi-head self-attention, the deformable attention network only uses a small part of the Key and Query for attention calculation, which effectively reduces the complexity of the model. For a given input feature ρ∈R C×H×W , let q index a variable with content feature z q and a two-dimensional reference point p q For a query element, the deformable attention is calculated as follows:

[0220]

[0221] Where k is the index of the sampling key and K is the total number of sampling keys. mqk and A mqk denote the sampling offset and attention weight of the kth sampling point in the mth attention head. mqk ∈R 2 is a two-dimensional real number with unconstrained range. The attention weight A mqk ∈[0,1], Normalize.

[0222] In order to better adapt to the multi-scale information carried in the memory, deformable attention is applied to the multi-scale features. in For the normalized coordinates of each Query reference point, the multi-scale deformable attention calculation formula is as follows:

[0223]

[0224] Where l is the index of the feature map level, Δp mlpk and A mqk They represent the sampling offset and attention weight of the kth sampling point of the mth attention head on the first feature layer. Attention weight A mlqk through Do normalization. Figure 4 (b) shows the Decoder structure in detail, where the parameters of the deformable attention are set to M=8 and K=4.

[0225] Further, the pre-processed industrial fire smoke image is obtained, including:

[0226] Acquire industrial fire smoke images;

[0227] Cleaning industrial fire smoke images;

[0228] The cleaned industrial fire smoke image is annotated with bounding boxes and categories and double-checked to obtain the preprocessed industrial fire smoke image.

[0229] Specifically, the present invention introduces in detail the industrial fire smoke dataset, experimental environment and evaluation indicators used for model training and evaluation. In order to ensure the reliability of the data, the images and videos in the dataset are all from real industrial fire scenes.

[0230] Considering that the current datasets dedicated to industrial fire smoke detection still have problems such as low clarity, poor data diversity, and insufficient data volume, more than 5,000 images and 5 fire monitoring images were collected from various real industrial fire smoke scenes for dataset construction. After data cleaning, 2047 images were retained, and LabelImg was used to annotate the bounding boxes and categories of the acquired images. The annotation process divided all samples into two categories: "flame" and "smoke", and then double-checked to ensure the quality of the annotation. Image samples in the dataset are as follows: Fig. 9 As shown in the figure, only a few representative images are shown here due to space limitations. After the label file is generated, the dataset is divided into training set, validation set and test set in a ratio of 7:1:2. Three image enhancement algorithms are applied to enhance the training set. These methods include: ① Randomly select 15% of the pictures, change the brightness of the image in the range of [0.7:1.3] and generate a new image; ② Randomly select 15% of the pictures, flip them horizontally once, and generate a new image; ③ Randomly select 10% of the pictures in the original image and add blur noise. The industrial fire smoke dataset finally established includes 4304 training set images, 409 validation set images and 206 test set images.

[0231] The training and testing of all models in the experiment were performed on the same server, which used the Ubuntu 16.04 operating system and was equipped with an Intel(R) Xeon(R) Platinum 8383C CPU. Four NVIDIA GeForce RTX 4090 24GB GPUs were used to train the models. All models were run under the PyTorch 1.12.1 framework and the programming language was Python 3.8.16. The models were optimized using the AdamW optimizer, and the initial learning rate was set to 1×10 -4 , batch size is 8, training epoch is 200, weight decay rate is 10 -4 .

[0232] In order to evaluate the model more comprehensively, the present invention uses precision (P), recall (R), average precision (AP), F1-Score and Accuracy (AR) to measure the detection accuracy of the model, and uses the operating frequency FPS and latency (ms) to evaluate the real-time performance of the model. The above-mentioned indicators can be calculated by the following formula:

[0233]

[0234] Among them, Prcision is precision, Recall is recall rate, AP is average precision, F1-Score is the geometric mean of precision and recall rate, Accuracy is accuracy, TP represents the number of times the sample is correctly predicted as a positive sample, FP represents the number of times the sample is incorrectly predicted as a positive sample, FN represents the number of times the sample is incorrectly predicted as a negative sample, and TN represents the number of times the sample is correctly predicted as a negative sample.

[0235] Since the model is designed for industrial application environments, the feasibility of deploying the model in resource-constrained embedded devices must be considered. Under this premise, models with lower GFLOPs and Parameters are preferred. GFLOPs is a giga floating-point operation per second, and its size can measure the computing resources required for model operation. In general, models with higher GFLOPs values ​​require stronger hardware support, while Parameters represent the number of parameters in the entire algorithm framework.

[0236] To further demonstrate the effectiveness of IFS-DETR in fire and smoke detection, all experimental results are presented and discussed. A series of detailed experiments are conducted on a self-built industrial fire smoke dataset and multiple public fire datasets to further evaluate and compare IFS-DETR with other state-of-the-art models.

[0237] In order to verify the effectiveness of each improved component in the IFS-DETR architecture, a series of ablation experiments were conducted on the test set of the industrial fire smoke dataset. The experimental results are shown in Table 1. The "√" symbol indicates that the corresponding improvement strategy is adopted. The baseline selected in the experiment uses the ResNet50 backbone network and the GIoU loss function. When AFFNet is not used, the feature maps of the last two stages of the backbone network are concatenated with the feature maps output by the encoder and sent to the decoder. At the same time, CBAM, CA, SE, and ECA are used to compare and verify the efficiency of EFCA.

[0238] Table 1

[0239]

[0240] Comparing Experiment 1 and Experiment 2 in the table, it is found that replacing ResNet50 with LeanNet significantly reduces the number of model parameters. In addition, comparing Experiments 4, 5, 6, 7, 8, 9 with Experiments 17, 18, 19, 20, 21, 22, it is found that LeanNet is compatible with various structural improvements and optimizations of the IFS-DETR detector, which improves the lightweight degree of the algorithm. By comparing Experiment 1 and Experiment 3, it can be seen that after IFS-DETR is added to AFFNet, the detector mAP50 has been significantly improved, but it has not caused a surge in the number of parameters. These experimental results show that feature fusion enhances the interaction between objects of different scales and helps the network to locate small-sized objects more accurately. At the same time, a moderate feature alignment scheme will not increase too much delay, which helps to better balance the accuracy and speed of the network. In addition, comparing Experiments 4, 5, 6, 7, 8, 9 with Experiments 17, 18, 19, 20, 21, 22, it is found that after the loss function is changed to Inner-SIoU, the accuracy of the detector has been significantly improved. The efficiency of EFCA was also verified by comparing CBAM, CA, SE, and ECA. In experiments 11, 12, 13, 14, 15, and 16, ECA and EFCA achieved cross-channel information interaction more efficiently with their unique channel-to-channel interaction methods. At the same time, thanks to the rich frequency component information brought by 2D-DCT, EFCA effectively improved the recognition ability of the detector and was more suitable for fire smoke detection. Finally, by comparing experiment 1 with experiments 2, 9, and 22, it was verified that the three improvement methods are not mutually exclusive, but compatible with each other.

[0241] In order to evaluate the effectiveness of Inner-SIoU, 5 different loss functions including CIoU were tested. The baseline of the experiment adopts the 15th group in Table 1, and the model is trained on the industrial fire smoke dataset for 200 epochs. Fig.10 The first 120 epochs of model training are selected for display.

[0242] The above is only a preferred specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A structured lightweight fire smoke detection method for petrochemical plants, characterized in that: The specific steps include: Get image data; Inputting the image data into an IFS-DETR model to obtain a detection result, wherein the IFS-DETR model is trained by a training set, and the training set includes pre-processed industrial fire smoke images and corresponding bounding boxes and category annotations; An alarm message is sent based on the detection result to realize structured lightweight fire smoke detection for petrochemical plants.

2. The structured lightweight fire smoke detection method for petrochemical plants as claimed in claim 1, characterized in that: The IFS-DETR model consists of a backbone feature extraction network LeanNet, an encoder, a feature fusion network AFFNet, a decoder, and a prediction head; The backbone feature extraction network LeanNet is used to extract features from the image data to obtain a plurality of image features, wherein the plurality of image features include a first image feature, a second image feature, a third image feature and a fourth image feature; The encoder is configured to obtain an output image feature of the encoder based on the fourth image feature and the position embedding; The feature fusion network AFFNet is used to fuse the second image feature, the third image feature and the output image feature of the encoder to form a Memory; The decoder is used to obtain output image features of the encoder based on the memory, content query and position query; The prediction head is used to classify the output image features of the encoder and predict the bounding box of the object to obtain the detection result.

3. The structured lightweight fire smoke detection method for petrochemical plants as claimed in claim 2, characterized in that: The backbone feature extraction network LeanNet includes several feature extraction stages, each of which is composed of a convolutional layer and several DS-C blocks, and each of which is composed of a depth-separable convolutional layer, a standard convolutional layer, a batch normalization layer, and an activation layer; The convolution layer is used to reduce the dimension of the image data; the DS-C block is used to extract features from the image data after the dimension reduction to obtain a number of image features; The DS-C block is: d1=Conv(DSConv(d)) d2=BN(ReLU(d1)) d3=Conv(d2) Among them, d is the input feature of the DS-C block, d1 is the feature after depth-separable convolution and standard convolution, d2 is the feature after ReLU activation and batch normalization, d3 is the feature after convolution operation, and d out is the output feature of the DS-C block, Conv(·) is the standard convolution operation, DSConv(·) is the depthwise separable convolution operation, BN(·) is the batch normalization operation, ReLU(·) is the activation function, For splicing operation.

4. The structured lightweight fire smoke detection method for petrochemical plants as claimed in claim 2, characterized in that: The encoder is composed of a multi-head self-attention network and a feedforward neural network, and each of the multi-head self-attention network and the feedforward neural network is followed by a residual link; Among them, the multi-head self-attention network is: Among them, MultiHeadAtten(·) is the multi-head self-attention operation, z q is query, q is z q The index of ρ is the input feature, m is the number of heads of multi-head attention, M is the total number of heads of multi-head attention, and W m , W' m is the learnable weight, k is the index of the sampling key, K is the total number of sampling keys, A mqk represents the mth head attention weight, ρ k is the feature represented by key.

5. The structured lightweight fire smoke detection method for petrochemical plants as claimed in claim 2, characterized in that: Before fusing the second image feature, the third image feature and the output image feature of the encoder, the method further includes: The output image features of the encoder are mapped, and the position information is supplemented using sinusoidal position coding, wherein the sinusoidal position coding is: Among them, PE(·) is the encoding result, pos represents the position of the feature, i is the value of the i-th dimension in the position encoding matrix, and d m is the dimension of the model embedding vector.

6. The structured lightweight fire smoke detection method for petrochemical plants as claimed in claim 5, characterized in that: The feature fusion network AFFNet consists of an information alignment module, a cross-layer information fusion module and an information refinement module; The information alignment module is used to unify the scales of the second image feature, the third image feature and the output image feature of the encoder to obtain aligned feature information; The information alignment module is: <h2 style=";text-align:left;direction:ltr">d<h2 style=";text-align:left;direction:ltr"> 11 <h2 style=";text-align:left;direction:ltr"> =Avgpool(S2) <h2 style=";text-align:left;direction:ltr">d<h2 style=";text-align:left;direction:ltr"> 22 <h2 style=";text-align:left;direction:ltr"> =S3 d 33 =Interpolation(F1) Among them, d 11 is the feature output by the high-dimensional feature S2 after dimensionality reduction by average pooling, d 22 For feature S3, d 33 is the feature output by the low-dimensional feature F1 after bilinear interpolation, S2 is the feature layer output by the second stage of the backbone network, S3 is the feature layer output by the third stage of the backbone network, Avgpool(·) is the average pooling operation, Interpolation(·) is the bilinear interpolation operation, F1 is the output feature of the encoder, For splicing operation; The cross-layer information fusion module is used to fuse the aligned feature information to obtain fused feature information; The cross-layer information fusion module is: L1=Conv 1×1 (D) L2=EFCA(L1) L=Conv 3×3 (L2) Among them, L1 is the feature after the 1×1 convolution kernel operation, L2 is the feature after the EFCA attention operation, L is the feature after the 3×3 convolution kernel operation, Conv 1×1 (·) is a convolution operation with a convolution kernel of 1×1, D is the input feature, EFCA(·) is the efficient frequency channel attention calculation, Conv 3×3 (·) is the convolution operation with a convolution kernel of 3×3; The information refinement module is used to refine the fused feature information and obtain the Memory in combination with the output image features of the encoder; The information refinement module is: E1=RepC3[Conv 1×1 (f1)] E2=RepC3[Conv 1×1 [Interpolation(f2)]] Among them, E1 is the first set of output features, E2 is the second set of output features, RepC3[·] is the reparameterization module, f1 is the first set of input features, Interpolation(·) is the bilinear interpolation operation, and f2 is the second set of input features.

7. The structured lightweight fire smoke detection method for petrochemical plants as claimed in claim 6, characterized in that: The cross-layer information fusion module consists of a first convolutional layer, an efficient frequency channel attention network and a second convolutional layer; The first convolution layer is used to perform preliminary extraction on the aligned feature information to obtain preliminary extracted feature information; The efficient frequency channel attention adopts an efficient frequency channel attention network based on 2D DCT to perform inter-channel information interaction on the initially extracted feature information and obtain feature information after information interaction; Wherein, the 2D DCT and its inverse are: In the formula, is the basis of 2D DCT, h is the height of 2D DCT basis, w is the width of 2D DCT basis, H is the height of 2D DCT spectrum, W is the width of 2D DCT spectrum, i is the number of rows of real matrix, j is the number of columns of real matrix, is the 2DDCT spectrum, 2D means two-dimensional, For input; The second convolutional layer is used to extract the feature information after the information interaction to obtain the fused feature information.

8. The structured lightweight fire smoke detection method for petrochemical plants as claimed in claim 2, characterized in that: Based on the memory, content query and position query, the output image features of the encoder are obtained, including: Introducing the IoU-Aware query into the IFS-DETR model, and performing an intersection-over-union query on the memory to obtain the content query; Performing comparative denoising training on the Memory to obtain the location query, wherein the comparative denoising training includes matching and denoising; The memory, the content query and the position query are input into the decoder to obtain output image features of the encoder.

9. The structured lightweight fire smoke detection method for petrochemical plants as claimed in claim 8, characterized in that: The decoder is composed of a deformable attention network, a multi-scale variability attention network and a feedforward neural network, and each of the deformable attention network, the multi-scale variability attention network and the feedforward neural network is followed by a residual link; The deformable attention network is: Among them, DeformAtten(·) is the deformable multi-head self-attention operation, k is the index of the sampling key, K is the total number of sampling keys, m is the number of heads of multi-head attention, and W m , W' m is the learnable weight, A mqk represents the mth head attention weight, Δp mqk represents the sampling offset of the kth sampling point in the mth attention head, z q For query, p q is a two-dimensional reference point, ρ is the input feature, and M is the total number of deformable multi-head self-attention heads; The multi-scale variable attention network is: Among them, MSDeformAtten(·) is a multi-scale deformable multi-head self-attention operation. is the normalized coordinate of the reference point of the query element, ρ l is the multi-scale input feature, L is the number of layers of input features, φ l is the normalization operation, l is the index of the feature map level, A mlqk is the attention weight, Δp mlqk Represents the sampling offset of the kth sampling point of the mth attention head on the first feature layer.

10. The structured lightweight fire smoke detection method for petrochemical plants as claimed in claim 1, characterized in that: Acquiring the preprocessed industrial fire smoke image, including: Acquire industrial fire smoke images; Cleaning the industrial fire smoke image; The cleaned industrial fire smoke image is annotated with a bounding box and a category, and double-checked to obtain the pre-processed industrial fire smoke image.

Images