Structured lightweight fire smoke detection method for petrochemical plant areas

Abstract

The application discloses a structured lightweight fire smoke detection method for a petrochemical plant area, and comprises the following steps: acquiring image data; inputting the image data into an IFS-DETR model to obtain a detection result, wherein the IFS-DETR model is obtained by training a training set, and the training set comprises preprocessed industrial fire smoke images and corresponding boundary boxes and class labels; and sending alarm information based on the detection result to realize the structured lightweight fire smoke detection for the petrochemical plant area. The application reduces the complexity of the network, improves the detection effect of the fire smoke, and increases the real-time detection performance of the network. By introducing a feature fusion network based on an alignment mechanism, the detection performance of the network on small targets and prominent targets is effectively improved, and the network precision is improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of fire smoke detection, and particularly relates to a structured lightweight fire smoke detection method for a petrochemical plant area. BACKGROUND

[0002] Traditional detection methods are mainly based on various sensors. However, due to the limited detection range of sensors, there is a large transmission delay and a high false alarm rate in complex industrial production environments. Therefore, in large outdoor spaces or places with high noise interference, sensor-based fire detection methods have been gradually replaced by other methods. In recent years, with the upgrading of industrial monitoring hardware devices, researchers have applied image-based fire detection methods in industrial production. Image-based fire detection methods can cover a larger range and are more suitable for early fire detection in open environments such as outdoors. Therefore, there is an urgent need for a structured lightweight fire smoke detection method for a petrochemical plant area to address the shortcomings of existing technologies. SUMMARY

[0003] The purpose of the present application is to provide a structured lightweight fire smoke detection method for a petrochemical plant area to solve the problems existing in the prior art.

[0004] To achieve the above purpose, the present application provides a structured lightweight fire smoke detection method for a petrochemical plant area, comprising the following steps:

[0005] Obtaining image data;

[0006] Inputting the image data into an IFS-DETR model to obtain a detection result, wherein the IFS-DETR model is obtained by training a training set, and the training set includes preprocessed industrial fire smoke images and corresponding bounding boxes and class labels;

[0007] Sending an alarm message based on the detection result to achieve a structured lightweight fire smoke detection method for a petrochemical plant area.

[0008] Optionally, the IFS-DETR model is composed of a backbone feature extraction network LeanNet, an encoder, a feature fusion network AFFNet, a decoder, and a prediction head;

[0009] The backbone feature extraction network LeanNet is used for feature extraction of image data to obtain a plurality of image features, wherein the plurality of image features include a first image feature, a second image feature, a third image feature, and a fourth image feature;

[0010] The encoder is used to obtain an output image feature of the encoder based on the fourth image feature and a position embedding;

[0011] The feature fusion network AFFNet is configured to fuse the second image feature, the third image feature and the output image feature of the encoder to form a Memory;

[0012] The decoder is configured to acquire the output image feature of the encoder based on the Memory, a content query and a location query.

[0013] The prediction head is configured to classify the output image feature of the encoder and acquire a bounding box of a predicted object to obtain the detection result.

[0014] Optionally, the backbone feature extraction network LeanNet comprises a plurality of feature extraction stages, each of which is composed of a convolutional layer and a plurality of DS-C blocks, and each of the DS-C blocks is composed of a depth separable convolutional layer, a standard convolutional layer, a batch normalization layer and an activation layer.

[0015] The convolutional layer is configured to reduce the dimension of the image data, and the DS-C block is configured to extract features from the reduced image data to obtain a plurality of image features.

[0016] The DS-C block is as follows:

[0017] d1 = Conv(DSConv(d))

[0018] d2 = BN(ReLU(d1))

[0019] d3 = Conv(d2)

[0020] d out = d

[0021] Wherein, d is the input feature of the DS-C block, d1 is the feature after the depth separable convolution and the standard convolution operation, d2 is the feature after the ReLU activation and the batch normalization, d3 is the feature after the convolution operation, d out is the output feature of the DS-C block, Conv(·) is the standard convolution operation, DSConv(·) is the depth separable convolution operation, BN(·) is the batch normalization operation, ReLU(·) is the activation function, and

[0022] Optionally, the encoder is composed of a multi-head self-attention network and a feedforward neural network, and a residual link is arranged after the multi-head self-attention network and the feedforward neural network.

[0023] The multi-head self-attention network is as follows:

[0024]

[0025] where MultiHeadAtten(·) is a multi-head self-attention operation, z q is the query, q is the index of z q , p is the input feature, m is the number of heads of the multi-head attention, M is the total number of heads of the multi-head attention, W m , W' m are learnable weights, k is the index of the sampled key, K is the total number of sampled keys, A mqk represents the mth head attention weight, p k is the feature represented by the key.

[0026] Optionally, before the fusion of the second image feature, the third image feature and the output image feature of the encoder, the method further comprises:

[0027] mapping the output image feature of the encoder and supplementing position information using a sinusoidal position encoding, wherein the sinusoidal position encoding is:

[0028]

[0029] where PE(·) is an encoding result, pos represents the position of the feature, i is the value of the ith dimension in the position encoding matrix, d m is the dimension of the model embedding vector.

[0030] Optionally, the feature fusion network AFFNet is composed of an information alignment module, a cross-layer information fusion module and an information refinement module.

[0031] The information alignment module is configured to unify the scales of the second image feature, the third image feature and the output image feature of the encoder, and obtain the aligned feature information.

[0032] The information alignment module is:

[0033] d 11 =Avgpool(S2)

[0034] d 22 =S3

[0035] d 33 =Interpolation(F1)

[0036] D=d 11 ⊕d 22 ⊕d 33

[0037] where d 11 is the feature output after the high-dimensional feature S2 is reduced in dimension by average pooling, d 22 is the feature S3, and d 33S2 is a feature layer output by the second stage of the backbone network, S3 is a feature layer output by the third stage of the backbone network, Avgpool(·) is an average pooling operation, Interpolation(·) is a bilinear interpolation operation, F1 is an output feature of the encoder, and is a splicing operation;

[0038] The cross-layer information fusion module is configured to fuse the aligned feature information to obtain fused feature information.

[0039] The cross-layer information fusion module is:

[0040] L1=Conv 1×1 (D)

[0041] L2=EFCA(L1)

[0042] L=Conv 3×3 (L2)

[0043] L1 is a feature after 1x1 size convolution kernel operation, L2 is a feature after EFCA attention operation, L is a feature after 3x3 size convolution kernel operation, Conv 1×1 (·) is a convolution operation with a 1x1 convolution kernel, D is an input feature, EFCA(·) is high-efficiency frequency channel attention calculation, Conv 3×3 (·) is a convolution operation with a 3x3 convolution kernel.

[0044] The information refinement module is configured to refine the fused feature information and combine the output image feature of the encoder to obtain the Memory.

[0045] The information refinement module is:

[0046] E1=RepC3[Conv 1×1 (f1)]

[0047] E2=RepC3[Conv 1×1 [Interpolation(f2)]]

[0048] E1 is a first group of output features, E2 is a second group of output features, RepC3[·] is a reparameterization module, f1 is a first group of input features, Interpolation(·) is a bilinear interpolation operation, and f2 is a second group of input features.

[0049] Optionally, the cross-layer information fusion module is composed of a first convolutional layer, a high-efficiency frequency channel attention network, and a second convolutional layer.

[0050] The first convolutional layer is used to perform preliminary extraction on the aligned feature information to obtain the preliminary extracted feature information;

[0051] The high-efficiency frequency channel attention adopts a high-efficiency frequency channel attention network based on 2D DCT, which is used to perform inter-channel information interaction on the initially extracted feature information and obtain the feature information after information interaction.

[0052] Wherein, the 2D DCT and its inverse are:

[0053]

[0054] In the formula, Let h be the height of the 2D DCT basis, w be the width of the 2D DCT basis, H be the height of the 2D DCT spectrum, Q be the width of the 2D DCT spectrum, i be the number of rows in the real matrix, and j be the number of columns in the real matrix. This is a 2D DCT spectrum, where 2D indicates two dimensions. For input;

[0055] The second convolutional layer is used to extract the feature information after the information interaction and obtain the fused feature information.

[0056] Optionally, based on the Memory, content query, and location query, the output image features of the encoder are obtained, including:

[0057] The IoU-Aware query is introduced into the IFS-DETR model, and the memory is filtered by intersection-union query to obtain the content query;

[0058] Perform contrastive denoising training on the memory to obtain the location query, wherein the contrastive denoising training includes matching and denoising;

[0059] The Memory, the content query, and the location query are input into the decoder to obtain the output image features of the encoder.

[0060] Optionally, the decoder consists of a deformable attention network, a multi-scale variable attention network, and a feedforward neural network, and each of the deformable attention network, the multi-scale variable attention network, and the feedforward neural network has a residual link after it.

[0061] The deformable attention network is:

[0062]

[0063] wherein DeformAtten(·) is deformable multi-head self-attention operation, j is the index of the sampling key, K is the total number of sampling keys, m is the number of multi-head attention heads, W m m is a learnable weight, A mqk represents the mth head attention weight, Δp mqk represents the sampling offset of the kth sampling point in the mth attention head, z q is a query, p q is a two-dimensional reference point, p is an input feature, and M is the total number of deformable multi-head attention heads.

[0064] The multi-scale deformable attention network is:

[0065]

[0066] wherein MSDeformAtten(·) is multi-scale deformable multi-head self-attention operation, is the normalized coordinate of the query element reference point, p l is a multi-scale input feature, L is the number of layers of the input feature, and p l is a normalization operation, l is the index of the feature map level, A mlqk is an attention weight, Δp mlqk represents the sampling offset of the kth sampling point in the mth attention head on the first feature layer.

[0067] Optionally, the preprocessed industrial fire smoke image is obtained, including:

[0068] An industrial fire smoke image is obtained.

[0069] The industrial fire smoke image is cleaned.

[0070] The cleaned industrial fire smoke image is labeled with a bounding box and a class, and double-checked, thereby obtaining the preprocessed industrial fire smoke image.

[0071] The present application has the following beneficial effects:

[0072] ​This invention makes targeted improvements to the DETR network. First, by employing a more efficient feature extraction network and combining it with an optimized Transformer Encoder-Decoder architecture, the complexity of the network is reduced, and the overall real-time performance of the network is improved. Second, by introducing a feature fusion network based on an alignment mechanism, the network's detection performance for small and salient targets is effectively improved, thus enhancing network accuracy. Simultaneously, considering the physical characteristics of the main targets (smoke / flame) in fire detection, a new loss function is introduced to obtain more accurate boundary loss. Finally, the network proposed in this invention is rapidly deployed in edge computing devices and successfully applied in industrial intelligent monitoring equipment and wheeled fire inspection robots, meeting real-world industrial fire protection needs. Attached Figure Description

[0073] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0074] Figure 1 This is a flowchart of a structured lightweight fire smoke detection method for petrochemical plant areas, as described in an embodiment of the present invention.

[0075] Figure 2 This is a structural diagram of the IFS-DETR model proposed in an embodiment of the present invention;

[0076] Figure 3 This is a structural diagram of LeanNet proposed in an embodiment of the present invention;

[0077] Figure 4 The diagram shows the encoder-decoder structure of the IFS-DETR proposed in this embodiment of the invention, where (a) is the encoder structure and (b) is the decoder structure.

[0078] Figure 5 This is a flowchart of the contrastive denoising training proposed in an embodiment of the present invention;

[0079] Figure 6 This is a structural diagram of the EFCA proposed in an embodiment of the present invention;

[0080] Figure 7 This is a structural diagram of the AFFNet proposed in an embodiment of the present invention;

[0081] Figure 8 This is a schematic diagram illustrating the relevant information of IoU proposed in the embodiments of the present invention;

[0082] Figure 9 This is a schematic diagram of image samples in the dataset proposed in an embodiment of the present invention;

[0083] Figure 10 This is a comparison curve of mAP50 under different loss functions proposed in the embodiments of the present invention. Detailed Implementation

[0084] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present application will now be described in detail with reference to the accompanying drawings and embodiments.

[0085] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0086] The DETR series of algorithms, by combining CNNs and Transformers, can effectively perform various object detection tasks. However, DETR-type models still lag behind mainstream detectors in real-time performance. Therefore, a simple yet efficient backbone network was designed based on depthwise separable convolutions. This simple design approach enables the detector to achieve better real-time performance. To overcome the shortcomings of DETR-type detectors in small object detection, a feature fusion network based on an alignment mechanism was introduced, embedding a channel attention mechanism within it. Addressing the difficulty of efficiently training DETR-type models, an Intersection over Union (IOU) query mechanism and aspect ratio-based contrastive denoising training were introduced into the detector, effectively improving the training strategy for DETR-type models. To address the deployment challenges caused by the high model complexity of DETR-type models, the number of layers in the encoder and decoder was optimized to achieve the best balance between model accuracy and speed. Finally, Inner-IoU and SIoU were combined as part of the localization loss function to accelerate model convergence.

[0087] like Figure 1 As shown, this embodiment provides a structured lightweight fire smoke detection method for petrochemical plant areas, including the following steps: acquiring image data; inputting the image data into the IFS-DETR model to obtain detection results, wherein the IFS-DETR model is trained from a training set, which includes preprocessed industrial fire smoke images and corresponding bounding boxes and category labels; and sending alarm information based on the detection results to achieve structured lightweight fire smoke detection for petrochemical plant areas.

[0088] Specifically, such as Figure 1As shown, taking an unmanned inspection robot as an application case, the IFS-DETR model is deployed in an embedded device. A high-definition camera on top of the embedded device captures real-time images of the surrounding environment, identifies flame and smoke features, and predicts the location of a fire. Specifically, when signs of fire (smoke / flame) appear on-site, the model quickly detects suspicious target areas and transmits the corresponding detection results and time points to the fire control computer to trigger an alarm. The specific operation process is as follows:

[0089] (1) The inspection robot moves along a predetermined route and uses the top high-definition camera to capture image data in real time.

[0090] (2) The image data is then transmitted to the Jetson OrinNano device and detected in real time using the IFS-DETR model;

[0091] (3) Once a fire or smoke feature is detected, the edge device will immediately send an alarm message containing the location, time and fire image of the suspicious information to the fire control computer via WIFI to complete the alarm and record it on the vehicle touch screen for subsequent inspection.

[0092] Furthermore, the IFS-DETR model consists of a backbone feature extraction network LeanNet, an encoder, a feature fusion network AFFNet, a decoder, and a prediction head;

[0093] The backbone feature extraction network LeanNet is used to extract features from image data and obtain several image features, including a first image feature, a second image feature, a third image feature, and a fourth image feature.

[0094] The encoder is used to obtain the output image features of the encoder based on the fourth image features and the position embedding;

[0095] The Feature Fusion Network (AFFNet) is used to fuse the second image features, the third image features, and the encoder's output image features to form a memory.

[0096] The decoder is used to obtain the output image features of the decoder based on memory, content query, and location query.

[0097] The prediction head is used to classify the features of the decoder's output image and predict the bounding boxes of objects to obtain detection results.

[0098] Specifically, such as Figure 2 As shown, the IFS-DETR model consists of five parts: the backbone feature extraction network LeanNet, the Encoder and Decoder, the feature fusion network AFFNet, and the prediction head.

[0099] The prediction head, the final part of the model, is used to classify the decoder's output and predict the bounding boxes of objects. It was also found that in the early stages of a fire, both flame and smoke samples consistently exhibited varied shapes and small size. Therefore, the loss function was improved, effectively increasing the model's convergence speed during training.

[0100] The feedforward network is used as the model's prediction head to output the location and category of the bounding boxes. Specifically, the final prediction can be viewed as being computed through a 1×1 convolution, which contains a three-layer perceptron with a linear projection layer, a ReLU activation function, and a hidden dimension d. The FFN predicts the center coordinates, height, and width of the bounding boxes, and Softmax is used to predict the category labels. However, due to the special nature of ensemble prediction, the number of bounding boxes predicted by the model is usually much larger than the actual number of targets in the image. Therefore, an additional category φ is used to represent that no targets were detected.

[0101] In the early stages of a fire, both flames and smoke are presented at a small scale, and the two types of samples differ significantly in color, shape, and texture. Therefore, during network training, a loss function that can adapt to tasks with high sample diversity is needed to more effectively learn the optimal weights for that task.

[0102] IoU has been widely used in calculating the localization loss for object detection. It represents the degree of overlap between the "predicted bounding box" and the "ground truth bounding box". When the two completely overlap, IoU = 1. Figure 8 The information related to IoU is displayed. DETR, consistent with many mainstream algorithms, uses a combination of GIoU and smoothed L1 to calculate the localization loss, as shown in the following formula:

[0103]

[0104] Where IoU is the intersection-union ratio between the predicted bounding box and the ground truth bounding box. Let GIoU be the loss function, where A is the area of ​​the predicted bounding box, B is the area of ​​the ground truth bounding box, and D is the area of ​​the smallest bounding rectangle of the two. GIoU is the lower bound of IoU; it is equivalent when the predicted and ground truth bounding boxes overlap infinitely, and it takes a minimum value of -1 when they have no intersection and are infinitely far apart. The value range of GIoU is bounded and symmetrical, and it will not fluctuate drastically during training. However, when the predicted and ground truth bounding boxes have overlapping regions in the same direction but at different distances, it cannot quickly respond to the loss function, causing the network to fail to converge quickly in the early stages of training.

[0105] To overcome the problem of slow convergence in DETR-type networks, Inner-SIoU is introduced into the localization loss. Gevorgyan et al. first introduced the concept of angle loss into IoU, which uses the angle between the predicted box and the ground truth box as a cost term. By calculating the angle between the predicted box and the ground truth box, the predicted box is quickly aligned with the nearest coordinate axis, thus completing the regression task. Specifically, SIoU consists of four cost functions, the first being the angle cost, which can be expressed by the following formula:

[0106]

[0107] Where Λ is the angle cost function, x is the sine of the angle between the center point of the predicted box and the center point of the ground truth box, and c h σ is the height difference between the center points of the predicted bounding box and the ground truth bounding box, α is the distance between the center points of the predicted bounding box and the ground truth bounding box, and α is the angle between the center points of the predicted bounding box and the ground truth bounding box. These are the coordinates of the center point of the prediction box. These are the coordinates of the center point of the ground truth bounding box. The angle cost shifts the predicted box to the nearest coordinate axis, considering whether to prioritize approaching the x-axis or y-axis depending on the angle. When α = 45°, Λ = 1. When the center point of the box is aligned on either the x-axis or y-axis, Λ = 0. After considering the angle cost, the distance cost can be defined by the following formula:

[0108]

[0109] γ=2-Λ

[0110]

[0111] Where Δ is the distance cost function, t is the x-axis or y-axis, e is the natural constant, γ is a constant, and ∈ t As part of the cost term, d w and d h These represent the width and height of the minimum bounding rectangle, respectively. Finally, shape cost is used to describe the dimensional difference between the predicted and ground truth bounding boxes, and the importance of shape cost in the overall calculation is set using the θ value (θ = 2), defined as follows:

[0112]

[0113] Where Ω is the shape cost function, t is the width or height, and ω is the shape cost function. w For the width cost loss, ω h For a high cost of loss, w A and h A w represents the width and height of the prediction box. B and h BRepresents the width and height of the ground truth bounding box. Taking all the above costs into account, the SIoU loss function is defined as follows:

[0114]

[0115] in, This is the SIoU loss function.

[0116] SIoU reduces the degrees of freedom in bounding box regression by adding geometric constraints, thereby accelerating model convergence. However, it doesn't consider the rationality of the IoU loss itself, which largely determines the quality of the detection results. To compensate for this deficiency, Inner-IoU is combined with SIoU, and an auxiliary bounding box is used for calculation. Figure 8 As shown, A represents the prediction box and These are the coordinates of its center point, where B represents the true bounding box. These are the coordinates of its center point. A and h A w represents the width and height of the prediction box. B and h B Represents the width and height of the actual bounding box. and Represents the width and height of the Inner prediction box. and This represents the width and height of the actual Inner bounding box. The Inner-IoU can be calculated using the following formula:

[0117]

[0118] inter = [min(A r B r )-max(A l B l )]×[min(A b B b )-max(A t B t )]

[0119] union = ξ 2 (w B ×h B )+ξ 2 (w A ×h A )-inter

[0120]

[0121] Among them, A l A r A t Ab B l B r B t B b inter, union, and inter are all functions used for computation, and IoU is also a function. inner Let ξ be the Inner Joint Intersection over Union (IoU), and ξ be a scaling factor, typically ξ ∈ [0.5, 1.5]. Inner-IoU compensates for the weak generalization and slow convergence of existing IoU losses in detection tasks. In summary, combining Inner-IoU with SIoU as an important component of localization loss is defined as follows:

[0122]

[0123] in, The loss function is Inner-SIoU.

[0124] Meanwhile, considering that a large initial value in the early stages of training can lead to an unstable model training due to an excessively large loss gradient, the loss gradient needs to be small enough to allow the model to converge to a global or local minimum as the training gradually stabilizes. Smoothing L1 loss is introduced as another part into the localization loss. This prevents the gradient value from becoming too large when the difference between the ground truth bounding box and the predicted bounding box is too great, and ensures that the gradient value is small enough when the difference between the ground truth bounding box and the predicted bounding box is small. The definition of smoothing L1 loss is as follows, where g is the coordinate deviation between the ground truth bounding box and the predicted bounding box:

[0125]

[0126] in, This is the smoothed L1 loss function.

[0127] Ultimately, the model's localization loss Mainly composed of and It consists of two parts, defined as follows:

[0128]

[0129] Where β and Here, β = 2 is taken as the weighting coefficient for the loss term.

[0130] Furthermore, the backbone feature extraction network LeanNet contains several feature extraction stages, each of which consists of convolutional layers and several DS-C blocks, and each DS-C block consists of depthwise separable convolutional layers, standard convolutional layers, batch normalization layers and activation layers.

[0131] The convolutional layer is used to reduce the dimensionality of the image data; the DS-C block is used to extract features from the reduced image data to obtain several image features.

[0132] The DS-C block is:

[0133] d1 = Conv(DSConv(d))

[0134] d2 = BN(ReLU(d1))

[0135] d3 = Conv(d2)

[0136] d out =d⊕d3

[0137] Where d represents the input features of the DS-C block, d1 represents the features after depthwise separable convolution and standard convolution, d2 represents the features after ReLU activation and batch normalization, and d3 represents the features after convolution. out For the output features of the DS-C block, Conv(·) is the standard convolution operation, DSConv(·) is the depthwise separable convolution operation, BN(·) is the batch normalization operation, ReLU(·) is the activation function, and ⊕ is the concatenation operation.

[0138] Specifically, traditional DETR-type detectors use a ResNet50 backbone network to initially extract image features. ResNet introduces residual connections into the network design, allowing the network to be designed to be deeper at different scales, thus driving the progress of deep learning. However, for object detection algorithms that require better real-time performance, an efficient and lightweight feature extraction network is a key factor in improving detection speed, which necessitates a more lightweight design scheme.

[0139] This invention presents LeanNet, a lightweight and efficient feature extraction network designed based on depthwise separable convolution. Similar to other mainstream backbone networks, LeanNet consists of numerous repetitive modules, but these modules are designed to avoid overly complex operations to ensure lower network complexity. Overall, LeanNet uses four stages to extract image features, with varying numbers of DS-C blocks in each stage to accommodate different computational depths.

[0140] like Figure 3 As shown, in each stage, convolution is first used to reduce the dimensionality of the feature map. After this step, the scale of the feature map is reduced from d before input. in ∈R C×H×W When changed to output Subsequently, feature extraction is completed sequentially through depthwise separable convolutions, standard convolutions, batch normalization layers, and activation layers within the DS-C block. Simultaneously, residual connections are incorporated within each DS-C block to reuse input features, compensating for the loss of inter-channel correlation information caused by the depthwise separation operation. The DS-C block can be expressed using the formula described above.

[0141] Furthermore, the encoder consists of a multi-head self-attention network and a feedforward neural network, and both the multi-head self-attention network and the feedforward neural network are followed by a residual link.

[0142] The multi-head self-attention network is as follows:

[0143]

[0144] Where MultiHeadAtten(·) is the multi-head self-attention operation, z q Let q be the query and z be the z. q The index is ρ, the input feature is m, the number of heads in the multi-head attention is M, and the total number of heads in the multi-head attention is W. m W' m For learnable weights, k is the index of the sampled key, K is the total number of sampled keys, and A mqk ρ represents the attention weight of the m-th head. k The features are represented by the key.

[0145] Specifically, such as Figure 4 As shown in (a), the Encoder structure consists of a multi-head self-attention network and a feed-forward neural network (FFN). Given a query element and a set of key elements, the multi-head self-attention network adaptively aggregates the content of the keys according to attention weights, which measure the compatibility between the two elements, ultimately completing the global information modeling. The computation method of the multi-head self-attention network is as follows:

[0146]

[0147] In the formula, q∈Ω q Index a character with representation feature z q ∈R C The query, k∈Ω k Index a character with representation feature ρ k ∈R C The keys, C is the feature dimension, Ω q and Ω k Specify the sets of Query and Keys respectively, where m is the number of heads for multi-head attention. This represents the attention weight of the m-th head, calculated by ∑k∈K A mqk =1 for normalization, All of these are learnable weights.

[0148] Furthermore, before fusing the second image features, the third image features, and the encoder's output image features, the following steps are also included:

[0149] The encoder's output image features are mapped, and sinusoidal position coding is used to supplement position information. The sinusoidal position coding is as follows:

[0150]

[0151] Where PE(·) is the encoding result, pos represents the position of the feature, i is the value of the i-th dimension in the position encoding matrix, and d m The dimension of the embedding vector for the model.

[0152] Specifically, in DETR-type networks, the encoder typically accounts for the largest portion of the network's computation. Taking Deformable-DETR as an example, its encoder's Gflop accounts for 49% of the network's total Gflop, but contributes only 11% of the AP. The main reason for this is that despite employing a more efficient attention calculation scheme, the excessively long input sequence still incurs significant computational overhead, severely impacting the detector's detection speed. To address this issue, an independent feature fusion network is used for multi-scale feature interaction. Specifically, the image features output by the CNN are fused... Mapped to Sine position coding is used to supplement position information.

[0153] Furthermore, the Feature Fusion Network (AFFNet) consists of an information alignment module, a cross-layer information fusion module, and an information refinement module.

[0154] The information alignment module is used to unify the scale of the second image features, the third image features, and the output image features of the encoder, and to obtain aligned feature information.

[0155] The information alignment module is:

[0156] d 11 =Avgpool(S2)

[0157] d 22 =S3

[0158] d 33 =Interpolation(F1)

[0159] D = d 11 ⊕d 22 ⊕d 33

[0160] Where, d 11 The output feature is the high-dimensional feature S2 after dimensionality reduction by average pooling, d 22 For feature S3, d 33 S1 is the output feature of low-dimensional feature F1 after bilinear interpolation, S2 is the feature layer output by the second stage of the backbone network, S3 is the feature layer output by the third stage of the backbone network, Avgpool(·) is the average pooling operation, Interplation(·) is the bilinear interpolation operation, F1 is the output feature of the encoder, and ⊕ is the concatenation operation.

[0161] The cross-layer information fusion module is used to fuse the aligned feature information to obtain the fused feature information;

[0162] The cross-layer information fusion module is as follows:

[0163] L1 = Conv 1×1 (D)

[0164] L2 = EFCA(L1)

[0165] L = Conv 3×3 (L2)

[0166] Where L1 represents the features after processing with a 1×1 convolution kernel, L2 represents the features after processing with EFCA attention, L represents the features after processing with a 3×3 convolution kernel, and Conv represents the features. 1×1 (·) represents a convolution operation with a 1×1 kernel, D represents the input features, EFCA(·) represents efficient frequency channel attention computation, and Conv 3×3 (·) represents a convolution operation with a 3×3 kernel;

[0167] The information refinement module is used to refine the fused feature information and, in conjunction with the output image features of the encoder, obtain the Memory.

[0168] The information refinement module is as follows:

[0169] E1 = RepC3[Conv] 1×1 (f1)]

[0170] E2=RepC3[Conv 1×1 [Interpolation(f2)]]

[0171] Wherein, E1 is the first set of output features, E2 is the second set of output features, RepC3[·] is the reparameterization module, f1 is the first set of input features, Interpolation(·) is the bilinear interpolation operation, and f2 is the second set of input features.

[0172] Specifically, multi-head self-attention mechanisms perform attention operations between features of the same scale, improving the network's ability to express features. However, they do not consider the correlation between features of different scales, resulting in a waste of useful information. Therefore, a feature fusion network based on an alignment mechanism is introduced into the model to effectively aggregate multi-scale information from different channels.

[0173] To compensate for the lack of cross-scale interaction capabilities in DETR-type networks, a feature fusion network based on an alignment mechanism was designed. This network mainly consists of an Information Alignment Module (IAM), a Cross-Layer Information Fusion Module (CIFM), and an Information Refinement Module (IRM). Aligning feature information to the same scale avoids the loss of useful information caused by cross-layer recursive computation in FPN networks, while enhancing the network's information fusion capability without significantly increasing latency. Figure 7 As shown, specifically, in order to better fuse global features with cross-scale features, we select... and The three layers of features are used as input to AFFNet, and these information are unified to the same scale by the alignment module. Subsequently, the aligned feature information is processed by CIFM to realize the information interaction of features at different levels. Finally, the fused feature information is further refined by IRM.

[0174] In IAM, average pooling is used to downsample high-dimensional feature information, and bilinear interpolation is used to upsample low-dimensional feature information. Finally, feature information at different scales is unified to the feature size of stage 3. Moderately sized feature maps retain more contextual information, helping the network to more accurately locate small objects, while also avoiding high computational latency, thus better balancing the network's accuracy and speed. Uniformly sized feature maps are concatenated into a single entity X, which is then fed into CIFM for cross-layer information fusion.

[0175] In CIFM, EFCA is used to guide the network to focus more on useful information and suppress irrelevant information, which improves the network's accuracy. At the same time, adaptive local cross-channel interaction ensures that EFCA is lightweight and efficient, without adding excessive inference latency to the network.

[0176] To enable the fused information to be utilized in subsequent networks, an efficient information refinement module (IRM) based on reparameterized convolutional blocks was designed. The IRM consists of two main paths. At the beginning of each path, a 1×1 convolution and upsampling are set to normalize the feature map scale. At the same time, a reparameterized convolutional block (RepC3) is used to further refine the feature texture. Finally, the feature maps of different scales are flattened and concatenated with the output of the encoder to form a memory, which compensates for the lack of multi-scale fusion capability of DETR-type models.

[0177] Furthermore, the cross-layer information fusion module consists of a first convolutional layer, an efficient frequency channel attention network, and a second convolutional layer;

[0178] The first convolutional layer is used to initially extract the aligned feature information and obtain the initially extracted feature information;

[0179] The high-efficiency frequency channel attention uses a high-efficiency frequency channel attention network based on 2D DCT to perform inter-channel information interaction on the initially extracted feature information and obtain the feature information after information interaction.

[0180] Wherein, 2D DCT and its inverse are:

[0181]

[0182] In the formula, Let h be the height of the 2D DCT basis, w be the width of the 2D DCT basis, H be the height of the 2D DCT spectrum, W be the width of the 2D DCT spectrum, i be the number of rows of the real matrix, and j be the number of columns of the real matrix. This is a 2D DCT spectrum, where 2D indicates two dimensions. For input;

[0183] The second convolutional layer is used to extract the feature information after information interaction and obtain the fused feature information.

[0184] Specifically, CIFM mainly consists of convolutions and Efficient Frequency Channel Attention (EFCA). Generally speaking, it first uses 1×1 convolutions to extract initial information, and the extracted information is then processed by the attention network for inter-channel interaction. Subsequently, 3×3 convolutions are used to output the feature information L fused by CIFM.

[0185] Traditional channel attention methods focus on constructing various weight functions to evaluate channel importance, requiring a scalar for each channel. To save computational overhead, Global Average Pooling (GAP) has become the standard choice for feature compression, but a potential problem is that averaging alone cannot effectively capture rich input information. Furthermore, considering that flames and smoke possess rich color patterns and boundary textures, using multiple frequency components of 2D DCT instead of GAP operations can better adapt to task requirements. Therefore, an efficient frequency channel attention network (EFCANet) based on a 2D DCT compression scheme is designed. 2D DCT and its inverse can be represented by the following formula:

[0186]

[0187] Where i∈{0,1,…,H-1}, j∈{0,1,…,W-1}, As a basis for 2D DCT, freq 2D ∈R H×W For 2DDCT spectrum, x 2D ∈R H×W For input. When h = w = 0 in the above formula, 2D DCT is equivalent to GAP.

[0188] like Figure 6 As shown, before size compression, the input feature x∈R C×H×W The channel direction is divided into {x} 1 ,x 2 ,…,x n Each part has a dimension of x. i ∈R C'×H×W ,in For any x i Each of these is assigned a corresponding 2D DCT frequency component. The information compression process can be represented by the following formula:

[0189]

[0190] f = Concat([f 1 ,f 2 ,…f n ])

[0191] Among them, u i v i It is x i The corresponding 2D frequency component index, f i ∈R C' It is the output component after information compression, f∈R C It is a multispectral vector after information compression. It is a two-dimensional discrete cosine transform, and Concat(·) is the concatenation operation.

[0192] After compressing the information in the channel, the input features are x∈R C×H×W Change to f∈R C×1×1 Wang et al. demonstrated through experiments that dimensionality reduction has side effects on channel attention prediction, and capturing all dependencies between channels is inefficient and unnecessary. Therefore, they borrowed the information interaction method from ECANet, namely, achieving cross-channel information interaction through 1D same convolutions of scale k, where K is the coverage of local cross-channel interaction. Clearly, by mapping ψ, higher-dimensional channels have a longer range of interactions. For a given number of channels C, the coverage k of cross-channel interaction can be determined by the following formula:

[0193]

[0194] Among them, |*| odd This indicates the operation of taking the nearest odd number.

[0195] Subsequently, the feature weights are activated by the Sigmoid function and multiplied channel by channel with the original features to obtain the output feature f'∈R of the attention mechanism. C×H×W This process can be expressed by the following formula:

[0196] f′=Sigmoid[Conv1D k (f)]

[0197]

[0198] Where Sigmoid[·] is the activation function, and Conv1D k (·) represents a one-dimensional convolution. This is the output feature of EFCA.

[0199] Furthermore, based on memory, content query, and location query, the output image features of the encoder are obtained, including:

[0200] Introduce IoU-Aware queries into the IFS-DETR model and perform intersection-union queries on the memory to filter and obtain content queries;

[0201] Perform contrastive denoising training on the memory to obtain location queries. The contrastive denoising training includes matching and denoising.

[0202] Input the memory, content query, and location query into the decoder to obtain the output image features of the encoder.

[0203] Specifically, optimizing query initialization addresses the difficulty of optimizing queries by the decoder. In traditional DETR, object queries are a set of learnable embeddings optimized by the decoder. Since these embeddings lack explicit physical meaning, they are difficult to interpret and optimize during training. Current improvements to object query initialization mainly include content-based and location-based methods, both using classification scores to select the top-K tokens from the encoder for query initialization. However, due to the difference in distribution between classification scores and location confidence, some boxes with high classification scores but low location confidence are selected, while boxes with low classification scores but high location confidence are discarded, which degrades model performance. To improve training quality, IoU-Aware queries are introduced into the model. This method constrains the model to generate high classification scores for features with high location confidence and low classification scores for features with low location confidence during training. After adding IoU-Aware query selection, the predicted boxes corresponding to the top-K encoder features selected by the model based on classification scores simultaneously possess both high classification scores and low location confidence, which can be expressed as:

[0204]

[0205] in, For IoU query constraint functions, Let y = {b, c} represent the predicted value, where c is the category and b is the bounding box.

[0206] Adding positive and negative samples during denoising training addresses the instability of Hungarian matching. DETR uses the Hungarian matching algorithm to achieve the optimal match between the predicted target and the true target. However, due to the discreteness of Hungarian matching and the randomness of model training, the matching of the query to the ground truth (GT) becomes a dynamic and unstable process. This leads to the same query matching different GTs in different training cycles, making the model optimization ambiguous. Therefore, the data with added noise to the GT is used as input to the decoder, and the decoder learns the denoising process, making the predicted output approximate the true GT.

[0207] like Figure 5As shown, denoising training can be divided into two parts: denoising and matching. Since Hungarian matching is not required, the denoising part can be regarded as a shortcut to help the decoder optimize parameters efficiently. When adding noise, each ground truth (GT) generates 5 sets of noise samples to improve training efficiency and robustness. Since the GT after adding noise is similar to the correct answer, in order to ensure that there is no information leakage, the Attention Mask is used to specify the rules for calculating the Attention between each Query and the key: (1) Queries in the Matching Part cannot interact with Denoising Queries; (2) Queries in the Denoising Part cannot interact with each other.

[0208] To better reflect the physical characteristics of flames and smoke, bounding box visualization was performed on multiple fire and smoke scene datasets. It was found that the ground truth (GT) of flame and smoke samples often exhibits a geometric shape with one side longer than the other. To ensure that the added positive samples are within a reasonable scale, a coordinate-based noise addition method based on aspect ratio was proposed. During data preprocessing, the coordinates of the GT were transformed from (x, y, w, h) to (x...). min ,y min ,x max ,y max ), and based on the sine values ​​of width and height The ground truth (GT) is divided into two categories for noise addition. The noisy coordinates (x') are then... min ,y' min ,x' max ,y' max The following formula can be used to calculate it:

[0209]

[0210] Here, μ1, μ2, μ3, and μ4 are perturbation coefficients sampled from a uniform distribution. The perturbation coefficients for positive samples range from {μ1, μ2, μ3, μ4} ∈ (0, 1), while those for negative samples range from {μ1, μ2, μ3, μ4} ∈ (1, 2). When tanα ≤ 1, λ = 2, v = 4. When tanα > 1, λ = 4, v = 2. During label denoising, the ground truth (GT) labels are randomly converted to other categories, and the category values ​​are encoded into embedding vectors. Finally, since the denoising process only occurs during model training, to ensure network compatibility, the noise query and the IoU-Aware query are concatenated and input into the Decoder, but they are independent and not coupled.

[0211] Furthermore, the decoder consists of a deformable attention network, a multi-scale variable attention network, and a feedforward neural network, and each of the deformable attention network, the multi-scale variable attention network, and the feedforward neural network has a residual link after it.

[0212] The deformable attention network is:

[0213]

[0214] Where DeformAtten(·) is the deformable multi-head self-attention operation, k is the index of the sampled key, K is the total number of sampled keys, m is the number of heads in the multi-head attention, and W m W' m For learnable weights, A mqk Let Δp represent the attention weight of the m-th head. mqk z represents the sampling offset of the k-th sampling point in the m-th attention header. q For query, p q Here, ρ is the two-dimensional reference point, M is the input feature, and M is the total number of deformable multi-head self-attention heads.

[0215] Multiscale variable attention network is:

[0216]

[0217] Among them, MSDeformAtten(·) is a multi-scale deformable multi-head self-attention operation. To query the normalized coordinates of the reference point of the element, ρ l For multi-scale input features, L is the number of layers of input features, and φ l For normalization operations, l is the index at the feature layer level, A mlqk For attention weights, Δp mlak This represents the sampling offset of the m-th attention head at the k-th sampling point on the first feature layer.

[0218] Specifically, during the model training phase, the Decoder's input consists of Memory, Class queries (i.e., content queries), and position queries. To improve the detector's real-time performance, multi-scale deformable attention is introduced as a computational scheme for the cross-attention part. Compared to traditional multi-head self-attention, the deformable attention network only uses a small portion of the Key and Query for attention computation, effectively reducing the model's complexity. For a given input feature ρ∈R... C×H×W Let q index a value with content characteristics z. q and a two-dimensional reference point p q For query elements, deformable attention is calculated as follows:

[0219]

[0220] Where k is the index of the sampling key, and K is the total number of sampling keys. Δp mqk and A mqk Let Δp represent the sampling offset and attention weight of the k-th sampling point in the m-th attention head, respectively. mqk ∈R 2 It is a two-dimensional real number with an unconstrained range. Attention weight A mqk ∈[0,1], via Normalize.

[0221] To better accommodate the multi-scale information carried in memory, deformable attention is applied to multi-scale features. For a given set of multi-scale features as input... in The formula for calculating multi-scale deformable attention, based on the normalized coordinates of each query reference point, is as follows:

[0222]

[0223] Where l is the index at the feature layer level, Δp mlpk and A mqk Let A represent the sampling offset of the m-th attention head at the k-th sampling point on the first feature layer, and A represent the attention weight, respectively. mlqk through Normalize. Figure 4 (b) The Decoder structure is shown in detail, where the parameters of the deformable attention are set to M=8 and K=4.

[0224] Furthermore, acquiring preprocessed industrial fire smoke images includes:

[0225] Acquire images of smoke from industrial fires;

[0226] Cleaning of industrial fire smoke images;

[0227] The cleaned industrial fire smoke images are labeled with bounding boxes and categories, and double-checked to obtain preprocessed industrial fire smoke images.

[0228] Specifically, this invention details the industrial fire smoke dataset, experimental environment, and evaluation metrics used for model training and evaluation. To ensure data reliability, the images and videos in this dataset are all from real industrial fire scenarios.

[0229] Considering that current datasets specifically designed for industrial fire smoke detection still suffer from low resolution, poor data diversity, and insufficient data volume, over 5,000 images and 5 fire monitoring videos were collected from various real-world industrial fire smoke scenarios for dataset construction. After data cleaning, 2047 images were retained, and LabelImg was used to annotate the acquired images with bounding boxes and categories. The annotation process categorized all samples into "flame" and "smoke," followed by double checking to ensure annotation quality. Image samples in the dataset are shown below. Figure 9 As shown, due to space limitations, only a few representative images are presented here. After the label files were generated, the dataset was divided into training, validation, and test sets in a 7:1:2 ratio. Three image enhancement algorithms were applied to enhance the training set: ① Randomly select 15% of the images, change their brightness within the range of [0.7:1.3], and generate new images; ② Randomly select 15% of the images, flip them horizontally once, and generate new images; ③ Randomly select 10% of the images from the original images and add blur noise. The final industrial fire smoke dataset includes 4304 training images, 409 validation images, and 206 test images.

[0230] All model training and testing were conducted on the same server, running Ubuntu 16.04 and powered by an Intel(R) Xeon(R) Platinum 8383C CPU. Four NVIDIA GeForce RTX 4090 24GB GPUs were used for training. All models were run on the PyTorch 1.12.1 framework, using Python 3.8.16. The models were optimized using the AdamW optimizer with an initial learning rate of 1×10⁻⁶. -4 Batch size is 8, training epochs are 200, and weight decay rate is 10. -4 .

[0231] To provide a more comprehensive evaluation of the model, this invention uses precision (P), recall (R), average precision (AP), F1-Score, and accuracy (AR) to measure the model's detection accuracy, and uses operation frequency (FPS) and latency (ms) to evaluate the model's real-time performance. The metrics mentioned above can be calculated using the following formulas:

[0232]

[0233] Wherein, Precision is precision, Recall is recall, AP is average precision, F1-Score is the geometric mean of precision and recall, Accuracy is accuracy, TP represents the number of times a sample is correctly predicted as a positive sample, FP represents the number of times a sample is incorrectly predicted as a positive sample, FN represents the number of times a sample is incorrectly predicted as a negative sample, and TN represents the number of times a sample is correctly predicted as a negative sample.

[0234] Since the model is designed for industrial applications, its feasibility for deployment in resource-constrained embedded devices must be carefully considered. Under these circumstances, models with lower GFLOPs and Parameters are preferred. GFLOPs, or gigabit-float operations per second, measures the computational resources required for model operation. Generally, models with higher GFLOPs require more robust hardware support, while Parameters represent the number of parameters in the overall algorithm framework.

[0235] To further demonstrate the effectiveness of IFS-DETR in fire and smoke detection, all experimental results are presented and discussed. A series of detailed experiments were conducted on a self-built industrial fire and smoke dataset and several public fire datasets to further evaluate and compare IFS-DETR with other state-of-the-art models.

[0236] To verify the effectiveness of each improved component in the IFS-DETR architecture, a series of ablation experiments were conducted on the test set of an industrial fire smoke dataset. The experimental results are shown in Table 1. The "√" symbol indicates that the corresponding improvement strategy was adopted. The baseline selected in the experiments used a ResNet50 backbone network and the GIoU loss function. When AFFNet was not used, the feature maps of the last two stages of the backbone network were concatenated with the feature map output by the encoder before being fed into the decoder. CBAM, CA, SE, and ECA were also used to compare and verify the efficiency of EFCA.

[0237] Table 1

[0238]

[0239] Comparing Experiments 1 and 2 in the table, it was found that replacing ResNet50 with LeanNet significantly reduced the number of model parameters. Furthermore, comparing Experiments 4, 5, 6, 7, 8, and 9 with Experiments 17, 18, 19, 20, 21, and 22, it was found that LeanNet is compatible with various structural improvements and optimizations to the IFS-DETR detector, improving the algorithm's lightweight nature. Comparing Experiments 1 and 3, it can be seen that adding AFFNet to IFS-DETR significantly improved the detector's mAP50 without causing a surge in the number of parameters. These experimental results indicate that feature fusion enhances the interaction between objects of different scales, which helps the network to more accurately locate small objects. At the same time, a moderate feature alignment scheme does not add excessive latency, helping to better balance the network's accuracy and speed. Furthermore, comparing Experiments 4, 5, 6, 7, 8, and 9 with Experiments 17, 18, 19, 20, 21, and 22, it was found that modifying the loss function to Inner-SIoU significantly improved the detector's accuracy. The efficiency of EFCA was further verified by comparing it with CBAM, CA, SE, and ECA. In experiments 11, 12, 13, 14, 15, and 16, ECA and EFCA achieved cross-channel information exchange more efficiently due to their unique inter-channel interaction methods. Meanwhile, thanks to the rich frequency component information provided by 2D-DCT, EFCA effectively improved the detector's recognition capability, making it more suitable for fire smoke detection. Finally, by comparing Experiment 1 with Experiments 2, 9, and 22, it was verified that the three improvement methods are not mutually exclusive but rather compatible.

[0240] To evaluate the effectiveness of Inner-SIoU, five different loss functions, including CIoU, were tested. The baseline for the experiments was set to group 15 in Table 1, and the model was trained for 200 epochs on an industrial fire smoke dataset. Figure 10 The first 120 epochs of model training were selected for demonstration.

[0241] The above description is merely a preferred embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A structured, lightweight fire smoke detection method for petrochemical plant areas, characterized in that, Specifically, the following steps are included: Acquire image data; The image data is input into the IFS-DETR model to obtain the detection results. The IFS-DETR model is trained from a training set, which includes preprocessed industrial fire smoke images and corresponding bounding boxes and category labels. The IFS-DETR model consists of a backbone feature extraction network LeanNet, an encoder, a feature fusion network AFFNet, a decoder, and a prediction head. The backbone feature extraction network LeanNet is used to extract features from image data and obtain several image features, wherein the several image features include a first image feature, a second image feature, a third image feature, and a fourth image feature; The encoder is used to obtain the output image features of the encoder based on the fourth image features and the position embedding; The feature fusion network AFFNet is used to fuse the second image features, the third image features, and the output image features of the encoder to form a memory. The decoder is used to obtain the output image features of the encoder based on the memory, content query, and location query. The prediction head is used to classify the output image features of the encoder and predict the bounding boxes of objects to obtain the detection results. Based on the detection results, alarm information is sent to achieve structured, lightweight fire smoke detection for petrochemical plant areas.

2. The structured lightweight fire smoke detection method for petrochemical plant areas as described in claim 1, characterized in that, The backbone feature extraction network LeanNet contains several feature extraction stages. Each feature extraction stage consists of a convolutional layer and several DS-C blocks. Each DS-C block consists of a depth-separable convolutional layer, a standard convolutional layer, a batch normalization layer, and an activation layer. The convolutional layer is used to reduce the dimensionality of the image data; the DS-C block is used to extract features from the reduced image data to obtain several image features. The DS-C block is: d1 = Conv(DSConv(d)) d2 = BN(ReLU(d1)) d3 = Conv(d2) Where d represents the input features of the DS-C block, d1 represents the features after depthwise separable convolution and standard convolution, d2 represents the features after ReLU activation and batch normalization, and d3 represents the features after convolution. out For the output features of the DS-C block, Conv(·) is the standard convolution operation, DSConv(·) is the depthwise separable convolution operation, BN(·) is the batch normalization operation, and ReLU(·) is the activation function. This is for concatenation operations.

3. The structured lightweight fire smoke detection method for petrochemical plant areas as described in claim 1, characterized in that, The encoder consists of a multi-head self-attention network and a feedforward neural network, and both the multi-head self-attention network and the feedforward neural network are followed by a residual link; The multi-head self-attention network is as follows: Where MultiHeadAtten(·) is the multi-head self-attention operation, z q Let q be the query and z be the z. q The index is ρ, the input feature is m, the number of heads in the multi-head attention is M, and the total number of heads in the multi-head attention is W. m W ' m For learnable weights, k is the index of the sampled key, K is the total number of sampled keys, and A mqk ρ represents the attention weight of the m-th head. k The features are represented by the key.

4. The structured lightweight fire smoke detection method for petrochemical plant areas as described in claim 1, characterized in that, Before fusing the second image feature, the third image feature, and the encoder's output image feature, the following steps are included: The output image features of the encoder are mapped, and position information is supplemented using sinusoidal position coding, wherein the sinusoidal position coding is: Where PE(·) represents the encoding result, pos represents the position of the feature, i is the value of the i-th dimension in the position encoding matrix, and d m The dimension of the embedding vector for the model.

5. The structured lightweight fire smoke detection method for petrochemical plant areas as described in claim 4, characterized in that, The feature fusion network AFFNet consists of an information alignment module, a cross-layer information fusion module, and an information refinement module. The information alignment module is used to unify the scale of the second image feature, the third image feature, and the output image feature of the encoder, and obtain aligned feature information. The information alignment module is: d 11 =Avgpool(S2) d 22 S3 d 33 =Interpolation(F1) Where, d 11 The output feature is the high-dimensional feature S2 after dimensionality reduction by average pooling, d 22 For feature S3, d 33 S1 represents the output features of the low-dimensional feature F1 after dimensionality upscaling via bilinear interpolation; S2 represents the feature layer output from the second stage of the backbone network; S3 represents the feature layer output from the third stage of the backbone network; Avgpool(·) is the average pooling operation; Interpolation(·) is the bilinear interpolation operation; and F1 represents the output features of the encoder. For concatenation operations; The cross-layer information fusion module is used to fuse the aligned feature information to obtain the fused feature information; The cross-layer information fusion module is: L1=Conv 1×1 (D) L2 = EFCA(L1) L=Conv 3×3 (L2) Where L1 represents the features after processing with a 1×1 convolution kernel, L2 represents the features after processing with EFCA attention, L represents the features after processing with a 3×3 convolution kernel, and Conv represents the features. 1×1 (·) represents a convolution operation with a 1×1 kernel, D represents the input features, EFCA(·) represents efficient frequency channel attention computation, and Conv 3×3 (·) represents a convolution operation with a 3×3 kernel; The information refinement module is used to refine the fused feature information and, in conjunction with the output image features of the encoder, obtain the Memory. The information refinement module is: E1=RepC3[Conv 1×1 (f1)] E2=RepC3[Conv 1×1 [Interpolation(f2)]] Wherein, E1 is the first set of output features, E2 is the second set of output features, RepC3[·] is the reparameterization module, f1 is the first set of input features, Interpolation(·) is the bilinear interpolation operation, and f2 is the second set of input features.

6. The structured lightweight fire smoke detection method for petrochemical plant areas as described in claim 5, characterized in that, The cross-layer information fusion module consists of a first convolutional layer, an efficient frequency channel attention network, and a second convolutional layer. The first convolutional layer is used to perform preliminary extraction on the aligned feature information to obtain the preliminary extracted feature information; The high-efficiency frequency channel attention adopts a high-efficiency frequency channel attention network based on 2D DCT, which is used to perform inter-channel information interaction on the initially extracted feature information and obtain the feature information after information interaction. Wherein, the 2D DCT and its inverse are: In the formula, Let h be the height of the 2D DCT basis, w be the width of the 2D DCT basis, H be the height of the 2D DCT spectrum, W be the width of the 2D DCT spectrum, i be the number of rows in the real matrix, and j be the number of columns in the real matrix. This is the 2DDCT spectrum, where 2D indicates two-dimensional. For input; The second convolutional layer is used to extract the feature information after the information interaction and obtain the fused feature information.

7. The structured lightweight fire smoke detection method for petrochemical plant areas as described in claim 1, characterized in that, Based on the Memory, content query, and location query, the output image features of the encoder are obtained, including: The IoU-Aware query is introduced into the IFS-DETR model, and the memory is filtered by intersection-union query to obtain the content query; Perform contrastive denoising training on the memory to obtain the location query, wherein the contrastive denoising training includes matching and denoising; The Memory, the content query, and the location query are input into the decoder to obtain the output image features of the encoder.

8. The structured lightweight fire smoke detection method for petrochemical plant areas as described in claim 7, characterized in that, The decoder consists of a deformable attention network, a multi-scale variable attention network, and a feedforward neural network, and each of the deformable attention network, the multi-scale variable attention network, and the feedforward neural network has a residual link after it. The deformable attention network is: Where DeformAtten(·) is the deformable multi-head self-attention operation, k is the index of the sampled key, K is the total number of sampled keys, m is the number of heads in the multi-head attention, and W m W ' m For learnable weights, A mqk Let Δp represent the attention weight of the m-th head. mqk z represents the sampling offset of the k-th sampling point in the m-th attention header. q For query, p q Here, ρ is the two-dimensional reference point, M is the input feature, and M is the total number of deformable multi-head self-attention heads. The multi-scale variable attention network is: Among them, MSDeformAtten(·) is a multi-scale deformable multi-head self-attention operation. To query the normalized coordinates of the reference point of the element, ρ l For multi-scale input features, L is the number of layers of input features, and φ l For normalization operations, l is the index at the feature layer level, A mlqk For attention weights, Δp mlqk This represents the sampling offset of the m-th attention head at the k-th sampling point on the first feature layer.

9. The structured lightweight fire smoke detection method for petrochemical plant areas as described in claim 1, characterized in that, Acquiring the preprocessed industrial fire smoke image includes: Acquire images of smoke from industrial fires; The industrial fire smoke image is cleaned; The preprocessed industrial fire smoke image is obtained by labeling the bounding box and category of the cleaned industrial fire smoke image and performing double checks.

Images