A smoke flame detection method, system and electronic device based on HNetMSFA

Abstract

This specification discloses a smoke and flame detection method, system, and electronic device based on HNetMSFA. The smoke and flame detection method includes: acquiring a smoke and flame image dataset, dividing it into a training set, a validation set, and a test set, and performing data preprocessing; constructing a detection network based on hierarchical multi-scale feature map modulation: the backbone network of the detection network includes a multi-scale feature map modulation visual transformer module; training and testing the constructed detection network using the training set, validation set, and test set to obtain a smoke and flame detection model; performing smoke and flame detection on the image to be detected based on the smoke and flame detection model; and the multi-scale feature map modulation visual transformer module performing global multi-scale feature aggregation. This solves the problem of insufficient detection accuracy and easy false negatives and missed detections in complex environments due to large differences in scale, shape, and texture in existing smoke and flame target detection methods.

Description

Technical Field

[0001] Several embodiments in this specification relate to the field of image detection technology, specifically to the optimization of smoke and flame detection models. Background Technology

[0002] Smoke and flames are the primary visual signs of fire, and accurate and timely automatic detection can buy valuable time for early warning and emergency response. However, due to the translucent, diffuse, and variable shape characteristics of smoke, and the significant differences in size, brightness, and morphology of flames caused by the combustible material and environmental factors, this high degree of diversity poses a serious challenge to the robustness and accuracy of detection algorithms.

[0003] To address the aforementioned challenges, existing research largely focuses on optimizing mainstream object detection frameworks. Many works revolve around the YOLO series of algorithms. For example, some studies combine the Transformer attention mechanism with YOLOv4 to enhance global feature modeling capabilities; others improve YOLOv5 by introducing dilated convolutions to enhance feature extraction for small-scale targets; still others focus on lightweighting the model, employing depthwise separable convolutions to simplify the network structure. Furthermore, to enhance the fusion capability of multi-scale features, some methods introduce a weighted bidirectional feature pyramid into the neck network of YOLOv8.

[0004] However, existing improvement schemes mostly involve the superposition or replacement of single modules, such as optimizing only the neck network or using only a specific lightweight convolution. This results in the model's feature extraction mechanism lacking specificity for the diverse morphologies of fire targets, making it difficult to effectively coordinate global semantics with local details. Consequently, when faced with small-sized, distorted, or low-contrast smoke and flames, existing algorithms have insufficient feature capture capabilities and are easily affected by complex backgrounds, leading to false positives and false negatives. They cannot reliably solve the bottleneck problem of detection accuracy caused by the diversity of target shapes and sizes. Summary of the Invention

[0005] This specification provides an embodiment of a smoke and flame detection method, system, and electronic device based on HNetMSFA. By introducing a multi-scale feature map modulation visual transformer module into the backbone network of the detection network, it enables the parallel aggregation of global context features at different scales, thereby effectively improving the model's detection accuracy for smoke and flame targets with varying shapes and scales.

[0006] The technical solution is as follows:

[0007] Firstly, this specification provides a smoke and flame detection method based on HNetMSFA, comprising the following steps:

[0008] Collect a dataset of smoke and flame images, divide it into training, validation and test sets, and perform data preprocessing.

[0009] A detection network based on hierarchical multi-scale feature map network modulation is constructed: the backbone network of the detection network includes a multi-scale feature map spectrum modulation visual transformer module;

[0010] The constructed detection network was trained and tested using the training set, validation set, and test set to obtain a smoke and flame detection model.

[0011] Smoke and flame detection is performed on the image to be detected based on the smoke and flame detection model.

[0012] The multi-scale feature map modulation visual transformer module performs global multi-scale feature aggregation, including:

[0013] Calculate the QKV matrix based on input features;

[0014] Multiple convolutional networks of different scales are used to aggregate spatial neighborhood information of QKV matrices to obtain multiple sets of aggregated QKV matrices corresponding to different scales;

[0015] A linear attention mechanism is employed to calculate attention features based on the QKV matrix and multiple sets of aggregated QKV matrices corresponding to different scales.

[0016] All attention features obtained through fusion.

[0017] As a preferred embodiment, the linear attention mechanism is employed, which calculates attention features based on the QKV matrix and multiple sets of aggregated QKV matrices corresponding to different scales, including:

[0018] Introduce a multi-scale common parameter matrix;

[0019] A linear attention mechanism including a multi-scale common parameter matrix is ​​adopted to calculate attention features based on the QKV matrix and multiple sets of aggregated QKV matrices corresponding to different scales.

[0020] As a preferred embodiment, during model training, the total loss function of the constructed detection network includes the target detection loss and the input feature map relation network loss;

[0021] The multi-scale feature map modulation visual transformer module performs global multi-scale feature aggregation and also includes:

[0022] The input features of the multi-scale feature map modulation visual transformer module are modeled using a feature relationship graph to obtain a similarity graph network matrix;

[0023] Based on the similarity graph network matrix, the loss term corresponding to each attention feature is calculated to obtain the input feature graph relationship network loss.

[0024] As a preferred embodiment, the loss term corresponding to each attention feature is calculated based on the similarity graph network matrix to obtain the input feature graph relationship network loss, including:

[0025] Based on the similarity graph network matrix, the loss term corresponding to each attention feature is calculated separately;

[0026] The loss term corresponding to each attention feature is weighted and calculated to obtain the input feature map relationship network loss.

[0027] As a preferred embodiment, the input features of the multi-scale feature map modulation visual transformer module include multiple row vectors;

[0028] The input features of the multi-scale feature map modulation visual transformer module are modeled using a feature relationship graph to obtain a similarity graph network matrix, including:

[0029] Calculate the Euclidean distance between each row vector in the input features of the multi-scale feature map modulation visual transformer module to obtain the similarity map network matrix.

[0030] As a preferred embodiment, the construction of the detection network based on hierarchical multi-scale feature spectral network modulation further includes: introducing a distributed offset convolutional network and a moving flippable convolutional block into the backbone network of the detection network.

[0031] As a preferred embodiment, the backbone network of the detection network includes a Stem module and a feature map pyramid structure formed by multiple processing stages;

[0032] The Stem module includes a standard convolutional network and a distributed offset convolutional network;

[0033] The feature map pyramid structure includes a processing stage consisting of a moving flippable convolutional block, and a subsequent processing stage consisting of a cascaded moving flippable convolutional block and a multi-scale feature map modulation visual transformer module.

[0034] As a preferred embodiment, the multi-scale feature map modulation visual transformer module includes a feedforward network and a depthwise separable convolutional layer;

[0035] The feedforward network uses the same expansion ratio as the moving flippable convolutional block;

[0036] All depthwise separable convolutional layers in the feature map pyramid structure have the same kernel size as the moving flippable convolutional blocks;

[0037] The activation functions used in the backbone of the detection network are all the same.

[0038] Secondly, embodiments of this specification provide a smoke and flame detection system based on HNetMSFA, which applies the method described in the first aspect of the above embodiments. The smoke and flame detection system includes:

[0039] The data acquisition and preprocessing module is used to acquire smoke and flame image datasets, divide them into training sets, validation sets, and test sets, and perform data preprocessing.

[0040] The model building module is used to build a detection network based on hierarchical multi-scale feature map modulation: the backbone network of the detection network includes a multi-scale feature map modulation visual transformer module;

[0041] The training and testing module is used to train and test the constructed detection network using the training set, validation set, and test set to obtain the smoke and flame detection model.

[0042] The detection module is used to perform smoke and flame detection on the image to be detected based on the smoke and flame detection model.

[0043] Thirdly, embodiments of this specification provide an electronic device, including a processor and a memory; the processor is connected to the memory; the memory is used to store executable program code; the processor reads the executable program code stored in the memory to run a program corresponding to the executable program code, so as to perform the steps described in the first aspect of the above embodiments.

[0044] Fourthly, embodiments of this specification provide a computer storage medium storing a plurality of instructions adapted for loading by a processor and executing the steps described in the first aspect of the above embodiments.

[0045] The beneficial effects of the technical solutions provided in some embodiments of this specification include at least the following:

[0046] The proposed HNetMSFA model consistently outperforms baseline models in key metrics such as accuracy, recall, and mean precision. In practical applications, it can more reliably identify real smoke and flames while reducing the likelihood of misclassifying background objects as fires.

[0047] By introducing a hierarchical multi-scale feature aggregation mechanism, the model can synchronously and effectively handle targets of various sizes, ranging from small ignition points to large spreading smoke columns. The multi-branch design of the NetMSViT module enables the model to adaptively fuse spatial context information of different ranges, thereby overcoming the limitations of traditional single-scale feature extraction when facing the diversity of target sizes.

[0048] Training with feature map relational network loss constrains the network to maintain semantic consistency of features during deep transformations, which helps improve the model's feature discrimination ability under adverse conditions such as weak contrast between target and background or motion blur, thereby reducing the false negative rate.

[0049] The introduction of the Distributed Offset Convolutional Network (DSConv) makes the model more robust in capturing the local structural features of non-rigid targets such as flames and smoke that are prone to geometric distortion.

[0050] The MBConv module, through its inverted bottleneck structure and channel attention mechanism, enables efficient and refined feature processing with low parameter count and computational cost. It excels at enhancing discriminative channel features related to fine smoke textures and bright core regions of flames, while suppressing irrelevant information. Its built-in residual connections effectively alleviate the vanishing gradient problem, accelerating model convergence.

[0051] The proposed HNetMSFA model network front-end, DSConv, leverages its distributed offset mechanism to robustly extract local features from irregular and distorted targets, laying a high-quality and adaptable feature foundation for subsequent processing. These features are then further refined and enhanced in depth by the MBConv module through an efficient inverted bottleneck and channel attention mechanism, particularly strengthening key information such as smoke texture and flame core.

[0052] The cascaded combination of MBConv and NetMSViT (i.e., the NetMBMSViT module) jointly achieves feature preprocessing and aggregation at a deep layer of the backbone network. MBConv first performs secondary refinement on the input features, and NetMSViT then simultaneously aggregates global context at multiple scales. This allows global attention computation to be built upon locally enhanced features, achieving a seamless connection between local detail awareness and global semantic modeling.

[0053] The solution employs a series of unified hyperparameter designs, which significantly reduces the complexity of model parameter tuning and enhances the regularity and reproducibility of the architecture. Without significantly increasing the number of model parameters or computational costs, the efficient combination of modules achieves a substantial improvement in detection performance, making the model both effective and efficient. Attached Figure Description

[0054] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0055] Figure 1 This is a schematic flowchart of a smoke and flame detection method based on HNetMSFA provided in the embodiments of this specification.

[0056] Figure 2 This is a schematic diagram of the architecture of the multi-scale feature map modulation visual transformer module in the HNetMSFA backbone network provided in the embodiments of this specification.

[0057] Figure 3 This is a schematic diagram of the architecture of the HNetMSFA backbone network provided in the embodiments of this specification.

[0058] Figure 4 This is a complete architecture diagram of the HNetMSFA-YOLOv10 model provided in the embodiments of this specification.

[0059] Figure 5 This is a schematic diagram of the structure of a smoke and flame detection system based on HNetMSFA provided in the embodiments of this specification.

[0060] Figure 6 This is a schematic diagram of the structure of an electronic device provided in the embodiments of this specification. Detailed Implementation

[0061] The technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings.

[0062] The terms "first," "second," "third," etc., in the description, claims, and accompanying drawings are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such processes, methods, products, or apparatus.

[0063] The following description provides examples and does not limit the scope, applicability, or examples set forth in the claims. Changes may be made to the function and arrangement of the described elements without departing from the scope of this specification. Various processes or components may be appropriately omitted, substituted, or added to the examples. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Furthermore, features described with respect to some examples may be combined into other examples.

[0064] See attached document Figure 1 , Figure 1A flowchart illustrating a smoke and flame detection method based on HNetMSFA, provided as an embodiment of this specification, may include at least the following steps:

[0065] Step 102: Collect a dataset of smoke and flame images, divide it into training, validation and test sets, and perform data preprocessing;

[0066] Step 104: Construct a detection network based on hierarchical multi-scale feature map network modulation: The backbone network of the detection network includes a multi-scale feature map modulation visual transformer module;

[0067] Step 106: Train and test the constructed detection network using the training set, validation set, and test set to obtain the smoke and flame detection model;

[0068] Step 108: Perform smoke and flame detection on the image to be detected based on the smoke and flame detection model;

[0069] The multi-scale feature map modulation visual transformer module performs global multi-scale feature aggregation, including:

[0070] Calculate the QKV matrix based on input features;

[0071] Multiple convolutional networks of different scales are used to aggregate spatial neighborhood information of QKV matrices to obtain multiple sets of aggregated QKV matrices corresponding to different scales;

[0072] A linear attention mechanism is employed to calculate attention features based on the QKV matrix and multiple sets of aggregated QKV matrices corresponding to different scales.

[0073] All attention features obtained through fusion.

[0074] This implementation method aims to address the problem of insufficient detection accuracy for flame and smoke targets in complex scenes due to significant differences in scale, shape, and texture. Its core lies in innovatively improving the backbone network of mainstream detection frameworks. By introducing a specially designed multi-scale feature map modulation visual transformer module, a detection model based on the HNetMSFA (Hierarchical network Multi-scale Feature Modulation and Aggregation) backbone network is constructed. This model can simultaneously capture local details and global semantics and is robust to multi-scale targets.

[0075] To illustrate, the process begins with data preparation: collecting and segmenting a dataset of smoke and flame images, followed by necessary preprocessing. Next, an improved detection network is constructed by integrating a multi-scale feature map modulation visual transformer module (NetMSViT) into the backbone network. This module replaces and enhances some feature extraction structures of the original network with a more powerful feature extraction and aggregation mechanism. Then, the improved network is trained and tested end-to-end using the prepared data, resulting in an optimized smoke and flame detection model. Finally, this model is applied to a real-world scenario; inputting an image to be detected yields the detection results for smoke and flames.

[0076] Explaining the basic idea of ​​the standard Vision Transformer (ViT), which is to segment an image into a sequence of blocks and establish global dependencies through a self-attention mechanism, this embodiment improves upon the standard Vision Transformer by obtaining a multi-scale feature map modulation Vision Transformer module, as shown in the attached diagram. Figure 2 This achieves the global feature aggregation process of "multi-scale convolution modulation".

[0077] Specifically, the process begins with a linear projection of the input feature map to generate the query (Q), key (K), and value (V) matrices necessary for attention computation. The Q, K, and V matrices respectively serve to propose the query, provide the keywords, and carry the feature values, forming the foundation for the attention mechanism to calculate association weights. However, these initial Q, K, and V matrices are not directly used for attention computation. Instead, multiple (e.g., two) convolutional networks with different receptive fields (such as 3×3 and 5×5) are used to perform parallel spatial neighborhood information aggregation on these three matrices—a process known as "multi-scale convolutional modulation." This ensures that each set of Q, K, and V matrices is pre-incorporated with local contextual information of different scales before participating in global computation. For example, the result of the Q, K, and V matrices after undergoing 3×3 depthwise separable convolutions is Q... 3×3 K 3 ×3 V 3×3 The result after a 5×5 depthwise separable convolution is Q. 5×5 K 5×5 V 5×5 .

[0078] Subsequently, an efficient linear attention mechanism (ReLU Linear Attention) is employed, which computes multi-path attention features in parallel based on the original Q, K, V matrices and multiple sets of Q, K, V matrices modulated by convolution at different scales. This linear attention mechanism reduces computational complexity from quadratic to linear levels while maintaining an approximate global receptive field, achieving a balance between efficiency and performance.

[0079] Of the calculated features, one path retains the most original global associations, while the others each contain global representations of local contexts at different granularities. Finally, these attention features calculated from different feature perspectives are fused (C) to generate an enhanced multi-scale global feature map that contains both detailed local structures and broad contextual semantics.

[0080] The feature map output by this module, as the high-level semantic output of the improved backbone network, is fed into the subsequent neck network and detection head, ultimately completing the accurate localization and classification of smoke and flame targets in the image.

[0081] While traditional vision (ViT) attention mechanisms can establish global associations, they are computationally expensive and relatively weak at capturing local details. The multi-scale feature map modulation vision transformer module combines multi-scale convolution aggregation with an efficient linear attention mechanism, effectively enhancing the model's ability to represent features of morphologically variable targets such as flames and smoke, thereby improving the accuracy and robustness of detection in complex environments.

[0082] In one embodiment of this specification, the linear attention mechanism is employed to calculate attention features based on the QKV matrix and multiple sets of aggregated QKV matrices corresponding to different scales, including:

[0083] Introduce a multi-scale common parameter matrix;

[0084] A linear attention mechanism including a multi-scale common parameter matrix is ​​adopted to calculate attention features based on the QKV matrix and multiple sets of aggregated QKV matrices corresponding to different scales.

[0085] Interpretationally, in a multi-branch parallel attention computation framework, multiple branches (original scale branch, 3×3 modulation branch, 5×5 modulation branch, etc.) independently perform feature transformation and attention computation. Although this can capture information under different receptive fields, it may also lead to differences or even fragmentation in the feature representations learned by each branch, which is not conducive to the model's cohesive understanding of the discriminative features of smoke and flame targets at different scales.

[0086] Therefore, in this embodiment, a multi-scale common parameter matrix W that can be trained and learned is explicitly introduced into the calculation formula of the linear attention mechanism.s This matrix is ​​shared throughout the attention calculation process of all branches. Its purpose is to mathematically constrain and guide attention branches at different scales, enabling them to learn and reinforce common features that are beneficial for object detection at all scales, thereby achieving feature alignment and knowledge sharing across scales.

[0087] Specifically, in conjunction with the appendix Figure 2 When calculating the attention features for each path (whether based on the original Q, K, V matrices or the Q, K, V matrices modulated by 3×3 or 5×5 convolution), a formula containing the common parameter matrix is ​​used. The attention features output based on the original Q, K, V matrices are matrix O, where each row of O is a vector O0. i (i=1,2,3...,N) is calculated using the following formula:

[0088] ;

[0089] Q i (i=1,2,3...,N) is the i-th row vector of the Q matrix. j V j (j=1,2,3...,N), where W represents the j-th row vector of the K and V matrices, respectively. s This refers to the introduced multi-scale common parameter matrix. ReLU is an activation function used to implement the linear attention mechanism to reduce computational complexity.

[0090] Interpretive, in which the multi-scale common parameter matrix W s by In the form of a global, position-independent additive bias term, it is added to the weighted sum of attention weights over the value vector V. This ensures that the added term is a positive semi-definite matrix with good mathematical properties. Multi-scale common parameter matrix W s A common, data-driven correction is applied to the feature aggregation at all locations. More importantly, this correction is not targeted at any particular local pattern, but rather aims to characterize the common information patterns that are ubiquitous across features at all scales.

[0091] By sharing the same W across all multi-scale attention branches s The matrix is ​​used, and forward and backward propagation training is performed using the above formula. The network can automatically optimize W. s This allows it to represent common semantic features that are important and consistent across scales for flame and smoke detection (e.g., the high-temperature core brightness pattern of flames, the diffusion texture edge features of smoke, etc.). Therefore, In the computation, this term acts as a feature consensus amplification or common feature basis, effectively promoting the consistency of multi-scale feature representations, enhancing the synergistic effect when the model integrates multi-scale information, and thus improving the robustness and accuracy of the final detection.

[0092] Similarly, the attention features based on the output of the Q, K, and V matrices after 3×3 and 5×5 convolution modulation are matrix O. 3 ×3 O 5×5 O 3×3 O 5×5 Each row vector O i 3×3 O i 5×5 (i=1,2,3...,N) is calculated using the following formula:

[0093] ;

[0094] ;

[0095] Among them, O i 3×3 O i 5×5 (i=1,2,3,...,N) are the i-th row vectors of the Q-matrix after being modulated by 3×3 and 5×5 convolutions respectively, and K j 3×3 K j 5×5 (j=1,2,3,...,N) are the j-th row vectors of the K matrix after being modulated by 3×3 and 5×5 convolutions respectively, V j 3×3 V j 5×5 (j=1,2,3...,N) is the j-th row vector of the V matrix after being modulated by 3×3 and 5×5 convolutions, respectively.

[0096] In one embodiment of this specification, during model training, the total loss function of the constructed detection network includes target detection loss and input feature map relation network loss;

[0097] The multi-scale feature map modulation visual transformer module performs global multi-scale feature aggregation and also includes:

[0098] The input features of the multi-scale feature map modulation visual transformer module are modeled using a feature relationship graph to obtain a similarity graph network matrix;

[0099] Based on the similarity graph network matrix, the loss term corresponding to each attention feature is calculated to obtain the input feature graph relationship network loss.

[0100] This implementation further defines the optimization objective of the model training process. In deep learning model training, the loss function guides model parameter updates. The total loss function optimized by the detection network constructed in this method includes not only the conventional object detection loss used to directly optimize object localization and classification accuracy, but also... Furthermore, an input feature map relational network loss is introduced. The resulting composite loss function guides the model simultaneously from two levels: outcome supervision and feature representation learning process supervision, enabling it to learn more discriminative feature representations with consistent internal structure.

[0101] Combined with appendix Figure 2 The computation of the input feature map relation network loss is deeply coupled with the workflow of the multi-scale feature map spectral modulation visual transformer module (NetMSViT). The construction logic begins with performing a relation modeling on the input features of this module.

[0102] Specifically, during training, whenever the features of an image sample are input into the NetMSViT module, a similarity graph network matrix G is first constructed based on these original input features. This matrix quantifies the pairwise similarity relationships between all basic units (visual lexical units) in the input features. In the input feature space, semantically similar regions (such as feature points belonging to the same flame region) should have high similarity, while regions with large semantic differences should have low similarity.

[0103] Subsequently, after the NetMSViT module computes the multi-path attention features in parallel, the constructed similarity graph network matrix G is used to measure whether the attention features of each output path inherit the semantic relationship structure that should exist in the input features. The computation process calculates a corresponding loss term for each attention feature output path. , , These loss terms numerically reflect the degree of deviation between the output and input feature structure of the path; the smaller the deviation, the lower the loss, which means that the features still maintain their semantic clustering characteristics well after scaling and transformation.

[0104] Finally, these loss terms, corresponding to different attention branches, are summed to form the total input feature map relational network loss. This loss is combined with the object detection loss and participates in the backpropagation of model training. This means that the model's optimization constraints not only aim for the accuracy of the final detection box but also consciously maintain the clarity and consistency of feature semantics during internal feature extraction and fusion. This design effectively alleviates the feature representation degradation or confusion that may occur in complex networks (especially multi-branch structures) during training, promoting more robust and interpretable results from multi-scale feature aggregation. This fundamentally improves the model's robustness in representing morphologically variable targets such as smoke and flames, ultimately contributing to improved detection accuracy.

[0105] In one embodiment of this specification, the step of calculating the loss term corresponding to each attention feature based on the similarity graph network matrix to obtain the input feature graph relationship network loss includes:

[0106] Based on the similarity graph network matrix, the loss term corresponding to each attention feature is calculated separately;

[0107] The loss term corresponding to each attention feature is weighted and calculated to obtain the input feature map relationship network loss.

[0108] This embodiment further clarifies the specific composition and calculation method of the input feature map relationship network loss. Using the similarity graph network matrix G as the benchmark ground value for measuring the consistency of feature relationships, the corresponding loss term is calculated for each attention feature output by the NetMSViT module.

[0109] Specifically, the initial attention feature matrix O output by the module and the attention feature matrix O obtained after 3×3 and 5×5 scale convolution modulation are calculated respectively. 3×3 and O 5×5 The difference between the similarity graph network matrix G and the similarity graph network matrix G is calculated using a trace form of the graph-based Laplacian matrix, which aims to constrain the relational structure of the output features to be as consistent as possible with the relational structure of the input features.

[0110] In addition, the Laplacian matrix of a graph is an important mathematical object used to describe the structure of a graph and the relationships between nodes. It characterizes the structure of a graph through its adjacency relationships and the degree of its nodes, and is widely used in graph theory, graph neural networks, spectral graph theory, and other fields.

[0111] Taking the initial attention feature O as an example, its corresponding loss term The calculation formula is:

[0112] ;

[0113] Where D is the degree matrix of G, and (D−G) is the Laplacian matrix of the graph. The smaller the value of this loss term, the smoother the row vectors of the feature matrix O (i.e., the representations of each feature point) are on the spectral space defined by the graph Laplacian matrix. This means that in feature O, similar points in the original input features remain close, while dissimilar points remain far apart. Similarly, the loss term corresponding to the 3×3 scale branch can be calculated separately. Loss term with 5×5 scale branch :

[0114] ;

[0115] .

[0116] In obtaining respectively , , Finally, the final loss function for the input feature maps is not a direct sum of the input features, but rather a weighted calculation using adjustable weight coefficients. The total loss function is represented as follows: for:

[0117] ;

[0118] in, The target detection loss is defined by α, β, and γ, which are preset weighting coefficients.

[0119] For targets with drastically varying shapes and scales, such as smoke and flames, a wider range of contextual information (provided by 5×5 convolutional aggregation) is more important for building robust feature representations. Therefore, the corresponding loss term can be assigned the highest weight, and during backpropagation, the model parameters will be adjusted to prioritize the quality of large-scale contextual feature representations. For example, the weight coefficients α, β, and γ are set to 0.5, 0.7, and 0.9, respectively. The setting of γ>β>α indicates that during training, the model prioritizes optimizing and constraining the feature relationships learned by the attention branch with the largest receptive field (5×5), ensuring a high degree of consistency with the semantic relationships of the input features. The 3×3 scale branch is then given secondary attention, and the original scale branch is the last consideration.

[0120] This embodiment not only imposes constraints on the learning of each multi-scale attention feature while maintaining the semantic relationship of the input, but also flexibly guides the model to perform focused optimization on the consistency learning of features at different scales through differentiated weight configuration (for example, higher weights can be assigned to scale branches that pay more attention to local details), thereby improving the representation quality of the multi-scale feature aggregation module more finely and effectively.

[0121] In one embodiment of this specification, the input features of the multi-scale feature map modulation visual transformer module include multiple row vectors;

[0122] The input features of the multi-scale feature map modulation visual transformer module are modeled using a feature relationship graph to obtain a similarity graph network matrix, including:

[0123] Calculate the Euclidean distance between each row vector in the input features of the multi-scale feature map modulation visual transformer module to obtain the similarity map network matrix.

[0124] In deep learning feature representation, a two-dimensional feature map is typically unfolded into a two-dimensional matrix in spatial dimension before being processed by a Transformer-type module. This means the input features consist of multiple row vectors, each corresponding to a "visual lexical" or a local feature region in the original input feature map, represented along the feature channel dimension. Therefore, the entire input feature can be represented as a matrix X with N rows (number of visual lexicals) and F columns (feature dimension), where the i-th row x... i This represents the feature vector of the i-th visual word.

[0125] In this embodiment, the similarity graph network matrix G is directly measured based on the pairwise Euclidean distance between row vectors (i.e., visual word features). This is achieved by calculating the distance between any two row vectors x. i With x j The square of the Euclidean distance between them (i.e., ||x) i -x j || 2 The distance can quantify the similarity between two feature points in the original high-dimensional feature space. The smaller the distance, the more similar the two feature points are in the initial feature representation; the larger the distance, the greater their difference.

[0126] To construct a graph network that represents "similarity" rather than "difference" (in graph representations, a larger edge weight usually indicates greater similarity between nodes), it is necessary to transform the Euclidean distance, a measure of difference, into a measure of similarity. This is typically achieved using a monotonically decreasing function.

[0127] For example, a Gaussian kernel function is used for transformation. Each element G in the similarity graph network matrix G... ij It can be calculated using the following formula:

[0128]

[0129] Here, σ is a scaling parameter used to adjust the sensitivity of distance to the influence of similarity. It is mapped to a value between 0 and 1 through exponential negative operations, thus directly obtaining the matrix elements G representing similarity. ij When two feature vectors are exactly the same, the distance is 0 and the similarity is 1; as the distance increases, the similarity index decays to close to 0.

[0130] In one embodiment of this specification, the construction of a detection network based on hierarchical multi-scale feature spectral network modulation further includes: introducing a distributed offset convolutional network and a moving flippable convolutional block into the backbone network of the detection network.

[0131] This embodiment further introduces and integrates two convolutional processing units, the Distributed Offset Convolutional Network (DSConv) and the Moving Flipped Convolutional Block (MBConv), into the backbone of the detection network.

[0132] Specifically, the Distributed Offset Convolutional Network (DSConv) is typically deployed at the front end of the network, responsible for performing basic and efficient local feature extraction on the original input image. Its "distributed offset" mechanism decomposes the traditional convolutional kernel into a variable quantization kernel and a distributed offset network. This design makes it more adaptable than standard convolution when dealing with targets with irregular shapes, blurred boundaries, and prone to geometric distortions, such as flames and smoke. It can more robustly capture the initial local structural features of such targets while maintaining low memory consumption and high computational speed, providing higher-quality and more discriminative low-level feature representations for subsequent deep processing.

[0133] Moving flippable convolutional blocks (MBConv) typically operate after the primary features extracted by DSConv, responsible for the deep representation, refinement, and channel information enhancement of these features. MBConv combines an inverted bottleneck structure, depthwise separable convolution, and channel attention mechanisms (such as the SE module), achieving an excellent balance between computational efficiency and representational power. It can perform non-linear transformations on input features, efficiently enhancing discriminative information related to smoke texture details, flame core regions, etc., while suppressing irrelevant features. Its built-in residual connections also help alleviate the vanishing gradient problem in deep networks, stabilizing the training process.

[0134] This embodiment designs the NetMSViT module in conjunction with DSConv and MBConv to form a complete backbone network. In this architecture, DSConv specializes in identifying the initial contours of irregular targets in complex backgrounds; MBConv refines and purifies the initial features; and NetMSViT performs global semantic association and decision-level fusion of multi-scale local cues provided by the aforementioned modules. This hierarchical processing paradigm enables the network to systematically address the core challenges of multi-scale, morphologically diverse, and detail- and semantically complex smoke and flame detection, thereby significantly enhancing the overall feature extraction capabilities of the backbone network.

[0135] In one embodiment of this specification, the backbone network of the detection network includes a Stem module and a feature map pyramid structure formed by multiple processing stages;

[0136] The Stem module includes a standard convolutional network and a distributed offset convolutional network;

[0137] The feature map pyramid structure includes a processing stage consisting of a moving flippable convolutional block, and a subsequent processing stage consisting of a cascaded moving flippable convolutional block and a multi-scale feature map modulation visual transformer module.

[0138] This embodiment provides a specific architectural definition for the improved backbone network, which consists of two main parts: an input stem module and a feature map pyramid structure that includes four processing stages.

[0139] Specifically, see attached document. Figure 3 The Stem module, the starting point of the network, consists of a standard convolutional network (Conv) and a distributed offset convolutional network (DSConv). The standard convolutional network performs fast, basic initial downsampling and feature mapping; while the subsequent DSConv utilizes its unique distributed offset mechanism to focus on robustly extracting local structural features of irregular targets such as flames and smoke. The Stem module efficiently transforms the original input image into a primary feature map rich in discriminative local information, suitable for deep network processing.

[0140] The feature map pyramid structure is the main body and core of the backbone network, responsible for progressively extracting multi-scale features from primary features, with increasingly deeper semantic levels and progressively smaller spatial scales. The pre-processing stages (such as the first and second stages) consist solely of a mobile flippable convolutional block (MBConv). In this stage, the network utilizes the efficient refinement and channel enhancement capabilities of the MBConv module to deepen the features from the Stem module, strengthening discriminative information relevant to the target. The subsequent processing stages (such as the third and fourth stages) consist of a cascaded mobile flippable convolutional block (MBConv) and a multi-scale feature map modulation visual transformer module (NetMSViT). This cascaded combination forms a more powerful composite module (NetMBMSViT). In this architecture, MBConv first preprocesses and enhances the input features before feeding them into the NetMSViT module. NetMSViT leverages its advantages in global modeling and multi-scale fusion to perform high-level semantic aggregation of the features. This cascaded design of local refinement (MBConv) and global multi-scale aggregation (NetMSViT) further improves the model's detection performance.

[0141] The feature maps output from these different processing stages form a multi-scale feature map pyramid. For example... Figure 4As shown, the outputs of stages 2, 3, and 4 constitute this pyramid. These feature maps, containing different levels of semantic and spatial details, will be fed into the neck network of the detection model for the final detection of smoke and flame targets at different scales.

[0142] In one embodiment of this specification, the multi-scale feature map modulation visual transformer module includes a feedforward network and a depthwise separable convolutional layer;

[0143] The feedforward network uses the same expansion ratio as the moving flippable convolutional block;

[0144] All depthwise separable convolutional layers in the feature map pyramid structure have the same kernel size as the moving flippable convolutional blocks;

[0145] The activation functions used in the backbone of the detection network are all the same.

[0146] This embodiment clarifies that the multi-scale feature map modulation visual transformer module (NetMSViT) internally includes a feedforward network and depthwise separable convolutional layers. (Refer to...) Figure 2 After completing multi-branch attention calculation and feature fusion, the features undergo non-linear interaction and dimensional transformation between channels through a feed-forward network (FNN), and lightweight spatial information fusion and enhancement through a deep separable convolutional layer (DWConv), ultimately outputting the processed features.

[0147] In this architecture:

[0148] The scaling ratio used by the feedforward network within the NetMSViT module is the same as that used within the Moving Flipped Convolutional Block (MBConv). The scaling ratio is a key parameter controlling the degree of dimensional inflation in the intermediate layers of these two modules. In MBConv, it determines the scaling factor of the first 1×1 convolution; in the feedforward network of the Transformer architecture, it determines the dimension of the output of the first fully connected layer. Forcing both to the same value (e.g., typically e=4) means that feature dimension transformations follow a consistent scaling strategy across different regions of the network (convolution-dominated regions and attention-dominated regions). This effectively reduces the complexity of hyperparameter search and promotes coordination in feature capacity across different parts of the network, achieving a unified and modular architecture.

[0149] In the feature map pyramid structure, all depthwise separable convolutional layers and all moving flippable convolutional blocks use the same kernel size (e.g., uniformly set to k=5, providing a moderate receptive field). This uniform kernel size ensures that the local neighborhood range relied upon for spatial aggregation is consistent across different depths and modules, enhancing the predictability and consistency of network behavior. It also simplifies hyperparameters.

[0150] In the backbone of the entire detection network, all learnable layers use the same activation function (e.g., a uniform hard swish function). The activation function is the core element introducing non-linearity. Using the same activation function across all layers eliminates the inconsistencies in training dynamics that could arise from different activation functions. This makes the forward and backward propagation processes of the entire network more regular, further simplifying training tuning and contributing to improved training stability.

[0151] In summary, this method follows the engineering design principles of structural regularity and parameter simplification. The unified design of the expansion ratio, convolution kernel size, and activation function significantly reduces the design complexity and parameter tuning burden of the model, allowing researchers and engineers to focus more on the macroscopic structure of the network. At the same time, these consistency constraints also enhance the synergy between the various components within the network, enabling the complex backbone network composed of heterogeneous modules such as DSConv, MBConv, and NetMSViT to be trained efficiently as a coordinated whole. Ultimately, while maintaining the model's powerful feature extraction capabilities, it ensures good trainability, reproducibility, and deployment friendliness.

[0152] For example, the HNetMSFA backbone network constructed in this method is integrated into the YOLOv10 detection architecture to construct the HNetMSFA-YOLOv10 model, as shown in the appendix. Figure 4 Extensive experiments on two fire datasets demonstrate that our proposed model consistently outperforms the baseline model, exhibiting a significant advantage in smoke and flame detection.

[0153] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0154] Please refer to the following. Figure 5 , Figure 5A schematic diagram of a smoke and flame detection system based on HNetMSFA provided in an embodiment of this specification is shown.

[0155] The smoke and flame detection system 500 includes:

[0156] The data acquisition and preprocessing module 501 is used to acquire smoke and flame image datasets, divide them into training set, validation set and test set and perform data preprocessing.

[0157] The model building module 502 is used to build a detection network based on hierarchical multi-scale feature map network modulation: the backbone network of the detection network includes a multi-scale feature map modulation visual transformer module;

[0158] The training and testing module 503 is used to train and test the constructed detection network using the training set, validation set, and test set to obtain a smoke and flame detection model.

[0159] The detection module 504 is used to perform smoke and flame detection on the image to be detected based on the smoke and flame detection model.

[0160] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the smoke and flame detection system embodiments are basically similar to the smoke and flame detection method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the smoke and flame detection method embodiments.

[0161] Please see Figure 6 The diagram shown is a structural schematic of an electronic device provided in an embodiment of this specification.

[0162] like Figure 6 As shown, the electronic device 600 may include: at least one processor 601, at least one network interface 604, user interface 603, memory 605, and at least one communication bus 602.

[0163] The communication bus 602 can be used to realize the connection and communication of the above components.

[0164] The user interface 603 may include buttons, and the optional user interface may also include a standard wired interface or a wireless interface.

[0165] The network interface 604 may include, but is not limited to, Bluetooth modules, NFC modules, Wi-Fi modules, etc.

[0166] The processor 601 may include one or more processing cores. The processor 601 connects to various parts within the electronic device 600 using various interfaces and lines. It executes various functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in the memory 605, and by calling data stored in the memory 605. Optionally, the processor 601 may be implemented using at least one hardware form selected from DSP, FPGA, and PLC. The processor 601 may integrate one or more of the following: CPU, GPU, and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the content required for display; and the modem handles wireless communication. It is understood that the modem may also not be integrated into the processor 601 and may be implemented as a separate chip.

[0167] The memory 605 may include RAM or ROM. Optionally, the memory 605 may include a non-transitory computer-readable medium. The memory 605 may be used to store instructions, programs, code, code sets, or instruction sets. The memory 605 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as touch function, sound playback function, image playback function, etc.), instructions for implementing the above-described method embodiments, etc.; the data storage area may store data involved in the above-described method embodiments, etc. Optionally, the memory 605 may also be at least one storage device located remotely from the aforementioned processor 601. As a computer storage medium, the memory 605 may include an operating system, a network communication module, a user interface module, and a smoke and flame detection application. The processor 601 may be used to call the smoke and flame detection application stored in the memory 605 and execute the steps of the smoke and flame detection method mentioned in the foregoing embodiments.

[0168] This specification also provides a computer-readable storage medium storing instructions that, when executed on a computer or processor, cause the computer or processor to perform one or more steps in the above-described smoke and flame detection method embodiments. If the constituent modules of the above-described electronic device are implemented as software functional units and sold or used as independent products, they can be stored in the computer-readable storage medium.

[0169] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this specification are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in or transmitted through a computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., Digital Versatile Discs (DVDs)), or semiconductor media (e.g., Solid State Disks (SSDs)).

[0170] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. The aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks. Unless otherwise specified, the technical features of this embodiment and its implementation can be combined arbitrarily.

[0171] The embodiments described above are merely preferred embodiments of this specification and are not intended to limit the scope of this specification. Any modifications and improvements made by those skilled in the art to the technical solutions of this specification without departing from the spirit of this specification should fall within the protection scope defined by the claims of this specification.

Claims

1. A smoke and flame detection method based on HNetMSFA, characterized in that, Includes the following steps: Collect a dataset of smoke and flame images, divide it into training, validation and test sets, and perform data preprocessing. A detection network based on hierarchical multi-scale feature map network modulation is constructed: the backbone network of the detection network includes a multi-scale feature map spectrum modulation visual transformer module; The constructed detection network was trained and tested using the training set, validation set, and test set to obtain a smoke and flame detection model. Smoke and flame detection is performed on the image to be detected based on the smoke and flame detection model. The multi-scale feature map modulation visual transformer module performs global multi-scale feature aggregation, including: Calculate the QKV matrix based on input features; Multiple convolutional networks of different scales are used to aggregate spatial neighborhood information of QKV matrices to obtain multiple sets of aggregated QKV matrices corresponding to different scales; A linear attention mechanism is employed to calculate attention features based on the QKV matrix and multiple sets of aggregated QKV matrices corresponding to different scales. All attention features obtained through fusion.

2. The smoke and flame detection method based on HNetMSFA according to claim 1, characterized in that: The linear attention mechanism is employed to calculate attention features based on the QKV matrix and multiple aggregated QKV matrices corresponding to different scales, including: Introduce a multi-scale common parameter matrix; A linear attention mechanism including a multi-scale common parameter matrix is ​​adopted to calculate attention features based on the QKV matrix and multiple sets of aggregated QKV matrices corresponding to different scales.

3. The smoke and flame detection method based on HNetMSFA according to claim 1, characterized in that: During model training, the total loss function of the constructed detection network includes the target detection loss and the input feature map relation network loss; The multi-scale feature map modulation visual transformer module performs global multi-scale feature aggregation and also includes: The input features of the multi-scale feature map modulation visual transformer module are modeled using a feature relationship graph to obtain a similarity graph network matrix; Based on the similarity graph network matrix, the loss term corresponding to each attention feature is calculated to obtain the input feature graph relationship network loss.

4. The smoke and flame detection method based on HNetMSFA according to claim 3, characterized in that: The loss term for each attention feature based on the similarity graph network matrix is ​​calculated to obtain the input feature graph relation network loss, including: Based on the similarity graph network matrix, the loss term corresponding to each attention feature is calculated separately; The loss term corresponding to each attention feature is weighted and calculated to obtain the input feature map relationship network loss.

5. The smoke and flame detection method based on HNetMSFA according to claim 3, characterized in that: The input features of the multi-scale feature map modulation visual transformer module include multiple row vectors; The input features of the multi-scale feature map modulation visual transformer module are modeled using a feature relationship graph to obtain a similarity graph network matrix, including: Calculate the Euclidean distance between each row vector in the input features of the multi-scale feature map modulation visual transformer module to obtain the similarity map network matrix.

6. The smoke and flame detection method based on HNetMSFA according to claim 1, characterized in that: The construction of the detection network based on hierarchical multi-scale feature spectral network modulation also includes: introducing a distributed offset convolutional network and a moving flippable convolutional block into the backbone network of the detection network.

7. The smoke and flame detection method based on HNetMSFA according to claim 6, characterized in that: The backbone of the detection network includes a Stem module and a feature map pyramid structure formed by multiple processing stages; The Stem module includes a standard convolutional network and a distributed offset convolutional network; The feature map pyramid structure includes a processing stage consisting of a moving flippable convolutional block, and a subsequent processing stage consisting of a cascaded moving flippable convolutional block and a multi-scale feature map modulation visual transformer module.

8. The smoke and flame detection method based on HNetMSFA according to claim 7, characterized in that: The multi-scale feature map modulation visual transformer module includes a feedforward network and a depthwise separable convolutional layer; The feedforward network uses the same expansion ratio as the moving flippable convolutional block; All depthwise separable convolutional layers in the feature map pyramid structure have the same kernel size as the moving flippable convolutional blocks; The activation functions used in the backbone of the detection network are all the same.

9. A smoke and flame detection system based on HNetMSFA, employing the smoke and flame detection method as described in any one of claims 1-8, characterized in that, The smoke and flame detection system includes: The data acquisition and preprocessing module is used to acquire smoke and flame image datasets, divide them into training sets, validation sets, and test sets, and perform data preprocessing. The model building module is used to build a detection network based on hierarchical multi-scale feature map modulation: the backbone network of the detection network includes a multi-scale feature map modulation visual transformer module; The training and testing module is used to train and test the constructed detection network using the training set, validation set, and test set to obtain the smoke and flame detection model. The detection module is used to perform smoke and flame detection on the image to be detected based on the smoke and flame detection model.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1-8.

Images