A method for constructing a lightweight flame recognition deep learning model integrating global and local features

Abstract

The present invention discloses a method for constructing a lightweight flame recognition deep learning model that integrates global and local features, belonging to the field of image recognition processing technology. In response to the problems of existing flame recognition network models, such as the large number of network parameters and high latency, the present invention provides a lightweight flame recognition deep learning model that integrates global and local features. The method first collects and gathers flame image data to construct a flame recognition dataset. Secondly, a lightweight Transformer module (E‑TB) is constructed. A CTB module based on the lightweight Transformer module (E‑TB) and a convolutional neural network (CNN) is then constructed. Finally, a ShuffleNet V2 module and a CTB module are stacked in series to achieve step-by-step feature extraction from shallow to deep layers, ultimately constructing a network framework (SConvTrans) for global and local feature extraction. Finally, the constructed flame recognition dataset is divided into a training set and a test set, and the constructed network framework (SConvTrans) is trained and tested using the data to obtain a lightweight flame recognition deep learning model.

Description

Technical Field

[0001] The present invention belongs to the technical field of image recognition and processing, and specifically relates to a method for constructing a lightweight flame recognition deep learning model that integrates global and local features. Background Art

[0002] In recent years, flame recognition technology based on image processing has provided a more accurate and efficient means of fire detection by comprehensively analyzing on-site images. Compared with traditional fire recognition methods based on physical sensors, flame recognition technology based on image processing offers a wider range and coverage, greater stability, and easier deployment.

[0003] Traditional flame detection methods based on machine vision mainly achieve flame recognition by capturing multiple features such as flame shape, color, and brightness. The specific methods are as follows:

[0004] 1. According to the characteristics of different color models, various flame image processing methods are used to effectively realize the recognition and detection of flame images based on RGB, HSV, YCbCr, YUV and other color models.

[0005] Second, based on the uniqueness of flame shape, it can also be used to assist in the recognition and detection of flame images. Flame shape features mainly include flame size, shape, edge, texture, contour and other features.

[0006] Third, the motion information of flame spread can be used for fire identification and detection. The motion characteristics of flame spread primarily include flame velocity, direction of movement, and morphological changes. These motion characteristics can be analyzed and extracted using digital image processing techniques, enabling the calculation and analysis of parameters such as the flame's trajectory, velocity, and acceleration. The most commonly used moving target detection methods include optical flow, frame difference, and background subtraction.

[0007] However, this early manual feature extraction had the following problems: difficulty in implementation, low recognition accuracy, poor model robustness, and it was difficult to meet practical application requirements.

[0008] Image processing methods based on deep learning are increasingly superior to traditional approaches. They offer two key advantages: first, manual flame feature extraction is no longer necessary; deep learning's automatic feature extraction significantly reduces the complexity of algorithm development; second, deep learning surpasses traditional algorithms in various visual performance metrics, including detection accuracy and recall. Classic deep convolutional neural networks such as AlexNet, VGG, GoogleNet, and ResNet have achieved significant results in many visual tasks. However, these large networks are computationally intensive and require a large number of parameters, making them difficult to deploy on edge devices. While lightweight networks such as MobileNet, ShuffleNet, and Xception have been proposed, their inherent local nature means they suffer from significant limitations in global feature extraction.

[0009] In short, establishing a lightweight, low-latency network model is urgent and crucial for the task of rapid flame recognition. Summary of the Invention

[0010] Currently, traditional flame recognition methods have low accuracy. Most deep neural network-based models cannot simultaneously extract global and local features of flame images to avoid false or missed detections. Furthermore, these models often have large parameter counts, making them difficult to deploy on mobile devices. To address these issues, this paper provides a method for constructing a lightweight flame recognition deep learning model that integrates global and local features.

[0011] In order to achieve the above object, the present invention adopts the following technical solutions:

[0012] A method for constructing a lightweight flame recognition deep learning model that integrates global and local features includes the following steps:

[0013] Step 1: Acquire and collect flame image data to construct a flame recognition dataset;

[0014] Step 2: Construct a lightweight Transformer module E-TB that can extract global features of flame images;

[0015] Step 3: Build a global and local feature extraction and fusion module CTB based on the lightweight Transformer module E-TB and convolutional neural network;

[0016] Step 4: By stacking the ShuffleNet V2 module and the CTB module in series, we achieve step-by-step feature extraction from shallow to deep, and finally build the network framework SConvTrans for global and local feature extraction of flame images;

[0017] Step 5: Divide the constructed flame recognition dataset into a training set and a test set, and use the data to train and test the constructed network framework SConvTrans to obtain a lightweight flame recognition deep learning model.

[0018] Furthermore, in step 2, a lightweight Transformer module E-TB (Efficient Transformer Block) capable of extracting global features of flame images is constructed; the specific method is as follows:

[0019] First, the input feature map Divide the patch block, where d is the number of feature map channels; H is the height of the feature map; W is the width of the feature map. After the Unfold operation, its dimension becomes , d is still the channel dimension, N is the number of patch blocks, that is, , P is the size of the patch after flattening, that is, ρ 2 ; Then use the efficient and lightweight attention mechanism to capture the global information of the data, and finally restore the feature dimension to .

[0020] Furthermore, the efficient and lightweight attention mechanism is specifically as follows: first, it passes through the efficient multi-head self-attention mechanism EMHSA (Efficient Multi-Head Self-Attention), and then reduces the dimensions of K and V through average pooling. On the one hand, it extracts the significant information in K and V, and on the other hand, it realizes the lightweight design of the self-attention mechanism; then it passes through the multilayer perceptron MLP (Multilayer Perceptron) to realize the extraction of global feature information.

[0021] Furthermore, the overall process of E-TB is expressed as:

[0022] (2)

[0023] (3)

[0024] Where: , , Represent the input features, EMHSA and MLP output features respectively.

[0025] Furthermore, the efficient multi-head self-attention mechanism EMHSA is expressed as:

[0026] (4)

[0027] (5)

[0028] (6)

[0029] Where: z represents the division of input features into multiple forms in the channel dimension; represents a low-computation self-attention operator; , , is a linear layer for context encoding; P s is a global pooling operation with a step size of s; Formula (4) is the standard multi-head self-attention calculation formula, where d k is the length of the token sequence;

[0030] Specifically, the input features are first linearly transformed to obtain a dimension of The three matrices Q, K, and V are respectively subjected to an average pooling operation with a step size of s to reduce the computational cost in the attention operation. After the average pooling operation, the dimensions of K and V are ; Finally, the multi-head self-attention mechanism MHSA (Multi-Head Self-Attention) is used to capture richer feature information.

[0031] Furthermore, the multilayer perceptron (MLP) is used to extract more basic and obvious features. Specifically, it is first normalized by a Batch Normalization (BN) layer, and then subjected to two 1×1 convolutions to perform dimensionality increase and then dimensionality reduction to better extract feature information. The activation function uses the SiLU activation function.

[0032] Furthermore, the CTB module consists of two branches: the upper branch is a Transformer branch for global representation processing, and the lower branch is a convolutional branch for local representation processing. Finally, the information extracted by the two branches is fused to achieve the interaction of global and local information.

[0033] The CTB module is represented as:

[0034] (1)

[0035] Where: is the input feature, is the shrinkage ratio for reducing the dimensionality of the input features in the channel dimension, is the output feature.

[0036] Furthermore, the CTB module is specifically:

[0037] First, the input dimension is Features Figure X, respectively through the upper branch 1×1 convolution and the lower branch A 1×1 convolution is used for dimensionality reduction. It is a shrinkage ratio greater than 0 and less than or equal to 1, thereby reducing the amount of calculation; at the same time, this approach is also to avoid the information loss on the branch caused by splitting directly on the channel dimension. Then, the upper branch Transformer operation is performed to obtain global representation information; the lower branch obtains local representation information through the SV2 convolution block operation. SV2 is the convolution block in ShuffleNet V2. Its module diagram is as follows Figure 3 As shown in the figure, the information of the two branches is concat- ed and then fused by 1×1 convolution, so that the global information and local information interact with each other. Finally, it is added to the original input feature X and the final feature Y is obtained by 1×1 convolution.

[0038] Furthermore, the network framework SConvTrans for global and local feature extraction of flame images;

[0039] First, deep feature extraction is achieved through 3×3 convolution, and redundant features are deleted through maximum downsampling MaxPool. The ShuffleNet v2 (SV2) module and the CTB module are repeatedly stacked three times. Then, channel adjustment is achieved through 1×1 convolution, and spatial feature dimensionality reduction is completed through global average pooling. Finally, the model completes classification through a fully connected layer to achieve rapid flame recognition.

[0040] Furthermore, the network framework SConvTrans for extracting global and local features of flame images is divided into five stages. In the first stage, the dimension is reduced to one-quarter of the original input image through convolution and pooling operations.

[0041] From the second to the fourth stages, the ShuffleNet V2 downsampling block and the CTB module are used to reduce the dimension of the original input image to one-eighth, one-sixteenth, and one-thirty-second, respectively, so that the overall pyramid structure is formed to obtain features at different scales, enabling the model to have better generalization and robustness. In the fifth stage, the fully connected layer FC is used to achieve the final output of the flame recognition category.

[0042] Compared with the prior art, the present invention has the following advantages:

[0043] First, self-made flame recognition dataset. This dataset combines three data sources:

[0044] 1) Public dataset (Fire Image Data Set for Dunnings 2018 study);

[0045] 2) A black powder combustion dataset collected on-site, which was augmented through random cropping, flipping, brightness changes, and other data enhancement methods;

[0046] 3) Network collection.

[0047] The self-made dataset for this experiment contains a total of 19,182 images, of which 9,887 are positive samples, including natural flames and black powder combustion flames; and 9,295 are negative samples, including non-fire natural images and fire-like images with high interference (such as sunset, sunrise, lamplight, stainless steel reflection, etc.).

[0048] Second, it achieves simultaneous extraction of global and local features of flame images. This patent independently designs a lightweight deep neural network model that fully integrates the local feature extraction capabilities of convolutional neural networks (CNNs) and the long-distance relationship modeling capabilities of Transformers.

[0049] Third, the lightweight deep neural network proposed in this patent can achieve superior performance metrics (such as accuracy and recall) while maintaining a relatively small model capacity. The model SConvTrans-S proposed in this patent has roughly the same number of parameters and computational complexity as MobileNetV2, yet achieves an accuracy of 93.5%, which is 2.4%, 0.4%, 1.7%, and 0.7% higher than VGG16, ResNet50, MobileNetV3, and ShuffleNetV2, respectively. Meanwhile, SConvTrans-L, with only 4.6M parameters and 0.7G computational complexity, achieves accuracy, precision, recall, and F1 of 0.945, 0.934, 0.962, and 0.948, respectively. These metrics all surpass those of classic lightweight models such as VGG16, ResNet50, MobileNetV3, and ShuffleNetV2. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 The structure of the lightweight flame recognition network (SConvTrans) that integrates convolution and Transformer;

[0051] Figure 2 The overall structure diagram of the CTB module;

[0052] Figure 3 Schematic diagram of the SV2 convolution module structure;

[0053] Figure 4 Schematic diagram of the lightweight Transformer block structure;

[0054] Figure 5 Schematic diagram of Unfold and Fold operations;

[0055] Figure 6 MLP layer structure diagram

[0056] Figure 7 Representative images of dataset samples;

[0057] Figure 8 Schematic diagram of accuracy changes during model iteration;

[0058] Figure 9 Schematic diagram of parameters at each stage of the model;

[0059] Figure 10 Schematic diagram showing the model confusion matrix;

[0060] Figure 11 Schematic diagram of the model's attention level when the shrinkage ratio r is different;

[0061] Figure 12 Schematic diagram of confusion matrix representation of different convolutional models;

[0062] Figure 13 Schematic diagram of the comparison of PR curves with different convolutional models;

[0063] Figure 14 Schematic diagram of confusion matrix representation of different Transformer models;

[0064] Figure 15 Schematic diagram comparing PR curves of different Transformer models;

[0065] Figure 16 Comparative diagram of the areas of interest in the final output of the model. DETAILED DESCRIPTION

[0066] This section aims to fully verify the effectiveness of the proposed model by presenting detailed experimental data and analyzing the results of ablation and comparative experiments. This in-depth exploration of the experimental results further illustrates the advantages of the proposed model in terms of performance and robustness, and provides strong support and reference for research and practical applications in related fields.

[0067] A method for constructing a lightweight flame recognition deep learning model that integrates global and local features includes the following steps:

[0068] Step 1: Acquire and collect flame image data to construct a flame recognition dataset;

[0069] Step 2: Construct a lightweight Transformer module E-TB (Efficient Transformer Block) that can extract the global features of the flame image; Figure 4As shown; first, the input feature map Divide the patch block, where d is the number of feature map channels; H is the height of the feature map; W is the width of the feature map. After the Unfold operation, its dimension becomes , d is still the channel dimension, N is the number of patch blocks, that is, , P is the size of the patch after flattening, that is, ρ 2 ; Then use the efficient and lightweight attention mechanism to capture the global information of the data, and finally restore the feature dimension to The Unfold and Fold operation process is as follows: Figure 5 shown.

[0070] Efficient and lightweight attention mechanism: The algorithm first uses the Efficient Multi-Head Self-Attention (EMMHSA) mechanism, then reduces the dimensions of K and V through average pooling. This not only extracts significant information from K and V, but also achieves a lightweight design of the self-attention mechanism. The algorithm then uses the Multilayer Perceptron (MLP) to extract global feature information.

[0071] The efficient multi-head self-attention mechanism EMHSA is expressed as:

[0072] (4)

[0073] (5)

[0074] (6)

[0075] Where: z represents the division of input features into multiple forms in the channel dimension; represents a low-computation self-attention operator; , , is a linear layer for context encoding; P s is a global pooling operation with a step size of s; Formula (4) is the standard multi-head self-attention calculation formula, where d k is the length of the token sequence;

[0076] Specifically, the input features are first linearly transformed to obtain a dimension of The three matrices Q, K, and V are respectively subjected to an average pooling operation with a step size of s to reduce the computational cost in the attention operation. After the average pooling operation, the dimensions of K and V are ; Finally, the multi-head self-attention mechanism MHSA (Multi-Head Self-Attention) is used to capture richer feature information.

[0077] Multilayer Perceptron (MLP) is used to extract more basic and obvious features. The specific process is as follows: Figure 6 As shown in the figure, the image is first normalized by the Batch Normalization (BN) layer, and then dimensionality is increased and then reduced by two 1×1 convolutions to better extract feature information. The SiLU activation function is used as the activation function.

[0078] The overall process of E-TB is expressed as:

[0079] (2)

[0080] (3)

[0081] Where: , , Represent the input features, EMHSA and MLP output features respectively.

[0082] Step 3: Build a global and local feature extraction and fusion module CTB based on the lightweight Transformer module E-TB and convolutional neural network;

[0083] The CTB module consists of two branches. The upper branch is the Transformer branch for global representation processing, and the lower branch is the convolution branch for local representation processing. Finally, the information extracted by the two branches is fused to achieve the interaction of global and local information. The overall structure of the CTB module is shown in the figure below. Figure 2 shown.

[0084] The CTB module is represented as:

[0085] (1)

[0086] Where: is the input feature, is the shrinkage ratio for reducing the dimensionality of the input features in the channel dimension, is the output feature.

[0087] The CTB module is specifically: first, the input dimension is Features Figure X , respectively through the upper branch 1×1 convolution and the lower branch A 1×1 convolution is used for dimensionality reduction. It is a shrinkage ratio greater than 0 and less than or equal to 1, thereby reducing the amount of calculation; at the same time, this approach is also to avoid the information loss on the branch caused by splitting directly on the channel dimension. Then, the upper branch Transformer operation is performed to obtain global representation information; the lower branch obtains local representation information through the SV2 convolution block operation. SV2 is the convolution block in ShuffleNet V2. Its module diagram is as follows Figure 3 As shown in the figure, the information of the two branches is concat- ed and then fused by 1×1 convolution, so that the global information and local information interact with each other. Finally, it is added to the original input feature X and the final feature Y is obtained by 1×1 convolution.

[0088] Step 4: By stacking the ShuffleNet V2 module and the CTB module in series, we achieve step-by-step feature extraction from shallow to deep, and finally build the network framework SConvTrans for global and local feature extraction of flame images;

[0089] First, deep feature extraction is achieved through 3×3 convolution, and redundant features are deleted through maximum downsampling MaxPool. The ShuffleNet v2 (SV2) module and the CTB module are repeatedly stacked three times. Then, channel adjustment is achieved through 1×1 convolution, and spatial feature dimensionality reduction is completed through global average pooling. Finally, the model completes classification through a fully connected layer to achieve rapid flame recognition.

[0090] The network framework SConvTrans for global and local feature extraction of flame images is divided into five stages. In the first stage, the dimension is reduced to one-quarter of the original input image through convolution and pooling operations.

[0091] From the second to the fourth stages, the ShuffleNet V2 downsampling block and the CTB module are used to reduce the dimension of the original input image to one-eighth, one-sixteenth, and one-thirty-second, respectively, so that the overall pyramid structure is formed to obtain features at different scales, enabling the model to have better generalization and robustness. In the fifth stage, the fully connected layer FC is used to achieve the final output of the flame recognition category.

[0092] Step 5: Divide the constructed flame recognition dataset into a training set and a test set, and use the data to train and test the constructed network framework SConvTrans to obtain a lightweight flame recognition deep learning model.

[0093] 1. The datasets used in this method come from three sources:

[0094] The first part is derived from a public dataset (Fire Image Data Set for Dunnings 2018 study);

[0095] The second part is a black powder combustion dataset collected on-site, which is augmented by random cropping, flipping, brightness change and other data enhancement methods.

[0096] The third part is network collection.

[0097] The self-made dataset for this experiment contains a total of 19,182 images. All images in the dataset are resized to 320×240 and are divided into two categories: fire (9,887 images) and nofire (9,295 images). Nofire mainly includes two categories of images: non-fire natural images and fire-like images with high interference (such as sunset, sunrise, lights, stainless steel reflections, etc.). Figure 7 The representative images in the dataset are also shown in the figure: (a) flame images collected by the network; (b) black powder flame images collected by ourselves; (c) natural images; (d) fire-like images. The specific number of samples is shown in Table 2.

[0098] Table 2 Dataset sample distribution

[0099]

[0100] During the experiment, the dataset was divided into training set and test set at an 8:2 ratio.

[0101] 2. Experimental Environment and Hyperparameter Settings

[0102] This experiment uses Windows 10 system, the deep learning framework is PyTorch framework, CPU model: Intel Core i7-10870, GPU model: NVIDIA GeForce RTX2060 (video memory 6GB).

[0103] During the model training process, the total training batch was set to 300 epochs, and the batch size was 4; the initial learning rate was set to 0.0001, and the learning rate was adjusted to 0.00001 after 240 epochs; the optimization algorithm used was the Adam optimizer; and the loss function used was the cross-entropy loss function.

[0104] 3. Model Training Results

[0105] The accuracy of the model changes with the number of iterations in 300 iterations as shown below: Figure 8As shown in the figure, the blue line represents the SConvTrans-S model, and the orange line represents the SConvTrans-L model. The curve gradually flattens out near 240 epochs. After 240 epochs, reducing the learning rate to one-tenth of the initial learning rate causes the loss function to converge, resulting in a small improvement in accuracy.

[0106] The number of parameters of SConvTrans-S and SConvTrans-L at different stages (i.e. different output space dimensions) is as follows Figure 9 As shown in the figure, the total number of parameters of the SConvTrans-S and SConvTrans-L models are 2.702M and 4.609M, respectively. In the first stage, when the output spatial dimension is 64×64, the two models have the least number of parameters. Since the number of output channels of the two models is equal, the number of parameters in the first stage is only 0.001M. In the third and fourth stages, when the output spatial dimensions are 16×16 and 8×8, respectively, the two models have the most parameters. In the fourth stage, the number of parameters of the SConvTrans-S and SConvTrans-L models is 1.509M and 2.676M, respectively.

[0107] In order to further verify the effectiveness of the model proposed in this paper, the performance of the two models SConvTrans-S and SConvTrans-L is demonstrated through the confusion matrix. Figure 10 As shown in the figure, the confusion matrix for the SConvTrans-S model is shown on the right, and the confusion matrix for the SConvTrans-L model is shown on the left. In the confusion matrix, the vertical axis represents the true label, and the horizontal axis represents the predicted label. Among 1977 flame images, SConvTrans-L effectively identified flames in 1901 of them, and SConvTrans-S also recognized 1886 of them, achieving good results. Among 1859 non-flame images, SConvTrans-L and SConvTrans-S correctly identified 135 and 158 non-flame images as flames, respectively. Because the test set contains over 400 highly distracting flame-like images, such as images of sunsets and sunrises, light from electric bulbs, and images of tiny flames and sparks, these images significantly interfere with the model's effective recognition. However, the model proposed in this paper can still effectively identify most fire-like images. Among them, the SConvTrans-L model has more channels and wider dimensions in the third to fifth stages, and can extract more feature information to identify fire-like images. Its performance is also better than that of SConvTrans-S.

[0108] 4. Validation of the Shrinkage Ratio r

[0109] The lightweight pyramid hybrid network architecture model (SconvTrans) proposed in this paper effectively combines the strengths of CNN and Transformer. The CTB block effectively fuses local information extracted by CNN with global information extracted by Transformer. To further reduce model parameters and computational complexity within the CTB block, thereby promoting model lightweighting, and to demonstrate the superior feature extraction performance of the combined CNN and Transformer architectures, a shrinkage ratio r was designed to control the feature dimensions of the input SV2 convolutional block and the Transformer block. Comparisons with different values ​​of r validated the effectiveness of the shrinkage ratio r in the SconvTrans model. The specific comparison results are shown in Table 3.

[0110] Table 3 Effect of SConvTrans-L under different values ​​of contraction ratio r

[0111]

[0112] Table 3 shows that the highest precision is achieved when r = 0.5 and r = 0.75, at 94.5% and 94.6%, respectively. The F1 index also reaches 94.8%. At r = 0.5, the recall rate reaches 96.2%, effectively identifying nearly all flame images. While the model evaluation metrics are roughly equivalent when r = 0.5 and r = 0.75 are roughly equivalent, the number of model parameters and computational complexity is significantly reduced when r = 0.5, which aligns with the lightweight design principle of this paper. The model ultimately settled on 0.5 as the standard value for r. At r = 0.25, the model also interacts with global and local information, achieving 94% precision. When r = 0 and r = 1, the CTB in the SconvTrans model only uses the SV2 convolutional block and the Transformer to extract local or global feature information, respectively, failing to effectively combine the two types of information. Its accuracy, precision, recall, and F1 harmonic mean are relatively low. Furthermore, when only the Transformer is present, feature information is primarily extracted using the fully connected layer, resulting in an excessive number of parameters and computations. Table 3 fully demonstrates the effectiveness of this method's feature extraction approach by combining CNN and Transformer.

[0113] This paper selects an appropriate r value to control the dimension of the feature channel of the input CNN and Transformer, which not only greatly reduces the number of model parameters and computational complexity, but also makes the model have better performance. Figure 11This section shows a visualization of how the model focuses on flame features when r is equal to different values. The first three images are examples of ordinary flames, and the last two images are examples of flames produced when gunpowder is ignited.

[0114] from Figure 11 It can be seen that when r is equal to 0.25, 0.5, and 0.75, the effect is better. This is mainly due to the interactive fusion of local and global information by the model, which enables the model to extract richer feature information for flame recognition. When r is equal to 0 or 1, the model recognition effect is poor, especially for small flame feature images.

[0115] V. Algorithm Comparison Experiment

[0116] In order to further verify the effectiveness of the model proposed in this paper, we first compare the model in this paper with different CNN-related models such as VGG, ResNet, MobileNet and ShuffleNet in terms of six aspects: parameter quantity, computational complexity, accuracy, precision, recall rate and F1 score. The comparison results are shown in Table 4. The confusion matrix of different convolution-related models is shown in Figure 12 shown.

[0117] Table 4 Comparison with different CNN related models

[0118]

[0119] As can be seen from Table 4, the proposed method SConvTrans-S has roughly the same number of parameters and computational complexity as MobileNetV2, yet its accuracy can reach 93.5%, which is 2.4%, 0.4%, 1.7%, and 0.7% higher than VGG16, ResNet50, MobileNetV3, and ShuffleNetV2, respectively. Meanwhile, SConvTrans-L has an accuracy of 94.5% with only 4.6M parameters and 0.7G computational complexity, achieving the highest values ​​in precision, recall, and F1 score compared to the other models in Table 4. Figure 12 From the confusion matrices of the models, we can see that compared with other convolution-related models, the proposed model has the highest TP value, that is, the proposed model correctly identifies the largest number of images with the true label of flame. This is mainly due to the effective combination of the local detail information features and the global overall information features of the flame image by the proposed model, which enables the model to effectively identify flames.

[0120] This paper effectively improves the model's ability to recognize flames by effectively combining global information with local information. Figure 13 The PR curve shows the comparison between the model in this paper and other convolution-related models. Figure 13 It can be seen that the two curves of the proposed model SconvTrans-L and SconvTrans-S are closer to the upper right corner, that is, when the recall rate increases, the precision rate of the proposed model can maintain the highest value compared with other convolution-related models, thus outperforming other models in the effect of identifying flames.

[0121] Table 5 Comparison with different Transformer related models

[0122]

[0123] Secondly, Table 5 shows the comparison between the proposed model and different Transformer-related models. The experiment selected MobileVitV1, MobileVitV2, Next-ViT, PVT and other related models for comparison. The proposed model SConvTrans-L achieved the highest values ​​in accuracy, recall rate and F1 score, and was close to the highest value in precision. The performance of different Transformer-related models was demonstrated through the confusion matrix. The relevant confusion matrix is ​​shown as follows: Figure 14 shown.

[0124] Depend on Figure 14 As can be seen, while the TN value of our model is average among the comparison models, its TP value is the highest. This is primarily due to our model's superior performance in identifying tiny Mars objects compared to other models. While the Transformer captures global representational information from features, our model not only captures global information but also adds a convolutional branch that captures local information. Finally, the information from these two branches is combined, enabling accurate recognition of even tiny Mars images.

[0125] exist Figure 15 The PR curve of this model and other Transformer related models is shown in Figure 15 It can be seen that the solid line represented by the model SconvTrans-L in this paper has better performance. When the recall rate gradually approaches 1, its precision rate is higher than that of other models.

[0126] 6. Algorithm Visualization

[0127] This paper achieves excellent results by improving ShuffleNet V2. Figure 16In this paper, we demonstrate the superiority of our method through several visual examples. The speckled area in the figure represents the area that the model focuses on in its final output, with darker colors indicating higher levels of focus. The image on the left shows an example of a typical flame, demonstrating that our method pays more attention to the flame region than ShuffleNet V2. The image on the right shows an example of a flame produced by the ignition of gunpowder. Since the sparks produced by gunpowder ignition are tiny and difficult to detect, this paper adds an attention module and a multi-scale module to ShuffleNet V2, enabling the model to further enhance its feature extraction of subtle flame regions, thereby detecting these tiny sparks.

[0128] The above analysis fully demonstrates the superiority of our approach. Compared with convolutional neural network models, our model can extract more global feature information, resulting in more accurate recognition results. Compared with Transformer-related models, our model can extract local features and retain more detailed information. By combining CNN and Transformer to extract feature information, global and local information interact, the model's flame recognition capabilities can be further improved.

[0129] Any matters not described in detail in this specification are prior art known to those skilled in the art. Although the above description of the present invention is based on specific embodiments to facilitate understanding of the present invention by those skilled in the art, it should be understood that the present invention is not limited to the scope of the specific embodiments. As long as various modifications are within the spirit and scope of the present invention as defined and determined by the appended claims, such modifications will be obvious to those skilled in the art, and all inventions and creations utilizing the concepts of the present invention are protected.

Claims

1. A method for constructing a lightweight flame recognition deep learning model that integrates global and local features, characterized by: The following steps are involved: Step 1: Acquire and collect flame image data to construct a flame recognition dataset; Step 2: Construct a lightweight Transformer module E-TB that can extract global features of flame images; Step 3: Build a global and local feature extraction and fusion module CTB based on the lightweight Transformer module E-TB and convolutional neural network; Step 4: ShuffleNet V2 modules and CTB modules are stacked in series to achieve step-by-step feature extraction from shallow to deep, and finally the network framework SConvTrans for global and local feature extraction of flame images is constructed; Step 5: Divide the constructed flame recognition dataset into a training set and a test set, and train and test the constructed network framework SConvTrans to obtain a lightweight flame recognition deep learning model; The CTB module consists of two branches: the upper branch is a Transformer branch for global representation processing, and the lower branch is a convolution branch for local representation processing. Finally, the information extracted by the two branches is fused to achieve the interaction of global and local information. The CTB module is represented as: ； Where: is the input feature, is the shrinkage ratio for reducing the dimensionality of the input features in the channel dimension, is the output feature; The CTB module specifically: first, the input dimension is The feature map X of the upper branch is respectively 1×1 convolution and the lower branch A 1×1 convolution is used for dimensionality reduction. It is a shrinkage ratio greater than 0 and less than or equal to 1, so as to reduce the amount of calculation. At the same time, this approach is also to avoid the information loss on the branch caused by splitting directly in the channel dimension. Then, the upper branch Transformer operation is performed to obtain global representation information; the lower branch obtains local representation information through the SV2 convolution block operation, SV2 is the convolution block in ShuffleNet V2; then the information of the two branches is concat spliced, and then 1×1 convolution is performed for information fusion, so that the global information and local information interact; finally, it is added to the original input feature X, and the information is fused through 1×1 convolution to obtain the final feature Y; The network framework SConvTrans for global and local feature extraction of flame images first implements deep feature extraction through 3*3 convolution, removes redundant features through maximum downsampling MaxPool, repeatedly stacks the ShuffleNet v2 module and the CTB module three times, then implements channel adjustment through 1*1 convolution, and completes spatial feature dimensionality reduction through global average pooling. Finally, the model completes classification through a fully connected layer to achieve rapid flame recognition.

2. The method for constructing a lightweight flame recognition deep learning model that integrates global and local features according to claim 1, characterized in that: The specific method of step 2, constructing a lightweight Transformer module E-TB capable of extracting global features of flame images, is as follows: First, the input feature map Divide the patch block, where d is the number of feature map channels; H is the height of the feature map; W is the width of the feature map. After the Unfold operation, its dimension becomes , d is still the channel dimension, N is the number of patch blocks, that is, , P is the size of the patch after flattening, that is, ρ 2 ; Then use the efficient and lightweight attention mechanism to capture the global information of the data, and finally restore the feature dimension to .

3. The method for constructing a lightweight flame recognition deep learning model that integrates global and local features according to claim 2, characterized in that: The efficient and lightweight attention mechanism is specifically: first passing through the efficient multi-head self-attention mechanism EMHSA, then reducing the dimensions of K and V through average pooling, and then passing through the multi-layer perceptron MLP to realize the extraction of global feature information.

4. The method for constructing a lightweight flame recognition deep learning model that integrates global and local features according to claim 3, characterized in that: The overall process of E-TB is expressed as: ； ； Where: , , Represent the input features, EMHSA and MLP output features respectively.

5. The method for constructing a lightweight flame recognition deep learning model integrating global and local features according to claim 4 is characterized in that: The efficient multi-head self-attention mechanism EMHSA is expressed as: ； ； ； Where: z represents the division of input features into multiple forms in the channel dimension; represents a low-computation self-attention operator; , , is a linear layer for context encoding; P s is a global pooling operation with a step size of s; EMHSA(z) is the standard multi-head self-attention calculation formula, where d k is the length of the token sequence; Specifically, the input features are first linearly transformed to obtain a dimension of The three matrices Q, K, and V are respectively subjected to an average pooling operation with a step size of s to reduce the computational cost in the attention operation. After the average pooling operation, the dimensions of K and V are ; Finally, the multi-head self-attention mechanism MHSA is used to capture richer feature information.

6. The method for constructing a lightweight flame recognition deep learning model integrating global and local features according to claim 5, characterized in that: The multi-layer perceptron (MLP) is used to extract more basic and obvious features. Specifically, it is first normalized through a BN layer, and then subjected to two 1×1 convolutions to perform dimensionality increase and then dimensionality reduction to better extract feature information. The activation function uses the SiLU activation function.

7. The method for constructing a lightweight flame recognition deep learning model integrating global and local features according to claim 1, characterized in that: The network framework SConvTrans for extracting global and local features of flame images is divided into five stages. In the first stage, the dimension is reduced to one-quarter of the original input image through convolution and pooling operations. In the second to fourth stages, the dimension is reduced to one-eighth, one-sixteenth, and one-thirty-second of the original input image through the ShuffleNet V2 downsampling block and the CTB module, respectively, so that the overall pyramid structure is formed to obtain features of different scales. In the fifth stage, the fully connected layer FC is used to output the final flame recognition category.

Images