A smoke-resistant forest fire detection method based on thermal infrared guided asymmetric fusion

Abstract

A smoke-resistant forest fire detection method based on thermal infrared-guided asymmetric fusion is proposed. This invention relates to the field of computer vision and multimodal image processing technology, addressing the problem of smoke interference and loss of details of small fire sources in visible light images of fire scenes. Multimodal fusion is widely used in fire monitoring; however, existing symmetric fusion methods are susceptible to smoke noise contamination, and downsampling leads to the loss of spatial information of fire sources. To improve this, this invention proposes a detection method. By constructing a dual-stream feature network, a smoke suppression-guided aggregation mechanism is introduced at a deep layer for asymmetric fusion to filter out noise; a detail aggregation module is used to aggregate features across layers to repair edges; and a topological consistency focus Tversky combination loss optimization model is constructed. This method can effectively suppress smoke and accurately reconstruct small fire sources. This invention is applied to smoke-resistant forest fire detection in complex environments.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and multimodal image processing technology, specifically relating to a method for detecting smoke and forest fires based on thermal infrared guided asymmetric fusion. Background Technology

[0002] Forest fires are extremely destructive, and accurate early detection is crucial for disaster prevention and mitigation. In recent years, computer vision technology has been widely used in fire monitoring. However, the environment at fire scenes is extremely complex, often accompanied by dense smoke. Existing multimodal methods typically employ symmetrical fusion strategies, ignoring the physical imaging differences at fire scenes. Visible light (RGB) has a short wavelength and cannot bypass smoke particles, resulting in strong scattering effects and filling RGB images with false texture noise. In deep feature extraction, this physical noise is amplified. If blind fusion is performed directly, the contaminated RGB features will disrupt the discrimination boundaries of thermal infrared (TIR) ​​features, causing the model to misidentify high-temperature smoke as a fire source.

[0003] Furthermore, while classic methods such as feature pyramids or dilated convolutions attempt to mitigate the spatial information loss caused by downsampling, they often fail in fire scenarios. Simple cross-level skip connections blindly introduce shallow, high-frequency smoke noise, contaminating deep semantic features. Simultaneously, the token oversmoothing effect unique to the Transformer architecture causes tiny fire sources, occupying very few pixels, to be annihilated by the mean squared effect of large-area background features during self-attention computation. This dual dilemma of noise interference and feature diffusion makes existing general decoding strategies unable to effectively remove environmental interference while reconstructing the details of tiny fire sources. Additionally, due to smoke occlusion leading to sparse effective supervision signals, fire pixels constitute a very small percentage of the image. Commonly used cross-entropy loss functions are easily overwhelmed by the dominant background pixel gradients during training, causing the model to get trapped in a local optimum predicting the entire background. Summary of the Invention

[0004] The purpose of this invention is to address the problems of how to suppress dense smoke noise interference in visible light images using thermal infrared information, and how to reconstruct the spatial details of small fire sources during deep network downsampling. A smoke-resistant forest fire detection method based on thermal infrared-guided asymmetric fusion is proposed. The above-mentioned objective is mainly achieved through the following technical solution: This invention provides a smoke-resistant forest fire detection method based on thermal infrared-guided asymmetric fusion, comprising the following steps:

[0005] S1. Multimodal Data Construction and Preprocessing: Acquire paired visible light and thermal infrared images of a fire scene, normalize and geometrically enhance the images, and construct a multimodal training set. The steps are as follows:

[0006] (1) Obtain paired data of visible light and thermal infrared images in a fire scene. To eliminate the differences in imaging dimensions between different sensors, a multimodal training set is constructed and the images are standardized, as shown in formula (1):

[0007] (1)

[0008] in, This represents the image data obtained after standardization processing. This represents the original image data input. This represents the statistical mean of all pixel values ​​in the dataset. This represents the statistical standard deviation of all pixel values ​​in the dataset. While performing standardization, online data augmentation is also performed to increase the diversity of the sample.

[0009] (2) During model training, online data augmentation is performed simultaneously on the input visible light image, thermal infrared image and corresponding label image; the augmentation methods include random horizontal flipping, random angle rotation and random scale scaling; if the thermal infrared image is a single-channel grayscale image, it needs to be mapped to three-channel data through channel copying in the preprocessing stage to match the network input dimension.

[0010] S2. Dual-stream hybrid feature encoding: A dual-stream feature extraction network based on the Transformer architecture is constructed to extract features from the visible light mode and the thermal infrared mode respectively. A hierarchical hybrid fusion strategy is adopted: conventional fusion is performed in the shallow stage of the network to preserve texture details, and a smoke suppression-guided aggregation mechanism is introduced in the deep stage to perform asymmetric anti-interference fusion. The steps are as follows:

[0011] (1) The dual-stream feature extraction network uses two Mix Transformers with shared structure but not shared weights as the backbone network. Each backbone network contains four feature extraction stages: the first, the second, the third and the fourth. The resolutions of the output feature maps are 1 / 4, 1 / 8, 1 / 16 and 1 / 32 of the original image, respectively.

[0012] (2) The hierarchical hybrid fusion strategy is divided into two parts: shallow fusion and deep fusion based on the above four stages. The first stage and the second stage constitute the shallow stage of the network, in which the feature fusion module is used for modal fusion. The third stage and the fourth stage constitute the deep stage of the network, in which the smoke suppression-guided aggregation module is used for modal fusion.

[0013] (3) For deep semantic features, the smoke-penetrating characteristics of the thermal infrared mode are used as a guiding signal to denoise and filter the visible light mode; specifically, the Re-Softmax strategy is introduced, as shown in formula (2):

[0014] (2)

[0015] in, This represents the feature vector output after computation through the attention mechanism. This represents the query vector obtained from thermal infrared feature mapping. This represents the key vector obtained from visible light feature mapping. The matrix representing the transpose of the key vector. This represents the feature dimension of the key vector. This is a scaling factor used to scale the dot product result to prevent gradient vanishing. Represents the normalized exponential function, This indicates that the constructed smoke suppression mask applies a negative infinite penalty to non-fire source areas. This represents the value vector obtained by visible light feature mapping. Through the above mechanism, asymmetric anti-interference fusion with the elimination of false data is achieved.

[0016] (4) For the feature fusion module in the shallow stage, the extracted shallow visible light feature map and thermal infrared feature map are directly concatenated and spliced ​​in the channel dimension to initially aggregate the spatial information of the two modes and obtain the preliminary mixed features. The splicing process is shown in formula (3):

[0017] (3)

[0018] in, This represents the hybrid feature map obtained after concatenation along the channel dimension. This indicates a concatenated stitching operation performed along the channel dimension of the feature map. This represents the visible light feature map extracted during the shallow stages of the network. This represents the thermal infrared feature map extracted in the shallow layer stage of the network;

[0019] (5) Global description information at the channel level is extracted through global average pooling and multilayer perceptron network. Finally, the channel attention weight vector is calculated by nonlinear mapping using the Sigmoid activation function. The weight calculation process is shown in formula (4):

[0020] (4)

[0021] in, This represents the calculated channel attention weight vector, used to measure the contribution of different feature channels to subsequent tasks. This represents the Sigmoid activation function, used to normalize the generated weight values ​​to a range of zero to one. This represents a multilayer perceptron network used to learn non-linear dependencies between feature channels. This represents the global average pooling operation, used to compress the spatial features of each channel into a global scalar. This represents a convolution operation performed on the feature map to achieve dimensionality reduction and recombination of the features. This represents the hybrid feature map obtained from the initial stitching in the above steps;

[0022] (6) The channel attention weight vectors obtained by calculation are used to perform channel-wise weighted modulation on the recombined hybrid features. Combined with the residual connection mechanism, the weighted modulated features are added element-wise to the unweighted recombined features to output the final shallow fusion features. The output process is shown in formula (5):

[0023] (5)

[0024] in, This represents the shallow fused feature map output by the feature fusion module. This represents the convolution operation performed on the feature map. This represents the hybrid feature map obtained from the initial splicing. This represents a channel-wise multiplication operation, which is an element-wise multiplication operation that applies channel attention weights to the corresponding feature channels. The first term on the right-hand side of the equation represents the channel attention weight vector calculated in the above steps. The second term represents the feature branch enhanced by the attention mechanism. This constitutes a residual direct connection branch that retains the original feature information.

[0025] S3. Multi-scale detail aggregation decoding: Using the detail aggregation module, the deep semantic features output by the dual-stream feature extraction network are aggregated across layers with the shallow high-resolution features to repair the loss of edge information during downsampling and generate a high-precision fire prediction mask. The steps are as follows:

[0026] (1) Extract the high-resolution feature map output from the first stage of the encoder as spatial detail guiding features. ;

[0027] (2) Extract the deep semantic feature map from the output of the fourth stage of the encoder. Its spatial scale is adjusted by bilinear interpolation upsampling, as shown in formula (6):

[0028] (6)

[0029] in, This represents the aggregated feature map obtained after cross-level concatenation operations. This indicates a cascading splicing operation performed at the channel level. This represents the shallow, high-resolution feature map output by the encoder. This indicates a bilinear interpolation upsampling operation. The deep semantic feature map output by the encoder is concatenated, and then the aggregated features are smoothed and the edges are sharpened through a convolutional layer to repair the edge information lost during the downsampling process and generate a high-precision fire prediction mask.

[0030] (3) The feature maps obtained in steps (1) and (2) are concatenated along the channel dimension and fed into consecutive convolutional layers for feature smoothing and edge sharpening. Finally, they are mapped to the class space through the MLP decoding head to generate a prediction mask. As shown in formula (7):

[0031] (7)

[0032] in, This represents a segmentation mask with the same resolution as the original image. This refers to the multilayer perceptron decoder head. This represents a 3x3 convolution operation. This indicates a channel splicing operation. This indicates high-resolution spatial detail guiding features. This represents the deep feature map after upsampling.

[0033] S4. Combined Loss Optimization Training: Construct a topologically consistent focus Tversky combined loss function. Address the issues of background dominant gradient and missed detection of small fire sources through dynamic weighting and boundary constraints. Optimize network parameters using the backpropagation algorithm. The steps are as follows:

[0034] (1) Construct a topologically consistent focus Tversky combined loss function, and solve the problem of missed detection of background dominant gradient and small fire source through dynamic weighting and boundary constraints, as shown in Equation (8):

[0035] (8)

[0036] in, This represents the total value of the combined loss function. This represents the value of the focus loss function. This represents the value of the Tversky loss function. This represents the value of the boundary loss function. This represents the balance coefficient used to adjust the boundary loss weights;

[0037] (2) By introducing a focusing factor The weight of simple background samples is dynamically reduced, forcing the model to focus on mining difficult-to-classify samples such as smoke edges, as shown in formula (9):

[0038] (9)

[0039] in, Indicates pixel index, To balance the weight hyperparameters of positive and negative samples, The model represents the first The predicted probability of the true class of each pixel. As a focusing factor, Represents the natural logarithm;

[0040] (3) The For the Tversky loss, the recall ability of the model for sparse fire targets is asymmetrically enhanced by adjusting the false negative penalty weights, as shown in Equation (10):

[0041] (10)

[0042] in, and These represent the model's prediction of the first... The probability of each pixel representing the fire source target and the background. and These represent the true label values ​​for the pixel as the fire source and the background, respectively. and These represent the weighting parameters that control the severity of penalties for false positives and false negatives, respectively. The boundary loss is used to calculate the distance difference between the predicted mask and the real label in terms of geometric topology, explicitly constraining the edge consistency of tiny fire sources, and achieving fine-grained segmentation and reconstruction.

[0043] S5. Test-Time Enhanced Inference: During the model inference phase, a test-time enhancement strategy is introduced to fuse the prediction results of the original image and the flipped image, outputting the final fire segmentation map. The steps are as follows:

[0044] (1) The acquired original input image is directly fed into the trained fire semantic segmentation network model for a single positive prediction to obtain the prediction probability map of the original spatial orientation. The positive prediction process is shown in formula (11):

[0045] (11)

[0046] in, This represents the predicted probability map of the original direction output by the model. This represents a fire detection model that has been trained, optimized, and has fixed parameters. The test image represents the original input;

[0047] (2) In order to obtain feature representations with spatial transformation invariance, a geometric transformation operation of horizontal flipping is performed on the same input image. The flipped image is then fed into the fire detection model for prediction to obtain a flipped probability map. Furthermore, the flipped probability map is subjected to a reverse horizontal flipping operation to ensure that it is strictly aligned back to the original direction in the spatial coordinate system. The above feature extraction and spatial alignment process is shown in formula (12):

[0048] (12)

[0049] in, This represents the probability map obtained after horizontal flipping prediction and then flipping it back to the original direction. This represents the function that is the inverse of the horizontal flip operation. This represents a fire detection model that has been trained and optimized. This function represents the horizontal flipping operation performed on the image. The test image represents the original input;

[0050] S53. Perform pixel-by-pixel fusion calculation on the predicted probability maps obtained from the two different inference paths. Specifically, take the arithmetic mean of the predicted probability map in the original direction and the probability map flipped back to the original direction, and use it as the final prediction result output by the method of this invention. The fusion calculation process is shown in formula (13):

[0051] (13)

[0052] in, This represents the high-precision fire segmentation probability map that is ultimately output by this method. This represents the weighting coefficient used to calculate the arithmetic mean of two corresponding probability graphs. This represents the predicted probability map of the original directions obtained in the above steps. This represents the probability diagram obtained after reversing and flipping back to the original direction in the above steps.

[0053] Invention Effects

[0054] This invention employs an asymmetric processing approach. In the shallow stage, conventional fusion is retained to utilize texture; in the deep stage, the smoke-penetrating properties of thermal infrared images are used to generate a high-temperature mask, which acts as a filter to forcibly suppress smoke noise regions in the visible light branch that have a strong response but lack thermal radiation verification, retaining only the color information of the true fire source, thus achieving anti-interference fusion with accurate data retention. Simultaneously, this invention constructs a high-resolution information path before decoding, directly aggregating the high-resolution spatial features from the encoder's first stage with deep semantic features across layers, significantly enhancing the perception of minute fire sources. Furthermore, the topological consistency focus Tversky loss strategy proposed in this invention dynamically reduces the weight of simple backgrounds to mine hard samples, asymmetrically penalizes missed detections, and introduces explicit boundary constraints to optimize the topological consistency of the prediction results, thereby achieving refined reconstruction of the edges of minute fire sources. Attached Figure Description

[0055] Figure 1 This is a schematic diagram of the overall process of the method of the present invention;

[0056] Figure 2 This is a schematic diagram of the structure of the dual-stream feature extraction network and the hierarchical hybrid fusion strategy in this invention;

[0057] Figure 3 This is a schematic diagram of the smoke suppression guided polymerization module SSAM in this invention;

[0058] Figure 4 This is a comparison chart showing the effectiveness of the present invention with other detection methods. Detailed Implementation Specific implementation method one:

[0060] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0061] like Figure 1 The proposed method for smoke and forest fire detection based on thermal infrared-guided asymmetric fusion, as shown in this paper, includes model building and training as well as model inference.

[0062] The model construction and training process includes the following steps:

[0063] S1. Multimodal data construction and preprocessing: Acquire paired data of visible light images and thermal infrared images in fire scenarios. In order to eliminate the differences in imaging dimensions of different sensors, the paired data is standardized and normalized. At the same time, online geometric data augmentation operations including random horizontal flipping, random angle rotation and random scale scaling are performed. The single-channel thermal infrared image is copied and expanded into three-channel data to strictly match the input dimension requirements of the subsequent network. Finally, a multimodal training set is constructed.

[0064] S2. Dual-stream hybrid feature encoding: A dual-stream feature extraction network based on the Transformer architecture is constructed. Two backbone networks with the same structure but independent weights are used to extract multi-level features of the visible light mode and the thermal infrared mode, respectively. A hierarchical hybrid fusion strategy is adopted for modal interaction. In the shallow stage of the dual-stream feature extraction network, a feature fusion module based on channel attention is used to splice and weightedly fuse the features to preserve the environmental context and texture details of the underlying layer. In the deep stage of the dual-stream feature extraction network, a smoke suppression guided aggregation mechanism is introduced. The thermal infrared features are used as a net heat source guiding signal to generate a smoke suppression mask, which forcibly filters out smoke noise regions in the visible light branch that have a strong response but lack thermal radiation verification. This achieves asymmetric anti-interference fusion with true and false features and outputs deep semantic features.

[0065] S3. Multi-scale detail aggregation decoding: The detail aggregation module is used to aggregate the shallow high-resolution features output from the first stage of the dual-stream feature extraction network with the deep semantic features output from the fourth stage across layers. First, the deep semantic features are interpolated and upsampled to align the spatial scale. Then, they are concatenated with the shallow features that retain rich spatial geometric information in the channel dimension. Subsequently, feature smoothing and edge sharpening are performed through continuous convolution operations to repair the loss of small fire source edge information during the downsampling process of the deep network. Finally, it is mapped to the category space to generate a high-precision fire prediction mask.

[0066] S4. Combined Loss Optimization Training: Construct a topology-consistent focus Tversky combined loss function, in which the focus loss component is used to dynamically reduce the dominant gradient weight of a large number of simple background samples, the Tversky loss component is used to asymmetrically enhance the model's recall penalty for sparse false negative fire targets, and the boundary loss component is used to predict the edge consistency between the mask and the real label in the geometric topology based on the distance transformation constraint. Based on this joint constraint, the backpropagation algorithm is used to calculate the loss gradient and iteratively optimize the network parameters.

[0067] S5. Test-time Enhanced Inference: A test-time enhancement strategy is introduced during the model inference stage. Forward inference prediction is performed on the original input test image and the test image after horizontal flipping geometric transformation. The obtained flipped prediction probability map is then aligned back to the original direction in reverse space. Finally, the arithmetic mean of the two probability maps is taken as the final fire segmentation probability map output to eliminate random errors generated by single prediction.

[0068] The embodiments of the present invention will be described in detail below:

[0069] The specific implementation of this invention is as follows.

[0070] S1, such as Figure 1 As shown, the multimodal data construction and preprocessing step aims to acquire paired visible light and thermal infrared images of a fire scene, normalize and geometrically enhance the images, thereby constructing a high-quality multimodal training set. The steps are as follows:

[0071] (1) Obtain paired data of visible light images and thermal infrared images in a fire scene. Let the obtained original multimodal paired dataset be set. ,in This represents the total number of image sample pairs in the dataset. Indicates the first The index number of each sample pair Indicates the first One visible light image sample, Indicates the corresponding first One thermal infrared image sample, This represents the pixel-level manually labeled ground truth value corresponding to the image sample pair;

[0072] To eliminate the dimensional differences between visible light sensors and thermal infrared sensors in imaging mechanisms and pixel value distribution, and to accelerate the convergence speed of deep learning network training, a multimodal training set is constructed and Z-Score normalization is applied to all input images. The mathematical calculation logic is shown in formula (1):

[0073] (1)

[0074] in, This represents the image data obtained after standardization processing. This represents the original image data input. This represents the statistical mean of all pixel values ​​in the corresponding modality dataset. This represents the statistical standard deviation of all pixel values ​​in the corresponding modality dataset. While performing the above standardization process, online data augmentation operations are executed to increase the diversity of the samples.

[0075] (2) In order to improve the generalization ability and anti-overfitting ability of the fire segmentation model in complex and variable forest fire scenarios during the model training process, online spatial alignment data augmentation operation is performed simultaneously on the input visible light image, thermal infrared image and corresponding label ground truth map; the online data augmentation means specifically include: random horizontal flip triggered by a preset probability, random angle rotation within a specified degree range, and random scale scaling;

[0076] In particular, since the thermal infrared images output by thermal infrared imaging devices are usually single-channel grayscale images, while the initial convolutional layers of deep feature extraction networks are usually preset to input three-channel data, in order to strictly match the input dimension requirements of subsequent dual-stream feature extraction networks, it is necessary to map them to three-channel data in the preprocessing stage by copying and expanding the channel dimension. This dimension mapping process is shown in formula (2):

[0077] (2)

[0078] in Represents the space of the real number set. and These represent the spatial height and width of the input image, respectively. This is a three-channel thermal infrared image obtained after channel replication processing.

[0079] S2, such as Figure 1 and Figure 2 As shown, the specific implementation process is dual-stream hybrid feature encoding, that is, constructing a dual-stream feature extraction network based on the Transformer architecture to extract features of the visible light mode and the thermal infrared mode respectively, and adopting a hierarchical hybrid fusion strategy. In the shallow stage of the network, conventional fusion is performed to preserve texture details, and in the deep stage, a smoke suppression-guided aggregation mechanism is introduced to perform asymmetric anti-interference fusion. The steps are as follows:

[0080] (1) Combination Figure 2 As shown, the dual-stream feature extraction network uses two MixTransformer models with shared network structures but not shared network weights as the backbone network, which are used to independently process visible light and thermal infrared inputs. Each backbone network contains four cascaded feature extraction stages, from the first stage to the fourth stage. As the network layers deepen, the spatial resolution of the output feature map gradually decreases, to 1 / 4, 1 / 8, 1 / 16 and 1 / 32 of the resolution of the original input image, respectively, thus forming a multi-scale feature pyramid representation.

[0081] (2) The hierarchical hybrid fusion strategy is based on the above four feature extraction stages and is clearly divided into two parts: shallow texture fusion and deep semantic denoising. The first and second stages of the backbone network constitute the shallow stage of the network, in which the following approach is adopted: Figure 2 The feature fusion module identified in the middle performs modal fusion to preserve the underlying environmental context information such as edges and textures of the fire scene to the greatest extent. The third and fourth stages constitute the deep stage of the network. In this stage, because the physical scattering noise of smoke in the deep features is severely amplified, a method is adopted. Figure 3 The smoke suppression-guided aggregation module shown performs asymmetric deep modal fusion.

[0082] (3) For the feature fusion module in the shallow stage, the extracted shallow visible light feature map and thermal infrared feature map are directly concatenated and spliced ​​in the channel dimension to initially aggregate the spatial information of the two modes and obtain the preliminary mixed features. The splicing operation process is shown in formula (3):

[0083] (3)

[0084] in, This represents the hybrid feature map obtained after concatenation along the channel dimension. This indicates a cascading concatenation operation performed along the channel dimension of the feature map. This represents the visible light feature map extracted in the shallow stage of the network. This represents the thermal infrared feature map extracted in the shallow layer stage of the network;

[0085] (4) The preliminary mixed features are input into a convolutional block containing convolutional layers, batch normalization layers, and linear rectified activation functions for dimensionality reduction and feature recombination. Then, global average pooling and multilayer perceptron network are used to extract channel-level global description information. Finally, the inverted function, i.e. the Sigmoid activation function, is used for nonlinear mapping to calculate the channel attention weight vector. The weight calculation process is shown in formula (4):

[0086] (4)

[0087] in, This represents the calculated channel attention weight vector, used to measure the contribution of different feature channels to subsequent fire detection tasks. This represents the Sigmoid activation function, used to normalize the generated weight values ​​to a continuous interval between zero and one. This represents a multilayer perceptron network used to learn the nonlinear dependencies between multimodal feature channels. This represents the global average pooling operation, used to compress the spatial two-dimensional features of each channel into a global scalar feature. This represents a convolution operation performed on the feature map to achieve dimensionality reduction and advanced semantic reorganization of the features. This represents the hybrid feature map obtained from the initial stitching in the above steps;

[0088] (5) The channel attention weight vectors obtained by calculation are used to perform channel-wise weighted modulation on the recombined hybrid features. Combined with the residual connection mechanism, the weighted modulated features are added element-wise to the unweighted recombined features to output the final shallow fusion features. The output process is shown in formula (5):

[0089] (5)

[0090] in, This represents the shallow fused feature map output by the feature fusion module. This represents the convolution operation performed on the feature map. This represents the hybrid feature map obtained from the initial splicing. This represents a channel-wise multiplication operation, which is an element-wise multiplication operation in which the channel attention weights are precisely applied to the corresponding feature channels. The first term on the right side of the equation represents the feature branch enhanced by the attention mechanism, while the second term constitutes the residual direct connection branch that retains the original feature information.

[0091] (6) Targeting deep semantic features, combined with Figure 3 The internal logic of the module shown uses the physical smoke penetration characteristics of the thermal infrared mode as the net heat source guiding signal to denoise and filter the visible light mode. Specifically, a re-optimized Softmax strategy is introduced. The calculation process of this deep attention aggregation mechanism is shown in formula (6):

[0092] (6)

[0093] in, This represents the refined feature vector output after computation through the attention mechanism. This represents the query vector obtained by mapping thermal infrared features. This represents the key vector obtained by mapping visible light features; The mathematical transpose matrix representing the key vector. This represents the feature dimension of the key vector. This is a scaling factor used to scale the dot product results to prevent gradient vanishing during network training. Represents the normalized exponential function, This indicates that the smoke suppression mask, constructed based on thermal infrared response, primarily functions to impose an extreme penalty of negative infinity on non-fire source areas lacking thermal radiation verification. This represents the value vector obtained by mapping visible light features. Through the masking mechanism mentioned above, the smoke noise region in the visible light branch that has a strong response but is not a real fire point is forcibly suppressed, thereby achieving asymmetric anti-interference fusion that removes false signals and preserves true signals.

[0094] S3, such as Figure 2 As shown on the right, the specific implementation process is multi-scale detail aggregation decoding. That is, using the detail aggregation module, the deep semantic features output by the dual-stream feature extraction network are aggregated across layers with the shallow high-resolution features to repair the loss of edge information during the downsampling process and generate a high-precision fire prediction mask. The steps are as follows:

[0095] (1) Extract the high-resolution feature map output of the first stage of the encoder as spatial detail guidance features. Since this stage is located in the shallow layer of the network, its feature map has a high spatial resolution and retains the richest spatial geometry, texture and edge detail information of the fire scene, which can provide accurate positioning guidance for subsequent mask generation.

[0096] (2) Extract the deep semantic feature map output from the fourth stage of the encoder. Since its spatial scale has been greatly reduced after multiple network downsampling, it is necessary to first adjust its spatial scale through bilinear interpolation upsampling operation to make its resolution consistent with the shallow features. Then, perform cross-level stitching. The calculation process of this cross-level stitching operation is shown in formula (7):

[0097] (7)

[0098] in, This represents the aggregated feature map obtained after cross-level concatenation operations. This indicates a cascading splicing operation performed at the channel level. This represents the shallow, high-resolution feature map output from the first stage of the encoder. This represents the bilinear interpolation upsampling operation used to enlarge the feature map size. This represents the deep semantic feature map output by the fourth stage of the encoder. After the concatenation is completed, the aggregated features are smoothed and the edges are sharpened by subsequent convolutional layers to repair the loss of small fire source edge information during the downsampling process, laying the foundation for generating a high-precision fire prediction mask.

[0099] (3) After the high-resolution spatial detail guiding features extracted above are concatenated with the upsampled deep feature map in the channel dimension, they are fed into a series of convolutional layers for deep feature smoothing and edge sharpening. Finally, a nonlinear mapping is performed through the multilayer perceptron decoder head to map it to the final pixel category space and generate a prediction mask. The decoding and mapping process is shown in Equation (8):

[0100] (8)

[0101] in, This indicates that the final output is a fire segmentation mask with the exact same resolution as the original input image. This represents the multilayer perceptron decoding head, used to perform the final pixel-level classification mapping. This indicates a 3x3 convolution operation with a spatial receptive field size, applied twice consecutively to expand the receptive field and smoothly fuse features. This indicates a channel splicing operation. Indicates high-resolution spatial detail guiding features, This represents the deep feature map after upsampling. Through this cross-level aggregation decoding mechanism, the model not only obtains the powerful semantic discrimination of the deep layers, but also recovers the subtle edge details of the shallow layers.

[0102] S4, Combination Figure 4 The diagram showing the performance comparison between this invention and other detection methods illustrates the specific implementation process: combined loss optimization training. This involves constructing a topologically consistent focal Tversky combined loss function, using dynamic weighting and boundary constraints to address the issues of missed detections due to background dominance gradients and small fire sources, and then optimizing network parameters using the backpropagation algorithm. Since the background area constitutes a large proportion of real fire scenarios, while real fire points are often extremely sparse, this severe data imbalance can easily lead to network optimization biased towards the background. Therefore, this invention designs a joint optimization strategy, the steps of which are as follows:

[0103] (1) Construct the topology-consistent focus Tversky combined loss function. This total loss function integrates pixel-level classification constraints and geometric topology-based structural constraints, guiding parameter optimization of deep networks from multiple dimensions. Its overall mathematical expression is shown in formula (9):

[0104] (9)

[0105] in, This represents the total value of the combined loss function. This represents the value of the focus loss function. This represents the value of the Tversky loss function. This represents the value of the boundary loss function. This represents the balance coefficient used to adjust the boundary loss weights;

[0106] (2) To address the problem that a large number of simple background samples dominate the network gradient update, a focusing factor is introduced to dynamically reduce the weight of easily classified background samples, forcing the model to focus on mining difficult-to-classify samples such as smoke edges during training. The mathematical calculation process is shown in formula (10):

[0107] (10)

[0108] in, Represents the pixel index in the image space. To balance the weight hyperparameters of the positive and negative class sample ratios, The model represents the first The predicted probability of each pixel belonging to its true class. This is a focusing factor used to dynamically amplify the penalty for difficult-to-differentiate samples. Represents a logarithmic function with the natural constant as its base;

[0109] (3) Traditional cross-entropy or Dice loss is often ineffective in addressing the problem that sparse and small fire points are easily missed. This component asymmetrically enhances the model's recall ability for sparse fire targets by independently adjusting the penalty weights for false negatives and false positives. Its mathematical expression is shown in formula (11):

[0110] (11)

[0111] in, and These represent the model's prediction of the first... The probability of each pixel representing the fire source target and the background. and These represent the true label values ​​for the pixel as the fire source and the background, respectively. and These represent the weighting parameters that control the severity of penalties for false positives and false negatives, respectively. The boundary loss is used to calculate the distance difference between the predicted mask and the real label in terms of geometric topology, explicitly constraining the edge consistency of tiny fire sources, and achieving fine-grained segmentation and reconstruction.

[0112] S5, such as Figure 1 As shown at the bottom of the overall flowchart, the specific implementation process is test-time augmented inference, that is, introducing a test-time augmentation strategy during the model inference stage. By effectively fusing the prediction results of the original image and the geometrically flipped image, a high-precision fire segmentation mask is finally output. The steps are as follows:

[0113] (1) The acquired original input image is directly fed into the dual-stream fire semantic segmentation network model that has been fully trained and optimized and whose network weight parameters have been completely fixed for a single forward prediction, thereby obtaining the prediction probability map of the corresponding original spatial direction. The mathematical expression of this forward prediction process is shown in formula (12):

[0114] (12)

[0115] in, This represents the predicted probability map of the original spatial orientation, which is the final output of the segmentation model. This represents a fire semantic segmentation and detection model that has been trained, optimized, and has fixed parameters. This represents the full-size data of the original test image input to the model.

[0116] (2) In order to obtain robust feature representations with spatial transformation invariance, the same input image is subjected to a strict horizontal flip geometric spatial transformation operation. Then, the horizontally flipped image is fed back into the fire semantic segmentation detection model for positive prediction to obtain the corresponding flip probability map. In order to ensure the accurate correspondence of spatial pixels, the flip probability map is further subjected to a reverse horizontal flip operation so that it is completely and strictly aligned back to the viewpoint direction of the original input image in the spatial coordinate system. The mathematical calculation expression of the above feature extraction and spatial reverse alignment process is shown in formula (13):

[0117] (13)

[0118] in, This represents the spatial alignment probability map obtained after the original image is horizontally flipped for prediction and then flipped back to its original orientation. This represents the inverse space operation function that is strictly corresponding to the horizontal flip transformation. This represents a fire semantic segmentation and detection model that has been trained, optimized, and has fixed parameters. This represents the function that performs a horizontal flip spatial geometry operation on the original input image. This represents the full-size data of the original input test image;

[0119] (3) Perform pixel-by-pixel precise fusion calculation on the spatial alignment prediction probability maps obtained based on the two different inference paths. Specifically, according to the principle of equal weight, take the arithmetic mean of the corresponding pixel positions of the prediction probability map in the original direction and the alignment probability map flipped back to the original direction, and use it as the prediction result finally output by the method proposed in this invention. The mathematical expression of this pixel-by-pixel fusion calculation process is shown in formula (14):

[0120] (14)

[0121] in, This represents the high-precision fire segmentation probability map output by the method of the present invention after enhancement processing during testing. The half-constant term on the right side of the equation represents a constant weighting coefficient used to balance and calculate the arithmetic mean of two corresponding spatial probability maps. This represents the predicted probability map of the original spatial orientation directly obtained in the first step above. This represents the spatial alignment probability map obtained after flipping back to the original orientation in the second step above.

[0122] This integrated inference strategy can significantly smooth out discrete noise at the target prediction edge and greatly eliminate random errors generated during a single network prediction process, thereby improving the stability and accuracy of the final forest fire detection.

[0123] This invention may have other embodiments. Without departing from the spirit and essence of this invention, those skilled in the art can make various corresponding changes and modifications according to this invention, but these corresponding changes and modifications should all fall within the scope of protection of this invention.

Claims

1. A method for detecting smoke and forest fires based on thermal infrared-guided asymmetric fusion, characterized in that, The method includes the following steps: S1. Multimodal data construction and preprocessing: Acquire paired data of visible light images and thermal infrared images in fire scenarios, normalize and geometrically enhance the images, and construct a multimodal training set; S2. Dual-stream hybrid feature encoding: Construct a dual-stream feature extraction network based on the Transformer architecture to extract features of the visible light mode and the thermal infrared mode respectively, and adopt a hierarchical hybrid fusion strategy: perform conventional fusion in the shallow stage of the network to preserve texture details, and introduce a smoke suppression-guided aggregation mechanism in the deep stage to perform asymmetric anti-interference fusion. S3, Multi-scale detail aggregation decoding: Using the detail aggregation module, the deep semantic features output by the dual-stream feature extraction network are aggregated across layers with the shallow high-resolution features to repair the loss of edge information during the downsampling process and generate a high-precision fire prediction mask. S4. Combined loss optimization training: Construct a topology-consistent focus Tversky combined loss function, solve the background dominant gradient and small fire source missed detection problems through dynamic weighting and boundary constraints, and optimize network parameters using the backpropagation algorithm; S5. Test-time Enhanced Inference: During the model inference stage, a test-time enhancement strategy is introduced to fuse the prediction results of the original image and the flipped image, and output the final fire segmentation map.

2. The method for detecting smoke and forest fires based on thermal infrared-guided asymmetric fusion as described in claim 1, characterized in that, Step S1 involves multimodal data construction and preprocessing: acquiring paired visible light and thermal infrared images of a fire scene, normalizing and geometrically enhancing the images, and constructing a multimodal training set, including the following steps: S11. The acquired visible light image and thermal infrared image are standardized and normalized as shown in formula (1): (1) in, This represents the image data after normalization. This represents the original image data input. This represents the statistical mean of all pixel values ​​in the corresponding modality dataset. This represents the statistical standard deviation of all pixel values ​​in the corresponding modality dataset; S12. During model training, online data augmentation is performed simultaneously on the input visible light image, thermal infrared image, and corresponding label image. The augmentation methods include random horizontal flipping, random angle rotation, and random scale scaling. If the thermal infrared image is a single-channel grayscale image, it needs to be mapped to three-channel data through channel duplication during the preprocessing stage to match the network input dimension.

3. The method for detecting smoke and forest fires based on thermal infrared-guided asymmetric fusion as described in claim 1, characterized in that, Step S2, dual-stream hybrid feature encoding: Construct a dual-stream feature extraction network based on the Transformer architecture to extract features from the visible light mode and the thermal infrared mode respectively, and adopt a hierarchical hybrid fusion strategy: Perform conventional fusion in the shallow stage of the network to preserve texture details, and introduce a smoke suppression-guided aggregation mechanism in the deep stage for asymmetric anti-interference fusion, including the following steps: S21. The dual-stream feature extraction network uses two Mix Transformers with shared structures but not shared weights as the backbone network. Each backbone network contains four feature extraction stages: the first, the second, the third, and the fourth. The resolutions of the output feature maps are 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the original image, respectively. S22. The hierarchical hybrid fusion strategy is divided into two parts based on the above four stages: shallow fusion and deep fusion. The first and second stages constitute the shallow stage of the network, in which feature fusion module is used for modal fusion. The third and fourth stages constitute the deep stage of the network, in which smoke suppression-guided aggregation module is used for modal fusion. S23. For deep semantic features, the smoke-penetrating characteristics of the thermal infrared mode are used as a guiding signal to denoise and filter the visible light mode; specifically, the Re-Softmax strategy is introduced, as shown in formula (5): (5) in, This represents the refined characteristics output after smoke suppression aggregation calculation. This represents the query vector generated by thermal infrared feature mapping. This represents the key vector generated by visible light feature mapping. The matrix representing the transpose of the key vector. This represents a value vector generated by visible light feature mapping. This represents the feature dimension of the key vector. Scaling factor Represents the normalized exponential function, This refers to a smoke suppression mask generated by taking the maximum negative value of the non-fire source area. This mask forcibly suppresses the smoke noise area in the visible light branch that has a strong response but lacks thermal radiation verification, and only retains the color information of the real fire source, thereby achieving asymmetric anti-interference fusion that removes false information and retains true information. S24. For the feature fusion module in the shallow stage, the extracted shallow visible light feature map and thermal infrared feature map are directly concatenated and stitched together in the channel dimension to initially aggregate the spatial information of the two modes and obtain preliminary mixed features. The stitching process is shown in formula (6): (6) in, This represents the hybrid feature map obtained after concatenation along the channel dimension. This indicates a concatenated stitching operation performed along the channel dimension of the feature map. This represents the visible light feature map extracted during the shallow stages of the network. This represents the thermal infrared feature map extracted in the shallow layer stage of the network; S25. Perform feature dimensionality reduction and high-level semantic reorganization on the concatenated hybrid features, and calculate the importance weights of each feature channel. Specifically, the hybrid features are input into a convolutional block containing convolutional layers, batch normalization layers, and ReLU activation functions for processing. Then, global average pooling and multilayer perceptron network are used to extract channel-level global description information. Finally, the Sigmoid activation function is used for nonlinear mapping to calculate the channel attention weight vector. The weight calculation process is shown in formula (7). (7) in, This represents the calculated channel attention weight vector, used to measure the contribution of different feature channels to subsequent tasks. This represents the Sigmoid activation function, used to normalize the generated weight values ​​to a range of zero to one. This represents a multilayer perceptron network used to learn non-linear dependencies between feature channels. This represents the global average pooling operation, used to compress the spatial features of each channel into a global scalar. This represents a convolution operation performed on the feature map to achieve dimensionality reduction and recombination of the features. This represents the hybrid feature map obtained from the initial stitching in the above steps; S26. The channel attention weight vectors obtained from the calculation are used to perform channel-wise weighted modulation on the recombined hybrid features. Combined with the residual connection mechanism, the weighted modulated features are added element-wise to the unweighted recombined features to output the final shallow fusion features. The output process is shown in formula (8): (8) in, This represents the shallow fused feature map that is the final output of the feature fusion module. This represents the convolution operation performed on the feature map. This represents the hybrid feature map obtained from the initial splicing. This represents a channel-wise multiplication operation, which is an element-wise multiplication operation that applies channel attention weights to the corresponding feature channels. The first term on the right-hand side of the equation represents the channel attention weight vector calculated in the above steps. The second term represents the feature branch enhanced by the attention mechanism. This constitutes a residual direct connection branch that retains the original feature information.

4. The method for detecting smoke and forest fires based on thermal infrared-guided asymmetric fusion as described in claim 1, characterized in that, Step S3, multi-scale detail aggregation decoding: Using the detail aggregation module, the deep semantic features output by the dual-stream feature extraction network are aggregated across layers with the shallow high-resolution features to repair the loss of edge information during downsampling and generate a high-precision fire prediction mask. This includes the following steps: S31. Extract the high-resolution feature map output from the first stage of the encoder as spatial detail guiding features. ; S32. Extract the deep semantic feature map from the fourth stage output of the encoder. Its spatial scale is adjusted by bilinear interpolation upsampling, as shown in formula (9): (9) in, This represents the feature map after upsampling. This indicates a bilinear interpolation upsampling operation. This represents a one-to-one convolution operation used to adjust the number of channels. Represents a deep semantic feature map; S33. The feature maps obtained in steps S31 and S32 are concatenated along the channel dimension and fed into consecutive convolutional layers for feature smoothing and edge sharpening. Finally, they are mapped to the class space through the MLP decoding head to generate a prediction mask. As shown in formula (10): (10) in, This represents a segmentation mask with the same resolution as the original image. This refers to the multilayer perceptron decoder head. This represents a 3x3 convolution operation. This indicates a channel splicing operation. This indicates high-resolution spatial detail guiding features. This represents the deep feature map after upsampling.

5. The method for detecting smoke and forest fires based on thermal infrared-guided asymmetric fusion as described in claim 1, characterized in that, Step S4 involves optimizing the training with combined loss: Constructing a topologically consistent focus Tversky combined loss function, addressing the issues of background dominant gradient and missed detection of small fire sources through dynamic weighting and boundary constraints, and optimizing network parameters using the backpropagation algorithm, as shown in Equation (11): (11) in, This represents the total value of the combined loss function. The balancing coefficient is used to adjust the weights of the boundary loss. To address focus loss, a focus factor is introduced. The weights of simple background samples are dynamically reduced, forcing the model to focus on mining difficult-to-classify samples such as smoke edges, as shown in formula (12): (12) in, Indicates pixel index, To balance the weight hyperparameters of positive and negative samples, The model represents the first The predicted probability of the true class of each pixel. As a focusing factor, Represents the natural logarithm; S41, the aforementioned For the Tversky loss, the recall ability of the model for sparse fire targets is asymmetrically enhanced by adjusting the false negative penalty weights, as shown in Equation (13): (13) in, and These represent the model's prediction of the first... The probability of each pixel representing the fire source target and the background. and These represent the true label values ​​for the pixel as the fire source and the background, respectively. and These represent the weighting parameters that control the severity of penalties for false positives and false negatives, respectively. The boundary loss is used to calculate the distance difference between the predicted mask and the real label in terms of geometric topology, explicitly constraining the edge consistency of tiny fire sources, and achieving fine-grained segmentation and reconstruction.

6. The method for detecting smoke and forest fires based on thermal infrared-guided asymmetric fusion as described in claim 1, characterized in that, Step S5, Test-Time Enhanced Inference: During the model inference phase, a test-time enhancement strategy is introduced to fuse the prediction results of the original image and the flipped image, outputting the final fire segmentation map. This includes the following steps: S51. The acquired original input image is directly fed into the trained fire semantic segmentation network model for a single positive prediction to obtain the prediction probability map of the original spatial direction. The positive prediction process is shown in formula (14): (14) in, This represents the predicted probability map of the original direction output by the model. This represents a fire detection model that has been trained, optimized, and has fixed parameters. The test image represents the original input; S52. In order to obtain feature representations with spatial transformation invariance, a geometric transformation operation of horizontal flipping is performed on the same input image. The flipped image is then fed into the fire detection model for prediction to obtain a flipped probability map. Furthermore, the flipped probability map is subjected to a reverse horizontal flipping operation to ensure that it is strictly aligned back to the original direction in the spatial coordinate system. The above feature extraction and spatial alignment process is shown in formula (12): (15) in, This represents the probability map obtained after horizontal flipping prediction and then flipping it back to the original direction. This represents the function that is the inverse of the horizontal flip operation. This represents a fire detection model that has been trained and optimized. This function represents the horizontal flipping operation performed on the image. The test image represents the original input; S53. Perform pixel-by-pixel fusion calculation on the predicted probability maps obtained from the two different inference paths. Specifically, take the arithmetic mean of the predicted probability map in the original direction and the probability map flipped back to the original direction, and use it as the final prediction result output by the method of the present invention. The fusion calculation process is shown in formula (16): (16) in, This represents the high-precision fire segmentation probability map that is ultimately output by this method. This represents the weighting coefficient used to calculate the arithmetic mean of two corresponding probability graphs. This represents the predicted probability map of the original directions obtained in the above steps. This represents the probability diagram obtained after reversing and flipping back to the original direction in the above steps.

Images