A lightweight multi-modal image fusion system and method for a smoke scene

Abstract

The present application provides a kind of lightweight multi-modal image fusion system for smoke scene, it is related to computer vision and multi-modal image processing technical field.The present application includes: encoder, for extracting the infrared multi-modal feature and visible light multi-modal feature of infrared image and visible light image under the same smoke scene to be fused;Encoder is hybrid structure, contains two parallel feature extraction branches: CNN branch and Transformer branch;Latent feature mapping network, for receiving infrared multi-modal feature and visible light multi-modal feature, and carries out smoke suppression and feature enhancement to infrared multi-modal feature and visible light multi-modal feature, respectively outputs corresponding smoke-free infrared feature and smoke-free visible light feature;Fusion module, for receiving smoke-free infrared feature and smoke-free visible light feature and integration, generates integrated feature containing infrared and visible light complementary information;Decoder, for receiving integrated feature and gradually reconstructing, generates and outputs the final fusion image after smoke removal.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and multimodal image processing technology, and more specifically, to a lightweight multimodal image fusion system and method for smoke scenes. Background Technology

[0002] Currently, multimodal image fusion technology enhances the comprehensiveness and robustness of a system's environmental understanding by integrating complementary information from different modalities. Infrared and visible light image fusion has demonstrated significant application value in fields such as security monitoring and autonomous driving. However, when application scenarios expand to extreme visual environments such as smoke, traditional fusion methods face severe challenges. Smoky environments cause severe loss of detail and color distortion in visible light images, while infrared images, although possessing certain penetrating power, are susceptible to uneven temperature distribution and noise interference, resulting in blurred target edges and a decreased signal-to-noise ratio.

[0003] In the field of traditional fusion methods, the DSFusion method enhances feature extraction capabilities by designing local attention modules and improving channel attention mechanisms. The FISCNet model innovatively performs fusion processing simultaneously in the frequency and spatial domains. The LENFusion network, designed for low-light nighttime environments, achieves interactive optimization of low-light enhancement and fusion through a brightness adjustment network and feedback loop structure.

[0004] For multimodal image fusion under severe weather conditions such as smoke, the FIVFusion method integrates the dehazing and fusion processes through aggregation and decoupling strategies; the HaIVFusion model designs a scene information recovery network to restore the visible light texture obscured by smoke and introduces color consistency loss to reduce color cast; DeDNet uses a decomposition-driven network to jointly optimize the fusion and denoising tasks.

[0005] Existing fusion systems for smoke scenarios suffer from the following key problems: First, most existing methods employ a step-by-step processing strategy, such as removing smoke before fusion. This process is not only complex but also prone to error accumulation, lacking a unified model that can simultaneously achieve smoke removal and fusion. Second, existing fusion systems typically have a large number of parameters and high computational complexity, making them difficult to deploy in resource-constrained real-time systems. Third, under a unified smoke removal and fusion framework, the impact mechanism of different fusion strategies on model performance is still unclear, lacking systematic analysis and comparison.

[0006] In view of the above, this application is hereby submitted. Summary of the Invention

[0007] The present invention provides a lightweight multimodal image fusion system for smoke scenes to improve at least one of the above-mentioned technical problems.

[0008] In a first aspect, the present invention provides a lightweight multimodal image fusion system for smoke scenes, comprising:

[0009] An encoder is used to extract infrared multimodal features and visible light multimodal features from infrared and visible light images of the same smoke scene to be fused; the encoder has a hybrid structure and contains two parallel feature extraction branches: a CNN branch and a Transformer branch; Furthermore, the CNN branch is used to extract local spatial features, and the Transformer branch is used to capture global contextual information; Furthermore, the encoder consists of 4 layers of CNN branches and 3 layers of Transformer branches, with the two branches set up in parallel.

[0010] A latent feature mapping network is used to receive the infrared multimodal features and the visible light multimodal features, perform smoke suppression and feature enhancement on the infrared multimodal features and the visible light multimodal features, and output the corresponding smoke-free infrared features and smoke-free visible light features respectively; The latent feature mapping network consists of three concatenated dual convolutional modules, each of which is composed of two convolutional layers connected in series. The convolutional layers are connected using the ReLU activation function to introduce non-linearity. Furthermore, residual connections are preserved between the convolutional layers.

[0011] A fusion module is used to receive and integrate the smokeless infrared features and the smokeless visible light features to generate an integrated feature containing complementary information of infrared and visible light. The fusion module employs any one or more of the following fusion strategies: An additive averaging fusion strategy integrates features by averaging the elements one by one. The channel attention-based fusion strategy achieves selective fusion of the channel dimension by adaptively learning the importance weights of different feature channels. A spatial attention-based fusion strategy focuses on the differences in importance of different spatial locations and performs weighted fusion in the spatial dimension. A fusion strategy based on axial attention is used to calculate attention by combining feature axis information. Preferably, the average fusion strategy is used to integrate the smokeless infrared features and the smokeless visible light features.

[0012] A decoder is used to receive the integrated features, perform stepwise reconstruction, generate and output the final fused image after smoke removal; The decoder is a hybrid structure that alternates between double convolution and single convolution. It gradually restores the spatial resolution of the integrated features through transposed convolution or upsampling operations, while gradually reducing the number of channels of the integrated features through convolution operations.

[0013] Furthermore, each layer of the decoder is provided with a skip connection to the corresponding layer of the encoder.

[0014] The encoder, latent feature map network, and decoder are pre-trained in two stages to obtain optimal parameters. The two-stage training includes: The first stage involves pre-training the encoder and decoder using a self-supervised learning approach. The original clear image is input into the encoder to extract features, and then the decoder reconstructs the image based on the features. The goal is to minimize the difference between the reconstructed image and the original clear image, thereby optimizing the parameters of the encoder and decoder. Second stage: Fix the encoder parameters pre-trained in the first stage, and use paired smoke infrared images, smoke visible light images, and corresponding smoke-free clear images to perform supervised training on the latent feature mapping network and the decoder.

[0015] On the other hand, the present invention also provides a lightweight multimodal image fusion method for smoke scenes, comprising the following steps: S1: Input the infrared image and the visible light image of the same smoke scene to be fused into the encoder respectively, and extract the infrared multimodal features of the infrared image and the visible light multimodal features of the visible light image; S2: Input the infrared multimodal features and the visible light multimodal features into a latent feature mapping network, and perform smoke suppression and feature enhancement on the infrared multimodal features and the visible light multimodal features through the latent feature mapping network, and output the corresponding smoke-free infrared features and smoke-free visible light features respectively; S3: Input the smokeless infrared feature and the smokeless visible light feature into the fusion module for integration, and generate an integrated feature containing complementary infrared and visible light information; S4: Input the integrated features into the decoder for stepwise reconstruction, generate and output the final fused image after smoke removal; By adopting the above technical solution, the present invention can achieve the following technical effects: The encoder adopts a parallel structure of shallow CNN and shallow Transformer, and the latent feature mapping network adopts a simple three-layer double convolution structure, realizing a lightweight network architecture design. Through an end-to-end unified framework, it simultaneously achieves effective smoke suppression and high-quality infrared-visible light information fusion, avoiding parameter expansion caused by deep stacking or complex attention modules. It overcomes the problems of complex process and error accumulation in existing step-by-step processing strategies, effectively solving the problems of complex process, error accumulation and lack of lightweight solution in existing solutions, and improving fusion efficiency and overall performance. Attached Figure Description

[0016] To more clearly illustrate the technical solution of the present invention, the accompanying drawings used in the specific embodiments of the present invention will be briefly introduced below. It should be understood that the following drawings only show some specific embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained from these drawings without creative effort.

[0017] Figure 1 This is a schematic diagram of the structure of a lightweight multimodal image fusion system for smoke scenes.

[0018] Figure 2 This is a schematic diagram of the latent feature mapping network in Example 1.

[0019] Figure 3 These are the SSIM values ​​of different lightweight structures in Example 1 at different training intervals.

[0020] Figure 4 This is a test loss curve of different lightweight structures under different training intervals in Example 1.

[0021] Figure 5 This is a graph showing the changes of each loss function and the overall objective function in Example 1.

[0022] Figure 6 This is a flowchart illustrating a lightweight multimodal image fusion method for smoke scenes.

[0023] Figure 7 This is a comparison curve of the smoke removal capabilities of all the comparative methods in Example 2 under medium smoke concentration scenarios.

[0024] Figure 8 This is a comparison curve of the smoke removal capabilities of all the comparative methods in Example 2 under high smoke concentration scenarios.

[0025] Figure 9 This is a comparison curve of the smoke removal capabilities of all the comparative methods in Example 2 under ultra-high smoke concentration scenarios.

[0026] Figure 10 This is a comparison chart of the smoke removal and fusion results of all the comparison methods in Example 2 under ultra-high smoke concentration scenarios.

[0027] Figure 11 This is a comparison chart of the smoke removal and fusion results of all the comparison methods in Example 2 under a high smoke concentration scenario.

[0028] Figure 12 This is a comparison chart of the smoke removal and fusion results of all the comparison methods in Example 2 under a medium smoke concentration scenario. Detailed Implementation

[0029] The technical solutions of the present invention will now be clearly and completely described with reference to the accompanying drawings in the embodiments of the present invention.

[0030] Example 1 See Figure 1 The first embodiment of the present invention provides a lightweight multimodal image fusion system for smoke scenes, comprising: The encoder is used to extract infrared multimodal features and visible light multimodal features from infrared and visible light images of the same smoke scene to be fused. The CNN branch consists of multiple stacked convolutional layers and uses a combination of initial feature extraction and residual connections to progressively extract local spatial features; the Transformer branch uses a multi-head self-attention mechanism to perform global modeling, capture the dependencies between distant pixels, and obtain global contextual information. Furthermore, the features extracted by the CNN branch and the Transformer branch are concatenated in the intermediate layer, and then the number of channels is compressed through a 1×1 convolutional layer to form a complementary feature representation that combines local details and global semantics. For example, the block size of the Transformer branch is set to 16, the embedding dimension is set to 16, the encoding depth is set to 3, the number of attention heads is set to 16, the feedforward network dimension is set to 1024, and the dropout rate is set to 0.1.

[0031] See Figure 2 The latent feature mapping network consists of three concatenated dual convolutional modules, each consisting of two convolutional layers connected in series. The convolutional layers are nonlinearly introduced using the ReLU activation function. The network receives the infrared multimodal features and the visible light multimodal features, performs smoke suppression and feature enhancement on the infrared and visible light multimodal features, and outputs the corresponding smoke-free infrared and smoke-free visible light features, respectively. Furthermore, residual connections are preserved between the convolutional layers; In this embodiment, the Latent Feature Mapping Network (LSMR) is one of the core structures of a lightweight multimodal image fusion system for smoke scenes. Its network depth and number of parameters directly affect the overall lightweight nature of the system. To further analyze the impact of different lightweight structures on the performance of the LMR, this embodiment selects three different lightweight structures for comparative experiments, named Lightweight Structure 1, Lightweight Structure 2, and Lightweight Structure 3, respectively. Lightweight Structure 1: It consists of a dual convolution module, namely a cascaded structure of single convolution, ReLU and single convolution; Lightweight Structure 2: It consists of two Lightweight Structure 1s connected in series, that is, two dual convolutional modules stacked together; Lightweight Structure 3: It consists of three Lightweight Structure 1s connected in series, that is, three dual convolutional modules stacked together; It should be noted that this embodiment does not select a larger number of modules (such as four or five) in order to control the scale of model parameters and computational complexity, avoid excessively increasing inference latency, and ensure the lightweight characteristics of the overall model.

[0032] See Figure 3 and Figure 4 The performance trend curves of training SSIM values ​​and testing losses for different lightweight structures at different training intervals are presented respectively. The experiments use the same training configuration (dataset, optimizer, learning rate, batch size, etc.), and only differ in the number of modules in the latent feature mapping network.

[0033] Experimental results show that: Under the same training conditions, lightweight architecture 3 achieved the highest final training SSIM value, slightly higher than lightweight architecture 2, while lightweight architecture 1 had the lowest performance. In terms of test losses, lightweight structure 3 is also at a low level, slightly lower than lightweight structure 2, and significantly better than lightweight structure 1. The overall trend shows that as the number of dual convolutional modules increases, the training performance of the model continues to improve, but the performance gain gradually decreases as the number of dual convolutional modules increases.

[0034] Based on the above comparative experimental results, this embodiment selects lightweight structure 3 as the default configuration of the latent feature mapping network. With the depth of three double convolutional modules, this structure can achieve the highest structural similarity and the lowest test loss under the premise of controllable parameter quantity, thus balancing performance and system lightweight requirements.

[0035] A fusion module is used to receive and integrate the smokeless infrared features and the smokeless visible light features to generate an integrated feature containing complementary information of infrared and visible light. The fusion module employs any one or more of the following fusion strategies: An additive averaging fusion strategy integrates features by averaging the elements one by one. The channel attention-based fusion strategy achieves selective fusion of the channel dimension by adaptively learning the importance weights of different feature channels. A spatial attention-based fusion strategy focuses on the differences in importance of different spatial locations and performs weighted fusion in the spatial dimension. A fusion strategy based on axial attention is used to calculate attention by combining feature axis information. The fusion module can dynamically select any of the above fusion strategies based on preset configurations or real-time computing resource constraints. In the default lightweight mode, preferably, the average fusion strategy is used to integrate the smokeless infrared features and the smokeless visible light features.

[0036] A decoder is used to receive the integrated features, perform stepwise reconstruction, generate and output the final fused image after smoke removal; The decoder is a hybrid structure that alternates between double convolution and single convolution. It gradually restores the spatial resolution of the integrated features through transposed convolution or upsampling operations, while gradually reducing the number of channels of the integrated features through convolution operations.

[0037] Furthermore, each layer of the decoder is connected to the corresponding layer of the encoder via skip connections. These skip connections concatenate the output feature map of the encoder layer with the feature map of the current layer of the decoder along the channel dimension to fuse deep semantic information with shallow detail information.

[0038] In this embodiment, a lightweight design is adopted: the encoder consists of 4 CNN branches and 3 Transformer branches, the latent feature mapping network adopts a three-layer dual convolutional structure, and the fusion module defaults to the average fusion strategy with the lowest computational cost. Therefore, the overall system weight is controlled at 19.64MB, and the processing speed for a single image pair can reach approximately 0.7 seconds. It should be noted that the above parameters are merely examples, and those skilled in the art can appropriately adjust the number of network layers or channels according to actual application scenarios, all of which fall within the scope of protection of this invention.

[0039] Furthermore, two-stage training is employed beforehand to obtain the optimal parameters for the encoder, latent feature map network, and decoder, specifically including: The first stage involves pre-training the encoder and decoder using a self-supervised learning approach. The original clear image is input into the encoder to extract features, and then the decoder reconstructs the image based on the features. The goal is to minimize the difference between the reconstructed image and the original clear image, thereby optimizing the parameters of the encoder and decoder. In the first stage, three complementary loss functions are used together to form the overall objective function. : ; In the formula, This is the pixel loss due to mean square error, used to ensure pixel-level reconstruction accuracy; Structural similarity loss is used to preserve the overall structural information of the original, clear image; This is a gradient loss function used to preserve edge and texture details of the original sharp image.

[0040] The mean square error loss The calculation formula is: ; In the formula, For the original clear image in the first The pixel value at each position. To reconstruct the image at the The pixel value at each position. The total number of pixels in the original, clear image; The mean square error loss By comparing pixels one by one, the network output is forced to numerically approximate the input.

[0041] The structural similarity loss The calculation formula is: ; In the formula, For the original clear image, To reconstruct the image, It is a structural similarity function; The structural similarity function Taking into account the similarity across three dimensions—brightness, contrast, and structure—the calculation formula is as follows: ; In the formula, and These represent the original sharp images. and reconstructed images Average pixel value within a local window (measures brightness); and These represent the original sharp images. and reconstructed images Standard deviation within a local window (a measure of contrast); Represents the original sharp image and reconstructed images Covariance within a local window (a measure of structure); , It is the stability constant; Those skilled in the art can adjust the stability constant according to the image bit depth or actual needs. , The value of , for example, , .

[0042] Gradient loss The calculation formula is: ; In the formula, The Laplacian gradient response of the original sharp image. The Laplacian gradient response of the reconstructed image, The number of pixels involved in the calculation; The original sharp image in spatial coordinates The pixel value (i.e., grayscale value or brightness value) at that location. The reconstructed image in spatial coordinates The pixel value (i.e., grayscale value or brightness value) at that location. The gradient loss By penalizing the differences in gradient responses, the network output is made to maintain the edge sharpness and texture details of the original clear image; See Figure 5 During the first stage of training, all three loss functions showed a rapid decrease and gradually converged to a stable trend as the number of training rounds increased. The loss value decreased significantly in the early stage of training. As the number of iterations increased, the rate of decrease of each loss curve slowed down and gradually oscillated and converged, indicating that the encoder and decoder parameters were gradually optimized to a stable state, and the system had the ability to accurately restore image pixels, overall structure and edge texture.

[0043] The second stage involves fixing the encoder parameters pre-trained in the first stage, using paired infrared images with smoke, visible light images with smoke, and corresponding clear images without smoke. The infrared images with smoke and the visible light images with smoke are used as input images, and the corresponding clear images without smoke are used as supervision signals. The output is a smoke-free fused image, which is used to supervise the training of the latent feature mapping network and the decoder. By allowing the network to learn the mapping relationship from smoke features to smoke-free features, end-to-end learning of the smoke suppression function is achieved.

[0044] In the second stage, the total training function is: ; In the formula, This represents the L1 loss between the original sharp image and the reconstructed image. For structural similarity loss, In order to perceive loss, To adjust the hyperparameters for the weights of various losses; L1 loss is more robust to outliers than L2 loss and produces clearer output results. The calculation formula is as follows: ; In the formula, For smoke-free clear images in the first The value at each pixel position To remove smoke and fuse images in the first The value at each pixel position; N The total number of pixels in the image; The structural similarity loss The calculation formula is the same as in the first stage. In the second stage, the original clear image in the formula is replaced with a smoke-free clear image, and the reconstructed image is replaced with a smoke-free fused image. Perceived loss The introduction of this method aims to make the fusion result closer to a smoke-free, clear image at the high-level semantic feature level, rather than just pixel-level matching. The calculation formula is as follows: ; In the formula, Indicates the pre-trained network's... Feature maps output by the layer , , These represent the number of channels, height, and width of the feature map, respectively. Perceptual loss extracts features using a pre-trained latent feature mapping network. By comparing features in a feature space at multiple scales, it can capture multi-level information from low-level texture to high-level semantics. It compares the differences between smoke-free fused images and smoke-free clear images in the feature space, generating visually more natural and semantically more accurate fusion results.

[0045] By adopting the above technical solution, the present invention can achieve the following technical effects: Two-stage training is adopted in advance. In the first stage, the encoder is made to have a strong feature extraction capability through self-supervised learning. In the second stage, a lightweight latent feature mapping network is specially optimized based on the trained encoder, which not only ensures the quality of feature extraction, but also improves the fusion performance while keeping the model lightweight. Compared to configurations using deep Transformers or large-scale convolutional networks, the encoder's three-layer dual-convolutional structure keeps the number of parameters and inference time at a low level while ensuring sufficient feature mapping capabilities, achieving a good balance between fusion performance and computational efficiency. The hybrid architecture achieves collaborative extraction of local and global features while controlling the number of parameters. The three-layer dual-convolutional structure of the latent feature mapping network completes the core smoke suppression function at minimal cost, and the simple design of the fusion module avoids the computational overhead caused by complex attention mechanisms. The overall lightweight design approach ensures that the system can meet the requirements of inference speed and deployment flexibility in practical applications while maintaining high-quality fusion results.

[0046] Example 2 See Figure 6 Based on the system of Embodiment 1, the second embodiment of the present invention provides a lightweight multimodal image fusion method for smoke scenes, including the following steps: S1: Input the infrared image and the visible light image of the same smoke scene to be fused into the encoder respectively, and extract the infrared multimodal features of the infrared image and the visible light multimodal features of the visible light image; S2: Input the infrared multimodal features and the visible light multimodal features into a latent feature mapping network, and perform smoke suppression and feature enhancement on the infrared multimodal features and the visible light multimodal features through the latent feature mapping network, and output the corresponding smoke-free infrared features and smoke-free visible light features respectively; S3: Input the smokeless infrared feature and the smokeless visible light feature into the fusion module for integration, and generate an integrated feature containing complementary infrared and visible light information; S4: Input the integrated features into the decoder for stepwise reconstruction, generate and output the final fused image after smoke removal.

[0047] To comprehensively evaluate the fusion performance of the method of the present invention under different smoke concentrations, this embodiment selects a variety of objective evaluation indicators, including peak signal-to-noise ratio (PSNR), mean square error (MSE), structural similarity (SSIM), phase consistency-based structural similarity (SCD), information entropy (EN), gradient-based fusion quality index (Qabf), and average smoke concentration (ASC). Among them, PSNR, MSE, SSIM, SCD, EN, and Qabf are all well-known image quality evaluation indicators in the field, and their calculation methods and physical meanings have been discussed in detail in a large number of published documents, so they will not be repeated here.

[0048] ASC is a novel evaluation metric proposed in this invention specifically for image fusion tasks in smoke scenes. It is used to quantitatively assess the concentration of residual smoke in the fused image; the lower the value, the more thorough the smoke removal effect. Let the... The test image at color channel c, pixel position The pixel value at is , For the first Given the total number of pixels in the images, where N is the total number of test images, the formula for calculating the smoke concentration SC is: ; In the formula, filter represents the filtering operation, where the image is filtered by 15×15, and RGB are the three color channels of the image. The formula for calculating ASC is: ; In the formula, j represents the j-th fused image during the test.

[0049] To verify the fusion performance of the method of this invention under different smoke concentration scenarios, this embodiment selected the method of this invention (denoted as Ours) and conducted a quantitative comparison experiment with four existing advanced algorithms—DSFusion (an infrared and visible light image fusion method combining detail and scene information), FISCNet (a frequency integral and spatial compensation network), LENFusion (a joint low-light enhancement and fusion network for nighttime infrared and visible light image fusion), and MMNet (a multi-scale network based on MixFormer). The experiment selected Peak Signal-to-Noise Ratio (PSNR), Mean Square Error (MSE), Structural Similarity (SSIM), and Average Smoke Concentration (ASC) as evaluation indicators, and evaluated the performance of each method under three scenarios: medium smoke concentration, high smoke concentration, and ultra-high smoke concentration. Higher PSNR and SSIM values ​​indicate better fused image quality; lower MSE and ASC values ​​indicate smaller errors and more thorough smoke removal.

[0050] See Figure 7 As shown in Table 1, in scenarios with medium smoke concentration, the method of this invention exhibits significant advantages in all evaluation indicators: the PSNR value of this invention reaches 63.57, the highest among all compared methods; the MSE value is as low as 0.035, the lowest among all methods; in terms of image structure preservation, the SSIM value of this invention is 1.405, significantly better than other compared algorithms; in addition, the average smoke concentration ASC value of this invention is only 0.319, far lower than other methods, indicating that its smoke removal effect is the most thorough.

[0051] Table 1 shows the fusion performance of each method under different smoke concentration scenarios.

[0052] See Figure 8 As shown in Table 2, the method of the present invention still maintains a comprehensive lead in high smoke concentration scenarios: the PSNR value of the method of the present invention is 63.21 and the MSE value is 0.036, both of which are the best; the SSIM value is 1.380 and the ASC value is 0.364, which are also the best among all the comparison methods, verifying the robustness of the method of the present invention under high interference conditions.

[0053] Table 2. Fusion performance of various methods in high smoke concentration scenarios.

[0054] See Figure 9 As shown in Table 3, the superiority of the method of the present invention is further verified in ultra-high smoke concentration scenarios: the PSNR value of the method of the present invention is 62.72, the SSIM value is 1.354, both of which are the highest; the MSE value is 0.038, the ASC value is 0.387, both of which are the lowest.

[0055] Table 3. Fusion performance of various methods in ultra-high smoke concentration scenarios.

[0056] See Figure 10 In the image, mode 1 is the infrared image, and mode 2 is the visible light image with smoke. This is a comparison of the smoke removal and fusion results of all comparison methods in a scene with extremely high smoke concentration. Figure 10 As can be seen, compared with other comparison methods, the fused image obtained by the method of the present invention has higher visual clarity and more complete detail preservation. To further verify the robustness of the method of the present invention under different smoke concentrations, Figure 11 and Figure 12 Comparison figures of smoke removal fusion results under medium and high smoke concentrations are presented respectively. Figure 11 and Figure 12 Mode 1 in the image is an infrared image, and mode 2 is a visible light image with smoke. Observation shows that... Figure 11 and Figure 12 The conclusions presented are consistent with Figure 10 Consistency means that the method of the present invention can obtain the best visual quality fusion results in different smoke concentration scenarios.

[0057] In summary, under different smoke concentration scenarios, the method of this invention outperforms existing advanced algorithms such as DSFusion, FISCNet, LENFusion, and MMNet in four metrics: PSNR, SSIM, MSE, and ASC. This fully demonstrates the effectiveness and advancement of the lightweight multimodal image fusion method for smoke scenarios proposed in this invention in improving image quality and enhancing smoke removal capabilities.

[0058] Because both the encoder and the latent feature mapping network adopt a lightweight design, the feature extraction and fusion speed of the method in this invention is relatively fast. To verify its lightweight effect, the average fusion speed of all comparative methods was tested under different smoke concentration conditions, and the weights of all comparative methods are given. The results are shown in Table 4.

[0059] Table 4. Weights of each method and average fusion speed under different smoke concentration scenarios.

[0060] As shown in Table 4, in medium smoke concentration scenarios, the average fusion speed of the method of this invention is 0.94 seconds / pair, significantly faster than DSFusion (1.84 seconds / pair), FISCNet (6.99 seconds / pair), LENFusion (13.91 seconds / pair), and MMNet (1.94 seconds / pair). In high smoke concentration and ultra-high smoke concentration scenarios, the method of this invention also maintains the fastest fusion speed, at 0.72 seconds / pair and 0.73 seconds / pair, respectively.

[0061] The aforementioned speed advantage is mainly attributed to two aspects: first, the use of an average fusion strategy instead of a complex attention transformation fusion significantly reduces the computational overhead of the fusion layer; second, the entire fusion system employs a lightweight design. The experimental results demonstrate the good practicality of the method in terms of lightweight design.

[0062] This invention conducts a systematic comparative study of different fusion strategies. Four representative fusion strategies were designed experimentally: average fusion with minimal computational cost, feature-weighted fusion based on channel attention mechanism, pixel-level weighted fusion based on spatial attention mechanism, and global dependency modeling fusion based on axial attention. These strategies aim to examine the impact of different feature integration methods on smoke removal fusion performance and employ multiple objective evaluation metrics for comprehensive assessment.

[0063] Table 5 illustrates the performance differences of different fusion strategies across various metrics. The data shows that channel attention fusion performs best in PSNR and SSIM, reaching 62.82 and 1.365 respectively, indicating its significant advantages in preserving image structural information and improving signal-to-noise ratio. The average fusion strategy achieves the best results in MSE and SCD, with an MSE as low as 0.038 and an SCD as high as 0.949, demonstrating its superior performance in suppressing errors and enhancing spectral consistency. In contrast, spatial fusion and axis attention fusion generally perform worse than the first two strategies, with axis attention fusion exhibiting a high MSE of 0.101 and an SSIM of only 1.120, indicating relatively poor performance.

[0064] Table 5 Performance Comparison of Different Fusion Strategies

[0065] In summary, channel attention fusion excels at refined feature selection and structural fidelity, while average fusion performs better in global error control and color fidelity. Considering that this embodiment needs to control mean square error and improve overall stability while ensuring fusion quality, average fusion was ultimately chosen as the core fusion strategy. This strategy not only achieved the lowest MSE value, but its SCD and EN indices are also at a leading level, achieving a good balance across key quality evaluation dimensions. This provides a reliable feature foundation for subsequent smoke suppression and image reconstruction.

[0066] Example 3 The third embodiment of the present invention provides a lightweight multimodal image fusion device for smoke scenes, including: a processor, a memory, and a computer program stored in the memory; the computer program can be executed by the processor to implement a lightweight multimodal image fusion method for smoke scenes as described in Embodiment 2.

[0067] Example 4 The fourth embodiment of the present invention provides a computer-readable storage medium, the computer-readable storage medium including a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to execute a lightweight multimodal image fusion method for smoke scenes as described in Embodiment 2.

[0068] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A lightweight multimodal image fusion system for smoke scenes, characterized in that, include: An encoder is used to extract infrared multimodal features from an infrared image and visible light multimodal features from a visible light image in the same smoke scene to be fused; the encoder has a hybrid structure and contains two parallel feature extraction branches: a CNN branch and a Transformer branch; A latent feature mapping network is used to receive the infrared multimodal features and the visible light multimodal features, perform smoke suppression and feature enhancement on the infrared multimodal features and the visible light multimodal features, and output the corresponding smoke-free infrared features and smoke-free visible light features respectively; The latent feature mapping network consists of three dual convolutional modules connected in series, and each dual convolutional module is composed of two convolutional layers connected in series. A fusion module is used to receive and integrate the smokeless infrared features and the smokeless visible light features to generate an integrated feature containing complementary information of infrared and visible light. A decoder is used to receive the integrated features, perform stepwise reconstruction, generate and output the final fused image after smoke removal.

2. The lightweight multimodal image fusion system for smoke scenes according to claim 1, characterized in that, The encoder consists of 4 CNN branches and 3 Transformer branches, with the two branches set in parallel. The CNN branches are used to extract local spatial features, and the Transformer branches are used to capture global context information.

3. A lightweight multimodal image fusion system for smoke scenes according to claim 1, characterized in that, A ReLU activation function is used between the convolutional layers to introduce nonlinearity.

4. A lightweight multimodal image fusion system for smoke scenes according to claim 3, characterized in that, Residual connections are also provided between the convolutional layers.

5. A lightweight multimodal image fusion system for smoke scenes according to claim 1, characterized in that, The fusion module employs any one or more of the following fusion strategies: An additive averaging fusion strategy integrates features by averaging the elements one by one. The channel attention-based fusion strategy achieves selective fusion of the channel dimension by adaptively learning the importance weights of different feature channels. A spatial attention-based fusion strategy focuses on the differences in importance of different spatial locations and performs weighted fusion in the spatial dimension. A fusion strategy based on axial attention is used to perform attention calculation by combining feature axis information.

6. A lightweight multimodal image fusion system for smoke scenes according to claim 1, characterized in that, The fusion module uses the average fusion strategy to integrate the smokeless infrared features and the smokeless visible light features.

7. A lightweight multimodal image fusion system for smoke scenes according to claim 1, characterized in that, The decoder is a hybrid structure that alternates between double convolution and single convolution. It gradually restores the spatial resolution of the integrated features through transposed convolution or upsampling operations, while gradually reducing the number of channels of the integrated features through convolution operations.

8. A lightweight multimodal image fusion system for smoke scenes according to claim 7, characterized in that, Each layer of the decoder is connected to the corresponding layer of the encoder via skip connections.

9. A lightweight multimodal image fusion system for smoke scenes according to claim 1, characterized in that, The encoder, latent feature map network, and decoder are pre-trained in two stages, including: The first stage involves pre-training the encoder and decoder using a self-supervised learning approach. The original clear image is input into the encoder to extract features, and then the decoder reconstructs the image based on the features. The goal is to minimize the difference between the reconstructed image and the original clear image, thereby optimizing the parameters of the encoder and decoder. Second stage: Fix the encoder parameters pre-trained in the first stage, and use paired smoke infrared images, smoke visible light images, and corresponding smoke-free clear images to perform supervised training on the latent feature mapping network and the decoder.

10. A lightweight multimodal image fusion method for smoke scenes based on the system of any one of claims 1 to 9, characterized in that, Includes the following steps: S1: Input the infrared image and the visible light image of the same smoke scene to be fused into the encoder respectively, and extract the infrared multimodal features of the infrared image and the visible light multimodal features of the visible light image; S2: Input the infrared multimodal features and the visible light multimodal features into a latent feature mapping network, and perform smoke suppression and feature enhancement on the infrared multimodal features and the visible light multimodal features through the latent feature mapping network, and output the corresponding smoke-free infrared features and smoke-free visible light features respectively; S3: Input the smokeless infrared feature and the smokeless visible light feature into the fusion module for integration, and generate an integrated feature containing complementary infrared and visible light information; S4: Input the integrated features into the decoder for stepwise reconstruction, generate and output the final fused image after smoke removal.

Images