Multi-modal fusion method and system for thermal diffusion suppression dynamic perception

Abstract

The invention provides a multi-modal fusion method and system for thermal diffusion suppression dynamic perception, and relates to the technical field of image fusion. According to the method and the system, firstly, a thermal diffusion suppression module is built, physical modeling is carried out on thermal target diffusion of an infrared image, the problem of boundary blurring caused by a thermal diffusion effect in the infrared image is solved, reconstruction of a blurred edge is realized, and the infrared image with thermal diffusion suppression is obtained; then a fusion network is built, and adaptive fusion is carried out on the infrared image and the visible light image subjected to thermal diffusion suppression to obtain a fused image; according to the method and the system, the problem of boundary blurring of the infrared image caused by thermal diffusion is solved; sharing features and specific features of different modes can be effectively separated and utilized through an orthogonal loss constraint feature decoupling module; the comparative learning strategy enhances the distinguishing ability of different modal specific features; the multi-scale feature dynamic weight fusion realizes the optimization and retention of details.

Description

Technical Field

[0001] The present invention relates to the field of image fusion technology, and in particular to a multimodal fusion method and system for thermal diffusion suppression dynamic perception. Background Art

[0002] Image fusion is an important image processing technique that effectively integrates image information captured by different sensors to generate a more informative image. In multimodal image fusion, the fusion of infrared and visible light images is particularly important because these two types of images have complementary properties: infrared images can capture thermal radiation information in low-light environments, while visible light images provide rich texture and color details.

[0003] In recent years, with the development of deep learning technology, deep learning-based image fusion methods have made significant progress. These methods can be mainly divided into CNN-based methods, autoencoder-based methods, and hybrid architecture methods. Although these methods have demonstrated excellent fusion performance, they still have the following problems: first, it is difficult to effectively separate and utilize the specific information of different modalities; second, the computational complexity of the feature extraction network is too high; third, there is a lack of effective fusion mechanisms for multi-scale features; and fourth, they cannot effectively handle the thermal boundary diffusion problem commonly found in infrared images, resulting in blurred object boundaries in the fused image.

[0004] Thermal boundary bleed is a typical phenomenon in infrared imaging, manifesting as blurred boundaries between objects of different materials. This phenomenon is primarily caused by heat conduction and diffusion. Specifically, when objects of different thermal properties are placed adjacent to each other, heat diffuses between them, blurring the boundaries between objects in the infrared image. This heat leakage not only degrades the detail quality of infrared images but also negatively impacts subsequent image fusion and target recognition tasks.

[0005] Therefore, designing a fusion method that can fully utilize the dual-modal characteristics of visible light and infrared, effectively deal with the thermal boundary diffusion problem and maintain computational efficiency has important theoretical and application value. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to address the deficiencies of the above-mentioned prior art and provide a multimodal method and system for dynamic perception of thermal diffusion suppression to achieve multimodal fusion of infrared images and visible light images.

[0007] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0008] In one aspect, the present invention provides a multimodal fusion method for dynamic perception of thermal diffusion suppression, comprising:

[0009] Build a thermal diffusion suppression module to perform physical modeling of thermal target diffusion in infrared images, solve the boundary blur problem caused by thermal diffusion effects in infrared images, reconstruct the blurred edges, and obtain infrared images with thermal diffusion suppression.

[0010] Build a fusion network to adaptively fuse the infrared image with thermal diffusion suppression and the visible light image to obtain a fused image;

[0011] The fusion network adopts a lightweight end-to-end fusion network as the basic framework, including an initial feature extraction module, a multi-scale feature extraction module, a feature decoupling module, a contrastive learning enhancement module and a detail-aware adaptive fusion module;

[0012] The initial feature extraction module obtains the initial feature representation of the infrared image and the visible light image; then the deep features of the infrared image and the visible light image at different scales are obtained through the multi-scale feature extraction module; the multi-scale features of the infrared image and the visible light image are specifically separated through the feature decoupling module; the contrastive learning enhancement module is used to perform contrastive learning enhancement on the specifically separated features and then perform the first stage of fusion; at the same time, the features of different scales are aligned to a uniform spatial resolution to provide the multi-scale features of a uniform format for the detail-perceived adaptive fusion module; finally, the multi-scale features of the uniform format are adaptively fused through the detail-perceived adaptive fusion module and combined with the fused features of the first stage to generate a fused image.

[0013] Furthermore, the specific method of building a heat diffusion suppression module, performing physical modeling on the diffusion of thermal targets in infrared images, solving the boundary blur problem caused by the heat diffusion effect in infrared images, and reconstructing the blurred edges to obtain an infrared image with heat diffusion suppression is as follows:

[0014] Combining the physical model of thermal target diffusion in infrared images with deep learning, a thermal diffusion suppression module is designed to restore the original thermal signal through a step-by-step denoising approach.

[0015] A heat diffusion model is established for infrared images, and a diffusion probability model based on material property perception is fully considered to gradually eliminate the heat diffusion effect in the infrared imaging process.

[0016] The thermal diffusion model is a method for modeling the boundary blur problem caused by thermal diffusion in the infrared imaging process as a convolution of the original thermal signal and the thermal diffusion of the atmospheric medium;

[0017] The thermal diffusion model also includes a diffusion probability model; the diffusion probability model models the influence of material properties on thermal diffusion and introduces material property parameters into the thermal diffusion model.

[0018] Furthermore, the physical model of thermal target diffusion in infrared images is combined with deep learning to design a thermal diffusion suppression module. The specific method of restoring the original thermal signal through gradual denoising is as follows:

[0019] The observed blurred infrared image I obs As the initial image I s =I obs ;

[0020] The image is iteratively updated using the diffusion model in deep learning to make the observed blurred infrared image close to the target image;

[0021] The iteration is terminated when the KL divergence is less than the preset threshold or the maximum number of iterations is reached.

[0022] Furthermore, the feature decoupling module explicitly models the shared features and specific features between infrared images and visible light modalities through orthogonal loss constraints, thereby achieving the separation and retention of unique information of different modalities; the orthogonal loss constraints are achieved by minimizing the correlation matrix norm between shared features and specific features.

[0023] Furthermore, the contrastive learning enhancement module adopts a contrastive learning strategy to enhance modality-specific feature extraction, so that the infrared thermal target and the visible light detail features are clearly distinguished, thereby improving the information extraction efficiency; the contrastive learning enhancement module includes an enhanced spatial attention unit, a channel attention unit and a contrastive learning module; the enhanced spatial attention unit and the channel attention unit enhance the features after specific separation based on the spatial attention mechanism and the channel attention mechanism respectively; the contrastive learning module enhances the discrimination ability between modalities by calculating the contrast loss between infrared features and visible light features.

[0024] Furthermore, the specific method of the contrastive learning enhancement module using the contrastive learning strategy to enhance the modality-specific feature extraction is:

[0025] The infrared feature F is transformed into IR and visible light characteristics F VIS Projection into low-dimensional feature space;

[0026] Randomly sample the spatial dimensions of infrared features and visible light features in the low-dimensional feature space to reduce computational overhead;

[0027] Calculate the similarity matrix between cross-modal features;

[0028] Construct a contrastive learning loss function to measure the similarity between infrared and visible light features;

[0029] The contrast loss is optimized through back-propagation, the projection network parameters are updated, and the projection network's ability to extract modality-specific features is enhanced.

[0030] Furthermore, the contrastive learning enhancement module combines the shared features and specific features between the infrared image and the visible light modality modeled by the feature decoupling module, and obtains multi-scale features after channel compression and spatial alignment. The multi-scale features are enhanced through the spatial attention mechanism; and the multi-scale features are converted into specific pixel values ​​to obtain multi-scale features in a unified format; at the same time, the shared features and specific features are modulated and further enhanced through the channel attention mechanism to obtain infrared features enhanced by contrastive learning. and visible light characteristics These two features are adaptively fused to obtain the fused features of the first stage.

[0031] Furthermore, the detail-aware adaptive fusion module adopts a detail-aware adaptive weighted fusion strategy to adaptively fuse multi-scale features in a unified format, and combines the fusion results of the first stage to generate a fused image, mapping the information in the high-dimensional feature space to the image pixel space; this module adopts an edge-aware mechanism based on Sobel filtering and a dynamic weight allocation strategy to achieve effective fusion of multi-scale features and detail retention.

[0032] On the other hand, the present invention also provides a multimodal fusion system for dynamic perception of thermal diffusion suppression, comprising: a thermal diffusion suppression module and a fusion network construction module;

[0033] The heat diffusion suppression module performs physical modeling on the diffusion of thermal targets in infrared images, solves the boundary blur problem caused by the heat diffusion effect in infrared images, reconstructs the blurred edges, and obtains an infrared image with heat diffusion suppressed;

[0034] The fusion network construction module builds a fusion network to adaptively fuse the infrared image with thermal diffusion suppression and the visible light image to obtain a fused image.

[0035] The beneficial effects of the above technical solution are as follows: the multimodal fusion method for dynamic perception of thermal diffusion suppression solves the problem of blurred infrared image boundaries caused by thermal diffusion; the feature decoupling module, constrained by orthogonal loss, effectively separates and utilizes shared and specific features of different modalities; the contrastive learning strategy enhances the ability to distinguish between modality-specific features; the dynamic weighted fusion of multi-scale features optimizes the preservation of details; and the lightweight design reduces computational complexity and improves processing efficiency. Experimental results demonstrate that the proposed method outperforms existing methods in terms of visual quality, objective evaluation indicators, and support for downstream visual tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 A flowchart of a multimodal fusion method for dynamic perception of thermal diffusion suppression provided by an embodiment of the present invention;

[0037] Figure 2 A schematic structural diagram of a heat diffusion suppression module provided in an embodiment of the present invention;

[0038] Figure 3 A block diagram of adaptive fusion of infrared images and visible light images with heat diffusion suppression provided by an embodiment of the present invention;

[0039] Figure 4 A schematic structural diagram of a feature decoupling module provided in an embodiment of the present invention;

[0040] Figure 5 A schematic diagram of the structure of a detail-aware adaptive fusion module provided in an embodiment of the present invention;

[0041] Figure 6 Comparison diagram of fusion effect diagrams between the method of the present invention and the existing fusion methods provided in the embodiments of the present invention, (a) is the fusion effect diagram of the Infrared method, (b) is the fusion effect diagram of the Visible method, (c) is the fusion effect diagram of the SeAFusion method, (d) is the fusion effect diagram of the GTF method, (e) is the fusion effect diagram of the DenseFuse method, (f) is the fusion effect diagram of the FusionGAN method, (g) is the fusion effect diagram of the IFCNN method, (h) is the fusion effect diagram of the GANMCC method, (i) is the fusion effect diagram of the SDNet method, (j) is the fusion effect diagram of the DeFusion method, (k) is the fusion effect diagram of the SPDFusion method, and (l) is the fusion effect diagram of the method of the present invention. DETAILED DESCRIPTION

[0042] The following embodiments of the present invention are described in further detail with reference to the accompanying drawings and examples. The following examples are used to illustrate the present invention but are not intended to limit the scope of the present invention.

[0043] In this embodiment, a multimodal fusion method for dynamic perception of thermal diffusion suppression is provided. Figure 1 As shown, the following steps are included:

[0044] Step S1: Build a heat diffusion suppression module, such as Figure 2 As shown in the figure, the thermal diffusion of infrared images is physically modeled to solve the boundary blur problem caused by the thermal diffusion effect in infrared images, reconstruct the blurred edges, and obtain infrared images with thermal diffusion suppressed.

[0045] Step S11: Combining the physical model of thermal target diffusion in infrared images with deep learning, a thermal diffusion suppression module is designed to restore the original thermal signal through a step-by-step denoising method. Specifically, the following steps are included:

[0046] Step S111: The observed blurred infrared image I obs As the initial image I s =I obs ;

[0047] Step S112: The image is iteratively updated using the diffusion model in deep learning to make the observed blurred infrared image close to the target image, as shown in the following formula:

[0048]

[0049] Among them, α t is the learning rate, D KL is the KL divergence; is a gradient operator used to calculate the rate of change of KL divergence in a specific direction and to find the direction in which KL divergence is reduced; t , I true Represent the image in the current iteration step and the ideal real infrared image, which is the target image that the algorithm hopes to approach;

[0050] The thermal diffusion suppression module of the present invention is designed to address the problems of "thermal leakage" and "spatial thermal gradient blurring" in infrared images. Thermal diffusion in infrared imaging stems from the scattering and absorption effects in the atmospheric transport equation (RTE). When thermal radiation propagates through the atmosphere, heat energy diffuses at the molecular scale, blurring the boundaries of thermal targets. This process can be described by a modified heat conduction equation:

[0051]

[0052] Where T(x,y,t) represents the temperature field at the spatial position (x,y) at time t, α is the thermal diffusion coefficient, and S(x,y,t) is the heat source term;

[0053] In the infrared imaging process, the boundary blur caused by thermal diffusion can be modeled as the convolution of the original thermal signal and the atmospheric medium thermal diffusion kernel:

[0054] I obs (x,y)=I true (x,y)*G σ (x,y)+η(x,y)

[0055] Among them, I obs is the observed blurred infrared image, I true is an ideal diffusion-free image, G σ is a Gaussian diffusion kernel with parameter σ, and η is the observation noise.

[0056] In order to reversely deduce the heat diffusion process, a thermal information inversion model based on the Laplace operator is introduced. Inspired by diffusion physics, heat diffusion satisfies Fourier's heat conduction law, and its time evolution follows the following partial differential equation:

[0057]

[0058] Where D is the diffusion coefficient. By solving the inverse process of this equation, the initial diffusion-free state can be estimated:

[0059]

[0060] Among them, λ i is a learnable inverse diffusion coefficient, adaptively estimated by a neural network.

[0061] Step S113: When the KL divergence D KL (I t ||I true ) is less than a preset threshold or the maximum number of iterations is reached to terminate the iteration;

[0062] Step S12: establishing a heat diffusion suppression model for the infrared image, and fully considering a diffusion probability model based on material property perception, to gradually eliminate the heat diffusion effect during the infrared imaging process;

[0063] The thermal diffusion model is a convolution of the original thermal signal and the thermal diffusion of the atmospheric medium to model the boundary blur problem caused by thermal diffusion during infrared imaging, as shown in the following formula:

[0064] I obs (x,y)=I true (x,y)*G σ (x,y)+η(x,y)

[0065] Among them, I obs is the observed blurred infrared image, x, y are the pixel space coordinates, I true is an ideal diffusion-free image, G σ is the Gaussian diffusion kernel with parameter σ, and η is the observation noise;

[0066] The thermal diffusion model also includes a diffusion probability model. The model is based on the influence of material properties on thermal diffusion. Different materials have significantly different effects on thermal diffusion. Material property parameters are introduced into the thermal diffusion model as shown in the following formula:

[0067]

[0068] M=[k,c,ρ,ε]

[0069] Among them, q(I t |I t-1,M) represents the given material property M and the image I at the previous moment t-1 Under the condition of t The conditional probability distribution of M represents the material property vector, which includes various material properties that affect the thermal diffusion process: thermal conductivity k: determines the heat conduction rate; heat capacity c: affects the material's ability to store heat; density ρ: affects thermal mass; emissivity ε: affects infrared radiation characteristics; β t (M) is the diffusion coefficient related to material properties and determines the rate of heat diffusion; represents the normal distribution (Gaussian distribution) function, which is used to model the probability distribution of the heat diffusion process; I is an infrared image containing thermal noise;

[0070] Step S2: Build a fusion network to adaptively fuse the infrared image with thermal diffusion suppression and the visible light image, such as Figure 3 As shown, the fused image is obtained;

[0071] The fusion network uses a lightweight end-to-end fusion network as its basic framework. The network has strong feature extraction capabilities, visible light and infrared dual-modal feature decoupling capabilities, and multi-scale dynamic weight fusion capabilities. The fusion network includes an initial feature extraction module, a multi-scale feature extraction module, a feature decoupling module, a contrastive learning enhancement module, and a detail-aware adaptive fusion module.

[0072] Specifically, the infrared image and visible light image with thermal diffusion suppression are respectively sent to the initial feature extraction module to obtain the initial feature representation of the infrared image and the visible light image; then the deep features of the infrared image and the visible light image at different scales are obtained through the multi-scale feature extraction module; the multi-scale features of the infrared image and the visible light image are specifically separated through the feature decoupling module; the contrastive learning enhancement module is used to perform contrastive learning enhancement on the specifically separated features and then perform the first stage of fusion; at the same time, features of different scales are aligned to a unified spatial resolution to provide multi-scale fusion features in a unified format for the detail-aware adaptive fusion module; finally, the detail-aware adaptive fusion module is used to perform adaptive fusion, and the fusion result of the first stage is combined to generate a fused image.

[0073] Specifically, for the input thermal diffusion suppressed infrared image I IR and visible light image I VIS , the multi-scale feature extraction module can be expressed as:

[0074]

[0075] Among them, l∈1,2,3,4 represents the level of feature extraction performed by the multi-scale feature extraction module, that is, the feature level, C l is the feature compression operation, Fl-1 Represents the features of the l-1 layer, IR represents the features of the infrared mode, and VIS represents the features of the visible light mode. Represents the features of layer l;

[0076] εl represents the encoder module of the lth layer of the multi-scale feature extraction module, which is composed of MobileBlock, as shown in the following formula:

[0077] εl(x)=MobileBlock(x)=PWConv°DWConv°PWConv(x)

[0078] Among them, PWConv represents point convolution and DWConv represents depth-wise separable convolution;

[0079] The feature decoupling module performs the specificity separation process as follows:

[0080]

[0081] in, represents shared features, represents the specific feature, Φ represents the feature decoupling module;

[0082] Shared Features and specific characteristics Satisfy the orthogonality loss constraint;

[0083] The adaptive fusion process of the detail-aware adaptive fusion module is combined with a color preservation mechanism, which is expressed as:

[0084]

[0085] Among them, I fused is the fused image, Y fused is the brightness channel of the fused image, Cr VIS and Cb VIS are the Cr and Cb chromaticity channels of the original visible light image, Represents a convolutional network.

[0086] In this embodiment, the feature decoupling module is as follows: Figure 4 As shown in the figure, the shared features and specific features between infrared images and visible light modalities are explicitly modeled through orthogonal loss constraints, achieving the separation and retention of unique information of different modalities. Specifically, the orthogonal loss constraint makes the shared features and modality-specific features independent of each other, ensuring that the feature information of different modalities is not confused, thereby more effectively preserving their unique texture, edge and semantic information.

[0087] The orthogonal loss constraint is achieved by minimizing the correlation matrix norm between shared features and specific features, which can be expressed mathematically as follows:

[0088]

[0089] in, and are the normalized shared features and specific features of the lth layer of the feature decoupling module, respectively. F represents the Frobenius norm.

[0090] Through this constraint, the feature decoupling module can construct an orthogonal subspace of the feature space to achieve <F shared ,F specific The feature separation target of >≈0 enables the thermal radiation features of the infrared modality and the color texture features of the visible light modality to be accurately separated and reasonably preserved in the subsequent fusion process.

[0091] In this embodiment, the contrastive learning enhancement module uses a contrastive learning strategy to enhance modality-specific feature extraction, making the infrared thermal target and visible light detail features more distinct, thereby improving information extraction efficiency;

[0092] The contrastive learning enhancement module includes: an enhanced spatial attention unit, a channel attention unit and a contrastive learning module; the enhanced spatial attention unit and the channel attention unit enhance the features after specific separation based on the spatial attention mechanism and the channel attention mechanism respectively; the contrastive learning module enhances the discrimination ability between modalities by calculating the contrast loss between infrared features and visible light features.

[0093] In this embodiment, the calculation of the enhanced spatial attention unit can be expressed as:

[0094] A spatial (F)=σ(Conv(Concat(Avg(F),Max(F))))

[0095] Where Avg and Max represent the channel average and maximum operations of the feature map respectively, Concat represents channel splicing, Conv is the convolution operation, and σ is the sigmoid activation function.

[0096] The channel attention unit can be expressed as:

[0097] A channel (F)=σ(FC2(ReLU(FC1(AvgPool(F)))))

[0098] Among them, AvgPool represents global average pooling, and FC1 and FC2 represent fully connected layers.

[0099] In this embodiment, the contrastive learning module enhances the ability to distinguish between modalities by calculating the contrast loss between infrared and visible light features. The specific process is as follows:

[0100] Step C1: Feature projection: First, the infrared feature F IR and visible light characteristics F VIS Projected into the low-dimensional feature space, denoted as f IR =P IR (F IR ) and f VIS =P VIS (F VIS ), where P IR and P VIS Both are projection networks composed of convolutional layers and normalization layers; the role of the projection network is to map infrared and visible light features into a common embedding space.

[0101] Step C2: Feature sampling: Randomly sample the spatial dimensions of infrared features and visible light features in the low-dimensional feature space to reduce computational overhead. The sampling ratio is r, and the sampling features are obtained. and Where N = H × W × r represents the number of sampling points, B is the batch, C is the number of feature map channels, H is the feature map height, and W is the feature map width;

[0102] Step C3: Similarity calculation: Calculate the similarity matrix between cross-modal features Where τ is the temperature parameter;

[0103] Step C4: Contrastive Learning Loss: Constructing Contrastive Learning Loss Function Used to measure the similarity between infrared and visible light features:

[0104]

[0105] Among them, sim represents cosine similarity, f ir and f vis are the characteristic representations of infrared and visible light, respectively.

[0106] Contrastive learning loss encourages similarity rather than differentiation of features at corresponding locations. The exp function in this formula maps similarity to exponential space, forming a probability distribution. The temperature parameter τ controls the smoothness of this distribution. When τ is large, the distribution is smoother, and the differences in the importance of each feature pair are reduced. When τ is small, the distribution is more concentrated, emphasizing the importance of positive pairs.

[0107] Step C5: Backpropagation update: Optimize the contrast loss through backpropagation, update the projection network parameters, and enhance the projection network's ability to extract modality-specific features, expressed as:

[0108]

[0109] Among them, θ new The updated projection network parameters, that is, the neural network weights and biases after training and updating; θ old The current (before update) network parameters, that is, the neural network weights and biases before this update; η is the learning rate, which controls the step size of each parameter update. is the gradient operator, which means the derivative of the parameter θ;

[0110] By updating the gradient descent of the contrast loss function, the projection network can better project similar cross-modal features to similar locations and dissimilar features to farther locations. By optimizing the parameters of this projection network, the features at corresponding positions in the infrared and visible light images are brought closer in the projection space, thereby improving the effect of feature alignment.

[0111] In this embodiment, infrared features and visible light features are decoupled into shared features and modality-specific features through a feature decoupling module. The shared features and modality-specific features are then concatenated after channel compression and spatial alignment to obtain multi-scale features, which are then enhanced using a spatial attention mechanism.

[0112] At the same time, shared features and modality-specific features are modulated and further enhanced through the channel attention mechanism to obtain infrared features enhanced by contrastive learning. and visible light characteristics These two features are adaptively fused to obtain the fusion features of the first stage, which improves the discriminability and effectiveness of the fusion features. The above process is shown in the following formula:

[0113] Channel Compression:

[0114]

[0115] Spatial alignment:

[0116]

[0117] Multi-scale features after splicing:

[0118]

[0119] Enhanced spatial attention:

[0120]

[0121] modulation:

[0122]

[0123] Channel Attention Enhancement:

[0124]

[0125] Weight generation:

[0126]

[0127] Enhanced feature fusion:

[0128] F fused =α·F IR * +(1-α)·F VIS *

[0129] Among them, Compress ir and Compress vis They are channel compression functions for infrared IR and visible light VIS modes, respectively, which ensure effective information fusion by compressing feature maps. It is the feature map of the infrared image (IR) and visible light image (VIS) after channel compression; the Align function is responsible for aligning features of different scales to a unified spatial resolution; this step provides multi-scale fusion features in a unified format for the subsequent detail-aware adaptive fusion module. are multiple feature maps compressed from different scale features of the infrared (IR) modality. Multiple feature maps compressed at different scales for the visible light (VIS) modality are concatenated together as input for subsequent fusion operations. Represents the feature maps of the infrared image (IR) and visible light image (VIS) after the alignment operation. In multimodal learning, since the spatial resolution and scale of different modalities (such as infrared and visible light images) are usually different, it is necessary to align the features of different modalities through the Align operation. In this way, the feature maps of the two modalities can match in space, making subsequent fusion or calculation more efficient. The aligned feature maps will be used in subsequent fusion steps, such as splicing and weighted fusion. The infrared and visible light features are modulated, W1, W2, W3, W4, and W5 are weight parameters dynamically adjusted according to the contrast loss; WeightNet is used to generate adaptive weight α to balance global information and local details; F fused It is the fusion feature of the first stage.

[0130] The contrastive learning enhancement module effectively distinguishes the specific features of infrared and visible light images through a contrastive learning strategy, thereby enhancing the thermal targets in infrared images and the detailed textures in visible light images, thereby improving the information retention efficiency and fusion quality in the subsequent fusion process.

[0131] The first stage of feature-level fusion produces rich multi-scale feature representations However, these features are still in the high-dimensional feature space. In order to generate the final fused image, it is necessary to convert these abstract features into specific pixel values ​​to obtain a unified feature representation F unified The conversion process is shown in the following formula:

[0132]

[0133] Among them, Detailbranch is specifically responsible for extracting multi-scale features Detailed information such as edges and textures, F unified It will serve as the input for the second stage image-level fusion.

[0134] In this embodiment, the unified feature representation F is obtained based on the contrastive learning enhancement module. unified ,The detail aware adaptive fusion module adopts the detail aware adaptive weighted fusion strategy to ,perform adaptive fusion and combines the combination results of the ,first stage to generate a fused image, mapping the information of the high-dimensional ,feature space to the image pixel space, e.g. Figure 5 As shown in Figure 2, this module uses an edge-aware mechanism based on Sobel filtering and a dynamic weight allocation strategy to achieve effective fusion of multi-scale features and detail preservation. This module includes the following key steps:

[0135] Step A1: Multi-scale feature compression: First, the features of different scales are channel compressed, which can be expressed as:

[0136]

[0137] in, represents the IR features and VIS features after channel compression, C IR and C VIS It is a channel compression function implemented by 1×1 convolution, l represents the feature level, and Represent the unified feature representation F unified The IR and VIS feature components of the lth layer in .

[0138] Step A2: Sobel edge detection: Apply the Sobel operator to the channel compressed features to extract edge information. The edge features of the infrared image and visible light image are expressed as:

[0139]

[0140] Among them, E IR 、E VIS are the edge features of infrared images and visible light images respectively, and S is the Sobel filter operation, which is defined as:

[0141]

[0142] Among them, G x and G y They are Sobel operators in the horizontal and vertical directions, F represents the image, F*G x Represents the convolution of image F and horizontal Sobel operator, F*G y Represents the convolution of image F with the vertical Sobel operator.

[0143] Step A3: Dynamic weight generation: Generate fusion weights based on edge features and channel compressed features, expressed as:

[0144]

[0145] Among them, W IR 、W VIS are the fusion weights of infrared images and visible light images respectively, σ is the sigmoid activation function to ensure that the sum of the weights is 1, Conv represents the convolution operation, and Concat represents the concatenation function.

[0146] Step A4: Feature alignment: Unify the features of different scales to the target size through bilinear upsampling operation, expressed as:

[0147]

[0148] in, are features of unified size for infrared images and visible light images, respectively, and Up represents a bilinear upsampling operation.

[0149] Step A5: Detail enhancement fusion: Use detail enhancement strategy to perform feature fusion, expressed as:

[0150]

[0151] Among them, ⊙ represents element-by-element multiplication, γ is the edge enhancement coefficient;

[0152] Step A6: Multi-scale feature integration: Perform weighted integration on the fusion features of different scales to obtain the multi-scale fusion features F of the infrared image and the visible light image. final , expressed as:

[0153]

[0154] Among them, α l is the weight coefficient of the l-th layer feature, generated by the global context-aware network.

[0155] The detail-aware adaptive fusion module effectively preserves the target contours in infrared images and the texture details in visible light images through Sobel edge detection and dynamic weight allocation strategy, realizes the adaptive fusion of multi-scale features, and significantly improves the visual quality and information integrity of the fused image.

[0156] In this embodiment, the performance comparison results of different image fusion methods on the Roadscene dataset are shown in Table 1 and Figure 6 As shown in the figure, the comparison methods include 8 existing methods such as GTF, DenseFuse, FusionGAN, IFCNN, GANMCC, SDNet, DeFusion, and SPDFusion, and the proposed method (Ours). Six objective indicators were used to evaluate the fusion results: information entropy (EN), peak signal-to-noise ratio (PSNR), mutual information (MI), visual information fidelity (VIF), correlation coefficient (CC), and edge-based quality index (Qabf). It can be seen from the results that the proposed method performs well in multiple key indicators, especially in PSNR (31.86), MI (3.87), VIF (0.80), CC (0.60), Qab (0.55) and other indicators. It achieves the best performance, proving the effectiveness and superiority of the proposed two-stage fusion strategy.

[0157] Table 1 Performance comparison of different image fusion methods

[0158]

[0159] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some or all of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope defined by the claims of the present invention.

Claims

1. A multimodal fusion method for dynamic perception of thermal diffusion suppression, characterized by: include: Build a thermal diffusion suppression module to perform physical modeling of thermal target diffusion in infrared images, solve the boundary blur problem caused by thermal diffusion effects in infrared images, reconstruct the blurred edges, and obtain infrared images with thermal diffusion suppression. Build a fusion network to adaptively fuse the infrared image with thermal diffusion suppression and the visible light image to obtain a fused image; The fusion network adopts a lightweight end-to-end fusion network as the basic framework, including an initial feature extraction module, a multi-scale feature extraction module, a feature decoupling module, a contrastive learning enhancement module and a detail-aware adaptive fusion module; The initial feature extraction module obtains the initial feature representation of the infrared image and the visible light image; then the multi-scale feature extraction module obtains the deep features of the infrared image and the visible light image at different scales; then the feature decoupling module specifically separates the multi-scale features of the infrared image and the visible light image; the contrastive learning enhancement module performs contrastive learning enhancement on the specifically separated features and then performs the first stage of fusion; and simultaneously aligns the features of different scales to a unified spatial resolution, providing the multi-scale features of a unified format for the detail perception adaptive fusion module; Finally, the multi-scale features in a unified format are adaptively fused through the detail-aware adaptive fusion module and combined with the fused features of the first stage to generate a fused image.

2. The multimodal fusion method for dynamic perception of thermal diffusion suppression according to claim 1, characterized in that: The specific method of building a heat diffusion suppression module, performing physical modeling on the diffusion of thermal targets in infrared images, solving the boundary blur problem caused by the heat diffusion effect in infrared images, and reconstructing the blurred edges to obtain an infrared image with heat diffusion suppressed is as follows: Combining the physical model of thermal target diffusion in infrared images with deep learning, a thermal diffusion suppression module is designed to restore the original thermal signal through a step-by-step denoising approach. A heat diffusion model is established for infrared images, and a diffusion probability model based on material property perception is fully considered to gradually eliminate the heat diffusion effect in the infrared imaging process. The thermal diffusion model is a method for modeling the boundary blur problem caused by thermal diffusion in the infrared imaging process as a convolution of the original thermal signal and the thermal diffusion of the atmospheric medium; The thermal diffusion model also includes a diffusion probability model; the diffusion probability model models the influence of material properties on thermal diffusion and introduces material property parameters into the thermal diffusion model.

3. The multimodal fusion method for dynamic perception of thermal diffusion suppression according to claim 2, characterized in that: The specific method of combining the physical model of thermal target diffusion in infrared images with deep learning to design a thermal diffusion suppression module and restore the original thermal signal through gradual denoising is as follows: The observed blurred infrared image I obs As the initial image I s =I obs ; The image is iteratively updated using the diffusion model in deep learning to make the observed blurred infrared image close to the target image; The iteration is terminated when the KL divergence is less than the preset threshold or the maximum number of iterations is reached.

4. The multimodal fusion method for dynamic perception of thermal diffusion suppression according to claim 1, characterized in that: The feature decoupling module explicitly models the shared features and specific features between infrared images and visible light modalities through orthogonal loss constraints, thereby separating and retaining the unique information of different modalities; the orthogonal loss constraints are achieved by minimizing the correlation matrix norm between shared features and specific features.

5. The multimodal fusion method for dynamic perception of thermal diffusion suppression according to claim 4 is characterized by: The contrastive learning enhancement module adopts a contrastive learning strategy to enhance modality-specific feature extraction, making the infrared thermal target and visible light detail features clear and improving information extraction efficiency; the contrastive learning enhancement module includes an enhanced spatial attention unit, a channel attention unit and a contrastive learning module; The enhanced spatial attention unit and channel attention unit enhance the features after specific separation based on the spatial attention mechanism and the channel attention mechanism respectively; The contrastive learning module enhances the discrimination ability between modalities by calculating the contrast loss between infrared features and visible light features.

6. The multimodal fusion method for dynamic perception of thermal diffusion suppression according to claim 5, characterized in that: The specific method of the contrastive learning enhancement module using the contrastive learning strategy to enhance the modality-specific feature extraction is: The infrared feature F is transformed into IR and visible light characteristics F VIS Projection into low-dimensional feature space; Randomly sample the spatial dimensions of infrared features and visible light features in the low-dimensional feature space to reduce computational overhead; Calculate the similarity matrix between cross-modal features; Construct a contrastive learning loss function to measure the similarity between infrared and visible light features; The contrast loss is optimized through back-propagation, the projection network parameters are updated, and the projection network's ability to extract modality-specific features is enhanced.

7. The multimodal fusion method for dynamic perception of thermal diffusion suppression according to claim 6, characterized in that: The contrastive learning enhancement module targets the shared features and specific features between the infrared image and the visible light modality modeled by the feature decoupling module, and splices them after channel compression and spatial alignment to obtain multi-scale features, which are enhanced by the spatial attention mechanism; and converts the multi-scale features into specific pixel values ​​to obtain multi-scale features in a unified format; at the same time, the shared features and specific features are modulated and further enhanced by the channel attention mechanism to obtain infrared features enhanced by contrastive learning. and visible light characteristics These two features are adaptively fused to obtain the fused features of the first stage.

8. The multimodal fusion method for dynamic perception of thermal diffusion suppression according to claim 7, characterized in that: The detail-aware adaptive fusion module adopts a detail-aware adaptive weighted fusion strategy to adaptively fuse multi-scale features in a unified format, and combines the fusion results of the first stage to generate a fused image, mapping the information in the high-dimensional feature space to the image pixel space; This module adopts the edge perception mechanism based on Sobel filtering and the dynamic weight allocation strategy to achieve effective fusion of multi-scale features and detail retention.

9. A multimodal fusion system for dynamic perception of thermal diffusion suppression, implemented based on the multimodal fusion method for dynamic perception of thermal diffusion suppression according to claim 1, characterized in that: include: Thermal diffusion suppression module and fusion network building module; The heat diffusion suppression module performs physical modeling on the diffusion of thermal targets in infrared images, solves the boundary blur problem caused by the heat diffusion effect in infrared images, reconstructs the blurred edges, and obtains an infrared image with heat diffusion suppressed; The fusion network construction module builds a fusion network to adaptively fuse the infrared image with thermal diffusion suppression and the visible light image to obtain a fused image.

Images