Infrared and visible image fusion method based on degradation perception and frequency integration
By using a degradation-sensing and frequency integration approach, the universality and generalization issues of infrared and visible light image fusion technology were addressed, enabling adaptive processing of complex degradation scenarios and improving image fusion quality and structural fidelity.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CHINA UNIV OF PETROLEUM (EAST CHINA)
- Filing Date
- 2025-08-28
- Publication Date
- 2026-06-09
AI Technical Summary
Existing infrared and visible light image fusion technologies lack universality and generalization capabilities, cannot effectively handle complex multi-source degradation phenomena, are difficult to extract discriminative structural spectra, and cannot automatically identify degradation patterns under unlabeled supervision.
A degradation-aware and frequency integration-based approach is adopted. Image phase and amplitude are obtained through fast Fourier transform and dynamic filtering. A nonlinear mapping module is used for channel stitching and fusion feature modulation. Combined with a degradation-aware expert prompting module, adaptive fusion is performed to generate a fusion prompt vector and enhance the image.
It significantly improves the quality and structural fidelity of fused images, effectively preserves key texture and geometric information, enhances the ability to retain edge details and high-frequency textures, and achieves effective modeling and robust adaptation to complex degraded scenes.
Smart Images
Figure CN121095079B_ABST
Abstract
Description
Technical Field
[0001] This invention discloses an infrared and visible light image fusion method based on degradation sensing and frequency integration, belonging to the field of image fusion technology. Background Technology
[0002] Infrared and visible light image fusion technology aims to integrate the thermal radiation features of infrared images with the texture and detail features of visible light images, and is widely used in nighttime surveillance, intelligent security, autonomous driving, and other fields. However, in real-world scenarios, image data is often affected by degradation factors such as blurring, low light, and sensor noise, severely reducing the quality and usability of the fused image. Traditional fusion methods rely on direct coupling of spatial domain features or fixed weight strategies, lacking the ability to model complex, multi-source degradation phenomena and failing to achieve adaptive enhancement. Furthermore, existing methods utilize frequency information in a relatively crude manner, making it difficult to extract discriminative structural spectra; in addition, most methods cannot automatically identify degradation patterns under unlabeled supervision, lacking universality and generalization ability. Summary of the Invention
[0003] The purpose of this invention is to provide an infrared and visible light image fusion method based on degradation sensing and frequency integration, so as to solve the problem that the infrared and visible light image fusion technology in the prior art lacks universality and generalization ability.
[0004] The infrared and visible light image fusion method based on degradation perception and frequency integration includes two branches. One branch involves acquiring an infrared image, inputting it into an encoder, performing a fast Fourier transform to obtain the phase, and then performing dynamic filtering. The other branch involves acquiring a visible light image, inputting it into an encoder, performing a fast Fourier transform to obtain the phase and amplitude, and then performing dynamic filtering on each. A nonlinear mapping module is used to concatenate the dynamic filtering results of the two phases to obtain a fused phase spectrum, which is then combined with the dynamic filtering result of the amplitude to perform an inverse fast Fourier transform to obtain the fused features.
[0005] The fused features are input into the degradation perception expert prompt module, which outputs a fused prompt vector. Then, two branches are generated. One branch performs convolution mapping on the fused prompt vector to obtain the scaling factor γ, and the other branch performs convolution mapping on the fused prompt vector to obtain the bias parameter β. The fused features are multiplied with the scaling factor and then added with the bias parameter to complete the modulation fused features. The modulated fused features are then input into the encoder to obtain the fused image.
[0006] Modulation fusion features include:
[0007] F f =γ·F′ f +β;
[0008] In the formula, F′ f For fusion features, Ff This refers to the fusion characteristics after modulation.
[0009] The encoder obtains the input image and sequentially performs feature extraction, downsampling, feature extraction, downsampling, feature extraction, downsampling, and feature extraction to obtain the features processed by the encoder.
[0010] The feature extraction is performed using a Transformer-based feature extraction block, and the downsampling process includes 3×3 convolution, ReLU activation function and 2×2 max pooling in sequence.
[0011] The nonlinear mapping module consists of two 1×1 convolutional layers and a Tanh activation function;
[0012] The convolutional mapping includes fully connected layers and convolutional layers, which project the cue vector from the semantic space to a parameter space that matches the number of feature channels.
[0013] Dynamic filtering of visible light images includes adjusting the infrared phase spectrum P ir A 1×1 convolutional layer is input to extract local spectral response features. These features are then nonlinearly mapped using the ReLU activation function. Global average pooling (GAP) is applied to these local spectral response features, which are then input into a multilayer perceptron (MLP) to generate corresponding channel attention weights W. Channel-level modulation is then used to enhance each spectral component in conjunction with W.
[0014]
[0015] In the formula, It is the P after dynamic filtering ir , ⊙ represents the dot product operation, Conv represents the convolution operation, and σ represents the activation function.
[0016] Dynamic filtering of infrared images includes adjusting the visible light amplitude spectrum A. vis Visible phase spectrum P vis A 1×1 convolutional layer is input to extract local spectral response features. These features are then nonlinearly mapped using the ReLU activation function. Global average pooling (GAP) is applied to these local spectral response features, which are then input into a multilayer perceptron (MLP) to generate corresponding channel attention weights W. Channel-level modulation is then used to enhance each spectral component in conjunction with W.
[0017]
[0018] In the formula, It is A after dynamic filtering vis , It is the P after dynamic filtering vis .
[0019] The degradation-aware expert suggestion module includes inputting fused features into an average pooling layer to obtain semantic vectors, and calculating the similarity W between the semantic vectors and the key vectors of each expert. n The expert response probability distribution P is obtained. exp A threshold for the minimum expert subset is set to filter the active expert subset, a low-rank dictionary is constructed, and the semantic vector and the low-rank dictionary are multiplied by a dot product to obtain the hint vector. The weighted aggregation is then used to obtain the fused hint vector.
[0020] Let the number of experts be N, and assign a set of key vectors to each expert to form the expert key set K:
[0021] K = {k1, ..., k} N};
[0022] In the formula, k1, ..., k N These are the N key vectors of K.
[0023] According to P exp The response probabilities are ranked, and the smallest subset of experts whose cumulative response probabilities exceed a set threshold is selected to form the active expert subset e. i .
[0024] For each e i Construct the corresponding low-rank cue dictionary D i :
[0025] D i =D+B i ;
[0026]
[0027] In the formula, B i For low-rank perturbation terms, U is a shared low-rank basis. For e i The transpose of the reconstructed weight matrix, D, is the basic hint dictionary shared by all experts.
[0028] The resulting hint vector includes:
[0029]
[0030] In the formula, α i It is e i The weight distribution, Softmax is the activation function, q is the semantic vector, p i It is the cue vector, α i,j M is the weight of the j-th suggestion vector of the i-th expert, where M is the maximum value of j, and D is the weight of the suggestion vector of the j-th expert. i [j] is the j-th suggestion vector in the low-rank suggestion dictionary corresponding to the i-th expert. It is D i The transpose of .
[0031] All cue vectors are weighted and aggregated according to their routing probabilities to obtain the fused cue vector P:
[0032] P = ∑P exp [i]·p i ;
[0033] In the formula, P exp [i] is the probability that the i-th expert in the degradation perception expert suggestion module is selected.
[0034] Compared with the prior art, the present invention has the following beneficial effects: The present invention significantly improves the fusion quality of infrared and visible light images, enhances the expressive power of structure-sensitive frequency bands, effectively preserves key textures and geometric information, and significantly improves the structural fidelity of the fused image; it realizes effective modeling and robust adaptation to complex degradation scenes, and realizes enhanced control of local degradation areas, thereby improving the ability to preserve edge details and high-frequency textures. Attached Figure Description
[0035] Figure 1 This is a flowchart of the technology of the present invention;
[0036] Figure 2 This is a schematic diagram of the encoder of the present invention;
[0037] Figure 3 This is a structural diagram of the degradation perception expert prompting module of the present invention;
[0038] Figure 4 This is the visualization result of removing the degradation perception expert prompt module;
[0039] Figure 5 It preserves the complete visualization results of the degradation perception expert prompt module. Detailed Implementation
[0040] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention are described clearly and completely below. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.
[0041] Infrared and visible light image fusion methods based on degradation sensing and frequency integration, such as Figure 1It includes two branches. One branch acquires an infrared image, inputs it into an encoder, performs a fast Fourier transform to obtain the phase, and performs dynamic filtering. The other branch acquires a visible light image, inputs it into an encoder, performs a fast Fourier transform to obtain the phase and amplitude, and performs dynamic filtering on each. The nonlinear mapping module concatenates the dynamic filtering results of the two phases to obtain a fused phase spectrum, and then performs an inverse fast Fourier transform with the dynamic filtering result of the amplitude to obtain the fused features.
[0042] The fused features are input into the degradation perception expert prompt module, which outputs a fused prompt vector. Then, two branches are generated. One branch performs convolution mapping on the fused prompt vector to obtain the scaling factor γ, and the other branch performs convolution mapping on the fused prompt vector to obtain the bias parameter β. The fused features are multiplied with the scaling factor and then added with the bias parameter to complete the modulation fused features. The modulated fused features are then input into the encoder to obtain the fused image.
[0043] Modulation fusion features include:
[0044] F f =γ·F′ f +β;
[0045] In the formula, F′ f For fusion features, F f This refers to the fusion characteristics after modulation.
[0046] The encoder is as follows Figure 2 The input image is obtained, and feature extraction, downsampling, feature extraction, downsampling, feature extraction, downsampling, feature extraction, and feature extraction are performed sequentially to obtain the features processed by the encoder.
[0047] The feature extraction is performed using a Transformer-based feature extraction block, and the downsampling process includes 3×3 convolution, ReLU activation function and 2×2 max pooling in sequence.
[0048] The nonlinear mapping module consists of two 1×1 convolutional layers and a Tanh activation function;
[0049] The convolutional mapping includes fully connected layers and convolutional layers, which project the cue vector from the semantic space to a parameter space that matches the number of feature channels.
[0050] Dynamic filtering of visible light images includes adjusting the infrared phase spectrum P ir A 1×1 convolutional layer is input to extract local spectral response features. These features are then nonlinearly mapped using the ReLU activation function. Global average pooling (GAP) is applied to these local spectral response features, which are then input into a multilayer perceptron (MLP) to generate corresponding channel attention weights W. Channel-level modulation is then used to enhance each spectral component in conjunction with W.
[0051]
[0052] In the formula, It is the P after dynamic filtering ir , ⊙ represents the dot product operation, Conv represents the convolution operation, and σ represents the activation function.
[0053] Dynamic filtering of infrared images includes adjusting the visible light amplitude spectrum A. vis Visible phase spectrum P vis A 1×1 convolutional layer is input to extract local spectral response features. These features are then nonlinearly mapped using the ReLU activation function. Global average pooling (GAP) is applied to these local spectral response features, which are then input into a multilayer perceptron (MLP) to generate corresponding channel attention weights W. Channel-level modulation is then used to enhance each spectral component in conjunction with W.
[0054]
[0055] In the formula, It is A after dynamic filtering vis , It is the P after dynamic filtering vis .
[0056] Degradation perception expert prompt module, such as Figure 3 This includes inputting the fused features into an average pooling layer to obtain a semantic vector, and calculating the similarity W between the semantic vector and each expert key vector. n The expert response probability distribution P is obtained. exp A threshold for the minimum expert subset is set to filter the active expert subset, a low-rank dictionary is constructed, and the semantic vector and the low-rank dictionary are multiplied by a dot product to obtain the hint vector. The weighted aggregation is then used to obtain the fused hint vector.
[0057] Let the number of experts be N, and assign a set of key vectors to each expert to form the expert key set K:
[0058] K = {k1, ..., k} N};
[0059] In the formula, k1, ..., k N These are the N key vectors of K.
[0060] According to P exp The response probabilities are ranked, and the smallest subset of experts whose cumulative response probabilities exceed a set threshold is selected to form the active expert subset e. i .
[0061] For each e i Construct the corresponding low-rank cue dictionary D i :
[0062] D i =D+B i;
[0063]
[0064] In the formula, B i For low-rank perturbation terms, U is a shared low-rank basis. For e i The transpose of the reconstructed weight matrix, D, is the basic hint dictionary shared by all experts.
[0065] The resulting hint vector includes:
[0066]
[0067] In the formula, α i It is e i The weight distribution, Softmax is the activation function, q is the semantic vector, p i It is the cue vector, α i，j M is the weight of the j-th suggestion vector of the i-th expert, where M is the maximum value of j, and D is the weight of the suggestion vector of the j-th expert. i [j] is the j-th suggestion vector in the low-rank suggestion dictionary corresponding to the i-th expert. It is D i The transpose of .
[0068] All cue vectors are weighted and aggregated according to their routing probabilities to obtain the fused cue vector P:
[0069] P = ∑P exp [i]·p i ;
[0070] In the formula, P exp [i] is the probability that the i-th expert in the degradation perception expert suggestion module is selected.
[0071] This invention was qualitatively compared with several mainstream fusion models on multiple test datasets. The invention effectively suppresses interference, restores structural details, and significantly improves local texture and overall contrast. In original images affected by low light or fog, ground or target outlines are almost indistinguishable, while the invention performs excellently in detail fidelity and brightness restoration. In samples with less degradation, the invention also achieved optimal visual effects, demonstrating good stability and generalization ability. Table 1 presents the quantitative evaluation results of the invention on four benchmark datasets. Although the ranking of some metrics fluctuates slightly across different datasets, the overall ranking is first across all datasets, fully demonstrating its stable cross-scene generalization ability.
[0072] Table 1. Quantitative results of existing methods on benchmark datasets
[0073]
[0074] Table 1 lists the abbreviations of various existing techniques. To further verify the performance of this invention under diverse degradation scenarios, it was combined with current mainstream image restoration and fusion methods to conduct a systematic comparative experiment on its anti-degradation performance. For different types of degradation, representative image restoration models were selected to specifically restore the source images before fusion: NeuralBR focuses on low-light image enhancement, DTR is used for raindrop removal under various severe weather conditions, AirNet integrates denoising and contrast enhancement functions, and IAT is responsible for exposure correction. All comparison methods used publicly available pre-trained models to ensure the fairness of the experiment.
[0075] This invention qualitatively compares with various mainstream restoration and fusion methods under different degradation scenarios. Other methods often suffer from details loss, texture blurring, or artifacts when facing complex degradation. In contrast, this invention can more effectively suppress noise and distortion caused by various degradations, significantly improving image clarity and structural integrity. Table 2 further verifies the robustness and superior performance of this invention in diverse degradation scenarios through quantitative indicators. Specifically, this invention achieves significant superiority in key indicators such as mutual information (MI) and visual information fidelity (VIF), fully demonstrating its ability to more effectively extract and retain key information from degraded images, improving the detail and overall quality of the fused image. Unlike traditional methods that rely on preset degradation models or fixed restoration procedures, this invention does not require explicit degradation type identification. It can dynamically perceive and adjust the fusion process for different degradation modes through an adaptive mechanism, thereby achieving a better trade-off between information preservation and image quality.
[0076] Table 2. Quantitative results of existing methods under different degradation scenarios
[0077]
[0078] In Table 2, the index row is an abbreviation for various existing technologies.
[0079] To evaluate the improvement effect of infrared-visible image fusion on multimodal target detection, experiments were conducted on the LLVIP dataset in a low-light street scene. The data was divided into 4000 training pairs, 450 validation pairs, and 163 test pairs to ensure fairness and reproducibility. Each fusion method corresponded to its own YOLOv5 detection model, and the training settings were unified. Performance was evaluated using metrics such as precision, recall, mAP@0.5, and mAP@0.5:0.95. The present invention showed superior performance in terms of target contour integrity and localization accuracy. Table 3 further presents the quantitative comparison results of each method on the LLVIP dataset. The present invention achieved mAP@0.5 of 0.99 and mAP@0.5:0.95 of 0.73, demonstrating excellent detection accuracy and overall robustness.
[0080] Table 3. Object detection results of various methods on the LLVIP dataset
[0081]
[0082] To visually demonstrate the improvement in feature discrimination capabilities of the degradation perception expert prompting module, t-SNE was used to perform visual analysis on the extracted features. Figure 4 The image shows the feature distribution after removing this module. Samples of different degradation types (noise, low contrast, raindrops, darkness, overexposure) are highly mixed in two-dimensional space, with blurred class boundaries and significant overlap, indicating poor feature discrimination. In contrast, Figure 5 To preserve the feature distribution after the degradation perception expert prompt module, similar sample points exhibit a tighter clustering, and different categories form a clearer cluster structure. Categories such as low contrast and darkness are significantly separated, fully demonstrating the effectiveness of this module in enhancing feature discriminability and helping the model to more accurately identify and adapt to different image degradation scenarios.
[0083] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.
Claims
1. An infrared and visible light image fusion method based on degradation sensing and frequency integration, characterized in that, It includes two branches. One branch acquires an infrared image, inputs it into an encoder, performs a fast Fourier transform to obtain the phase, and performs dynamic filtering. The other branch acquires a visible light image, inputs it into an encoder, performs a fast Fourier transform to obtain the phase and amplitude, and performs dynamic filtering on each. A nonlinear mapping module is used to concatenate the dynamic filtering results of the two phases to obtain a fused phase spectrum, and then performs an inverse fast Fourier transform with the dynamic filtering result of the amplitude to obtain the fused features. The fused features are input into the degradation-aware expert prompting module, which outputs a fused prompt vector. Then, two branches are generated. One branch performs a convolution mapping on the fused prompt vector to obtain a scaling factor. One branch performs a convolution mapping on the fused cue vector to obtain the bias parameters. The fusion features are multiplied by the scaling factor and then added to the bias parameter to complete the modulation fusion features, which are then input into the encoder to obtain the fused image. Modulation fusion features include: ； In the formula, As a feature of fusion, The fusion features are modulated; Dynamic filtering of visible light images includes adjusting the infrared phase spectrum. A 1×1 convolutional layer is input to extract local spectral response features. These features are then nonlinearly mapped using the ReLU activation function. Global average pooling (GAP) is applied to these local spectral response features, which are then input into a multilayer perceptron (MLP) to generate corresponding channel attention weights. , combined Each spectral component is enhanced using channel-level modulation: ； In the formula, It is after dynamic filtering , This represents the dot product operation. This represents the convolution operation. Indicates the activation function; Dynamic filtering of infrared images includes adjusting the visible light amplitude spectrum. Visible phase spectrum A 1×1 convolutional layer is input to extract local spectral response features. These features are then nonlinearly mapped using the ReLU activation function. Global average pooling (GAP) is applied to these local spectral response features, which are then input into a multilayer perceptron (MLP) to generate corresponding channel attention weights. , combined Each spectral component is enhanced using channel-level modulation: ； ； In the formula, It is after dynamic filtering , It is after dynamic filtering ; The degradation-aware expert suggestion module includes inputting fused features into an average pooling layer to obtain semantic vectors, and calculating the similarity between the semantic vectors and the key vectors of each expert. The probability distribution of expert responses was obtained. A threshold for the minimum expert subset is set to filter the active expert subset, a low-rank dictionary is constructed, and the semantic vector and the low-rank dictionary are multiplied by a dot product to obtain the hint vector. The weighted aggregation is then used to obtain the fused hint vector.
2. The infrared and visible light image fusion method based on degradation sensing and frequency integration according to claim 1, characterized in that, The encoder obtains the input image and sequentially performs feature extraction, downsampling, feature extraction, downsampling, feature extraction, downsampling, and feature extraction to obtain the features processed by the encoder. The feature extraction is performed using a Transformer-based feature extraction block, and the downsampling process includes 3×3 convolution, ReLU activation function and 2×2 max pooling in sequence. The nonlinear mapping module consists of two 1×1 convolutional layers and a Tanh activation function; The convolutional mapping includes fully connected layers and convolutional layers, which project the cue vector from the semantic space to a parameter space that matches the number of feature channels.
3. The infrared and visible light image fusion method based on degradation sensing and frequency integration according to claim 2, characterized in that, Set the number of experts to be Assign a set of key vectors to each expert to form an expert key set. : ； In the formula, yes of A number of key vectors.
4. The infrared and visible light image fusion method based on degradation sensing and frequency integration according to claim 3, characterized in that, according to The response probabilities are ranked, and the smallest subset of experts whose cumulative response probabilities exceed a set threshold is selected to form the active expert subset. .
5. The infrared and visible light image fusion method based on degradation sensing and frequency integration according to claim 4, characterized in that, For each Construct the corresponding low-rank cue dictionary : ； ； In the formula, For low-rank perturbation terms, To share the low-rank basis, for The transpose of the reconstructed weight matrix, It is a basic prompt dictionary shared by all experts.
6. The infrared and visible light image fusion method based on degradation sensing and frequency integration according to claim 5, characterized in that, The resulting hint vector includes: ； ； In the formula, yes The weight distribution It is an activation function. It is a semantic vector. It is a cue vector. It is the first The first expert The weights of each cue vector, yes The maximum value, It is the first The first expert's corresponding low-rank cue dictionary A prompt vector, yes The transpose of .
7. The infrared and visible light image fusion method based on degradation sensing and frequency integration according to claim 6, characterized in that, All cue vectors are weighted and aggregated according to their routing probabilities to obtain the fused cue vector. : ； In the formula, It is the first in the degradation perception expert prompt module The probability of an expert being selected.