A semantically guided wavelet domain infrared and visible light image fusion method and system

By proposing a semantically guided wavelet domain infrared and visible light image fusion method, a visual-language model is used to generate visible light bias ratio modulation low-frequency fusion weights. Combined with self-attention and cross-attention mechanisms, the problem of fixed fusion strategies and insufficient utilization of modal characteristic differences in existing methods is solved, and high-quality infrared and visible light image fusion is achieved.

CN123118380APending Publication Date: 2026-11-17CHANGCHUN UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611561224.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-10-10
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing infrared and visible light image fusion methods cannot dynamically switch fusion behavior according to scene semantic description, lack semantic adaptive capability, fail to effectively utilize the differences in information characteristics of images in different frequency subbands, and lack a decoupling mechanism between modal-specific information and cross-modal shared information, resulting in noise amplification and artifact generation.

Method used

A semantically guided wavelet domain infrared-visible image fusion method is adopted. The visible light bias ratio is generated by a pre-trained visual-language model, the low-frequency fusion weight function is modulated, the frequency domain fusion is performed by discrete wavelet transform, and the cross-task feature interaction and explicit separation of modality-specific information are achieved through self-attention and cross-attention mechanisms.

Benefits of technology

It achieves the switching of optimal fusion strategy under different scenarios, balances global perception quality and local detail fidelity, avoids the boundary artifact problem of hard threshold segmentation, and provides multimodal context for semantic tasks with fused features, ensuring the quality of fused images.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123118380A_ABST
    Figure CN123118380A_ABST
Patent Text Reader

Abstract

This invention relates to the field of image processing technology, specifically disclosing a semantically guided wavelet domain infrared and visible light image fusion method and system. The method includes fusing infrared and visible light images to obtain fused features, a first fused image; performing discrete wavelet transforms on the infrared and visible light images respectively to obtain low-frequency subbands and high-frequency subbands; fusing the low-frequency subbands using a modulated low-frequency fusion weight function to obtain a fused low-frequency subband; fusing the high-frequency subband pairs to obtain a fused high-frequency subband; performing inverse discrete wavelet transforms on the fused low-frequency and fused high-frequency subbands to obtain a second fused image; and mixing the first and second fused images to obtain a final fused image. This invention achieves an optimal balance between controllable global perception quality and stable local detail fidelity by performing semantic modulation on the low-frequency subbands while maintaining a fixed strategy on the high-frequency subbands.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of image processing technology, specifically to a semantically guided wavelet domain infrared-visible image fusion method and system. Background Technology

[0002] Infrared and visible light image fusion aims to complementarily combine the thermal radiation information captured by infrared sensors with the texture detail information captured by visible light sensors to generate high-quality fused images that possess both high target saliency and clear scene texture. This technology can be widely applied in fields such as security monitoring, autonomous driving, military reconnaissance, and remote sensing. However, existing infrared and visible light image fusion methods still suffer from the following technical problems:

[0003] Existing fusion methods are mainly divided into three categories:

[0004] Deep learning-based spatial domain fusion methods have fixed fusion strategies in the network parameters after training. During inference, they can only perform fusion according to the fixed patterns learned during training and cannot dynamically switch fusion behavior according to scene semantic description. For example, night scenes need to enhance infrared weights to highlight thermal targets, while daytime scenes need to prioritize the preservation of visible light textures. Existing methods cannot automatically adapt to such differentiated needs within the same model.

[0005] Fusion methods based on frequency domain multi-scale analysis employ preset fixed fusion rules (such as maximum absolute value, regional energy, etc.) and lack semantic adaptive capabilities.

[0006] The initial attempt at text-guided fusion, although it introduced text descriptions as conditional signals, only used text as an additional conditional feature input to the spatial domain network. It lacked an interpretable mapping from semantics to specific fusion parameters and failed to utilize the physical characteristics of different frequency bands in the frequency domain to achieve differentiated modulation.

[0007] Most existing deep learning methods perform pixel-level or feature-level fusion in the spatial domain, failing to take advantage of the differences in information characteristics of images in different frequency sub-bands: infrared thermal target information is concentrated in low-frequency components, while visible light texture details are concentrated in high-frequency components. Spatial domain methods cannot design targeted fusion strategies for these two frequency bands; infrared high-frequency components are accompanied by sensor noise, and equal-weighted fusion will amplify the noise.

[0008] Existing methods lack explicit feature interaction mechanisms between auxiliary tasks such as fusion and semantic segmentation, preventing the two tasks from working together to improve performance. At the same time, they lack decoupling mechanisms between modality-specific information and cross-modal shared information, leading to excessive suppression of information in a particular modality or the generation of cross-modal interference artifacts.

[0009] In summary, there is an urgent need for an infrared and visible light image fusion method that can interpretably correlate textual semantic descriptions with frequency domain fusion parameters to achieve semantic-driven frequency domain differential fusion. Summary of the Invention

[0010] The purpose of this invention is to provide a semantically guided wavelet domain infrared and visible light image fusion method and system to solve the problems mentioned in the background art.

[0011] To achieve the above objectives, the present invention provides the following technical solution:

[0012] A semantically guided wavelet domain infrared-visible image fusion method, the method comprising:

[0013] Acquire infrared and visible light images, as well as scene description text;

[0014] Multi-scale feature extraction is performed on the infrared image and the visible light image respectively to obtain infrared feature maps and visible light feature maps. Cross-modal feature fusion is performed on the infrared feature maps and the visible light feature maps to obtain fused features. The fused features are then decoded to obtain a first fused image.

[0015] The scene description text is encoded using a pre-trained visual-language model text encoder to obtain a semantic embedding vector. Based on the semantic similarity between the semantic embedding vector and at least two pre-stored reference text embeddings, a visible light bias ratio is generated.

[0016] In parallel, discrete wavelet transforms are performed on the infrared image and the visible light image respectively to obtain corresponding low-frequency sub-bands and high-frequency sub-bands. Based on at least one control parameter of the low-frequency fusion weight function modulated by the visible light bias ratio, the low-frequency sub-bands are fused using the modulated low-frequency fusion weight function to obtain a fused low-frequency sub-band. The high-frequency sub-bands are fused based on a bounded compression weighted fusion strategy to obtain a fused high-frequency sub-band. The fused low-frequency sub-band and the fused high-frequency sub-band are subjected to inverse discrete wavelet transforms to obtain a second fused image.

[0017] The first fused image and the second fused image are blended to obtain the final fused image.

[0018] As a further embodiment of the present invention, the multi-scale feature extraction process employs a dual-channel encoder, which includes an infrared encoder and a visible light encoder. The infrared encoder and the visible light encoder have the same network topology and their parameters are independent of each other.

[0019] As a further embodiment of the present invention, the at least two reference text embeddings include a thermal imaging bias reference embedding and a visible light bias reference embedding.

[0020] As a further embodiment of the present invention, the low-frequency fusion weighting function is an exponential function with the normalized infrared low-frequency coefficient as the independent variable.

[0021] As a further embodiment of the present invention, the control parameters include the transition steepness parameter of the exponential function and the infrared enhancement intensity parameter;

[0022] The transition steepness parameter is used to control the steepness of the transition from the cold zone to the hot zone in the low-frequency fusion weighting function, wherein the smaller the visible light bias ratio, the smaller the transition steepness parameter, and the larger the visible light bias ratio, the larger the transition steepness parameter.

[0023] The infrared enhancement intensity parameter is used to control the contribution intensity of infrared information in the fusion, wherein the smaller the visible light bias ratio, the larger the infrared enhancement intensity parameter, and the larger the visible light bias ratio, the smaller the infrared enhancement intensity parameter.

[0024] As a further embodiment of the present invention, the step of decoding the fusion features to obtain the first fused image includes:

[0025] Extract semantic discriminant features from the fused features;

[0026] By using a multi-head self-attention mechanism to perform cross-task global context interaction on the concatenated fusion features and semantic discriminative features, cross-enhanced fusion features and cross-enhanced semantic features are obtained.

[0027] Using a multi-head cross-attention mechanism, the fusion feature and the semantic discriminative feature are each used as queries, and the cross-enhanced fusion feature and the cross-enhanced semantic feature are used as keys and values, respectively, to perform targeted feature refinement, resulting in refined fusion features and refined semantic features.

[0028] Based on the feature subtraction operation guided by the spatial attention of the modal decoupling block, infrared-specific features and visible light-specific features are explicitly separated from the refined fused features;

[0029] The decoupled infrared-specific features and visible light-specific features are sent to a shared decoder for independent reconstruction. The corresponding original infrared and visible light images are used as self-supervised targets for reconstruction constraints to obtain the first fused image.

[0030] As a further aspect of the present invention, the specific steps of explicitly separating infrared-specific features and visible light-specific features from refined fused features based on the feature subtraction operation guided by the spatial attention of the modal decoupling block include:

[0031] For the refined fusion features, spatial attention is used to extract modal salient regions from the visible light feature map and the infrared feature map, respectively;

[0032] The infrared-specific feature is obtained by subtracting the spatial attention-weighted visible light component from the refined fusion feature.

[0033] The visible light-specific features are obtained by subtracting the infrared component weighted by the spatial attention from the refined fusion features.

[0034] As a further embodiment of the present invention, the first fused image and the second fused image are mixed, including: generating a spatially adaptive mixing weight based on the thermal saliency distribution of the infrared image, increasing the mixing weight of the second fused image in the thermally saliency region, and increasing the mixing weight of the first fused image in the cold background region.

[0035] Different dynamic range mapping parameters are used for regions with different thermal salience. A wide dynamic range mapping is used for thermally salience regions to release the internal contrast of infrared thermal targets, while a narrow dynamic range mapping is used for cold background regions to ensure accurate reproduction of visible light brightness.

[0036] High-frequency detail residuals are extracted from the original infrared and visible light images respectively, and either infrared detail residuals or visible light detail residuals are adaptively injected according to their thermal saliency distribution.

[0037] The present invention also provides a semantically guided wavelet domain infrared-visible image fusion system, the system comprising:

[0038] The image acquisition module is used to acquire infrared images, visible light images, and scene description text.

[0039] The feature extraction and fusion module is used to perform multi-scale feature extraction on the infrared image and the visible light image respectively to obtain an infrared feature map and a visible light feature map, perform cross-modal feature fusion on the infrared feature map and the visible light feature map to obtain a fused feature, and decode the fused feature to obtain a first fused image;

[0040] The visible light bias ratio generation module is used to encode the scene description text using a pre-trained visual-language model text encoder to obtain a semantic embedding vector, and generate a visible light bias ratio based on the semantic similarity between the semantic embedding vector and at least two pre-stored reference text embeddings.

[0041] The discrete wavelet transform module is used to perform discrete wavelet transform on the infrared image and the visible light image respectively to obtain corresponding low-frequency sub-bands and high-frequency sub-bands. Based on at least one control parameter of the low-frequency fusion weight function modulated by the visible light bias ratio, the low-frequency sub-bands are fused using the modulated low-frequency fusion weight function to obtain a fused low-frequency sub-band. The high-frequency sub-bands are fused based on a bounded compression weighted fusion strategy to obtain a fused high-frequency sub-band. The fused low-frequency sub-band and the fused high-frequency sub-band are subjected to inverse discrete wavelet transform to obtain a second fused image.

[0042] The image fusion module is used to blend the first fused image and the second fused image to obtain the final fused image.

[0043] Compared with the prior art, the beneficial effects of the present invention are: the present invention establishes for the first time a complete interpretable control chain from natural language scene description to frequency domain fusion parameters. Each link of the control chain has a clear physical meaning and mathematical traceability. During the inference stage, the user only needs to input the scene description text, and the low-frequency fusion parameters can be automatically modulated through semantic bias estimation to achieve the switching of the optimal fusion strategy under different scenarios without retraining the model parameters.

[0044] By performing semantic modulation in the low-frequency subband while maintaining a fixed strategy in the high-frequency subband, an optimal balance is achieved between the controllability of global perceptual quality (low frequency) and the stability of local detail fidelity (high frequency). Under semantic modulation, the exponential dynamic weighting function of the low-frequency subband achieves a smooth and progressive fusion transition from cold background to hot target, avoiding the boundary artifact problem of hard threshold segmentation.

[0045] Through a cascaded mechanism of self-attention and cross-attention (CTF+CTQ), bidirectional feature refinement is achieved for fusion and semantic segmentation tasks. The fused features provide multimodal context for the semantic task, while the semantic features, in turn, guide the fused features to focus on the semantic structure of the scene. CTQ's asymmetric cross-attention design, using the original features as the query, ensures that the refined output is a semantically enhanced version of the features rather than newly generated features, naturally possessing residual stability.

[0046] Explicit separation of modality-specific information is achieved through feature subtraction guided by spatial attention, without the need for complex decoupling methods such as mutual information loss, adversarial networks, or orthogonal constraints. It relies solely on a learnable, lightweight attention layer and basic tensor subtraction operations. The self-supervised reconstructed signal provides an effective learning driver for decoupling, eliminating the need for additional modality decoupling annotation data. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention.

[0048] Figure 1 The diagram shows the overall network structure of the semantically guided wavelet domain infrared-visible image fusion method provided in this embodiment of the invention.

[0049] Figure 2 This is a diagram of the internal structure of the cross-modal fusion block provided in an embodiment of the present invention.

[0050] Figure 3 The diagram shows the structure of the Cross-Task Interaction Module (CTF) and Cross-Task Refinement Module (CTQ) provided in this embodiment of the invention.

[0051] Figure 4 This is a structural diagram of a modal decoupling block provided in an embodiment of the present invention.

[0052] Figure 5 The complete structure of the wavelet semantic fusion head provided in the embodiments of the present invention.

[0053] Figure 6 The flowchart illustrates the three-stage adaptive post-processing provided in this embodiment of the invention.

[0054] Figure 7 The qualitative comparison results of six representative scenarios provided in the embodiments of the present invention are shown in the figure. Detailed Implementation

[0055] To make the technical problems to be solved, the technical solutions, and the beneficial effects of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present invention and are not intended to limit the present invention.

[0056] In this embodiment of the invention, a semantically guided wavelet domain infrared-visible image fusion method is provided, the method comprising:

[0057] Acquire infrared and visible light images, as well as scene description text;

[0058] Multi-scale feature extraction is performed on the infrared image and the visible light image respectively to obtain infrared feature maps and visible light feature maps. Cross-modal feature fusion is performed on the infrared feature maps and the visible light feature maps to obtain fused features. The fused features are then decoded to obtain a first fused image.

[0059] The scene description text is encoded using a pre-trained visual-language model text encoder to obtain a semantic embedding vector. Based on the semantic similarity between the semantic embedding vector and at least two pre-stored reference text embeddings, a visible light bias ratio is generated.

[0060] In parallel, discrete wavelet transforms are performed on the infrared image and the visible light image respectively to obtain corresponding low-frequency sub-bands and high-frequency sub-bands. Based on at least one control parameter of the low-frequency fusion weight function modulated by the visible light bias ratio, the low-frequency sub-bands are fused using the modulated low-frequency fusion weight function to obtain a fused low-frequency sub-band. The high-frequency sub-bands are fused based on a bounded compression weighted fusion strategy to obtain a fused high-frequency sub-band. The fused low-frequency sub-band and the fused high-frequency sub-band are subjected to inverse discrete wavelet transforms to obtain a second fused image.

[0061] The first fused image and the second fused image are blended to obtain the final fused image.

[0062] like Figure 1 As shown, in this embodiment, the core concept is:

[0063] Semantic encoding: The text encoder of a pre-trained vision-language model (such as CLIP) maps arbitrary scene descriptions to a semantic embedding space. This embedding space has the property that "semantically similar text embeddings are adjacent to each other." For example, the embedding vectors of "night," "thermal imaging," and "dark light" are adjacent to each other, while the embedding vectors of "daytime," "sunlight," and "outdoors" cluster in another region. This property is the basis for achieving zero-shot semantic generalization. Even if a specific scene word has not been seen during training, as long as its embedding vector is adjacent to a scene word that has been seen, the system can make a reasonable fusion strategy inference.

[0064] Semantic bias estimation: By pre-setting a pair of semantic anchors (thermal imaging reference embedding and visible light reference embedding) in the semantic space, the similarity between the query text embedding and the temperature-scaled Softmax of the two anchors is calculated, compressing the high-dimensional semantic embedding (e.g., 512-dimensional) into a continuous scalar. The introduction of temperature scaling addresses the issue of insufficient discriminative power caused by excessively small differences in the original cosine similarity (e.g., 0.53 vs 0.59) – specifically, the temperature coefficient. It amplifies the original similarity differences, making the bias of the Softmax output more explicit.

[0065] Parameter modulation: Visible light deflection ratio via dual paths (transition steepness) and infrared enhancement intensity This affects low-frequency fusion behavior. The modulation has clear physical interpretability; it is not generated by a black box but rather adjusts the mathematical form of the fusion function. For example, when a user inputs "night street with pedestrians," the visible light bias is... steepness of transition The weighting is reduced, which flattens the fusion weighting function curve and assigns higher infrared weights to more intermediate brightness areas, making the thermal features of pedestrians more prominent in the fusion result.

[0066] Fusion execution: Under the parameter control of semantic modulation, the exponential dynamic weight function calculates the infrared fusion weight pixel by pixel, realizing fine semantic control of "where infrared is incorporated and how much infrared is incorporated" in the low-frequency sub-band.

[0067] The fundamental difference from existing technologies lies in the fact that instead of "encoding text as conditional features input to a black-box network," this invention "utilizes semantic bias estimation results as explicit control signals to dynamically modulate specific parameters of low-frequency fusion in the frequency domain, achieving traceable and interpretable causal control from semantics to fusion behavior." This difference elevates the role of visual-language models such as CLIP in this invention from "feature extractor" to "fusion strategy controller." Text semantics is not merely another input data source for the network, but becomes the control signal source for the frequency domain fusion process.

[0068] In a preferred embodiment of the present invention, the step of decoding the fusion features to obtain the first fused image includes:

[0069] Extract semantic discriminant features from the fused features;

[0070] By using a multi-head self-attention mechanism to perform cross-task global context interaction on the concatenated fusion features and semantic discriminative features, cross-enhanced fusion features and cross-enhanced semantic features are obtained.

[0071] Using a multi-head cross-attention mechanism, the fusion feature and the semantic discriminative feature are each used as queries, and the cross-enhanced fusion feature and the cross-enhanced semantic feature are used as keys and values, respectively, to perform targeted feature refinement, resulting in refined fusion features and refined semantic features.

[0072] Based on the feature subtraction operation guided by the spatial attention of the modal decoupling block, infrared-specific features and visible light-specific features are explicitly separated from the refined fused features;

[0073] The decoupled infrared-specific features and visible light-specific features are sent to a shared decoder for independent reconstruction. The corresponding original infrared and visible light images are used as self-supervised targets for reconstruction constraints to obtain the first fused image.

[0074] In this embodiment, the multi-scale feature extraction steps of the dual-channel encoder are as follows:

[0075] Two encoders with identical structures but independent parameters are used to process the infrared images. And the brightness channel of the visible light image Multi-scale feature extraction is performed to obtain infrared feature maps at multiple scales. and visible light feature map The infrared encoder and the visible light encoder have the same network topology but their parameters are independent (non-weight sharing), which allows the two encoders to learn modality-specific low-level feature extraction strategies: the infrared encoder learns a filter that is sensitive to the smooth distribution of thermal radiation, and the visible light encoder learns a filter that is sensitive to the texture direction, thus maintaining modality independence from the source of feature extraction.

[0076] Each encoder contains multiple EBlocks, and each EBlock is composed of stacked residual blocks (ResBlocks). Bilinear interpolation downsampling is used instead of convolution with a stride of 2 for spatial size compression to reduce the number of parameters and avoid checkerboard artifacts.

[0077] The encoder's specific structure and parameter configuration:

[0078] The encoder employs a multi-level downsampling pyramid structure, mapping the input single-channel image to a multi-scale feature pyramid. Each encoder consists of a feature extraction convolutional layer, an encoding block (EBlock), and a downsampling layer (DownSample) concatenated. Infrared encoder ( ) and visible light encoder ( They have the exact same topology, but their parameters are completely independent (without weight sharing) to capture the underlying statistical properties unique to each mode.

[0079] In this embodiment, the encoder employs a three-stage processing flow. For the input image... :

[0080] First stage: The initial convolutional layer maps the single-channel input to a basic number of channels. Subsequently through (Depend on Feature refinement is performed on a stack of ResBlocks.

[0081] ;

[0082] Level Two: (This is incomplete and likely refers to a specific level or category.) Apply convolution operation (increase the number of channels from 32 to 64), then After refining, the space size is reduced by half using DownSample:

[0083] ;

[0084] Level 3: [This is incomplete and likely refers to a specific level or category.] Apply convolution operation (increase the number of channels from 64 to 128), then After refinement, the space size is further compressed using DownSample:

[0085] ;

[0086] DownSample is implemented by downsampling via bilinear interpolation followed by... Convolution preserves channel dimension:

[0087] ;

[0088] In the formula, Indicates the input feature map; This indicates a bilinear interpolation downsampling operation; This indicates a convolutional layer with a kernel size of 1×1; This indicates bilinear interpolation upsampling with a scaling factor of 0.5.

[0089] This embodiment chooses bilinear interpolation downsampling instead of convolution with a stride of 2 because: the former reduces the number of parameters while avoiding checkerboard artifacts that stride-2 Conv may introduce; and the smoothness of bilinear interpolation helps to preserve the spatial continuity of infrared thermal targets during downsampling.

[0090] Basic building block (ResBlock): EBlock consists of It is composed of stacked residual blocks (in this embodiment) (This value can be adjusted to other values ​​depending on computing resources). Each ResBlock contains two layers. Convolution with identity residual connection:

[0091] ;

[0092] In the formula, Indicates the input feature map; Represents a residual block; This indicates a convolutional layer with a kernel size of 3×3; This represents a 3×3 convolutional layer without an activation function.

[0093] Two layers The effective receptive field after convolution stacking is The first convolutional layer is followed by ReLU activation to introduce nonlinear transformation capability, while the second layer does not have an activation function to ensure the numerical additivity of the residual connections. The identity mapping path of the residual connections allows gradients to propagate directly to shallow layers, effectively alleviating the gradient vanishing problem in deep networks.

[0094] Infrared encoders and visible light encoders use independent parameters rather than weighted algorithms, based on the following three levels of systemic considerations:

[0095] At the underlying statistical level: infrared images exhibit a smooth thermal radiation distribution, with grayscale values ​​primarily determined by object temperature and emissivity, resulting in blurred edges and a low signal-to-noise ratio; visible light images, on the other hand, contain rich reflective textures and high-frequency details. If weights are shared, the network will be forced to learn a compromise feature extraction strategy, failing to fully capture modality-specific information.

[0096] At the convolution kernel learning level: parameter independence makes infrared encoders tend to learn Gaussian-like filters with a large spatial smoothing range to suppress strip noise and fixed-pattern noise of infrared sensors; visible light encoders tend to learn Gabor-like edge detection kernels with orientation selectivity to efficiently capture spatial orientation information.

[0097] Feature diversity level: Two independent encoders provide two complementary "views" for the fusion network. Each view extracts the most discriminative information from its own modality, providing richer feature diversity for subsequent cross-modal fusion.

[0098] The above design maintains modal independence from the source of feature extraction (encoder stage), laying a feature foundation for subsequent modal decoupling (FDB) and differential fusion.

[0099] The steps for cross-modal feature fusion are as follows:

[0100] Infrared and visible light feature maps are fused at multiple scales using a CrossModalFusionBlock (FBB). The core of FFB lies in its learnable dual-channel attention mechanism, which generates mode-specific channel recalibration weights for each modality. In the infrared features, channels representing thermal targets receive higher fusion weights than background noise channels, while in the visible light features, channels representing texture structures are preferentially preserved. The recalibrated two-modality features are then concatenated and subjected to convolutional dimensionality reduction and projection to obtain fused features at multiple scales. .

[0101] The internal structure of FFB is as follows Figure 2 As shown, the CrossModalFusionBlock (FFB) is the core unit for feature fusion. In this embodiment, there is one independent FFB instance at each of the three scales. , , The number of input and output channels is the same.

[0102] While directly concatenating infrared and visible light features along the channel dimension and performing convolution (early fusion) is simple, it ignores the differences in channel importance between modes. In infrared features, channels representing thermal targets should receive higher fusion weights than background noise channels, and in visible light features, channels representing texture structures should be preferentially preserved. FFB learns the mode-specific channel importance distribution end-to-end through a learnable dual-channel attention mechanism.

[0103] Calculation process: Let the input infrared feature be... Visible light characteristics are ;

[0104] Channel splicing and dual-path shallow fusion: splicing two modal features along the channel dimension into Two sets of three-layer convolutional sequences with identical structures and independent parameters are fed into each other to extract two complementary shallow fusion modes:

[0105] ;

[0106] ;

[0107] in, and These are two parameter-independent three-layer convolutional sequences. Each layer has an output channel count of 1. (The middle layer will be placed first) Compress to The dual-path design enables the network to learn two complementary, fused perspectives: The focus is on generating recalibrated weights for infrared features. The focus is on generating recalibrated weights for visible light features.

[0108] Channel attention weight generation: Applying different weights to the two sets of shallow fusion features respectively. Convolution (cross-channel linear combination) generates a modality-specific spatial-channel attention weight map:

[0109] ;

[0110] In the formula, and These represent the shallow fusion features of the first and second paths, respectively. Channel attention weight map representing infrared modes; Channel attention weight map representing visible light modes; Let represent a tensor space of size C×H×W in the real number field.

[0111] Convolution performs a linear combination along the channel dimension, and its parameters implicitly learn during training which channels contribute more to the fusion quality. The weight map preserves the spatial dimension, so attention is spatially adaptive, and the same channel can be assigned different importance at different spatial locations.

[0112] Channel-level recalibration: Element-wise multiplication is performed between the generated weight map and the original modal features.

[0113] , ;

[0114] In the formula, Represents infrared feature map; This represents a visible light characteristic map.

[0115] After recalibration, the amplitude of channels related to thermal targets in the infrared features is enhanced and the noise channels are suppressed; in the visible light features, channels with rich textures are preserved and smooth areas are appropriately attenuated.

[0116] Stitching and Projection: The recalibrated features of the two modalities are stitched together, through... Convolution projection to target number of channels :

[0117] ;

[0118] FFB's design differs fundamentally from the following three common fusion strategies:

[0119] Compared to simple feature concatenation (Concat+Conv), which assumes all channels are equally important in the fusion, FFB breaks this assumption by learning channel attention weights. Compared to element-wise addition, which forces infrared and visible light features to align on the same channel and ignores the differences in channel correspondence between modalities, FFB solves this problem with its dual-path independent channel weight generation path.

[0120] The steps for assisted semantic feature extraction and cross-task bidirectional interaction are as follows:

[0121] like Figure 3 As shown, semantic discriminative features are extracted from the fused features using auxiliary segmentation blocks (SFBs). .

[0122] Subsequently, through the cascading of the CrossTask Module (CTF) and the CrossTask Refinement Module (CTQ), bidirectional information interaction between fused features and semantic features is achieved.

[0123] CTF employs a multi-head self-attention mechanism to model the global context of the concatenated fused features and semantic features. Before computation, CTF adaptively pools the feature maps to compress them to a preset compact spatial size, thereby reducing the computational complexity of self-attention from... The computation is reduced to a manageable scale; after completion, the results are upsampled to restore the original spatial size. Through self-attention, fusion, and semantic tokens interact equally in the attention space, cross-enhanced fusion features and cross-enhanced semantic features are generated.

[0124] The CrossTaskModule (CTF) establishes global contextual interaction between fused features and segmented features through a multi-head self-attention (MSA) mechanism.

[0125] The design of CTF stems from a deep understanding of the potential conflict between the optimization goals of fusion and semantic segmentation tasks. Specifically, the optimization goal of fusion tasks is to maximize the preservation of texture details and the enhancement of the saliency of hot targets, and its feature representation tends to emphasize low-level visual information such as local texture gradients, edge directions, and brightness contrast. The optimization goal of semantic segmentation tasks is to maximize the semantic consistency of pixel-level category discrimination, and its feature representation tends to emphasize high-level semantic information such as category prototypes, semantic boundaries, and global context. The feature representations of the two tasks may conflict in gradient direction. For example, in order to enhance edge contrast, the fusion branch may amplify the gradient difference between infrared and visible light at the target boundary, while the segmentation branch, in order to achieve smooth semantic boundaries, tends to suppress gradient changes within the same semantic region. If the features of the two tasks are optimized separately in independent spaces, this inherent conflict will lead to: insufficient semantic discriminative power in the fused features, making it difficult to distinguish foreground targets from background regions; and loss of texture detail information in the segmentation features, resulting in decreased boundary localization accuracy.

[0126] CTF establishes a cross-task global context interaction mechanism by performing multi-head self-attention computation in the joint sequence space, effectively mitigating the aforementioned conflicts. Its core mechanism lies in the fact that the fusion token and the segmentation token participate in the interaction equally in the Query-Key-Value space of self-attention. The attention output of each token is a weighted sum of the values ​​of all tokens (from both tasks), with the weights determined by the dot product similarity between the query and all keys. This mechanism enables: the fusion token to adaptively extract high-level semantic information from the segmentation features based on its semantic similarity with the segmentation token, enhancing its ability to distinguish foreground / background regions; the segmentation token to supplement multimodal texture context from the fusion features based on its texture similarity with the fusion token, improving boundary localization accuracy; and the features of the two tasks achieve bidirectional information flow in the attention space, with the fusion features providing multimodal low-level information support for the segmentation features, and the segmentation features providing semantic structure guidance for the fusion features, transforming their potentially conflicting relationship into a synergistic enhancement relationship.

[0127] The computational complexity of standard self-attention versus sequence length It is proportional to the square of ( ,in (where the feature dimension is O(n), and the storage complexity of the attention matrix is ​​O(n). In high-resolution feature maps, The number of features can reach tens of thousands to hundreds of thousands, and directly calculating self-attention would lead to memory overflow. CTF addresses this contradiction through the following strategy: adaptive average pooling compresses feature maps of arbitrary resolution to a fixed, compact size. ( ), reducing complexity from Reduced to The fixed pooling size design enables CTF to handle any input resolution without needing to adjust the network structure for different input sizes.

[0128] In this embodiment, different pooling target sizes are used for the three scales to match their respective feature resolution levels: Scale 1 ( ) is used for global context modeling at the finest scale; scale 2 ( ) is used for medium-scale interactions; Scale 3 ( This is used for the coarsest-scale interactions. After pooling compression, the total attention computation across the three scales is orders of magnitude lower than that calculated directly at the original resolution.

[0129] CTF operates independently at three scales. Taking scale 1 as an example:

[0130] Fusion features and segmentation features Each sample was downsampled to an adaptive average pooling level. And apply batch normalization (BatchNorm):

[0131] ;

[0132] ;

[0133] Flatten the two feature spaces into sequences and concatenate them along the sequence dimensions:

[0134] ;

[0135] before sequence Each token comes from a fusion feature, after which... Each token comes from the segmentation features. .

[0136] Will Mapped to Query, Key, Value triples using independent linear projection matrices:

[0137] ;

[0138] In the formula, Indicates the features of the concatenated sequence; These represent the linear projection matrices of Query, Key, and Value, respectively.

[0139] For each attention head, a scaled dot product attention is calculated. In this embodiment, the number of attention heads is set to a preset number. (In this embodiment, the exemplary value is 8):

[0140] ;

[0141] In the formula, This represents the output of the i-th attention head; This represents the feature dimension of each attention head.

[0142] After all heads are spliced ​​together, the output projection is as follows:

[0143] ;

[0144] In the formula, This represents the output characteristics of multi-head self-attention; This represents the output of all attention heads; This indicates the output projection matrix.

[0145] Since Q, K, and V all originate from the same concatenation sequence, the attention matrix... Each element in Characterizing the first The token is paired with the first The level of attention given to each token is independent of whether the two tokens originate from the same task. This enables true bidirectional information flow between fusion and splitting.

[0146] Feature separation and residual superposition. Layer normalization (LayerNorm) is applied to the attention output, and the fused enhancement features and segmentation enhancement features are separated after reshaping. Each is processed through an MLP (Multi-Level Processing). -ReLU- Generate residual correction terms, then perform bilinear upsampling back to the original feature size:

[0147] ;

[0148] ;

[0149] In the formula, This represents the fusion features after cross-enhancement; This indicates the operation of bilinear upsampling to the original spatial size (H×W); A multilayer perceptron representing a fusion branch; [,0,:,:,:] represents the fusion token portion extracted from the attention output. This represents the segmentation features after cross-enhancement; This represents a multilayer perceptron with split branches; ,1,:,:,:] represents the segmented token portion extracted from the attention output.

[0150] CTQ employs a multi-head cross-attention mechanism, using the original task features as the query and CTF cross-enhanced features as the key and value, to refine targeted features. The query originates from the original features rather than learnable embeddings. When cross-attention fails to capture useful cross-task associations, the output naturally degenerates into an identity mapping of the original features; when cross-attention discovers meaningful associations, the output is refined with semantic guidance based on the original features. The refined features are then stacked using residuals (…). , This indicates the characteristics after CTQ refinement. The ellipsis represents the original task features; the ellipsis represents the residual correction term calculated by CTQ cross-attention. Combined with the original features, this provides stability during the initial training phase and robustness to unseen scenarios during inference.

[0151] After the above interactive refinement, refined fusion features were obtained respectively. and refined semantic features The fusion features provide multimodal contextual information for semantic tasks, while the semantic features in turn guide the fusion features to focus on the semantic structure of the scene, thus improving the synergy between the two tasks.

[0152] The CrossTask Refinement Module (CTQ) employs a multi-head cross-attention mechanism to perform targeted feature query refinement based on the global interaction of CTF. The cascaded structure of CTF and CTQ is as follows: Figure 3 As shown.

[0153] CTF enables all tokens to interact equally through self-attention, with fused and segmented features serving as both queries and key / value pairs. CTQ employs an asymmetric design: the original features (fused or segmented) act as queries, while the cross-enhanced features output by CTF serve as keys and values. This means that the query features "question" the key / value features, selectively extracting useful information from them.

[0154] Calculation process (in fusion branch) For example): Let As a feature of the original fusion, This refers to the cross-enhancement features output by CTF.

[0155] Will Downsampling is performed using adaptive average pooling. ( (Already at this size), flatten into sequence format: , .

[0156] In the formula, Indicates the original fusion characteristics; Indicates CTF cross-enhancement features; and These represent the format characteristics of the flattened sequence.

[0157] The query comes from the original features, and the key and value come from CTF cross-enhanced features:

[0158] ;

[0159] Cross-attention calculation:

[0160] ;

[0161] In the formula, The output features representing cross-attention, This represents the dimension of each attention head.

[0162] After restoring the spatial dimensions through layer normalization and bilinear upsampling, the original features are superimposed as residuals.

[0163] ;

[0164] The key role of residual connections is that when cross-attention fails to capture useful cross-task associations (attention weights tend to be evenly distributed), the output naturally degenerates into the original features; when cross-attention discovers meaningful associations, the output is enhanced with semantically guided fine-tuning based on the original features.

[0165] Branching It operates in a completely symmetrical manner. The CTQ parameters for fusion and splitting are independent to accommodate the different emphases of information requirements in the two tasks.

[0166] The cascaded structure of CTF and CTQ is analogous to the encoder-decoder architecture in the standard Transformer, but there are the following key differences: CTQ's query is not a learnable embedding vector, but the original feature itself. This makes CTQ's output a "semantically enhanced version of the original feature" rather than a "sequence generated from zero", which is naturally suitable for feature refinement rather than generation tasks. Residual stacking ensures that the output degenerates into an identity mapping of the original feature when cross attention fails, providing stability in the early stages of training and robustness to unseen scenarios during inference.

[0167] As a preferred embodiment of the present invention, the step of explicitly separating infrared-specific features and visible light-specific features from refined fused features based on the feature subtraction operation guided by the spatial attention of the modal decoupling block specifically includes:

[0168] For the refined fusion features, spatial attention is used to extract modal salient regions from the visible light feature map and the infrared feature map, respectively;

[0169] The infrared-specific feature is obtained by subtracting the spatial attention-weighted visible light component from the refined fusion feature.

[0170] The visible light-specific features are obtained by subtracting the infrared component weighted by the spatial attention from the refined fusion features.

[0171] In this embodiment, as Figure 5 As shown, using a Modal Disentanglement Block (FDB), infrared-specific features and visible light-specific features are explicitly separated from the refined fused features through a feature subtraction operation guided by spatial attention.

[0172] The basic principle of FDB is: Refining and fusing features. It can be regarded as infrared-specific information Visible light-specific information and cross-modal information sharing The three are superimposed. Spatial attention locates modality-specific salient regions (texture, edges, structure) from visible light features. The visible light component weighted by spatial attention is subtracted from the fused features, and the remainder is the infrared-specific feature (approximately equal to...). Symmetrically, the spatially attention-weighted infrared component is subtracted from the fused features to obtain the visible light-specific features. This subtraction decoupling strategy relies solely on a learnable, lightweight attention layer and basic tensor subtraction operations, without introducing mutual information loss terms, adversarial networks, or orthogonal constraints.

[0173] The Spatial Attention Layer (SALayer) is the core computational unit of FDB, employing a lightweight, compression-excitation structure.

[0174] ;

[0175] in The channel compression ratio (a configurable preset parameter; in this embodiment, the exemplary value is...) This value can be determined based on the channel dimension. (And flexible adjustment of computing resource constraints). Change the number of channels from Compress to , Change the number of channels from Restore to After activation by the Parametric Modified Linear Unit (PReLU), it is activated by the sigmoid activation function. (Sigmoid) normalizes the attention weights to .

[0176] SALayer uses PReLU instead of the standard ReLU as the activation function between channel compression and recovery. The reason is that ReLU sets all neurons to zero on the negative half-axis, which may cause some neurons in the compressed low-dimensional representation to become permanently inactive, losing the information needed for subsequent channel recovery. PReLU retains a learnable slope parameter on the negative half-axis, allowing for moderate propagation of negative features and effectively avoiding information loss in the bottleneck layer.

[0177] FDB contains two branches with independent parameters:

[0178] (Infrared-specific information decoupling): Decoupling visible light feature maps Apply spatial attention to extract salient regions (texture, edges, structure), and subtract these from the refined fused features:

[0179] ;

[0180] (Decoupling of visible light-specific information): Infrared feature maps Apply spatial attention to extract salient regions (heat sources, targets), and subtract them from the refined fused features:

[0181] ;

[0182] The intuitive principle of this design can be understood from the perspective of signal decomposition: refined fusion features. It can be viewed as a superposition of infrared-specific information, visible light-specific information, and cross-modal shared information. Spatial attention. The salient region in the visible light domain was extracted, and after subtracting this part from the fused features, the residual... It is approximately equal to a shared structure that superimposes infrared-specific information and cannot be interpreted by visible light attention. The technical advantage of the "subtraction decoupling" strategy is that it only relies on the attention mapping of two single modes, does not require explicit modeling of complex intermodal interaction relationships, and does not require additional modal decoupling annotation data.

[0183] To further clarify the theoretical basis of the FDB decoupling mechanism, this invention provides the following analysis from the perspective of information projection and feature interpretability.

[0184] From the perspective of information projection, spatial attention mapping This can be understood as using prior information from the visible light modality to estimate the components in the fused features that are "explainable by the visible light modality". Specifically, the spatial attention layer learns the salient region distribution of the visible light features to generate a spatial-channel adaptive weight map, which represents "how much information at each spatial-channel location of the fused feature can be explained by the feature patterns of the visible light modality". This estimation process is equivalent to performing a non-orthogonal projection of the fused features onto the feature subspace of the visible light modality.

[0185] From the perspective of feature interpretability, Constituting fusion characteristics An interpretable component is defined as "the portion that can be explained by visible light modal features." Subtracting this interpretable component is not a simple numerical subtraction, but rather the removal of the visible light modal interpretable subspace component from the joint feature space. The remaining component after the subtraction operation... Two types of information are preserved: information specific to the infrared mode (such as thermal radiation distribution patterns), which cannot be explained by the characteristic patterns of the visible light mode; and the part of the cross-modal shared information that cannot be explained by visible light attention alone (such as geometric structures that appear in two modes at the same time, but whose multimodal joint expression cannot be fully covered by the projection of a single mode).

[0186] Based on the above theoretical framework, the decoupling process of FDB can be rigorously described as modal residual separation: fusing features... Decomposed into: This represents the interpretable components (projection components) of the visible light modes. Indicates infrared modal residuals (the remaining information after removing the interpretable components of visible light); Indicates the interpretable components (projected components) of the infrared mode. This represents the residual of the visible light mode (the remaining information after removing the interpretable infrared components).

[0187] The significance of this theoretical framework lies in the following: it elevates modal decoupling from an empirical "feature subtraction" to a "removal of interpretable projections" operation with rigorous mathematical interpretation, thus strengthening the creative justification for the technical solution; it explains why FDB does not require mutual information loss or adversarial training, because the decoupling process essentially involves gradually removing interpretable subspace components of non-target modalities from the fused features, rather than constraining modal independence at the loss function level; and it provides theoretical support for the lightweight design of FDB (requiring only two learnable attention layers), since the decoupling capability mainly comes from the computational paradigm of "interpretable component estimation-removal," rather than the complex mapping of a large number of parameters.

[0188] The decoupled infrared-specific features and visible light-specific features are fed into a shared decoder for independent reconstruction. The corresponding original infrared and visible light images are used as self-supervised targets for reconstruction constraints, providing effective learning driving signals for decoupling.

[0189] The decoupled features are then fed into a shared decoder for independent reconstruction.

[0190] ;

[0191] In the formula, and These represent the output images reconstructed by the infrared decoder and the visible light decoder, respectively. Indicates a shared decoder; and These represent the infrared mode decoupling characteristics and the visible light mode decoupling characteristics, respectively.

[0192] During training, Using the raw infrared image as the monitoring target, The original visible light brightness channel is used as the supervision target. This self-supervised mechanism requires no additional annotation and can provide an effective learning signal for modal decoupling using only the input image itself.

[0193] The decoder employs a multi-level upsampling pyramid structure symmetrical to the encoder to reconstruct a spatial domain image from the refined multi-scale features. The decoder is reused in TWFNet in three modes: fusion decoding mode, input refined fusion features... Output brightness image Segmentation decoding mode, input refined segmentation features Output semantic segmentation map Decoupled reconstruction mode: Input the decoupled modal features and output the reconstructed image.

[0194] The decoder restores resolution step by step, starting from the coarsest scale. Each stage first undergoes non-linear refinement using DBlock (stacked ResBlock), then uses transposed convolution and independent bilinear upsampling modules to double the spatial size and halve the number of channels.

[0195] ;

[0196] In the formula, This represents the upsampled output characteristics of the l-th level decoder block; This represents the input features of the l-th level decoder block; This indicates a 2x upsampling operation; This represents a transposed convolution with a kernel size of 3×3 and a stride of 1. This represents the l-th level decoder block.

[0197] Multi-scale skip connections: The decoder fuses cross-scale information from the encoder at each stage through a Skip module. The Skip module receives encoder features at three scales (matching features at the current scale, upsampled alignment of features at a coarser level, and downsampled alignment of features at a finer level), concatenates them along the channels, and then passes them through the Skip module. Convolutional sequence fusion. The Skip output and upsampled features are concatenated along the channels and then processed... Convolutional fusion enables multi-granular integration of cross-scale information.

[0198] The decoder has three multiplexing modes and parameter sharing: all three modes share all decoder parameters (including DBlock, upsampling layer, Skip module, and intermediate convolutional layer), except that the splitting mode adds an additional independent parameter at the end. Convolutional classifier. The advantages of parameter sharing are: reduced total number of parameters (saving approximately 2 / 3 of the decoder parameters); and forcing the decoder to learn a general "feature". The image "mapping" capability, rather than overfitting to the input distribution of a specific task; the gradients of the segmentation and reconstruction tasks are backpropagated to the fusion encoder through shared parameters, forming an implicit multi-task regularization effect.

[0199] In a preferred embodiment of the present invention, the at least two reference text embeddings include a thermal imaging bias reference embedding and a visible light bias reference embedding.

[0200] The low-frequency fusion weighting function is an exponential function with the normalized infrared low-frequency coefficient as the independent variable.

[0201] The control parameters include the transition steepness parameter of the exponential function and the infrared enhancement intensity parameter;

[0202] The transition steepness parameter is used to control the steepness of the transition from the cold zone to the hot zone in the low-frequency fusion weighting function, wherein the smaller the visible light bias ratio, the smaller the transition steepness parameter, and the larger the visible light bias ratio, the larger the transition steepness parameter.

[0203] The infrared enhancement intensity parameter is used to control the contribution intensity of infrared information in the fusion, wherein the smaller the visible light bias ratio, the larger the infrared enhancement intensity parameter, and the larger the visible light bias ratio, the smaller the infrared enhancement intensity parameter.

[0204] In this embodiment, the complete structure of the wavelet semantic fusion head is as follows: Figure 4 As shown, the wavelet semantic fusion head is used to perform frequency domain fusion of the brightness channels of the original infrared image and the visible light image. This highlights the essential difference between the present invention and the prior art: instead of simply using visual-language models such as CLIP to extract text features to assist fusion, the present invention uses semantic bias estimation results as control signals to dynamically modulate the core parameters of the frequency domain fusion process.

[0205] infrared images and visible light brightness image Multi-level Discrete Wavelet Transform (DWT) is performed. In a preferred embodiment, a two-level decomposition is performed using the Daubechies-4 (db4) wavelet basis: the first-level DWT decomposes the input image into low-frequency subbands. and three directional high-frequency sub-bands ,in This is the first-level low-frequency sub-band; (Horizontal edge) (Vertical edge) (Diagonal edge) is the first-level high-frequency sub-band, and the spatial dimensions are all... Second-level DWT pair Further decomposition yields ,in This is the second-level low-frequency sub-band. This is the second-level high-frequency sub-band, and all dimensions are... The above decomposition is performed on both the infrared and visible light inputs. The reason for choosing the db4 wavelet basis is that its tight support and orthogonality ensure the numerical stability of the forward DWT and the inverse IDWT; the fourth-order vanishing moment can effectively capture the singularity of the edges of infrared thermal targets and the boundaries of visible light textures.

[0206] Low-frequency subband It includes the global brightness distribution and overall contrast at the coarsest scale of the image. In infrared images, Characterizing the overall temperature distribution of the thermal target, the heat source region exhibits a high coefficient value, while the cold background is close to zero. In visible light images, Characterizes the overall lighting and tonal structure of the scene. The fusion directly determines the global brightness and thermal target saliency of the fused image, thus making it the optimal site for semantic modulation.

[0207] (size : Located at the mid-frequency scale, it contains structural edges of medium size (building outlines, road boundaries, and boundaries of large object shapes).

[0208] (size Infrared images are located at the highest frequency scale and contain the finest texture details (leaf textures, road surface particles, clothing wrinkles, etc.). Infrared images have a low signal-to-noise ratio at this scale, therefore a high-frequency fusion strategy dominated by visible light is adopted.

[0209] The complete process of two-level decomposition-fusion-reconstruction:

[0210]

[0211] in It is a set of high-frequency subbands in three directions.

[0212] The fusion of low-frequency subbands employs an exponential dynamic weighting mechanism based on infrared brightness. The normalized infrared low-frequency coefficients are: (Divided by DC gain to restore pixel value range) ,in (for decomposition of series), infrared weighted graph Given by a smoothed exponential function:

[0213] ;

[0214] In a preferred embodiment, the lower limit of the cold zone infrared weighting upper limit of infrared weight in hot zone and basic transition steepness These are configurable preset parameters. As an example value, (A cold background introduces almost no infrared information). (The fusion of heat sources is dominated by infrared radiation.) These parameter values ​​can be adjusted according to specific application scenarios and sensor characteristics to adapt to the data distribution characteristics of different infrared-visible image pairs.

[0215] The property of the exponential function is: when (Heat source), weights converge ;when (Cold background), weights converge The transition region (intermediate brightness) exhibits a smooth exponential decay with no hard threshold.

[0216] Apply mean filtering to the weighted graph to suppress boundary artifacts that may occur in the exponential transition region:

[0217] ;

[0218] in, Weighted graph after applying mean filtering

[0219] The basic weighted fusion is as follows:

[0220] ;

[0221] In the formula, This represents the basic fusion result of the low-frequency subband of the second-level wavelet decomposition; and These represent the second-level low-frequency sub-bands of visible light and infrared light, respectively. This represents the weighted graph of the infrared brightness index after mean filtering and smoothing.

[0222] An additional visible light brightness protection feature is added to prevent the fusion result of cold areas from falling below the original visible light brightness due to weak infrared signals, which could lead to color saturation and contrast collapse.

[0223] ;

[0224] In the formula, This represents the second-level low-frequency sub-band fusion result after brightness protection correction; This represents the visible light luminance protection factor.

[0225] In a preferred embodiment, This is a configurable preset parameter (exemplarily set to 0.15 in this embodiment). The protection term is activated only when the fused base brightness is lower than the visible light brightness (ReLU), restoring a portion of the difference. This step-by-step calculation, first calculating the base weighted fusion and then applying the brightness protection correction, avoids numerical ambiguity caused by implicit equations.

[0226] Using a text encoder in a pre-trained vision-language model, the user-input scene description text is processed. Encoded as semantic embedding vector In a preferred embodiment, the pre-trained visual-language model is a Contrastive Language-Image Pre-training (CLIP) model, and the text encoder is the CLIP text encoder.

[0227] By utilizing a pre-trained CLIP (Contrastive Language-Image Pre-training) model, specifically the CLIP ViT-B / 32 text encoder, natural language scene descriptions are mapped to the semantic space, dynamically modulating low-frequency fusion behavior without training parameter updates. This is the most critical technical innovation of this invention, and its core contribution lies in: establishing for the first time a complete and interpretable control chain from natural language semantics to frequency domain fusion parameters. Each link in this control chain has a clear physical meaning and mathematical traceability: text semantic description (user input) CLIP text encoding (semantic embedding space mapping) Semantic bias estimation (temperature-scaled Softmax similarity calculation) Visible light polarization (Continuous scalar bias parameter) Dual-path parameter modulation (transition steepness) Infrared Enhancement Intensity ) Changes in the shape of the exponential dynamic weighting function (changes in low-frequency fusion behavior) The global brightness and thermal target significance of the fusion result can be controlled. This complete control chain upgrades the fusion behavior of the present invention from "data-driven black-box generation" to "semantic-driven white-box control". Users can predictably and smoothly control the fusion result at any position between "aggressive infrared enhancement" and "conservative visible light protection" by changing the semantics of the input text.

[0228] Pre-setting semantic anchors and pre-storing a pair of semantically opposing reference text embeddings: reference hot text embedding and reference visible text embedding The text encoder pre-encodes semantically contradictory reference text descriptions (such as "thermalinfrared image..." and "visible light image...") and stores them in the network as non-trainable parameters.

[0229] ;

[0230] ;

[0231] The reference text is for illustrative purposes only; other semantically contradictory reference texts may be used in practice.

[0232] Dynamic semantic biased computation, during inference, the user-provided scene description text Encoded as query embedding The cosine similarity between the query embedding and two semantic anchors is calculated, and the semantic discriminative power is enhanced by temperature-scaled Softmax to generate the visible light bias ratio. :

[0233] ;

[0234] ;

[0235] in, and These represent the thermal imaging bias weight and the visible light bias weight, respectively. Temperature coefficient, a configurable discrimination enhancement parameter. (in Attention intensity (configurable parameter, default value is 1.0), temperature coefficient. It can be adjusted according to the dimensions of the embedding space, the selection of semantic anchors, and the target bias discrimination requirements. Visible light bias ratio For the range of values ​​within The value is a continuous scalar parameter between 1 and 0. The closer the value is to 1, the more the scene semantics are biased towards visible light (such as daytime, outdoors), and the closer the value is to 0, the more the scene semantics are biased towards thermal imaging (such as nighttime, low light).

[0236] The necessity of temperature scaling: The original cosine similarity differences in the CLIP embedding space are typically small (e.g., a query has a similarity of 0.53 with a thermal anchor and 0.59 with a visible light anchor). Without temperature scaling, the difference in bias weights in the Softmax output is insufficient to produce an effective modulation effect. Temperature coefficient This amplifies the original differences, giving the Softmax output a clear bias.

[0237] Using visible light polarization By using a preset modulation function, dual-path adaptive modulation is performed on the core parameters of low-frequency subband fusion:

[0238] Path 1, steep transition modulation:

[0239] ;

[0240] in, Base transition steepness (configurable parameter). For text guidance strength (a configurable parameter), the recommended range in the preferred embodiment is [range missing]. .when (Scene semantics biased towards thermal imaging / nighttime) Reducing the weighting curve flattens the curve, allowing more intermediate brightness areas to receive infrared enhancement; when (Scene semantics biased towards visible light / daytime). Increasing the value sharpens the curve, and only the hottest region receives infrared weight.

[0241] Path 2, Infrared Enhancement Intensity modulation:

[0242] ;

[0243] in, For attention intensity (configurable parameter). When Infrared enhancement intensity increases; when The infrared enhancement intensity decreases.

[0244] when (Scene semantics biased towards thermal imaging / nighttime): Path 1 reduced This flattens the weighting curve, allowing more intermediate brightness areas to receive infrared enhancement; path two increases the infrared enhancement intensity, making the infrared contribution more dominant. (Scene semantics biased towards visible light / daytime): Path 1 increases The curve is sharpened, and only the hottest region receives infrared weight; path two reduces the infrared enhancement intensity to protect visible light information.

[0245] The entire semantic bias computation is performed during the inference phase within the `torch.no_grad()` context, and the CLIP text encoder parameters are not included in training updates. If no text embeddings are provided or semantic guidance is disabled, then... , The system degenerates into unguided baseline fusion, and this fallback mechanism ensures the robustness of the system.

[0246] The aforementioned dual-path modulation mechanism has clear physical interpretability. It does not generate fusion results in a black box, but rather adjusts the mathematical morphological parameters of the fusion function to make the relationship between semantics and fusion behavior transparent and traceable.

[0247] The selection of the dynamic weighting function for the infrared brightness index is based on the following mathematical properties:

[0248] Smooth transition characteristics: exponential function Provides a cool background ( ) to thermal target ( The continuous and smooth transition avoids the binary boundary artifacts produced by hard thresholding in the transition region. The steepness of the transition is determined by the parameter... Precise control The larger the value, the faster the transition, and the closer the fusion strategy is to the hard threshold; The smaller the value, the smoother the transition, and the closer the fusion strategy is to the global average.

[0249] Semantic moduloability: The parameters are dynamically modulated using the visible light deflection ratio. Its modulation function... by At the neutral point, Linearity is produced when deviating from the neutral point. Offset. This linear modulation relationship makes semantic-parameter changes predictable—gradual changes in textual semantics (such as from “night” to “dusk” to “day”) result in a continuous, smooth gradient in the fusion result, rather than abrupt changes.

[0250] Boundedness guarantee: The range of the weighting function is strictly limited to... Within the range. Ensure that even in the coldest background areas, trace amounts of infrared information are preserved to avoid complete loss; Ensure that fundamental brightness information of visible light is preserved even in the hottest regions to avoid brightness collapse. This applies regardless of changes in semantic modulation. The values ​​and weights will never overflow the bounded interval, ensuring the physical rationality of the fusion result.

[0251] High-frequency subband fusion:

[0252] The high-frequency subbands (LH / HL / HH) carry the structural edges and texture details of the image. While the infrared high-frequency components contain the boundaries of thermal targets, they are accompanied by significant noise; the visible light high-frequency components have a high signal-to-noise ratio. In this embodiment, a strategy of using visible light as the main component and selectively incorporating infrared high frequencies is adopted for the high-frequency subbands, and reconstruction artifacts are suppressed by tanh bounded compression.

[0253] For high-frequency subband ( A bounded compression weighted fusion strategy is adopted. To suppress Gibbs ringing artifacts during inverse wavelet transform, all fused high-frequency subbands are compressed using a bounded activation function before being fed into IDWT.

[0254] ;

[0255] in, This represents the high-frequency subband after bounded compression. This represents the fused high-frequency subband. This is the compression steepness coefficient (configurable parameter). This is the high-frequency scaling factor (configurable parameter). Map the input to Naturally limits the upper limit of amplitude; Control the steepness of the compression zone transition; The compressed amplitude is restored to a reasonable level suitable for IDWT reconstruction.

[0256] This operation has three functions: Map the input to Naturally limits the upper limit of amplitude; Control the steepness of the compression transition (configurable parameter, the default value in this embodiment is 1.5, which strikes a balance between noise reduction and detail preservation); The scaling factor (a configurable parameter, with a default value of 2.0 in this embodiment) restores the compressed amplitude to a reasonable level suitable for IDWT reconstruction.

[0257] The high-frequency subband supports multiple switchable fusion modes, including visible light-dominated weighted mode, average mode, and learnable fusion mode, to adapt to different application scenarios.

[0258] Visible light-dominated weighted mode (default), i.e., linear weighted mixing:

[0259] ;

[0260] in, Indicates the first Level directional sub-band The average fusion result, and These represent the infrared and visible light at the 1st and 2nd dpole, respectively. Level directional sub-band High frequency coefficients, , , The infrared high-frequency mixing weight is a configurable parameter; the default value in this embodiment is 0.6.

[0261] Average pattern: It has no training parameters and is suitable for fast inference.

[0262] Learnable fusion pattern: Each pair of sub-bands consists of independent Convolution fusion, convolution kernel initialization (Equal weighted average), end-to-end training adaptive learning optimal mixing ratio.

[0263] Inverse wavelet reconstruction:

[0264] The fused low-frequency and high-frequency subbands are reconstructed step by step using Inverse Discrete Wavelet Transform (IDWT): first, the second-level fused subbands are used... Reconstructing the first-level low frequency using IDWT ; and then utilize the first-level fusion subband Wavelet domain fused image obtained by IDWT reconstruction .

[0265] The semantic segmentation branch serves as an auxiliary task, providing additional semantic supervision signals for the fused features. The input is the segmentation features refined through both CTF and CTQ stages. By sharing the structure of the fusion decoder but with an independent classifier, the output of the segment decoder is obtained. Class semantic segmentation prediction graph .

[0266] Auxiliary Segmentation Block (SFB): Before cross-task interaction in CTF, the SFB extracts preliminary semantic discriminative features from the fused features. Independent depthwise separable convolutional residual blocks (DSCResBlocks) are used for each of the three scales.

[0267] ;

[0268] ;

[0269] Depthwise Convolution extracts spatial features pointwise. Convolution enables information exchange between channels, while residual connections ensure training stability.

[0270] Training Strategy: TWFNet supports both labeled and unlabeled training modes. When semantic segmentation labels are available, the segmentation decoder's classifier receives complete pixel-level supervision signals (Dice + Cross-Entropy), and the segmentation loss is backpropagated to CTF / CTQ and the fusion encoder, driving the network to learn fusion features with stronger semantic discriminative power. When semantic segmentation labels are unavailable, the segmentation branch is in "unsupervised exploration" mode. Although there are no explicit class labels, the self-attention mechanism in CTF allows segmentation feature tokens to interact freely with fusion feature tokens. The segmentation features automatically cluster in the attention space to identify patterns that are significantly structurally related to the fusion features.

[0271] As a preferred embodiment of the present invention, mixing the first fused image and the second fused image includes: generating a spatially adaptive mixing weight based on the thermal saliency distribution of the infrared image, increasing the mixing weight of the second fused image in thermally saliency regions, and increasing the mixing weight of the first fused image in cold background regions.

[0272] Different dynamic range mapping parameters are used for regions with different thermal salience. A wide dynamic range mapping is used for thermally salience regions to release the internal contrast of infrared thermal targets, while a narrow dynamic range mapping is used for cold background regions to ensure accurate reproduction of visible light brightness.

[0273] High-frequency detail residuals are extracted from the original infrared and visible light images respectively, and either infrared detail residuals or visible light detail residuals are adaptively injected according to their thermal saliency distribution.

[0274] In this embodiment, the complete process of the three-stage adaptive post-processing is as follows: Figure 6 As shown, the three-stage adaptive post-processing fuses the decoder output. Output fused with wavelet Adaptive blending and enhancement are performed. The two sources have complementary properties: the decoder output preserves the contrast and brightness information learned by the network (high SSIM), while the wavelet output preserves multi-scale edges and details (high EN and MI).

[0275] The fused image output from the network decoder The image fused with the wavelet domain obtained in step five A three-stage adaptive blending and enhancement process is performed to obtain the final fused image. The two sources have complementary properties: the decoder output preserves the contrast and brightness information learned by the network, while the wavelet output preserves multi-scale edges and details.

[0276] The three-stage adaptive blending and enhancement processing includes:

[0277] Phase 1: Regional adaptive spatial weight allocation, using brightness information from infrared images to calculate thermal saliency maps. :

[0278] ;

[0279] in, It is the Sigmoid activation function; the larger the value, the more the spatial location is inclined to the hot target region. The infrared brightness threshold is a configurable parameter, set to 0.35 in this embodiment. This is the thermal sensitivity coefficient (a configurable parameter, set to 10 in this example). Noise is eliminated through mean pooling smoothing. .

[0280] Spatial Adaptive Hybrid Weights :

[0281] ;

[0282] in The base mixing ratio (a configurable parameter, the default value in this embodiment is 0.60, which can be further modulated by G-Atten). This is the amount of heat enhancement (a configurable parameter, 0.30 in this embodiment). The amplitude is limited to within a preset upper limit (in this embodiment, it is...). This is to prevent the decoder information from being completely discarded in extreme hot zones.

[0283] Blended Images for:

[0284] ;

[0285] In areas with significant heat, wavelet fusion results are preferred to preserve details of the hot target, while in areas with a cold background, decoder fusion results are preferred to maintain visible light structure fidelity.

[0286] Phase Two: Dynamic Range Mapping for Different Hot Zones, using Different S-Type Activation Function Mapping Parameters:

[0287] ;

[0288] ;

[0289] in, This represents the effective mapped temperature at spatial location x. This represents the pixel value at spatial location x after being mapped by the partitioned dynamic range. The base mapping temperature (a configurable parameter, 6.5 in this example) is used. This is the temperature increment for the hot zone (a configurable parameter, 2.5 in this embodiment). Mapping center (configurable parameter, 0.50 in this example). Physical meaning: Hot zone ( , Larger areas show a smoother mapping transition, releasing the internal contrast of infrared thermal targets; cold areas ( , (Smaller) transitions are sharper, ensuring accurate reproduction of visible light brightness.

[0290] A wider dynamic range mapping is used in the thermally significant regions to release the internal contrast of the infrared thermal target, while a narrower dynamic range mapping is used in the cold background regions to ensure accurate reproduction of visible light brightness.

[0291] Phase 3: Modal complementarity detail enhancement, extracting high-frequency detail residuals from the original image (subtracting the smoothing component from large-kernel mean pooling):

[0292] ;

[0293] in, and These represent the high-frequency detail residuals in infrared and visible light, respectively. The pooling kernel size is used; the residual is the high-frequency texture.

[0294] Thermally significant guided spatial adaptive injection:

[0295] ;

[0296] in, This represents the fused image after detail enhancement; This represents the image after partitioned dynamic range mapping; The infrared detail enhancement factor (a configurable parameter, set to 0.10 in this embodiment) Visible light detail enhancement factor (configurable parameter, 0.06 in this embodiment). Hot zone ( Injecting infrared details to supplement missing edge information of hot targets; cold areas ( Inject visible light details to enhance the recognizability of background textures.

[0297] Finally, residual connections are used to preserve the statistical properties of the source image, and the image is limited to... :

[0298] ;

[0299] in, This represents the final fused output image; This is the residual strength coefficient (a configurable parameter, 0.08 in this example). The residual term backpropagates the pre-mapped mixed information with minimal strength, which helps maintain the numerical stability of the mutual information (MI) index.

[0300] High-frequency detail residuals are extracted from the original infrared and visible light images respectively, and infrared or visible light detail residuals are adaptively and selectively injected according to thermal saliency distribution, which enhances texture while avoiding mutual interference of cross-modal details.

[0301] The steps for multi-task joint training are as follows:

[0302] G-Atten relies on the dynamic encoding of the CLIP text encoder. To avoid repeatedly calling the CLIP model during training, an offline pre-caching strategy is adopted: before training, the pre-trained CLIP ViT-B / 32 text encoder is used to batch compute the scene description text embedding vectors (dimension 512) for all training samples at once, and store them in a compressed format. During training, embeddings are loaded by image file name index, and only the embedding of a single sample is transferred to the GPU.

[0303] CLIP text embeddings are projected into a trainable projection head onto a semantic space compatible with network features:

[0304] ;

[0305] in, This represents the semantic embedding vector after projection. For the projection matrix, This is a learnable scaling factor (initialized to 1.0). This represents a trainable bias vector. This represents the text embedding output by the CLIP text encoder. The ProjectionHead is updated end-to-end during training, while the CLIP text encoder parameters remain unchanged.

[0306] Visible light RGB images are first converted to the luminance-chrominance color space (YCbCr): Only for the luminance channel. Perform blending processing on the chroma channels. Retain without processing. The merged version will be used during the reconstruction phase. With the original Combine and reverse convert back to RGB to ensure complete preservation of color information.

[0307] During the training phase, an end-to-end multi-task loss function is used for optimization.

[0308] ;

[0309] in, The total loss function is defined as the sum of the L1 loss and the structural similarity loss (SSIM). The sum of Dice loss and cross-entropy loss for the segmentation task (enabled when semantic segmentation labels are available); To incorporate supervised loss (enabled when paired ground truth images are available); To incorporate the regularization loss, including the range constraint term, the mean constraint term, and the highlight penalty term; and These represent the reconstruction losses of the infrared and visible light encoders, respectively. and These represent the reconstruction losses of the infrared and visible light decoders, respectively. This represents the regularization weight coefficient.

[0310] The four self-supervised reconstruction branches share the same loss function form, a weighted combination of L1 loss and structural similarity loss (SSIM):

[0311] ;

[0312] ;

[0313] In the formula, Representing an image with truth value The loss function between; and Let i and j represent the predicted value and the true value at spatial location (i,j), respectively. ,·) represents a structural similarity measure; and Let x and y represent the mean values ​​of the image, respectively. and Let x and y represent the variances of the image, respectively. Represents the covariance of the x and y axes of the image; and It is the stability constant.

[0314] The combination of L1 and SSIM is complementary: L1 loss imposes a linear penalty on large-scale brightness shifts but is highly tolerant of small displacements in local structures; SSIM loss is sensitive to contrast flips and structural damage, making up for the shortcomings of L1 in perceived quality.

[0315] The four monitoring objectives are as follows: , This verifies that the encoder has correctly extracted the features of each modality; , This verifies that FDB successfully decouples modality-specific information from the fused features. The latter two loss terms are key driving signals for FDB's self-supervised learning.

[0316] Segmentation loss (specifically for labeled datasets): Dice loss directly optimizes the spatial overlap between the predicted class probability map and the binary ground truth mask. Its core advantage lies in its natural robustness to class imbalance. Cross-entropy loss provides independent pixel-by-pixel classification constraints. The combination of Dice and cross-entropy balances the optimization dimensions of maximizing global region overlap and local pixel classification accuracy.

[0317] Fusion supervision loss (optional, enabled when paired ground truth images exist): consists of L1 loss, SSIM loss, gradient fidelity term, and overexposure penalty term. The gradient fidelity term constrains the fusion edges to align with the stronger of the two source images; the overexposure penalty term imposes an additional penalty on highlighted areas.

[0318] Fusion regularization loss. Constrain the fusion output in... Physically reasonable range:

[0319] ;

[0320] In the formula, ReLU represents the fusion regularization loss, and ReLU(·) represents the linear rectified function. This indicates the fused output image. This represents the mean of the image.

[0321] The four items correspond to: lower limit constraint ( Penalties), upper limit constraints ( Penalty), mean constraint (penalty only) To prevent overall over-brightness, a highlight penalty (3 times the weight) is applied. Regularization weights. During training, the value is linearly increased from 0.1 to 0.5 (interpolated by global steps).

[0322] Training configuration:

[0323] Adaptive Moment Estimation Optimizer (Adam) is used, with hyperparameters... , , Initial learning rate This is combined with cosine annealing (LR) learning rate scheduling and linear warm-up (Gradual WarmupScheduler) for the first few epochs. The warm-up mechanism is necessary because the randomly initialized attention weights in the early stages of training may produce large gradient variances, and directly using the full learning rate may lead to unstable parameter update directions.

[0324] Automatic Mixed Precision (AMP) training is used to accelerate convergence. Special attention needs to be paid to the precision handling of the DWT / IDWT path: the wavelet transform filter bank weights are of type float32, and autocast must be explicitly disabled in the DWT / IDWT calculation to force this path to maintain fp32 precision. The rest of the network uses AMP for acceleration as normal.

[0325] To comprehensively evaluate the fusion performance of the method of this invention, a systematic quantitative and qualitative comparative experiment was conducted on the publicly available infrared-visible light fusion dataset M3FD. The M3FD dataset contains 4,200 pairs of rigorously registered infrared-visible light images, covering three typical outdoor scenes: campus, main urban roads, and coastal scenic areas, and encompassing various lighting conditions such as daytime, dusk, and nighttime. This dataset can effectively test the generalization ability of the fusion method in multiple scenarios.

[0326] Six representative mainstream fusion methods were selected as baselines for comparison: YDTR (a transform domain-based fusion method), CDDFuse (a CNN-based fusion method), PIAFusion (a light-aware fusion method), Text-IF (a text-guided fusion method), U2Fusion (a unified framework-based fusion method), and FreeFusion (the current leading-performing fusion method). Evaluation metrics included entropy (EN), standard deviation (SD), spatial frequency (SF), average gradient (AG), structural difference correlation (SCD), and visual information fidelity (VIF), which were quantitatively evaluated from six dimensions: information richness, contrast distribution, spatial detail preservation, edge sharpness, structural fidelity, and visual information fidelity.

[0327] Table 1. Average quantitative indicators of each method on the M3FD test set.

[0328]

[0329] As shown in Table 1, the TWFNet of this invention achieved the best results in four of the six metrics (EN, SD, SF, AG), and the specific analysis is as follows:

[0330] EN and SD reflect the global information content and contrast distribution range of the fused image. TWFNet achieves an EN of 7.242, surpassing the previous best, FreeFusion (7.228); its SD reaches 45.455, slightly ahead of FreeFusion (45.382). This advantage stems from the exponential dynamic weight fusion strategy implemented by the wavelet semantic fusion head in the low-frequency subband. By assigning progressively increasing infrared weights to hot source regions and maintaining visible light dominance in cold background regions, it effectively preserves the global brightness and contrast levels of the scene.

[0331] SF and AG are core metrics for measuring the quality of edge texture preservation in fused images. TWFNet achieved a significant lead in these two metrics: SF reached 26.349, surpassing the second-place FreeFusion (20.171) by 30.6%; AG reached 13.365, nearly 1.88 times that of FreeFusion (7.128). This substantial improvement directly verifies the effectiveness of the two core designs of this invention: modal complementary detail enhancement in the three-stage adaptive post-processing, which supplements the boundary information of infrared thermal targets in hot regions and enhances the texture discernibility of visible light backgrounds in cold regions through thermally saliency-guided spatial adaptive detail residual injection; and the tanh bounded compression fusion strategy for high-frequency subbands, which effectively suppresses cross-modal amplification of infrared noise while preserving multi-scale edge details.

[0332] In terms of the structure-oriented metrics SCD and VIF, TWFNet scores (1.290 and 0.585) lag behind leading methods. This phenomenon can be explained by the inherent characteristics of frequency domain fusion: while wavelet domain fusion enhances edge contrast and texture details (manifested as a significant improvement in SF and AG), it may slightly affect the spatial continuity of the global structure. This is essentially a trade-off between "detail enhancement" and "structural smoothing." Notably, FreeFusion's lead in SCD (1.916) and VIF (0.769) comes at the cost of a mere 20.171 in SF, validating the aforementioned trade-off. The design orientation of this invention clearly prioritizes ensuring the saliency of thermal targets and the preservation of texture details, which has greater practical value in applications such as security monitoring and military reconnaissance where target identifiability is crucial.

[0333] From the perspective of method categories: early methods such as YDTR and U2Fusion generally performed poorly across various metrics, indicating that traditionally hand-designed fusion rules are difficult to adapt to diverse scenarios; deep learning-based methods such as CDDFuse and PIAFusion showed stable performance at the median level; after introducing text guidance, Text-IF showed a slight improvement in EN, SF, and AG compared to PIAFusion, verifying the positive effect of text semantics on fusion quality, but its improvement was far less than that of this invention, which precisely confirms the "semantic bias estimation" of this invention. Parameter modulation The interpretable control chain of "frequency domain fusion" has significant advantages over the "text as a conditional feature black box input" scheme of Text-IF. FreeFusion performs well in SCD and VIF, but lags far behind TWFNet in SF and AG, reflecting its relative deficiency in texture detail. Overall, TWFNet has established a clear leading position in information richness and texture detail, especially in SF and AG, the two most perceptually distinguishing metrics, where it significantly surpasses existing methods.

[0334] Qualitative comparison results of the M3FD dataset. The figure shows... The grid layout consists of a set of infrared-visible image pairs representing a typical scene in each row, and each column contains: an infrared source image, a visible light source image, and the fusion results of YDTR, CDDFuse, PIAFusion, Text-IF, U2Fusion, FreeFusion, and the TWFNet of this invention.

[0335] like Figure 7 As shown, this further validates the conclusions of the quantitative analysis from the perspective of visual perception.

[0336] From the perspective of thermal target saliency: In nighttime and low-light scenes, the TWFNet fusion results show that thermal targets such as pedestrians and vehicles have the highest brightness contrast and the clearest contour boundaries. This is due to the selective enhancement of infrared information in the heat source region by the low-frequency exponential dynamic weighting function, and the automatic reduction of transition steepness by the G-Atten semantic modulation mechanism in thermal imaging biased scenarios. Expanding the infrared enhancement range. In comparison, while FreeFusion and Text-IF can render thermal targets, the contrast between the target and the background is not as sharp as that of TWFNet; YDTR and U2Fusion suffer from blurred boundaries of thermal targets, which blend into the background.

[0337] From the perspective of visible light texture preservation: In daytime and well-lit scenes, TWFNet fully preserves visible light features such as building facade textures, road markings, and vegetation details, while appropriately incorporating infrared thermal information to enhance scene depth. This is thanks to the spatial adaptive weight allocation strategy in the three-stage post-processing that prioritizes cool background areas based on the decoder fusion results. Text-IF and PIAFusion perform reasonably well in visible light texture preservation, but lack visual prominence of thermal targets; CDDFuse exhibits the problem of excessive suppression of visible light textures by infrared information in some scenes.

[0338] Based on both qualitative and quantitative analysis, TWFNet demonstrates stable performance advantages in diverse street scenes (campus, urban main roads, coastal scenic areas) and under different lighting conditions (daytime, dusk, nighttime), verifying the generalization ability and robustness of this invention under multiple scene conditions. Experimental results show that the text semantic dynamic modulation wavelet low-frequency fusion strategy can effectively solve the core problem of existing methods' inability to adaptively switch fusion strategies under different scenes, achieving an optimal balance between texture details and thermal target saliency while maintaining reasonable structural fidelity.

[0339] This invention also provides a semantically guided wavelet domain infrared-visible image fusion system, the system comprising:

[0340] The image acquisition module is used to acquire infrared images, visible light images, and scene description text.

[0341] The feature extraction and fusion module is used to perform multi-scale feature extraction on the infrared image and the visible light image respectively to obtain an infrared feature map and a visible light feature map, perform cross-modal feature fusion on the infrared feature map and the visible light feature map to obtain a fused feature, and decode the fused feature to obtain a first fused image;

[0342] The visible light bias ratio generation module is used to encode the scene description text using a pre-trained visual-language model text encoder to obtain a semantic embedding vector, and generate a visible light bias ratio based on the semantic similarity between the semantic embedding vector and at least two pre-stored reference text embeddings.

[0343] The discrete wavelet transform module is used to perform discrete wavelet transform on the infrared image and the visible light image respectively to obtain corresponding low-frequency sub-bands and high-frequency sub-bands. Based on at least one control parameter of the low-frequency fusion weight function modulated by the visible light bias ratio, the low-frequency sub-bands are fused using the modulated low-frequency fusion weight function to obtain a fused low-frequency sub-band. The high-frequency sub-bands are fused based on a bounded compression weighted fusion strategy to obtain a fused high-frequency sub-band. The fused low-frequency sub-band and the fused high-frequency sub-band are subjected to inverse discrete wavelet transform to obtain a second fused image.

[0344] The image fusion module is used to blend the first fused image and the second fused image to obtain the final fused image.

[0345] This invention establishes for the first time a complete and interpretable control chain from natural language scene description to frequency domain fusion parameters. Each link in the control chain has a clear physical meaning and mathematical traceability. During the inference stage, users only need to input scene description text, and low-frequency fusion parameters can be automatically modulated through semantic bias estimation to achieve optimal fusion strategy switching under different scenarios without retraining the model parameters. The technical basis of this capability lies in utilizing the semantic similarity in the semantic embedding space of a pre-trained visual-language model. The property of "proximity" is used to compress high-dimensional semantic information into continuous scalar control parameters that can be executed by the fusion system through temperature scaling Softmax. Then, by using a dual-path modulation function, the scalar parameter is mapped to a specific control variable of the frequency domain fusion function, thus achieving a complete and interpretable connection from "high-level semantic cognition" to "low-level signal processing".

[0346] By performing semantic modulation in the low-frequency subband while maintaining a fixed strategy in the high-frequency subband, an optimal balance is achieved between the controllability of global perceptual quality (low frequency) and the stability of local detail fidelity (high frequency). Under semantic modulation, the exponential dynamic weighting function of the low-frequency subband achieves a smooth and progressive fusion transition from cold background to hot target, avoiding the boundary artifact problem of hard threshold segmentation.

[0347] Through a cascaded mechanism of self-attention and cross-attention (CTF+CTQ), bidirectional feature refinement is achieved for fusion and semantic segmentation tasks. The fused features provide multimodal context for the semantic task, while the semantic features, in turn, guide the fused features to focus on the semantic structure of the scene. CTQ's asymmetric cross-attention design, using the original features as the query, ensures that the refined output is a semantically enhanced version of the features rather than newly generated features, naturally possessing residual stability.

[0348] Explicit separation of modality-specific information is achieved through feature subtraction guided by spatial attention, without the need for complex decoupling methods such as mutual information loss, adversarial networks, or orthogonal constraints. It relies solely on a learnable, lightweight attention layer and basic tensor subtraction operations. The self-supervised reconstructed signal provides an effective learning driver for decoupling, eliminating the need for additional modality decoupling annotation data.

[0349] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A semantically guided wavelet domain infrared-visible image fusion method, characterized in that, The method includes: Acquire infrared and visible light images, as well as scene description text; Multi-scale feature extraction is performed on the infrared image and the visible light image respectively to obtain infrared feature maps and visible light feature maps. Cross-modal feature fusion is performed on the infrared feature maps and the visible light feature maps to obtain fused features. The fused features are then decoded to obtain a first fused image. The scene description text is encoded using a pre-trained visual-language model text encoder to obtain a semantic embedding vector. Based on the semantic similarity between the semantic embedding vector and at least two pre-stored reference text embeddings, a visible light bias ratio is generated. In parallel, discrete wavelet transforms are performed on the infrared image and the visible light image respectively to obtain corresponding low-frequency sub-bands and high-frequency sub-bands. Based on at least one control parameter of the low-frequency fusion weight function modulated by the visible light bias ratio, the low-frequency sub-bands are fused using the modulated low-frequency fusion weight function to obtain a fused low-frequency sub-band. The high-frequency sub-bands are fused based on a bounded compression weighted fusion strategy to obtain a fused high-frequency sub-band. The fused low-frequency sub-band and the fused high-frequency sub-band are subjected to inverse discrete wavelet transforms to obtain a second fused image. The first fused image and the second fused image are blended to obtain the final fused image.

2. The semantically guided wavelet domain infrared-visible image fusion method according to claim 1, characterized in that, The multi-scale feature extraction process employs a dual-channel encoder, which includes an infrared encoder and a visible light encoder. The infrared encoder and the visible light encoder have the same network topology and their parameters are independent of each other.

3. The semantically guided wavelet domain infrared-visible image fusion method according to claim 1, characterized in that, The at least two reference text embeddings include a thermal imaging bias reference embedding and a visible light bias reference embedding.

4. The semantically guided wavelet domain infrared-visible image fusion method according to claim 1, characterized in that, The low-frequency fusion weighting function is an exponential function with the normalized infrared low-frequency coefficient as the independent variable.

5. The semantically guided wavelet domain infrared-visible image fusion method according to claim 4, characterized in that, The control parameters include the transition steepness parameter of the exponential function and the infrared enhancement intensity parameter; The transition steepness parameter is used to control the steepness of the transition from the cold zone to the hot zone in the low-frequency fusion weighting function, wherein the smaller the visible light bias ratio, the smaller the transition steepness parameter, and the larger the visible light bias ratio, the larger the transition steepness parameter. The infrared enhancement intensity parameter is used to control the contribution intensity of infrared information in the fusion, wherein the smaller the visible light bias ratio, the larger the infrared enhancement intensity parameter, and the larger the visible light bias ratio, the smaller the infrared enhancement intensity parameter.

6. The semantically guided wavelet domain infrared-visible image fusion method according to claim 1, characterized in that, The step of decoding the fusion features to obtain the first fused image includes: Extract semantic discriminant features from the fused features; By using a multi-head self-attention mechanism to perform cross-task global context interaction on the concatenated fusion features and semantic discriminative features, cross-enhanced fusion features and cross-enhanced semantic features are obtained. Using a multi-head cross-attention mechanism, the fusion feature and the semantic discriminative feature are each used as queries, and the cross-enhanced fusion feature and the cross-enhanced semantic feature are used as keys and values, respectively, to perform targeted feature refinement, resulting in refined fusion features and refined semantic features. Based on the feature subtraction operation guided by the spatial attention of the modal decoupling block, infrared-specific features and visible light-specific features are explicitly separated from the refined fused features; The decoupled infrared-specific features and visible light-specific features are sent to a shared decoder for independent reconstruction. The corresponding original infrared and visible light images are used as self-supervised targets for reconstruction constraints to obtain the first fused image.

7. The semantically guided wavelet domain infrared-visible image fusion method according to claim 6, characterized in that, Based on the feature subtraction operation guided by the spatial attention of the modal decoupling block, the specific steps for explicitly separating infrared-specific features and visible light-specific features from the refined fused features include: For the refined fusion features, spatial attention is used to extract modal salient regions from the visible light feature map and the infrared feature map, respectively; The infrared-specific feature is obtained by subtracting the spatial attention-weighted visible light component from the refined fusion feature. The visible light-specific features are obtained by subtracting the infrared component weighted by the spatial attention from the refined fusion features.

8. The semantically guided wavelet domain infrared-visible image fusion method according to claim 1, characterized in that, Blending the first fused image and the second fused image includes: generating spatially adaptive blending weights based on the thermal saliency distribution of the infrared image, increasing the blending weight of the second fused image in thermally saliency regions, and increasing the blending weight of the first fused image in cold background regions; Different dynamic range mapping parameters are used for regions with different thermal salience. A wide dynamic range mapping is used for thermally salience regions to release the internal contrast of infrared thermal targets, while a narrow dynamic range mapping is used for cold background regions to ensure accurate reproduction of visible light brightness. High-frequency detail residuals are extracted from the original infrared and visible light images respectively, and either infrared detail residuals or visible light detail residuals are adaptively injected according to their thermal saliency distribution.

9. A semantically guided wavelet domain infrared-visible image fusion system, used to implement the semantically guided wavelet domain infrared-visible image fusion method according to any one of claims 1-8, characterized in that, The system includes: The image acquisition module is used to acquire infrared images, visible light images, and scene description text. The feature extraction and fusion module is used to perform multi-scale feature extraction on the infrared image and the visible light image respectively to obtain an infrared feature map and a visible light feature map, perform cross-modal feature fusion on the infrared feature map and the visible light feature map to obtain a fused feature, and decode the fused feature to obtain a first fused image; The visible light bias ratio generation module is used to encode the scene description text using a pre-trained visual-language model text encoder to obtain a semantic embedding vector, and generate a visible light bias ratio based on the semantic similarity between the semantic embedding vector and at least two pre-stored reference text embeddings. The discrete wavelet transform module is used to perform discrete wavelet transform on the infrared image and the visible light image respectively to obtain corresponding low-frequency sub-bands and high-frequency sub-bands. Based on at least one control parameter of the low-frequency fusion weight function modulated by the visible light bias ratio, the low-frequency sub-bands are fused using the modulated low-frequency fusion weight function to obtain a fused low-frequency sub-band. The high-frequency sub-bands are fused based on a bounded compression weighted fusion strategy to obtain a fused high-frequency sub-band. The fused low-frequency sub-band and the fused high-frequency sub-band are subjected to inverse discrete wavelet transform to obtain a second fused image. The image fusion module is used to blend the first fused image and the second fused image to obtain the final fused image.