Image fusion method based on semantic chain feedback and reinforcement learning driving

Abstract

The present application relates to a semantic chain feedback and reinforcement learning driven image fusion method, belonging to the technical field of computer vision and image processing. It includes: inputting the preprocessed input image into the frozen dual-path hierarchical encoder, extracting multi-scale visual features and shallow prior features in the prior injection branch; on each intermediate scale, cross-modal collaborative fusion is performed on the infrared features, visible light features, the last layer propagation features and shallow prior features in the multi-scale visual features; chain semantic enhancement is performed on the multi-scale fusion features: obtaining several candidate fusion images; optimizing the unfrozen model parameters; inputting the infrared image and visible light image to be processed into the optimized network, and selecting the noise-free reference candidate as the final fusion image output. This method cooperatively optimizes through chain semantic progressive modulation and difference driven reinforcement learning, improving the visual quality and information expression ability of the fusion image.

Description

Technical Field

[0001] This invention relates to an image fusion method based on semantic chain feedback and reinforcement learning, belonging to the field of computer vision and image processing technology. Background Technology

[0002] With the widespread application of infrared and visible light imaging technologies, infrared and visible light image fusion has become an important means to improve visual perception capabilities in complex environments. Infrared images can highlight thermal targets and are unaffected by illumination, but lack texture details; visible light images contain rich structural and detailed information, but their performance is limited under low-light or occlusion conditions. Fusing the two into an image that combines target saliency and visual detail is of great significance for tasks such as autonomous driving, object detection, and semantic segmentation. In recent years, deep learning-based fusion methods have made progress in network structure and representation modeling, such as encoder-decoder structures, Transformer modeling (i.e., sequence modeling structures based on self-attention mechanisms), and generative priors, which have effectively improved fusion performance.

[0003] However, existing methods still have shortcomings in three aspects: First, it is difficult to effectively introduce large-scale pre-trained visual priors under frozen conditions; second, semantic information is mostly used as input conditions, lacking the ability to dynamically feedback and close-loop control of the fusion process; third, the optimization process still relies on fixed weighted loss and lacks a difference-driven mechanism based on candidate comparison, which limits the expressive power of the fused image quality. Based on the above problems, there is an urgent need for an image fusion method that can effectively integrate prior knowledge, realize semantic process control, and have stronger optimization capabilities.

[0004] Specifically, this invention proposes an infrared and visible light image fusion method based on semantic chain feedback and reinforcement learning. It innovatively constructs a fusion representation mechanism guided by a frozen visual basic model, a chain semantic progressive modulation mechanism, and a difference-driven reinforcement learning optimization framework, thereby improving the quality and robustness of the fused image. Summary of the Invention

[0005] The technical problem solved by this invention is that it provides an image fusion method based on semantic chain feedback and reinforcement learning to overcome the problems of insufficient visual priors in existing deep learning fusion methods under small-scale data training, the difficulty of multimodal semantic coding models intervening in the intermediate fusion process, and the difficulty of achieving adaptive optimization due to the dependence of traditional deterministic optimization paradigms based on weighted loss on absolute error supervision. The method of this invention provides prior representation guidance through the collaboration of a frozen DINOv3 dual-path hierarchical encoder and a frozen Qwen3-VL-Embedding encoder, and combines chain semantic progressive modulation and candidate policy reinforcement to make full use of large-scale pre-trained visual priors and multimodal semantic priors without retraining the backbone network, thereby significantly improving the visual quality of the fused image and the performance of downstream tasks.

[0006] The technical solution of this invention is: an image fusion method based on semantic chain feedback and reinforcement learning driven by semantic chain feedback, the method comprising:

[0007] After preprocessing the input infrared and visible light image pairs, the input is frozen into a dual-channel layered encoder and a prior injection branch to extract multi-scale visual features and shallow prior features, respectively.

[0008] The multi-source collaborative fusion module performs cross-modal collaborative fusion of infrared features, visible light features, upper-layer propagation features and shallow prior features in multi-scale visual features at various intermediate scales.

[0009] Chain-based semantic enhancement is performed on multi-scale fused features;

[0010] Then acquire several candidate fusion images;

[0011] Optimize the parameters of the unfrozen model;

[0012] The infrared image and the visible light image to be processed are input into the optimized network, and the optimized network is used for the fusion processing of the infrared image and the visible light image.

[0013] Furthermore, the method includes the following specific steps:

[0014] Step 1: Obtain infrared and visible light image pairs and preprocess the input images;

[0015] Step 2: Input the preprocessed infrared image and visible light image into the frozen dual-channel layered encoder to extract the multi-scale visual features of the infrared image and visible light image respectively.

[0016] The preprocessed infrared image and the visible light image are used as input priors to extract shallow prior features;

[0017] Step 3: Through the multi-source collaborative fusion module, cross-modal collaborative fusion is performed on infrared features, visible light features, upper-layer propagation features and shallow prior features in multi-scale visual features at each intermediate scale.

[0018] When performing cross-modal collaborative fusion, the fusion features are propagated from high to low levels along a top-down path, and then the fusion features of each scale are unified to the same number of channels and spatial resolution to form a multi-scale fusion feature dictionary.

[0019] Step 4: Perform chained semantic enhancement on the multi-scale fused features using the chained semantic progressive modulation module:

[0020] Multi-scale fusion features are reconstructed step-by-step into intermediate fusion images using a shared lightweight decoder;

[0021] The intermediate fused image is then input into the frozen large model encoder in the chain semantic progressive modulation module for staged semantic analysis. The obtained continuous semantic embedding vector is linearly projected and fused with the joint global visual vector in the multi-scale visual features. After activation function processing, the channel gating coefficients are obtained. The channel gating coefficients are used to adaptively and differentially enhance the fused features of the next layer, forming a chain progressive semantic modulation process from coarse to fine. Finally, the chain semantically enhanced fused features are output.

[0022] Step 5: The chain-semantic-enhanced fusion features are treated as states through the candidate policy sampling module. The baseline action mean is learned through a lightweight policy head composed of two 3×3 convolutions. A Gaussian policy distribution is constructed with a fixed standard deviation. Several candidate features are sampled from this distribution. The 0th candidate is a noise-free baseline candidate. The policy mean neighborhood of the remaining candidates is then randomly explored. The candidate features are concatenated with the original bimodal image and sent to the shared lightweight decoder to reconstruct several candidate fusion images.

[0023] Step 6: Optimize the unfrozen model parameters from Steps 2-5 using a multidimensional composite reward function and policy gradient algorithm; the optimization process is as follows:

[0024] A baseline is constructed based on the average reward of several candidates within the sample. The relative advantage of each candidate is calculated. The policy gradient loss is constructed by multiplying the relative advantage by the log probability, which drives the policy distribution to converge toward the high-quality fused image region.

[0025] Step 7: Input the infrared image and visible light image to be processed into the optimized network, and select the noise-free benchmark candidate as the final fused image output.

[0026] Further, Step 1 includes:

[0027] The input infrared and visible light image pairs are normalized in terms of channels and size, and the brightness channels of single-channel infrared and visible light images are expanded into a three-channel format consistent with the backbone network requirements.

[0028] During the training phase, images are randomly cropped and normalized.

[0029] During the inference stage, the original resolution is preserved, and the visible light image is separated according to the YCbCr color space, where Y represents the luminance channel, Cb represents the blue differential channel, and Cr represents the red differential channel. Only the Y channel and the infrared image are sent to the fusion network. After fusion, the output is recombined with the Cb and Cr channels and converted back to the RGB color space, where RGB represents the red, green, and blue three-channel color space, to obtain the final fused image.

[0030] Furthermore, Step 2 includes:

[0031] Step 2.1: Extract multi-scale visual features from the preprocessed infrared and visible light images into the frozen dual-channel layered encoder, specifically including:

[0032] First, input the infrared image With visible light images Each channel is expanded into three channels through a channel copy operation, resulting in... and The expression is , Indicates a channel copy operation;

[0033] Next, and The signals are fed into a dual-channel layered encoder with completely frozen parameters. Perform reasoning to extract multi-scale visual features ,in It is a low-level structural feature. These are mid-level semantic features. For high-level semantic features, For global semantic features;

[0034] Finally, the backbone network of the dual-channel hierarchical encoder remains frozen throughout the training and inference process and does not participate in parameter updates.

[0035] Step 2.2: Extract shallow prior features from the input prior injection branch by combining the preprocessed infrared image and the visible light image, including:

[0036] First, and After being spliced ​​along the channel dimension, the data is fed into a shallow network consisting of two 3×3 convolutional layers. Each convolutional layer is followed by a normalization layer and an activation function, resulting in a basic prior feature representation with 32 channels. :

[0037]

[0038] Where Cat represents the concatenation operation along the channel dimension, B represents the batch size of the features, and H and W represent the height and width of the feature map, respectively.

[0039] Next, the basic prior features are represented. Interpolate to a spatial size that adapts to features at various scales, and then pass it through a mapping network consisting of 3×3 convolutions and 1×1 convolutions. Upgrade to the target number of channels at each scale to form hierarchical shallow prior features. ;

[0040]

[0041] in, Corresponding to 0, 1, 2, and 3, Stage 0 represents the original size hierarchical feature layer, where the feature size extracted from Stage 0 is the same as the original input image size. Stage 1, Stage 2, and Stage 3 represent the low-level, middle-level, and high-level hierarchical feature layers, respectively. Finally, the shallow prior features... The number of channels is kept consistent with the corresponding backbone network, which is used to compensate for the gradual decay of low-level visual responses during deep coding in subsequent fusion processes.

[0042] Furthermore, in Step 3, the operation process of the multi-source collaborative fusion module includes:

[0043] First, for arbitrary scale Stage indexing The corresponding scale, multi-source collaborative fusion module Infrared features at the corresponding scale With visible light characteristics After being concatenated along the channel dimension, the data is fed into a cross-modal complementary modeling operator consisting of two 3×3 convolutional layers. , transfer the channel from Mapped to :

[0044]

[0045] Where Cat represents the splicing operation along the channel dimension;

[0046] Next, if there are propagation characteristics at the next higher level... Then it is constructed from 1×1 convolutions. Feature alignment layer, which propagates features from the previous layer. Features at the current scale Align along the channel dimension and inject the current feature:

[0047]

[0048] in, Assignment operation for information transmission;

[0049] Then, the shallow prior features of the corresponding scale are... Inject the current representation to compensate for the attenuation of low-level visual information during stepwise downsampling:

[0050]

[0051] Then, a thinning operator consisting of a single 3×3 convolution layer is used. Reshape the features:

[0052]

[0053] Finally, a top-down progressive fusion path is constructed through a multi-source collaborative fusion module: starting from the semantically richest Stage3, it propagates layer by layer to Stage0; the fused features at each scale are refined through a channel compression layer. A unified mapping is performed to 128 channels, thus obtaining a multi-scale fusion feature dictionary.

[0054] Furthermore, in Step 4, the operation process of the chain-like semantic progressive modulation module includes:

[0055] Step 4.1: Initialize the chained modulation state: merging the multi-scale feature dictionary. The features of each layer in the chain modulation are used as the initial state, where the multi-scale fusion feature dictionary is denoted as 𝒟, and the initial state is denoted as 𝒟. Where k represents the Stage index that participates in semantic analysis and generates channel gating coefficients, and the values ​​are decremented by 3, 2, and 1. Stage0 is the output layer after modulation in stage k=1.

[0056] Step 4.2, Intermediate Image Fusion and Reconstruction: In the modulation stage corresponding to Stagek, the current state is... Compared with the original dual-modal infrared image and visible light images After concatenation along the channel dimension, the input is shared by a lightweight decoder. Reconstructed into an intermediate blended image And scale it to a fixed resolution. The input format was copied to three channels to adapt to the large model encoder frozen in the chained semantic progressive modulation module. ; This represents the fixed side length parameter of the large model encoder input frozen in the adaptation chain semantic progressive modulation module:

[0057]

[0058] Here, `clamp` represents the truncation function, used to limit the value from exceeding a preset upper bound, and `Repeat` represents the channel copy operation. This represents the bilinear interpolation operator;

[0059] Step 4.3, Chained Semantic Feedback: The semantic analysis is passed layer by layer from Stage 3 to Stage 2 to Stage 1, and the modulation result is then passed to Stage 0. Here, k takes values ​​of 3, 2, and 1, corresponding to the three semantic analysis stages of Stage 3, Stage 2, and Stage 1 respectively. Stage 0 does not perform semantic analysis using k alone, but receives the modulation result generated in the stage where k=1. In the modulation stage corresponding to Stage k, the frozen large model encoder Ψ receives the current intermediate fused image. The system uses fixed text commands and corresponding hierarchical prompts for Stagek, outputting continuous semantic embedding vectors. :

[0060]

[0061] Where, instruction represents a fixed text instruction, and prompt_k represents the hierarchical prompt text corresponding to Stagek;

[0062] Among them, Stage 3 prompts focus on global salient targets and structural consistency, Stage 2 prompts focus on boundaries and contrast, and Stage 1 prompts focus on detail texture and artifact suppression.

[0063] Step 4.4, Visual-Semantic Alignment and Chained Channel Modulation: Global semantic features of infrared and visible light images respectively. and Perform mean pooling along the token dimension and sum them to obtain a joint global visual vector. Wherein, token represents a single sequence unit in the global semantic features. and Global semantic features representing the corresponding position :

[0064]

[0065] semantic embedding vector By linear projection Aggregated feature vectors aligned to the same dimension :

[0066]

[0067] Add it to the joint global visual vector and project it. Mapped to channel gating coefficient :

[0068]

[0069] The channel gating coefficients are further obtained by centering the channel using the Sigmoid activation function. :

[0070]

[0071] in, This represents the Sigmoid activation function;

[0072] Will The next lower-level feature corresponding to Stage(k-1) is applied. Channel-level adaptive modulation is applied to it; when k=1, the modulated object is the feature corresponding to Stage0:

[0073]

[0074] in, This represents the Hadamard product, used for element-wise feature modulation.

[0075] Step 4.5: Output the fused features after chained semantic enhancement: Perform three chained modulations corresponding to k=3, k=2, and k=1 in sequence, and obtain the Stage 0 modulation result. Finally, the final chained semantically enhanced fused features are output. :

[0076] .

[0077] Furthermore, Step 5 includes:

[0078] Step 5.1, Lightweight Strategy Head and Strategy Mean Estimation: Lightweight Strategy Head It consists of two 3×3 convolutional layers. The first convolutional layer is followed by a normalization layer and an activation function, while the second convolutional layer directly outputs the mean of the policy distribution as a residual. Make an estimate:

[0079]

[0080] Step 5.2, Construction of Gaussian Policy Distribution and Candidate Sampling: Construct a Gaussian policy distribution for the mean and a fixed standard deviation σ. and from the Gaussian strategy distribution Mid-sampling yields N candidate features:

[0081]

[0082] in These are noise-free baseline candidates used for inference output and baseline reward calculation. For the nth candidate feature, It is the identity matrix. Let n be the nth random noise; the remaining candidates, except for the 0th candidate, randomly explore the neighborhood of the policy mean, and explicitly construct a candidate set with differences;

[0083] Step 5.3, Calculate the log probability of the policy: For each candidate feature, calculate its log probability under the current policy distribution. ,in This represents the action in the current state, and is normalized by averaging the channel and spatial dimensions:

[0084]

[0085] Where C represents the channel dimension of the feature map, and H and W represent the height and width of the feature map, respectively. These are elements corresponding to the passage and spatial location. This represents the log probability of the nth candidate feature under the current policy distribution;

[0086] Step 5.4, Candidate Fusion Image Reconstruction with Shared Lightweight Decoder: Shared Lightweight Decoder This process is repeated at this stage, with N candidate features... The unified decoding reconstructs candidate fused images; the parameters of this shared lightweight decoder are shared and trained through reward gradients, which indirectly affects the quality of the intermediate fused images.

[0087] The features obtained by concatenating the candidate features with the original bimodal image are represented as follows: :

[0088]

[0089] Subsequently, the N candidate features are fed in parallel into the shared lightweight decoder. The candidate fused images are obtained as follows: This shared lightweight decoder consists of two 3×3 convolutional layers, each followed by a normalization layer and a LeakyReLU activation function to progressively extract fusion features; the last layer uses a 1×1 convolution to map the features to a single-channel output, directly generating N candidate fused images. :

[0090]

[0091] Finally, it is restored to a tensor shape of [B,N,1,H,W], where B represents the batch size, N represents the number of candidate features, and H and W represent the image height and width, respectively.

[0092] Furthermore, Step 6 includes:

[0093] Step 6.1: Construct a multidimensional composite reward function in product form:

[0094]

[0095] Implicit joint constraints are constructed without the need for manual weight setting; where Representative fused image Total reward These represent the fused images in sequence. The rewards include pixel response consistency, structural similarity, and gradient detail preservation.

[0096] Step 6.2, Pixel Response Consistency Reward Pixel-wise maximum response map of two modes For reference, the fused image should inherit at least the brightness response of the stronger modality at each location:

[0097]

[0098] Where max represents the maximum value operation. Represents an exponential function. It is an L1 norm;

[0099] Step 6.3, Structural Similarity Reward The SSIM similarity values ​​between the candidate image and the two modalities are calculated separately, converted into rewards, and then averaged. A lower bound truncation ε is used to ensure the stability of the training values. Here, SSIM represents the structural similarity measure, and ε represents the lower bound truncation constant. Representative structural similarity loss:

[0100]

[0101]

[0102] Step 6.4, Gradient Details Retention Reward Gradient magnitudes are extracted using the Sobel operator, and the pixel-wise maximum gradients of the two modes are used. For reference, image fusion is encouraged to preserve the edge and structural details of the sharpest modality; where the Sobel operator represents a gradient extraction operator based on local difference. , For the corresponding gradient value:

[0103]

[0104]

[0105] Step 6.5, Advantage Value Estimation and Policy Gradient Optimization: Using the average reward of N candidates within the same sample. Using this as a baseline, calculate the relative advantage of each candidate. :

[0106]

[0107] Step 6.6, Policy Gradient Loss Optimization: The policy gradient loss is optimized using the REINFORCE policy gradient objective, where REINFORCE represents the policy gradient algorithm based on Monte Carlo reward estimation. Maximizing the expected reward is equivalent to minimizing the following loss:

[0108]

[0109] in, Let N represent the logarithmic probability of the nth candidate feature under the current policy distribution, and let N represent the number of candidate features. To enhance the loss, it is used to update the parameters of the entire network, including the model parameters that were not frozen in Steps 2-5;

[0110] The parameter update rule is as follows:

[0111]

[0112] in, These are the learnable parameters for the entire network. For learning rate, This represents the gradient of the corresponding learnable parameter;

[0113] During training, the lightweight policy head parameters and the shared lightweight decoder parameters are jointly updated using policy gradient loss, while the parameters of the dual-path hierarchical encoder and the large model encoder remain frozen. During the inference phase, noise-free baseline candidates are selected. By sharing a lightweight decoder reconstruction As the final fused image output.

[0114] Furthermore, Step 7 includes:

[0115] First, the infrared image and the visible light image to be processed are input into the optimized network;

[0116] Next, multi-scale visual features are extracted using the frozen dual-path hierarchical encoder and prior injection branch, and a multi-scale fusion feature dictionary is generated from top to bottom through a multi-source collaborative fusion module.

[0117] Then, through the chain semantic progressive modulation module based on the frozen large model encoder, semantic analysis is performed step by step from high-level features to low-level features, and the channel gating coefficient is passed to the next low-level feature for layer-by-layer modulation to obtain the chain semantically enhanced fusion feature.

[0118] Then, the policy mean is estimated by the lightweight policy head, and its noise-free benchmark candidate is reconstructed into the final fused image by a shared lightweight decoder;

[0119] Finally, the final fused image is output.

[0120] The present invention also provides an image fusion system based on semantic chain feedback and reinforcement learning, the system comprising: a module for executing the above-described image fusion method based on semantic chain feedback and reinforcement learning.

[0121] The beneficial effects of this invention are:

[0122] 1. Significantly improves the generalization and stability of fused representations: This invention proposes a multi-scale prior injection fusion framework based on a frozen DINOv3 visual base model with ConvNeXt-Base as the backbone network. Without retraining the backbone network, it directly introduces the hierarchical visual representations obtained from large-scale pre-training into the fusion process. By combining a top-down hierarchical progressive fusion path with a prior injection branch, it effectively compensates for the attenuation of low-level visual responses during deep encoding. Under the small data fusion setting, it retains as much of the generalized representations obtained by the base model on large-scale data as possible, significantly reducing the risk of overfitting and achieving stable fused feature representations with good cross-dataset transferability.

[0123] 2. Constructing a chain-like semantic progressive modulation mechanism for the fusion process: This invention proposes a chain-like semantic progressive modulation method based on a frozen Qwen3-VL-Embedding encoder. The frozen Qwen3-VL-Embedding encoder is embedded inside the fusion path. By performing hierarchical semantic parsing on the multi-scale intermediate fused images and mapping the semantic feedback into cross-layer modulation signals, the layer-by-layer closed-loop optimization of the fusion process is achieved. The coarse-to-fine chain-like progressive modulation enables high-level semantics, mid-level structure and low-level details to be synergistically enhanced in a unified representation space. This solves the bottleneck of semantic priors being difficult to intervene in the intermediate fusion process in traditional methods, and significantly improves the overall quality of the fused image in terms of target saliency, structural consistency and detail texture preservation.

[0124] 3. Breaking through the weighted loss paradigm with a difference-driven reinforcement learning mechanism: Image fusion is modeled as a policy optimization problem under reinforcement learning. By constructing candidate sampling and updating based on the policy gradient of relative advantage, an adaptive optimization mechanism driven by difference is realized, breaking through the deterministic training paradigm dominated by traditional weighted loss. The multidimensional composite reward function constructs implicit joint constraints in the form of a product, avoiding the extreme optimization of a single index from masking other defects. The relative advantage is driven by the relative comparison between candidates, which significantly reduces the dependence on fixed weights and improves the model's adaptability.

[0125] 4. Improve the adaptability and reliability of practical applications: The method of this invention can stably output a higher quality final fused image under different lighting conditions, different scene complexities and different downstream task requirements, and has good potential for practical applications. In applications such as visual target detection, semantic segmentation, autonomous driving perception and UAV target reconnaissance, this invention can provide a more reliable and detailed cross-modal fusion solution, which enhances the applicability and practicality of the system.

[0126] 5. This invention addresses the key challenges of infrared and visible light image fusion by unifying "chain-based semantic progressive modulation, candidate policy sampling, and reward-driven optimization" into an end-to-end fusion framework. This significantly enhances the guiding role of large-scale pre-trained priors for the fusion task, while achieving consistency between training and inference in an end-to-end manner. It solves the problems of insufficient utilization of visual priors and limited deterministic optimization expression in cross-modal fusion, and has important theoretical value and broad practical application prospects. Attached Figure Description

[0127] Figure 1 This is a comparison diagram of the overall process of the method of the present invention and existing traditional image fusion paradigms;

[0128] Subgraph (a) represents the existing paradigm, which uses a single encoder-decoder path and is trained by minimizing weighted loss; Subgraph (b) represents the fusion paradigm based on semantic chain feedback and reinforcement learning proposed in this invention, which uses a frozen DINOv3 dual-path hierarchical encoder for feature extraction, a frozen Qwen3-VL-Embedding encoder for chain semantic analysis, and is optimized by policy-based candidate feature sampling and reward maximization.

[0129] Figure 2 This is a diagram showing the overall network framework structure of the method of the present invention;

[0130] Figure 3 The figure shows the qualitative comparison experimental results of the method of the present invention on the M3FD dataset;

[0131] Figure 4 The figure shows the qualitative comparison experimental results of the method of the present invention on the MSRS dataset. Detailed Implementation

[0132] Example 1: An image fusion method based on semantic chain feedback and reinforcement learning, the method comprising the following specific steps:

[0133] Step 1: Obtain infrared and visible light image pairs and preprocess the input images;

[0134] Further, Step 1 includes:

[0135] The input infrared and visible light image pairs are normalized in terms of channels and size, and the brightness channels of single-channel infrared and visible light images are expanded into a three-channel format consistent with the backbone network requirements.

[0136] During the training phase, images are randomly cropped and normalized.

[0137] During the inference stage, the original resolution is preserved, and the visible light image is separated according to the YCbCr color space, where Y represents the luminance channel, Cb represents the blue differential channel, and Cr represents the red differential channel. Only the Y channel and the infrared image are sent to the fusion network. After fusion, the output is recombined with the Cb and Cr channels and converted back to the RGB color space, where RGB represents the red, green, and blue three-channel color space, to obtain the final fused image.

[0138] Step 2: Input the preprocessed infrared image and visible light image into the frozen dual-channel layered encoder to extract the multi-scale visual features of the infrared image and visible light image respectively.

[0139] The preprocessed infrared image and the visible light image are used as input priors to extract shallow prior features;

[0140] Furthermore, the dual-channel hierarchical encoder can adopt the DINOv3 dual-channel hierarchical encoder, where DINOv3 is a pre-trained visual base model, and the ConvNeXt-Base backbone network of DINOv3 is kept frozen, with ConvNeXt-Base being a convolutional neural network backbone structure.

[0141] Furthermore, Step 2 includes:

[0142] Step 2.1: Extract multi-scale visual features from the preprocessed infrared and visible light images into the frozen dual-channel layered encoder, specifically including:

[0143] First, input the infrared image With visible light images Each channel is expanded into three channels through a channel copy operation, resulting in... and The expression is , Indicates a channel copy operation;

[0144] Next, and The signals are fed into a dual-channel layered encoder with completely frozen parameters. Perform reasoning to extract multi-scale visual features ,in It is a low-level structural feature. These are mid-level semantic features. For high-level semantic features, For global semantic features;

[0145] The spatial resolution, channel dimensions, and uses of each layer's output are shown in Table 1.

[0146] Table 1 shows the hierarchical feature information of DINOv3.

[0147] Finally, the backbone network of the dual-channel hierarchical encoder remains frozen during training and inference and does not participate in parameter updates; the extracted multi-scale features serve as the unified feature basis for subsequent cross-modal fusion and semantically guided modulation.

[0148] During the downsampling process of deep encoders, the fine-grained brightness, edges, and local textures in the original image attenuate as the receptive field increases. To address this, this invention designs a priori injection branch to perform shallow feature modeling on the bimodal image in the input space, injecting low-level prior features into features at each scale to compensate for the gradually decaying original visual response during deep encoding.

[0149] Step 2.2: Extract shallow prior features from the input prior injection branch by combining the preprocessed infrared image and the visible light image, including:

[0150] First, and After being spliced ​​along the channel dimension, the data is fed into a shallow network consisting of two 3×3 convolutional layers. Each convolutional layer is followed by a normalization layer and an activation function, resulting in a basic prior feature representation with 32 channels. :

[0151]

[0152] Where Cat represents the concatenation operation along the channel dimension, B represents the batch size of the features, and H and W represent the height and width of the feature map, respectively.

[0153] Next, the basic prior features are represented. Interpolation is performed to a spatial size that adapts to features at various scales, followed by a mapping network consisting of 3×3 convolutions and 1×1 convolutions (both with GroupNorm normalization layers and LeakyReLU activation functions). Upgrade to the target number of channels at each scale to form hierarchical shallow prior features. ;

[0154]

[0155] in, Corresponding to 0, 1, 2, and 3, Stage 0 represents the original size hierarchical feature layer, where the feature size extracted from Stage 0 is the same as the original input image size. Stage 1, Stage 2, and Stage 3 represent the low-level, middle-level, and high-level hierarchical feature layers, respectively. Finally, the shallow prior features... The number of channels is kept consistent with the corresponding backbone network, which is used to compensate for the gradual decay of low-level visual responses during deep coding in subsequent fusion processes.

[0156] Step 3: Through the multi-source collaborative fusion module, cross-modal collaborative fusion is performed on infrared features, visible light features, upper-layer propagation features and shallow prior features in multi-scale visual features at each intermediate scale.

[0157] When performing cross-modal collaborative fusion, the fusion features are propagated from high to low levels along a top-down path, and then the fusion features of each scale are unified to the same number of channels and spatial resolution to form a multi-scale fusion feature dictionary.

[0158] Furthermore, in Step 3, the operation process of the multi-source collaborative fusion module includes:

[0159] First, for arbitrary scale Stage indexing The corresponding scale, multi-source collaborative fusion module Infrared features at the corresponding scale With visible light characteristics After being concatenated along the channel dimension, the data is fed into a cross-modal complementary modeling operator consisting of two 3×3 convolutional layers (a GroupNorm normalization layer and a LeakyReLU activation function). , transfer the channel from Mapped to :

[0160]

[0161] Where Cat represents the splicing operation along the channel dimension;

[0162] Next, if there are propagation characteristics at the next higher level... Then it is constructed from 1×1 convolutions. Feature alignment layer, which propagates features from the previous layer. Features at the current scale Align along the channel dimension and inject the current feature:

[0163]

[0164] in, Assignment operation for information transmission;

[0165] Then, the shallow prior features of the corresponding scale are... Inject the current representation to compensate for the attenuation of low-level visual information during stepwise downsampling:

[0166]

[0167] Then, a thinning operator consisting of a single 3×3 convolution layer is used. Reshape the features:

[0168]

[0169] Finally, a top-down progressive fusion path is constructed through a multi-source collaborative fusion module: starting from the semantically richest Stage3, it propagates layer by layer to Stage0; the fused features at each scale are refined through a channel compression layer. The unified mapping is to channel 128, and the complete path is as follows:

[0170] Stage 3 (coarse-grained semantics, 512 channels):

[0171]

[0172] Where Interp represents the bilinear interpolation operator, used to ensure the consistency of the spatial dimension of the features; An empty input representing the propagation characteristics of a layer above. Represents the feature space.

[0173] Stage 2 (Mid-level structure, 256 channels, introducing deep semantics):

[0174]

[0175] Stage 1 (Fine layer, 128 channels, introduces general features):

[0176]

[0177] Stage 0 (Restore to input resolution and inject low-level prior features):

[0178]

[0179] This yields a multi-scale fusion feature dictionary. All features are aligned to the input resolution [H, W], and the number of channels is uniformly 128, serving as the input basis for chained semantic progressive modulation and candidate policy sampling.

[0180] Step 4: Perform chained semantic enhancement on the multi-scale fused features using the chained semantic progressive modulation module:

[0181] Multi-scale fusion features are reconstructed step-by-step into intermediate fusion images using a shared lightweight decoder;

[0182] The intermediate fused image is then input into the frozen large model encoder in the chain semantic progressive modulation module for staged semantic analysis. The obtained continuous semantic embedding vector is linearly projected and fused with the joint global visual vector in the multi-scale visual features. After activation function processing, the channel gating coefficients are obtained. The channel gating coefficients are used to adaptively and differentially enhance the fused features of the next layer, forming a chain progressive semantic modulation process from coarse to fine. Finally, the chain semantically enhanced fused features are output.

[0183] Furthermore, the frozen large model encoder in the chained semantic progressive modulation module can be a frozen Qwen3-VL-Embedding encoder, where Qwen3-VL-Embedding is a multimodal semantic coding model;

[0184] Existing methods typically use multimodal semantic coding models for result evaluation or auxiliary prompts, limiting their role to outside the fusion process and making it difficult to directly participate in the optimization of intermediate features. This invention embeds a frozen Qwen3-VL-Embedding encoder within the fusion path, generating continuous modulation signals through layer-by-layer semantic analysis of intermediate states, thereby achieving dynamic guidance and progressive optimization of fused features. This invention uses a frozen Qwen3-VL-Embedding encoder as the semantic analysis module for intermediate fusion states. In the three semantic analysis stages of Stage 3, Stage 2, and Stage 1, the network first decodes the current features into an intermediate fused image, then feeds it into the frozen Qwen3-VL-Embedding encoder for staged semantic analysis. The resulting continuous semantic embedding vector, after being aligned with the DINOv3 joint global visual vector, is mapped to the channel gating coefficient of the next lower-level feature corresponding to Stage (k-1); when k=1, the modulated object is the feature corresponding to Stage 0.

[0185] Furthermore, in Step 4, the operation process of the chain-like semantic progressive modulation module includes:

[0186] Step 4.1: Initialize the chained modulation state: merging the multi-scale feature dictionary. The features of each layer in the chain modulation are used as the initial state, where the multi-scale fusion feature dictionary is denoted as 𝒟, and the initial state is denoted as 𝒟. Where k represents the Stage index that participates in semantic analysis and generates channel gating coefficients, and the values ​​are decremented by 3, 2, and 1. Stage0 is the output layer after modulation in stage k=1.

[0187] Step 4.2, Intermediate Image Fusion and Reconstruction: In the modulation stage corresponding to Stagek, the current state is... Compared with the original dual-modal infrared image and visible light images After concatenation along the channel dimension, the input is shared by a lightweight decoder. Reconstructed into an intermediate blended image And scale it to a fixed resolution. The input format was copied to three channels to adapt to the large model encoder frozen in the chained semantic progressive modulation module. ; This represents the fixed side length parameter of the large model encoder input frozen in the adaptation chain semantic progressive modulation module:

[0188]

[0189] Here, `clamp` represents the truncation function, used to limit the value from exceeding a preset upper bound, and `Repeat` represents the channel copy operation. This represents the bilinear interpolation operator;

[0190] It is worth noting that the frozen Qwen3-VL-Embedding encoder is used only for semantic analysis and does not participate in parameter updates; the shared lightweight decoder The parameters are optimized using the reward signal from candidate reconstruction, thereby achieving policy-driven feature learning.

[0191] Step 4.3, Chained Semantic Feedback: The semantic analysis is passed layer by layer from Stage 3 to Stage 2 to Stage 1, and the modulation result is then passed to Stage 0. Here, k takes values ​​of 3, 2, and 1, corresponding to the three semantic analysis stages of Stage 3, Stage 2, and Stage 1 respectively. Stage 0 does not perform semantic analysis using k alone, but receives the modulation result generated in the stage where k=1. In the modulation stage corresponding to Stage k, the frozen large model encoder Ψ receives the current intermediate fused image. The system uses fixed text commands and corresponding hierarchical prompts for Stagek, outputting continuous semantic embedding vectors. :

[0192]

[0193] Where, instruction represents a fixed text instruction, and prompt_k represents the hierarchical prompt text corresponding to Stagek;

[0194] Among them, Stage 3 prompts focus on global salient targets and structural consistency, Stage 2 prompts focus on boundaries and contrast, and Stage 1 prompts focus on detail texture and artifact suppression.

[0195] Step 4.4, Visual-Semantic Alignment and Chained Channel Modulation: Global semantic features of infrared and visible light images respectively. and Perform mean pooling along the token dimension and sum them to obtain a joint global visual vector. Wherein, token represents a single sequence unit in the global semantic features. and Global semantic features representing the corresponding position :

[0196]

[0197] semantic embedding vector By linear projection Aggregated feature vectors aligned to the same dimension :

[0198]

[0199] Add it to the joint global visual vector and project it. Mapped to channel gating coefficient :

[0200]

[0201] The channel gating coefficients are further obtained by centering the channel using the Sigmoid activation function. :

[0202]

[0203] in, This represents the Sigmoid activation function;

[0204] Will The next lower-level feature corresponding to Stage(k-1) is applied. Channel-level adaptive modulation is applied to it; when k=1, the modulated object is the feature corresponding to Stage0:

[0205]

[0206] in, This represents the Hadamard product, used for element-wise feature modulation.

[0207] The updated next low-level features will be further reconstructed into an intermediate fused image via a shared lightweight decoder and fed into a frozen Qwen3-VL-Embedding encoder for semantic analysis. This serves as the input for the next step of semantic perception and feature modulation, forming a continuous chain-like transmission relationship between layers, i.e.:

[0208]

[0209] Step 4.5: Output the fused features after chained semantic enhancement: Perform three chained modulations corresponding to k=3, k=2, and k=1 in sequence, and obtain the Stage 0 modulation result. Finally, the final chained semantically enhanced fused features are output. :

[0210] .

[0211] Therefore, semantic information is no longer injected all at once, but propagates and accumulates step by step along the cross-scale path, forming a chain-like modulation process, achieving gradual refinement from high-level semantics to low-level details; and the frozen Qwen3-VL-Embedding encoder is only used for semantic analysis and does not participate in parameter updates, sharing a lightweight decoder. The parameters are optimized through reward gradients to achieve policy-driven feature learning.

[0212] Step 5: The chain-semantic-enhanced fusion features are treated as states through the candidate policy sampling module. The baseline action mean is learned through a lightweight policy head composed of two 3×3 convolutions. A Gaussian policy distribution is constructed with a fixed standard deviation. Several candidate features are sampled from this distribution. The 0th candidate is a noise-free baseline candidate. The policy mean neighborhood of the remaining candidates is then randomly explored. The candidate features are concatenated with the original bimodal image and sent to the shared lightweight decoder to reconstruct several candidate fusion images.

[0213] Existing fusion methods typically generate a single fused image based on a weighted loss function, which struggles to characterize the potential solution space corresponding to different fusion strategies. This limits the ability to model fusion diversity and optimal trade-offs, and also incurs high manual parameter tuning costs. This invention models the final fused image generation step as a policy learning problem: it integrates the chained semantically enhanced fusion features... The model is treated as a state, and the mean of the baseline action is learned through a lightweight policy head. Multiple candidate fusion images are then sampled within their neighborhoods to construct a differential candidate set. Based on this differential set, the reward function evaluates the different candidates, thereby providing a relative comparison signal for policy optimization and driving the model to gradually converge to a high-quality fusion image.

[0214] Furthermore, Step 5 includes:

[0215] Step 5.1, Lightweight Strategy Head and Strategy Mean Estimation: Lightweight Strategy Head It consists of two 3×3 convolutional layers. The first convolutional layer is followed by a normalization layer and an activation function, while the second convolutional layer directly outputs the mean of the policy distribution as a residual. Make an estimate:

[0216]

[0217] The residual form ensures that the policy is close to the identity transformation during initialization, which helps with stability in the early stages of training;

[0218] Step 5.2, Construction of Gaussian Policy Distribution and Candidate Sampling: Construct a Gaussian policy distribution for the mean and a fixed standard deviation σ. and from the Gaussian strategy distribution Mid-sampling yields N candidate features:

[0219]

[0220] in These are noise-free baseline candidates used for inference output and baseline reward calculation. For the nth candidate feature, It is the identity matrix. Let n be the nth random noise; the remaining candidates, except for the 0th candidate, randomly explore the neighborhood of the policy mean, and explicitly construct a candidate set with differences;

[0221] Step 5.3, Calculate the log probability of the policy: For each candidate feature, calculate its log probability under the current policy distribution. ,in This represents the action in the current state, and normalization is performed by averaging the channel and spatial dimensions to eliminate the influence of feature map size on the numerical scale.

[0222]

[0223] Where C represents the channel dimension of the feature map, and H and W represent the height and width of the feature map, respectively. These are elements corresponding to the passage and spatial location. This represents the log probability of the nth candidate feature under the current policy distribution. The log probability characterizes the likelihood of generating a candidate feature under the current policy distribution. In the reinforcement learning framework, it serves as a key term of the policy gradient and works together with the advantage function to feed the reward signal back to the policy distribution.

[0224] Step 5.4, Candidate Fusion Image Reconstruction with Shared Lightweight Decoder: Shared Lightweight Decoder This process is repeated at this stage, with N candidate features... The unified decoding reconstructs candidate fused images; the parameters of this shared lightweight decoder are shared and trained through reward gradients, which indirectly affects the quality of the intermediate fused images.

[0225] The features obtained by concatenating the candidate features with the original bimodal image are represented as follows: :

[0226]

[0227] Subsequently, the N candidate features are fed in parallel into the shared lightweight decoder. The candidate fused images are obtained as follows: This shared lightweight decoder consists of two 3×3 convolutional layers, each followed by a normalization layer and a LeakyReLU activation function to progressively extract fusion features; the last layer uses a 1×1 convolution to map the features to a single-channel output, directly generating N candidate fused images. :

[0228]

[0229] Finally, it is restored to a tensor shape of [B,N,1,H,W], where B represents the batch size, N represents the number of candidate features, and H and W represent the image height and width, respectively, to make full use of the parallel computing efficiency of the graphics processing unit (GPU).

[0230] Step 6: Optimize the unfrozen model parameters from Steps 2-5 using a multidimensional composite reward function and policy gradient algorithm; the optimization process is as follows:

[0231] A baseline is constructed based on the average reward of several candidates within the sample. The relative advantage of each candidate is calculated. The policy gradient loss is constructed by multiplying the relative advantage by the log probability, which drives the policy distribution to converge toward the high-quality fused image region.

[0232] Furthermore, the multidimensional composite reward function is composed of pixel response consistency reward, structural similarity reward, and gradient detail preservation reward in a product form;

[0233] The policy gradient algorithm is the REINFORCE policy gradient algorithm, which represents a policy gradient algorithm based on Monte Carlo return estimation.

[0234] Furthermore, Step 6 includes:

[0235] Step 6.1: Construct a multidimensional composite reward function in product form:

[0236]

[0237] The product form requires good performance across all dimensions to achieve a high reward, preventing extreme optimization in a single dimension from masking other shortcomings. This is equivalent to constructing implicit joint constraints without the need for manual weight setting. Representative fused image Total reward These represent the fused images in sequence. The rewards include pixel response consistency, structural similarity, and gradient detail preservation.

[0238] Step 6.2, Pixel Response Consistency Reward Pixel-wise maximum response map of two modes For reference, the fused image should inherit at least the brightness response of the stronger modality at each location:

[0239]

[0240] Where max represents the maximum value operation. Represents an exponential function. It is an L1 norm;

[0241] Step 6.3, Structural Similarity Reward The SSIM similarity values ​​between the candidate image and the two modalities are calculated separately, converted into rewards, and then averaged. A lower bound truncation ε is used to ensure the stability of the training values. Here, SSIM represents the structural similarity measure, and ε represents the lower bound truncation constant. Representative structural similarity loss:

[0242]

[0243]

[0244] Step 6.4, Gradient Details Retention Reward Gradient magnitudes are extracted using the Sobel operator, and the pixel-wise maximum gradients of the two modes are used. For reference, image fusion is encouraged to preserve the edge and structural details of the sharpest modality; where the Sobel operator represents a gradient extraction operator based on local difference. , For the corresponding gradient value:

[0245]

[0246]

[0247] Step 6.5, Advantage Value Estimation and Policy Gradient Optimization: Using the average reward of N candidates within the same sample. Using this as a baseline, calculate the relative advantage of each candidate. :

[0248]

[0249] This baseline transforms absolute rewards into relative evaluations, reducing variance while maintaining the unbiasedness of policy gradients. When the relative advantage value A is greater than 0, where A represents the relative advantage value, the corresponding candidate is strengthened and its sampling probability is increased; otherwise, it is suppressed.

[0250] Step 6.6, Policy Gradient Loss Optimization: The policy gradient loss is optimized using the REINFORCE policy gradient objective, where REINFORCE represents the policy gradient algorithm based on Monte Carlo reward estimation. Maximizing the expected reward is equivalent to minimizing the following loss:

[0251]

[0252] in, Let N represent the logarithmic probability of the nth candidate feature under the current policy distribution, and let N represent the number of candidate features. To enhance the loss, it is used to update the parameters of the entire network, including the model parameters that were not frozen in Steps 2-5;

[0253] The parameter update rule is as follows:

[0254]

[0255] in, These are the learnable parameters for the entire network. For learning rate, This represents the gradient of the corresponding learnable parameter;

[0256] During training, the lightweight policy head parameters and the shared lightweight decoder parameters are jointly updated using policy gradient loss, while the parameters of the dual-path hierarchical encoder and the large model encoder remain frozen. During the inference phase, noise-free baseline candidates are selected. By sharing a lightweight decoder reconstruction As the final fused image output.

[0257] Step 7: Input the infrared image and visible light image to be processed into the optimized network, and select the noise-free benchmark candidate as the final fused image output.

[0258] Furthermore, Step 7 includes:

[0259] First, the infrared image and the visible light image to be processed are input into the optimized network;

[0260] Next, multi-scale visual features are extracted using the frozen dual-path hierarchical encoder and prior injection branch, and a multi-scale fusion feature dictionary is generated from top to bottom through a multi-source collaborative fusion module.

[0261] Then, through the chain semantic progressive modulation module based on the frozen large model encoder, semantic analysis is performed step by step from high-level features to low-level features, and the channel gating coefficient is passed to the next low-level feature for layer-by-layer modulation to obtain the chain semantically enhanced fusion feature.

[0262] Then, the policy mean is estimated by the lightweight policy head, and its noise-free benchmark candidate is reconstructed into the final fused image by a shared lightweight decoder;

[0263] Finally, the final fused image is output.

[0264] Unlike existing methods that place the multimodal semantic coding model after the generation of the fused image for posterior evaluation, or use it only as an external textual cue condition, this invention embeds the frozen Qwen3-VL-Embedding encoder into the fusion feature formation process, constructing a closed-loop mechanism consisting of visual fusion, semantic analysis, difference enhancement, candidate generation, and reward optimization.

[0265] Furthermore, this embodiment also provides an image fusion system based on semantic chain feedback and reinforcement learning, the system comprising:

[0266] The image acquisition and preprocessing module is used to acquire infrared and visible light image pairs, and to preprocess the input images, including unifying image size and standardizing and normalizing channels.

[0267] The frozen dual-path hierarchical feature extraction module is used to construct a frozen DINOv3 dual-path hierarchical encoder and a priori injection branch, which extracts multi-scale visual features from infrared and visible light images respectively, and compensates for low-level spatial details for deep encoding through the priori injection branch.

[0268] The multi-source collaborative fusion module is used to perform cross-modal complementary modeling and collaborative fusion of infrared features, visible light features, upper-level propagation features and lower-level prior features at various scales, and propagates step by step along a top-down path to obtain a multi-scale fused feature dictionary.

[0269] The chain-like semantic progressive modulation module is used to reconstruct the multi-scale fusion features into intermediate fusion images step by step, and feed them into the frozen Qwen3-VL-Embedding encoder for staged semantic analysis; and after fusing the semantic embedding vector with the DINOv3 joint global visual vector, it is mapped into channel gating coefficients to differentially enhance the next low-level features.

[0270] The candidate policy sampling module is used to take the fused features after chained semantic enhancement as the state, learn the mean of the baseline action through a lightweight policy head, construct a Gaussian policy distribution with a fixed standard deviation, and sample N candidate features from the distribution.

[0271] A shared lightweight decoder module is used to concatenate N candidate features with the original bimodal image and then decode and reconstruct N candidate fused images in parallel. During the chain-like semantic modulation process, intermediate state features are reconstructed into intermediate fused images for semantic analysis by the Qwen3-VL-Embedding encoder.

[0272] The multidimensional composite reward and policy gradient optimization module is used to construct a multidimensional composite reward function in the form of a product of pixel response consistency reward, structural similarity reward and gradient detail preservation reward; the relative advantage is calculated with the in-sample candidate average reward as the baseline, and the end-to-end parameter update of the lightweight policy head and the shared lightweight decoder is performed based on the REINFORCE policy gradient loss.

[0273] The training module is used to achieve end-to-end training through the above-mentioned multi-dimensional composite reward and policy gradient optimization module. The parameters of DINOv3 and Qwen3-VL-Embedding are always frozen, and only the learnable projection, lightweight policy head and shared lightweight decoder in the multi-source collaborative fusion module and chained semantic progressive modulation module are updated.

[0274] The generation module is used to input the infrared and visible light images to be processed into the optimized network and select noise-free benchmark candidates as the final fused image output.

[0275] This invention innovatively embeds a frozen large-scale visual foundation model and multimodal semantic priors into the fusion process and constructs a difference-driven reinforcement learning optimization mechanism. First, based on the frozen DINOv3 visual foundation model with ConvNeXt-Base as the backbone network, a DINOv3 dual-path hierarchical encoder and a prior injection branch are constructed. A top-down hierarchical progressive fusion path is used to perform cross-modal collaborative fusion of infrared and visible light features, resulting in a multi-scale fused feature dictionary. Second, a frozen Qwen3-VL-Embedding encoder is embedded into the fusion path, performing step-by-step chain-like semantic analysis on intermediate fusion states. The semantic embedding vectors are mapped to channel gating coefficients to achieve differential enhancement of the next lower-level features. Finally, the fusion is modeled as a policy learning problem. Candidate sampling and updates based on the relative advantage of the REINFORCE policy gradient are implemented to achieve a difference-driven adaptive optimization mechanism, robustly improving the visual quality of the fused image and the performance of downstream tasks without requiring manual weight setting.

[0276] To verify the effectiveness of the method of this invention, training and evaluation sets are constructed using the MSRS dataset and the M3FD dataset, both of which are publicly available benchmark datasets for infrared and visible light image fusion. During the training phase, random cropping and normalization are used as data augmentation strategies. The parameters of the DINOv3 visual base model with ConvNeXt-Base as the backbone network and the Qwen3-VL-Embedding encoder are frozen, and only the multi-source collaborative fusion module, semantic-visual projection parameters, lightweight policy head, and shared lightweight decoder are updated. The number of candidates, N, is set to 4, where N represents the number of candidate features. The standard deviation σ of the Gaussian distribution of the policy is set to 0.1, where σ is a fixed standard deviation. The default value is 256, which corresponds to the fixed side length parameter of the semantic encoder input; the Adam optimizer is used, which is an adaptive moment estimation optimization algorithm, and the initial learning rate is set to 0.0001.

[0277] To further explain, this invention employs mutual information (Q) when evaluating image fusion quality. MI Gradient-based fusion evaluation index (Q) AB / F Visual information fidelity (Q) VIF ), based on feature contrast metrics (Q cv ), based on local structural similarity index (Q y ) and structural similarity measure (Q SSIMThe quality of the fused image is evaluated using key indicators such as LUT-Fuse, DATFuse, FreeFusion, ISFuse, MetaLFusion, SMLNet, TIMFusion, and U2Fusion in the table. Ours represents the method of this invention.

[0278] Among them, Q MI This value measures the amount of information from the source images contained in the fused image; a higher value indicates that the fused image retains more effective information from the source images. AB / F Q reflects the ability of the fusion process to preserve edge structure information of the source image; a higher value indicates that the edge information is preserved more completely. VIF Used to evaluate the degree to which the fused image retains the features of the source image at the visual perception level; a higher value indicates that the fused image is visually closer to the source image. cv As a contrast evaluation index based on the characteristics of the human visual system, Q characterizes the contrast loss by calculating the local contrast difference between the fused image and the source image; y It reflects the degree of similarity between the fused image and the source image in local region structure, and can effectively characterize the structural integrity of the fused image; Q SSIM This reflects the comprehensive similarity between the fused image and the source image in terms of brightness, contrast, and structure. Up indicates a higher index, corresponding to better method performance; down indicates a lower index, corresponding to better method performance; bold black text represents the optimal value of the current index; and underlined text represents the second-best value of the current index.

[0279] Table 2 shows the quantitative comparison experimental results of this invention on multiple datasets.

[0280] On the MSRS and M3FD datasets, the performance of our proposed method (Ours) is compared with that of existing representative methods such as LUT-Fuse, DATFuse, FreeFusion, ISFuse, MetaLFusion, SMLNet, TIMFusion, and U2Fusion. Our proposed method achieves superior performance across all metrics on the aforementioned datasets, particularly in information preservation, structure preservation, and visual quality, where it significantly outperforms the comparative methods.

[0281] Figure 3 and Figure 4The results of qualitative comparative experiments on the MSRS and M3FD datasets of this invention, compared with other methods, demonstrate that the method of this invention can obtain higher quality fused images compared to existing methods, and effectively alleviates the problems of thermal target weakening, texture loss, and edge artifacts in the fused images. These experimental results fully verify the effectiveness of the method of this invention under different datasets and scene conditions, showing good fusion performance and excellent cross-dataset generalization ability, further demonstrating the application potential and practical value of this invention in infrared and visible light image fusion tasks.

[0282] The specific embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention.

Claims

1. An image fusion method based on semantic chain feedback and reinforcement learning, characterized in that: The method includes: After preprocessing the input infrared and visible light image pairs, the input is frozen into a dual-channel layered encoder and a prior injection branch to extract multi-scale visual features and shallow prior features, respectively. The multi-source collaborative fusion module performs cross-modal collaborative fusion of infrared features, visible light features, upper-layer propagation features and shallow prior features in multi-scale visual features at various intermediate scales. Chain-based semantic enhancement is performed on multi-scale fused features; Then acquire several candidate fusion images; Optimize the parameters of the unfrozen model; The infrared image and the visible light image to be processed are input into the optimized network, and the optimized network is used for the fusion processing of the infrared image and the visible light image.

2. The image fusion method based on semantic chain feedback and reinforcement learning driven by claim 1, characterized in that: The method includes the following specific steps: Step 1: Obtain infrared and visible light image pairs and preprocess the input images; Step 2: Input the preprocessed infrared image and visible light image into the frozen dual-channel layered encoder to extract the multi-scale visual features of the infrared image and visible light image respectively. The preprocessed infrared image and the visible light image are used as input priors to extract shallow prior features; Step 3: Through the multi-source collaborative fusion module, cross-modal collaborative fusion is performed on infrared features, visible light features, upper-layer propagation features and shallow prior features in multi-scale visual features at each intermediate scale. When performing cross-modal collaborative fusion, the fusion features are propagated from high to low levels along a top-down path, and then the fusion features of each scale are unified to the same number of channels and spatial resolution to form a multi-scale fusion feature dictionary. Step 4: Perform chained semantic enhancement on the multi-scale fused features using the chained semantic progressive modulation module: Multi-scale fusion features are reconstructed step-by-step into intermediate fusion images using a shared lightweight decoder; The intermediate fused image is then input into the frozen large model encoder in the chain semantic progressive modulation module for staged semantic analysis. The obtained continuous semantic embedding vector is linearly projected and fused with the joint global visual vector in the multi-scale visual features. After activation function processing, the channel gating coefficients are obtained. The channel gating coefficients are used to adaptively and differentially enhance the fused features of the next layer, forming a chain progressive semantic modulation process from coarse to fine. Finally, the chain semantically enhanced fused features are output. Step 5: The chain-semantic-enhanced fusion features are treated as states through the candidate policy sampling module. The baseline action mean is learned through a lightweight policy head composed of two 3×3 convolutions. A Gaussian policy distribution is constructed with a fixed standard deviation. Several candidate features are sampled from this distribution. The 0th candidate is a noise-free baseline candidate. The policy mean neighborhood of the remaining candidates is then randomly explored. The candidate features are concatenated with the original bimodal image and sent to the shared lightweight decoder to reconstruct several candidate fusion images. Step 6: Optimize the unfrozen model parameters from Steps 2-5 using a multidimensional composite reward function and policy gradient algorithm; the optimization process is as follows: A baseline is constructed based on the average reward of several candidates within the sample. The relative advantage of each candidate is calculated. The policy gradient loss is constructed by multiplying the relative advantage by the log probability, which drives the policy distribution to converge toward the high-quality fused image region. Step 7: Input the infrared image and visible light image to be processed into the optimized network, and select the noise-free benchmark candidate as the final fused image output.

3. The image fusion method based on semantic chain feedback and reinforcement learning driven by claim 2, characterized in that: Step 1 includes: The input infrared and visible light image pairs are normalized in terms of channels and size, and the brightness channels of single-channel infrared and visible light images are expanded into a three-channel format consistent with the backbone network requirements. During the training phase, images are randomly cropped and normalized. During the inference stage, the original resolution is preserved, and the visible light image is separated according to the YCbCr color space, where Y represents the luminance channel, Cb represents the blue differential channel, and Cr represents the red differential channel. Only the Y channel and the infrared image are sent to the fusion network. After fusion, the output is recombined with the Cb and Cr channels and converted back to the RGB color space, where RGB represents the red, green, and blue three-channel color space, to obtain the final fused image.

4. The image fusion method based on semantic chain feedback and reinforcement learning driven by claim 2, characterized in that: Step 2 includes: Step 2.1: Extract multi-scale visual features from the preprocessed infrared and visible light images into the frozen dual-channel layered encoder, specifically including: First, input the infrared image With visible light images Each channel is expanded into three channels through a channel copy operation, resulting in... and The expression is , Indicates a channel copy operation; Next, and The signals are fed into a dual-channel layered encoder with completely frozen parameters. Perform reasoning to extract multi-scale visual features ,in It is a low-level structural feature. These are mid-level semantic features. For high-level semantic features, For global semantic features; Finally, the backbone network of the dual-channel hierarchical encoder remains frozen throughout the training and inference process and does not participate in parameter updates. Step 2.2: Extract shallow prior features from the input prior injection branch by combining the preprocessed infrared image and the visible light image, including: First, and After being spliced ​​along the channel dimension, the data is fed into a shallow network consisting of two 3×3 convolutional layers. Each convolutional layer is followed by a normalization layer and an activation function, resulting in a basic prior feature representation with 32 channels. : ； Where Cat represents the concatenation operation along the channel dimension, B represents the batch size of the features, and H and W represent the height and width of the feature map, respectively. Next, the basic prior features are represented. Interpolate to a spatial size that adapts to features at various scales, and then pass it through a mapping network consisting of 3×3 convolutions and 1×1 convolutions. Upgrade to the target number of channels at each scale to form hierarchical shallow prior features. ; ； in, Corresponding to 0, 1, 2, and 3, Stage 0 represents the original size hierarchical feature layer, where the feature size extracted from Stage 0 is the same as the original input image size. Stage 1, Stage 2, and Stage 3 represent the low-level, middle-level, and high-level hierarchical feature layers, respectively. Finally, the shallow prior features... The number of channels is kept consistent with the corresponding backbone network, which is used to compensate for the gradual decay of low-level visual responses during deep coding in subsequent fusion processes.

5. The image fusion method based on semantic chain feedback and reinforcement learning driven by claim 2, characterized in that: In Step 3, the operation process of the multi-source collaborative fusion module includes: First, for arbitrary scale Stage indexing The corresponding scale, multi-source collaborative fusion module Infrared features at the corresponding scale With visible light characteristics After being concatenated along the channel dimension, the data is fed into a cross-modal complementary modeling operator consisting of two 3×3 convolutional layers. , transfer the channel from Mapped to : ； Where Cat represents the splicing operation along the channel dimension; Next, if there are propagation characteristics at the next higher level... Then it is constructed from 1×1 convolutions. Feature alignment layer, which propagates features from the previous layer. Features at the current scale Align along the channel dimension and inject the current feature: ； in, Assignment operation for information transmission; Then, the shallow prior features of the corresponding scale are... Inject the current representation to compensate for the attenuation of low-level visual information during stepwise downsampling: ； Then, a thinning operator consisting of a single 3×3 convolution layer is used. Reshape the features: ； Finally, a top-down progressive fusion path is constructed through a multi-source collaborative fusion module: starting from the semantically richest Stage3, it propagates layer by layer to Stage0; the fused features at each scale are refined through a channel compression layer. A unified mapping is performed to 128 channels, thus obtaining a multi-scale fusion feature dictionary.

6. The image fusion method based on semantic chain feedback and reinforcement learning driven by claim 2, characterized in that: In Step 4, the operation process of the chain-like semantic progressive modulation module includes: Step 4.1: Initialize the chained modulation state: merging the multi-scale feature dictionary. The features of each layer in the chain modulation are used as the initial state, where the multi-scale fusion feature dictionary is denoted as 𝒟, and the initial state is denoted as 𝒟. Where k represents the Stage index that participates in semantic analysis and generates channel gating coefficients, and the values ​​are decremented by 3, 2, and 1. Stage0 is the output layer after modulation in stage k=1. Step 4.2, Intermediate Image Fusion and Reconstruction: In the modulation stage corresponding to Stagek, the current state is... Compared with the original dual-modal infrared image and visible light images After concatenation along the channel dimension, the input is shared by a lightweight decoder. Reconstructed into an intermediate blended image And scale it to a fixed resolution. The input format was copied to three channels to adapt to the large model encoder frozen in the chained semantic progressive modulation module. ; This represents the fixed side length parameter of the large model encoder input frozen in the adaptation chain semantic progressive modulation module: ； Here, `clamp` represents the truncation function, used to limit the value from exceeding a preset upper bound, and `Repeat` represents the channel copy operation. This represents the bilinear interpolation operator; Step 4.3, Chained Semantic Feedback: The semantic analysis is passed layer by layer from Stage 3 to Stage 2 to Stage 1, and the modulation result is then passed to Stage 0. Here, k takes values ​​of 3, 2, and 1, corresponding to the three semantic analysis stages of Stage 3, Stage 2, and Stage 1 respectively. Stage 0 does not perform semantic analysis using k alone, but receives the modulation result generated in the stage where k=1. In the modulation stage corresponding to Stage k, the frozen large model encoder Ψ receives the current intermediate fused image. The system uses fixed text commands and corresponding hierarchical prompts for Stagek, outputting continuous semantic embedding vectors. : ； Where, instruction represents a fixed text instruction, and prompt_k represents the hierarchical prompt text corresponding to Stagek; Among them, Stage 3 prompts focus on global salient targets and structural consistency, Stage 2 prompts focus on boundaries and contrast, and Stage 1 prompts focus on detail texture and artifact suppression. Step 4.4, Visual-Semantic Alignment and Chained Channel Modulation: Global semantic features of infrared and visible light images respectively. and Perform mean pooling along the token dimension and sum them to obtain a joint global visual vector. Wherein, token represents a single sequence unit in the global semantic features. and Global semantic features representing the corresponding position : ； semantic embedding vector By linear projection Aggregated feature vectors aligned to the same dimension : ； Add it to the joint global visual vector and project it. Mapped to channel gating coefficient : ； The channel gating coefficients are further obtained by centering the channel using the Sigmoid activation function. : ； in, This represents the Sigmoid activation function; Will The next lower-level feature corresponding to Stage(k-1) is applied. Channel-level adaptive modulation is applied to it; when k=1, the modulated object is the feature corresponding to Stage0: ； in, This represents the Hadamard product, used for element-wise feature modulation. Step 4.5: Output the fused features after chained semantic enhancement: Perform three chained modulations corresponding to k=3, k=2, and k=1 in sequence, and obtain the Stage 0 modulation result. Finally, the final chained semantically enhanced fused features are output. : 。 7. The image fusion method based on semantic chain feedback and reinforcement learning driven by claim 2, characterized in that: Step 5 includes: Step 5.1, Lightweight Strategy Head and Strategy Mean Estimation: Lightweight Strategy Head It consists of two 3×3 convolutional layers. The first convolutional layer is followed by a normalization layer and an activation function, while the second convolutional layer directly outputs the mean of the policy distribution as a residual. Make an estimate: ； Step 5.2, Construction of Gaussian Policy Distribution and Candidate Sampling: Construct a Gaussian policy distribution for the mean and a fixed standard deviation σ. and from the Gaussian strategy distribution Mid-sampling yields N candidate features: ； in These are noise-free baseline candidates used for inference output and baseline reward calculation. For the nth candidate feature, It is the identity matrix. Let n be the nth random noise; the remaining candidates, except for the 0th candidate, randomly explore the neighborhood of the policy mean, and explicitly construct a candidate set with differences; Step 5.3, Calculate the log probability of the policy: For each candidate feature, calculate its log probability under the current policy distribution. ,in This represents the action in the current state, and is normalized by averaging the channel and spatial dimensions: ； Where C represents the channel dimension of the feature map, and H and W represent the height and width of the feature map, respectively. These are elements corresponding to the passage and spatial location. This represents the log probability of the nth candidate feature under the current policy distribution; Step 5.4, Candidate Fusion Image Reconstruction with Shared Lightweight Decoder: Shared Lightweight Decoder This process is repeated at this stage, with N candidate features... The unified decoding reconstructs candidate fused images; the parameters of this shared lightweight decoder are shared and trained through reward gradients, which indirectly affects the quality of the intermediate fused images. The features obtained by concatenating the candidate features with the original bimodal image are represented as follows: : ； Subsequently, the N candidate features are fed in parallel into the shared lightweight decoder. The candidate fused images are obtained as follows: This shared lightweight decoder consists of two 3×3 convolutional layers, each followed by a normalization layer and a LeakyReLU activation function to progressively extract fusion features; the last layer uses a 1×1 convolution to map the features to a single-channel output, directly generating N candidate fused images. : ； Finally, it is restored to a tensor shape of [B,N,1,H,W], where B represents the batch size, N represents the number of candidate features, and H and W represent the image height and width, respectively.

8. The image fusion method based on semantic chain feedback and reinforcement learning driven by claim 2, characterized in that: Step 6 includes: Step 6.1: Construct a multidimensional composite reward function in product form: ； Implicit joint constraints are constructed without the need for manual weight setting; where Representative fused image Total reward These represent the fused images in sequence. The rewards include pixel response consistency, structural similarity, and gradient detail preservation. Step 6.2, Pixel Response Consistency Reward Pixel-wise maximum response map of two modes For reference, the fused image should inherit at least the brightness response of the stronger modality at each location: ； Where max represents the maximum value operation. Represents an exponential function. It is an L1 norm; Step 6.3, Structural Similarity Reward The SSIM similarity values ​​between the candidate image and the two modalities are calculated separately, converted into rewards, and then averaged. A lower bound truncation ε is used to ensure the stability of the training values. Here, SSIM represents the structural similarity measure, and ε represents the lower bound truncation constant. Representative structural similarity loss: ； ； Step 6.4, Gradient Details Retention Reward Gradient magnitudes are extracted using the Sobel operator, and the pixel-wise maximum gradients of the two modes are used. For reference, image fusion is encouraged to preserve the edge and structural details of the sharpest modality; where the Sobel operator represents a gradient extraction operator based on local difference. , For the corresponding gradient value: ； ； Step 6.5, Advantage Value Estimation and Policy Gradient Optimization: Using the average reward of N candidates within the same sample. Using this as a baseline, calculate the relative advantage of each candidate. : ； Step 6.6, Policy Gradient Loss Optimization: The policy gradient loss is optimized using the REINFORCE policy gradient objective, where REINFORCE represents the policy gradient algorithm based on Monte Carlo reward estimation. Maximizing the expected reward is equivalent to minimizing the following loss: ； in, Let N represent the logarithmic probability of the nth candidate feature under the current policy distribution, and let N represent the number of candidate features. To enhance the loss, it is used to update the parameters of the entire network, including the model parameters that were not frozen in Steps 2-5; The parameter update rule is as follows: ； in, These are the learnable parameters for the entire network. For learning rate, This represents the gradient of the corresponding learnable parameter; During training, the lightweight policy head parameters and the shared lightweight decoder parameters are jointly updated using policy gradient loss, while the parameters of the dual-path hierarchical encoder and the large model encoder remain frozen. During the inference phase, noise-free baseline candidates are selected. By sharing a lightweight decoder reconstruction As the final fused image output.

9. The image fusion method based on semantic chain feedback and reinforcement learning driven by claim 2, characterized in that: Step 7 includes: First, the infrared image and the visible light image to be processed are input into the optimized network; Next, multi-scale visual features are extracted using the frozen dual-path hierarchical encoder and prior injection branch, and a multi-scale fusion feature dictionary is generated from top to bottom through a multi-source collaborative fusion module. Then, through the chain semantic progressive modulation module based on the frozen large model encoder, semantic analysis is performed step by step from high-level features to low-level features, and the channel gating coefficient is passed to the next low-level feature for layer-by-layer modulation to obtain the chain semantically enhanced fusion feature. Then, the policy mean is estimated by the lightweight policy head, and its noise-free benchmark candidate is reconstructed into the final fused image by a shared lightweight decoder; Finally, the final fused image is output.

10. An image fusion system based on semantic chain feedback and reinforcement learning, characterized in that, The system includes a module for executing the image fusion method based on semantic chain feedback and reinforcement learning as described in any one of claims 1 to 9.

Images