An image fusion controllable modal modulation method based on classifier-free guidance

Abstract

The application belongs to the technical field of image fusion, and proposes an image fusion controllable modal modulation method based on non-classifier guidance: firstly, a double-branch neural network containing an unconditional reconstruction branch and a conditional fusion branch is constructed, the unconditional branch is used for high-quality fidelity reconstruction of the source image to suppress pre-background noise in the absence of conditional modal guidance, and the conditional branch is used for extracting features from the source image and the conditional modal image, and performing dynamic modulation and fusion in a multi-scale space; secondly, a three-stage joint training strategy is used to train the constructed double-branch neural network; then, in the inference stage, the source image, the conditional modal image and the set guide scale factor are received, unconditional prediction and conditional prediction are performed respectively, and continuous controllable modal injection without retraining is realized through a linear extrapolation formula, and the final fused image is output. The application effectively improves the precision and flexibility of the existing image fusion algorithm in the integration of heterogeneous modal features.

Description

Technical Field

[0001] This invention relates to the field of image fusion technology, and in particular to a classifier-free image fusion controllable mode modulation method. Background Technology

[0002] In complex and dynamic scenarios such as all-weather autonomous driving and security monitoring, single vision sensors often face severe perception bottlenecks: visible light images have rich textures but are easily affected by poor lighting conditions, while infrared images can accurately highlight heat sources but lack scene context. Therefore, multimodal image fusion technology has become the key to breaking through the limits of all-weather perception.

[0003] Early deep learning fusion solutions largely relied on CNNs or GANs for end-to-end direct deterministic mapping. This paradigm is prone to losing high-frequency details or producing unnatural artifacts. In recent years, the introduction of diffusion models has brought about a revolutionary paradigm shift in this field. It abandons traditional direct mapping and implicitly models the prior distribution of high-quality fused images through a progressive denoising mechanism. This generative mechanism endows the model with the ability to deeply understand and reconstruct deep structures, high-frequency textures, and multimodal energy distributions during the noise recovery process, thereby generating highly natural, artifact-free, and detail-rich fused images.

[0004] However, existing heterogeneous modality fusion methods based on diffusion models are generally limited by the rigidity of modal condition injection. In these existing methods, the injection intensity of conditional modalities (such as infrared features) is fixed during the training phase, forming a static coupling mapping. Once the model is trained, it cannot flexibly adjust the fusion intensity during the inference phase according to the dynamic and changing scene requirements (such as sudden changes in lighting during day and night, or the prominence of specific high-value targets), which severely restricts the algorithm's adaptability in real-world complex environments. Summary of the Invention

[0005] The purpose of this invention is to provide a classifier-free image fusion controllable modal modulation method, which can dynamically and adaptively adjust the network fusion intensity during image fusion using a dual-branch neural network, so that the model does not need any retraining during the inference stage, and can endow the fusion process with a high degree of scene adaptability and subjective controllability.

[0006] The technical solution adopted by this invention to solve its technical problem is as follows:

[0007] A classifier-free image fusion controllable modal modulation method includes the following steps:

[0008] A two-branch neural network is constructed, comprising an unconditional reconstruction branch and a conditional fusion branch. The unconditional reconstruction branch is a basic image feature reconstruction branch independent of the conditional modality, used to perform high-quality, high-fidelity reconstruction of the source image and pre-background noise suppression when the conditional modality guidance is lacking. The conditional fusion branch is a condition-based cross-modal feature adaptive injection branch, used to extract features from the source image and the conditional modality image and perform dynamic modulation and fusion in a multi-scale space.

[0009] A three-stage joint training strategy is used to train the constructed dual-branch neural network;

[0010] During the inference phase, the source image, conditional modality image, and set guiding scale factor are received. Unconditional prediction and conditional prediction are performed respectively, and continuous controllable modality injection without retraining is achieved through linear extrapolation formula, outputting the final fused image.

[0011] In some embodiments, the dual-branch neural network employs a dual-branch encoder-decoder architecture comprising an independent visual image feature extraction backbone and a conditional feature extraction backbone.

[0012] In the feature transfer process of the bottleneck layer and decoder of the dual-branch neural network, a fine-grained spatial conditional attention mechanism is used to modulate the backbone features with conditional features.

[0013] In some embodiments, the use of a fine-grained spatial conditional attention mechanism to modulate the backbone features using conditional features includes the following steps:

[0014] Spatial alignment of the input conditional features;

[0015] Spatial dimensional affine transformation parameters γ and β are generated through spatial convolution mapping;

[0016] The backbone features are modulated using the FiLM mechanism, and a feature map is output. The formula for calculating the output feature map is as follows:

[0017] ;

[0018] in, Indicates the characteristics of the backbone network. This represents the conditional features after alignment. This indicates element-wise multiplication.

[0019] In some embodiments, the three-stage joint training strategy includes: unconditional reconstruction pre-training in the first stage, conditional dropout training based on random masks in the second stage, and joint fine-tuning of pre-feature orthogonal constraint training in the third stage.

[0020] In some embodiments, the unconditional reconstruction pre-training in the first stage refers to: training the unconditional reconstruction branch using natural images, with conditional modality images not participating in the training, and introducing learnable empty conditional features, which are then used to replace the real conditional modality features for dimensional expansion and spatial alignment.

[0021] The second stage of conditional dropout training based on random masks refers to: establishing a conditional dropout mechanism based on random masks, generating unconditional masks and conditional masks with preset probabilities, and simultaneously performing weighted calculations of unconditional reconstruction loss and conditional fusion loss on the same batch of data;

[0022] The third stage of joint fine-tuning and orthogonal constraint training refers to the following: during the forward propagation of the network, unconditional features and conditional features are output simultaneously. By blocking the gradient backpropagation of unconditional features, the feature increment between conditional and unconditional features is calculated. Orthogonal constraint loss is introduced to minimize the cosine similarity between the unconditional feature and the feature increment, so that the conditional branch focuses on learning the unique incremental information of the conditional mode. The network is then globally jointly optimized by combining noise loss, reconstruction loss and fusion loss.

[0023] In some embodiments, the unconditional reconstruction pre-training in the first stage is implemented using the following steps:

[0024] Set the input condition to an all-zero tensor and train only the unconditional reconstruction branch;

[0025] The parameters of the backbone network are updated by calculating the reconstruction loss between the denoised output and the original image features.

[0026] In some embodiments, the second stage of conditional dropout training based on random masks is trained using a randomized binary crossover training strategy, and the implementation steps are as follows:

[0027] The probability of discarding under certain conditions is defined as follows: ;

[0028] Generate a boolean mask in each training iteration, so as to The probability of setting the input conditional features to null;

[0029] The total loss function for training a bi-branch neural network is a weighted sum of the unconditional reconstruction branch loss and the conditional fusion branch loss, expressed as:

[0030] ;

[0031] Where λ is the weight coefficient of the conditional fusion branch. Conditional fusion of branch losses, This represents the loss from unconditionally rebuilding the branch.

[0032] In some embodiments, the third stage of joint fine-tuning and feature orthogonal constraint training is implemented through the following steps:

[0033] The unconditional reconstruction branch and the conditional fusion branch are jointly optimized, and feature orthogonal loss is introduced to achieve feature decoupling;

[0034] For a given multi-source image, input it into a two-branch neural network for unconditional forward propagation and conditional forward propagation respectively to obtain unconditional multi-scale features. and conditional multiscale features ;

[0035] right Gradient truncation yields unconditional multiscale features. This allows the conditional branch to learn only the incremental information of the conditional mode and calculate the conditional feature increment. ;

[0036] Calculate the squared cosine similarity between the gradient-trunculated unconditional multi-scale features and the conditional feature increments, and use it as the orthogonal loss, denoted as . The calculation formula is as follows:

[0037] ;

[0038] Where CosSim represents the cosine similarity function.

[0039] In some embodiments, the third stage of joint fine-tuning and feature orthogonality constraint training has a total loss function of unconditional reconstruction loss. Conditional fusion loss and characteristic orthogonal loss The weighted sum of the terms, the formula for calculating the total loss function is:

[0040] ;

[0041] in, The weights for unconditional reconstruction loss, The weights for the conditional fusion loss, The weights are the features of the orthogonal loss.

[0042] In some embodiments, the step of receiving the source image, the conditional modality image, and a set guiding scale factor during the inference phase, performing unconditional prediction and conditional prediction respectively, and achieving continuous and controllable modality injection without retraining through a linear extrapolation formula to output the final fused image, refers to:

[0043] During the inference phase of the dual-branch neural network, source images and conditional modality images are received;

[0044] The unconditional prediction results are obtained by performing one unconditional prediction at a time using a two-branch neural network. And the conditional prediction result obtained from a single conditional prediction. ;

[0045] A linear extrapolation formula without a classifier guidance mechanism is used, and a set guidance scale factor s is introduced to combine features of the unconditional prediction results and the conditional prediction results. The linear extrapolation formula is expressed as follows:

[0046] ;

[0047] The salience of the conditional mode in the final fused image is continuously controlled by adjusting the size of s.

[0048] The beneficial effects of this invention are:

[0049] I. Achieving independent extraction and efficient fusion of bi-branch features:

[0050] This invention introduces learnable null-conditional features and a conditional compression-incentivized attention mechanism to achieve feature extraction and attention fusion for both unconditional and conditional generation paths. This mechanism effectively handles the differences in feature representation between unconditional and conditional states, ensuring accurate transmission and efficient integration of multimodal information in the dual-branch architecture.

[0051] II. Achieving dynamic controllability during the inference period:

[0052] This invention, by introducing a random conditional discarding mechanism during the training phase and combining it with a guiding scaling factor during the inference phase, completely breaks the barrier that the parameters of traditional fusion models are fixed once trained, and realizes continuous and flexible dynamic control of the injection intensity of different modalities without the need for retraining.

[0053] III. Eliminating feature redundancy and improving expression purity:

[0054] This invention applies orthogonal constraints to features during joint training, forcing conditional branches to focus on learning incremental information specific to the conditional modality (such as infrared). Therefore, it can effectively remove feature redundancy between multiple modalities, avoid interference from repetitive information, and significantly improve the purity of heterogeneous feature representation.

[0055] Therefore, while inheriting the excellent generation quality of the diffusion model, this invention can achieve "on-demand control" and "dynamic adaptation" in the field of heterogeneous image fusion, thus greatly expanding the application potential and deployment value of multimodal fusion algorithms in complex real-world environments. Attached Figure Description

[0056] Figure 1This is a flowchart of a classifier-free image fusion controllable mode modulation method according to Embodiment 1 of the present invention;

[0057] Figure 2 This is a schematic diagram of the model structure of the dual-branch neural network in Embodiment 1 of the present invention;

[0058] Figure 3 This is a schematic diagram of the final fused image effect after controlling the guiding scale factor to achieve controllable conditional injection in Embodiment 1 of the present invention;

[0059] Figure 4 This is a schematic diagram showing the comparison effect of the final fused image obtained on Roadscene using various existing image fusion methods and the method provided in Embodiment 1 in Embodiment 2 of the present invention. Detailed Implementation

[0060] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0061] Example 1

[0062] This embodiment provides a classifier-free image fusion controllable modal modulation method, the flowchart of which can be found in [link to flowchart]. Figure 1 The method may include the following steps:

[0063] S1. Construct a two-branch neural network containing an unconditional reconstruction branch and a conditional fusion branch. The unconditional reconstruction branch is a basic image feature reconstruction branch independent of the conditional modality, used to perform high-quality, high-fidelity reconstruction of the source image and pre-background noise suppression when there is a lack of conditional modality guidance. The conditional fusion branch is a condition-based cross-modal feature adaptive injection branch, used to extract features from the source image and the conditional modality image, and to perform dynamic modulation and fusion in a multi-scale space.

[0064] S2. A three-stage joint training strategy is used to train the constructed dual-branch neural network;

[0065] S3. During the inference phase, the source image, conditional modality image, and set guiding scale factor are received. Unconditional prediction and conditional prediction are performed respectively. Continuous and controllable modality injection without retraining is achieved through linear extrapolation formula, and the final fused image is output.

[0066] The method described in this embodiment breaks the constraint of the traditional network fusion strength being unadjustable, enabling continuous and flexible adjustment of the weights injected into different modal features without any retraining during the inference phase. Furthermore, this embodiment innovatively introduces decoupled conditional and unconditional branches and applies feature orthogonality constraints during joint training, forcing the conditional branches to focus on learning the incremental information unique to the conditional modality (such as the infrared modality). Therefore, it not only effectively eliminates information redundancy in the multimodal feature extraction process but also endows the fusion process with a high degree of scene adaptability and subjective controllability. The method described in this embodiment is particularly suitable for adaptive fusion and intelligent perception detection of multimodal images such as visible light (source image) and infrared (conditional modality image).

[0067] See Figure 2 In this embodiment, the dual-branch neural network uses the same model for both conditional and unconditional paths. When unconditional, the conditional input (ir in the diagram) is a tensor with all zeros; when conditional, the conditional input is the corresponding modality data. Specifically... Figure 2 The symbols marked in the diagram (e.g., r-resblock is a residual module with a time interface, a common structure in diffusion models; depthwise is a separable convolution; conv is a regular convolution; the cond_fusion module is shown in the small diagram below) indicate that this embodiment does not emphasize the specific structure of each layer of the neural network, but rather focuses on the unconditional reconstruction branch and the conditional fusion branch, as well as their respective image processing procedures. That is, it emphasizes the concept of controllable injection fusion of two branches. Since the specific structure of the convolutional block is not an improvement point in this embodiment, existing convolutional blocks can be used to implement the method of this embodiment.

[0068] In practical applications, for the construction of a two-branch neural network, this embodiment first constructs an unconditional reconstruction branch, in... Figure 2 In the network architecture, it is a visual backbone network independent of the conditional modality. The core purpose of this branch is to still be able to reconstruct visible light images with high fidelity and suppress background noise when there is a lack of guidance from conditional modalities such as infrared. Therefore, in order to ensure the consistency of the two branches in terms of computational dimension, this embodiment introduces learnable null conditions to replace the real conditional modal features in the unconditional generation path.

[0069] Secondly, see Figure 2This embodiment can construct the conditional fusion branch in a dual-branch neural network. Specifically, in this embodiment, the dual-branch neural network adopts a dual-branch encoder-decoder architecture containing an independent visual image feature extraction backbone and a conditional feature extraction backbone. During the feature transfer process in the bottleneck layer and decoder of the dual-branch neural network, a fine-grained spatial conditional attention mechanism can be used to modulate the backbone features using conditional features. The modulation of the backbone features using the fine-grained spatial conditional attention mechanism includes the following steps:

[0070] Spatial alignment is performed on the input conditional features (infrared conditional features);

[0071] Spatial dimensional affine transformation parameters γ and β are generated through spatial convolution mapping;

[0072] The backbone features are modulated using the FiLM mechanism, and a feature map is output. The formula for calculating the output feature map is as follows:

[0073] ;

[0074] in, Indicates the characteristics of the backbone network. This represents the conditional features after alignment. This indicates element-wise multiplication.

[0075] Therefore, the above mechanism can effectively realize the dynamic modulation and fusion of infrared features and visible light image features in a multi-scale space.

[0076] To enable the aforementioned dual-branch neural network model to be dynamically adjustable during inference and to remove feature redundancy, this embodiment divides the training process into three stages, employing a three-stage joint training strategy. Specifically, this includes: a first stage of unconditional reconstruction pre-training, a second stage of conditional discarding training based on random masks, and a third stage of joint fine-tuning pre-feature orthogonal constraint training.

[0077] In the first stage, unconditional branches are trained in large-scale natural images (conditional modalities do not participate in training). Learnable empty conditional features are introduced and used to replace real conditional modal features for dimensional expansion and spatial alignment, ensuring the computational consistency of the dual-branch network structure. Therefore, the model can achieve high-fidelity image reconstruction even in the absence of conditional modal guidance.

[0078] In the second stage, a conditional drop mechanism based on random masks (CFG Drop) is established to generate unconditional masks and conditional masks with preset probabilities, and to simultaneously perform weighted calculations of unconditional reconstruction loss and conditional fusion loss on the same batch of data.

[0079] In the third stage, joint fine-tuning and orthogonal constraint training are performed. During the network's forward propagation, both unconditional and conditional features are output simultaneously. By blocking the gradient backpropagation of the unconditional features, the feature increment between the conditional and unconditional features is calculated. An orthogonal constraint loss is introduced to minimize the cosine similarity between the unconditional feature and the feature increment, prompting the conditional branch to focus on learning the unique incremental information of the conditional mode (such as the infrared mode), avoiding feature learning redundancy, and combining noise loss, reconstruction loss, and fusion loss for global joint optimization of the network.

[0080] Specifically, the above three training phases can be achieved through the following steps:

[0081] The first stage (unconditional reconstruction pre-training): The input condition is set to an all-zero tensor, and training is performed only on the backbone feature encoding and decoding branch. This stage endows the model with basic denoising and visible light reconstruction capabilities by calculating the reconstruction loss between the denoised output and the original image features.

[0082] Phase 2 (Conditional Drop Training Based on Random Mask): Introducing the CFG Drop mechanism and setting the conditional drop probability. (e.g., 0.1 or 0.2). A Boolean mask is generated in each iteration, with the infrared conditional features set to null with this probability. This stage jointly optimizes the unconditional reconstruction loss and the conditional fusion loss:

[0083] ;

[0084] Where λ is the weight coefficient of the conditional fusion branch. Conditional fusion of branch losses, This represents the loss from unconditionally rebuilding the branch.

[0085] The third stage (joint fine-tuning and feature orthogonality constraints): simultaneously outputting unconditional multi-scale features during forward propagation. and conditional multiscale features To force the conditional branch to learn only incremental information such as heat sources specific to the infrared mode, for Perform gradient detachment and calculate conditional feature increments. Then, the squared cosine similarity between the two is calculated as the orthogonality loss. Here, CosSim represents the cosine similarity function, and it is used for global optimization in conjunction with reconstruction and fusion losses. This step completely solves the problem of information redundancy in multimodal feature extraction.

[0086] In this embodiment, the total loss function of the third stage joint fine-tuning and feature orthogonality constraint training is the unconditional reconstruction loss. Conditional fusion loss and characteristic orthogonal loss The weighted sum of the terms, the formula for calculating the total loss function is:

[0087] ;

[0088] in, The weights for unconditional reconstruction loss, The weights for the conditional fusion loss, The weights are the features of the orthogonal loss.

[0089] like Figure 1 Flowcharts and Figure 3 As shown in the demonstration image, no retraining of any network parameters is required during the model deployment and inference phases. After receiving visible light and infrared images, the dual-branch neural network model performs an unconditional prediction for each. And a single conditional prediction Subsequently, a linear extrapolation formula without a classifier guidance mechanism is used, and a user-defined guidance scale factor s is introduced to combine features of the unconditional prediction results and the conditional prediction results. The linear extrapolation formula is expressed as follows:

[0090] ;

[0091] The salience of conditional modes (such as infrared features) in the final fused image can be continuously controlled by adjusting the value of s.

[0092] like Figure 3 As shown, when s gradually increases from 0.5 to 2.5, the integration of conditional branch information becomes more significant, meaning the infrared features become more prominent. Therefore, the salience of infrared targets (such as pedestrian heat sources) in the final fused image exhibits a continuous and smooth enhancement. Thus, through... Figure 3 As can be seen, the on-demand control capability of the method provided in this embodiment enables this embodiment to perfectly adapt to complex real environments such as day-night cycles and sudden changes in light intensity.

[0093] Example 2

[0094] See Figure 4 Based on Example 1, this example compares the results on Roadscene. Ir and vi represent the original modal data. TSDT, MURF, TUFusion, MUFusion, URFusion, LFDT-Fusion, DDFM, Dif-Fusion, and AVDMUFusion are the final fused image results of their respective existing image fusion methods. OUR is the final fused image result of the method provided in Example 1. Figure 4It can be seen that, under glare (complex lighting) conditions, the target in the final fused image is clearer and more prominent when using the method provided in Example 1.

[0095] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A classifier-free guided image fusion controllable modal modulation method, characterized in that, Includes the following steps: A two-branch neural network is constructed, comprising an unconditional reconstruction branch and a conditional fusion branch. The unconditional reconstruction branch is a basic image feature reconstruction branch independent of the conditional modality, used to perform high-quality, high-fidelity reconstruction of the source image and pre-background noise suppression when the conditional modality guidance is lacking. The conditional fusion branch is a condition-based cross-modal feature adaptive injection branch, used to extract features from the source image and the conditional modality image and perform dynamic modulation and fusion in a multi-scale space. A three-stage joint training strategy is used to train the constructed dual-branch neural network; During the inference phase, the source image, conditional modality image, and set guiding scale factor are received. Unconditional prediction and conditional prediction are performed respectively, and continuous controllable modality injection without retraining is achieved through linear extrapolation formula, outputting the final fused image.

2. The image fusion controllable modal modulation method based on non-classifier guidance according to claim 1, characterized in that, The dual-branch neural network adopts a dual-branch encoder-decoder architecture that includes an independent visual image feature extraction backbone and a conditional feature extraction backbone. In the feature transfer process of the bottleneck layer and decoder of the dual-branch neural network, a fine-grained spatial conditional attention mechanism is used to modulate the backbone features with conditional features.

3. The image fusion controllable modality modulation method based on non-classifier guidance according to claim 2, characterized in that, The method of using a fine-grained spatial conditional attention mechanism to modulate the backbone features based on conditional features includes the following steps: Spatial alignment of the input conditional features; Spatial dimensional affine transformation parameters γ and β are generated through spatial convolution mapping; The backbone features are modulated using the FiLM mechanism, and a feature map is output. The formula for calculating the output feature map is as follows: ； wherein, represents a backbone network feature, represents an aligned conditional feature, represents an element-wise multiplication.

4. The image fusion controllable modal modulation method based on non-classifier guidance according to claim 1, characterized in that, The three-stage joint training strategy includes: unconditional reconstruction pre-training in the first stage, conditional discarding training based on random masks in the second stage, and joint fine-tuning of pre-feature orthogonal constraint training in the third stage.

5. The image fusion controllable modal modulation method based on non-classifier guidance according to claim 4, characterized in that, The first stage of unconditional reconstruction pre-training refers to: training the unconditional reconstruction branch using natural images, with conditional modality images not participating in the training, and introducing learnable empty conditional features, which are then used to replace the real conditional modality features for dimensional expansion and spatial alignment. The second stage of conditional dropout training based on random masks refers to: establishing a conditional dropout mechanism based on random masks, generating unconditional masks and conditional masks with preset probabilities, and simultaneously performing weighted calculations of unconditional reconstruction loss and conditional fusion loss on the same batch of data; The third stage of joint fine-tuning and orthogonal constraint training refers to the following: during the forward propagation of the network, unconditional features and conditional features are output simultaneously. By blocking the gradient backpropagation of unconditional features, the feature increment between conditional and unconditional features is calculated. Orthogonal constraint loss is introduced to minimize the cosine similarity between the unconditional feature and the feature increment, so that the conditional branch focuses on learning the unique incremental information of the conditional mode. The network is then globally jointly optimized by combining noise loss, reconstruction loss and fusion loss.

6. The image fusion controllable modal modulation method based on non-classifier guidance according to claim 4, characterized in that, The first stage of unconditional reconstruction pre-training is implemented through the following steps: Set the input condition to an all-zero tensor and train only the unconditional reconstruction branch; The parameters of the backbone network are updated by calculating the reconstruction loss between the denoised output and the original image features.

7. The classifier-free image fusion controllable modal modulation method according to claim 4, characterized in that, The second stage of conditional dropout training based on random masks is trained using a randomized binary crossover training strategy, and its implementation steps are as follows: The probability of discarding under certain conditions is defined as follows: ; Generate a boolean mask in each training iteration, so as to The probability of setting the input conditional features to null; The total loss function for training a bi-branch neural network is a weighted sum of the unconditional reconstruction branch loss and the conditional fusion branch loss, expressed as: ； Where λ is the weight coefficient of the conditional fusion branch. Conditional fusion of branch losses, This represents the loss from unconditionally rebuilding the branch.

8. The classifier-free image fusion controllable mode modulation method according to claim 4, characterized in that, The third stage of joint fine-tuning and feature orthogonal constraint training is implemented through the following steps: The unconditional reconstruction branch and the conditional fusion branch are jointly optimized, and feature orthogonal loss is introduced to achieve feature decoupling; For a given multi-source image, input it into a two-branch neural network for unconditional forward propagation and conditional forward propagation respectively to obtain unconditional multi-scale features. and conditional multiscale features ; right Gradient truncation yields unconditional multiscale features. This allows the conditional branch to learn only the incremental information of the conditional mode and calculate the conditional feature increment. ; Calculate the squared cosine similarity between the gradient-trunculated unconditional multi-scale features and the conditional feature increments, and use it as the orthogonal loss, denoted as . The calculation formula is as follows: ； Where CosSim represents the cosine similarity function.

9. The method for controllable mode modulation of image fusion based on classifier-free guidance as described in claim 8, characterized in that, The third stage of joint fine-tuning and feature orthogonality constraint training has a total loss function of unconditional reconstruction loss. Conditional fusion loss and characteristic orthogonal loss The weighted sum of the terms, the formula for calculating the total loss function is: ； in, The weight for unconditional reconstruction loss, The weights for the conditional fusion loss, The weights are the features of the orthogonal loss.

10. A classifier-free image fusion controllable mode modulation method according to any one of claims 1-9, characterized in that, The process of receiving the source image, conditional modality image, and a set guiding scale factor during the inference phase, performing unconditional and conditional predictions respectively, and achieving continuous and controllable modality injection without retraining through a linear extrapolation formula to output the final fused image, refers to: During the inference phase of the dual-branch neural network, source images and conditional modality images are received; The unconditional prediction results are obtained by performing one unconditional prediction at a time using a two-branch neural network. And the conditional prediction result obtained from a single conditional prediction. ; A linear extrapolation formula without a classifier guidance mechanism is used, and a set guidance scale factor s is introduced to combine features of the unconditional prediction results and the conditional prediction results. The linear extrapolation formula is expressed as follows: ； The salience of the conditional mode in the final fused image is continuously controlled by adjusting the size of s.

Images