Universal image fusion method based on task-customized hybrid adapter network

Abstract

The application discloses a general image fusion method based on a task customization mixed adapter network, which comprises the following steps: a general image fusion network uses a pre-trained Vision Transformer as a backbone network to extract multi-source image features in a frozen double-flow network structure, obtains global attention through window cycle movement, and inputs a subsequent task customization mixed adapter network; in a training process, the task customization mixed adapter network customizes fusion prompts according to current sample features and a task to which a current sample belongs; a fusion layer obtains locally biased fusion features according to task customization prompts and source coding of multiple source images; the locally biased fusion features are returned to each source network backbone branch, the fusion features are injected into backbone features, and the features are reconstructed into a fusion image; the value of a fusion image loss and mutual information regularization is calculated respectively, the sum of the two values is obtained, and then back propagation is performed to train the network; and various fusion images are obtained by scaling and shifting the fusion prompts.

Description

Technical Field

[0001] This invention relates to the field of image fusion, and more particularly to a general image fusion method based on a task-customized hybrid adapter network. Background Technology

[0002] The core of image fusion lies in integrating diverse image information acquired from different sensors within the same scene to form a comprehensive, single image. This process not only helps highlight key information within the image but also significantly enhances its overall visual quality. The importance of image fusion technology in image analysis and computer vision is self-evident. By intelligently combining multi-source data, more accurate and comprehensive decision-making information can be obtained than from a single sensor. Simultaneously, the fused image better aligns with human and machine visual perception habits, laying a solid foundation for subsequent tasks such as target recognition and detection.

[0003] Image fusion technology has a wide range of applications, covering multiple fields such as object tracking, medical diagnosis, and remote sensing monitoring. Currently, image fusion technologies mainly include various forms such as multimodal, multi-exposure, and multifocal image fusion, each with its unique fusion mechanism. For example, multi-exposure image fusion aims to integrate a sequence of images with different exposure levels into a single, high-quality, fully exposed image, where each original image provides unique illumination and structural information for the final fused image. Visible-infrared image fusion is a typical multimodal fusion technique that combines complementary information from infrared and visible light images to generate a robust and information-rich fused image. Infrared images provide rich intensity information, while visible light images provide detailed texture and gradient information. Furthermore, multifocal image fusion focuses on generating a fully focused image from a series of partially focused images, where each sharp region depends solely on learning from a specific original image.

[0004] With the rapid advancement of deep learning technology, the field of image fusion has seen significant breakthroughs in recent years. However, current mainstream methods are often limited to specific single image fusion scenarios, designing complex network architectures or specific loss functions for a particular task, thus restricting their flexibility for cross-task applications. Given that different fusion tasks are essentially aimed at integrating key information from multiple source images, recent research has begun to attempt to build a unified model to handle diverse fusion tasks, thereby achieving a more general image fusion method.

[0005] However, the challenges these unified models face when handling multiple tasks cannot be ignored. Some methods may suffer performance degradation on other tasks due to excessive bias towards a dominant task, while others may sacrifice the unique characteristics of each task in order to cater to the commonalities of multiple tasks, ultimately exhibiting suboptimal overall performance. Therefore, there is an urgent need to explore a more compatible fusion paradigm that can dynamically and adaptively adapt to different fusion scenarios, thereby achieving better balance and fusion effects across different tasks. Summary of the Invention

[0006] This invention provides a general image fusion method based on a task-customized hybrid adapter network. Leveraging the performance advantages of existing large-scale pre-trained models, this invention designs an efficient fine-tuning method for the task-customized hybrid adapter network, enabling the model to achieve dynamic compatibility with multiple fusion tasks, reducing inter-task conflicts, and ultimately improving the performance of fusion tasks. This achieves unified, general image fusion, as detailed below:

[0007] A general image fusion method based on task-customized hybrid adapter networks, the method comprising:

[0008] The general image fusion network uses a pre-trained Vision Transformer as the backbone network, extracts multi-source image features with a frozen two-stream network structure, obtains global attention through window cyclic movement, and inputs it into a subsequent task-customized hybrid adapter network.

[0009] During training, the task-customized hybrid adapter network customizes fusion prompts based on the features of the current sample and the task to which the current sample belongs; the fusion layer obtains local bias fusion features based on the task-customized prompts and source encodings of multiple source images.

[0010] The locally biased fusion features are returned to the backbone branches of the network from each source, the fusion features are injected into the backbone features, and then the features are reconstructed into a fusion image.

[0011] The values ​​of the fused image loss and mutual information regularization are calculated separately, and the sum of the two is used to perform backpropagation to train the network. By scaling and shifting the fused cues, a variety of fused images are obtained.

[0012] The task-customized hybrid adapter network includes: a task-customized prompt generation module and a prompt-driven feature fusion module connected in series;

[0013] The task customization prompt generation module includes a task-specific routing network library and a task-shared adapter library. Based on the characteristics of the current sample and the task to which the current sample belongs, it selects the routing network and the adapter network to generate dynamically customized fusion prompts.

[0014] The cue-driven feature fusion module contains several convolutional layers and learnable source encoding parameters, and obtains fused features with local bias based on fusion cues and source encoding.

[0015] The window cyclically moves in front of all frozen Vision Transformer blocks, cuts the features according to the window size Win, and adds learnable local position information; the cutting positions before the i-th block and the (i+1)-th block are offset by Win / 2 lengths.

[0016] The mutual information regularization is defined as follows: by constraining the fusion hints, the fused features are constrained to a linear combination of the source features.

[0017] min|prompt x +prompt y -1|

[0018] Among them, prompt x For fusion hints targeting X features of the source image, prompt y This provides a fusion suggestion for the Y-features of the source image.

[0019] Specifically, scaling and offsetting the fusion prompts involves:

[0020] During the inference phase, the generated fusion cues are scaled and offset. When α = 0, the fusion cues for different regions of the image have the same value.

[0021] When α = 1, the task customization prompt generation module predicts the value of the fusion prompt for each region, and the offset coefficient β controls the similarity between the fused image and the source image X or Y;

[0022] When β = 1, the fused image is very similar to the X image; when β = 0, the fused image is very similar to the Y image.

[0023] By adjusting coefficients α and β, various fused images can be obtained. The scaling and offset formulas are as follows:

[0024] prompt x ′=μ+α(prompt x -μ)+β

[0025] prompt y ′=μ+α(prompt y -μ)-β.

[0026] Where μ is a fixed constant, set to 0.5.

[0027] The beneficial effects of the technical solution provided by this invention are:

[0028] 1. This invention leverages the performance advantages of existing large-scale pre-trained models to design an efficient fine-tuning method for a hybrid adapter network based on task customization. This method enables the model to achieve dynamic compatibility with multiple fusion tasks during training, reduces inter-task conflicts, promotes inter-task cooperation, and ultimately achieves leading fusion results on all tasks, realizing a high-performance, unified, and general image fusion model.

[0029] 2. This invention designs a mutual information regularization method, which constrains the fusion features to a linear combination of source features by using constraint fusion prompts, ensuring complementarity for different sources, establishing a feature space and feature representation of linear combination of source features, thereby improving fusion performance;

[0030] 3. This invention controls the strength of network selection features and feature mode bias by scaling and shifting the fusion prompts, thereby obtaining a series of diverse fused images. This achieves controllability of fusion results for known tasks and generalizability of fusion results for unknown tasks, thus enhancing the practical application capability of image fusion.

[0031] 4. This invention designs a method to accelerate training and inference speed based on trainable local location information. For the training and inference process of Vision Transformer, the features are divided into several windows, and the windows are moved cyclically as the network depth increases, thereby reducing the computational load of Transformer blocks and indirectly realizing global attention of features; thus reducing energy consumption and increasing practical application significance. Attached Figure Description

[0032] Figure 1 This is a flowchart of a general image fusion method based on a task-customized hybrid adapter network proposed in this invention;

[0033] Figure 2 This is a schematic diagram of the task-customized hybrid adapter network module of the present invention;

[0034] Figure 3 This is a schematic diagram of the controllable and generalizable method of the present invention;

[0035] Figure 4 This is a schematic diagram of the method for accelerating training and inference speed based on trainable local location information according to the present invention. Detailed Implementation

[0036] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below.

[0037] To address the technical problems existing in general image fusion methods, this invention proposes a general image fusion network framework based on a task-customized hybrid adapter network. This network enhances the representation and reconstruction capabilities of image features by adding learnable additional parameters to a frozen pre-trained model. The network achieves compatibility with multiple fusion tasks and the ability to cooperate between tasks by designing a task-specific routing network library and a task-shared adapter library. The network improves the stability of model training by designing a mutual information regularization method to constrain the feature representation range of fused features. The network solves the technical problem of unique fused images in fusion tasks by scaling and shifting fusion cues, achieving controllability and generalization. The network utilizes a local window cyclic movement method to enhance the inference speed of the network framework and reduce computational load.

[0038] like Figure 1 As shown, a general image fusion method based on task-customized hybrid adapter networks consists of three components, each with the following functions:

[0039] In the multi-source image feature extraction component, this embodiment of the invention uses the Vision Transformer network structure, imports the pre-trained parameters of the MAE (Masked AutoEncoder) network, performs multi-source image feature extraction, and performs a local window cyclic movement operation every other Vision Transformer block.

[0040] Furthermore, multi-source image feature extraction specifically includes: each image can be viewed as being composed of N p×p image blocks stitched together; for each image block, a labeled feature can be obtained, thus each image can obtain N labeled features. Images from different sources are defined as X and Y, respectively. The structure prior to the j-th customized hybrid adapter network is denoted as E. j (·). Based on this, the features used for prompt generation are extracted, denoted as f. x and f y To fuse these image features from different sources, f x and f y The features are represented as multi-source image pairs and combined using the feature concatenation operation Cat(·). This step aims to enable information exchange and fusion between features from different sources in subsequent network processing. A fully connected layer L(·) is then used to reduce the dimensionality of the combined features. After this step, the processed multi-source feature Ф is obtained, which will serve as the input to the subsequent network.

[0041] f x =E j (X),f y=E j (Y)

[0042] Ф=L(Cat(f x ,f y ))

[0043] Furthermore, such as Figure 2 As shown, the local window looping operation specifically includes: before all VisionTransformer blocks, the features are cut according to the window size Win, and learnable local position information is added. The cutting positions before the i-th block and the (i+1)-th block are offset by Win / 2 lengths, ultimately achieving indirect global attention.

[0044] In the task-customized prompt generation component, this embodiment of the invention uses a task-specific routing network library. Based on the task to which the current sample belongs, a weight distribution of adapter hybridization is customized and applied to the task-shared adapter library to obtain task-customized fused prompts. These fused prompts use a mutual information regularization (MIR) constraint.

[0045] Furthermore, such as Figure 3 As shown, the weight distribution of the adapter hybrid specifically includes: based on the task to which Ф belongs, from the routing network library (including: multimodal task routing G) V Multi-exposure task routing G E and multi-focused task routing G F In the context of the algorithm, a task-specific routing network G is selected to customize the routing weights for the current sample. The indices of the K highest weights are chosen from these weights to determine which pair of source features should be input into the adapter library (which includes N adapters {A1, ..., A...}). N Which adapters in}). The route weights for the custom hybrid adapter are obtained as follows. Where Softmax(·) and Softplus(·) represent activation functions. W g and W noise Represents the weights of the routing network. N(0,1) represents Gaussian distribution sampling.

[0046] G(Ф) = Softmax(TopK(Ф·W) g +N(0,1)·SoftplusФ·W noise )))

[0047] Furthermore, such as Figure 3 As shown, the specific process of customized task-based fusion prompts is as follows: the outputs of the adapters are weighted and summed according to the routing weights of the customized hybrid adapters, which is the process of customizing the adapters, ultimately obtaining the fusion prompts. The calculation method is shown below. Where `prompt` represents the fusion prompt:

[0048]

[0049] Furthermore, the specific process of the Mutual Information Regularization (MIR) method is as follows: by constraining fusion hints, the fused features are constrained to a linear combination of the source features. As shown below:

[0050] min|prompt x +prompt y -1|

[0051] In the prompt-driven fusion component, this embodiment of the invention uses source bias encoding to record source-related information, uses a multi-source feature fusion layer F, introduces local inductive bias, and returns multi-source features to the main branch of the model.

[0052] Furthermore, such as Figure 3 As shown, the source bias encoding and fusion layer are specifically: a dot product of multi-source features and task-customized cues, followed by a source bias encoding S associated with the source. x and S y The features are added together and then passed through a fusion layer F with local inductive bias to obtain multi-source features f with local feature representations. TC-MOA The process is as follows:

[0053] h x =prompt x ·f x +S x

[0054] h y =prompt y ·f y +S y

[0055] f TC-MoA =F(h) x +h y )

[0056] Furthermore, such as Figure 3 As shown, the main branch of the multi-source feature return pre-trained model specifically involves: processing the multi-source features f... TC-MOA The features are weighted and summed with those of the backbone network before the adapter input, allowing the model to gradually extract important fusion features through several adapter modules. The calculation process is as follows:

[0057] f x ′=λ f f x +(1-λ f )f TC-MoA

[0058] f y ′=λ f f y +(1-λ f)f TC-MoA

[0059] During the inference phase, embodiments of the present invention employ a controllable and generalizable method for fused images. This method includes: controlling the intensity of network-selected features and the intensity of feature mode bias by scaling and shifting the fusion cues, thereby obtaining a series of diverse fused images. The controllable fusion result is as follows: Figure 4 As shown.

[0060] Furthermore, scaling and offset are specifically defined as follows:

[0061] prompt x ′=μ+α(prompt x -μ)+β

[0062] prompt y ′=μ+α(prompt y -μ)-β

[0063] Example 1

[0064] This invention provides a general image fusion method based on a task-customized hybrid adapter network, the method comprising the following steps:

[0065] 101: The general image fusion network uses a pre-trained Vision Transformer as the backbone network, extracts multi-source image features with a frozen two-stream network structure, indirectly obtains global attention through a window looping strategy, and then inputs it into a subsequent task-customized hybrid adapter network.

[0066] Specifically, the two branches in the two-stream network structure share frozen pre-trained parameters. Each branch has a structure consistent with the Vision Transformer structure and initializes the network using weights pre-trained using the MAE (Masked AutoEncoder) method. Before all Vision Transformer blocks, features are cut by a window size Win and learnable local location information is added. The cut positions before the i-th and (i+1)-th blocks are cyclically offset by a length Win / 2. This ultimately achieves indirect global attention.

[0067] 102: During training, the task-customized hybrid adapter network samples adaptively and the task adaptively dynamically customizes the fusion prompts;

[0068] After obtaining the features of the multi-source images, in order to perform general image fusion, it is necessary to select the appropriate routing network according to the current image and the current task, dynamically customize the weight distribution of the adapter library, and weight it with the output of the adapter library to finally obtain the task-customized fusion prompt.

[0069] 103: During training, the fusion layer obtains local bias fusion features based on task-customized cues and source encodings from multiple source images;

[0070] Considering that feature representations should include source-related biases (e.g., visible or infrared images), each source introduces an input-independent learnable parameter, i.e., a source code. The features of the multi-source images are multiplied by a task-customized fusion cue, and then the source codes are added to obtain the refined multi-source features. These features are then passed through a fusion layer F consisting of convolutional layers to obtain fused features with local inductive biases.

[0071] 104: Return the fusion features of local inductive bias to the main branches of the network from each source to realize the injection of fusion features into the main features;

[0072] To encourage the model to progressively extract important information, the main branch features output to the next Transformer block are obtained by weighting the original main features and the fused features, with a weighting ratio λ. f It is a learnable parameter.

[0073] 105: Repeat steps 101-104 until the features have passed through the entire network, then reconstruct the features into a fused image;

[0074] The Vision Transformer network structure incorporates several trainable, task-customizable hybrid adapter networks at equal intervals. Therefore, the network includes several 10¹-10⁴ steps. Finally, the network uses fully connected layers and deblocking methods to reconstruct the fused image from the features.

[0075] 106: After obtaining the fused image, calculate the fused image loss and the value of the mutual information regularization method MIR respectively, sum the two, and then perform backpropagation to train the network.

[0076] After obtaining the fused image, the corresponding image reconstruction loss can be calculated based on the input multi-source images and the current task. The loss functions for the multimodal task include: structural consistency loss (MS-SSIM), maximum pixel value loss (Max-Pixel), and maximum gradient loss (Max-Grad); the loss functions for the multi-exposure task include: structural consistency loss for multi-focus fusion (MEF-SSIM), average pixel value loss (Avg-Pixel), and maximum gradient loss (Max-Grad); and the loss functions for the multi-focus task include: structural consistency loss (MS-SSIM), mask pixel loss (Mask-Pixel), and mask gradient loss (Mask-Grad). The network's shared losses include the auxiliary loss (aux) for maintaining adapter selection balance and the mutual information regularization constraint (MIR).

[0077] 107: Deploying the trained network, the model can perform multi-focus, multi-exposure, and multi-modal image fusion tasks. By scaling and shifting the fusion cues, diverse fused images can be obtained, thus ensuring the controllability of the fusion results. The model can also perform unknown image fusion tasks, demonstrating generalizability. This allows for efficient use of existing data resources, saves computational overhead, and promotes the development of the image fusion field.

[0078] In summary, this invention generates a fused image step by step through three parts: multi-source feature extraction, task-customized fusion prompts, and prompt-driven feature fusion. Diverse fusion results are generated by scaling the prompts. A large model and window cyclic shift strategy are used to extract multi-source features, achieving efficient and global feature extraction. Task-customized fusion prompts enable multi-task compatibility, thus promoting performance across tasks. Prompt-driven feature fusion imports source and local information, resulting in better fused image generation. Scaling the prompts makes the results controllable and generalizable, making it easier to obtain high-quality images that conform to human visual perception and reducing computational overhead.

[0079] Example 2

[0080] This invention provides a method for local window cyclic movement. To reduce the computational cost of the Transformer architecture, most previous methods only supported fixed-size inputs. These methods employed additional preprocessing and post-processing steps to crop and stitch the fused images, resulting in a checkerboard effect. This invention improves upon the frozen-parameter ViT architecture by using local window cyclic movement to accelerate the inference speed of the fused network. The method divides features into multiple windows of 14×14 patches (to maintain consistency with the label sequence length during pre-training). Then, learnable local location embeddings are introduced to address the aforementioned problem. Local location embeddings can perceive the position of the labels within the window, ensuring the model's spatial awareness of the local window. Subsequently, a global receptive field is indirectly obtained by cyclically moving the windows between Transformer blocks. Therefore, the local window cyclic movement method not only allows for efficient handling of different input sizes but also captures the global receptive field of the image, effectively avoiding the checkerboard effect.

[0081] As shown in Table 1, the embodiments of the present invention were tested on the test sets of three tasks. Although the VisionTransformer architecture has poor support for large input sizes, the base model and the large model were accelerated by 178% and 167% respectively through the local window loop movement design. The results show that the embodiments of the present invention still achieve acceptable running speeds compared to other methods. Here, M represents millions of parameters. FPS represents the number of image frames output by the network per second.

[0082] Table 1

[0083]

[0084]

[0085] Example 3

[0086] The scheme in Example 1 will be further described below with specific examples and calculation formulas:

[0087] I. Data Preparation

[0088] The embodiments of this invention can be extended to general fusion methods. The embodiments of this invention validate the effectiveness of the proposed method on test datasets for multi-exposure fusion, multi-modal fusion, and multi-focus fusion. The training set includes: the multi-exposure dataset SCIE, the multi-focus datasets MFI-WHU and RealMFF, and the multi-modal dataset LLVIP. The test sets include: the multi-exposure dataset MEFB, the multi-focus dataset MFFB, and the multi-modal dataset LLVIP test set.

[0089] SCIE is a dataset of multiple exposure images consisting of pairs of low-contrast and high-contrast images. It includes 589 multiple exposure sequences of indoor and outdoor scenes. Each sequence contains 3 to 18 low-contrast images at different exposure levels, resulting in a total of 4413 multiple exposure images. 589 high-quality reference images were selected from the results of 13 representative enhancement algorithms. That is, each multiple exposure image uses the same high-contrast reference image. Image resolutions are 3000*2000 and 6000*4000, and the images are saved in RGB format.

[0090] The MFI-WHU dataset contains 120 color image pairs for multifocus image fusion. It's worth noting that the fully sharp source images come from several public datasets. 120 images were selected from the COCO dataset as sharp images, and masking them resulted in blurred regions with distinct boundaries between in-focus and out-of-focus areas, leading to unnatural transitions.

[0091] Real-MFF is a large and realistic multifocal dataset containing 710 pairs of source images and corresponding ground truth images. The dataset is generated from light field images, and both the source and ground truth images are real. This dataset serves as a benchmark for existing multifocal image fusion algorithms and is also a suitable training dataset for deep learning-based methods. It includes various scenes, such as buildings, plants, people, and shopping malls.

[0092] LLVIP is a visible-infrared pairing dataset for low-light vision. The dataset contains 33,672 images, or 16,836 pairs, most of which were taken in very dark scenes, and all images are strictly aligned temporally and spatially. Pedestrians in the dataset are labeled. This dataset has a high degree of detail and normalization, and can be used to estimate and improve the performance of existing image fusion algorithms, low-light human detection, and image translation methods.

[0093] MEFB is the first benchmark in the field of multi-exposure image fusion (MEF), aiming to provide a fair and comprehensive performance comparison platform for MEF methods. Currently, MEFB integrates 100 image pairs, 21 fusion algorithms, and 20 evaluation metrics, facilitating convenient performance comparison. All fusion results are also readily available to users.

[0094] MFFB is a benchmark dataset in the field of multifocal imaging. It includes three datasets: Lytro, MFFW, and MFI-WHU, totaling 63 image pairs. The Lytro dataset contains 20 multifocal color images and 4 color images with three different exposures, used for image fusion. MFFW contains 19 pairs of multifocal images collected from the internet. Compared to the Lytro dataset, the images in MFFW are significantly affected by defocus diffusion. Additionally, the scenes in MFFW are more complex.

[0095] II. General Image Fusion Network Structure Based on Task-Customized Hybrid Adapter Network

[0096] The general image fusion network structure based on task-customized hybrid adapter networks in this embodiment of the invention, such as... Figure 1 As shown, it consists of three components, and the functions of each part are as follows:

[0097] In the multi-source image feature extraction component, this embodiment of the invention uses the Vision Transformer network structure, imports the pre-trained parameters of the MAE (Masked AutoEncoder) network, performs multi-source image feature extraction, and performs a local window cyclic movement operation every other Vision Transformer block.

[0098] Furthermore, multi-source image feature extraction specifically includes: each image can be viewed as being composed of N p×p image blocks stitched together; for each image block, a labeled feature can be obtained, thus each image can obtain N labeled features. Images from different sources are defined as X and Y, respectively. The structure prior to the j-th customized hybrid adapter network is denoted as E. j (·). Based on this, features used to prompt the generation steps are extracted, denoted as f. x and f y To fuse these image features from different sources, f x and f y The features are represented as multi-source image pairs and combined using the feature concatenation operation Cat(·). This step aims to enable information exchange and fusion between features from different sources in subsequent network processing. A fully connected layer L(·) is then used to reduce the dimensionality of the combined features. After this step, the processed multi-source feature Ф is obtained, which will serve as the input to the subsequent network.

[0099] f x =E j (X),f y =E j (Y)

[0100] Ф=L(Cat(f x ,f y ))

[0101] Furthermore, such as Figure 2 As shown, the local window looping operation specifically includes: before all VisionTransformer blocks, cutting the features according to the window size Win, and adding learnable local position information. The cutting positions before the i-th block and the (i+1)-th block are offset by a length Win / 2. This ultimately achieves indirect global attention.

[0102] In the task-customized prompt generation component, this embodiment of the invention uses a task-specific routing network library. Based on the task to which the current sample belongs, a weight distribution of adapter hybridization is customized and applied to the task-shared adapter library to obtain task-customized fused prompts. These fused prompts use a mutual information regularization (MIR) constraint.

[0103] Furthermore, such as Figure 3 As shown, the weight distribution of the adapter hybrid specifically includes: based on the task to which Ф belongs, from the routing network library (including multimodal task routing G) V Multi-exposure task routing G E and multi-focused task routing G FIn the context of the algorithm, a task-specific routing network G is selected to customize the routing weights for the current sample. The indices of the K highest weights are chosen from these weights to determine which pair of source features should be input into the adapter library (which includes N adapters {A1, ..., A...}). N Which adapters in}). The route weights for the custom hybrid adapter are obtained as follows. Where Softmax(·) and Softplus(·) represent activation functions. W g and W noise Represents the weights of the routing network. N(0,1) represents Gaussian distribution sampling.

[0104] G(Ф) = Softmax(TopK(Ф·W) g +N(0,1)·Softplus(Ф·W noise )))

[0105] Furthermore, such as Figure 3 As shown, the specific process of customized task-based fusion prompts is as follows: the outputs of the adapters are weighted and summed according to the routing weights of the customized hybrid adapters, which is the process of customizing the adapters, ultimately obtaining the fusion prompts. The calculation method is shown below. Where `prompt` represents the fusion prompt:

[0106]

[0107] Furthermore, the specific process of the Mutual Information Regularization (MIR) method is as follows: by constraining fusion hints, the fused features are constrained to a linear combination of the source features. As shown below:

[0108] min|prompt x +prompt y -1|

[0109] In the prompt-driven fusion component, this embodiment of the invention uses source bias encoding to record source-related information; this embodiment of the invention uses a multi-source feature fusion layer F to introduce local inductive bias and return multi-source features to the main branch of the model.

[0110] Furthermore, such as Figure 3 As shown, the source bias encoding and fusion layer are specifically: a dot product of multi-source features and task-customized cues, followed by a source bias encoding S associated with the source. x and S y The features are added together and then passed through a fusion layer F with local inductive bias to obtain multi-source features f with local feature representations. TC-MoA The process is as follows:

[0111] h x =prompt x ·f x +S x

[0112] h y =prompt y ·f y +S y

[0113] f TC-MoA =F(h) x +h y )

[0114] Furthermore, such as Figure 3 As shown, the main branch of the multi-source feature return pre-trained model specifically involves: processing the multi-source features f... TC-MoA The features are weighted and summed with those of the backbone network before the adapter input, allowing the model to gradually extract important fusion features through several adapter modules. The calculation process is as follows:

[0115] f x ′=λ f f x +(1-λ f )f TC-MoA

[0116] f y ′=λ f f y +(1-λ f )f TC-MoA

[0117] During the inference phase, embodiments of the present invention employ a controllable and generalizable method for fused images, comprising: scaling and shifting the fusion cues to control the strength of the network's selected features and the strength of the feature mode bias, thereby obtaining a series of diverse fused images.

[0118] Furthermore, scaling and offset are specifically defined as follows:

[0119] prompt x ′=μ+α(prompt x -μ)+β

[0120] prompt y ′=μ+α(prompt y -μ)-β

[0121] III. Evaluation Indicators and Agreements

[0122] Since the commonly used evaluation metrics for different fusion tasks are not entirely the same, a set of metrics has been customized for each task. Existing metrics can be mainly divided into the following four categories: ◇ Information theory-based metrics: Entropy (EN), Peak Signal-to-Noise Ratio (PSNR), Mutual Information (MI), Normalized MI (NMI), Nonlinear Related Information Entropy (Qncie), Feature Mutual Information (FMI). ◇ Image feature-based metrics: Phase consistency-based metrics Qp, Standard Deviation (SD), Gradient-based metrics Qabf and Qg. ◇ Structural similarity-based metrics: Qc, Qw, Qs, Structural Similarity (SSIM), Multiscale Structural Similarity (MS-SSIM), and MEF-SSIM. ◇ Human perception-inspired metrics: Visual Information Fidelity (VIF), Qcb, and Qcv.

[0123] IV. Details on Model Usage

[0124] 1. Data Supplementation: This embodiment of the invention uses the lowest exposure and overexposure images from the SCIE sequence as input. Although the model supports inputs of arbitrary size, all samples are randomly cropped to 448×448 to ensure that each fusion task receives an equal amount of data in each iteration. A pre-trained MAE large model with GAN loss as its backbone is employed.

[0125] 2. Model Optimization: In this embodiment of the invention, the batch size for each fusion task is set to 3. The AdamW optimizer is used, with an initial learning rate of 1.5 × 10⁻⁴. To ensure training stability, this embodiment of the invention applies exponential moving average (EMA) to the optimization of the routing network and adapters.

[0126] 3. Hyperparameter settings: Each task-customized hybrid adapter network in this embodiment consists of N (N=4) adapters, but only the K (K=2) adapters with the highest activation values ​​are activated.

[0127] 4. Loss Setting: In this embodiment of the invention, an unsupervised loss function is customized for each fusion task. L is added to the loss function of each task. aux To ensure the training of TC-MoA. aux This is an auxiliary loss used to avoid learning imbalance in the adapter. On the other hand, to generate high-quality fused images, embodiments of the present invention incorporate structural information (L...) of the fused images for different fusion tasks. ssim Intensity information (L) Pixel ) and gradient information (L Grad Constraints were imposed. For the VIF task, the objective of this embodiment is to retain the most significant high-frequency and low-frequency information from the source images in the fused image. Therefore, this embodiment designs L... MaxPixel and LMaxGrad To avoid obfuscation of gradients, the sign of the gradient values ​​is preserved in all loss functions related to gradient information. For the MEF task, this embodiment of the invention assumes that the brightness of the fused image should be the average level of all gradient information. Therefore, a loss function with L... AvgPixel and L MaxGrad The MEF loss function is used. Furthermore, an L-type loss function specifically designed for MEF tasks is employed. mefssim For the MFF task, it is assumed that each region patch in the fused image should depend on only a single source image with the largest gradient. This is to prevent the edges of objects in the out-of-focus image from being preserved, thus affecting the quality of the fused image. Therefore, only one source image is selected to compute the loss function, L, for each patch in the image. MaskPixel and L MaskGrad .

[0128] To address the technical problems existing in general image fusion methods, this invention proposes a general image fusion network framework based on a task-customized hybrid adapter network. This network enhances the representation and reconstruction capabilities of image features by adding learnable additional parameters to a frozen pre-trained model. The network achieves compatibility with multiple fusion tasks and the ability to cooperate between tasks by designing a task-specific routing network library and a task-shared adapter library. The network improves the stability of model training by designing a mutual information regularization method to constrain the feature representation range of fused features. The network solves the technical problem of unique fused images in fusion tasks by scaling and shifting fusion cues, achieving controllability and generalization. The network utilizes a local window cyclic movement method to enhance the inference speed of the network framework and reduce computational load.

[0129] The embodiments of this invention have the following four key inventive points:

[0130] I. A general image fusion method based on task-customized hybrid adapter networks is proposed.

[0131] Technical Results: This network leverages the performance advantages of existing large-scale pre-trained models and designs an efficient fine-tuning method for a task-customized hybrid adapter network. This enables the model to achieve dynamic compatibility with multiple fusion tasks during training, reducing inter-task conflicts, promoting inter-task cooperation, and ultimately achieving leading fusion results on all tasks, thus realizing a high-performance, unified, and general image fusion model.

[0132] II. Propose a mutual information regularization method

[0133] Technical effect: This mutual information regularization method constrains the fusion features to a linear combination of source features by constraining fusion prompts, ensuring complementarity for different sources, realizing the model's perception of the dominant intensity of different source images and the establishment of feature space for different source feature combinations, thereby improving fusion performance;

[0134] III. A controllable and generalizable method for fusing images is proposed.

[0135] Technical effect: By scaling and shifting the fusion cue, the strength of the network's selected features and the strength of the feature mode bias are controlled, thereby obtaining a series of diverse fused images. This achieves controllability of the fusion results for known tasks and generalizability of the fusion results for unknown tasks, thus enhancing the practical application capability of image fusion.

[0136] IV. A method for accelerating training and inference speed using trainable local location information is proposed.

[0137] Technical Effects: For the training and inference process of Vision Transformer, by dividing features into several windows and cyclically moving these windows as the network depth increases, the computational cost of Transformer blocks is reduced, indirectly achieving global feature attention. This reduces energy consumption and increases practical application value.

[0138] In summary, this invention provides a general image fusion method based on a task-customized hybrid adapter network. A mutual information regularization method is designed to enable the model to perceive the dominant intensity of different source images and establish a feature space for different source feature combinations. Through a controllable and generalizable method for fusion images, a series of diverse fused images are obtained. A method for accelerating training and inference speed by using trainable local location information is designed, allowing the training model to better utilize existing computing resources to quickly obtain fused images.

[0139] Example 4

[0140] The method proposed in this invention has been compared with several other techniques on multimodal datasets, multi-exposure datasets, and multi-focus datasets.

[0141] I. Comparison of Multimodal Datasets

[0142] For the multimodal dataset LLVIP, the experimental results are shown in Table 1. The embodiments of this invention evaluated the fusion performance on eight quantitative metrics. The task-customized hybrid adapter network method outperforms all general image fusion methods, demonstrating its superior compatibility across multiple tasks. The embodiments of this invention also make significant improvements to most task-specific image fusion methods, despite their complex task-specific designs. Specifically, the embodiments of this invention achieve significant advantages in VIF, Qcv, and EN metrics related to human perception or information theory. This indicates that the fused image is more consistent with human visual perception and contains more information from the source images. The results of MS-SSIM, SD, and Qw metrics also explain that the fusion result has a sufficient amount of structural and gradient information.

[0143] Table 1

[0144]

[0145]

[0146] II. Comparison of Multi-Exposure Datasets

[0147] For the multi-exposure dataset, the experimental results are shown in Table 2. This embodiment of the invention used eight quantitative metrics to evaluate the fusion performance. The task-customized hybrid adapter network model achieved state-of-the-art performance in general image fusion methods and competitive results in task-specific methods. For example, due to the compatibility of the task-customized hybrid adapter network with different tasks, the model significantly improved the MEF-SSIM score, achieving task-specific optimization while reducing task conflicts. MEF-SSIM primarily focuses on the structural and contrast distortion of the image, while Qp primarily measures the phase consistency between the source and fused images. The highest MEF-SSIM, PSNR, and Qp values ​​demonstrate the effectiveness in preserving structural information, detailed features, and image quality in the fused image.

[0148] Table 2

[0149]

[0150]

[0151] III. Comparison of Multi-Focus Datasets

[0152] For the multi-focus dataset, the experimental results are shown in Table 3. This embodiment of the invention used eight quantitative metrics to evaluate the fusion performance. Compared with most image fusion methods, the task-customized hybrid adapter network method exhibits good performance. For general image fusion, this method model achieves superior performance on all metrics. Furthermore, the task-customized hybrid adapter network method directly generates the fused image without requiring a prediction decision map. Moreover, this method still significantly outperforms IFCNN on NMI, MI, Qcb, and Qcv metrics. This indicates that, compared with supervised methods, the fusion result of the task-customized hybrid adapter network retains more source image details and demonstrates significant human visual perception.

[0153] Table 3

[0154]

[0155]

[0156] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of a preferred embodiment, and the sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0157] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A general image fusion method based on task-customized hybrid adapter networks, characterized in that, The method includes: The general image fusion network uses a pre-trained Vision Transformer as the backbone network, extracts multi-source image features with a frozen two-stream network structure, obtains global attention through window cyclic movement, and inputs it into a subsequent task-customized hybrid adapter network. During training, the task-customized hybrid adapter network customizes fusion prompts based on the features of the current sample and the task to which the current sample belongs; the fusion layer obtains local bias fusion features based on the task-customized prompts and source encodings of multiple source images. The locally biased fusion features are returned to the backbone branches of the network from each source, the fusion features are injected into the backbone features, and then the features are reconstructed into a fusion image. The values ​​of the fused image loss and mutual information regularization are calculated separately, and the sum of the two is used to perform backpropagation to train the network. By scaling and shifting the fused cues, various fused images are obtained. The task-customized hybrid adapter network includes: a task-customized prompt generation module and a prompt-driven feature fusion module connected in series; The task customization prompt generation module includes a task-specific routing network library and a task-shared adapter library. Based on the characteristics of the current sample and the task to which the current sample belongs, it selects the routing network and the adapter network to generate dynamically customized fusion prompts. The cue-driven feature fusion module contains several convolutional layers and learnable source encoding parameters, and obtains fused features with local bias based on fusion cues and source encoding.

2. The general image fusion method based on a task-customized hybrid adapter network according to claim 1, characterized in that, The window cyclically moves in front of all frozen Vision Transformer blocks, cuts the features according to the window size Win, and adds learnable local position information; where the cutting positions before the i-th block and the (i+1)-th block are offset by Win / 2 lengths.

3. The general image fusion method based on a task-customized hybrid adapter network according to claim 1, characterized in that, The mutual information regularization is achieved by constraining the fusion cue to a linear combination of the source features: ； in, This provides fusion suggestions for the X features of the source image. This provides a fusion suggestion for the Y-features of the source image.

4. The general image fusion method based on a task-customized hybrid adapter network according to claim 3, characterized in that, The scaling and offsetting of the fusion prompts specifically involves: During the inference phase, the generated fusion hints are scaled and offset, when The fusion of different regions of the image indicates that they all have the same value; when The task customization prompt generation module predicts the value of the fused prompt for each region, and the offset coefficient. Control the degree of similarity between the fused image and the source image X or Y; when At that time, the fused image is very similar to the X image. At that time, the fused image is very similar to the Y image; Adjustment coefficient and To obtain diverse fused images, the scaling and offset formulas are as follows: ； 。

Images