A unified image fusion method and system for multi-task separation-mixing-purification

Abstract

The present invention relates to the field of image processing technology and specifically discloses a unified image fusion method and system for multi-task separation-mixing-purification. First, a reconstruction network model is constructed for each specific task to obtain a specific task separation network, thereby explicitly extracting task-specific features. Subsequently, a multi-task hybrid fusion modeling strategy is adopted to construct a hybrid task fusion network based on the specific task separation network to promote fine-grained collaboration between tasks and extract more discriminative fusion components. Finally, a knowledge distillation strategy is used to construct a unified model purification network to achieve model purification, adaptively unify task-specific features and cross-task shared features into a fusion output, and obtain a unified image fusion network. Extensive experiments conducted on four typical image fusion tasks demonstrate that the proposed method and system significantly outperform existing advanced unified image fusion methods in terms of performance and effectively narrow the gap with specially optimized single-task models.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a multi-task separation-mixing-purification unified image fusion method and system. Background Art

[0002] Image fusion aims to integrate complementary information from multiple source images, including but not limited to infrared and visible fusion (IVF), multimodal medical fusion (MMF), multi-exposure fusion (MEF), and multi-focus fusion (MFF). Most existing methods develop task-specific fusion models for single-task source image pairs. However, this specificity limits their generalization capabilities in multi-task scenarios. Therefore, "unified image fusion" (UIF) has emerged to address multiple image fusion tasks simultaneously.

[0003] In recent years, research on UIF has mainly explored two major categories of methods. The first category uses a unified network structure for different fusion tasks, but independently optimizes task-specific model parameters for each fusion task during the training phase. During the inference phase, it is necessary to manually switch the corresponding model parameters to adapt to different tasks. However, since this method does not consider the intrinsic correlation between tasks, it is easy to cause the model to overfit to single-task features and have insufficient generalization performance. The second category of methods is committed to extracting common features or task-independent general features between multiple fusion tasks, thereby achieving cross-task knowledge transfer and enhancing model adaptability. However, this type of method is difficult to effectively control the balance between capturing task-general features and retaining task-specific details. In particular, in multi-task training, some key information is often lost due to gradient conflicts, which limits the overall performance of the model in diverse fusion tasks.

[0004] At present, how to achieve the generalization and adaptability of the model to multi-task scenarios while ensuring the optimization accuracy of a single task is still one of the core challenges facing UIF. Summary of the Invention

[0005] The present invention provides a unified image fusion method and system for multi-task separation-mixing-purification, which solves the technical problem of how to achieve the generalization and adaptability of the model to multi-task scenarios while ensuring the optimization accuracy of a single task, so as to achieve a dual improvement in fusion accuracy and generalization performance.

[0006] To solve the above technical problems, the present invention provides a unified image fusion method of multi-task separation-mixing-purification, comprising the following steps:

[0007] Build and train a task-specific separation network for more than two image fusion tasks;

[0008] Building a hybrid task fusion network based on the trained specific task separation network and training the network;

[0009] Based on the trained mixed task fusion network, a model is constructed to purify the unified network and train it;

[0010] Purifying the trained model and performing unified network purification to obtain a unified image fusion network;

[0011] The unified image fusion network is used to fuse the image pairs to be fused, and a fused image for two or more image fusion specific tasks is obtained.

[0012] Furthermore, the specific task separation network includes multiple isomorphic encoder-decoder reconstruction networks adapted to different image fusion specific tasks, and the encoder-decoder reconstruction network includes a specific task encoder, a specific task decoder and a specific task head; the specific task encoder extracts task-related features of the source image pair of the image fusion specific task and inputs the task-related features into the corresponding specific task decoder and the specific task head, and the specific task decoder performs image reconstruction based on the task-related features to generate a reconstructed image pair; the specific task head maps the task-related features into two semantic vectors.

[0013] Furthermore, during the training process, the loss function of the specific task separation network is defined as the weighted sum of the reconstruction loss and semantic contrast loss of all image fusion specific tasks, where the reconstruction loss is defined as the difference between the source image pair and the reconstructed image pair; and the semantic contrast loss is defined as the InfoNCE loss based on the semantic vector.

[0014] Furthermore, the hybrid task fusion network includes multiple isomorphic multi-task fusion networks adapted to different image fusion specific tasks, and also includes a component ratio estimation network; the component ratio estimation network is used to calculate the weight of each image fusion specific task based on the source image pairs of all image fusion specific tasks; the multi-task fusion network inherits the task-specific encoder of the specific task separation network with frozen parameters after training, extracts the source image pairs of the image fusion specific task to generate feature images, and then fuses the feature images of each image fusion specific task in combination with the task weights determined by the component ratio estimation network.

[0015] Furthermore, the component scale estimation network jointly processes all source image pairs of the image fusion specific tasks, generates semantic features and a probability vector representing the weight of each image fusion specific task;

[0016] The multi-task fusion network selects a source image pair of a specific image fusion task, and extracts features of each specific image fusion task through the task-specific encoder with frozen parameters; then, the features of each specific image fusion task, as well as the semantic features and probability vectors generated by the component ratio estimation network, are input into the fusion network of each specific image fusion task for fusion, thereby obtaining fused features of each specific image fusion task;

[0017] The fusion features of each image fused with the specific task are input into the decoder for decoding, so as to generate a fusion image of each image fused with the specific task for the selected specific task.

[0018] Furthermore, during the training process, the loss function of the hybrid task fusion network is defined as the weighted sum of the fusion losses of each image fusion specific task and the sum of the cross entropy loss of the classification; the fusion loss of each image fusion specific task is designed according to its task information, and the weight of the fusion loss of each image fusion specific task adopts the weight of each image fusion specific task calculated by the component ratio estimation network.

[0019] Furthermore, the model purification unified network inherits the encoder, component ratio estimation network, fusion network and decoder of the trained hybrid task fusion network, and also introduces a student fusion module and a student decoder; the task features output by the inherited encoder are spliced ​​and input into the student fusion module, and the student fusion module interactively fuses the task features according to the task weights output by the component ratio estimation network to generate interactive fusion features, which are input into the student decoder; the student decoder decodes and generates the corresponding fused image;

[0020] During the training process, the loss function of the model purification unified network is defined as the weighted sum of the feature layer distillation loss and the pixel layer distillation loss; the feature layer distillation loss is defined as the difference between the output of the student fusion module and the output of the inherited fusion network; the pixel layer distillation loss is defined as the difference between the output of the student decoder and the output of the inherited decoder.

[0021] Furthermore, the unified image fusion network inherits the trained model to purify other structures in the unified network except the encoder and the fusion network.

[0022] During the training process, the loss function of the model purification unified network is defined as the sum of the feature layer distillation losses.

[0023] Furthermore, specific image fusion tasks include infrared and visible light fusion, multimodal medical fusion, multi-exposure fusion, and multi-focus fusion.

[0024] The present invention also provides a unified image fusion system of multi-task separation-mixing-purification, the key of which is that it includes a model construction unit, a model training unit and a model application unit; the model construction unit is used to construct a specific task separation network for more than two image fusion specific tasks, and the model training unit is used to train the specific task separation network; the model construction unit is also used to construct a mixed task fusion network based on the trained specific task separation network, and the model training unit is also used to train the mixed task fusion network; the model construction unit is also used to construct a model purification unified network based on the trained mixed task fusion network, and the model training unit is also used to train the model purification unified network; the model construction unit is also used to purify the trained model purification unified network to obtain a unified image fusion network; the model application unit is used to use the unified image fusion network to fuse the image pairs to be fused to obtain a fused image of more than two image fusion specific tasks.

[0025] The present invention provides a unified image fusion method and system for multi-task separation-mixing-purification. First, a reconstruction network model is constructed separately for each specific task to obtain a specific task separation network, thereby explicitly extracting task-specific features; then, it adopts a multi-task mixed fusion modeling strategy to construct a mixed task fusion network based on the specific task separation network to promote fine-grained collaboration between tasks and extract more discriminative fusion components; finally, it constructs a model purification unified network through a knowledge distillation strategy to achieve model purification, adaptively unify task-specific features and cross-task shared features into a fusion output, and obtain a unified image fusion network. Extensive experiments conducted on four typical image fusion tasks show that the proposed method and system significantly surpass existing advanced unified image fusion methods in performance and effectively narrow the gap with specially optimized single-task models. In addition, the unified image fusion network proposed in the present invention exhibits excellent generalization ability in complex multi-task fusion scenarios, fully verifying its effectiveness and practicality. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 This is a flowchart of a unified image fusion method of multi-task separation-mixing-purification provided by an embodiment of the present invention;

[0027] Figure 2 is a structural diagram of a specific task separation network provided by an embodiment of the present invention;

[0028] Figure 3 is a structural diagram of a hybrid task fusion network provided by an embodiment of the present invention;

[0029] Figure 4 This is a structural diagram of a unified network for model purification provided by an embodiment of the present invention;

[0030] Figure 5 1 is a qualitative comparison result diagram of the method in this paper and two types of methods on the MEFB dataset provided by an embodiment of the present invention;

[0031] Figure 6 This is a qualitative comparison result diagram of the method in this paper and two types of methods on the MFIFB dataset provided by an embodiment of the present invention;

[0032] Figure 7 This is a qualitative comparison result diagram of the method in this paper and two types of methods on the TNO and RoadScene datasets provided by an embodiment of the present invention;

[0033] Figure 8 This is the qualitative comparison result of the method in this paper and two types of methods on the Harvard Medical Dataset provided by the embodiment of the present invention;

[0034] Figure 9 This is the qualitative comparison result of five unified image fusion methods on a multi-task fusion dataset provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0035] The following describes the embodiments of the present invention in detail with reference to the accompanying drawings. The embodiments are provided for illustrative purposes only and are not to be construed as limiting the present invention. The accompanying drawings are provided for reference and illustration only and do not constitute a limitation on the scope of protection of the present invention. Many changes may be made to the present invention without departing from the spirit and scope of the present invention.

[0036] The embodiment of the present invention provides a unified image fusion method of multi-task separation-mixing-purification, such as Figure 1 As shown in the flowchart, the steps include:

[0037] Build and train a task-specific separation network for more than two image fusion tasks;

[0038] Building a hybrid task fusion network based on the trained specific task separation network and training the network;

[0039] Based on the trained mixed task fusion network, a model is constructed to purify the unified network and train it;

[0040] The trained model is purified and unified to obtain a unified image fusion network.

[0041] The unified image fusion network is used to fuse the image pairs to be fused, and a fused image for two or more image fusion specific tasks is obtained.

[0042] (1) Phase 1: Separation of specific tasks

[0043] Specifically, the constructed task-specific separation network includes multiple isomorphic encoder-decoder reconstruction networks adapted to different image fusion tasks. The encoder-decoder reconstruction network includes a task-specific encoder, a task-specific decoder, and a task-specific head, which are represented as and The subscript d indicates different tasks. The task-specific encoder and decoder each consist of six densely connected convolutional layers, while the head consists of a two-layer multilayer perceptron. During training, the first-stage loss function is constructed by combining reconstruction loss and contrastive learning loss.

[0044] The first stage aims to achieve explicit separation between tasks at the feature level. Task-specific encoder-decoder reconstruction networks are trained for different tasks to extract key features of each task. Separation of feature distributions between tasks is achieved by constructing a semantic contrastive learning loss. The task-specific features provide feature-level initialization for subsequent stages.

[0045] This embodiment takes the typical MEF, MFF, IVF and MMF tasks as examples. The four scenario-related tasks are represented as follows: , the structure diagram of the specific task separation network is as follows Figure 2 As shown. Figure 2 As shown in Figure 2, the constructed task-specific separation network consists of four isomorphic encoder-decoder reconstruction networks, and image reconstruction is performed separately in four tasks. , use the encoder to extract task-related features and record them as , and reconstructed by a specific task decoder, the reconstructed image pairs are recorded as , the whole process of image reconstruction can be expressed as: , , then the image reconstruction loss for a specific task can be calculated as:

[0046] .

[0047] In order to enhance the discriminative ability of the task-specific encoder, this method also designs a semantic contrast loss function , aims to maximize the similarity between features of the same specific task, while minimizing the similarity between features of different tasks. Specifically, this method first maps the task-related features extracted by the task-specific encoder into two semantic vectors (one-dimensional vectors) through the task-specific head, and then uses the InfoNCE loss to achieve contrastive learning between paired semantic vectors.

[0048] The loss of the image fusion specific task is defined as the weighted sum of the reconstruction loss and the semantic contrast loss. The overall loss function of the first stage is defined as the sum of the losses of all image fusion specific tasks, which is expressed as follows:

[0049] ,

[0050] ,

[0051] in, and are the weight coefficients of reconstruction loss and contrast loss, Represents the 1-norm. T Indicates the number of tasks, N represents the total number of samples, Representation sample i On the mission t No. a The semantic vector obtained on the branch, d represents the dimension of the semantic vector, Represents a Relative branch index, represents the cosine similarity, represents the temperature parameter, controlling the "smoothness" of the contrast distribution, Represents exponential operation.

[0052] (2) The second stage: hybrid task fusion

[0053] The second phase aims to achieve effective interaction of intra-task and inter-task knowledge through a hybrid task fusion network by performing refined task collaboration modeling for specific fusion requirements within each task to achieve hybrid fusion.

[0054] Specifically, the hybrid task fusion network includes multiple isomorphic multi-task fusion networks adapted to different image fusion-specific tasks, as well as a component proportion estimation (CPE) network. The component proportion estimation network is specifically used to quantify the performance of the source image on different task features, thereby effectively evaluating the distribution of task weights interwoven in the image. The multi-task fusion network is based on the pre-trained encoder trained in the first stage. It uses the task-specific encoder inherited from the first stage to extract features from the source image pairs of a specific task, and then fuses the feature images with the task weights in the image pairs determined by the component proportion estimation network. During the training process, the model is dynamically adjusted to adapt to the fusion requirements of multiple tasks by constructing a task-specific loss function using weights.

[0055] This embodiment takes the typical four types of tasks, MEF, MFF, IVF and MMF, as examples. The structure of the multi-task fusion network is as follows: Figure 3 As shown. Figure 3 As shown, the multi-task fusion network selects a pair of source images and (a specific task), four sets of task-specific features are extracted through four pre-trained encoders (the encoder of the encoder-decoder reconstruction network, the parameters are frozen after the first stage of training), which are and At the same time, the encoder and classification header The CPE network composed of the two processes jointly processes all the source image pairs for the image fusion specific task. and , generate semantic features and the probability vectors representing the four tasks The training process of the CPE network is optimized by minimizing the cross entropy loss between the actual task labels and the predicted probabilities.

[0056] Next, the corresponding task features and , and semantic features and the probability vector Input to four task-specific fusion networks (denoted as ). Each fusion network consists of densely connected low-level feature fusion blocks, mid-level feature fusion blocks, and high-level feature fusion blocks, which are used to fuse low-level features, mid-level features, and high-level features respectively. Finally, the low-level, mid-level, and high-level features are fused to obtain the fusion features of each image fusion specific task. Specifically, This is a multi-scale fusion module. The low-level feature fusion block uses convolutional layers to fuse low-level features. The mid-level feature fusion block uses SEBlock to fuse mid-level features. SEBlock includes the following components: Global Average Pooling (GAP), Multilayer Perceptron (MLP) network, 3×3 Convolutional Layer (Conv), Batch Normalization (BN), and Leaky Rectified Linear Unit (LReLU). The high-level feature fusion block fuses high-level features through the Transformer module, which includes the following components: multi-dimensional convolutional head transposed attention and a gated convolutional feedforward network. The input dimension of the Transformer module is 64, and the number of attention heads is 4.

[0057] It is worth noting that the fusion process is adaptively guided by the task weights identified in the source images, ensuring that task-specific features are integrated in a contextually appropriate manner. is input into the decoder to generate a fused image customized for a specific task .

[0058] On this basis, this method designs different fusion loss functions for different task information in the source image to achieve adaptive fusion. Specifically, for the multi-exposure image fusion (MEF) task, the hybrid task fusion network focuses on balancing the brightness level of the image, so the brightness loss is introduced. , contrast loss And the structural similarity loss at the image block level For the multi-focus fusion (MFF) task, the goal of the hybrid task fusion network is to maintain the local maximum gradient in the source image pair, so the image patch level gradient loss is introduced. and contrast loss For infrared and visible image fusion (IVF) and medical image fusion (MMF) tasks, the goal of the hybrid task fusion network is to preserve the contrast and texture details in the image pair, so the brightness loss is used. , contrast loss and weighted maximum gradient loss The loss function of the second-stage hybrid task fusion network is defined as the sum of the weighted sum of the specific task fusion losses of each image fusion and the cross entropy loss of the classification: , Indicates the corresponding task The weight of , the value range is (0,1), Indicates the The loss function of the fusion task, represents the cross entropy loss for classification.

[0059] Specifically, for the multi-exposure fusion (MEF) task, this method focuses more on balancing the brightness level of the image, thus introducing a brightness loss , contrast loss and structural similarity loss at the image patch level .

[0060] By utilizing loss, the contrast of the fused image is significantly improved. The loss is defined as: , represents the structure tensor, Represents the fused image of all fusion tasks The structure tensor of Indicated by Two-channel image formed by cascade The structure tensor of represents the F-norm.

[0061] The loss is defined as:

[0062] ,

[0063] in, 、 Represents specific tasks d The source image pair and The structure tensor of 、 represents the corresponding weight, represents the convolution operation, express Activation function.

[0064] Defined as:

[0065] ,

[0066] in, Represent the height and width of the fused image Z respectively.

[0067] Therefore, the loss function of the MEF task is defined as: , 、 Respectively represent the MEF tasks Loss and loss.

[0068] For the multi-focus image fusion (MFF) task, the goal is to preserve the local maximum gradient in the source image pair, involving a gradient loss at the patch level, denoted as Therefore, the loss of the multi-focus image fusion task consists of three parts, defined as: , gradient loss Defined as: ,in represents the number of patches in the image, , Indicates the source image pair in d Under the task, corresponding to k The gradient of the patch area, Indicates that the fused image Z is k The gradient of the patch area, represents the 2-norm. 、 Represent the loss and loss.

[0069] For infrared and visible image fusion (IVF) and medical image fusion (MMF) tasks, the goal is to preserve the contrast and texture details in the image pair. Therefore, this method introduces an intensity loss , contrast loss and weighted maximum gradient loss , which is defined as:

[0070] ,

[0071] in, represents the gradient of the fused image, and Represents specific tasks d The source image pair and absmax(·) means taking the absolute values ​​of the corresponding pixels of the two gradient maps and selecting the larger value.

[0072] Therefore, the loss function for the infrared and visible image fusion (IVF) and medical image fusion (MMF) tasks consists of these three parts, defined as follows:

[0073] ,

[0074] ,

[0075] in, 、 Respectively represent the IVF task Loss and loss, 、 Respectively represent the MMF tasks Loss and loss. and Represents the IVF task and MMF task respectively. loss.

[0076] (3) The third stage: model purification and unification

[0077] In this stage, a model purification unified network is proposed. This model purification unified network uses knowledge distillation and, based on the proportion of each task information in the source image, aggregates the fusion components from different tasks obtained in the second stage to generate the final fused image. This distillation process is performed at both the feature layer and the pixel layer. Figure 4 As shown, the model purification unified network inherits the encoder, component ratio estimation network, fusion network and decoder of the hybrid task fusion network obtained by the second stage training. The parameters of the encoder, component ratio estimation network and fusion network have been frozen and can generate task-specific features. , fusion features of each task And the fusion image of each task. The model purification unified network introduces a student fusion module , which performs feature layer adaptive distillation and uses a student decoder to perform pixel-level adaptive distillation.

[0078] Student Integration Module It is composed of densely connected convolutional layers and is an efficient feature layer aggregator that can adaptively interact and fuse specific task features to generate the final fused features. Specifically, this method combines the specific task characteristics To stitch, input Interactive fusion is performed in . The training strategy aims to make and fusion features The similarity between them is as close as possible, and this similarity is evaluated by the frozen CPE network. Through feature layer distillation, features from different tasks are adaptively aggregated according to the proportional weight of their respective information in the source image. Formally, the feature layer distillation loss Defined as:

[0079] ,

[0080] represents the output of the student encoder for the source image pair ( and ) generated fusion features, Represents the pair of source images output by the fusion network of the inherited hybrid task fusion network ( and ) generated fusion features.

[0081] Next, Input to student decoder In this method, a training strategy is used to train , ensuring that the final fused image is as similar as possible to the fused image of each task, and the similarity is also evaluated by the frozen CPE network. Through the pixel-level distillation strategy, the knowledge from multiple task-specific decoders is transferred to the student decoder, and the weights Dynamically adjust the transfer process. This method helps to generate the final pixel-level fused image. Formally, the pixel-level distillation loss Defined as:

[0082] ,

[0083] in, Indicates student decoder Output source image pair ( and ) generated fusion image, Denotes the decoder output of the inherited mixed task fusion network for the source image pair ( and ) generated fused image.

[0084] Finally, the overall objective loss function of the third stage is defined as the weighted sum of the feature layer distillation loss and the pixel layer distillation loss:

[0085] ,

[0086] in, and are the weighting coefficients of feature-level distillation loss and pixel-level distillation loss, respectively.

[0087] (4) Model application

[0088] After training the unified model purification network, the network is refined, retaining all structures except the encoder and fusion network to form a unified image fusion network. In practical applications, multiple pairs of task-specific images to be fused are fed into the unified image fusion network for fusion, resulting in multiple task-specific fused images.

[0089] It should be noted that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved. This embodiment is not limited here.

[0090] Corresponding to the above-mentioned multi-task separation-mixing-purification unified image fusion method, this embodiment also provides a multi-task separation-mixing-purification unified image fusion system, which includes a model construction unit, a model training unit, and a model application unit. The model construction unit is used to construct a specific task separation network for two or more image fusion specific tasks, the model training unit is used to train the specific task separation network, the model construction unit is also used to construct a mixed task fusion network based on the trained specific task separation network, the model training unit is also used to train the mixed task fusion network, the model construction unit is also used to construct a model purification unified network based on the trained mixed task fusion network, the model training unit is also used to train the model purification unified network, and the model construction unit is also used to purify the trained model purification unified network to obtain a unified image fusion network. The model application unit is used to fuse the image pairs to be fused using the unified image fusion network to obtain a fused image for two or more image fusion specific tasks.

[0091] The embodiments described herein can be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0092] Computer programs for implementing the methods and systems of the present invention can be written in any combination of one or more programming languages ​​and stored in a computer-readable storage medium. These computer programs can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The computer program can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0093] Computer-readable storage media can be tangible media that can contain or store a computer program for use with an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. Computer-readable storage media can include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, computer-readable storage media can be a machine-readable signal medium. More specific examples of machine-readable storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, compact disc read-only memories (CD ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0094] In summary, the present invention provides a unified image fusion method and system of multi-task separation-mixing-purification. First, a reconstruction network model is constructed separately for each specific task to obtain a specific task separation network, thereby explicitly extracting task-specific features; then, it adopts a multi-task mixed fusion modeling strategy to construct a mixed task fusion network based on the specific task separation network to promote fine-grained collaboration between tasks and extract more discriminative fusion components; finally, it constructs a model purification unified network through a knowledge distillation strategy to achieve model purification, adaptively unify task-specific features and cross-task shared features into a fusion output, and obtain a unified image fusion network. A large number of experiments conducted on four typical image fusion tasks show that the proposed method and system significantly surpass existing advanced unified image fusion methods in performance, and effectively narrow the gap with specially optimized single-task models. In addition, the unified image fusion network obtained by the present invention exhibits excellent generalization ability in complex multi-task fusion scenarios, fully verifying its effectiveness and practicality.

[0095] In order to verify the effect of the present invention, a large number of experiments were carried out below.

[0096] (1) Dataset Introduction

[0097] 1) Training set

[0098] The training dataset consists of a mixture of datasets from multiple image fusion tasks. Specifically, for the multi-exposure image fusion task, 589 image pairs were selected from the SCIE dataset, using underexposed and overexposed images in the SCIE sequence as input. For the multi-focus image fusion task, 710 image pairs from the RealMFF dataset were used. For the infrared-visible image fusion task, 1,000 image pairs were selected from the LLVIP dataset, and 2,000 image pairs were selected from the M3FD dataset. For the medical image fusion task, 600 image pairs were selected from the Harvard Medical Dataset, which covers three categories: MRI-CT, MRI-CT, and MRI-PET.

[0099] 2) Benchmark evaluation dataset

[0100] This example evaluates the proposed method and system on two image dataset settings:

[0101] Single-Task Fusion Datasets: Evaluations were conducted on four existing benchmark datasets, covering multi-exposure image fusion, multi-focus image fusion, infrared and visible light image fusion, and medical image fusion. For multi-exposure image fusion, the MEFB dataset was used. For multi-focus image fusion, the MFIFB benchmark setting was followed. For infrared and visible light image fusion, TNO and RoadScene were used for testing. Finally, for medical image fusion, testing was conducted on MRI-CT, MRI-PET, and MRI-SPECT from the Harvard Medical Dataset.

[0102] Multi-task fusion dataset: This example is also tested on a multi-task fusion dataset to evaluate its adaptability when multiple task features exist simultaneously.

[0103] (2) Evaluation indicators

[0104] This experiment selected six metrics from four categories of evaluation indicators to quantitatively measure the fusion results. These metrics include AG (average gradient), EN (information entropy), Qabf (fusion quality index), SF (spatial frequency), MS-SSIM (multi-scale structural similarity), and FMI (feature mutual information).

[0105] (3) Comparative method

[0106] Two experimental settings are used: (1) unified fusion method, and (2) task-specific fusion method. Specifically, the baselines for the unified fusion method include PMGI, CU-Net, SDNet, DIF-Net, U2Fusion, SwinFusion, DeFusion, MUFusion, and TC-MoA. In addition, for the comparison of fusion methods for specific tasks, in the multi-focus image fusion task, the method in this chapter is compared with the ZMFF and MFF-GAN methods; in the multi-exposure image fusion task, it is compared with the TransMEF and BHFMEF methods; in the infrared and visible light fusion task, it is compared with the recent DDFM and LRRNet methods; in the medical image fusion task, the performance is compared with the MATR and DDFM methods.

[0107] (4) Implementation details

[0108] The experiments were conducted on a server equipped with an NVIDIA GeForce RTX 1080Ti GPU. The training samples of the four fusion tasks were randomly cropped into 128×128 image blocks during training. The training iterations were 90 times, divided into three stages, with 10 iterations in the first and third stages and 70 iterations in the second stage. The batch size was set to 8, and the Adam optimizer was used with an initial learning rate of 10. -4, decreasing by 0.5 every 10 epochs. To augment the training data, 20,000 image patches were cropped for each dataset. Furthermore, the classes were assigned values ​​of 0, 1, 2, and 3, corresponding to the MFF, IVF, MEF, and MMF tasks, respectively.

[0109] (5) Comparison with advanced methods

[0110] 1) Multi-exposure image fusion

[0111] The quantitative results on the MEFB dataset are shown in Table 1. Our method (Ours) consistently achieves state-of-the-art results among unified image fusion methods and task-specific fusion methods, especially in AG and SF metrics (focusing on details and textures). The qualitative comparison results of our method with the two types of methods on the MEFB dataset are shown in Table 1. Figure 5 ,from Figure 5 It can be seen that our method can well preserve texture details and achieve uniform lighting distribution, while other methods are insufficient in detail preservation or exposure consistency.

[0112] Table 1: Quantitative results on the MEFB dataset

[0113]

[0114] 2) Multi-focus image fusion

[0115] Table 2 shows the quantitative results of the MFF task on the MFIFB dataset. It is obvious that the proposed method performs well in SF, Qabf and FMI indicators, and is close to the optimal value in other indicators. Further qualitative results can be found in Figure 6 : Our method can maintain excellent texture and color consistency in both near-focus and far-focus areas, while CU-Net and DIF-Net are prone to blurring.

[0116] Table 2: Quantitative results of the MFF task on the MFIFB dataset

[0117]

[0118] 3) Infrared-visible light image fusion

[0119] The quantitative results of the IVF task on the TNO dataset are shown in Table 3, which surpasses the unified fusion method and the task-specific fusion method in terms of detail preservation, edge preservation, and structural fidelity. Figure 7 : While other methods often perform poorly in saliency and contrast, resulting in blurred edges and loss of details, our method stably produces clear, high-contrast fused images.

[0120] Table 3: Quantitative results of the IVF task on the TNO dataset

[0121]

[0122] 4) Medical Image Fusion

[0123] The qualitative and quantitative results of the MMF task are shown in Figure 8 and Table 4. The proposed method shows significant advantages in preserving image contrast and details.

[0124] Table 4: Quantitative results of the MMF task

[0125]

[0126] 5) Multi-task fusion

[0127] The quantitative results on the multi-task fusion dataset are shown in Table 5. It can be seen from Table 5 that even in the scenario where task-specific information is intertwined, the proposed model still outperforms the state-of-the-art methods in various fusion tasks. Figure 9 Qualitative comparison results of five unified image fusion methods are given on a multi-task fusion dataset, showing that the proposed model can recover more edge and texture details from the scene.

[0128] Table 5: Quantitative results on multi-task fusion dataset

[0129]

[0130] 6) Ablation experiment

[0131] The ablation experiment results on the MEFB task are shown in Table 6. As can be seen from Table 6, the unified image fusion network obtained through the three stages designed by the present invention has the best performance.

[0132] Table 6: Ablation experiment results on the MEFB task

[0133]

[0134] Extensive experiments on four typical image fusion tasks demonstrate that the proposed method and system significantly outperform existing state-of-the-art unified image fusion methods and effectively narrow the performance gap with specially optimized single-task models. Furthermore, the proposed unified image fusion network demonstrates excellent generalization capabilities in complex multi-task fusion scenarios, fully demonstrating its effectiveness and practicality.

[0135] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be considered as equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. A unified image fusion method of multi-task separation-mixing-purification, characterized by: Including steps: Build and train a task-specific separation network for more than two image fusion tasks; The specific task separation network includes multiple isomorphic encoder-decoder reconstruction networks adapted to different image fusion specific tasks, and the encoder-decoder reconstruction network includes a specific task encoder, a specific task decoder, and a specific task head; the specific task encoder extracts task-related features of the source image pair of the image fusion specific task and inputs them into the corresponding specific task decoder and the specific task head; the specific task decoder performs image reconstruction based on the task-related features to generate a reconstructed image pair; the specific task head maps the task-related features into two semantic vectors; Building a hybrid task fusion network based on the trained specific task separation network and training the network; The hybrid task fusion network includes multiple isomorphic multi-task fusion networks adapted to different image fusion specific tasks, and also includes a component ratio estimation network; the component ratio estimation network is used to calculate the weight of each image fusion specific task based on the source image pairs of all image fusion specific tasks; the multi-task fusion network inherits the task-specific encoder of the specific task separation network with frozen parameters after training, extracts the source image pairs of the image fusion specific task to generate feature images, and then fuses the feature images of each image fusion specific task in combination with the task weights determined by the component ratio estimation network; Based on the trained mixed task fusion network, a model is constructed to purify the unified network and train it; The model purification unified network inherits the encoder, component ratio estimation network, fusion network and decoder of the trained hybrid task fusion network, and also introduces a student fusion module and a student decoder; the task features output by the inherited encoder are spliced ​​and input into the student fusion module, and the student fusion module interactively fuses the task features according to the task weights output by the component ratio estimation network to generate interactive fusion features, which are input into the student decoder; the student decoder decodes and generates the corresponding fused image; the trained model purification unified network is purified to obtain a unified image fusion network; The unified image fusion network inherits the trained model to purify other structures in the unified network except the encoder and the fusion network; The unified image fusion network is used to fuse the image pairs to be fused, and a fused image for two or more image fusion specific tasks is obtained.

2. The unified image fusion method of multi-task separation-mixing-purification according to claim 1, characterized in that: During the training process, the loss function of the specific task separation network is defined as the weighted sum of the reconstruction loss of all image fusion specific tasks and the semantic contrast loss, wherein the reconstruction loss is defined as the difference between the source image pair and the reconstructed image pair; and the semantic contrast loss is defined as the InfoNCE loss based on the semantic vector.

3. The unified image fusion method of multi-task separation-mixing-purification according to claim 1, characterized in that: The component scale estimation network jointly processes all source image pairs of the image fusion specific task, generates semantic features and a probability vector representing the weight of each image fusion specific task; The multi-task fusion network selects a source image pair of a specific image fusion task, and extracts features of each specific image fusion task through the task-specific encoder with frozen parameters; then, the features of each specific image fusion task, as well as the semantic features and probability vectors generated by the component ratio estimation network, are input into the fusion network of each specific image fusion task for fusion, thereby obtaining fused features of each specific image fusion task; The fusion features of each image fused with the specific task are input into the decoder for decoding, so as to generate a fusion image of each image fused with the specific task for the selected specific task.

4. The unified image fusion method of multi-task separation-mixing-purification according to claim 3, characterized in that: During the training process, the loss function of the hybrid task fusion network is defined as the weighted sum of the fusion losses of each image fusion specific task and the sum of the cross entropy loss of the classification; the fusion loss of each image fusion specific task is designed according to its task information, and the weight of the fusion loss of each image fusion specific task adopts the weight of each image fusion specific task calculated by the component ratio estimation network.

5. The unified image fusion method of multi-task separation-mixing-purification according to claim 4, characterized in that: During the training process, the loss function of the model purification unified network is defined as the weighted sum of the feature layer distillation loss and the pixel layer distillation loss; the feature layer distillation loss is defined as the difference between the output of the student fusion module and the output of the inherited fusion network; the pixel layer distillation loss is defined as the difference between the output of the student decoder and the output of the inherited decoder.

6. The unified image fusion method of multi-task separation-mixing-purification according to any one of claims 1 to 5, characterized in that: Specific image fusion tasks include infrared and visible light fusion, multimodal medical fusion, multi-exposure fusion, and multi-focus fusion.

7. A multi-task separation-mixing-purification unified image fusion system, applied to the multi-task separation-mixing-purification unified image fusion method according to any one of claims 1 to 6, characterized in that: It includes a model construction unit, a model training unit and a model application unit; the model construction unit is used to construct a specific task separation network for more than two image fusion specific tasks, and the model training unit is used to train the specific task separation network; the model construction unit is also used to construct a mixed task fusion network based on the trained specific task separation network, and the model training unit is also used to train the mixed task fusion network; the model construction unit is also used to construct a model purification unified network based on the trained mixed task fusion network, and the model training unit is also used to train the model purification unified network; the model construction unit is also used to purify the trained model purification unified network to obtain a unified image fusion network; the model application unit is used to use the unified image fusion network to fuse the image pairs to be fused to obtain a fused image for more than two image fusion specific tasks.

Images