A unified image fusion method and system based on adaptive distribution difference perception

Abstract

The present invention relates to the field of image processing technology, and specifically discloses a unified image fusion method and system with adaptive distribution difference perception. In view of the feature distribution differences between source images of different tasks, a unified image fusion network is designed. The network is designed with a distribution difference perception fuser to dynamically distinguish the distribution differences between low-frequency and high-frequency features in the image, thereby fine-grainedly adjusting the fusion strategy to adapt to the feature differences of different source images. The present invention also proposes a weight-adaptive distribution difference perception loss function to balance the contribution of different loss terms to the fusion task, ensuring the robustness and generalization ability of the network in complex scenes. The trained unified image fusion network can not only effectively control the dominant deviations in different fusion tasks, but also integrate multiple fusion tasks such as multi-modality, multi-exposure and multi-focus into a unified framework. Experimental results show that the fusion effect of the present invention in multi-task scenarios is better than that of existing methods.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a unified image fusion method and system for adaptive distribution difference perception. Background Art

[0002] Image fusion aims to integrate complementary information from the same scene captured by multiple sensors into a single image. It has broad applications in medical image enhancement and diagnosis, intelligent traffic environment perception, intelligent security, high dynamic range photography, and remote sensing image processing. One of the main challenges facing current image fusion methods is how to effectively capture the correlation between source images for different fusion tasks and integrate heterogeneous information from different sources. Most existing methods, while adopting a correlation paradigm based on static fusion rules, ignore the dynamic adaptability of modalities across different tasks, resulting in a model biased towards specific tasks and unable to generalize to a wider range of task scenarios.

[0003] In recent years, researchers have begun to focus on the correlation paradigm of adaptive dynamic fusion to adapt to diverse task scenarios. However, these methods either rely on hand-crafted optimization objectives, which limits their stability in complex scenes; or rely on certain assumptions that fail to fully reflect the distribution relationships between images in multiple task scenarios. In reality, scene distributions vary significantly between different tasks, and even for the same task, the distributions from different datasets may differ. For example, the feature distributions of multi-exposure images vary significantly, and the heterogeneity between infrared and visible light images leads to greater scene distribution complexity. Summary of the Invention

[0004] The present invention provides a unified image fusion method and system with adaptive distribution difference perception, which solves the technical problem that existing multi-source image fusion methods are insufficiently adaptable in complex task scenarios.

[0005] To solve the above technical problems, the present invention provides a unified image fusion method with adaptive distribution difference perception, comprising the steps of:

[0006] Constructing a unified image fusion network; training the unified image fusion network; and fusing the source image pairs to be fused using the trained unified image fusion network to obtain a unified fused image.

[0007] The unified image fusion network includes a cascaded first encoder, a first decoder, a cascaded second encoder, a second decoder, and also includes a distributed difference perception fusion device and an output layer; the first encoder and the second encoder have the same structure, both including a plurality of cascaded encoding layers; the first decoder and the second decoder have the same structure, both including a plurality of cascaded decoding layers; the distributed difference perception fusion device performs distributed difference perception fusion on the output of at least one encoding layer of the same level, and then splices the outputs with the outputs of the two encoding layers of the same level, and then inputs the encoding layer or decoding layer of the next level; the distributed difference perception fusion device also performs distributed difference perception fusion on the output of at least one decoding layer of the same level, and then splices the outputs with the outputs of the two decoding layers of the same level, and then inputs the decoding layer or output layer of the next level; the output layer is used to obtain a unified fused image according to the outputs of the first decoder and the second decoder.

[0008] Furthermore, the distribution difference perception fusion device is used to input two features and Perform distribution difference perception fusion to obtain high and low frequency fusion features , specifically including the steps:

[0009] calculate and The difference weight of low-frequency feature distribution between and high-frequency feature distribution difference weight ;

[0010] based on right 、 Low-frequency characteristics 、 After adaptive modulation of different brightness channels, weighted fusion is performed to obtain low-frequency fusion features. ;

[0011] based on right 、 The high frequency part 、 After performing edge feature enhancement, weighted fusion is performed to obtain high-frequency fusion features. ;

[0012] right and Perform weighted fusion to obtain high and low frequency fusion features .

[0013] Furthermore, the calculation and The difference weight of low-frequency feature distribution between and high-frequency feature distribution difference weight , specifically including:

[0014] extract The low-frequency components and high-frequency components ,as well as The low-frequency components and high-frequency components ;

[0015] Calculate low-frequency components and The mean and variance between them are used to calculate the low-frequency feature distribution difference weights. ;

[0016] Calculate high-frequency components and The mean and variance between them are used to calculate the high-frequency feature distribution difference weights. .

[0017] Furthermore, the low-frequency feature distribution difference weight is calculated based on the mean and variance Specifically include:

[0018] calculate 、 The mean of the low-frequency mean difference is obtained by taking the absolute value of the difference. ;

[0019] calculate 、 The variance of the low-frequency variance is obtained by taking the absolute value of the difference ;

[0020] Will and Splicing is performed to obtain the low-frequency feature distribution difference vector ;

[0021] Will After passing through two layers of fully connected layers and corresponding activation functions, the low-frequency feature distribution difference weight is obtained ;

[0022] Based on high-frequency components and The mean and variance between The same process calculation .

[0023] Furthermore, the right 、 Low-frequency characteristics 、 After adaptive modulation of different brightness channels, weighted fusion is performed to obtain low-frequency fusion features. , specifically including:

[0024] Using hybrid expert modules and Perform adaptive brightness normalization respectively and get 、 ; The router weight of the hybrid expert module is Apply Softmax and Top-K strategies to obtain;

[0025] Through the fully connected layer and activation function 、 Process them separately to get the corresponding channel attention weights 、 ;

[0026] based on right 、 Perform weighted fusion to obtain ;based on right 、 Perform weighted fusion to obtain ;

[0027] Will and After splicing, the low-frequency fusion weight is obtained through two fully connected layers and corresponding activation functions. ;

[0028] based on right 、 Perform weighted fusion to obtain low-frequency fusion features .

[0029] Furthermore, the right 、 The high frequency part 、 After performing edge feature enhancement, weighted fusion is performed to obtain high-frequency fusion features. , specifically including:

[0030] By designing the variance filter 、 Generate the corresponding significance probability map and ;

[0031] Will 、 With their respective and Multiply to get the detail enhancement feature map and ;

[0032] based on right 、 Perform weighted summation to obtain the final edge enhancement feature ;based on right 、 Perform weighted summation to obtain the final edge enhancement feature ;

[0033] Will and After splicing, the high-frequency fusion weight is obtained through two fully connected layers and corresponding activation functions ;

[0034] based on right 、 Perform weighted fusion to obtain high-frequency fusion features .

[0035] Furthermore, the and Perform weighted fusion to obtain high and low frequency fusion features , specifically including:

[0036] right and Generate initial fusion features through global average pooling layer and fully connected layer ;

[0037] right The low-frequency channel weight is obtained through the low-frequency component fully connected layer and the high-frequency component fully connected layer and high-frequency channel weights ;

[0038] Will 、 After splicing, the Softmax function is used to normalize the splicing result, and then the normalized result is split into channels to obtain the low-frequency channel fusion weight and high-frequency channel fusion weights ;

[0039] based on 、 right and Perform weighted fusion to obtain high and low frequency fusion features .

[0040] Furthermore, in the process of training the unified image fusion network, the loss function used is designed to be intensity loss , detail loss and contrast loss The weighted sum of the strength loss , detail loss , contrast loss are the brightness difference, high frequency component difference, and contrast difference between the source image and the unified fused image, respectively.

[0041] Furthermore, the strength loss , detail loss and contrast loss Corresponding weight , calculated by the following steps:

[0042] Calculate the mean absolute difference between pairs of source images The absolute value of the difference from the standard deviation ;

[0043] Will and After multi-layer perceptron and Softmax, channel-level separation is performed to obtain weights .

[0044] The present invention also provides a unified image fusion system with adaptive distribution difference perception, the key of which is that it includes a model construction module, a model training module and a model application module. The model construction module is used to construct a unified image fusion network, and the model training module is used to train the unified image fusion network; the model application module is used to apply the trained unified image fusion network to fuse the source image pairs to obtain a unified fused image.

[0045] The present invention provides a unified image fusion method and system with adaptive distribution difference perception. Aiming at the feature distribution differences between source images of different tasks, a unified image fusion network is designed. The network is designed with a distribution difference perception fuser to dynamically distinguish the distribution differences between low-frequency and high-frequency features in the image, thereby fine-grainedly adjusting the fusion strategy to adapt to the feature differences of different source images. On this basis, the present invention also proposes a weight-adaptive distribution difference perception loss function to balance the contribution of different loss terms to the fusion task, ensuring the robustness and generalization ability of the network in complex scenes. The trained unified image fusion network can not only effectively control the dominant deviations in different fusion tasks, but also integrate multiple fusion tasks such as multi-modality, multi-exposure and multi-focus into a unified framework. Experimental results show that the fusion effect of the present invention in multi-task scenarios is better than that of existing methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 is a structural diagram of a unified image fusion network provided by an embodiment of the present invention;

[0047] Figure 2 is a structural diagram of a distribution difference perception fusion device provided by an embodiment of the present invention;

[0048] Figure 3 This is a qualitative comparison diagram of the multi-exposure image fusion task on the MEFB dataset provided by an embodiment of the present invention;

[0049] Figure 4 is a qualitative comparison diagram of the multi-focus image fusion task on the MFIFB dataset provided by an embodiment of the present invention;

[0050] Figure 5 This is a qualitative comparison chart of the infrared and visible light image fusion task on the TNO dataset provided by an embodiment of the present invention;

[0051] Figure 6 This is a qualitative comparison chart of the medical image fusion task on the Harvard Medical Dataset provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0052] The following describes the embodiments of the present invention in detail with reference to the accompanying drawings. The embodiments are provided for illustrative purposes only and are not to be construed as limiting the present invention. The accompanying drawings are provided for reference and illustration only and do not constitute a limitation on the scope of protection of the present invention. Many changes may be made to the present invention without departing from the spirit and scope of the present invention.

[0053] In order to achieve adaptive fusion of paired source images in different fusion task scenarios, an embodiment of the present invention provides a unified image fusion method with adaptive distribution difference perception, which includes the following steps:

[0054] Build a unified image fusion network;

[0055] Train the unified image fusion network;

[0056] The trained unified image fusion network is used to fuse the source image pairs to obtain a unified fused image.

[0057] Figure 1 This is the structural diagram of the unified image fusion network. Figure 1As shown, the network includes a cascaded first encoder, a first decoder, a cascaded second encoder, a second decoder, and a distribution difference perception fusion device (DDWF) and an output layer; the first encoder and the second encoder have the same structure and share parameters, and both include a plurality of cascaded encoding layers; the first decoder and the second decoder have the same structure and both include a plurality of cascaded decoding layers; the distribution difference perception fusion device performs distribution difference perception fusion on the output of at least one encoding layer of the same level, and then splices the outputs of the two encoding layers of the same level, and then inputs the encoding layer or decoding layer of the next level; the distribution difference perception fusion device also performs distribution difference perception fusion on the output of at least one decoding layer of the same level, and then splices the outputs of the two decoding layers of the same level, and then inputs the decoding layer or output layer of the next level; the output layer is used to obtain a unified fused image according to the outputs of the first decoder and the second decoder.

[0058] like Figure 1 As shown, given a pair of source images ( Represent the height and width of the image respectively, and 3 represents the number of channels), and the fusion result is recorded as In the unified image fusion network, the source image pair is first Each input is fed into a patch embedding layer to extract feature tokens, which are then fed into the first and second encoders, respectively. The entire network is built based on the Vision Transformer (ViT) architecture, with both the encoder and decoder consisting of multiple Transformer blocks. This means that both the encoding and decoding layers utilize Transformer blocks. To perceive and control data distribution differences in multi-task scenarios, this method proposes a distribution difference-aware fusion agent. By introducing a distribution difference sensor, a brightness control module, and an edge control module, this method fine-grainedly adjusts the feature fusion process. Furthermore, this method designs a distribution difference-aware loss function to guide feature extraction and fusion.

[0059] Figure 2 This is the structural diagram of the distribution difference perception fusion. Figure 2 As shown in the figure, the distribution difference perception fusion includes a distribution difference sensor, a brightness control module, an edge control module and a high-low frequency feature modulator (HF-LFFM). Overall, the distribution difference perception fusion is used to transform the two input features into a single pixel. and Perform distribution difference perception fusion to obtain high and low frequency fusion features , specifically including the steps:

[0060] S1, calculated using distribution difference perceptron and The difference weight of low-frequency feature distribution between and high-frequency feature distribution difference weight ;

[0061] S2, using brightness control module based on right 、 Low-frequency characteristics 、 After adaptive modulation of different brightness channels, weighted fusion is performed to obtain low-frequency fusion features. ;

[0062] S3, using edge control module based on right 、 The high frequency part 、 After performing edge feature enhancement, weighted fusion is performed to obtain high-frequency fusion features. ;

[0063] S4, using high and low frequency characteristic modulator and Perform weighted fusion to obtain high and low frequency fusion features .

[0064] In this embodiment, step S1 (calculating and The difference weight of low-frequency feature distribution between and high-frequency feature distribution difference weight ) specifically includes the following steps:

[0065] S11, extract using feature decoupling module The low-frequency components and high-frequency components ,as well as The low-frequency components and high-frequency components ;

[0066] S12. Calculate low-frequency components using distribution difference weight generation module and The mean and variance between them are used to calculate the low-frequency feature distribution difference weights. , calculate the high frequency components and The mean and variance between them are used to calculate the high-frequency feature distribution difference weights. .

[0067] In step S11, in order to achieve dynamic decoupling of feature maps, this method uses learnable low-pass and high-pass filters to form a feature decoupling module to effectively separate low-frequency and high-frequency components. In addition, by sharing filters in the group dimension, the model complexity can be effectively reduced while maintaining feature diversity. Specifically, given any input feature , represents the spatial dimension, is the number of channels. After passing through the filter generation layer, a low-pass filter is generated for each set of inputs. The formula of the filter generation layer is as follows:

[0068] ,

[0069] in, from Transformed into , is the kernel size of the low-pass filter, Indicates the number of groups. BN is batch normalization, W is the convolution parameter, and GAP is the global average pooling. The filter is applied with the Softmax function. Group-based operation has fewer parameters and lower complexity than generating filters for each pixel. In addition, to obtain a high-pass filter, this method uses the identity matrix (whose center value is 1 and the rest are 0) Subtracting the low-pass filter gives:

[0070] .

[0071] Next, we will input the features Divided by group, each group feature is recorded as ,in is the group index, and , apply a low-pass filter to each set of features and high-pass filter (All sizes are ), thereby obtaining the corresponding low-frequency and high-frequency components, the calculation formula is:

[0072] ,

[0073] ,

[0074] in, is the channel index, and is the spatial coordinate, Indicates the offset of the filter kernel.

[0075] In summary, according to the definition of the feature decoupling module, given two source images and , after the feature decoupling module The low-frequency and high-frequency components are obtained respectively, and the formula is:

[0076] ,

[0077] .

[0078] In step S12, after decomposing the feature map into different frequency components, the distribution difference weight generation module generates the corresponding perceptual weight by calculating the mean and variance differences of the low-frequency and high-frequency features. This weight essentially reflects the degree of difference between the two source images in key features (such as brightness, edge information, etc.). When there are large differences in these features between the source images, it indicates that more sufficient adjustment and adaptation are required during the fusion process to ensure that the final fusion result is of high quality and consistency. Therefore, this weight can be regarded as a quantitative indicator of the difficulty of fusion across tasks and different scenarios. Specifically, this method first constructs a feature distribution difference vector and quantifies the degree of feature change between the source images by calculating the mean and variance differences of the low-frequency and high-frequency components of the image, thereby reflecting their differences.

[0079] Specifically, step S12 includes the following steps:

[0080] S121, calculation 、 The mean of the low-frequency mean difference is obtained by taking the absolute value of the difference. ;

[0081] S122, calculation 、 The variance of the low-frequency variance is obtained by taking the absolute value of the difference ;

[0082] S123, will and Splicing is performed to obtain the low-frequency feature distribution difference vector ;

[0083] S124, will After passing through two layers of fully connected layers and corresponding activation functions, the low-frequency feature distribution difference weight is obtained ;

[0084] S125, based on high frequency components and The mean and variance between The same process calculation .

[0085] Taking low-frequency features as an example, low-frequency components mainly capture the global structure and brightness information of the image. Its mean represents the overall brightness distribution, while the variance reflects the contrast or texture uniformity of the image. By calculating the mean and variance of two low-frequency feature maps respectively and taking their absolute difference, a low-frequency difference vector can be obtained. :

[0086] ,

[0087] in, is a channel-level splicing operation, Represents the dimension of the low-frequency feature map.

[0088] Similarly, for high-frequency feature distribution difference vector , its calculation formula is as follows:

[0089] ,

[0090] in, Represents the dimension of the high-frequency feature map, 、 represent the high-frequency mean difference and high-frequency variance difference, respectively.

[0091] In order to further generate distribution difference weights, this method designs a dynamic perception module. This module takes the feature difference vectors of high-frequency and low-frequency as input, passes through the fully connected layer and the activation layer, and generates channel-level adaptive weights. Through these weights, the framework can adjust the fine-grained fusion of high-frequency and low-frequency features according to the size of the difference during the fusion process. Specifically, the dynamic perception module is based on 、 Generate low-frequency and high-frequency feature distribution difference weights 、 The process is expressed by the formula:

[0092] ,

[0093] ,

[0094] in, It is a linear mapping. The value of the Sigmoid activation function is between 0 and 1.

[0095] In this embodiment, step S2 specifically includes the following steps:

[0096] S21, using hybrid expert module and Perform adaptive brightness normalization respectively and get 、 ; The router weight of the hybrid expert module is Apply Softmax and Top-K strategies to obtain;

[0097] S22, through the full connection layer and activation function 、 Process them separately to get the corresponding channel attention weights 、 ;

[0098] S23, based on right 、 Perform weighted fusion to obtain ;based on right 、 Perform weighted fusion to obtain ;

[0099] S24, will and After splicing, the low-frequency fusion weight is obtained through two fully connected layers and corresponding activation functions. ;

[0100] S25, based on right 、 Perform weighted fusion to obtain low-frequency fusion features .

[0101] In step S21, for the low-frequency components, this method uses an adaptive brightness normalization module to adjust the normalization intensity of the brightness channel using distribution difference weights, aiming to eliminate the interference caused by brightness changes while retaining robust structural information. The core idea is to decide whether and to what extent to perform brightness normalization based on the distribution difference of the low-frequency features of the input image. To achieve this goal, this method designs a normalization module based on the Mixture-of-Experts (MoE) structure, which draws on the "dynamic routing" and "expert selection" mechanisms in MoE. Specifically, the MoE layer receives two features from the low-frequency part. As input, the feature distribution weights generated adaptively are assigned to a group of brightness normalization experts. Each expert processes the input through a predefined brightness normalization function to achieve brightness modulation. The definition of the expert is as follows:

[0102] ,

[0103] in, and represents a learnable parameter that controls the scaling and translation of normalized features, and Represents the mean and standard deviation calculated independently on the spatial dimension of each channel and instance respectively. Based on the above definition, given two input low-frequency features and , the process of adaptive brightness normalization performed by the hybrid expert module can be defined as:

[0104] ,

[0105] ,

[0106] in, represents the number of experts, Represents low-frequency features The router weight is calculated by Softmax and Top-K strategy, that is, This routing mechanism is used to guide the intensity of brightness normalization in the framework and complete adaptive brightness normalization.

[0107] In steps S22 to S25, this method is based on the normalized low-frequency features and , a dynamic fusion module is designed. This module transforms the original low-frequency features and The corresponding normalized features are used as joint inputs, and a set of neural networks are used to learn and generate dynamic weights to achieve adaptive modulation for different brightness channels, and finally the modulated features are fused. Specifically, the module first connects the and Here, the global average of each channel is mapped through the full connection, and the output is limited to range, thus obtaining the weight . Subsequently, these weights are combined with the original low-frequency features Multiply and perform weighted combination to obtain the dynamically modulated channel characteristics , the formula of this module is defined as:

[0108] ,

[0109] ,

[0110] The design of this module effectively alleviates the problem of information loss during normalization. Splicing is performed to form preliminary fusion features, and the weights of the fusion features are learned through two fully connected layers and corresponding activation functions. Finally, these weights are used to perform weighted fusion on the two modulation features to generate the final output feature The process is defined as:

[0111] ,

[0112] ,

[0113] in, is the sigmoid activation function, is the Relu activation function.

[0114] In this embodiment, step S3 specifically includes the following steps:

[0115] S31, through the feature design variance filter 、 Generate the corresponding significance probability map and ;

[0116] S32, will 、 With their respective and Multiply to get the detail enhancement feature map and ;

[0117] S33, based on right 、 Perform weighted summation to obtain the final edge enhancement feature ;based on right 、 Perform weighted summation to obtain the final edge enhancement feature ;

[0118] S34, will and After splicing, the high-frequency fusion weight is obtained through two fully connected layers and corresponding activation functions ;

[0119] S35, based on right 、 Perform weighted fusion to obtain high-frequency fusion features .

[0120] Similar to the design of fusion rules for low-frequency components, high-frequency components primarily contain image details and edge information, reflecting texture, boundaries, and local variations. To meet the need for differentiated processing of high-frequency information in different task scenarios, this method proposes an adaptive control strategy based on distribution difference weights. This strategy consists of an adaptive edge enhancement module and an adaptive edge channel fusion module.

[0121] The adaptive edge enhancement module implements steps S31 to S33. For high-frequency components, distribution difference weights are used to adjust edge enhancement. The purpose of edge enhancement is to enhance the significant edge areas in the image to make the details of the image clearer and more prominent. In high-frequency components, edges usually represent key structures and texture information in the image, and play an important role in the visual perception and analysis of the image. Therefore, accurately adjusting the enhancement strength of the edge area can significantly improve the quality of the image. This method uses distribution difference weights to adjust the intensity of edge enhancement. When there is a large distribution difference between the two feature maps, it indicates that there is a large difference in texture, structure or details between the two. At this time, the edge enhancement strength should be increased to highlight these detail areas. This enhancement helps to better distinguish subtle structural differences and improve the visibility of details. Conversely, when the distribution difference between the two feature maps is small, the edge enhancement strength can be appropriately weakened to prevent over-enhancement and alleviate noise interference. Based on this, this method proposes an adaptive edge enhancement module. The input of this module is two high-frequency feature maps. and , extracting the edge information of both by calculating the saliency edge probability map. Specifically, this module designs a variance filter for the two features and generates a saliency edge probability map In order to calculate the significance probability map of two high-frequency features and , based on local variance and global variance, a saliency measure for each feature is generated. Local variance reflects the variation of the image within a local area, while global variance provides information about the variability of the image as a whole. Combining these two variances can effectively extract edge regions or significant details. The calculation formula for the saliency edge probability map is as follows:

[0122] ,

[0123] ,

[0124] in, is the size of the sliding window. The local variance is calculated through a very small receptive field, which makes it impossible to accurately distinguish between significant edge areas and noise detail images, and easily causes the loss of significant edges. To address this problem, this method further calculates a stable global variance for the two high-frequency features. Through learnable parameters ,The framework can automatically adjust the relationship between global and local variances according to the characteristics of the data, and improve the adaptability of the framework.

[0125] Next, the two high-frequency features are multiplied with the corresponding significant edge probability map to obtain the detail enhanced feature. The enhanced strength is adjusted using the weight distribution difference. It is multiplied with the enhanced feature and added to the original feature map to obtain the final edge enhanced feature of the two high-frequency features. and :

[0126] ,

[0127] ,

[0128] ,

[0129] ,

[0130] The adaptive edge channel fusion module implements steps S34 and S35. and The spliced ​​features are processed by two fully connected layers to learn the fusion weights. :

[0131] .

[0132] Then, the weight is used to perform weighted fusion on the two modulated features, and finally the fused high-frequency features are generated. :

[0133] .

[0134] In this embodiment, step S4 specifically includes the following steps:

[0135] S41, yes and Generate initial fusion features through global average pooling layer and fully connected layer ;

[0136] S42, yes The low-frequency channel weight is obtained through the low-frequency component fully connected layer and the high-frequency component fully connected layer and high-frequency channel weights ;

[0137] S43, will 、 After splicing, the Softmax function is used to normalize the splicing result, and then the normalized result is split into channels to obtain the low-frequency channel fusion weight and high-frequency channel fusion weights ;

[0138] S44, based on 、 right and Perform weighted fusion to obtain high and low frequency fusion features .

[0139] After obtaining the fused high-frequency features and low-frequency characteristics Finally, in order to highlight the truly useful information in the reconstruction process, this method introduces a high-frequency and low-frequency feature modulator (HF-LFFM). Given two feature maps Fusion Features The calculation process is defined as:

[0140] ,

[0141] in, Represents the weight parameter of the fully connected layer. To generate channel weights, this method uses two other fully connected layers, concatenates their outputs, and normalizes the concatenation using the Softmax function. The formula is as follows:

[0142] ,

[0143] in, and Represent the parameters of the low-frequency and high-frequency component fully connected layers, Indicates the splitting of features in the channel dimension. Finally, through adaptive fusion , and obtain the final high- and low-frequency fusion features , the process is defined as:

[0144] .

[0145] Existing unified image fusion methods are mainly divided into two categories: one uses a fixed optimization objective to uniformly optimize different task scenarios, and the other relies on task-related loss functions to achieve data-driven adaptive optimization. However, these two strategies often lead to task deviation or ignore the commonalities between multiple tasks, thereby restricting the performance of the fusion model in complex and diverse scenarios. Therefore, it is particularly important to design a dynamic universal loss function that can take into account the requirements of various tasks and scenarios. To this end, this method proposes a distribution difference-aware loss function, which dynamically adapts to multi-task scenarios by dynamically adjusting the weights of each loss. Its formula is as follows:

[0146] ,

[0147] Among them, the strength loss , detail loss , contrast loss are the brightness difference, high frequency component difference and contrast difference between the source image and the unified fusion image, respectively. Strength loss , detail loss and contrast loss Adaptive weights of . Strength loss It is used to ensure that the brightness of the fused image is consistent with the source image, reflecting the commonality of brightness information between tasks. The Sobel gradient operator is used to extract high-frequency information (such as edges) and compare the high-frequency components of the fused image with the source image to meet the personalized needs of different tasks in terms of detail preservation. By comparing the histogram distribution of the fused image, its global and local contrast is optimized, emphasizing the differentiated requirements of specific tasks on global visual quality.

[0148] In multi-task image fusion scenarios, there are both features that reflect the commonality of tasks (such as brightness consistency) and features that reflect the differences between tasks (such as detail preservation and contrast enhancement). The statistical distribution differences of different scenes within the same task further increase the difficulty of optimization. To this end, this method proposes a weight generation mechanism based on distribution differences, which dynamically generates adaptive weights based on the statistical characteristics of the source images. When the input is the absolute value of the mean difference and the absolute value of the standard deviation When , the formula for generating weight is:

[0149] ,

[0150] in, , represents the mean calculation function, Represents the standard deviation calculation function, through the multi-layer perceptron After obtaining a single output, it is subjected to Softmax operation and then channel-level separation. , the generated weights are non-negative and normalized.

[0151] It should be noted that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved. This embodiment is not limited here.

[0152] To apply the above-mentioned unified image fusion method, embodiments of the present invention further provide a unified image fusion system with adaptive distributed difference perception, which includes a model construction module, a model training module, and a model application module. The model construction module is used to construct a unified image fusion network, the model training module is used to train the unified image fusion network, and the model application module is used to use the trained unified image fusion network to fuse the source image pairs to be fused, thereby obtaining a unified fused image. Since the above-mentioned method has already described in detail the operations implemented by each module in the system, they will not be repeated here.

[0153] The embodiments described herein can be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0154] Computer programs for implementing the methods and systems of the present invention can be written in any combination of one or more programming languages ​​and stored in a computer-readable storage medium. These computer programs can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The computer program can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0155] Computer-readable storage media can be tangible media that can contain or store a computer program for use with an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. Computer-readable storage media can include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, computer-readable storage media can be a machine-readable signal medium. More specific examples of machine-readable storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, compact disc read-only memories (CD ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0156] In summary, the embodiments of the present invention provide a unified image fusion method and system with adaptive distribution difference perception. Aiming at the feature distribution differences between source images of different tasks, a unified image fusion network is designed. The network is designed with a distribution difference perception fuser to dynamically distinguish the distribution differences between low-frequency and high-frequency features in the image, thereby fine-grainedly adjusting the fusion strategy to adapt to the feature differences of different source images. On this basis, the present invention also proposes a weight-adaptive distribution difference perception loss function to balance the contribution of different loss terms to the fusion task, ensuring the robustness and generalization ability of the network in complex scenes. The trained unified image fusion network can not only effectively control the dominant deviations in different fusion tasks, but also integrate multiple fusion tasks such as multi-modality, multi-exposure and multi-focus into a unified framework, and can simultaneously retain the salient information of the target and the texture details of the visible light image, thereby achieving a more natural and balanced fusion effect and outputting high-quality super-resolution results.

[0157] The following experimental verification is carried out.

[0158] This example conducts experiments in four fusion task scenarios: multi-exposure image fusion (MEF), multi-focus image fusion (MFF), visible and infrared image fusion (IVF), and medical image fusion (MMF). The training dataset used is a mixture of multiple image fusion task datasets. Specifically, for the multi-exposure image fusion task, 589 image pairs were selected from the SCIE dataset, and the underexposed and overexposed images in this sequence were used as input. For the multi-focus image fusion task, 710 image pairs from the RealMFF dataset were used. For the infrared image fusion task, 1,000 image pairs were selected from the LLVIP dataset and 2,000 image pairs were selected from the M3FD dataset. Finally, for the medical image fusion task, 600 image pairs from the Harvard Medical Dataset were selected, covering the three categories of MRI-CT, MRI-SPECT, and MRI-PET.

[0159] To comprehensively evaluate the performance of the proposed method, this example was tested on multiple existing benchmark datasets, covering four categories: MEF, MFF, IVF, and MMF. Specifically, the MEF task was evaluated using the MEFB dataset; the MFF task followed the MFIFB benchmark setup; the IVF task was tested on the TNO and RoadScene datasets; and the MMF task was evaluated on the Harvard Medical Dataset for the MRI-CT, MRI-PET, and MRI-SPECT categories.

[0160] This embodiment selects six evaluation indicators from four categories to quantitatively measure the performance of the fusion result. These indicators are: AG (Average Gradient, which measures image clarity or edge sharpness), EN (Entropy, which reflects the texture complexity or information content of the image), Q abf (edge ​​Preservation Information Transfer Factor, which describes the degree of edge information retained in the fused image), SF (Spatial Frequency, which measures the texture and detail performance of the fused image), MS-SSIM (Multi-Scale Structural Similarity, which evaluates the structural similarity of two images), and FMI (Feature Mutual Information, which measures the correlation of two image features such as edges and textures by calculating the mutual information between them).

[0161] This example uses two experimental settings to comprehensively compare the proposed methods: one is the unified image fusion method, and the other is the task-specific fusion method. Specifically, for the unified fusion method, the baseline includes U2Fusion [1] 、SDNet [1] 、SwinFusion [1] 、DeFusion [1] MUFusion [2] 、DDBFusion [1] 、TC-MoA [3] ; In the comparison of specific task fusion methods, for the MFF task, MFF-GAN is selected [4] 、ZMFF [5] and DB-MFIF [6] method as the comparison baseline; for MEF tasks, TransMEF [7] , BHFMEF [8] and HSDS-MEF [9] As a comparison method; for the IVF task, the proposed method is compared with LRRNet [1] ,DDFM [1] and EMMA [1] Methods are compared; in the MMF task, MATR is selected

[10] , DDFM and EMMA methods are used as comparison methods.

[0162] The experiments were conducted on a high-performance server equipped with four NVIDIA GeForce RTX 3090 GPUs. For different image fusion tasks, all training samples were randomly cropped to a size of 224×224 during the preprocessing stage. This operation helps to standardize the input image and reduce the task deviation that may be caused by image size differences. During the training process, the model was trained for a total of 60 iterations to ensure sufficient training and convergence. To optimize the training process, the batch size was set to 2 and the optimizer used was AdamW. The initial learning rate was set to 8.0×10 -5 As the training progresses, the learning rate will gradually decrease to 1.0×10 -6 ,Through this strategy of dynamically adjusting the learning rate, it can effectively prevent overfitting and help the model adjust the weights more finely in the later stage.

[0163] The model incorporates the Transformer architecture to enhance image feature capture and fusion performance. A distribution difference-aware fusion module is embedded in the Transformer blocks of the encoder's fourth to eighth layers and the decoder's first to fifth layers. This module enables the model to dynamically capture the distribution characteristics of different source images and flexibly adjust the fusion strategy by deeply analyzing the differences between each pair of source images in the feature space.

[0164] The experiment comprehensively evaluates the fusion performance of the proposed method through six quantitative indicators. The specific results of this method and other comparison methods on the MEFB dataset are shown in Tables 1 and Figure 3 . Figure 3 Among them, (a) exposed image, (b) underexposed image, (c) TransMEF, (d) BHFMEF, (e) HSDS-MEF, (f) U2Fusion, (g) SDNet, (h) DeFusion, (i) SwinFusion, (g) MUFusion, (k) DDBFusion, (l) TC-MoA, and (m) this method.

[0165] Table 1: Quantitative comparison of multi-exposure image fusion tasks on the MEFB dataset

[0166]

[0167] As can be seen from Table 1, in comparison with the most advanced unified image fusion methods, this method (Ours) performs the best and shows excellent compatibility under multiple fusion tasks. Especially in terms of information theory indicators (such as EN, MS-SSIM), this method has obvious advantages, indicating that the fused image it generates can retain more source image information and is more in line with human visual perception characteristics. Although many image fusion methods designed specifically for a single task adopt complex task-specific strategies, this method has also achieved excellent results in competition with the multi-exposure fusion method HSDS-MEF for specific tasks. In addition, the good performance of this method in indicators such as MS-SSIM and FMI further demonstrates its advantages in preserving structure and gradient information. Figure 3 As shown in Figure 2, our model significantly outperforms other methods in terms of visual quality: for example, the outlines of the trees outside the window are clearer, and the background texture is more delicate. It is worth noting that our model can not only directly generate natural color images, but also achieve color reconstruction by post-processing grayscale images.

[0168] This method is systematically compared with three task-specific methods (MFF-GAN, ZMFF and DB-MFIF) and eight unified image fusion methods on the multi-focus image fusion task. On the MFIFB dataset, this method uses six quantitative indicators to conduct a detailed evaluation of the performance of each method. The evaluation results are shown in Table 2 and Figure 4 shown. Figure 4 In the figure, (a) near-focus image, (b) far-focus image, (c) MFF-GAN, (d) ZMFF, (e) DB-MFIF, (f) U2Fusion, (g) SDNet, (h) DeFusion, (i) SwinFusion, (g) MUFusion, (k) DDBFusion, (l) TC-MoA, and (m) this method.

[0169] Table 2: Quantitative comparison of multi-focus image fusion tasks on the MFIFB dataset

[0170]

[0171] Table 2 and Figure 4 The experimental results show that the proposed method is highly competitive in most indicators, especially in the unified image fusion task, achieving excellent performance. At the same time, the proposed method also performs well in comparison with the latest task-specific multi-focus image fusion method DB-MFIF, which shows that the proposed method has a clear advantage in preserving the unique details of the source images and presents higher fusion quality in human visual perception. Qualitative comparison results (see Figure 4 ) further verified this conclusion: the fused image generated by this method is superior to other methods in terms of texture and color consistency, while U2Fusion has color deviation in the far focus area, and DDBFusion is blurred in the near focus area. At the same time, these methods are difficult to effectively maintain font clarity and color information.

[0172] The proposed model and the comparison methods were comprehensively evaluated on the infrared and visible light image fusion task using six quantitative indicators on the TNO dataset. The experimental results are shown in Table 3 and Figure 5 shown. Figure 5 In, (a) visible light image, (b) infrared image, (c) LRRNet, (d) DDFM, (e) EMMA, (f) U2Fusion, (g) SDNet, (h) DeFusion, (i) SwinFusion, (g) MUFusion, (k) DDBFusion, (l) TC-MoA, and (m) this method.

[0173] Table 3: Quantitative comparison of infrared and visible light image fusion tasks on the TNO dataset

[0174]

[0175] The experimental results in Table 3 show that the fusion framework proposed in this method not only performs on par with or even surpasses the existing unified image fusion methods and task-specific methods, but also significantly outperforms other state-of-the-art methods. Figure 5 It is further demonstrated that in low-light environments, this method can simultaneously preserve the salient information of the target and the texture details of the visible light image, thereby achieving a more natural and balanced fusion effect; other methods often have problems such as the scene being too dark, salient information or texture details being lost, and it is difficult to achieve an ideal balance between details and saliency.

[0176] On the SPECT-MRI fusion task in the Harvard Medical Dataset, this method systematically evaluates the performance of eleven methods using six quantitative indicators. The experimental results are shown in Table 4 and Figure 6 shown. Figure 6 middle,

[0177] (a) SPECT image, (b) MRI image, (c) MATR, (d) DDFM, (e) EMMA, (f) U2Fusion, (g) SDNet, (h) DeFusion, (i) SwinFusion, (g) MUFusion, (k) DDBFusion, (l) TC-MoA, (m) this method.

[0178] Table 4: Quantitative comparison of medical image fusion tasks on the Harvard Medical Dataset

[0179]

[0180] The experimental results in Table 4 show that this method achieves the best performance in all indicators, fully demonstrating its excellent effectiveness in preserving edge details, maintaining source image information, and capturing key features. Figure 6 As shown in the figure, while U2Fusion and SDNet perform well in processing functional and structural information, they are prone to introducing noise; SwinFusion can fully utilize functional information but often ignores key structural details; MUFusion attempts to strike a balance between structure and function but still suffers from noticeable artifacts; and DDBFusion exhibits significant color deviation. In comparison, this method demonstrates superior visual quality, effectively preserving the complementary information of the source images and demonstrating superior fusion performance.

[0181] Experimental results show that the fusion effect of the proposed method in multi-task scenarios is better than that of existing methods, achieving a more natural and balanced fusion effect.

[0182] References:

[0183] 1.Liu J, Wu G, Liu Z, et al. Infrared and visible image fusion: Fromdata compatibility to task adaption[J]. IEEE Transactions on Pattern Analysisand Machine Intelligence, 2024;

[0184] 2.Cheng C, Xu T, Wu X J. MUFusion: A general unsupervised imagefusion network based on memory unit[J]. Information Fusion, 2023, 92: 80-92;

[0185] 3.Zhu P, Sun Y, Cao B, et al. Task-customized mixture of adapters forgeneral image fusion[C] / / Proceedings of the IEEE / CVF conference on computervision and pattern recognition. 2024: 7099-7108;

[0186] 4.Zhang H, Le Z, Shao Z, et al. MFF-GAN: An unsupervised generativeadversarial network with adaptive and gradient joint constraints for multi-focus image fusion[J]. Information Fusion, 2021, 66: 40-53;

[0187] 5.Hu X, Jiang J, Liu X, et al. ZMFF: Zero-shot multi-focus imagefusion[J]. Information Fusion, 2023, 92: 127-138;

[0188] 6.Zhang J, Liao Q, Ma H, et al. Exploit the best of both end-to-endand map-based methods for multi-focus image fusion[J]. IEEE Transactions onMultimedia, 2024, 26: 6411-6423;

[0189] 7.Qu L, Liu S, Wang M, et al.TransMEF: A transformer-based multi-exposure image fusion framework using self-supervised multi-task learning[C] / / Proceedings of the AAAI conference on artificial intelligence. 2022, 36(2): 2126-2134;

[0190] 8.Mu P, Du Z, Liu J, et al. Little strokes fell great oaks: Boostingthe hierarchical features for multi-exposure image fusion[C] / / Proceedings ofthe 31st ACM International Conference on Multimedia. 2023: 2985-2993;

[0191] 9.Wu G, Fu H, Liu J, et al. Hybrid-supervised dual-search: Leveragingautomatic learning for loss-free multi-exposure image fusion[C] / / Proceedingsof the AAAI conference on artificial intelligence. 2024, 38(6): 5985-5993;

[0192] 10.Tang W, He F, Liu Y, et al. MATR: Multimodal medical image fusion via multiscale adaptive transformer[J]. IEEE Transactions on ImageProcessing, 2022, 31: 5134-5149.

[0193] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be considered as equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. A unified image fusion method based on adaptive distribution difference perception, characterized in that: Including steps: Constructing a unified image fusion network; training the unified image fusion network; and fusing the source image pairs to be fused using the trained unified image fusion network to obtain a unified fused image. The unified image fusion network includes a cascaded first encoder and a first decoder, a cascaded second encoder and a second decoder, and also includes a distributed difference perception fusion device and an output layer; the first encoder and the second encoder have the same structure and both include a plurality of cascaded encoding layers; the first decoder and the second decoder have the same structure and both include a plurality of cascaded decoding layers; the distributed difference perception fusion device performs distributed difference perception fusion on the output of at least one encoding layer of the same level, and then splices the outputs with the outputs of the two encoding layers of the same level, and then inputs the outputs into the encoding layer or decoding layer of the next level; the distributed difference perception fusion device also performs distributed difference perception fusion on the output of at least one decoding layer of the same level, and then splices the outputs with the outputs of the two decoding layers of the same level, and then inputs the output layer or output layer of the next level; the output layer is used to obtain a unified fused image according to the outputs of the first decoder and the second decoder; The distribution difference perception fusion is based on the two features of the input and Perform distribution difference perception fusion to obtain high and low frequency fusion features , specifically including the steps: calculate and The difference weight of low-frequency feature distribution between and high-frequency feature distribution difference weight ; based on right 、 Low-frequency characteristics 、 After adaptive modulation of different brightness channels, weighted fusion is performed to obtain low-frequency fusion features. ; based on right 、 The high frequency part 、 After performing edge feature enhancement, weighted fusion is performed to obtain high-frequency fusion features. ; right and Perform weighted fusion to obtain high and low frequency fusion features .

2. The unified image fusion method based on adaptive distribution difference perception according to claim 1, characterized in that: The calculation and The difference weight of low-frequency feature distribution between and high-frequency feature distribution difference weight , specifically including: extract The low-frequency components and high-frequency components ,as well as The low-frequency components and high-frequency components ; Calculate low-frequency components and The mean and variance between them are used to calculate the low-frequency feature distribution difference weights. ; Calculate high-frequency components and The mean and variance between them are used to calculate the high-frequency feature distribution difference weights. .

3. The unified image fusion method based on adaptive distribution difference perception according to claim 2, characterized in that: The low-frequency feature distribution difference weight is calculated based on the mean and variance Specifically include: calculate 、 The mean of the low-frequency mean difference is obtained by taking the absolute value of the difference. ; calculate 、 The variance of the low-frequency variance is obtained by taking the absolute value of the difference ; Will and Splicing is performed to obtain the low-frequency feature distribution difference vector ; Will After passing through two layers of fully connected layers and corresponding activation functions, the low-frequency feature distribution difference weight is obtained ; Based on high-frequency components and The mean and variance between The same process calculation .

4. The unified image fusion method based on adaptive distribution difference perception according to claim 1, characterized in that: The based right 、 Low-frequency characteristics 、 After adaptive modulation of different brightness channels, weighted fusion is performed to obtain low-frequency fusion features. , specifically including: Using hybrid expert modules and Perform adaptive brightness normalization respectively and get 、 ; The router weight of the hybrid expert module is Apply Softmax and Top-K strategies to obtain; Through the fully connected layer and activation function 、 Process them separately to get the corresponding channel attention weights 、 ; based on right 、 Perform weighted fusion to obtain ;based on right 、 Perform weighted fusion to obtain ; Will and After splicing, the low-frequency fusion weight is obtained through two fully connected layers and corresponding activation functions. ; based on right 、 Perform weighted fusion to obtain low-frequency fusion features .

5. The unified image fusion method based on adaptive distribution difference perception according to claim 1, characterized in that: The based right 、 The high frequency part 、 After performing edge feature enhancement, weighted fusion is performed to obtain high-frequency fusion features. , specifically including: By designing the variance filter 、 Generate the corresponding significance probability map and ; Will 、 With their respective and Multiply to get the detail enhancement feature map and ; based on right 、 Perform weighted summation to obtain the final edge enhancement feature ;based on right 、 Perform weighted summation to obtain the final edge enhancement feature ; Will and After splicing, the high-frequency fusion weight is obtained through two fully connected layers and corresponding activation functions ; based on right 、 Perform weighted fusion to obtain high-frequency fusion features .

6. The unified image fusion method based on adaptive distribution difference perception according to claim 5, characterized in that: The pair and Perform weighted fusion to obtain high and low frequency fusion features , specifically including: right and Generate initial fusion features through global average pooling layer and fully connected layer ; right The low-frequency channel weight is obtained through the low-frequency component fully connected layer and the high-frequency component fully connected layer and high-frequency channel weights ; Will 、 After splicing, the Softmax function is used to normalize the splicing result, and then the normalized result is split into channels to obtain the low-frequency channel fusion weight and high-frequency channel fusion weights ; based on 、 right and Perform weighted fusion to obtain high and low frequency fusion features .

7. The unified image fusion method based on adaptive distribution difference perception according to any one of claims 1 to 6, characterized in that: In the process of training the unified image fusion network, the loss function used is designed as intensity loss , detail loss and contrast loss The weighted sum of the strength loss , detail loss , contrast loss are the brightness difference, high frequency component difference, and contrast difference between the source image and the unified fused image, respectively.

8. The unified image fusion method based on adaptive distribution difference perception according to claim 7, characterized in that: Strength loss , detail loss and contrast loss Corresponding weight , calculated by the following steps: Calculate the mean absolute difference between pairs of source images The absolute value of the difference from the standard deviation ; Will and After multi-layer perceptron and Softmax, channel-level separation is performed to obtain weights .

9. A unified image fusion system with adaptive distributed difference perception, used to implement the unified image fusion method with adaptive distributed difference perception according to any one of claims 1 to 8, characterized in that: It includes a model construction module, a model training module and a model application module. The model construction module is used to construct a unified image fusion network, and the model training module is used to fuse the unified image fusion network; the model application module is used to apply the trained unified image fusion network to fuse the source image pairs to obtain a unified fused image.

Images