Image fusion method and device and storage medium

Abstract

The embodiment of the invention provides an image fusion method and device and a storage medium, and the method comprises the steps: obtaining a target image which comprises an infrared image and a visible light image; performing IPT pre-extraction on the target image to obtain an infrared feature and a visible light feature; performing normalization and one-dimensional convolution on the infrared features and the visible light features to generate a first query vector, a first key vector, a first value vector, a second query vector, a second key vector and a second value vector; performing weighted fusion on the infrared features and the visible light features in combination with an attention collaboration mechanism to obtain a first fusion result and a second fusion result; performing one-dimensional convolution on the first fusion result and the second fusion result to obtain feature mapping, and performing residual connection on the feature mapping, the infrared features and the visible light features to obtain an output feature map; and performing cross-modal feature distribution alignment on the output feature map to obtain a fused image. According to the method, the definition and reality of the fused image are improved.

Description

A method, apparatus and storage medium for image fusion Technical Field

[0001] This invention relates to the field of computer technology, and more specifically to a method, apparatus, and storage medium for image fusion. Background Technology

[0002] Image fusion technology is fundamentally about merging image information from different sources or modalities to generate more comprehensive and higher-quality fused images, which is of great significance in fields such as remote sensing monitoring, security surveillance, and medical imaging. In the field of remote sensing, the fusion of multi-sensor data can achieve comprehensive capture of environmental information, supporting resource exploration and disaster assessment. In the field of surveillance, the fusion of visible light and infrared images can overcome monitoring blind spots under conditions such as low light, improving security reliability. In the medical field, the fusion of CT, MRI, ultrasound, and other modal data can help doctors accurately locate lesions, diagnose lesions, and optimize clinical diagnosis and treatment plans.

[0003] With the development of image acquisition technology, combining infrared and visible light images has become an important approach for image enhancement and information extraction. However, in practical applications, due to environmental interference, equipment limitations, and transmission effects, the images to be fused often suffer from degradation problems such as low illumination, overexposure, noise, and blurring. This not only reduces the quality of single-modal images but also poses challenges to fusion processing, making it difficult for the fusion results to meet the actual requirements of information integrity, visual clarity, and realism.

[0004] In existing studies on degraded image fusion, traditional methods focus on improving brightness and contrast, such as multi-scale transformation and wavelet transform. Although these methods can improve the visual effect to a certain extent, they cannot capture modal details and enhance the characteristics of degraded scene images in depth, resulting in blurred details and inaccurate information representation in the fused images, making them difficult to adapt to complex degraded scenes.

[0005] The development of deep learning technology has provided new ideas for degraded image fusion. The application of convolutional neural networks, self-attention mechanisms, and other technologies has the ability to learn and fuse complex features driven by data, with significant advantages. However, existing deep learning solutions still have shortcomings: they tend to focus on single-modal feature extraction, neglecting the complementarity of multimodal features (such as infrared and visible light) in degraded scenes, and cannot effectively suppress interference from irrelevant information; at the same time, there is the problem of insufficient alignment of multimodal feature spaces, which can easily lead to color shifts and uneven illumination in the fusion results, affecting visual effects and information reliability, and failing to fully leverage the advantages of multimodal data.

[0006] In summary, existing technologies still face three major bottlenecks in dealing with complex degradation scenarios: First, insufficient feature extraction, with inadequate mining of fine features such as edges and textures in visible light images, and limited robust extraction capability for key features such as infrared thermal targets; second, flawed fusion strategies, susceptible to interference from irrelevant information, and unable to fully utilize multimodal complementary information; and third, insufficient feature space alignment, leading to color shifts, uneven illumination, and a lack of visual clarity and realism in the fusion results. Therefore, leveraging the advantages of large-scale model semantic understanding and generation to construct a text-guided, efficient degradation image fusion scheme to address the poor fusion quality of existing technologies has become a critical technical challenge in this field, possessing significant research and application value. Summary of the Invention

[0007] The purpose of this invention is to provide an image fusion method, apparatus, and storage medium, which improves the clarity and realism of the fused image.

[0008] To achieve the above objectives, embodiments of the present invention provide an image fusion method, the method comprising: acquiring a target image, the target image including an infrared image and a visible light image; performing IPT pre-extraction on the target image to obtain infrared features and visible light features; performing normalization and one-dimensional convolution on the infrared features and visible light features respectively to generate a first query vector, a first key vector, a first value vector, and a second query vector, a second key vector, and a second value vector; performing weighted fusion of the infrared features and visible light features based on the first query vector, the first key vector, the first value vector, and the second query vector, the second key vector, and the second value vector, combined with an attention collaboration mechanism to obtain a first fusion result and a second fusion result; performing one-dimensional convolution on the first fusion result and the second fusion result to obtain a feature map; performing residual concatenation on the feature map, the infrared features, and the visible light features to obtain an output feature map; and performing cross-modal feature distribution alignment on the output feature map to obtain a fused image.

[0009] Optionally, the step of performing IPT pre-extraction on the target image to obtain infrared and visible light features includes: performing feature pre-extraction and enhancement on the visible light image through a reversible neural network to obtain visible light features; and performing feature pre-extraction and enhancement on the infrared image through a PoolFormer encoder to obtain infrared features.

[0010] Optionally, the step of pre-extracting and enhancing the visible light image using a reversible neural network to obtain visible light features includes: separating the visible light image into two 32-channel sub-features and a second sub-feature, wherein the first sub-feature is subjected to a basic feature transformation to obtain a transformed feature; the second sub-feature is modulated to obtain an exponential modulation coefficient; the second sub-feature is subjected to residual fusion based on the exponential modulation coefficient to obtain supplementary information; and the transformed feature and supplementary information are used for cyclic interaction to obtain visible light features.

[0011] Optionally, the step of pre-extracting and enhancing the infrared features of the infrared image using the PoolFormer encoder includes: performing dimensional transformation on the infrared image through an initial convolutional layer to obtain a transformed image; and performing pooling operations and channel dimension fusion on the transformed image to obtain infrared features.

[0012] Optionally, the step of weightedly fusing the infrared features and visible light features based on the first query vector, the first key vector, the first value vector, and the second query vector, the second key vector, and the second value vector, combined with an attention collaboration mechanism, to obtain a first fusion result and a second fusion result includes: calculating a first spatial attention weight based on the first key vector and the second query vector, performing normalization and splitting processing on the first spatial attention weight, and weighted fusion with the first value vector to obtain a first fusion result; calculating a second spatial attention weight based on the second key vector and the first query vector, performing normalization and splitting processing on the second spatial attention weight, and weighted fusion with the second value vector to obtain a second fusion result.

[0013] Optionally, the step of performing a one-dimensional convolution on the first fusion result and the second fusion result to obtain a feature map, and then performing a residual concatenation on the feature map, infrared features, and visible light features to obtain an output feature map, includes: performing a one-dimensional convolution operation on the first fusion result and the second fusion result respectively to obtain a first feature map and a second feature map; concatenating the first feature map and the second feature map along the channel dimension to obtain a fused feature map; performing a residual addition operation on the fused feature map with the infrared features and the visible light features respectively to obtain two residuals, and then performing channel-level fusion on the result of adding the two residuals to obtain an output feature map.

[0014] Optionally, if the number of feature channels does not match during the residual connection process, the infrared and visible light features are adapted by 1×1 convolution.

[0015] Optionally, the loss function for cross-modal feature distribution alignment includes a spatial consistency loss function and a multi-kernel maximum mean difference loss function; the spatial consistency loss function for:

[0016] The multi-core maximum mean difference loss function for:

[0017] in, For the expected operation, For pixels in the image, For image region, This represents the direction of image gradient calculation. For the image in direction gradient on, The source image, For the fused images, n is the number of visible light feature samples, m is the number of infrared feature samples, and H is the Hilbert space of the regeneration kernel. To map the input to the feature map of the reproducing kernel Hilbert space, Let be a visible light feature sample, and i be the index of the visible light feature sample. Let j be the index of the infrared feature sample.

[0018] On the other hand, this application also proposes an image fusion apparatus, which includes: an acquisition module for acquiring a target image, the target image including an infrared image and a visible light image; a first processing module for performing IPT pre-extraction on the target image to obtain infrared features and visible light features; a second processing module for normalizing and one-dimensional convolution on the infrared features and visible light features respectively to generate a first query vector, a first key vector, a first value vector, and a second query vector, a second key vector, and a second value vector; a third processing module for performing weighted fusion on the infrared features and visible light features based on the first query vector, the first key vector, the first value vector, and the second query vector, the second key vector, and the second value vector, combined with an attention coordination mechanism to obtain a first fusion result and a second fusion result; a fourth processing module for performing one-dimensional convolution on the first fusion result and the second fusion result to obtain a feature map, and performing residual connection on the feature map, the infrared features, and the visible light features to obtain an output feature map; and a fifth processing module for performing cross-modal feature distribution alignment on the output feature map to obtain a fused image.

[0019] On the other hand, this application also proposes a machine-readable storage medium storing instructions that, when executed by a processor, configure the processor to perform the image fusion method according to the above.

[0020] An image fusion method of the present invention includes: acquiring a target image, the target image including an infrared image and a visible light image; performing IPT pre-extraction on the target image to obtain infrared features and visible light features; performing normalization and one-dimensional convolution on the infrared features and visible light features respectively to generate a first query vector, a first key vector, a first value vector, and a second query vector, a second key vector, and a second value vector; performing weighted fusion on the infrared features and visible light features based on the first query vector, the first key vector, the first value vector, and the second query vector, the second key vector, and the second value vector, combined with an attention collaboration mechanism to obtain a first fusion result and a second fusion result; performing one-dimensional convolution on the first fusion result and the second fusion result to obtain a feature map; performing residual concatenation on the feature map, the infrared features, and the visible light features to obtain an output feature map; and performing cross-modal feature distribution alignment on the output feature map to obtain a fused image. This method performs preliminary processing on visible light and infrared images using modules based on reversible neural networks and PoolFormer, respectively, to extract fine and key features. Simultaneously, a frozen CLIP model is introduced, leveraging its semantic guidance to enhance the feature extraction process, allowing image features to be optimized based on contextual understanding. The extracted features are further processed, fusing information from different modalities to prepare for subsequent image reconstruction. The fusion layer employs multiple attention mechanisms to deeply fuse feature information from various modalities, enhancing image detail and semantic consistency. Dynamic integration of textual prompts and multimodal image features achieves refined image reconstruction, ultimately outputting a high-quality visible light and infrared fused image, improving the clarity and realism of the fused image.

[0021] Other features and advantages of the embodiments of the present invention will be described in detail in the following detailed description section. Attached Figure Description

[0022] The accompanying drawings are provided to further illustrate the embodiments of the present invention and constitute a part of the specification. They are used together with the following detailed description to explain the embodiments of the present invention, but do not constitute a limitation on the embodiments of the present invention. In the drawings: Figure 1 is a flowchart illustrating an image fusion method according to the present invention; Figure 2 is a schematic diagram of a degradation-oriented multimodal image fusion network structure according to the present invention; Figure 3 is a schematic diagram of a reversible neural network structure according to the present invention; Figure 4 is a schematic diagram of a network structure according to the present invention; Figure 5 is a schematic diagram of a PoolFormer network structure according to the present invention; Figure 6 is a schematic diagram of an unrelated attention network structure according to the present invention; Figure 7 is a schematic diagram of a convolutional layer of a reversible neural network according to the present invention; Figure 8 is a schematic diagram of a convolutional layer of a PoolFormer network according to the present invention; Figure 9 is a schematic diagram of a cross-attention module structure according to the present invention; Figure 10 is a schematic diagram of an unrelated attention module structure according to the present invention; Figure 11 is a schematic diagram of a spatial attention module structure according to the present invention; Figure 12 is a schematic diagram of an image fusion apparatus according to the present invention.

[0023] Explanation of reference numerals in the attached drawings: 100 - Image fusion apparatus; 200 - Acquisition module; 300 - First processing module; 400 - Second processing module; 500 - Third processing module; 600 - Fourth processing module; 700 - Fifth processing module. Detailed Implementation

[0024] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit the scope of the present invention.

[0025] It should be noted that the acquisition, transmission, storage, use, and processing of data in the technical solution of this application all comply with relevant laws and regulations. In the embodiments of this application, certain existing industry solutions such as software, components, and models may be mentioned. These should be considered exemplary, intended only to illustrate the feasibility of implementing the technical solution of this application, and do not imply that the applicant has already used or necessarily used such solutions.

[0026] Example 1 Figure 1 is a flowchart of an image fusion method of the present invention. As shown in Figure 1, the image fusion method of the present invention includes: step S101 is to acquire a target image, wherein the target image includes an infrared image and a visible light image.

[0027] In one specific implementation, visible light images are acquired using a charge-coupled device (CCD) or complementary metal-oxide-semiconductor (CMOS), while infrared images are acquired using an uncooled focal plane array.

[0028] Step S102 involves performing IPT pre-extraction on the target image to obtain infrared and visible light features. IPT stands for image processing tasks.

[0029] According to one specific implementation, the step of performing IPT pre-extraction on the target image to obtain infrared and visible light features includes: performing feature pre-extraction and enhancement on the visible light image through a reversible neural network to obtain visible light features; and performing feature pre-extraction and enhancement on the infrared image through a PoolFormer encoder to obtain infrared features.

[0030] Specifically, the step of pre-extracting and enhancing the visible light image using a reversible neural network to obtain visible light features includes: separating the visible light image into two 32-channel sub-features and a second sub-feature; performing a basic feature transformation on the first sub-feature to obtain a transformed feature; modulating the second sub-feature to obtain an exponential modulation coefficient; performing residual fusion on the second sub-feature based on the exponential modulation coefficient to obtain supplementary information; and using the transformed feature and supplementary information in a cyclical interaction to obtain visible light features.

[0031] The step of obtaining infrared features by pre-extracting and enhancing the infrared image through the PoolFormer encoder includes: performing dimensional transformation on the infrared image through an initial convolutional layer to obtain a transformed image; and performing pooling operation and channel dimension fusion on the transformed image to obtain infrared features.

[0032] As shown in Figure 2, this invention acquires infrared and visible light features through the IPT pre-extraction module. As shown in Figure 3, the IPT pre-extraction module consists of a reversible neural network, a PoolFormer encoder, and a text semantic encoder. In the feature pre-extraction stage, the INN module is specifically designed for visible light image degradation, utilizing its core characteristic of reversible transformation to ensure maximum preservation and reconstruction of the original information flow during feature enhancement. As shown in Figure 4, the 64-channel input 3D Z-feature is first separated into two 32-channel sub-features, Z1 and Z2: Z1 undergoes basic feature transformation through the feature transformation unit Φ module, while Z2 first passes through the modulation unit ρ module to generate exponential modulation coefficients. Multiplicative modulation controls the feature scaling intensity, and then the residual fusion unit η module supplements detailed information at the addition nodes. Simultaneously, the features output by the Φ module are fed back through the addition nodes, achieving cyclic interaction and continuously strengthening the feature expression. The Φ, ρ, and η modules are shown in Figure 3. Finally, the two updated sub-features Z1^' and Z2^' are concatenated to complete the fusion.

[0033] Specifically, the visible light input image first passes through an initial 3×3 convolutional layer, with 64 output channels, a stride of 1, and padding of 1, thus maintaining the image's spatial dimensions. Next, the image undergoes feature extraction via the Φ, ρ, η module, which consists of multiple convolutions and activation operations. After these modules, the spatial dimensions remain unchanged. Finally, the image passes through a 3×3 convolutional layer, restoring the number of channels to 3, while maintaining the spatial dimensions of the output image. By using an INN network with parameters as shown in Figure 7 to extract image features, the spatial dimensions of the input and output images are kept consistent, with only the number of channels being transformed. The final output image has the same height and width as the input image, but the number of channels changes from 3 to 64 and then back to 3, completing the image processing.

[0034] The infrared input image is passed through a 3×3 convolutional layer, resulting in an output with 64 channels, a stride of 1, and padding of 1, thus maintaining the image's spatial dimensions. Next, the image enters the PoolFormer module, whose parameters are shown in Figure 8. This module contains multiple layers, each extracting features through Pooling and PoolMLP layers. The Pooling layer performs two-dimensional average pooling to reduce the spatial dimensions while preserving important features. The PoolMLP layer further enhances feature representation through 1×1 convolutions. After these processes, the feature map still maintains 64 channels. Finally, the image passes through a 3×3 convolutional layer, restoring the number of channels to 3, and the output image's spatial dimensions remain unchanged. Through this entire process, the PoolFormer network maintains the input image's spatial dimensions while downsampling using pooling operations and enhancing features through PoolMLP operations, ultimately restoring the output image with the same spatial dimensions as the input image.

[0035] This invention freezes the weights of the text encoder trained in CLIP. Let {}e denote the frozen weights, and this process can be represented as:

[0036] in ∈RN×L represents the high-dimensional semantic feature vector corresponding to the text description. Semantically similar text descriptions are mapped to neighboring positions in the CLIP embedding space; that is, in the reduced-dimensional Euclidean feature space, the distance between them is small. Subsequently, a multilayer perceptron (MLP) is used to further explore the deep connection between semantic features and the image fusion task, mapping the text information into two sets of modulation coefficients:

[0037] in, and These represent two parallel branches in the MLP: the former generates a scaling factor, and the latter generates an offset factor to form semantic information. These two coefficients interact with the features of the fusion network, injecting semantic priors into the image fusion process to achieve high-quality, controllable fusion in degraded scenarios. This can be described as follows:

[0038] in, As a feature of the original fusion, This represents the Hadamard product, which ensures that the product is perfect through element-wise multiplication. Each element is scaled individually, allowing for more precise adjustments. This refers to the fusion features of modulated text with semantic meaning.

[0039] A PoolFormer module was designed for infrared images, utilizing a collaborative enhancement mechanism of pooling and Transformer to optimize infrared feature representation. As shown in Figure 5, the PoolFormer encoder consists of two layers of normalization, pooling, and PoolMLP. First, the initial convolutional layer performs dimensional transformation of image features. Then, through pooling operations, it actively mines the implicit spatial context information in the image, effectively filling in the details lost due to the infrared imaging principle. Subsequently, the PoolMLP module further enhances the expressive power of features through deep fusion of channel dimensions. Finally, a text-driven semantic guidance mechanism is adopted. Users provide a brief external text description through task presets to provide high-level semantic prior information to the network to guide the image fusion process. Faced with the degradation combination of visible light overexposure and infrared noise, users can input the overexposure degradation of the visible light image and the noise interference of the infrared image to be processed, ensuring natural colors and clear thermal targets after fusion. For infrared low contrast and visible light low light scenes, the text can be described as needing to improve the low contrast of the infrared image to highlight the thermal target, while increasing the brightness of the low-light areas of the visible light image. Such descriptions allow the network to clearly understand which degradation problem needs to be solved. This large model-guided design effectively enhances the features of both modalities, ensuring higher quality fusion input.

[0040] Step S103 involves normalizing and performing one-dimensional convolution on the infrared and visible light features respectively to generate a first query vector, a first key vector, a first value vector, a second query vector, a second key vector, and a second value vector.

[0041] In one specific implementation, layer normalization is used to address the different distribution characteristics of infrared / visible light features. Batch normalization can also be selected depending on the scenario. Layer normalization is suitable for small-batch training with variable feature lengths (such as infrared temporal features); batch normalization is suitable for large-batch training with fixed feature lengths (such as one-dimensional features after flattening a visible light image). The function of one-dimensional convolution is to map the normalized features to Q / K / V vectors of the same dimension.

[0042] Step S104 involves using the first query vector, the first key vector, the first value vector, the second query vector, the second key vector, and the second value vector, combined with an attention collaboration mechanism, to perform weighted fusion of the infrared features and the visible light features to obtain a first fusion result and a second fusion result.

[0043] According to a specific implementation, the step of weightedly fusing the infrared features and visible light features based on the first query vector, the first key vector, the first value vector, and the second query vector, the second key vector, and the second value vector, combined with an attention collaboration mechanism to obtain a first fusion result and a second fusion result, includes: calculating a first spatial attention weight based on the first key vector and the second query vector; normalizing and splitting the first spatial attention weight; and weightedly fusing it with the first value vector to obtain a first fusion result; calculating a second spatial attention weight based on the second key vector and the first query vector; normalizing and splitting the second spatial attention weight; and weightedly fusing it with the second value vector to obtain a second fusion result.

[0044] Specifically, this invention proposes an unrelated fusion layer branch, mainly including a cross-attention module, an unrelated attention module, and a spatial attention module. The cross-attention layer drives deep coupling between the visible light modality query and the infrared modality key / value by constructing a cross-modal bidirectional feature channel. The query generated from infrared image features is matched with the key in the visible light image features to calculate attention weights. These weights are then used to weighted sum the values ​​of the visible light image features to obtain fused information. Simultaneously, the visible light query, key, and value are processed in the same way as the infrared image features to obtain fused information. The unrelated fusion layer optimizes the fusion process by extracting common and specific features from the two modalities and integrating spatial information. As shown in Figure 6, a new activation function is embedded in the cross-attention mechanism to calculate cross-modal attention weights using a reverse softmax mechanism, suppressing regions with strong intermodal correlation, thereby enhancing features such as visible light color details and infrared temperature distribution. Spatial attention dynamically adjusts the weights of each location in the feature map based on the spatial information of the image. By weighting each spatial location in the feature map, the model can pay more attention to meaningful regions in the image and further optimize the expression of spatial features after cross-modal fusion.

[0045] The cross-attention module receives two feature maps x_A and x_B of the same size as input. The processing flow is shown in Figure 9: First, x_A is normalized using GroupNorm, then passed through a 1×1 convolution with three input channels and an output channel that is three times the number of input channels, a kernel size of 1×1, and a stride of 1 to generate the tensor qkv_A. Next, qkv_A is split into three parts: query_A, key_A, and value_A. Similarly, after applying GroupNorm normalization to x_B, another 1×1 convolution is passed to generate qkv_B, which is then split into query_B, key_B, and value_B. Cross-attention calculation is then performed: attention weights are calculated using query_B and key_A, normalized using softmax, and then weighted and fused with value_A. The result is passed through a 1×1 convolution with a kernel size of 1×1 and a stride of 1, adjusted, and then concatenated with the residual of the original x_A to obtain the output out_A. Symmetrically, attention weights are calculated using query_A and key_B, fused with value_B, adjusted via a 1×1 convolution, and concatenated with the residual of the original x_B to obtain out_B. The final output consists of two feature maps of the same size as the input.

[0046] The processing flow of the irrelevant attention module is the same as that of the cross-attention module, but the attention calculation method is different. Given two feature maps x_A and x_B as input (Figure 10), the processing steps are as follows: First, x_A is normalized using GroupNorm, then qkv_A is generated through a 1×1 convolution and split into three parts. Similarly, x_B is generated through a 1×1 convolution using GroupNorm and split into qkv_B. The key difference lies in the attention calculation stage: after calculating the similarity between query_B and key_A, re_softmax (1-softmax) is applied instead of standard softmax for normalization, and then weighted and fused with value_A; the result is adjusted through a 1×1 convolution and connected to the residual of the original x_A to obtain out_A. The other branch is processed symmetrically to obtain out_B. Finally, two feature maps of the same size as the input are output, but the focus is on irrelevant feature regions.

[0047] The spatial attention module receives a single feature map input. As shown in Figure 11, the processing flow is as follows: First, GroupNorm normalization is applied to the input. Then, a 1×1 convolution is used to generate a qkv tensor containing the query, key, and value, which is then split into three parts: query, key, and value. Next, self-attention is calculated: spatial attention weights are calculated using the query and key, normalized with softmax, and then weighted and fused with the value to capture global spatial context information. The result is adjusted by a 1×1 convolution and then residually connected to the original input. The final output is a feature map of the same size as the input, but with global spatial information incorporated at each location.

[0048] Step S105 involves performing a one-dimensional convolution on the first fusion result and the second fusion result to obtain a feature map, and then performing a residual connection on the feature map, infrared features, and visible light features to obtain an output feature map.

[0049] According to a specific implementation, the step of performing a one-dimensional convolution on the first fusion result and the second fusion result to obtain a feature map, and then performing residual concatenation on the feature map, infrared features, and visible light features to obtain an output feature map, includes: performing a one-dimensional convolution operation on the first fusion result and the second fusion result respectively to obtain a first feature map and a second feature map; concatenating the first feature map and the second feature map along the channel dimension to obtain a fused feature map; performing residual addition operations on the fused feature map with the infrared features and the visible light features respectively to obtain two residuals, and then performing channel-level fusion on the results of the two residual additions to obtain an output feature map.

[0050] During the residual connection process, if the number of feature channels does not match, the infrared and visible light features are adapted by 1×1 convolution.

[0051] Step S106 involves aligning the output feature map across modal feature distributions to obtain a fused image.

[0052] According to one specific implementation, the loss function for cross-modal feature distribution alignment includes a spatial consistency loss function and a multi-kernel maximum mean difference loss function; the spatial consistency loss function for:

[0053] The multi-core maximum mean difference loss function for:

[0054] in, For the expected operation, For pixels in the image, For image region, This represents the direction of image gradient calculation. For the image in direction gradient on, The source image, For the fused images, n is the number of visible light feature samples, m is the number of infrared feature samples, and H is the Hilbert space of the regeneration kernel. To map the input to the feature map of the reproducing kernel Hilbert space, Let be a visible light feature sample, and i be the index of the visible light feature sample. Let j be the index of the infrared feature sample.

[0055] The loss function utilizes MK-MMD to align cross-modal feature distributions, ensuring efficient integration of information from visible and infrared images and significantly improving the fusion effect of multimodal information. Furthermore, spatial consistency ensures the preservation of image structure and details from the pixel level to the feature level, effectively avoiding edge blurring and texture loss. Gradient loss and intensity maximization loss enhance details in dark areas, while intensity consistency loss restores the true brightness of overexposed areas. By dynamically adjusting the weights of each loss term, especially optimizing MK-MMD and spatial consistency loss, this loss function can flexibly handle different degradation scenarios, significantly improving the visual naturalness and detail preservation of the fused image, while effectively solving problems such as noise amplification and distortion in overexposed areas, thus significantly improving the visual naturalness and detail preservation of the fused image.

[0056] Spatial consistency loss, as the cornerstone of structure preservation, is a key guarantee against illumination inconsistencies, geometric distortions, and edge blurring, ensuring that the fused image retains the basic structural information of the source images. This application employs a gradient-based difference-based metric method:

[0057] in, Indicates the image in orientation gradient on, Represents the source image. Indicates a fused image. Represents the image region. The expectation operation is represented by this loss. This loss measures and quantifies the degree of structural inconsistency between the source image and the fused image by calculating the expected value of the sum of squared gradient differences in four directions. This guides the model to adjust its parameters so that the fused image maintains a high degree of structural consistency with the source image.

[0058] By employing a multi-kernel strategy, MK-MMD can capture distribution differences at different scales, achieving feature alignment between images of different modalities. The multi-kernel maximum mean difference loss can be defined as:

[0059] in, This represents a feature map that maps the input to the reproducing kernel Hilbert space (RKHS). and These represent samples with visible light and infrared characteristics, respectively. To characterize distribution differences at different scales, this application employs a multi-kernel combination strategy using Gaussian and Laplace kernels, enabling the kernel function to capture both local detail similarity and measure global distribution consistency. Based on the properties of regenerating kernels, the kernel function... The inner product of RKHS has the following relationship:

[0060] This indicates that the kernel function It can be used to compute features mapped The inner product of the later samples in the RKHS. Based on this, in actual calculations, the RKHS inner product can be expressed by a kernel function as:

[0061] in, The number of kernel functions; For the first The bandwidth parameters of each core. Utilizing Gaussian kernel and The combinatorial strategy of the Laplace kernel accurately measures and minimizes the differences in deep feature distributions between heterogeneous modes such as visible and infrared in the regenerating kernel Hilbert space (RKHS).

[0062] During training, the AdamW optimizer was used with an initial learning rate of 1e-4, a training batch size of 8, and the model was trained for 120 epochs. Validation and the best model were saved every 2 epochs. The experimental environment consisted of a server with a single NVIDIA GeForce RTX 3090 GPU, 24GB of VRAM, an Intel(R) Core(TM) i7-14700KF CPU, and PyTorch as the experimental framework. The testing environment was the same as the training phase. This was done in RoadScene, MSRS, LLVIP, and M... 3 This application was tested on four benchmarks in FD. Conventional methods often result in grayish sky colors and distorted building textures after fusion in the RoadScene dataset. This invention preserves the color levels of the visible light image (such as blue sky and road surface details) while enhancing vehicle outlines in dark areas with infrared enhancement, resulting in natural transitions between light and dark. In the MSRS dataset, used for nighttime scenes, this invention achieves a balance between the saliency of thermal targets and environmental texture, with sharp vehicle edges and distinct road surface layers. In the aerial scene (LLVIP) dataset, infrared-annotated vehicle and pedestrian thermal targets are easily confused with the visible light background (road surface, vegetation). By fusing irrelevant features, the thermal target annotations are clearly highlighted while maintaining the natural harmony between road surface texture and vegetation color tones. In the foggy degradation scene (M3FD), conventional methods often result in noticeable fog residue (grayish background) or harsh infrared background fusion, leading to color banding. This invention effectively suppresses degradation, resulting in sharp outlines of people and trees, and background colors that are closer to the real scene. In summary, this invention more efficiently balances the complementary information of infrared and visible light in multiple scenarios, and outperforms the comparative methods in terms of detail preservation, color reproduction, and target highlighting.

[0063] Embodiment 2 Figure 12 is a schematic diagram of an image fusion device according to the present invention. As shown in Figure 12, this application also proposes an image fusion device 100, which includes: an acquisition module 200 for acquiring a target image, the target image including an infrared image and a visible light image; a first processing module 300 for performing IPT pre-extraction on the target image to obtain infrared features and visible light features; and a second processing module 400 for normalizing and performing one-dimensional convolution on the infrared features and visible light features respectively to generate a first query vector, a first key vector, a first value vector, and a second query vector, a second key vector, and a second value vector. The third processing module 500 is used to perform weighted fusion of the infrared features and visible light features based on the first query vector, the first key vector, the first value vector, the second query vector, the second key vector, and the second value vector, combined with an attention collaboration mechanism, to obtain a first fusion result and a second fusion result; the fourth processing module 600 is used to perform one-dimensional convolution on the first fusion result and the second fusion result to obtain a feature map, and perform residual connection on the feature map, the infrared features, and the visible light features to obtain an output feature map; the fifth processing module 700 is used to perform cross-modal feature distribution alignment on the output feature map to obtain a fused image.

[0064] An image fusion method of the present invention includes: acquiring a target image, the target image including an infrared image and a visible light image; performing IPT pre-extraction on the target image to obtain infrared features and visible light features; performing normalization and one-dimensional convolution on the infrared features and visible light features respectively to generate a first query vector, a first key vector, a first value vector, and a second query vector, a second key vector, and a second value vector; performing weighted fusion on the infrared features and visible light features based on the first query vector, the first key vector, the first value vector, and the second query vector, the second key vector, and the second value vector, combined with an attention collaboration mechanism to obtain a first fusion result and a second fusion result; performing one-dimensional convolution on the first fusion result and the second fusion result to obtain a feature map; performing residual concatenation on the feature map, the infrared features, and the visible light features to obtain an output feature map; and performing cross-modal feature distribution alignment on the output feature map to obtain a fused image. This method performs preliminary processing on visible light and infrared images using modules based on reversible neural networks and PoolFormer, respectively, to extract fine and key features. Simultaneously, a frozen CLIP model is introduced, leveraging its semantic guidance to enhance the feature extraction process, allowing image features to be optimized based on contextual understanding. The extracted features are further processed, fusing information from different modalities to prepare for subsequent image reconstruction. The fusion layer employs multiple attention mechanisms to deeply fuse feature information from various modalities, enhancing image detail and semantic consistency. Dynamic integration of textual prompts and multimodal image features achieves refined image reconstruction, ultimately outputting a high-quality visible light and infrared fused image, improving the clarity and realism of the fused image.

[0065] The image fusion device 100 includes a processor and a memory. The aforementioned acquisition module 200, first processing module 300, second processing module 400, third processing module 500, fourth processing module 600 and fifth processing module 700 are all stored in the memory as program units. The processor executes the aforementioned program units stored in the memory to realize the corresponding functions.

[0066] The processor contains a kernel, which retrieves the corresponding program unit from memory. One or more kernels can be configured; adjusting kernel parameters enhances the clarity and realism of the fused image.

[0067] The memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0068] This invention provides a storage medium storing a program that, when executed by a processor, implements the image fusion method.

[0069] This invention provides a processor for running a program, wherein the program executes the image fusion method during runtime.

[0070] This invention provides a device including a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, it performs the following steps: acquiring a target image, the target image including an infrared image and a visible light image; performing IPT pre-extraction on the target image to obtain infrared features and visible light features; normalizing and performing one-dimensional convolution on the infrared features and visible light features respectively to generate a first query vector, a first key vector, a first value vector, and a second query vector, a second key vector, and a second value vector; weightedly fusing the infrared features and visible light features based on the first query vector, the first key vector, the first value vector, and the second query vector, the second key vector, and the second value vector, using an attention-based collaborative mechanism to obtain a first fusion result and a second fusion result; performing one-dimensional convolution on the first fusion result and the second fusion result to obtain a feature map; performing residual concatenation on the feature map, the infrared features, and the visible light features to obtain an output feature map; and aligning the output feature map across modal feature distributions to obtain a fused image. The device described herein can be a server, PC, PAD, mobile phone, etc.

[0071] This application also provides a computer program product, which, when executed on a data processing device, is suitable for executing an initialization program having the following method steps: acquiring a target image, the target image including an infrared image and a visible light image; performing IPT pre-extraction on the target image to obtain infrared features and visible light features; normalizing and performing one-dimensional convolution on the infrared features and visible light features respectively to generate a first query vector, a first key vector, a first value vector, and a second query vector, a second key vector, and a second value vector; performing weighted fusion of the infrared features and visible light features based on the first query vector, the first key vector, the first value vector, and the second query vector, the second key vector, and the second value vector, combined with an attention collaboration mechanism to obtain a first fusion result and a second fusion result; performing one-dimensional convolution on the first fusion result and the second fusion result to obtain a feature map; performing residual concatenation on the feature map, the infrared features, and the visible light features to obtain an output feature map; and performing cross-modal feature distribution alignment on the output feature map to obtain a fused image.

[0072] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0073] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in one or more flowchart illustrations and / or one or more block diagrams.

[0074] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in one or more flowcharts and / or one or more block diagrams.

[0075] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide steps for implementing the functions specified in one or more flowcharts and / or one or more block diagrams.

[0076] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0077] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0078] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0079] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0080] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. An image fusion method, characterized in that, The method includes: acquiring a target image, the target image including an infrared image and a visible light image; performing IPT pre-extraction on the target image to obtain infrared features and visible light features; normalizing and performing one-dimensional convolution on the infrared features and visible light features respectively to generate a first query vector, a first key vector, a first value vector, and a second query vector, a second key vector, and a second value vector; performing weighted fusion of the infrared features and visible light features based on the first query vector, the first key vector, the first value vector, and the second query vector, the second key vector, and the second value vector, combined with an attention collaboration mechanism to obtain a first fusion result and a second fusion result; performing one-dimensional convolution on the first fusion result and the second fusion result to obtain a feature map; performing residual concatenation on the feature map, the infrared features, and the visible light features to obtain an output feature map; and performing cross-modal feature distribution alignment on the output feature map to obtain a fused image.

2. The method according to claim 1, characterized in that, The step of performing IPT pre-extraction on the target image to obtain infrared and visible light features includes: pre-extracting and enhancing the visible light image using a reversible neural network to obtain visible light features; and pre-extracting and enhancing the infrared image using a PoolFormer encoder to obtain infrared features.

3. The method according to claim 2, characterized in that, The step of obtaining visible light features by pre-extracting and enhancing the visible light image using a reversible neural network includes: separating the visible light image into two 32-channel sub-features and a second sub-feature; performing a basic feature transformation on the first sub-feature to obtain a transformed feature; modulating the second sub-feature to obtain an exponential modulation coefficient; performing residual fusion on the second sub-feature based on the exponential modulation coefficient to obtain supplementary information; and using the transformed feature and supplementary information in a cyclic interaction to obtain visible light features.

4. The method according to claim 2, characterized in that, The step of obtaining infrared features by pre-extracting and enhancing the infrared image through the PoolFormer encoder includes: performing dimensional transformation on the infrared image through an initial convolutional layer to obtain a transformed image; and performing pooling operation and channel dimension fusion on the transformed image to obtain infrared features.

5. The method according to claim 1, characterized in that, The step of weightedly fusing the infrared features and visible light features based on the first query vector, first key vector, first value vector, and second query vector, second key vector, and second value vector, combined with an attention collaboration mechanism, to obtain a first fusion result and a second fusion result includes: calculating a first spatial attention weight based on the first key vector and the second query vector; normalizing and splitting the first spatial attention weight; and weightedly fusing it with the first value vector to obtain a first fusion result; calculating a second spatial attention weight based on the second key vector and the first query vector; normalizing and splitting the second spatial attention weight; and weightedly fusing it with the second value vector to obtain a second fusion result.

6. The method according to claim 1, characterized in that, The step of performing a one-dimensional convolution on the first and second fusion results to obtain a feature map, and then performing residual concatenation on the feature map, infrared features, and visible light features to obtain an output feature map, includes: performing a one-dimensional convolution operation on the first and second fusion results respectively to obtain a first feature map and a second feature map; concatenating the first and second feature maps along the channel dimension to obtain a fused feature map; performing residual addition operations on the fused feature map with the infrared features and visible light features respectively to obtain two residuals, and then performing channel-level fusion on the results of the two residual additions to obtain an output feature map.

7. The method according to claim 6, characterized in that, During the residual connection process, if the number of feature channels does not match, the infrared and visible light features are adapted by 1×1 convolution.

8. The method according to claim 1, characterized in that, The loss function for cross-modal feature distribution alignment includes a spatial consistency loss function and a multi-kernel maximum mean difference loss function; the spatial consistency loss function for: The multi-core maximum mean difference loss function for: in, For the expected operation, For pixels in the image, For image region, This represents the direction of image gradient calculation. For the image in direction gradient on, The source image, For the fused images, n is the number of visible light feature samples, m is the number of infrared feature samples, and H is the Hilbert space of the regeneration kernel. To map the input to the feature map of the reproducing kernel Hilbert space, Let be a visible light feature sample, and i be the index of the visible light feature sample. Let j be the index of the infrared feature sample.

9. An image fusion apparatus, characterized in that, The device includes: an acquisition module for acquiring a target image, the target image including an infrared image and a visible light image; a first processing module for performing IPT pre-extraction on the target image to obtain infrared features and visible light features; a second processing module for normalizing and performing one-dimensional convolution on the infrared features and visible light features respectively to generate a first query vector, a first key vector, a first value vector, and a second query vector, a second key vector, and a second value vector; a third processing module for performing weighted fusion of the infrared features and visible light features based on the first query vector, the first key vector, the first value vector, and the second query vector, the second key vector, and the second value vector, combined with an attention collaboration mechanism to obtain a first fusion result and a second fusion result; a fourth processing module for performing one-dimensional convolution on the first fusion result and the second fusion result to obtain a feature map, and performing residual concatenation on the feature map, the infrared features, and the visible light features to obtain an output feature map; and a fifth processing module for performing cross-modal feature distribution alignment on the output feature map to obtain a fused image.

10. A machine-readable storage medium storing instructions thereon, characterized in that, When executed by a processor, this instruction causes the processor to be configured to perform the image fusion method according to any one of claims 1 to 8.

Images