Two-stage image fusion method

Abstract

This application relates to the field of image processing technology and provides a two-stage image fusion method, comprising: acquiring a visible light image and an infrared light image, and then inputting the visible light image and the infrared light image into a semantic fusion network; wherein, the semantic fusion network fuses the visible light image and the infrared light image to generate control conditions, iteratively processes the control conditions and noise feature sequences to obtain a fusion feature sequence, and performs patch sequence recombination on the fusion feature sequence to obtain a first fused image; the first fused image, the visible light image, and the infrared light image are input into a detail enhancement network, and the detail enhancement network performs texture detail enhancement on the first fused image based on the visible light image and the infrared light image to generate a target fused image. The method generates a first fused image with semantic information through the semantic fusion network, and the detail enhancement network enhances the details of this image. The generation process no longer relies on a single end-to-end training, thereby achieving high-quality fused image generation.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a two-stage image fusion method. Background Technology

[0002] Image fusion is an image processing technique that integrates multimodal image information to generate a more comprehensive and higher-quality composite image. In the fusion of visible light and infrared images, visible light images contain rich texture and structural information, but their imaging is limited by illumination and occlusion; infrared images can reflect the thermal radiation characteristics of a target, but their ability to capture texture details is weak and they are prone to introducing noise. The user's need is to generate a high-quality image that simultaneously preserves the thermal target and clear texture details through fusion processing.

[0003] To achieve effective fusion of multimodal images, an autoencoder-based fusion method can be used, combined with an end-to-end fusion network based on an attention mechanism and a generative adversarial network (GAN) approach. The GAN approach employs a generator to fuse multi-source information and output an image, while a discriminator evaluates the similarity of the probability distribution between the generated image and the source images, demonstrating advantages in generation quality.

[0004] However, generative adversarial networks (GANs) suffer from instability and pattern collapse during training, resulting in irrational distribution and insufficient detail preservation in the generated fused images. Summary of the Invention

[0005] This application provides a two-stage image fusion method to solve the problems of unreasonable image distribution and insufficient detail retention in fused images.

[0006] This application provides a two-stage image fusion method, including:

[0007] Acquire visible light and infrared light images;

[0008] The visible light image and the infrared light image are input into a semantic fusion network;

[0009] The semantic fusion network is configured to: fuse the visible light image and the infrared light image to generate control conditions; perform iterative processing on the control conditions and the noise feature sequence to obtain a fused feature sequence; and perform patch sequence recombination on the fused feature sequence to obtain a first fused image, wherein the first fused image includes semantic information.

[0010] The first fused image, the visible light image, and the infrared light image are input into the detail enhancement network;

[0011] The detail enhancement network is configured to perform texture detail enhancement on the first fused image based on the visible light image and the infrared light image to generate a target fused image.

[0012] As can be seen from the above technical solutions, the two-stage image fusion method provided in this application solves the technical problems of unstable training and easy mode collapse of generative adversarial networks in image fusion by constructing a two-stage architecture of semantic fusion network and detail enhancement network. Specifically, the semantic fusion network is responsible for generating a first fused image with complete semantic information, while the detail enhancement network enhances the texture details of the image based on the source image. This eliminates the reliance on a single end-to-end adversarial training process, thereby avoiding unreasonable distribution and missing details, and achieving stable and controllable high-quality fused image generation. Attached Figure Description

[0013] To more clearly illustrate the technical solution of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1 A schematic flowchart illustrating the two-stage image fusion method provided in this application embodiment;

[0015] Figure 2 This is a schematic diagram of the processing procedure within the semantic fusion network provided in the embodiments of this application;

[0016] Figure 3 This is a schematic diagram of the processing within the detail enhancement network provided in the embodiments of this application. Detailed Implementation

[0017] The embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following examples do not represent all embodiments consistent with this application.

[0018] Image fusion technology is used to combine image information acquired by different modal sensors in the same scene to generate a more comprehensive and higher quality fused image.

[0019] Taking the fusion of visible light images and infrared images as an example, visible light images have rich textures but are easily affected by lighting, while infrared images can capture thermal radiation but have more noise. The fusion of the two can simultaneously obtain target contour and detail information in complex environments, and has wide application value in fields such as security monitoring, remote sensing detection, and autonomous driving.

[0020] Early image fusion methods were mostly based on handcrafted feature design, such as wavelet transform and pyramid decomposition. Their fusion rules were fixed and difficult to adapt to diverse scenarios. With the development of deep learning, various neural network-based fusion methods have emerged, including autoencoder-based fusion methods, end-to-end fusion networks incorporating attention mechanisms, and generative adversarial networks.

[0021] For example, autoencoder-based fusion methods extract features from the source image through an encoder, fuse them in the feature space, and then reconstruct the image using a decoder. While their structure is simple, their feature representation capabilities are limited. End-to-end fusion networks incorporating attention mechanisms focus on important features through adaptive channel attention or spatial attention, which can improve fusion quality, but still struggle to achieve a balance between global semantics and local details.

[0022] For example, generative adversarial networks (GANs) have shown significant advantages in image fusion, consisting of a generator and a discriminator: the generator fuses visible light and infrared light image information to generate a fused image, and the discriminator evaluates the similarity of the probability distribution between the generated image and the source image. Through adversarial training, the generator can learn a fusion result that more closely approximates the real data distribution, outperforming other methods in visual quality and detail preservation.

[0023] However, the training process of generative adversarial networks (GANs) has inherent flaws. The game between the discriminator and the generator is difficult to balance, often resulting in training instability and artifacts or distortions in the generated images. In addition, the generator may fall into mode collapse, generating only a limited number of samples that cannot cover the real data distribution, resulting in insufficient diversity and detail retention in the fused images, making it difficult to meet the requirements of high-quality fused images in practical applications.

[0024] To address the aforementioned issues, this application provides a two-stage image fusion method. By decomposing the fusion task into two stages—semantic generation and detail enhancement—each stage focuses on its own objective, making the overall fusion process more stable and controllable. This solves the problems of unstable training and easy mode collapse in existing generative adversarial networks during image fusion, as well as the difficulty for a single network to simultaneously ensure semantic integrity and texture detail.

[0025] like Figure 1 As shown, the method includes the following steps S100-S300.

[0026] S100: Acquire visible light and infrared light images.

[0027] Visible light images can provide fine texture details when the light is sufficient, but they perform poorly at night or in occluded environments; infrared light images can penetrate interference such as smoke and low light and highlight thermal targets, but they lack texture information. The complementary characteristics of the two enable the fused image to have both clear structure and prominent thermal targets, making it suitable for all-weather scene understanding tasks.

[0028] Visible light images and infrared images can be acquired through camera devices or existing datasets. Visible light images are color or grayscale images, while infrared images are radiation images acquired by a thermal imager. The two types of images need to be registered to ensure pixel correspondence, providing aligned source images for subsequent fusion. For example, after registration, both the visible light and infrared images would have a resolution of 256×256 pixels.

[0029] S200: Input visible light images and infrared light images into the semantic fusion network.

[0030] In this embodiment, the final target fused image is obtained through a first stage and a second stage. In the first stage, a semantic fusion network is used for processing, and in the second stage, a detail enhancement network is used for processing. The two stages are executed in series, and the output of the first stage is input to the second stage. This ensures the overall consistency of the fused image and improves the realism of local details, making the image fusion process interpretable. The function of each stage is clear, and it is also easy to debug and optimize.

[0031] The following describes the first stage of the process. In the first stage, a semantic fusion network is used to generate a preliminary fusion result, namely the first fused image.

[0032] The semantic fusion network is a deep neural network that extracts semantic information from two images and generates a preliminary fusion result. The semantic fusion network includes a tile embedding layer, a location encoding module, a conditional generation module, multiple cascaded multimodal semantic fusion modules, and a tile reconstruction layer.

[0033] The input image is divided into patches by a patch embedding layer, which are then mapped to feature vectors to obtain a patch sequence. A position encoding module adds positional information to each patch sequence. A condition generation module concatenates the visible light patch sequence and the infrared light patch sequence and performs a linear transformation to generate control conditions. The multimodal semantic fusion module employs a Transformer architecture to iteratively process the control conditions and noisy feature sequences. Each module sequentially performs layer normalization, multi-head attention computation, and feedforward nonlinear transformation. The patch reconstruction layer linearly transforms each feature vector in the iteratively obtained fused feature sequence into pixel values ​​and concatenates them according to their positions in the original image to output the first fused image.

[0034] Specifically, the semantic fusion network is configured to: fuse visible light images and infrared light images to generate control conditions; perform iterative processing on the control conditions and noise feature sequences to obtain a fused feature sequence; and perform patch sequence recombination on the fused feature sequence to obtain the first fused image.

[0035] The control conditions are compressed representations of image information and can be multidimensional feature tensors. The semantic fusion network maps visible light and infrared light images into feature vectors and further combines them into control conditions. The control conditions carry the semantic content of the visible light and infrared light images, such as object categories, outlines, and spatial layouts.

[0036] Noise feature sequences are feature sequences extracted from random noise. Semantic fusion networks sample noisy images from a Gaussian distribution and divide the noisy images into multiple patches. Each patch is linearly mapped to obtain a feature vector, thus forming a noise feature sequence. The noise feature sequence provides randomness in the generation process, enabling the network to explore diverse fusion results.

[0037] The semantic fusion network performs iterative processing on the control conditions and the noisy feature sequence. That is, the current features are input into the transformation module in the semantic fusion network multiple times. Each time, the feature representation is updated, and the control conditions are used as conditional information to guide the noisy feature sequence to gradually evolve into fused features with semantic content. The number of iterations can be set according to the network depth, for example, 12 or 24 times.

[0038] In each iteration, the network uses an attention mechanism and a feedforward network to fuse control conditions and current features, so that the generated features gradually approximate the feature distribution of the real fused image, and finally obtains a fused feature sequence. The fused feature sequence is a set of feature vectors, each feature vector corresponding to a local region of the image. The fused feature sequence contains the texture structure of the visible light image and the thermal target information of the infrared light image, and is an intermediate representation for generating the first fused image.

[0039] The semantic fusion network then performs patch sequence recombination on the fused feature sequence. Patch sequence recombination is the process of rearranging the feature vectors in spatial order and mapping them back to pixel values.

[0040] Specifically, each feature vector is transformed into pixel values ​​of a patch through a linear layer, and then the patches are stitched together according to their positions in the original image to obtain the first fused image.

[0041] As can be seen from the above process, the first stage uses control conditions to guide the iterative evolution of noise feature sequences, which can extract high-level semantics from visible light images and infrared light images, thereby ensuring the correctness of the macroscopic structure of objects such as contours and categories in the fused image. The semantic fusion network adopts an iterative processing method to gradually refine features, which can avoid overfitting or single pattern caused by direct end-to-end mapping.

[0042] The first fused image is an image with complete semantic information, that is, the outlines of objects in the image are clear and the overall structure is reasonable, but the texture details may not be rich enough. Therefore, the next step S300, that is, the second stage of processing, is performed.

[0043] S300: Input the first fused image, visible light image, and infrared light image into the detail enhancement network.

[0044] In the second stage of processing, the first fused image, the visible light image, and the infrared light image are input into the detail enhancement network.

[0045] The detail enhancement network is a deep neural network used to improve image details, comprising a backbone feature encoder, a reference feature encoder, and a residual connection module. Both the backbone and reference feature encoders are convolutional layers. The backbone feature encoder extracts the baseline features of the first fused image, while the reference feature encoder extracts reference features from the visible light image and the infrared light image, respectively. The residual connection module contains multiple cascaded residual connection groups. Each residual connection group includes a channel attention unit and a spatial attention unit, used to modulate the baseline features based on the reference features to generate detail residuals. The detail residuals are added to the first fused image to obtain the target fused image.

[0046] The detail enhancement network is configured to perform texture detail enhancement on the first fused image based on visible light and infrared light images, generating the target fused image. In the second stage, using the source image as a reference, an attention mechanism is used to locate detailed regions and supplement high-frequency information. The detail enhancement network learns only in the residual domain, predicting only the details that need to be added without changing the overall structure of the first fused image. This reduces the learning difficulty, makes the network converge more easily, and generates details that more closely resemble the real texture of the visible light and infrared light images.

[0047] The detail enhancement network uses a reference-guided approach to enhance texture. The network first extracts the basic features of the first fused image, and simultaneously extracts reference features from the visible light image and the infrared light image. The basic features represent the overall structure of the first fused image, while the reference features carry high-frequency detail information from the visible light image and the infrared light image. The detail enhancement network fuses the details in the reference features into the basic features through an attention mechanism or residual learning to obtain the enhanced features. Finally, the enhanced features are added to the first fused image to obtain the target fused image.

[0048] The target fusion image retains the complete semantics of the first fusion image and supplements the texture details in the visible light image and infrared light image. The target fusion image can be used for downstream tasks such as target detection and scene understanding.

[0049] Figure 2This is a schematic diagram of the processing procedure within the semantic fusion network provided in the embodiments of this application.

[0050] In some embodiments, the semantic fusion network generates control conditions for extracting and encoding information from the visible light and infrared images, providing guidance for subsequent fusion processes, when performing the fusion of visible light and infrared images. Specifically, it is configured to perform the following steps S301-S305:

[0051] S301: Divide the visible light image into multiple first blocks; and divide the infrared light image into multiple second blocks.

[0052] The first patch is a non-overlapping local area in the visible light image. Each first patch covers a fixed-size pixel area. The entire visible light image is divided into multiple first patches by using a sliding window or grid partitioning method. These patches are arranged in spatial order and completely cover the original image.

[0053] The second image is divided in the same way as the first image. The second patch is a local area in the infrared image that corresponds to the spatial position of the first patch, and its size is the same as that of the first patch.

[0054] S302: Map each first patch to a first feature vector to obtain a visible light patch sequence; and map each second patch to a second feature vector to obtain an infrared light patch sequence.

[0055] Steps S302 and S303 are performed by the image embedding layer in the semantic fusion module.

[0056] The first image patch is a pixel matrix, which needs to be converted into a feature vector before it can be input into the neural network. The mapping is achieved through a linear layer. The linear layer performs a weighted combination of the pixel values ​​after the image patch is flattened and outputs a feature vector of fixed dimensions. Each first image patch corresponds to a first feature vector. All first feature vectors are arranged in the order of the images patch to form a visible light image patch sequence. The visible light image patch sequence contains the content information of each local region in the visible light image.

[0057] The mapping method for obtaining the infrared light patch sequence is the same as that for the first patch, and will not be repeated here. The infrared light patch sequence contains the content information of each local region in the infrared light image.

[0058] S303: Sample the noisy image from the Gaussian distribution, divide the noisy image into multiple noisy patches, and map each noisy patch to a feature vector to obtain a sequence of noisy patches.

[0059] The size of the noisy image is the same as that of the visible light image to ensure a consistent number of patches after partitioning. The noisy patch partitioning method is the same as that of the visible light and infrared images, and each noisy patch is mapped to a feature vector through a linear layer.

[0060] S304: Add position codes to the visible light patch sequence, infrared light patch sequence, and noise patch sequence respectively to obtain the visible light patch sequence with position codes, the infrared light patch sequence with position codes, and the noise patch sequence with position codes.

[0061] Step S304 is executed by the position encoding module in the semantic fusion module.

[0062] Positional encoding is location information appended to feature vectors, enabling the model to distinguish the spatial locations of different tiles within an image. Positional encoding can be sinusoidal positional encoding or learnable positional embeddings. Each feature vector is added to its corresponding positional encoding to form a new feature vector. After adding positional encoding, each feature vector in the tile sequence contains both tile content information and tile location information.

[0063] S305: Perform a linear transformation on the visible light patch sequence with position encoding and the infrared light patch sequence with position encoding to generate control conditions.

[0064] Step S305 is executed by the condition generation module in the semantic fusion module.

[0065] The concatenation operation merges two sequences along the feature dimension to form a longer feature sequence. The concatenated feature sequence contains all the information from both the visible light and infrared images. The linear transformation can be achieved through a linear layer, mapping the concatenated feature sequence to a specified dimension to obtain the control conditions. The control conditions carry the overall semantic and structural information of the visible light and infrared images, and are used to guide the evolution of the noisy feature sequence.

[0066] By generating control conditions with location information through tile partitioning, feature mapping, and position encoding, the semantic fusion network can capture the structure and content of visible light and infrared light images, laying the foundation for stable generation and avoiding pattern collapse.

[0067] In some embodiments, the semantic fusion network includes multiple cascaded multimodal semantic fusion modules. These modules are neural network layers built on the Transformer architecture, each with the same internal structure. Each module can transform input features, with the output of one module serving as the input of the next. The feature representation is gradually optimized through multiple transformations. The number of multimodal semantic fusion modules can be set according to the network depth, for example, 12 or 24.

[0068] The semantic fusion network executes to obtain the first fused image, specifically configured to perform the following steps S306-S308:

[0069] S306: Input the noise patch sequence with position encoding and control conditions into multiple cascaded multimodal semantic fusion modules to perform iterative processing and generate a fused feature sequence.

[0070] The position-coded sequence of noisy patches serves as the initial features, while control conditions act as conditional information, guiding feature evolution. Each multimodal semantic fusion module receives the feature sequence and control conditions output from the previous module, performs internal calculations, and outputs an updated feature sequence. During iterative processing, control conditions participate in the calculations in each module, continuously providing information from both visible and infrared images.

[0071] Each multimodal semantic fusion module sequentially performs layer normalization, multi-head attention computation, and feedforward nonlinear transformation.

[0072] Layer normalization standardizes the input features, adjusting them to zero mean and unit variance, making training more stable. Multi-head attention allows each element in the feature sequence to interact with other elements, calculating correlations and weighted fusion of information. Multi-head attention allows the model to focus on different features from multiple subspaces, capturing rich dependencies. Feedforward nonlinear transformation, consisting of two linear layers and a nonlinear activation function, further nonlinearly maps the attention output, enhancing the model's expressive power.

[0073] S307: Perform a linear transformation on each patch feature in the fused feature sequence to obtain the pixel value of the patch.

[0074] The fused feature sequence is a set of feature vectors obtained after multiple iterations. Each feature vector corresponds to a patch. The linear transformation maps the feature vectors to the flattened form of the patch pixel matrix. The dimension of the mapped feature vector is equal to the number of pixels in the patch. For example, for a 16×16 patch, the output dimension is 256.

[0075] S308: Reassemble the pixel values ​​of the patches according to their positions in the original image to obtain the first fused image.

[0076] Step S308 is performed by the tile reassembly layer in the semantic fusion module.

[0077] The reconstruction operation involves stitching together the pixel matrices of each patch in spatial order to restore the complete image. For example, the pixel values ​​of each patch are expanded row by row and then filled into the corresponding positions according to the patch's row and column indices.

[0078] The reconstructed image has the same size as the input image, with pixel values ​​continuously covering the entire image area. The first fused image contains semantic information extracted from the source image, with clear object outlines and a complete overall structure.

[0079] By iteratively processing control conditions and noise feature sequences through a series of multi-modal semantic fusion modules, the fusion features are gradually refined to generate a semantically complete first fused image, replacing the Generative Adversarial Network Generator (GAN) and avoiding the instability of adversarial training.

[0080] Figure 3 This is a schematic diagram of the processing within the detail enhancement network provided in the embodiments of this application.

[0081] In some embodiments, the detail enhancement network includes a backbone feature encoder, two reference feature encoders, and a residual connection module. The backbone feature encoder and the reference feature encoders are both convolutional layers used to extract features from the input image, and the residual connection module is used to perform multi-round modulation on the features to generate detail enhancement features.

[0082] The detail enhancement network performs texture detail enhancement on the first fused image based on the visible light image and the infrared light image to generate the target fused image. Specifically, it is configured to perform the following steps S401-S404:

[0083] S401: Extract the baseline features of the first fused image through the backbone feature encoder.

[0084] The first fused image contains complete semantic information but may lack texture details. The backbone feature encoder uses convolutional layers to downsample or extract features from the first fused image while maintaining the resolution, and outputs baseline features.

[0085] The baseline features preserve the overall structure and brightness distribution of the first fused image, and their channel numbers represent different feature types, such as edges, textures, or brightness.

[0086] S402: Encode the visible light image using a reference feature encoder to obtain visible light reference features, and encode the infrared light image using a reference feature encoder to obtain infrared light reference features.

[0087] The reference feature encoder uses convolutional layers to extract texture and structural information from the visible light image, outputting visible light reference features. These features retain high-frequency details of the original visible light image, providing a texture reference for subsequent enhancement. Simultaneously, the reference feature encoder encodes the infrared image, and the infrared reference features retain thermal target and contour information from the infrared image, providing a structural reference for the enhancement process.

[0088] The reference feature encoder has the same structure as the backbone feature encoder.

[0089] S403: Through the residual connection module, based on the visible light reference feature and the infrared light reference feature, multiple rounds of residual modulation are performed on the reference feature to obtain the detail enhancement feature.

[0090] Multi-round residual modulation involves modulating and updating the features multiple times. Each round of modulation generates a residual, which is then added to the current feature to form a new feature. The modulation process utilizes detailed information from the reference features to supplement the missing high-frequency components in the baseline features. The detail enhancement features represent the texture details that need to be added to the first fused image and exist in the form of feature maps.

[0091] In some embodiments, the residual connection module includes multiple cascaded residual connection groups, which can perform one round of modulation on the input features. The number of residual connection groups can be set according to the network depth, for example, 4, 6 or 8.

[0092] Each residual connection group includes a channel attention unit and a spatial attention unit. The channel attention unit is used to calculate the channel attention weights, and the spatial attention unit is used to calculate the spatial attention weights. These two units work together to modulate the features from different dimensions.

[0093] The residual connection module performs multi-round residual modulation on the reference features based on visible light and infrared light reference features to obtain detail enhancement features. Specifically, it is configured as follows: the channel attention unit calculates the channel attention weights based on the visible light and infrared light reference features; the spatial attention unit calculates the spatial attention weights based on the visible light and infrared light reference features; the residual connection group performs weighted modulation on the reference features of the current round using the channel attention weights and spatial attention weights to obtain intermediate features; the residual features are parsed from the intermediate features; and the residual features are added to the reference features of the current round to obtain the detail enhancement features.

[0094] For each residual connection group, its input includes a visible light reference feature F. vis Infrared reference feature F ir and the output feature F of the previous residual connection group i For the first residual connection group, F i These are the baseline features output by the backbone feature encoder. Within each residual connection group, channel attention calculation, spatial attention calculation, joint modulation, residual prediction, and residual connection operations are performed sequentially.

[0095] In some embodiments, the augmentation-details network further includes a first fully connected layer and a second fully connected layer;

[0096] The channel attention unit calculates channel attention weights based on visible light reference features and infrared light reference features. Specifically, it is configured to: concatenate the visible light reference features and infrared light reference features to obtain concatenated features; perform convolution processing on the concatenated features to obtain convolutional features; perform global average pooling on the convolutional features to obtain pooled features; and pass the pooled features sequentially through the first fully connected layer and the second fully connected layer, and then through the sigmoid activation function to obtain the channel attention weights.

[0097] Specifically, F vis and F ir By splicing along the channel dimension, the splicing feature [F] is obtained. vis F ir The concatenated features are fused using a 3×3 convolutional layer to obtain a fused feature map, i.e., the convolutional features. Then, global average pooling is performed on this feature map to obtain the feature vector z, i.e., the pooled features. Global average pooling compresses the feature map of each channel into a scalar, where the dimension of z is equal to the number of channels in the fused feature map.

[0098] The formula for calculating the global average pooling operation is as follows:

[0099] ;

[0100] Where H and W are the height and width of the feature map, respectively.

[0101] z is passed sequentially through the first fully connected layer W1 and the second fully connected layer W2. A ReLU activation function is applied after the first fully connected layer, and finally a sigmoid activation function is passed to obtain the channel attention weight A. c A c It is a one-dimensional vector, where each element corresponds to the weight of a feature channel, with values ​​ranging from 0 to 1. This weight is used to amplify reliable detail channels and suppress noise channels, thereby learning which channels contain reliable detail information from the reference features.

[0102] In some embodiments, the spatial attention unit performs the calculation of spatial attention weights based on visible light reference features and infrared light reference features, specifically configured to: calculate a first gradient magnitude of the visible light reference features and a second gradient magnitude of the infrared light reference features; obtain a visible light gradient map based on the first gradient magnitude and an infrared light gradient map based on the second gradient magnitude; stitch the visible light gradient map and the infrared light gradient map together to obtain a gradient stitched feature; perform convolution on the gradient stitched feature and process it through a sigmoid function to obtain the spatial attention weights.

[0103] Specifically, the spatial gradient magnitude G of the visible light reference feature is first calculated. vis That is, the first gradient magnitude and the spatial gradient magnitude G of the infrared reference feature.ir The second gradient magnitude is calculated by taking the modulus of the difference between the feature map in the horizontal and vertical directions, reflecting the intensity of edges and structural discontinuities in the image. The spatial gradient magnitude is the gradient map. After obtaining the visible light gradient map and the infrared light gradient map, the two are stitched together along the channel dimension to obtain the gradient stitching feature.

[0104] G vis With G ir By concatenating along the channel dimension, we obtain the gradient concatenation feature [G]. vis G ir The spliced ​​features are fused using a 3×3 convolutional layer, and then the spatial attention weights A are obtained through a sigmoid activation function. s A s It is a two-dimensional graph, where each spatial location corresponds to a weight value, ranging from 0 to 1. This weight is used to enhance locations in the reference image that have significant details, i.e., edges and textured areas receive higher weights, while smooth areas receive lower weights.

[0105] Get A c and A s Then, the residual connection group applies the current input feature F i Perform joint modulation, which first converts A c With F i Element-wise multiplication, i.e., scaling each channel, yields the modulated features of each channel. Then, A... s Multiply the result element-wise and scale it for each spatial location to obtain the intermediate feature F. i '.

[0106] The formula for calculating joint modulation is as follows:

[0107] ;

[0108] in, This indicates element-wise multiplication.

[0109] Through this dual modulation, the intermediate features retain information from reliable detail channels and rich detail regions, while suppressing noise and unimportant areas.

[0110] Intermediate feature F i 'This is then used to generate residual features, and the residual connection group will F i The residual feature R is obtained by passing the data through two 3×3 convolutional layers sequentially and applying the ReLU activation function after the first convolutional layer. i See the following formula for details:

[0111] ;

[0112] In the first 3×3 convolutional layer, the features are mapped to a higher dimension or kept dimensionless. ReLU introduces non-linearity. In the second 3×3 convolutional layer, the features are mapped back to the input features F. i The same number of channels is used to produce residuals, and the residual characteristic R is... i This represents the detail increments that need to be added to the baseline feature in the current round.

[0113] Finally, the residual link set will incorporate the residual characteristics R i With the current input features F i The summation yields the output characteristic F of the residual connection group. i+1 The addition operation adds elements one by one, supplementing the output features with detailed information extracted from the reference features while preserving the original content. The calculation formula is as follows:

[0114] ;

[0115] After multiple cascaded residual connection groups, the feature output by the last residual connection group is the detail enhancement feature. This detail enhancement feature is denoted as... , representing the texture detail features learned from the reference features.

[0116] S404: Add the detail enhancement features to the first fused image to obtain the target fused image.

[0117] Detail augmentation networks enhance detail features With the first fused image I F Add them together to obtain the target fused image F. refine The addition operation is performed pixel-by-pixel, preserving the semantic structure of the first fused image while supplementing texture details. The calculation formula is as follows:

[0118] ;

[0119] The target fusion image retains the overall structure while significantly enhancing texture details.

[0120] By jointly modulating the baseline features using channel attention and spatial attention units, the residuals are parsed from the intermediate features and added to the current features, achieving multi-round interpretable detail enhancement, improving texture richness without destroying the overall structure.

[0121] The detail enhancement features are the same size as the first fused image. After pixel-by-pixel addition, the overall structure of the first fused image remains unchanged, but the texture details are enhanced. The target fused image is the final output, which preserves the semantic integrity of the first fused image and contains the rich textures from both the visible light and infrared light images.

[0122] The baseline and reference features are extracted by the backbone encoder and the reference encoder, and then modulated by the residual connection module and added to the first fused image. This enhances texture details while maintaining the overall structure, thus solving the problem of insufficient detail retention.

[0123] In this embodiment, the semantic fusion network is trained using the flow-matching method, which enables the network to learn the mapping from noise distribution to real data distribution, thereby generating a high-quality first fused image during the inference stage.

[0124] The flow matching method includes: acquiring a training sample set, which includes sample visible light images, sample infrared images, and corresponding sample fused images; sampling sample noise images from a Gaussian distribution; constructing a continuous interpolation path between the sample noise images and the sample fused images, where the interpolation path is defined as a linear interpolation between the sample fused images and the sample noise images; sampling intermediate states on the interpolation path; and training a neural network to fit the velocity field in the intermediate states using the intermediate states as input and the true velocity field as the supervision label; wherein the true velocity field is defined as the difference between the sample fused images and the sample noise images.

[0125] Since there are few existing high-quality image fusion datasets, this application uses a data cleaning scheme to obtain a training sample set in order to solve this problem.

[0126] In some embodiments, the method further includes: generating candidate fused images through a fusion model; manually annotating the candidate fused images to obtain labeled data; training a binary classification model with the labeled data, wherein the binary classification model is a convolutional neural network, and the candidate fused images are used as input and the target labels are used as output for training; and selecting sample fused images from the candidate fused images through the binary classification model.

[0127] In this embodiment, the fusion model is a state-of-the-art (SOTA) model. The SOTA model can be, for example, a fusion network based on an autoencoder, a fusion method based on a generative adversarial network, or a fusion architecture based on a Transformer. The SOTA model has been pre-trained on a public dataset and has the ability to generate high-quality fused images.

[0128] Using sample visible light images and sample infrared light images as input, corresponding fused images are generated through the SOTA model. Since the SOTA model itself may have insufficient generalization or defects in specific scenarios, the quality of the generated candidate fused images is not ideal. Some images may have clear details and reasonable structure, while some images may have artifacts, blur or information loss. A large number of candidate fused images are generated through the SOTA model to provide sufficient material for subsequent screening.

[0129] The quality of each candidate fused image is judged by the annotators, such as texture sharpness, structural integrity, and noise level, and a label is assigned. The label is binary, such as good or bad, which indicates whether the image meets the criteria for a high-quality fused image.

[0130] The binary classification model is a convolutional neural network that is trained using candidate fused images as input and object annotations as output. The structure of the binary classification model can adopt common image classification networks, such as Residual Networks (ResNet), Visual Geometry Group Networks (VGG), or lightweight convolutional networks. The input layer receives candidate fused images, passes them through multiple convolutional layers, pooling layers, and fully connected layers, and outputs probability values ​​representing the probability that the image belongs to the high-quality category.

[0131] During training, the cross-entropy loss function can be used to optimize network parameters, enabling the model to accurately distinguish between high-quality and low-quality fused images. After training, the binary classification model has the ability to automatically evaluate the quality of fused images.

[0132] All candidate fused images are input into the trained binary classification model. The model outputs a quality probability value for each image. Based on a preset threshold, for example, a probability greater than 0.8, high-quality fused images are selected. These selected sample fused images serve as the training ground values ​​for the semantic fusion network. Together with their corresponding sample visible light images and sample infrared light images, they constitute the training sample set.

[0133] After obtaining the training sample set, the semantic fusion network is trained using a flow-matching-based method. The training process of flow-matching involves first constructing a continuous interpolation path between a simple prior distribution and the real data distribution, then sampling intermediate states along the path, and supervising the neural network to fit the real velocity field (vector field) in that state. This allows the model to gradually evolve noise into target data by solving ordinary differential equations during inference.

[0134] Define the actual path as: ;in, These are data labels, i.e., the merged image. It is random noise, where t is the current time step. .

[0135] The true velocity field (training labels) is: The model predicts: The training loss is: .

[0136] in, The speed at which the model predicts.

[0137] A flow matching method is used to construct a linear interpolation path between the sample fusion image and noise. The semantic fusion network is trained with the real velocity field as supervision to achieve stable and high-quality image generation and avoid the instability of GAN training.

[0138] In some embodiments, the detail enhancement network is trained using a loss function, which includes:

[0139] Structural similarity loss is used to measure the structural similarity between the generated target fused image and the visible light image and the infrared light image. The structural similarity loss is defined as the product of the first weight and the first structural similarity plus the product of the second weight and the second structural similarity. The first structural similarity is the structural similarity between the target fused image and the visible light image, and the second structural similarity is the structural similarity between the target fused image and the infrared light image.

[0140] The structural similarity is calculated as follows: For two images x and y, first calculate the mean of the local regions of the images. and ,variance and and covariance The formula for calculating structural similarity is as follows:

[0141] ;

[0142] in, , It is a stability constant used to avoid calculation instability when the denominator is close to zero.

[0143] The structural similarity value ranges from 0 to 1, with a larger value indicating greater similarity between the two images.

[0144] The structural similarity loss is calculated using the following formula:

[0145] ;

[0146] in, For the target image after fusion, Infrared light image, Visible light image, As the first weight, The first and second weights are preset hyperparameters used to balance the influence of visible light images and infrared light images on the fusion result.

[0147] By minimizing structural similarity loss, the target fused image is made consistent with the visible light image and the infrared light image in terms of brightness, contrast and structure.

[0148] Texture loss is used to measure the texture difference between the generated target fused image and the visible light image and infrared light image; where texture loss is defined as the difference between the gradient magnitude of the generated target fused image and the maximum gradient magnitude, and the maximum gradient magnitude is the maximum value between the gradient magnitude of the visible light image and the gradient magnitude of the infrared light image.

[0149] Texture loss is calculated using the following formula:

[0150] ;

[0151] in, The gradient magnitude of the target image is fused. The gradient magnitude of the visible light image. The gradient amplitude of the infrared light image. The height of the image. Let be the width of the image, and ||·||1 represent the L1 norm.

[0152] By minimizing texture loss, the gradient magnitude of the target fused image is made as close as possible to the one with the larger gradient in the source image, thereby preserving visible light texture and infrared light contours while avoiding the introduction of additional noise or blurring.

[0153] The total loss function is the sum of the structural similarity loss and the texture loss, as shown in the following formula:

[0154] ;

[0155] By employing structural similarity loss to maintain the structural consistency between the fused image and the source image, and combining it with texture loss to guide the network to retain more high-frequency details, the training of the detail enhancement network becomes more accurate, directly addressing the problem of insufficient detail retention.

[0156] Similar parts between the embodiments provided in this application can be referred to mutually. The specific implementation methods provided above are only a few examples under the overall concept of this application and do not constitute a limitation on the scope of protection of this application. For those skilled in the art, any other implementation methods extended from the solution of this application without creative effort shall fall within the scope of protection of this application.

Claims

1. A two-stage image fusion method, characterized in that, include: Acquire visible light and infrared light images; The visible light image and the infrared light image are input into a semantic fusion network; The semantic fusion network is configured to: fuse the visible light image and the infrared light image to generate control conditions; perform iterative processing on the control conditions and the noise feature sequence to obtain a fused feature sequence; and perform patch sequence recombination on the fused feature sequence to obtain a first fused image, wherein the first fused image includes semantic information. The semantic fusion network, in performing the fusion of the visible light image and the infrared light image, generates control conditions, which are specifically configured as follows: The visible light image is divided into multiple first blocks; and the infrared light image is divided into multiple second blocks; Map each of the first image patches to a first feature vector to obtain a visible light image patch sequence; Each of the second patch is mapped to a second feature vector to obtain an infrared light patch sequence; A noisy image is sampled from a Gaussian distribution, the noisy image is divided into multiple noisy patches, and each noisy patch is mapped to a third feature vector to obtain a sequence of noisy patches; Position codes are added to the visible light patch sequence, infrared light patch sequence, and noise patch sequence respectively to obtain a visible light patch sequence with position codes, an infrared light patch sequence with position codes, and a noise patch sequence with position codes; The control conditions are generated by performing a linear transformation after splicing the visible light patch sequence with position encoding and the infrared light patch sequence with position encoding. The first fused image, the visible light image, and the infrared light image are input into the detail enhancement network; The detail enhancement network is configured to perform texture detail enhancement on the first fused image based on the visible light image and the infrared light image to generate a target fused image.

2. The two-stage image fusion method according to claim 1, characterized in that, The semantic fusion network includes multiple cascaded multimodal semantic fusion modules; The semantic fusion network executes to obtain a first fused image, which is specifically configured as follows: The noise patch sequence with position encoding and the control conditions are input into multiple cascaded multimodal semantic fusion modules to perform iterative processing and generate a fused feature sequence. Each of the multimodal semantic fusion modules sequentially performs layer normalization, multi-head attention calculation and feedforward nonlinear transformation. Perform a linear transformation on each patch feature in the fused feature sequence to obtain the pixel value of the patch; The pixel values ​​of the image patches are recombined according to their positions in the original image to obtain the first fused image.

3. The two-stage image fusion method according to claim 1, characterized in that, The detail enhancement network includes a backbone feature encoder, a reference feature encoder, and a residual connection module, wherein the backbone feature encoder and the reference feature encoder are both convolutional layers; The detail enhancement network performs texture detail enhancement on the first fused image based on the visible light image and the infrared light image to generate a target fused image, specifically configured as follows: The baseline features of the first fused image are extracted using the backbone feature encoder; The visible light image is encoded by the reference feature encoder to obtain visible light reference features, and the infrared light image is encoded to obtain infrared light reference features; Through the residual connection module, based on the visible light reference feature and the infrared light reference feature, multiple rounds of residual modulation are performed on the reference feature to obtain the detail enhancement feature; The detail enhancement features are added to the first fused image to obtain the target fused image.

4. The two-stage image fusion method according to claim 3, characterized in that, The residual connection module includes multiple residual connection groups connected in series, and each residual connection group includes a channel attention unit and a spatial attention unit; The residual connection module performs multi-round residual modulation on the reference features based on the visible light reference features and the infrared light reference features to obtain detail enhancement features, specifically configured as follows: The channel attention unit calculates the channel attention weights based on the visible light reference features and the infrared light reference features; The spatial attention unit calculates the spatial attention weights based on the visible light reference features and the infrared light reference features. Through the residual connection group, the intermediate features are obtained by performing weighted modulation on the baseline features of the current round using the channel attention weights and the spatial attention weights. The residual features are extracted from the intermediate features; The residual feature is added to the baseline feature of the current round to obtain the detail enhancement feature.

5. The two-stage image fusion method according to claim 4, characterized in that, The augmentation-details network further includes a first fully connected layer and a second fully connected layer; The channel attention unit calculates the channel attention weights based on the visible light reference features and the infrared light reference features, and is specifically configured as follows: The visible light reference feature and the infrared light reference feature are spliced ​​together to obtain the spliced ​​feature; Perform convolution processing on the spliced ​​features to obtain convolutional features; Global average pooling is performed on the convolutional features to obtain pooled features; The pooling features are passed sequentially through the first fully connected layer and the second fully connected layer, and then through the sigmoid activation function to obtain the channel attention weights.

6. The two-stage image fusion method according to claim 4, characterized in that, The spatial attention unit performs the calculation of spatial attention weights based on the visible light reference features and the infrared light reference features, specifically configured as follows: Calculate the first gradient magnitude of the visible light reference feature, and calculate the second gradient magnitude of the infrared light reference feature; Based on the first gradient magnitude, a visible light gradient map is obtained, and based on the second gradient magnitude, an infrared light gradient map is obtained; By stitching together the visible light gradient map and the infrared light gradient map, a gradient stitching feature is obtained; The gradient concatenation features are convolved and processed by a sigmoid function to obtain spatial attention weights.

7. The two-stage image fusion method according to claim 1, characterized in that, The semantic fusion network is trained using a stream matching method, which includes: Obtain a training sample set, which includes sample visible light images, sample infrared images, and corresponding sample fused images; Sample noise images from a Gaussian distribution; A continuous interpolation path is constructed between the sample noise image and the sample fused image, and the interpolation path is defined as a linear interpolation between the sample fused image and the sample noise image; Sample intermediate states along the interpolation path; Using the intermediate state as input and the real velocity field as the supervision label, train a neural network to fit the velocity field under the intermediate state; The true velocity field is defined as the difference between the fused sample image and the sample noise image.

8. The two-stage image fusion method according to claim 7, characterized in that, The method further includes: Candidate fused images are generated using a fusion model; The candidate fused images are manually annotated to obtain annotation data; A binary classification model is trained using the labeled data. The binary classification model is a convolutional neural network, which is trained by taking the candidate fused image as input and the target label as output. The binary classification model is used to filter sample fused images from the candidate fused images.

9. The two-stage image fusion method according to claim 1, characterized in that, The detail enhancement network is trained using a loss function, which includes: Structural similarity loss is used to measure the structural similarity between the generated target fused image and the visible light image and the infrared light image; wherein, the structural similarity loss is defined as the product of a first weight and a first structural similarity plus the product of a second weight and a second structural similarity, where the first structural similarity is the structural similarity between the target fused image and the visible light image, and the second structural similarity is the structural similarity between the target fused image and the infrared light image. Texture loss is used to measure the texture difference between the generated target fused image and the visible light image and the infrared light image; wherein, the texture loss is defined as the difference between the gradient magnitude of the generated target fused image and the maximum gradient magnitude, and the maximum gradient magnitude is the maximum value between the gradient magnitude of the visible light image and the gradient magnitude of the infrared light image.

Images