An Infrared-Visible Image Fusion Method Based on Pseudo-Twin Networks

Abstract

This invention belongs to the field of computer vision and image fusion technology, and discloses an infrared-visible light image fusion method based on a pseudo-twin network. Infrared and visible light images are fused in a pseudo-twin network fusion model. The pseudo-twin network fusion model is divided into an encoding stage, a feature fusion stage, and a decoding stage. The encoding stage includes a feature extraction module, a hybrid attention module, and two densely connected residual modules. The feature fusion stage includes three hybrid attention modules: a hybrid attention module for the visible light channel, a hybrid attention module for the infrared channel, and a hybrid attention module for fusing the two channels. The decoding stage includes four hierarchically arranged convolutional blocks. The infrared-visible light image fusion method of this invention can significantly improve the image fusion quality, balancing detail preservation and global information acquisition. It is highly efficient and scalable, and can be widely applied in computer vision fields such as autonomous driving, video surveillance, object detection, and person re-identification.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and image fusion technology, and relates to an infrared-visible light image fusion method based on pseudo-twin networks. Background Technology

[0002] Image fusion technology plays a crucial role in various aspects of social life, including intelligent surveillance, autonomous driving, and social security. Due to the inherent limitations of imaging sensors, single-modal sensors struggle to comprehensively reflect information in complex scenes. For example, infrared sensors effectively capture thermal radiation information in low-light and harsh environments, while visible light sensors offer high resolution and rich texture details but are susceptible to changes in lighting conditions. Therefore, Infrared and Visible Image Fusion (IVIF) combines the advantages of both, enhancing visual perception and image reliability to better meet the needs of human observation and machine vision. This technology has already been applied in scenarios such as autonomous driving, object detection, and person re-identification.

[0003] Image fusion algorithms can be broadly categorized into two types: traditional methods, including those based on multi-scale transformation, sparse representation, subspace analysis, and saliency detection; and deep learning methods, which have emerged in recent years. Traditional methods largely rely on mathematical models, resulting in computational complexity and limited fusion effectiveness. Deep learning methods, with their powerful feature extraction and representation capabilities, have gradually become a research hotspot in the field of image fusion.

[0004] Existing deep learning methods include autoencoders (AEs), convolutional neural networks (CNNs), generative adversarial networks (GANs), and Transformer-based models. While these methods have achieved good results in fusion, they still have shortcomings: (1) CNNs are better at capturing local information but have difficulty modeling global dependencies. (2) The computational complexity of Transformer-based models increases quadratically with the length of the input sequence, making it difficult to meet real-time requirements on embedded platforms; at the same time, their insufficient utilization of local detail features leads to blurred target edges in the fused image, thus affecting the accuracy of subsequent target detection algorithms.

[0005] Therefore, this paper proposes a target detection system based on FPGA, and simultaneously designs a hardware acceleration method with excellent reusability, low resource consumption, and high performance to solve the aforementioned problems. This enables the network to efficiently utilize FPGA resources and computing power during computation, which has significant practical implications in the field of image target detection. Summary of the Invention

[0006] To address the aforementioned issues, this invention proposes an infrared-visible light image fusion method based on a pseudo-Siamese network. This invention presents a pseudo-Siamese network fusion model that combines multi-scale feature extraction, attention mechanisms, and a novel fusion framework with residual structures to better balance the modeling capabilities of local and global features. This improves computational efficiency while maintaining fusion effectiveness, resulting in stronger feature extraction capabilities and higher efficiency. Specifically, a Siamese encoder equipped with an improved residual module and a hybrid attention mechanism is employed to acquire rich shallow features. Subsequently, a densely connected module is constructed for feature reuse. To further optimize feature fusion, infrared, visible light, and cross-modal specific hybrid attention modules are designed to capture long-range dependencies. Furthermore, a loss function is proposed that combines pixel loss, gradient loss, and structural similarity loss to achieve unsupervised learning.

[0007] The technical solution of the present invention:

[0008] An infrared-visible image fusion method based on pseudo-Twin networks is proposed, given a pair of input images, where the infrared image is represented as... Visible light images are represented as Infrared and visible light images are fused in a pseudo-twin network fusion model. Before being input into the pseudo-twin network fusion model, the visible light image is converted from an RGB color space image to a YCbCr color space image, and the Y channel, i.e. the luminance component, is extracted and fused with the infrared image. The fusion result is then spliced ​​with the Cb and Cr information in the original YCbCr color space.

[0009] The pseudo-twin network fusion model is divided into an encoding stage, a feature fusion stage, and a decoding stage. The encoding stage includes a feature extraction module, a hybrid attention module, and two densely connected residual modules. The feature fusion stage includes three hybrid attention modules: a hybrid attention module for the visible light channel, a hybrid attention module for the infrared channel, and a hybrid attention module that fuses the two channels. The decoding stage includes four hierarchically arranged convolutional blocks.

[0010] (1) Feature extraction module: Based on the input feature map, it includes two parallel branches. The output results of the two branches are added element by element to obtain the output features;

[0011] The first branch contains two cascaded residual units, each with four parallel convolutional paths. The first path uses a 1×1 convolutional kernel, the second path uses a 3×3 convolutional kernel, the third path uses two consecutive 3×3 convolutions, and the fourth path uses three consecutive 3×3 convolutions. All four parallel convolutional paths use a leaky linear rectified activation function. The outputs of the four parallel convolutional paths are concatenated along the channel dimension to form a comprehensive feature map integrating information from different scales. After the two residual units are cascaded, they are then passed through a 1×1 convolutional layer for channel fusion and linear combination.

[0012] The second branch performs channel compression and linear mapping on the input feature map through a 1×1 convolutional layer. As a residual connection path, it is added to the output of the first branch element-wise to obtain the output of the feature extraction module, realizing information reuse and efficient gradient propagation.

[0013] (2) Hybrid attention module: It is set after the feature extraction module. The feature maps obtained by the infrared channel and the visible light channel are stitched together along the channel direction and used as the input of the hybrid attention module. It is used to perform interactive modeling of infrared features and visible light features. The hybrid attention module consists of three parallel branches: channel attention block, spatial attention block and residual connection path.

[0014] The channel attention block consists of five layers. The first layer is a global average pooling layer, used to optimize spatial representation; the second layer is a 1×1 convolution, used to reduce channel dimension; the third and fourth layers are LReLU activation function and a 1×1 convolution, used to increase channel dimension; the last layer is sigmoid activation function, which restricts the output to between 0 and 1.

[0015] The spatial attention block consists of four layers, including a 3×3 convolutional layer, an LReLU activation layer, a 3×3 convolutional layer, and a sigmoid function layer, which are used to mine feature information in the spatial dimension.

[0016] The residual connection path is used to establish jump connections between the input and output of the hybrid attention module, enabling the fusion of static and dynamic attention information; finally, the results of the channel attention block, spatial attention block, and residual connection path are added element by element for fusion.

[0017] (3) Densely connected residual module: After the hybrid attention module, a two-layer series densely connected residual block structure is adopted; the densely connected residual block includes the main branch and the residual connection branch;

[0018] In the main branch, multiple convolutional layers and activation layers are sequentially arranged. The first convolutional layer uses a 3×3 convolutional kernel followed by an LReLU activation function to extract local features. The second convolutional layer also uses a 3×3 convolutional kernel followed by an LReLU activation function to further extract deep features. The output of the second convolutional layer is fused and compressed via a 1×1 convolutional layer to reduce computational complexity and enhance information aggregation capabilities. Cross-layer connections are introduced between the convolutional layers in the main branch: while the output of the first layer is passed to the next layer, it is also directly added to the output of the subsequent layer through cross-layer connections to achieve the fusion of multi-level features.

[0019] The residual connection branch uses a 1×1 convolutional layer to adjust the channels of the features output from the hybrid attention module, making them consistent with the output of the main branch in dimension. Finally, the output of the main branch and the output of the residual branch are fused together by element-wise addition to obtain the output of the densely connected residual module, realizing feature reuse and efficient gradient propagation.

[0020] (4) Feature fusion module: Located between the encoding and decoding stages, it takes the output of the densely connected residual module as input and is used to realize the multimodal fusion of infrared and visible light features. The feature fusion module is placed after the densely connected residual module and is divided into three parallel hybrid attention modules.

[0021] Infrared and visible light features are respectively input to the hybrid attention module of the infrared channel and the hybrid attention module of the visible light channel; the infrared and visible light features are spliced ​​in the channel dimension and further input to the hybrid attention module that fuses the two channels to achieve deep interactive fusion between modalities;

[0022] In the above fusion process, the outputs of the three parallel hybrid attention modules are then concatenated along the channel dimension. The concatenated feature map is then fed into the decoder as input to reconstruct the output image.

[0023] (5) Decoding and Reconstruction Module: This module is used to aggregate and reconstruct the image after fusion by the feature fusion module to output the final image. The decoding and reconstruction module adopts a hierarchical decoder network structure, consisting of four sequentially arranged convolutional blocks: the first three convolutional blocks are composed of convolution operations, batch normalization layers and LReLU activation functions, which are used to gradually extract and restore the detailed features of the image; the last convolutional block is composed of convolution operations, batch normalization layers and Tanh activation functions, which are used to limit the output value to the range of [−1,1].

[0024] (6) Composite Loss Function: Design a composite loss function consisting of pixel loss, texture loss, and structural similarity loss; the overall form of the composite loss function is:

[0025]

[0026] in, , and These represent pixel loss, texture loss, and SSIM loss, respectively. and It is a weighting factor used to balance the contributions of each entity.

[0027] In the composite loss function,

[0028] Pixel loss The expression is: ,in It is the Frobenius norm. As a balance factor, To merge images, Visible light image;

[0029] Texture loss The expression is ,in It is the L2 norm. For gradient calculation, Infrared image;

[0030] Structural similarity loss The expression is ,in, This represents the structural similarity index.

[0031] The beneficial effects of this invention are:

[0032] 1. Strong multi-scale feature extraction capability: By combining the pseudo-twin encoder with the improved Inception module, it can extract local and global feature information simultaneously, fully preserving the complementarity of infrared and visible light images.

[0033] 2. Full cross-modal information interaction: By utilizing the channel-space hybrid attention mechanism, the feature synergy of infrared and visible light images is effectively enhanced, avoiding information redundancy and loss of key features.

[0034] 3. Superior fusion effect: The proposed composite loss function takes into account pixel consistency, texture details and structural similarity, so that the fusion result performs well in visual perception and subsequent target detection, recognition and other tasks.

[0035] The infrared and visible light image fusion method proposed in this invention can significantly improve the image fusion quality, while taking into account both detail preservation and global information acquisition. It is highly efficient and scalable, and can be widely applied in computer vision-related fields such as autonomous driving, video surveillance, target detection, and person re-identification. Attached Figure Description

[0036] Figure 1 This is a flowchart of the image fusion model workflow;

[0037] Figure 2 This is a diagram of a pseudo-twin network fusion model;

[0038] Figure 3 This is a schematic diagram of the feature extraction module structure;

[0039] Figure 4 This is a schematic diagram of the hybrid attention module structure;

[0040] Figure 5 This is a schematic diagram of the densely connected residual module structure. Detailed Implementation

[0041] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings and technical solutions.

[0042] The workflow of an infrared-visible image fusion method based on pseudo-twin networks is as follows: Figure 1 As shown, given a pair of input images, where the infrared image is represented as Visible light images are represented as Infrared and visible light images are fused in a pseudo-twin network fusion model. Before being input into the pseudo-twin network fusion model, the visible light image is converted from an RGB color space image to a YCbCr color space image, and the Y channel, i.e. the luminance component, is extracted and fused with the infrared image. The fusion result is then spliced ​​with the Cb and Cr information in the original YCbCr color space.

[0043] The pseudo-twin network fusion model is divided into an encoding stage, a feature fusion stage, and a decoding stage. The encoding stage includes a feature extraction module, a hybrid attention module, and two densely connected residual modules. The feature fusion stage includes three hybrid attention modules: a hybrid attention module for the visible light channel, a hybrid attention module for the infrared channel, and a hybrid attention module that fuses the two channels. The decoding stage includes four hierarchically arranged convolutional blocks.

[0044] The specific implementations of each part of the pseudo-twin network fusion model are as follows:

[0045] (1) Feature extraction module: Based on the input feature map, it includes two parallel branches. The output results of the two branches are added element by element to obtain the output features;

[0046] The first branch contains two cascaded residual units, each with four parallel convolutional paths. The first path uses a 1×1 convolutional kernel, the second path uses a 3×3 convolutional kernel, the third path uses two consecutive 3×3 convolutions, and the fourth path uses three consecutive 3×3 convolutions. All four parallel convolutional paths use a leaky linear rectified activation function. The outputs of the four parallel convolutional paths are concatenated along the channel dimension to form a comprehensive feature map integrating information from different scales. After the two residual units are cascaded, they are then passed through a 1×1 convolutional layer for channel fusion and linear combination.

[0047] The second branch performs channel compression and linear mapping on the input feature map through a 1×1 convolutional layer. As a residual connection path, it is added to the output of the first branch element-wise to obtain the output of the feature extraction module, realizing information reuse and efficient gradient propagation.

[0048] The residual element can be represented as:

[0049]

[0050]

[0051]

[0052]

[0053]

[0054] The feature extraction module can be represented as:

[0055]

[0056] (2) Hybrid attention module:

[0057] Located after the feature extraction module, the feature maps obtained from the infrared and visible light channels are concatenated along the channel direction and used as input to the hybrid attention module for interactive modeling of infrared and visible light features. The hybrid attention module achieves effective cross-modal fusion by dynamically allocating weights to highlight effective features and suppress redundant information. The hybrid attention module consists of three parallel branches: a channel attention block, a spatial attention block, and a residual connection path.

[0058] Specifically:

[0059] The channel attention block consists of five layers. The first layer is a global average pooling layer, used to optimize spatial representation; the second layer is a 1×1 convolution, used to reduce channel dimension; the third and fourth layers are LReLU activation function and a 1×1 convolution, used to increase channel dimension; the last layer is sigmoid activation function, which restricts the output to between 0 and 1.

[0060] The Spatial Attention Block (SAB) consists of four layers: a 3×3 convolutional layer, an LReLU activation layer, another 3×3 convolutional layer, and a sigmoid function layer. These layers are used to extract feature information in the spatial dimension. Specifically, SAB can be described as follows: .

[0061] The residual connection path is used to establish skip connections between the input and output of the hybrid attention module, enabling the fusion of static and dynamic attention information. Finally, the results of the channel attention block, spatial attention block, and residual connection path are added element by element and fused to improve training stability and feature preservation.

[0062] (3) Densely connected residual module: After the hybrid attention module, a two-layer series densely connected residual block structure is adopted; the densely connected residual block includes the main branch and the residual connection branch;

[0063] In the main branch, multiple convolutional layers and activation layers are sequentially arranged. The first convolutional layer uses a 3×3 convolutional kernel followed by an LReLU activation function to extract local features. The second convolutional layer also uses a 3×3 convolutional kernel followed by an LReLU activation function to further extract deep features. The output of the second convolutional layer is fused and compressed via a 1×1 convolutional layer to reduce computational complexity and enhance information aggregation capabilities. Cross-layer connections are introduced between the convolutional layers in the main branch: while the output of the first layer is passed to the next layer, it is also directly added to the output of the subsequent layer through cross-layer connections to achieve the fusion of multi-level features.

[0064] The residual connection branch uses a 1×1 convolutional layer to adjust the channels of the features output from the hybrid attention module, making them consistent with the output of the main branch in dimension. Finally, the output of the main branch and the output of the residual branch are fused together by element-wise addition to obtain the output of the densely connected residual module, realizing feature reuse and efficient gradient propagation.

[0065] (4) Feature fusion module: Located between the encoding and decoding stages, it takes the output of the densely connected residual module as input and is used to realize the multimodal fusion of infrared and visible light features. The feature fusion module is placed after the densely connected residual module and is divided into three parallel hybrid attention modules.

[0066] Infrared and visible light features are respectively input to the hybrid attention module of the infrared channel and the hybrid attention module of the visible light channel; the infrared and visible light features are spliced ​​in the channel dimension and further input to the hybrid attention module that fuses the two channels to achieve deep interactive fusion between modalities;

[0067] In the above fusion process, the outputs of the three parallel hybrid attention modules are then concatenated along the channel dimension. The concatenated feature map is then fed into the decoder as input to reconstruct the output image.

[0068] (5) Decoding and Reconstruction Module: This module is used to aggregate and reconstruct the image fused by the feature fusion module to output the final image. The decoding and reconstruction module adopts a hierarchical decoder network structure, consisting of four sequentially arranged convolutional blocks: the first three convolutional blocks are composed of convolution operations, batch normalization (BN) layers and LReLU activation functions, which are used to gradually extract and restore the detailed features of the image; the last convolutional block is composed of convolution operations, batch normalization layers and Tanh activation functions, which are used to limit the output value to the range of [−1,1].

[0069] Its mathematical expression is as follows:

[0070]

[0071]

[0072]

[0073]

[0074] Here, Tanh represents the hyperbolic tangent activation function. Regarding convolutional parameter settings, all convolutional layers in the decoder employ a stride of 1 and padding of 1 to ensure that the input and output maintain spatial consistency. This design enables the fusion network of this invention to seamlessly process inputs of arbitrary shapes and sizes while generating output images of the same size as the input, thus ensuring the system's flexibility and adaptability.

[0075] (6) Composite Loss Function: To effectively constrain the network training process under unsupervised conditions, a composite loss function composed of pixel loss, texture loss, and structural similarity loss is designed. This composite loss function can achieve comprehensive optimization in terms of brightness distribution preservation, texture detail enhancement, and overall structural consistency, thereby ensuring the quality of the fused image.

[0076] The overall form of the composite loss function is:

[0077]

[0078] in, , and These represent pixel loss, texture loss, and SSIM loss, respectively. and It is a weighting factor used to balance the contributions of each entity.

[0079] Pixel loss is used to measure the difference in pixel intensity between the fused image and the source image, ensuring that the fused result maintains a brightness distribution consistent with the source image. The expression is:

[0080]

[0081] in It is the Frobenius norm. As a balance factor;

[0082] Texture loss measures the consistency of texture details between the fused image and the source image by introducing gradient information, resulting in a fused image with sharper edges and textures. The expression is:

[0083]

[0084] in It is the L2 norm. This is for gradient calculation;

[0085] Structural similarity loss (SSIM Loss) measures the similarity between the fused image and the source image in terms of brightness, contrast, and structural information, further ensuring the preservation of the global structure of the fused image. The expression is ,in, This represents the structural similarity index.

[0086] This invention proposes an infrared-visible light image fusion method based on a pseudo-twin network, which includes the following steps:

[0087] Step 1: Construct a pseudo-twin network image fusion model. The encoding stage, feature fusion stage, and decoding stage together constitute the construction of the pseudo-twin network image fusion model.

[0088] Please see Figure 2 , Figure 3 , Figure 4 and Figure 5 In step 1, a pseudo-twin encoder feature extraction is constructed. The encoding stage, feature fusion stage, and decoding stage together construct a twin encoder, specifically including the following steps:

[0089] The encoding stage includes a feature extraction module, a hybrid attention module, and a densely connected residual module.

[0090] The feature extraction module comprises two parallel branches, with the outputs being element-wise summed to obtain the feature map. The first branch contains two cascaded residual units, each with four parallel convolutional paths employing 1×1, 3×3, two 3×3, and three 3×3 convolutions respectively, and using a leaky linear rectified activation function. The outputs of the four paths are concatenated along the channel dimension to form a multi-scale feature map, which is then fused with channel information via a 1×1 convolution. The second branch performs channel compression and linear mapping through a 1×1 convolution, and is added to the first branch as a residual connection, ultimately achieving information reuse and efficient gradient propagation.

[0091] The hybrid attention module, located after the feature extraction module, receives stitched infrared and visible light feature maps as input for interactive modeling. The module consists of three parallel branches: a channel attention block, a spatial attention block, and a residual connection path. The channel attention block comprises five layers: a global average pooling layer to optimize spatial representation, a 1×1 convolution to reduce channel dimensionality, an LReLU activation function and a 1×1 convolution to increase channel dimensionality, and finally, a sigmoid activation to limit the output range. The spatial attention block consists of four layers: two 3×3 convolutional layers and an LReLU activation layer, used to extract spatial features. The residual connection path connects the input and output, fusing static and dynamic attention information. Finally, the outputs of the three branches are fused element-wise.

[0092] The densely connected residual module follows the hybrid attention module and employs a two-layer cascaded structure. Each densely connected residual block includes a main branch and a residual connection branch. In the main branch, local features are first extracted using a 3×3 convolutional layer and an LReLU activation function, followed by a second 3×3 convolutional layer and LReLU to further extract deeper features. The output is then processed by a 1×1 convolutional layer for channel fusion and feature compression. Feature fusion is achieved between the convolutional layers in the main branch through cross-layer connections, with the output of the first layer directly added to the outputs of subsequent layers via these connections. The residual connection branch uses a 1×1 convolution to adjust the channels of the input features, making them consistent with the output dimension of the main branch. Finally, the outputs of the main branch and the residual branch are fused element-wise to obtain the output of the densely connected residual module, achieving feature reuse and efficient gradient propagation.

[0093] The feature fusion stage includes three hybrid attention modules. Infrared features and visible light features are input to the hybrid attention modules for the infrared channel and the visible light channel, respectively. In addition, the infrared features and visible light features are spliced ​​along the channel dimension and then input to the hybrid attention module that fuses the two channels to achieve deep interactive fusion between modalities.

[0094] The decoding stage comprises four layered convolutional blocks. The first three convolutional blocks consist of convolutional operations, batch normalization layers, and LReLU activation functions, used to progressively extract and restore detailed features of the image. The last convolutional block consists of convolutional operations, batch normalization layers, and Tanh activation functions, used to limit the output values ​​to the range [−1, 1]. The fused features are then reconstructed to recover the final fused image. Through the decoding and reconstruction module, the model uses a series of convolutional operations and activation functions to restore the fused feature map into a complete image.

[0095] Step 2: Acquire infrared and visible light images, and preprocess the infrared and visible light images to obtain normalized infrared and visible light images respectively.

[0096] Please see Figure 1 In step 1, infrared and visible light images are acquired, and the infrared and visible light images are preprocessed to obtain normalized infrared and visible light images, respectively. Specifically, this includes the following steps:

[0097] Color space conversion is performed on the visible light image: the input dimension of the infrared image is H×W×1, and the input dimension of the Y channel of the visible light image is H×W×1. The RGB image of the visible light is converted into a YCbCr image; the Y channel is extracted as intensity information for fusion with the infrared image; the Cb and Cr channels are retained as color information in subsequent reconstruction.

[0098] The infrared image and the visible light brightness channel image are linearly normalized to obtain the normalized infrared image and the visible light brightness channel image, ensuring that the data of different modes are comparable in numerical range.

[0099] Step 3: Input the infrared image and the visible light image into the pseudo-twin network image fusion model. Concatenate the fused image of the brightness channel output by the model with the Cb and Cr chromaticity channels extracted from the visible light image and convert them to RGB space to obtain a color fused image. Train the model using a composite loss function. Input the infrared image and the visible light image into the optimized image enhancement fusion model to obtain the final color fused image.

[0100] Please see Figure 1In step 3, the infrared image and the visible light image are input into the pseudo-twin network image fusion model. The fused image of the luminance channel output by the model is concatenated with the Cb and Cr chromaticity channels extracted from the visible light image and converted to the RGB space to obtain a color fused image. Specifically, this includes the following steps:

[0101] The fused image of the luminance channel output by the model and the Cb and Cr chromaticity channels extracted from the visible light image are concatenated to obtain the concatenated features. The concatenated features are then converted from YCbCr space to RGB space using a color space conversion function to obtain the color fused image.

[0102] The network is trained using unsupervised learning. The composite loss function consists of the following three parts:

[0103] Pixel loss: Constrains the consistency of brightness distribution between the fused image and the source image; pixel loss The expression is:

[0104]

[0105] in It is the Frobenius norm. As a balance factor;

[0106] Texture loss: Preserving edge and texture details using gradient operators; texture loss The expression is:

[0107]

[0108] in It is the L2 norm. This is for gradient calculation;

[0109] Structural similarity loss: Ensures structural consistency and improves image quality in terms of visual perception. The expression is:

[0110]

[0111] in, Represents the structural similarity index;

[0112] The overall form of the loss function is:

[0113]

[0114] and It is a weighting factor used to balance the contributions of each entity.

[0115] During training, the learning rate is adjusted round by round, and iterative optimization is performed using mini-batch gradient descent until convergence.

[0116] Guided by the loss function, this embodiment can preserve rich detail and texture information while achieving optimal pixel intensity distribution. The final output image combines infrared sensing capabilities with visible light visual realism, meeting the multimodal image enhancement needs in complex low-light scenes.

Claims

1. An infrared-visible image fusion method based on pseudo-twin networks, characterized in that, Given a pair of input images, where the infrared image is represented as Visible light images are represented as Infrared and visible light images are fused in a pseudo-twin network image fusion model. Before being input into the pseudo-twin network image fusion model, the visible light image is converted from an RGB color space image to a YCbCr color space image, and the Y channel, i.e. the luminance component, is extracted and fused with the infrared image. The fusion result is then stitched together with the Cb and Cr channels in the original YCbCr color space. The pseudo-twin network image fusion model is divided into a pseudo-twin encoder, a feature fusion module, and a decoder. The pseudo-twin encoder adopts a dual-path parallel structure, with each path accepting the corresponding modal input. It consists of a feature extraction module, a hybrid attention module, and two cascaded densely connected residual modules. The feature fusion module includes three parallel hybrid attention modules: a hybrid attention module for the visible light channel, a hybrid attention module for the infrared channel, and a hybrid attention module that fuses the two channels. The decoder includes four hierarchically arranged convolutional blocks. (1) Feature extraction module: Visible light image and infrared image are input to two parallel feature extraction modules of pseudo twin encoder along two paths respectively; each feature extraction module is based on a single-modal feature map as input and contains two processing branches; the output results of the two branches are added element by element to obtain the output of the feature extraction module; The first branch contains two cascaded residual units, each with four parallel convolutional paths. The first path uses a 1×1 convolutional kernel, the second path uses a 3×3 convolutional kernel, the third path uses two consecutive 3×3 convolutions, and the fourth path uses three consecutive 3×3 convolutions. All four parallel convolutional paths use a leaky linear rectified activation function. The outputs of the four parallel convolutional paths are concatenated along the channel dimension to form a comprehensive feature map integrating information from different scales. After the two residual units are cascaded, they are then passed through a 1×1 convolutional layer for channel fusion and linear combination. The second branch performs channel compression and linear mapping on the input feature map through a 1×1 convolutional layer. As a residual connection path, it is added to the output of the first branch element-wise to obtain the output of the feature extraction module, realizing information reuse and efficient gradient propagation. (2) Hybrid attention module: It is set after the feature extraction module. The input is the feature after the output of the infrared feature extraction module and the output of the visible light feature extraction module are spliced ​​along the channel direction. It is used to perform interactive modeling of the two modal features. The hybrid attention module consists of three parallel branches: channel attention block, spatial attention block and residual connection path. The final output is the element-wise sum of the outputs of the three parallel branches. The channel attention block consists of five layers. The first layer is a global average pooling layer, which is used to optimize the spatial representation. The second layer is a 1×1 convolution to reduce the channel dimension; the third and fourth layers are LReLU activation functions and a 1×1 convolution to increase the channel dimension; the last layer is a sigmoid activation function to restrict the output to between 0 and 1. The spatial attention block consists of four layers, including a 3×3 convolutional layer, an LReLU activation layer, a 3×3 convolutional layer, and a sigmoid function layer, which are used to mine feature information in the spatial dimension. The residual connection path is used to establish jump connections between the input and output of the hybrid attention module, enabling the fusion of static and dynamic attention information; finally, the results of the channel attention block, spatial attention block, and residual connection path are added element by element for fusion. (3) Dense connection residual module: It is set after the addition node of the pseudo-twin encoder and adopts a two-layer series dense connection residual block structure. On the two parallel paths of infrared and visible light, the dense connection residual module takes the feature of the corresponding modal feature extraction module and the hybrid attention module after element-wise summation as input. A single densely connected residual block consists of a main branch and residual connection branches, and the output is the element-wise sum of the results of the two branches; In the main branch, multiple convolutional layers and activation layers are sequentially arranged. The first convolutional layer uses a 3×3 convolutional kernel followed by an LReLU activation function to extract local features. The second convolutional layer also uses a 3×3 convolutional kernel followed by an LReLU activation function to further extract deep features. The output of the second convolutional layer is channel-fused and feature-compressed via a 1×1 convolutional layer. Cross-layer connections are introduced between the convolutional layers in the main branch: while the output of the first layer is passed to the next layer, it is also directly added to the output of the subsequent layer through cross-layer connections to achieve the fusion of multi-level features. The residual connection branch uses a 1×1 convolutional layer to adjust the channels of the features output from the hybrid attention module, making them consistent with the output of the main branch in dimension. Finally, the output of the main branch and the output of the residual branch are fused together by element-wise addition to obtain the output of the densely connected residual module, realizing feature reuse and efficient gradient propagation. (4) Feature fusion: Located between the pseudo-twin encoder and decoder, it takes the output features of the residual module densely connected by two paths as input to realize multimodal fusion of infrared and visible light features; the feature fusion includes three parallel hybrid attention modules, and its data flow is as follows: Infrared and visible light features are respectively fed into the single-modal hybrid attention module of the corresponding path; the infrared and visible light features are spliced ​​in the channel dimension and further input into the hybrid attention module that fuses the two channels to achieve deep interactive fusion between modes; In the above fusion process, the outputs of the three parallel hybrid attention modules are then concatenated along the channel dimension to obtain the feature fusion output; this fused feature is fed into the decoder as input to reconstruct the output image; (5) Decoder: Located after the feature fusion module, it is used to aggregate and reconstruct the image after feature fusion and output the final fused image. The input of the decoder is the fused features after the output of the three parallel hybrid attention modules of feature fusion are spliced ​​along the channel dimension. The output is the reconstructed fused image. The decoder adopts a hierarchical network structure, which consists of four sequentially arranged convolutional blocks: the first three convolutional blocks are composed of convolution operations, batch normalization layers and LReLU activation functions, which are used to gradually extract and restore the detailed features of the image; the last convolutional block is composed of convolution operations, batch normalization layers and Tanh activation functions, which are used to limit the output value to the range of [-1,1]. (6) Composite Loss Function: Design a composite loss function consisting of pixel loss, texture loss, and structural similarity loss; the overall form of the composite loss function is: in, , and These represent pixel loss, texture loss, and SSIM loss, respectively. and It is a weighting factor used to balance the contributions of each entity.

2. The infrared-visible image fusion method based on pseudo-twin networks according to claim 1, characterized in that, In the composite loss function, Pixel loss The expression is: ,in It is the Frobenius norm. As a balance factor, To merge images, Visible light image; Texture loss The expression is ,in It is the L2 norm. For gradient calculation, Infrared image; Structural similarity loss The expression is ,in, This represents the structural similarity index.

Images