Infrared and visible image fusion method based on information interaction and edge guidance

Abstract

The application discloses an infrared and visible light image fusion method based on information interaction and edge guidance, visible light images and infrared images in a data set are simultaneously input into a fusion network, five pairs of features with different granularities are obtained through a multi-granularity feature separation encoder, are input into a three-branch scene fidelity module for image fusion operation, and five fusion features with different granularities are obtained; the fusion features are input into a progressive cross-granularity interaction feature enhancement module, five enhanced features with different granularities are obtained through operation and processing of the module; the enhanced features are sequentially input into an image reconstruction decoder, are subjected to splicing and dimension reduction processing, and a fusion image is reconstructed; the fusion network is trained based on a visible light image, an infrared image, the fusion image and an edge-content guidance loss function, and real-time visible light image and infrared image fusion is performed based on the trained fusion network, and the overall contrast and visual quality of the fusion image are improved.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and in particular relates to a method for fusing infrared and visible light images based on information interaction and edge guidance. Background Technology

[0002] Due to hardware limitations, images acquired by a single-modal sensor or in a single shooting setting contain limited information and can only describe the imaging scene from a single perspective. Furthermore, single-modal images are susceptible to extreme weather conditions, leading to a significant deterioration in their visual quality. To acquire high-quality image data, image fusion technology, which is simpler and more efficient than hardware improvements, has sparked heated and widespread discussion in the field of visual imaging. This technology aims to extract complementary information from images of different modalities and combine them to obtain information-rich fused images that are beneficial for subsequent applications.

[0003] In the field of image fusion, infrared and visible light image fusion is the most widely used image fusion technique. Visible light images have excellent spatial resolution and can provide rich scene information. However, in dark or harsh environments, they cannot distinguish targets that need to be highlighted, such as pedestrians or vehicles. Conversely, even under harsh conditions, infrared images can highlight key targets such as pedestrians through thermal radiation information, but they usually lack texture detail. Based on the complementarity between infrared and visible light images, the fusion result often contains both rich scene information and significant thermal radiation targets.

[0004] To date, numerous methods for fusion of infrared and visible light images have been proposed. Generally, traditional methods involve three steps. First, features are extracted from the source image based on mathematical transformations. Then, the extracted features are fused using specific fusion strategies, such as adaptive fusion strategies and weighted average fusion strategies. Finally, the fused features are inversely transformed to obtain the fused image. Traditional image fusion methods can be categorized into five types based on the mathematical transformations used: multi-scale methods, sparse representation-based methods, subspace-based methods, optimization-based methods, and hybrid methods. While these traditional methods can yield relatively satisfactory fused images, the increasing complexity of feature transformations and representations in these methods makes them insufficient for the requirements of image fusion, and manually designed fusion rules cannot fully integrate complementary information from the source images.

[0005] In recent years, the rapid development of deep learning has provided more possibilities and room for development in the fusion of infrared and visible light images. Deep learning-based methods, with their powerful feature extraction and learning capabilities, often achieve more ideal fusion results. Depending on the network framework used, deep learning-based image fusion methods can be broadly classified into four categories: methods based on convolutional neural networks (CNNs), methods based on autoencoders (AEs), methods based on generative adversarial networks (GANs), and methods based on Transformers. To achieve more ideal image fusion results, some scholars have proposed a representative CNN-based image fusion method, which for the first time treats the image fusion problem as a problem of preserving the texture and intensity ratio of the source images. Other scholars have proposed an end-to-end residual network for infrared and visible light image fusion, solving the problem of the unlearnable fusion strategy and further improving the network's fusion performance. Still others have proposed a multi-classification constrained generative adversarial network, transforming image fusion into a multi-distribution simultaneous estimation problem, and leveraging the excellent performance of generative adversarial mechanisms in the unsupervised learning field to achieve good image fusion results.

[0006] While these deep learning-based methods have made some progress, several obstacles remain to be overcome. Image fusion based on convolutional neural networks and autoencoders often employs only the simplest operations such as superposition and stitching for feature fusion, while Transformer-based methods tend to extract contextual information from the source image within a single modality. These methods typically neglect cross-modal information interaction, which is closely related to the image fusion problem. Existing multi-granularity transform-based methods, although avoiding information loss by extracting multi-granularity features, often only consider operations between features of the same granularity, lacking information interaction between cross-granularity features, thus failing to further extract valuable information from the source image. Edges are one of the most fundamental features of an image, containing rich information. However, existing image fusion methods often ignore the impact of edges on the visual quality of the image, resulting in insufficient ability of the designed fusion network to capture edge detail features, thereby negatively affecting the quality of the fused image. Summary of the Invention

[0007] To address the above technical problems, this invention provides a method for fusing infrared and visible light images based on information interaction and edge guidance.

[0008] The technical solution adopted by this invention to solve its technical problem is:

[0009] An infrared and visible light image fusion method based on information interaction and edge guidance, the method comprising the following steps:

[0010] S100: Acquire visible light and infrared image datasets, build a fusion network, which includes a multi-granularity feature separation encoder, a three-branch scene fidelity module, a progressive cross-granularity interactive feature enhancement module, and an image reconstruction decoder;

[0011] S200: The visible light images and infrared images in the dataset are simultaneously input into the fusion network, and five pairs of features of different granularities are obtained through the multi-granularity feature separation encoder.

[0012] S300: The five pairs of features with different granularities are input into the three-branch scene fidelity module for image fusion operation to obtain five fusion features with different granularities;

[0013] S400: The five fused features of different granularities are input into the progressive cross-granularity interactive feature enhancement module. The module processes the features to obtain five enhanced features of different granularities. The input fused features correspond one-to-one with the output enhanced features, and the granularity remains unchanged.

[0014] S500: The five enhancement features of different granularities are sequentially input into the image reconstruction decoder for splicing and dimensionality reduction to reconstruct the fused image;

[0015] S600: The fusion network is trained based on visible light images, infrared images, fused images, and an edge-content guided loss function. When the edge-content guided loss function converges, the trained fusion network is obtained. Real-time fusion of visible light images and infrared images is performed based on the trained fusion network.

[0016] Preferably, the multi-granularity feature separation encoder in S200 includes two parallel infrared and visible light multi-granularity feature extraction branches, each branch consisting of a convolutional layer, a pooling layer and four residual blocks to obtain multi-granularity features of the source image.

[0017] Preferably, the three-branch scene fidelity module in S300 includes an original feature information branch, a difference information branch, and an infrared salient information branch.

[0018] The original feature information branch is used to perform convolution operations on features of the same granularity extracted from infrared and visible light images, then perform a stitching operation, and finally perform a convolution operation on the stitched features to obtain the original information features.

[0019] The difference information branch is used to perform an element-oriented maximum value operation on features of the same granularity extracted from infrared and visible light images. The resulting features are then subjected to two convolution operations to obtain the difference information features.

[0020] The infrared salient information branch is used to perform element-oriented subtraction on features of the same granularity extracted from infrared and visible light images, and then perform two convolution operations on the resulting features to obtain infrared salient information features.

[0021] The fused features are obtained based on the original information features, the difference information features, and the infrared salient information features.

[0022] Preferably, the formula for the original feature information branch can be expressed as:

[0023] F raw =Conv(Concat(Conv(F) ir ), Conv(F vis )))

[0024] Among them, F raw Represents the original information features, Conv represents the convolution operation, and Concat represents the concatenation operation;

[0025] The formulas for the difference information branch and the infrared saliency information branch can be expressed as:

[0026] F dif =Conv(Conv(Sub(F) ir F vis )))

[0027] F max =Conv(Conv(Max(F) ir F vis )))

[0028] Among them, F dif F represents the difference information feature. max Represents significant infrared information features, each Conv represents a convolution operation, and Sub and Max represent element-oriented subtraction and element-oriented maximum value operations, respectively.

[0029] Fusion feature F f The calculation process can be expressed by the following formula:

[0030] F f =Add(F raw F dif F max )

[0031] Add represents the element-oriented summation operation.

[0032] Preferably, the specific granularity feature of the last layer of the progressive cross-granularity interactive feature enhancement module is itself used as the enhancement feature of the last layer, and the feature granularity features of the remaining layers are superimposed with the cross-granularity difference features of the same granularity to obtain the enhancement feature of the current layer. S400 is specifically as follows:

[0033]

[0034] CDF(x, y) = Add(Sub(Conv1(F x ) (1×1) Conv1(F y ) (1×1) ),

[0035] Sub(Conv1(F x ) (3×3) Conv1(F y ) (3×3) ),

[0036] Sub(Conv1(F x ) (5×5) Conv1(F y ) (5×5) ))

[0037] Where EF(x) represents enhanced features of different granularities, SGF represents specific granularity features obtained through feature fusion, CDF represents cross-granularity difference features obtained by multi-scale subtraction between different granularities, and F x and F y CDF(x, y) represents the mapping of adjacent features of different granularities from the previous layer. x and F y The cross-granularity difference features obtained by performing multi-scale feature subtraction operations, Conv1(·) (n×n) Sub represents a convolutional layer of size n×n with all parameters set to 1, while Sub represents an element-oriented subtraction operation.

[0038] Preferably, S500 specifically comprises:

[0039] In the process of image fusion reconstruction, the fifth and fourth layer enhancement features are first concatenated. After upsampling and convolution, the resulting features are then connected to the third layer enhancement features. After upsampling and convolution, the resulting features are then connected to the second layer enhancement features. After upsampling and convolution, the resulting features are then connected to the first layer enhancement features. Finally, two convolution operations are performed on the 64-channel fusion features to reconstruct the fusion image.

[0040] Preferably, the edge-content guided loss function in S600 is as follows:

[0041] L total =L content +β·L edge

[0042] Among them, L content Let L be the content loss function. edge Let β be the edge loss function, and let β be a hyperparameter that balances content loss and edge loss.

[0043] Preferably, the content loss function includes an intensity loss function, a gradient loss function, and a structural similarity loss function, specifically:

[0044] L content =L int +α1·L grad +α2·L ssim

[0045]

[0046]

[0047] L ssim =(1-ssim(I f I ir ))+(1-ssim(I f I vis ))

[0048] Among them, L int For the intensity loss function, L grad For gradient loss function, L ssim Here, α1 and α2 are the structural similarity loss function, α1 and α2 are the hyperparameters used to balance the three loss functions, H and W represent the height and width of the input and output images, respectively, and ||·||1 represents the l1-norm regularization operation. The Sobel gradient operator is represented by |·|, which represents the absolute value operation, and ssim(·) represents the structural similarity operation. f Indicates the fused image, I ir and I vis These represent infrared images and visible light images, respectively.

[0049] Preferably, the edge loss function is as follows:

[0050]

[0051] Where Canny(·) represents the edge information extraction operation based on the Canny operator, I f Indicates the fused image, I ir and I vis Representing infrared and visible light images respectively, Canny(I)ir )+Canny(I vis ) and Canny (I f Canny(I) represents the edge information set used as the real reference and the edge information set of the fused image, respectively. f ) i and (Canny(I) ir )+Canny(I vis )) i Let represent the i-th element in each of the two edge information sets, * denotes the element-wise multiplication of the two matrices, and τ is used to avoid a denominator of 0.

[0052] The aforementioned infrared and visible light image fusion method based on information interaction and edge guidance achieves information interaction between cross-modal features through a three-branch scene fidelity module. During the feature fusion stage, it mines the difference information between cross-modal features and the infrared saliency information within them, improving the visual fidelity of the fused image. This fully captures scene information and salient target information from the extracted features, resulting in a fused image rich in scene information and prominent salient targets. The progressive cross-granularity interactive feature enhancement module achieves information interaction between cross-granularity features, mining the difference information between them and enriching the texture and structural information of the fused features, avoiding the loss of detail information in the fused image. The edge loss function guides the training of the fusion network, improving its ability to extract edge details from the source image. This guides the fusion network to retain edge detail information from the source image, reducing the loss of edge details in the fused image and improving the overall contrast and visual quality of the fused image. Furthermore, the fusion network has relatively few parameters and can be easily deployed to various advanced vision tasks, such as pedestrian detection and vehicle tracking. Attached Figure Description

[0053] Figure 1 This is a flowchart of an infrared and visible light image fusion method based on information interaction and edge guidance in one embodiment of the present invention;

[0054] Figure 2 This is a structural block diagram of an infrared and visible light image fusion method based on information interaction and edge guidance in one embodiment of the present invention;

[0055] Figure 3 This is a schematic diagram illustrating the principle of a progressive cross-granularity interactive feature enhancement module in one embodiment of the present invention.

[0056] Figure 4 This is a schematic diagram illustrating the principle of an image reconstruction decoder in one embodiment of the present invention; Detailed Implementation

[0057] To enable those skilled in the art to better understand the technical solution of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings.

[0058] In one embodiment, such as Figure 1 As shown, an infrared and visible light image fusion method based on information interaction and edge guidance includes the following steps:

[0059] S100: Acquire visible light and infrared image datasets, build a fusion network, which includes a multi-granularity feature separation encoder, a three-branch scene fidelity module, a progressive cross-granularity interactive feature enhancement module, and an image reconstruction decoder;

[0060] S200: The visible light images and infrared images in the dataset are simultaneously input into the fusion network, and five pairs of features of different granularities are obtained through the multi-granularity feature separation encoder.

[0061] S300: The five pairs of features with different granularities are input into the three-branch scene fidelity module for image fusion operation to obtain five fusion features with different granularities;

[0062] S400: The five fused features of different granularities are input into the progressive cross-granularity interactive feature enhancement module. The module processes the features to obtain five enhanced features of different granularities. The input fused features correspond one-to-one with the output enhanced features, and the granularity remains unchanged.

[0063] S500: The five enhancement features of different granularities are sequentially input into the image reconstruction decoder for splicing and dimensionality reduction to reconstruct the fused image;

[0064] S600: The fusion network is trained based on visible light images, infrared images, fused images, and an edge-content guided loss function. When the edge-content guided loss function converges, the trained fusion network is obtained. Real-time fusion of visible light images and infrared images is performed based on the trained fusion network.

[0065] Specifically, to address the aforementioned problems, this invention proposes a novel infrared and visible light image fusion method based on information interaction and edge guidance. The proposed infrared and visible light image fusion method's network consists of four parts: a multi-granularity feature separation encoder, a three-branch scene fidelity module, a progressive cross-granularity interactive feature enhancement module, and an image reconstruction decoder. Furthermore, to enhance the network's ability to extract edge information, a novel edge loss function is designed to guide the training of the fusion network. Figure 2 The overall framework of the proposed infrared and visible light image fusion method mainly consists of two parts: the specific network structure and the loss function guiding network training. Furthermore, the specific process of the proposed fusion method is shown below:

[0066] 1. First, infrared and visible light images are simultaneously input into the fusion network, and five pairs of features of different granularities are obtained through a multi-granularity feature separation encoder.

[0067] 2. Next, the five pairs of features with different granularities are input into the three-branch scene fidelity module for image fusion operation to obtain five fusion features with different granularities.

[0068] 3. Then, the five fused features with different granularities are input into the progressive cross-granularity interactive feature enhancement module. The module processes the features to obtain five enhanced features with different granularities. The input fused features correspond one-to-one with the output enhancements, and the granularity remains unchanged.

[0069] 4. Finally, the five enhancement features of different granularities obtained in the previous step are sequentially input into the image reconstruction decoder for splicing and dimensionality reduction to reconstruct the final fused image.

[0070] In one embodiment, the multi-granularity feature separation encoder in S200 includes two parallel infrared and visible light multi-granularity feature extraction branches, each consisting of a convolutional layer, a pooling layer, and four residual blocks to obtain multi-granularity features of the source image.

[0071] Specifically, the encoder aims to extract multi-granular features from the source image. For example... Figure 2 As shown, the encoder of this fusion network includes two parallel branches for infrared and visible light multi-granularity feature extraction. Each branch uses Res2net-50 (residual network) as its backbone, and each branch consists of convolutional layers, pooling layers, and four residual blocks. Res2net-50 is chosen as the backbone due to its powerful feature extraction capabilities and ease of training. Based on the structural characteristics of Res2net-50, each branch is divided into five layers to obtain multi-granularity features from the source image.

[0072] In one embodiment, the three-branch scene fidelity module in S300 includes an original feature information branch, a difference information branch, and an infrared salient information branch.

[0073] The original feature information branch is used to perform convolution operations on features of the same granularity extracted from infrared and visible light images, then perform a stitching operation, and finally perform a convolution operation on the stitched features to obtain the original information features.

[0074] The difference information branch is used to perform an element-oriented maximum value operation on features of the same granularity extracted from infrared and visible light images. The resulting features are then subjected to two convolution operations to obtain the difference information features.

[0075] The infrared salient information branch is used to perform element-oriented subtraction on features of the same granularity extracted from infrared and visible light images, and then perform two convolution operations on the resulting features to obtain infrared salient information features.

[0076] The fused features are obtained based on the original information features, the difference information features, and the infrared salient information features.

[0077] In one embodiment, the formula for the original feature information branch can be expressed as:

[0078] F raw =Conv(Concat(Conv(F) ir ), Conv(F vis )))

[0079] Among them, F raw Represents the original information features, Conv represents the convolution operation, and Concat represents the concatenation operation;

[0080] The formulas for the difference information branch and the infrared saliency information branch can be expressed as:

[0081] F dif =Conv(Conv(Sub(F) ir F vis )))

[0082] F max =Conv(Conv(Max(F) ir F vis )))

[0083] Among them, F dif F represents the difference information feature. max Represents significant infrared information features, each Conv represents a convolution operation, and Sub and Max represent element-oriented subtraction and element-oriented maximum value operations, respectively.

[0084] Fusion feature F f The calculation process can be expressed by the following formula:

[0085] F f =Add(F raw F dif F max )

[0086] Add represents the element-oriented summation operation.

[0087] Specifically, to achieve information interaction between cross-modal features and more fully integrate complementary information in the source images, this invention designs a triple-branch scene fidelity module (TBSF) in the network structure. Here, F... ir and F vis These represent features of the same granularity extracted from infrared and visible light images, respectively. Before the feature concatenation operation and after the difference and maximum value operations are completed, this invention uses 3×3 convolution to reduce the number of channels in the initial features or the features obtained after the operation, thereby reducing the number of parameters in the overall network. In the first information branch, i.e., the original feature information branch, two features of the same granularity are concatenated to fully preserve the original information of the extracted features. In addition to the original feature information branch, two additional information branches, an infrared saliency information branch and a difference information branch, are added to achieve information interaction between cross-modal features. The second information branch, i.e., the difference information branch, can fully explore the difference information between cross-modal features, such as scene information in visible images. In addition, the third information branch, i.e., the infrared saliency information branch, can obtain significant infrared information from the source image, such as thermal radiation targets represented in the infrared image. As compensation for the original feature information branch, these additional information branches can ensure that the fused features retain more meaningful information, thereby helping the fusion network reconstruct a fused image with higher visual quality.

[0088] In one embodiment, the specific granularity feature of the last layer of the progressive cross-granularity interactive feature enhancement module is itself used as the enhancement feature of the last layer, and the feature granularity features of the remaining layers are superimposed with the cross-granularity difference features of the same granularity to obtain the enhancement feature of the current layer. S400 is specifically as follows:

[0089]

[0090] CDF(x, y) = Add(Sub(Conv1(F x ) (1×1) Conv1(F y ) (1×1) ),

[0091] Sub(Conv1(F x ) (3×3) Conv1(F y ) (3×3) ),

[0092] Sub(Conv1(F x ) (5×5) Conv1(F y ) (5×5) ))

[0093] Where EF(x) represents enhanced features of different granularities, SGF represents specific granularity features obtained through feature fusion, CDF represents cross-granularity difference features obtained by multi-scale subtraction between different granularities, and F x and F Y CDF(x, y) represents the mapping of adjacent features of different granularities from the previous layer. x and F y The cross-granularity difference features obtained by performing multi-scale feature subtraction operations, Conv1(·) (n×n) This represents an n×n convolutional layer with all parameters set to 1, while Sub represents an element-oriented subtraction operation. Through this multi-scale processing method, the feature subtraction unit can acquire more differential information across granular features, thereby enriching the texture and structural information contained in the fused features.

[0094] Specifically, the Progressive Cross-granularity Interaction Feature Enhancement Module (PCIFE) aims to achieve information interaction between cross-granularity features, thereby enriching the texture and structural information of the fused features. For example... Figure 3 As shown, SGF is used here to represent the specific granularity feature obtained through feature fusion, and CDF represents the cross-granularity difference feature obtained by multi-scale subtraction between different granularities. Except for the specific granularity feature SSF(5,1) of the last layer, which itself serves as the enhancement feature EF(5), the specific granularity features of the remaining layers are superimposed with the cross-granularity difference features of the same granularity to obtain the enhancement features.

[0095] The multi-scale feature subtraction unit employed in this module can fully exploit the differences between cross-granularity features, thus avoiding subtraction operations on single, isolated pixel-level regions. This multi-scale feature subtraction unit uses convolutional layers with all parameters set to 1×1, 3×3, and 5×5 to calculate the details and structural differences of features at the pixel and region levels, respectively, thereby obtaining rich texture and structural information. Using convolutional layers with all parameters set to 1 allows for efficient network training without introducing additional parameter overhead. It is important to emphasize that deeper features have already been upsampled before performing subtraction operations on cross-granularity features.

[0096] In one embodiment, S500 specifically refers to:

[0097] In the process of image fusion reconstruction, the fifth and fourth layer enhancement features are first concatenated. After upsampling and convolution, the resulting features are then connected to the third layer enhancement features. After upsampling and convolution, the resulting features are then connected to the second layer enhancement features. After upsampling and convolution, the resulting features are then connected to the first layer enhancement features. Finally, two convolution operations are performed on the 64-channel fusion features to reconstruct the fusion image.

[0098] Specifically, the decoder is used to reconstruct the fused image, with a specific structure as follows: Figure 2 As shown, EF(x) represents the input enhanced feature, and the numbers represent the number of channels of the output feature.

[0099] In one embodiment, the edge-content guidance loss function in S600 is specifically as follows:

[0100] L total =L content +β·L edge

[0101] Among them, L content Let L be the content loss function. edge Let β be the edge loss function, and let β be a hyperparameter that balances content loss and edge loss.

[0102] Specifically, the loss function plays a crucial role in image fusion networks. To improve the visual quality and various quantitative evaluation metrics of the fused image, this invention designs a loss function combining content loss and edge loss. On the one hand, the content loss function drives the fusion network to fully integrate complementary information from the source images, such as texture details in visible light images and significant thermal radiation information in infrared images. On the other hand, a novel edge loss function is proposed to guide the fusion network in preserving edge information from the source images that is closely related to the visual quality of the fused image.

[0103] In one embodiment, the content loss function includes an intensity loss function, a gradient loss function, and a structural similarity loss function, specifically:

[0104] L content =L int +α1·L grad +α2·L ssim

[0105]

[0106]

[0107] L ssim =(1-ssim(I f ,I ir))+(1-ssim(I f ,I vis ))

[0108] Among them, L int For the intensity loss function, L grad For gradient loss function, L ssim Here, α1 and α2 are the structural similarity loss function, α1 and α2 are the hyperparameters used to balance the three loss functions, H and W represent the height and width of the input and output images, respectively, and ||·||1 represents the l1-norm regularization operation. The Sobel gradient operator is represented by |·|, which represents the absolute value operation, and ssim(·) represents the structural similarity operation. f Indicates the fused image, I ir and I vis These represent infrared images and visible light images, respectively.

[0109] Specifically, the content loss function, composed of intensity loss, gradient loss, and structural similarity loss, helps the fusion network integrate richer and more meaningful information. Typically, the intensity loss and the other two loss functions are not on the same order of magnitude; therefore, hyperparameters α1 and α2 are designed to balance these three loss functions. The intensity loss function measures the pixel-level intensity difference between the fused image and the source image. The gradient loss function is introduced to encourage the fusion network to extract rich texture details from the source image. The structural similarity loss function measures the structural similarity between the fused image and the source image based on brightness, contrast, and structural information.

[0110] In one embodiment, the edge loss function is specifically:

[0111]

[0112] Where Canny(·) represents the edge information extraction operation based on the Canny operator, I f Indicates the fused image, I ir and I vis Representing infrared and visible light images respectively, Canny(I) ir )+Canny(I vis ) and Canny (I f Canny(I) represents the edge information set used as the real reference and the edge information set of the fused image, respectively. f ) i and (Canny(I) ir )+Canny(I vis )) i Let represent the i-th element in each of the two edge information sets, * denotes the multiplication of corresponding elements of the two matrices, and τ is used to avoid a denominator value of 0.

[0113] Specifically, edge details in an image are closely related to its visual quality, thus having a crucial impact on the application of fused images in various advanced vision tasks. This invention proposes a novel edge loss function to guide the training of a fusion network, aiming to obtain fused images with superior visual quality and improve the performance of various advanced vision tasks. This edge loss function drives the fusion network to preserve edge details in the source images by evaluating the similarity between two edge information sets. The principle for calculating the similarity between edge information sets can be expressed as follows:

[0114]

[0115] Here, A and B represent the edge information set used as the real reference and the edge information set of the fused image, respectively. The larger the intersection area of ​​A and B, the higher the similarity between the two edge information sets.

[0116] In this invention, the Canny operator is used to obtain the edge information of the source image and the fused image. It should be noted that the edge information sets of the source image and the fused image only contain pixels with values ​​of 0 and 1. When calculating the similarity of the edge information sets, this invention only considers pixels with a value of 1, which represent edge information. Since the main information contained in infrared images and visible light images differs to some extent and lacks a true reference, when calculating this loss function, the information set obtained by adding the edge information sets of the infrared image and the visible light image is used as the true reference, as shown in the formula above.

[0117] Existing infrared and visible light image fusion frameworks typically neglect information interaction between cross-modal features and cross-granularity features. This results in the fused image failing to fully integrate complementary information from the source images, leading to a loss of scene information and texture detail, and blurring of salient targets. Furthermore, existing fusion frameworks do not consider the impact of edge details on the quality of the fused image, causing it to fail to retain sufficient edge information from the source images, resulting in reduced overall contrast and weakened visual quality. All of these factors indicate that the resulting fused image cannot reliably improve the performance of various advanced vision tasks. Therefore, this invention proposes an infrared and visible light image fusion method based on information interaction and edge guidance. The main advantages of this invention can be summarized as follows:

[0118] (1) The network of the present invention is an end-to-end trainable network. The network extracts valuable information from features from the perspective of cross-modal interaction and cross-granularity interaction, thereby fully acquiring texture details and salient targets in the source image. Then, based on the extracted features, it performs fusion and reconstruction to obtain a fused image.

[0119] (2) This invention proposes a three-branch scene fidelity module, which realizes information interaction between cross-modal features, mines the difference information between cross-modal features and the infrared salient information in cross-modal features during the feature fusion stage, and improves the visual fidelity of the fused image. It can fully acquire scene information and salient target information in the extracted features, so that the fused image contains rich scene information and prominent salient targets.

[0120] (3) The present invention proposes a progressive cross-granularity interactive feature enhancement module, which realizes information interaction between cross-granularity features, mines the difference information between cross-granularity features, enriches the texture and structural information of the fused features, and avoids the loss of detail information in the fused image.

[0121] (4) The present invention designs a novel edge loss function, which guides the training of the fusion network to improve the network’s ability to extract edge details in the source image, guides the fusion network to retain edge detail information in the source image, reduces the loss of edge details in the fusion image, and improves the overall contrast and visual quality of the fusion image.

[0122] (5) The network of the present invention has fewer parameters and can be easily deployed to various advanced vision tasks, such as pedestrian detection and vehicle tracking.

[0123] The infrared and visible light image fusion method based on information interaction and edge guidance provided by this invention has been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this invention. The descriptions of the embodiments above are only for the purpose of helping to understand the core ideas of this invention. It should be noted that those skilled in the art can make several improvements and modifications to this invention without departing from the principles of this invention, and these improvements and modifications also fall within the protection scope of the claims of this invention.

Claims

1. A method for fusing infrared and visible light images based on information interaction and edge guidance, characterized in that, The method includes the following steps: S100: Acquire visible light and infrared image datasets, build a fusion network, which includes a multi-granularity feature separation encoder, a three-branch scene fidelity module, a progressive cross-granularity interactive feature enhancement module, and an image reconstruction decoder; S200: The visible light images and infrared images in the dataset are simultaneously input into the fusion network, and five pairs of features of different granularities are obtained through the multi-granularity feature separation encoder. S300: The five pairs of features with different granularities are input into the three-branch scene fidelity module for image fusion, resulting in five fused features with different granularities. The three-branch scene fidelity module in S300 includes the original feature information branch, the difference information branch, and the infrared saliency information branch. The original feature information branch is used to perform convolution operations on features of the same granularity extracted from infrared and visible light images, then perform a stitching operation, and finally perform a convolution operation on the stitched features to obtain the original information features. The difference information branch is used to perform an element-oriented maximum value operation on features of the same granularity extracted from infrared and visible light images. The resulting features are then subjected to two convolution operations to obtain the difference information features. The infrared salient information branch is used to perform element-oriented subtraction on features of the same granularity extracted from infrared and visible light images, and then perform two convolution operations on the resulting features to obtain infrared salient information features. The fused features are obtained based on the original information features, the difference information features, and the infrared salient information features; S400: The five fused features of different granularities are input into the progressive cross-granularity interactive feature enhancement module. This module processes the features to obtain five enhanced features of different granularities. The input fused features correspond one-to-one with the output enhanced features, and the granularity remains unchanged. The specific granularity feature of the last layer of the progressive cross-granularity interactive feature enhancement module itself serves as the enhanced feature of the last layer. The granularity features of the remaining layers are superimposed with the cross-granularity difference features of the same granularity to obtain the enhanced feature of the current layer. Specifically, S400 is as follows: in, Representing enhancement features at different granularities, SGF This represents a specific granularity feature obtained through feature fusion. CDF This represents the cross-granularity difference feature obtained by performing multi-scale subtraction operations between different granularities. and This represents a mapping of adjacent features at different granularities from the previous layer. Indicates by and Cross-granularity difference features obtained by performing multi-scale feature subtraction. This represents a convolutional layer of size n×n with all parameters set to 1. Sub This indicates an element-oriented subtraction operation; S500: The five enhancement features of different granularities are sequentially input into the image reconstruction decoder for splicing and dimensionality reduction to reconstruct the fused image; S600: The fusion network is trained based on visible light images, infrared images, fused images, and an edge-content guided loss function. When the edge-content guided loss function converges, the trained fusion network is obtained. Real-time fusion of visible light images and infrared images is performed based on the trained fusion network.

2. The method according to claim 1, characterized in that, The S200 multi-granularity feature separation encoder includes two parallel infrared and visible light multi-granularity feature extraction branches. Each branch consists of a convolutional layer, a pooling layer, and four residual blocks to obtain multi-granularity features of the source image.

3. The method according to claim 2, characterized in that, The formula for the original feature information branch can be expressed as: in, Represents the characteristics of the original information. Conv This represents the convolution operation. Concat This indicates a splicing operation. and These represent features of the same granularity extracted from infrared and visible light images, respectively. The formulas for the difference information branch and the infrared saliency information branch can be expressed as: in, Indicates the characteristics of difference information. Representing salient infrared information features, each Conv This represents the convolution operation. Sub and Max These represent element-oriented subtraction and element-oriented maximum value operations, respectively. Fusion features The calculation process can be expressed by the following formula: in Add This represents an element-oriented summation operation.

4. The method according to claim 3, characterized in that, The S500 specifically refers to: In the process of image fusion reconstruction, the fifth and fourth layer enhancement features are first concatenated. After upsampling and convolution, the resulting features are then connected to the third layer enhancement features. After upsampling and convolution, the resulting features are then connected to the second layer enhancement features. After upsampling and convolution, the resulting features are then connected to the first layer enhancement features. Finally, two convolution operations are performed on the 64-channel fusion features to reconstruct the fusion image.

5. The method according to claim 4, characterized in that, The edge-content guided loss function in S600 is as follows: in, For content loss function, For edge loss function, Hyperparameters to balance content loss and edge loss.

6. The method according to claim 5, characterized in that, Content loss functions include intensity loss function, gradient loss function, and structural similarity loss function, specifically: in, For the intensity loss function, For gradient loss function, Let S be the structural similarity loss function. and These are the hyperparameters used to balance the three loss functions. H and W These represent the height and width of the input and output images, respectively. express - Norm regularization operation This represents the Sobel gradient operator. This indicates the absolute value operation. ssim ( ) represents the structural similarity operation. Indicates a fused image. and These represent infrared images and visible light images, respectively.

7. The method according to claim 6, characterized in that, The edge loss function is as follows: in, This indicates that edge information extraction is performed based on the Canny operator. Indicates a fused image. and These represent infrared images and visible light images, respectively. and These represent the edge information set used as the real reference and the edge information set of the fused image, respectively. and These represent the i-th elements in the two edge information sets, respectively. This represents the element-wise multiplication of two matrices. Used to avoid a denominator of 0.

Images