Infrared and visible light vision information fusion method based on gradient transform prior

Abstract

The application discloses an infrared and visible light visual information fusion method based on gradient transformation prior, and the method comprises the following steps: acquiring a registered infrared image and a registered visible light source image, wherein the registered infrared image and the registered visible light source image represent a consistent geometric transformation relationship in space; constructing a gradient transformation prior network; inputting the registered infrared image and the registered visible light source image into the gradient transformation prior network for feature extraction image fusion processing to obtain a reconstructed fusion image. By using the application, the detailed texture information in the visible light and the thermal radiation target information in the infrared image can be accurately extracted, and the fusion quality of the image is improved. The application can be widely applied to the field of image fusion technology as the infrared and visible light visual information fusion method based on the gradient transformation prior.

Description

Technical Field

[0001] This invention relates to the field of image fusion technology, and in particular to a method for fusing infrared and visible light visual information based on gradient transformation priors. Background Technology

[0002] Infrared and visible light fusion refers to the fusion of infrared and visible light images to generate a richer and clearer image. Many image fusion methods have been developed, including traditional methods and end-to-end learning methods. Traditional methods utilize mathematical transformations to convert the source image into the transform domain, and then perform active-level measurements and design fusion rules within the transform domain to achieve image fusion. Traditional image fusion techniques include methods based on multi-scale decomposition, subspace clustering, sparse representation, optimization, and hybrid methods. Images synthesized using these methods can meet the needs of subsequent tasks in specific scenarios. However, the development of traditional methods is currently experiencing bottlenecks. On the one hand, the transformations or representations used in traditional methods are becoming increasingly complex to achieve more impressive fusion performance, failing to meet the requirements of real-time computing applications. On the other hand, these traditional methods often require manual extraction of image features or reliance on professional knowledge for model building, thus lacking adaptability and expressiveness, and unable to handle large-scale, high-dimensional, and nonlinear data well. Moreover, the fusion results of these methods are often not refined enough to meet the requirements of high-precision applications. Therefore, with the development of deep learning technology, deep learning-based methods are gradually becoming the mainstream approach for image fusion tasks.

[0003] In recent years, the boom in deep learning has spurred the image fusion community to explore end-to-end learning-driven image fusion solutions. Mainstream end-to-end learning methods can be broadly categorized into three types: autoencoder (AE)-based methods, convolutional neural network (CNN)-based methods, and generative adversarial network (GAN)-based methods. AE-based methods first train an autoencoder as a feature extractor and image reconstructor on a natural image dataset. Then, the feature extractor extracts complementary information from multimodal images, and these features are merged using specific fusion rules such as stitching, element-wise addition, element-weighted summation, and element-maximization. Finally, the image reconstructor bears the burden of reconstructing the fused image from the fused features. However, the AE-based fusion framework is not fully learnable because it uses handcrafted fusion rules to merge deep features. Therefore, other researchers have focused on exploring image fusion networks based on end-to-end CNNs, which rely on superior network architecture and fine loss functions to ensure fusion performance. Given the lack of fundamental facts in the image fusion task, some works have attempted to define image fusion as a game between a generator and a discriminator. More specifically, they force the fused image to have rich texture details by constraining the probability distribution between the fused image and the source image. However, too strong a constraint may introduce artificial textures into the fused image, resulting in unsatisfactory final results. Summary of the Invention

[0004] To address the aforementioned technical problems, the present invention aims to provide a method for fusing infrared and visible light visual information based on gradient transformation priors, which can accurately extract detailed texture information in visible light and thermal radiation target information in infrared images, thereby improving the fusion quality of images.

[0005] The first technical solution adopted in this invention is: a method for fusing infrared and visible light visual information based on gradient transformation prior, comprising the following steps:

[0006] Acquire a registered infrared image and a registered visible light source image, wherein the registered infrared image and the registered visible light source image represent a consistent geometric transformation relationship in space;

[0007] Construct a gradient transformation prior network;

[0008] The registered infrared image and the registered visible light source image are input into the gradient transformation prior network for feature extraction and image fusion processing to obtain the reconstructed fused image.

[0009] Furthermore, the step of constructing the gradient transformation prior network specifically includes:

[0010] A gradient transformation prior network is constructed by introducing a residual convolution module, a multi-layer self-attention interaction module, and a feature decoding module.

[0011] The residual convolution module includes a first branch residual convolution and a second branch residual convolution. The first branch residual convolution includes a first residual convolution block, a second residual convolution block, a third residual convolution block, and a fourth residual convolution block. The second branch residual convolution includes a fifth residual convolution block, a sixth residual convolution block, a seventh residual convolution block, and an eighth residual convolution block. The residual convolution blocks are connected in a dense connection manner.

[0012] The multi-layer self-attention interaction module includes a first multi-layer self-attention interaction block, a second multi-layer self-attention interaction block, a third multi-layer self-attention interaction block, and a fourth multi-layer self-attention interaction block, and the multi-layer self-attention interaction blocks are connected in series.

[0013] The input end of the residual convolution module is connected to the image input information, the output end of the residual convolution block is connected to the input end of the multi-layer self-attention interaction module, the output end of the multi-layer self-attention interaction module is fed back to the output end of the residual convolution block, and the output end of the residual convolution block is also connected to the input end of the feature decoding module.

[0014] Furthermore, the step of inputting the registered infrared image and the registered visible light image into a gradient transform prior network for feature extraction and image fusion processing to obtain the reconstructed fused image specifically includes:

[0015] The registered infrared image and the registered visible light source image are input into the gradient transformation prior network;

[0016] The first branch of the residual convolution module in the gradient transform prior network is used to perform image feature extraction processing on the registered infrared image to obtain the infrared image features.

[0017] The second branch of the residual convolution module in the gradient transform prior network performs image feature extraction processing on the registered visible light source image to obtain the visible light source image features.

[0018] Based on the multi-layer self-attention interaction module in the gradient transformation prior network, feature fusion processing is performed on visible light source image features and infrared image features to obtain a feature fused image.

[0019] The feature decoding module in the gradient transform prior network performs stepwise image reconstruction on the feature fusion image to obtain the reconstructed fusion image.

[0020] Furthermore, the step of performing feature fusion processing on visible light source image features and infrared image features to obtain a feature fused image, based on the multi-layer self-attention interaction module in the gradient transformation prior network, specifically includes:

[0021] The visible light source image features and infrared image features are input into the multi-layer self-attention interaction module;

[0022] The attention unit based on the multi-layer self-attention interaction module further extracts and processes the features of visible light source images and infrared images to obtain visible light source feature maps and infrared feature maps.

[0023] The channel attention module based on the multi-layer self-attention interaction module calculates the weights of the visible light source feature map and the infrared feature map, generates a weight matrix, and combines it with the visible light source feature map and the infrared feature map to obtain a feature map with enhanced attention.

[0024] The spatial attention module based on the multi-layer self-attention interaction module performs spatial convolution processing on the feature map to enhance attention, and obtains the feature fusion image.

[0025] Furthermore, the channel attention module based on the multi-layer self-attention interaction module calculates weights for the visible light source feature map and the infrared feature map, generates a weight matrix, and combines it with the visible light source feature map and the infrared feature map to obtain a feature map with enhanced attention. This step specifically includes:

[0026] Input the visible light feature map and the infrared feature map into the channel attention module;

[0027] Based on the channel attention module, the average pooling aggregation feature module aggregates the spatial information of the visible light source feature map and the infrared feature map to generate a spatial context descriptor.

[0028] The shared network module based on the channel attention module maps the spatial context descriptor to generate the channel attention mapping weight matrix.

[0029] By multiplying the channel attention mapping weight matrix with the visible light feature map and the infrared feature map element by element, a feature map with enhanced attention is obtained.

[0030] Furthermore, the channel attention module processes the visible light feature map and the infrared feature map as follows:

[0031]

[0032] In the above formula, MLP(·) represents the multilayer perceptron operation, and Av g Pool(·) represents the average pooling aggregation feature operation. The descriptor represents the spatial context, W2 and W1 represent the weight matrices of the fully connected layer, F represents the image of the input channel attention module, σ represents the sigmoid function, and M represents the spatial context descriptor. c (F) represents the channel attention mapping weight matrix.

[0033] Furthermore, the step of the spatial attention module based on the multi-layer self-attention interaction module performing spatial convolution processing on the feature map to obtain the feature fusion image specifically includes:

[0034] The enhanced attention feature map is input into the spatial attention module;

[0035] The feature maps for enhanced attention are processed sequentially by max pooling and average pooling, and the generated results are concatenated to generate feature descriptors.

[0036] The feature descriptors are convolved by the convolutional layer of the spatial attention module to generate a two-dimensional spatial attention map, i.e., a feature fusion image.

[0037] Furthermore, the expression for the feature map processing procedure of the spatial attention module to enhance attention is as follows:

[0038]

[0039] In the above formula, This represents a two-dimensional spatial attention map, where F represents the image of the input spatial attention module. 7×7 (·) represents a convolution operation with a kernel size of 7, σ represents the sigmoid function, and M max (F) represents the result of the max pooling operation in the spatial attention module, M avg (F) represents the result of the average pooling operation of the spatial attention module.

[0040] Furthermore, the step of performing stepwise image reconstruction processing on the feature fusion image by the feature decoding module in the gradient transform prior network to obtain the reconstructed fusion image specifically includes:

[0041] The feature fusion image is input into the feature decoding module, and the weight matrix of the feature fusion image is calculated to obtain the attention weight matrix;

[0042] The attention weight matrix is ​​multiplied by the feature fusion image to obtain the feature fusion image with enhanced attention.

[0043] The feature fusion image after attention enhancement is combined with the feature fusion image and then subjected to three consecutive convolutional decoding processes to obtain the reconstructed fusion image.

[0044] Furthermore, it also includes constructing a gradient transformation prior network loss function to enrich the details of the reconstructed fused image. The gradient transformation prior network loss function includes an intensity loss function, a joint gradient loss function, and a guided filter loss function. The intensity loss function is used to maintain a clear intensity distribution in the reconstructed fused image; the joint gradient loss function is used to enhance the texture details of the reconstructed fused image; and the guided filter loss function is used to smooth the detailed parts of the reconstructed fused image. The expression for the gradient transformation prior network loss function is:

[0045]

[0046] In the above formula, L total This represents the gradient transformation prior network loss function. Represents the intensity loss function. Denotes the joint gradient loss function. Let γ1, γ2, and γ3 represent the guided filter loss function, and let γ1, γ2, and γ3 represent the corresponding weight coefficients.

[0047] The beneficial effects of this invention are as follows: By acquiring registered infrared images and registered visible light images, this invention ensures that they have a consistent geometric transformation relationship in space. Registration aims to eliminate differences in position, rotation, scale, and other transformations between images, enabling effective fusion operations. A gradient transformation prior network is further constructed, using a pair of infrared and visible light images as input, and sequentially performing feature extraction and multi-feature fusion to generate the fusion result. The feature extraction module uses skip connections to update model parameters, making the training process more efficient. To further enhance the effectiveness of skip connections, channel attention and spatial attention are introduced to promote high-quality interaction between multiple network channels. The multi-feature fusion module adaptively calculates the weight matrix of each feature path, multiplying the weight matrix with the input feature map to obtain a feature map with enhanced attention, enabling the network to more accurately represent the target scene. Finally, considering the excellent edge information extraction capability of guided filtering in edge-preserving filters, a loss function is introduced to promote richer details in the fused image. This invention can accurately extract detailed texture information from visible light and thermal radiation target information from infrared images, improving the image fusion quality. Attached Figure Description

[0048] Figure 1 This is a flowchart of the steps of the infrared and visible light visual information fusion method based on gradient transformation prior of the present invention;

[0049] Figure 2 This is a block diagram of the gradient transformation prior network constructed in this invention;

[0050] Figure 3 This is a schematic diagram of the curve of the LeakyRelu function of this invention;

[0051] Figure 4 This is a schematic diagram of the structure of the multi-layer self-attention feature interaction module in the gradient transformation prior network constructed in this invention;

[0052] Figure 5 This is a schematic diagram of the channel attention module in the multi-layer self-attention feature interaction module of the present invention;

[0053] Figure 6 This is a schematic diagram of the spatial attention module in the multi-layer self-attention feature interaction module of the present invention;

[0054] Figure 7 This is a schematic diagram of the multi-feature fusion module in the gradient transformation prior network constructed in this invention;

[0055] Figure 8 This is a schematic diagram showing the results of an image fusion experiment comparing the method of this invention with five existing image fusion methods. Detailed Implementation

[0056] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments. The step numbers in the following embodiments are only for ease of explanation and do not limit the order of the steps. The execution order of each step in the embodiments can be adapted according to the understanding of those skilled in the art.

[0057] Reference Figure 1 This invention provides a method for fusing infrared and visible light visual information based on gradient transformation priors, the method comprising the following steps:

[0058] S1. Obtain the registered infrared image I ir And the registered visible light source image I vis The registered infrared image and the registered visible light image represent a consistent geometric transformation relationship in space;

[0059] Specifically, in the field of image fusion, a "registered source image" refers to an input image that has undergone registration processing. Registration refers to aligning multiple images using a certain algorithm or technique to ensure they have consistent geometric transformations in space. The purpose of registration is to eliminate differences in position, rotation, scale, and other transformations between images to enable effective fusion. If an unregistered pair of images is fused, the final fused image will exhibit feature misalignment and severe artifacts, violating the original intention of image fusion to "integrate information from the original images to generate richer and more accurate images."

[0060] S2. Construct a gradient transformation prior network;

[0061] Specifically, refer to Figure 2This invention uses skip connections to update model parameters. These skip connections can transfer more details to later layers to make full use of available information and make our training process efficient.

[0062] Furthermore, this invention constructs a gradient transformation prior network, which includes a residual convolution module, a multi-layer self-attention interaction module, and a feature decoding module. The residual convolution module further includes a first-branch residual convolution and a second-branch residual convolution. The first-branch residual convolution includes a first residual convolution block, a second residual convolution block, a third residual convolution block, and a fourth residual convolution block. The second-branch residual convolution includes a fifth residual convolution block, a sixth residual convolution block, a seventh residual convolution block, and an eighth residual convolution block. The connections between the residual convolution blocks are dense connections. The multi-layer self-attention interaction module includes a first multi-layer self-attention interaction block, a second multi-layer self-attention interaction block, and a third multi-layer self-attention interaction block. The first branch of the residual convolutional network receives the registered visible light image as input, and the second branch of the residual convolution receives the registered infrared image as input. In the first branch of the residual convolution, the first, second, third, and fourth residual convolutional blocks are connected sequentially. In the second branch of the residual convolution, the fifth, sixth, seventh, and eighth residual convolutional blocks are connected sequentially. The output of the first residual convolutional block is connected to the input of the second residual convolutional block and the input of the first multi-layer self-attention interaction block. The output of the residual convolutional block is connected to the input of the third residual convolutional block and the input of the second multi-layer self-attention interaction block. The output of the third residual convolutional block is connected to the input of the fourth residual convolutional block and the input of the third multi-layer self-attention interaction block. The output of the fourth residual convolutional block is connected to the input of the feature decoding module and the input of the fourth multi-layer self-attention interaction block. The output of the fourth multi-layer self-attention interaction block is also connected to the input of the feature decoding module. The output of the fifth residual convolutional block is connected to the input of the first multi-layer self-attention interaction block and the input of the sixth residual convolutional block. The output of the sixth residual convolutional block is connected to the input of the second multi-layer self-attention interaction block and the input of the seventh residual convolutional block. The seventh residual convolutional block... The output of the first multi-layer self-attention interaction block is connected to the input of the third multi-layer self-attention interaction block and the input of the eighth residual convolutional block. The output of the eighth residual convolutional block is connected to the input of the feature decoding module. The output of the first multi-layer self-attention interaction block is connected to the output of the first residual convolutional block and feeds back to the input of the second residual convolutional block. The output of the second multi-layer self-attention interaction block is connected to the output of the second residual convolutional block and feeds back to the input of the third residual convolutional block. The output of the third multi-layer self-attention interaction block is connected to the output of the third residual convolutional block and feeds back to the input of the fourth residual convolutional block. The output of the fourth multi-layer self-attention interaction block is connected to the output of the fourth residual convolutional block and feeds back to the input of the feature decoding module.

[0063] S3. Input the registered infrared image and the registered visible light source image into the gradient transformation prior network for feature extraction and image fusion processing to obtain the reconstructed fused image.

[0064] Specifically, the gradient transformation prior network of this invention includes residual convolutional blocks (rHDC), a multi-layer self-attention interaction module (MSFI), and a feature decoding module. The residual convolutional blocks extract features from the source image; their structure involves stacking three complete convolutional layers and using residual connections. The multi-layer self-attention module fuses features between two original images; its structure is obtained by cascading units with channel attention and spatial attention mechanisms. The feature decoding module gradually reconstructs a complete image from the feature maps obtained by the residual convolutional blocks and the multi-layer self-attention module. Its structure includes a two-layer convolution for calculating the weight map and a three-layer convolution for reconstructing the image. The gradient transformation prior network is divided into two branches, corresponding to two different input images: an infrared image and a visible light image. For the visible light image branch, the input image undergoes feature extraction through four residual convolutional blocks, which use dense connections to enhance network training stability. The feature maps extracted by the four residual convolutional blocks are input to the next level and also to the multi-layer self-attention feature interaction module. Similar to the visible light image branch, the infrared image branch also extracts features through four residual convolutional blocks. However, since infrared images only contain heat source information and have simple features, and to reduce computational complexity, the infrared image branch does not use dense connections but only simple concatenation. The feature maps extracted by the four residual convolutional blocks of the infrared image branch are input to the next stage and also to the multi-layer self-attention feature interaction module. Since there are four residual convolutional blocks in the dual-branch architecture, there are also four multi-layer self-attention feature interaction modules. The fused feature map input to the multi-layer self-attention feature interaction module is returned to the infrared branch. This completes the feature extraction and feature fusion of the original image. For the feature decoding module, the specific process is to pass the dual-input feature maps through two convolutional layers, then pass them through the sigmoid function to obtain a weight map, multiply it with the original input feature map, and finally output the reconstructed fused image through three consecutive convolutional layers.

[0065] S31, Based on residual convolution module;

[0066] Specifically, the feature extraction network of this invention has four residual convolutional blocks (rHDC), each consisting of three convolutional layers connected by residuals. The kernel size is 3×3, and the stride and padding are set to 1. Each layer extracts 64 feature maps. To make the model more convergent during training and enhance its stability, this invention sets all activation functions to LeakyReLU, except for Tanh in the last layer. LeakyReLU is a variant of the classic ReLU activation function, such as... Figure 3 As shown, it has a very small slope for negative inputs. This means that when the input is less than 0, the information is not completely lost, but rather preserved accordingly. Furthermore, this invention employs batch normalization except for the last layer, which overcomes the sensitivity to data initialization and avoids the problems of vanishing and exploding gradients.

[0067] S32, Based on a multi-layer self-attention interaction module;

[0068] Specifically, refer to Figure 4 To make the extracted features more accurate and effective, this invention designs a multi-layer self-attention feature interaction module (MSFI), which consists of two cascaded attention units. Its unique architecture can enhance the network's ability to extract features, and the skip connection structure can strengthen the model's stability during training.

[0069] Furthermore, channel attention maps are generated by utilizing the inter-channel relationships of the features. Since each channel of the feature map is considered a feature detector, the channel attention module focuses on "what" is meaningful in a given input image. The specific computational flow of channel attention is as follows: Figure 5 As shown, it calculates a weight matrix and then operates on this weight matrix with the feature map to obtain a feature map with enhanced attention. This mechanism can retain valuable features and discard worthless features.

[0070] S321, Channel attention module based on multi-layer self-attention interaction module;

[0071] Specifically, the present invention first generates a spatial context descriptor by using the spatial information of the aggregated feature map through average pooling. This represents the average pooling feature. The descriptor is then forwarded to a shared network to generate a channel attention map M. c ∈R C×1×1 This shared network consists of a multilayer perceptron (MLP) and a hidden layer. To reduce parameter overhead, the hidden activation size is set to R. C / r×1×1 Where r is the reduction ratio, after the shared network is applied to each descriptor, this invention uses element-wise multiplication to combine the weight matrix with the original feature map to obtain the enhanced feature map. Let the input feature map be F∈g. C ×H×W Where C is the number of channels, and H and W are the height and width of the feature map, respectively, the channel attention process can be summarized as follows:

[0072]

[0073] In the above formula, MLP(·) represents the multilayer perceptron operation, and AvgPool(·) represents the average pooling feature aggregation operation. The descriptor represents the spatial context, W2 and W1 represent the weight matrices of the fully connected layer, F represents the image of the input channel attention module, σ represents the sigmoid function, and M represents the spatial context descriptor. c (F) represents the channel attention mapping weight matrix;

[0074] Feature map after channel attention enhancement It can be represented as follows:

[0075]

[0076] In the above formula, This represents an element-wise multiplication operation between pixels.

[0077] S322, Spatial attention module based on multi-layer self-attention interaction module;

[0078] Specifically, the spatial attention module primarily generates a spatial attention map by utilizing the spatial relationships between features. Unlike channel attention, spatial attention focuses on "where" important information is contained in the image, thus complementing channel attention. The specific computational process of spatial attention is as follows: Figure 6 As shown;

[0079] To compute spatial attention, this invention first applies average pooling and max pooling operations along the channel axis and concatenates them to generate an effective feature descriptor. Applying pooling operations along the channel axis effectively highlights informational regions. On the cascaded feature descriptor, we apply convolutional layers to generate a spatial attention map Ms(F)∈RH×W, which encodes the locations of emphasis or suppression. We describe the detailed operation below. This invention aggregates the channel information of a feature map through two pooling operations, generating two two-dimensional maps: M avg M max These represent the average pooling feature and the max pooling feature across the channels, respectively. These are then concatenated and convolved by a standard convolutional layer to produce a two-dimensional spatial attention map. In short, the spatial attention process can be summarized as follows:

[0080]

[0081] In the above formula, This represents a two-dimensional spatial attention map, where F represents the image of the input spatial attention module. 7×7 (·) represents a convolution operation with a kernel size of 7, σ represents the sigmoid function, and M max (F) represents the result of the max pooling operation in the spatial attention module, M avg (F) represents the result of the average pooling operation of the spatial attention module.

[0082] S33, Feature-based decoding module;

[0083] Specifically, such as Figure 7 As shown, in order to enable the network to adaptively select features from visible and infrared light, this invention proposes a multi-feature decoding module. Since the feature extraction part of this invention extracts numerous and complex features, and the differences between infrared and visible light feature maps are significant, in order to select the more valuable feature maps to be fused, this invention first needs to calculate the weight matrix of each feature map, the expression of which is:

[0084]

[0085] In the above formula, A w Represents the attention weight matrix;

[0086] Next, the attention weight matrix is ​​multiplied by the input feature map to obtain the enhanced attention feature map, which is expressed as:

[0087]

[0088]

[0089] In the above formula, f 1×1 F represents a convolution operation with a kernel size of 1. vi and F ir These represent the input feature maps, and the enhanced feature maps will be compared with the original feature maps. Figure 1 The final fusion result is decoded through three consecutive convolutional layers.

[0090] S4. Construct a gradient transformation prior network loss function to enrich the details of the reconstructed and fused image.

[0091] Specifically, the primary goal of infrared and visible light fusion is to preserve as much texture information as possible from the visible light image and brightness information from the infrared image. To preserve brightness information, direct regression under the supervision of the common similarity of the visible and infrared images is undoubtedly an effective method. SSIM (Structural Similarity Simulation) considers brightness, contrast, and structural indices, which aligns with human visual perception and outperforms L1 and L2 constraints. Therefore, to ensure the intensity distribution of the fused image remains clear and consistent with the original image, we introduce an intensity loss. The loss function is expressed as follows:

[0092]

[0093] Furthermore, we expect the fused image to maintain the optimal intensity distribution while preserving rich texture details. Based on extensive experiments, we find that the optimal texture of the fused image can be represented as the maximum aggregation of the textures from the infrared and visible light images. We model the gradient distribution and develop a joint gradient loss, expressed as:

[0094]

[0095] Where ||(·)||1 represents the l1 norm, The gradient map represents the gradient in the horizontal and vertical directions. This invention uses the Sobel operator to calculate the gradient map, and max(·) represents the element-wise maximum selection.

[0096] Furthermore, the main idea of ​​the network in this invention is to introduce additional effective constraints on texture details. Therefore, to obtain clearer textures, this invention also introduces guided filtering to enhance the constraints, ensuring that the detail information of visible light and the target information of infrared images are preserved to the maximum extent. Specifically, guided filtering (GIF) is an edge-preserving method based on a local linear model, often used as a filter for image smoothing. It can effectively smooth the details in an image while preserving strong edges. GIF is similar to bilateral filtering, but it can be optimized into a radius-independent method, thus having higher computational efficiency. In addition, guided filtering can solve the problem of bilateral filtering, which only preserves edges but not gradients (gradient maps). Assuming the guided image is G, the input image is f, and the output image is O, the linear model is as follows:

[0097]

[0098] Among them, Ω ψ1 (p′) represents a square window with radius ψ1 centered at pixel p′, and ψ1 is set to 15 in this invention. The parameters of α and β are determined by the minimum cost function, as follows:

[0099]

[0100] in Is it constraint α p′ The regularization parameter is set to 3 in this invention. Furthermore, for ease of reading, this invention represents the GIF process as GIF(·), therefore, this constraint can be expressed as:

[0101]

[0102] Finally, the gradient transformation prior network loss function of this invention can be summarized as follows:

[0103]

[0104] In the above formula, L total This represents the gradient transformation prior network loss function. Represents the intensity loss function. Denotes the joint gradient loss function. Let γ1, γ2, and γ3 represent the guided filter loss function, and let γ1, γ2, and γ3 represent the corresponding weight coefficients.

[0105] In summary, this invention proposes a general-purpose infrared and visible light image fusion method. First, a pair of infrared and visible light images are used as input, and feature extraction and multi-feature fusion are performed sequentially to generate the fused result. The feature extraction module uses skip connections to update model parameters, making the training process more efficient. Simultaneously, to further enhance the effectiveness of skip connections, channel attention and spatial attention are introduced to promote high-quality interaction between multiple channels in the network. The multi-feature fusion module adaptively calculates the weight matrix of each feature path, multiplying the weight matrix with the input feature map to obtain a feature map with enhanced attention, enabling the network to more accurately represent the target scene. Finally, considering the excellent edge information extraction capability of guided filtering in edge-preserving filters, this invention incorporates it into the loss function, allowing the fused image to express richer details. This invention can accurately extract detailed texture information from visible light and thermal radiation target information from infrared images, providing high-quality fusion results.

[0106] Furthermore, the simulation results of this invention are shown below:

[0107] To further demonstrate the advantages and effectiveness of this invention, a comparative experiment was conducted with five existing image fusion methods, and the performance of each method was analyzed based on subjective visual evaluation. Figure 8 (a) and Figure 8 (b) shows two images of visible light and infrared sources. Figure 8 (c) to (h) represent, respectively, TEMF: a perceptual framework for infrared-visible image fusion based on multi-scale structural decomposition and biological vision; MFEIF: an image fusion edge attention-guided method based on deep multi-scale feature integration; SeAFusion: a real-time infrared and visible light image fusion network based on semantic awareness; Perceptual Gan: an image fusion method based on a perceptually consistent generative adversarial network; and U2Fusion: a unified unsupervised image fusion network. The fusion results obtained by the proposed methods are also shown below. Figure 8 As shown, Figure 8 Methods (c)-(g) fail to preserve the original image's outline well in overly bright scenes, resulting in blurry cars, while lacking sufficient contrast and detail in normal environments. Through... Figure 8(h) It can be seen that the method proposed in this invention has a strong ability to extract target information from infrared images and texture information from visible light images, and can maintain a reasonable contrast in the fusion result. In summary, the method of this scheme has better fusion performance than the five most advanced contrast methods.

[0108] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. A method for fusing infrared and visible visual information based on gradient transform prior, characterized in that, Includes the following steps: Acquire a registered infrared image and a registered visible light source image, wherein the registered infrared image and the registered visible light source image represent a consistent geometric transformation relationship in space; Constructing a gradient transformation prior network specifically includes: A gradient transformation prior network is constructed by introducing a residual convolution module, a multi-layer self-attention interaction module, and a feature decoding module. The residual convolution module includes a first branch residual convolution and a second branch residual convolution. The first branch residual convolution includes a first residual convolution block, a second residual convolution block, a third residual convolution block, and a fourth residual convolution block. The second branch residual convolution includes a fifth residual convolution block, a sixth residual convolution block, a seventh residual convolution block, and an eighth residual convolution block. The four residual convolution blocks of the first branch residual convolution are connected in a dense connection manner, and the first branch residual convolution is a visible light image branch. The four residual convolution blocks of the second branch residual convolution are connected in a series connection manner, and the second branch residual convolution is an infrared image branch. The multi-layer self-attention interaction module includes a first multi-layer self-attention interaction block, a second multi-layer self-attention interaction block, a third multi-layer self-attention interaction block, and a fourth multi-layer self-attention interaction block, and the multi-layer self-attention interaction blocks are connected in series. The input end of the residual convolution module is connected to the image input information, the output end of the residual convolution block is connected to the input end of the multi-layer self-attention interaction module, the output end of the multi-layer self-attention interaction module is fed back to the output end of the residual convolution block, and the output end of the residual convolution block is also connected to the input end of the feature decoding module. The registered infrared image and the registered visible light source image are input into the gradient transformation prior network; The first branch of the residual convolution module in the gradient transform prior network is used to perform image feature extraction processing on the registered visible light source image to obtain the visible light source image features. The second branch of the residual convolution module in the gradient transform prior network is used to perform image feature extraction processing on the registered infrared image to obtain infrared image features. Based on the multi-layer self-attention interaction module in the gradient transformation prior network, feature fusion processing is performed on visible light source image features and infrared image features to obtain a feature fused image. The feature fusion image is input into the feature decoding module, and the weight matrix of the feature fusion image is calculated to obtain the attention weight matrix; The attention weight matrix is ​​multiplied by the feature fusion image to obtain the feature fusion image with enhanced attention. The feature fusion image after attention enhancement is combined with the feature fusion image and then subjected to three consecutive convolutional decoding processes to obtain the reconstructed fusion image; It also includes constructing a gradient transformation prior network loss function to enrich the details of the reconstructed fused image. The gradient transformation prior network loss function includes an intensity loss function, a joint gradient loss function, and a guided filter loss function. The intensity loss function is used to maintain a clear intensity distribution in the reconstructed fused image; the joint gradient loss function is used to enhance the texture details of the reconstructed fused image; and the guided filter loss function is used to smooth the detailed parts of the reconstructed fused image. The expression for the gradient transformation prior network loss function is: ； In the above formula, This represents the gradient transformation prior network loss function. Represents the intensity loss function. Denotes the joint gradient loss function. This represents the guided filter loss function. , and This represents the corresponding weighting coefficient.

2. The infrared and visible visual information fusion method based on gradient transformation prior as described in claim 1, characterized in that, The step of the multi-layer self-attention interaction module in the gradient transformation prior network performing feature fusion processing on visible light source image features and infrared image features to obtain a feature fused image specifically includes: The visible light source image features and infrared image features are input into the multi-layer self-attention interaction module; The attention unit based on the multi-layer self-attention interaction module further extracts and processes the features of visible light source images and infrared images to obtain visible light source feature maps and infrared feature maps. The channel attention module based on the multi-layer self-attention interaction module calculates the weights of the visible light source feature map and the infrared feature map, generates a weight matrix, and combines it with the visible light source feature map and the infrared feature map to obtain a feature map with enhanced attention. The spatial attention module based on the multi-layer self-attention interaction module performs spatial convolution processing on the feature map to enhance attention, and obtains the feature fusion image.

3. The infrared and visible visual information fusion method based on gradient transformation prior as described in claim 2, characterized in that, The channel attention module based on the multi-layer self-attention interaction module calculates weights for the visible light source feature map and the infrared feature map, generates a weight matrix, and combines it with the visible light source feature map and the infrared feature map to obtain a feature map with enhanced attention. This step specifically includes: Input the visible light feature map and the infrared feature map into the channel attention module; Based on the channel attention module, the average pooling aggregation feature module aggregates the spatial information of the visible light source feature map and the infrared feature map to generate a spatial context descriptor. The shared network module based on the channel attention module maps the spatial context descriptor to generate the channel attention mapping weight matrix. By multiplying the channel attention mapping weight matrix with the visible light feature map and the infrared feature map element by element, a feature map with enhanced attention is obtained.

4. The infrared and visible visual information fusion method based on gradient transformation prior as described in claim 3, characterized in that, The channel attention module processes the visible light feature map and infrared feature map as follows: ； In the above formula, This indicates multilayer perceptron operation. This indicates the average pooling aggregation feature operation. Represents the spatial context descriptor, and This represents the weight matrix of the fully connected layer. The image represents the input channel attention module. This represents the sigmoid function. This represents the channel attention mapping weight matrix.

5. The infrared and visible visual information fusion method based on gradient transformation prior as described in claim 4, characterized in that, The step of the spatial attention module based on the multi-layer self-attention interaction module performing spatial convolution processing on the feature map to obtain the feature fusion image specifically includes: The enhanced attention feature map is input into the spatial attention module; The feature maps for enhanced attention are processed sequentially by max pooling and average pooling, and the generated results are concatenated to generate feature descriptors. The feature descriptors are convolved by the convolutional layer of the spatial attention module to generate a two-dimensional spatial attention map, i.e., a feature fusion image.

6. The infrared and visible visual information fusion method based on gradient transformation prior as described in claim 5, characterized in that, The expression for the feature map processing procedure of the spatial attention module to enhance attention is as follows: ； In the above formula, Representing a two-dimensional spatial attention diagram, The image represents the input spatial attention module. This indicates a convolution operation with a kernel size of 7. This represents the sigmoid function. This represents the result of the max pooling operation in the spatial attention module. This represents the result of the average pooling operation in the spatial attention module.

Images