An Infrared and Visible Image Fusion Method Based on Multi-Layer Convolution

Abstract

This invention discloses a method for fusing infrared and visible light images based on multi-layer convolution. The network structure includes an encoder network, a decoder network, and a multi-layer convolutional fusion network. The encoder is composed of nested multi-layer convolutional blocks and an ECA attention mechanism. The decoder is mainly composed of decoding blocks, each of which consists of two convolutional layers. The multi-layer convolutional fusion network is mainly composed of gradient convolutional blocks, downsampling convolutional blocks, a convolutional spatial channel attention mechanism, and several convolutional layers. The method includes the following steps: S1, the registered infrared source image and visible light source image are fed into the encoder in pairs, and the encoder extracts the source image features; S2, the multi-layer convolutional fusion network fuses the source image features to obtain the fused features; S3, the decoder reconstructs the fused features and outputs the image. The fused image produced by this invention has prominent targets, clear details, obvious contours, and significantly improved performance indicators, conforming to human visual perception.

Description

Technical Field

[0001] This invention relates to the field of image processing, and more particularly to a method for fusing infrared and visible light images based on multi-layer convolution. Background Technology

[0002] Infrared and visible light images reflect the different characteristics of a target scene under different imaging instruments. Infrared images have strong penetrating power and are unaffected by light intensity, but lack texture information. Visible light images contain rich structural information and have better visual effects, but are easily affected by factors such as weather and lighting conditions, and have poor anti-interference capabilities. Therefore, fusing infrared and visible light images to fully utilize their complementary information has broad application value in various fields.

[0003] Existing image fusion methods mainly include traditional methods and deep learning-based methods. A typical example of traditional methods is the multi-scale transformation-based approach. This method is widely used to extract multi-scale features from source images. It obtains multi-scale features from the input image by using multi-scale transformation, and then fuses the multi-scale features of different images according to specific rules. Finally, the fused image is reconstructed through inverse multi-scale transformation. This type of method can capture features at different scales, allowing for a more comprehensive understanding of the image. However, selecting and designing appropriate transformation rules is difficult, and the fusion rules are complex, requiring the derivation and calculation of numerical formulas.

[0004] Currently, deep learning-based infrared and visible light image fusion methods are divided into three types: convolutional neural networks, generative adversarial networks, and autoencoders. Autoencoder-based methods encode the source image, fuse features, decode and reconstruct the image, and finally generate a fused image. In 2018, Li et al. (Li H, Wu X J. DenseFuse: A Fusion Approach to Infrared and Visible Images.[J].IEEE Trans. Image Processing, 2019, 28(5)) introduced dense connections into the encoder network to extract image depth features, proposing DenseFuse for infrared and visible light image fusion. In 2020, Li et al. (Li H, Wu XJ, Durani T. NestFuse: An Infrared and Visible Image Fusion Architecture based on Nest Connection and Spatial / Channel Attention Models[J]. IEEE Transactions on Instrumentation and Measurement, 2020, PP(99) designed a fusion strategy that can fuse multi-scale features based on spatial attention and channel attention, and proposed the NestFuse network. In 2021, Li et al. (Li H, Wu XJ, Kittler J. RFN-Nest: An end-to-end residual fusion network for infrared and visible images[J]. Information Fusion, 2021, 73, 72-86) proposed an end-to-end RFN-Nest network based on NestFuse. Subsequently, many scholars proposed more fusion methods for infrared and visible light images based on autoencoders. However, the current fusion methods based on autoencoders still have the following shortcomings:

[0005] 1) The target in the fused image is not prominent, and details are lost;

[0006] 2) The visual experience of the fused images is poor;

[0007] 3) In complex backgrounds, the texture details of the fused image are easily lost. Summary of the Invention

[0008] Purpose of the invention: The purpose of this invention is to provide a method for fusing infrared and visible light images based on multi-layer convolution, which can better preserve the thermal radiation information in infrared images and the texture details in visible light images under complex backgrounds, and improve the visual effect.

[0009] Technical Solution: This invention relates to an infrared and visible light image fusion method based on multi-layer convolution. The network structure employed includes an encoder network, a decoder network, and a multi-layer convolutional fusion network. The encoder is composed of nested multi-layer convolutional blocks and an ECA attention mechanism. The decoder is mainly composed of decoding blocks, each of which consists of two convolutional layers. The multi-layer convolutional fusion network is mainly composed of gradient convolutional blocks, downsampling convolutional blocks, a convolutional spatial channel attention mechanism, and several convolutional layers. The steps are as follows:

[0010] S1, The registered infrared source image and visible light source image are sent to the encoder in pairs, and the encoder extracts the source image features;

[0011] S2, the source image features are fused by a multi-layer convolutional fusion network to obtain the fused features;

[0012] S3, the decoder reconstructs the fused features and outputs the image.

[0013] Furthermore, the encoder will extract features from infrared source images and visible light source images at four scales;

[0014] In the ECA attention mechanism, C, k, W, and H represent the channel dimension, kernel size, feature map width, and height, respectively; the kernel size is determined by the following formula:

[0015] k = |log2(c) / γ + b / γ| odd

[0016] Among them, || odd This indicates that k can only be an odd number, and b and γ are used to change the ratio between the number of channels and the kernel size, respectively.

[0017] Furthermore, in the multi-layer convolutional fusion network, the downsampling convolutional block is composed of a max pooling layer, a 3×3 convolutional layer, and a convolutional layer with an activation function intersecting each other; after the input image passes through the max pooling layer, the feature information is processed twice by the 3×3 convolution and the convolutional layer with the activation function.

[0018] A convolutional block consisting of a 3×3 convolutional layer and a 3×3 convolutional layer with LReLU activation function is used to directly extract features from the source image information; and the features extracted from the source image information are integrated with the feature information extracted by the gradient convolutional block and the downsampling convolutional block.

[0019] Furthermore, the gradient convolutional block is mainly composed of convolutional layers with LReLU activation, 3×3 convolutional layers, 1×1 convolutional layers, and gradient operators. The main body adopts dense connections, and features are extracted using blocks concatenated from two 3×3 convolutional layers and a 3×3 convolutional layer with LReLU activation. The residual flow uses gradient operations to calculate the gradient magnitude of the features and uses a 1×1 regularized convolutional layer to eliminate channel dimension differences. Finally, the deep features extracted by the main dense flow and the fine-grained detail information obtained by the residual gradient flow are integrated.

[0020] Furthermore, in the decoder, each decoding block consists of two 3×3 convolutional layers; short connections are used to connect them in each row.

[0021] Furthermore, the autoencoder loss function L is adopted. auto To train an autoencoder network, the autoencoder loss function L auto The definition is as follows:

[0022] L auto =L pixel +100L ssim

[0023]

[0024] L ssim = 1 - SSIM(Output, Input)

[0025] In the formula, L pixel L represents the pixel loss between the input and output images. ssim This represents the structural similarity loss between the input and output images; It is the Frobenius norm; SSIM(.) is a structural similarity measure that quantifies the structural similarity between two images.

[0026] Furthermore, a fusion strategy loss function L is adopted. MCFN To train a multi-layer convolutional fusion network, the fusion strategy loss function L MCFN The definition is as follows:

[0027] L MCFN =αL detail +L feature

[0028] L detail =1-SSIM(O,I) vi )

[0029]

[0030] Among them, L detail L featureThese represent the background detail preservation loss function and the target feature enhancement loss function, respectively; α is a tradeoff parameter; m is the number of fusion networks; w1 is a vector of tradeoff parameters used to balance the magnitude of the loss, in order to balance the difference in magnitude at different scales; w vi Control fusion feature map The relative influence of infrared features in the w ir Control fusion feature map The relative influence of visible light characteristics in the light.

[0031] Compared with the prior art, the significant advantages of this invention are as follows:

[0032] 1. This invention introduces the ECA attention mechanism into the encoder and designs CSCA, GCB and DSCB fusion blocks. Based on this, an MCFN fusion network is constructed, which to a certain extent solves the problems of unclear target prominence, loss of texture detail information in complex backgrounds, and poor visual experience in fused images.

[0033] 2. The fusion network designed in this invention can better preserve the thermal radiation information in infrared images and the texture details in visible light images under complex backgrounds. By comparing it with five existing fusion algorithms on two public datasets in terms of subjective and objective aspects, the experimental results show that: in terms of objective aspects, the image evaluation index of the fused image is significantly improved; in terms of subjective aspects, this invention has certain advantages in highlighting target information, preserving texture details in complex backgrounds, and improving visual effects. Attached Figure Description

[0034] Figure 1 This is a schematic diagram of the overall network structure of the present invention;

[0035] Figure 2 A detailed structure diagram of the encoding and decoding;

[0036] Figure 3 Diagram of the ECA attention mechanism structure;

[0037] Figure 4 Here is a structural diagram of MCFN;

[0038] Figure 5 Here is a structural diagram of CB;

[0039] Figure 6 Here is a structural diagram of CSCA;

[0040] Figure 7 Graphs showing the results for different α values;

[0041] Figure 8 Images of helicopters fused using different algorithms;

[0042] Figure 9Images showing the fusion results of soldiers under different algorithms;

[0043] Figure 10 The images show the fusion results of different algorithms applied to the road.

[0044] Figure 11 Images showing the fusion results of the tent and the person under different algorithms;

[0045] Figure 12 The images show the fusion results of streets under different algorithms. Detailed Implementation

[0046] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0047] This invention proposes an infrared and visible light image fusion algorithm based on multi-layer convolution. In the encoder stage, an efficient channel attention (ECA) mechanism is introduced to improve the quality of the fused image. In the multi-layer convolutional fusion network (hereinafter referred to as the "fusion network") MCFN, gradient convolution blocks (GCBs), downsampling convolution blocks (DSCBs), and convolutional spatial channel attention (CSCA) are designed. The fusion network, containing numerous blocks, can better preserve the texture details of the image against complex backgrounds and highlight infrared targets. Finally, the decoder performs decoding and reconstruction output.

[0048] (I) Network Structure Design

[0049] like Figure 1 As shown, the overall network structure of this invention consists of an encoding network, a decoding network, and a fusion network. First, the registered infrared and visible light images are fed into the encoder in pairs, where the encoder extracts the features of the source images. Then, the fusion network MCFN fuses the features of the source images, and finally, the decoder reconstructs and outputs the results.

[0050] This invention employs a two-stage training method. The first stage trains the encoder, fusion network, and decoder as a whole. The second stage directly uses the parameters and weights from the first stage to train the fusion network MCFN separately. To better preserve the detail and background information of the fused image and improve its visual effect, the fusion network MCFN was designed. Ablation experiments demonstrate that the designed MCFN significantly improves the objective evaluation metrics and visual effect of the fused image.

[0051] The detailed structure of encoding and decoding is as follows: Figure 2 As shown, the encoder is on the left, I vis and I ir The input consists of a visible light source image and an infrared source image. The encoder is composed of nested convolutional blocks and an ECA attention mechanism. In the figure, labels similar to "(in out)" on the convolutional layers indicate that the input channel is "in" and the output channel is "out". For example, (168) indicates that the input channel is 16 and the output channel is 8. The encoder will extract features from the infrared source image and the visible light source image at four scales, and the extracted feature information will be fused through a fusion network.

[0052] The detailed structure of the decoder is as follows: Figure 2 As shown on the right, the network is mainly composed of decoder blocks (DBs), each consisting of two 3×3 convolutional layers. Within each row, these blocks are connected by short connections similar to a dense block architecture. Furthermore, the decoder structure employs cross-layer connections to preserve more multi-scale deep features and detailed information from the source image. The network output is a fused map reconstructed from multi-scale features.

[0053] The detailed structure of the ECA attention mechanism is as follows: Figure 3 As shown, a lightweight ECA attention mechanism was introduced into the encoder. Ablation experiments showed that this attention mechanism has a positive effect on improving the fusion image metrics.

[0054] The structural diagram of the ECA attention mechanism is as follows: Figure 3 As shown in the figure, C, k, W, and H represent the channel dimension, the kernel size, the width of the feature map, and the height, respectively. The kernel size is determined by the following formula:

[0055] k = |log2(c) / γ + b / γ| odd

[0056] In the formula, || odd This indicates that k can only be an odd number; b and γ are set to 2 and 1 respectively to change the ratio between the number of channels and the kernel size.

[0057] The structure of the converged network MCFN is as follows Figure 4As shown, MCFN mainly consists of a gradient convolutional block (GCB), a downsampling convolutional block (DSCB), a convolutional spatial channel attention mechanism (CSCA), and several convolutional layers. The downsampling convolutional block (DSCB) is composed of interleaved max-pooling layers, 3×3 convolutional layers, and convolutional layers with activation functions. This design reduces the large computational cost caused by the gradient convolutional block (GCB), and the max-pooling layer better preserves detailed texture information. After passing through the max-pooling layer, the feature information is processed twice, through 3×3 convolutions and convolutional layers with activation functions, further enhancing detailed features and preserving texture information.

[0058] In addition to the main body of the fusion network MCFN, convolutional blocks consisting of 3×3 convolutional layers and 3×3 convolutional layers with LReLU activation function are used to directly extract features from the source image information. The features extracted from the source image information are then integrated with the features extracted by the gradient convolutional block (GCB) and the downsampling convolutional block (DSCB). This operation can retain more source information and enrich the information content of the fused image.

[0059] A detailed structural diagram of the gradient convolution block (GCB) is shown below. Figure 5 As shown, the Gradient Convolutional Block (GCB) mainly consists of convolutional layers with LReLU activation, 3×3 convolutional layers, 1×1 convolutional layers, and gradient operators. LReLU is a non-saturating activation function; its application can solve the gradient vanishing problem, while also accelerating convergence and improving efficiency. Therefore, this operation is first applied to the feature information to extract shallow information from the feature map. The main body of the GCB uses dense connections, employing a block of two 3×3 convolutional layers and a 3×3 convolutional layer with LReLU activation for feature extraction. Introducing dense connections into the main body allows for full utilization of the features extracted by each convolutional layer. In addition, the residual flow uses gradient operations to calculate the gradient magnitude of the features and uses a 1×1 regularized convolutional layer to eliminate channel dimension differences. Finally, the deep features extracted by the main dense flow and the fine-grained detail information obtained by the residual gradient flow are integrated.

[0060] Detailed structure diagram of the downsampled convolutional block CSCA is shown below. Figure 6 As shown, the attention mechanism included in this network structure can reduce attention to channel- and spatially irrelevant information, focusing on high-value information, solving the information overload problem, and improving the efficiency and accuracy of task processing. Applying it simultaneously with convolutional blocks can better capture detailed information of the target regions that require focused attention, achieving the goal of preserving more detailed information. Through ablation experiments, this module has significant implications for improving the quality of fused images.

[0061] The loss function in this embodiment is divided into an autoencoder loss function and a fusion strategy loss function.

[0062] (A) Autoencoder loss function

[0063] This embodiment uses a loss function, L, to train the autoencoder network. auto The definition is as follows:

[0064] L auto =L pixel +100L ssim (1)

[0065] In the formula, L pixel L represents the pixel loss between the input and output images. ssim This represents the structural similarity loss between the input and output images.

[0066] L pixel The loss is calculated as follows:

[0067]

[0068] In the formula, It is the Frobenius norm.

[0069] The pixel-level similarity between the output image and the input image is affected by L. pixel Constraints. L ssim The loss is calculated by equation (3).

[0070] L ssim =1-SSIM(Output,Input) (3)

[0071] In the formula, SSIM(.) is a structural similarity measure that quantifies the structural similarity between two images.

[0072] (B) Loss Function of Fusion Strategy

[0073] The training of the fusion network MCFN aims to implement a fully learnable fusion strategy. In the second stage, with the encoder and decoder fixed, the fusion network MCFN is trained with an appropriate loss function. To better train the fusion network MCFN, this embodiment uses a loss function L. MCFN Its definition is as follows:

[0074] L MCFN =αL detail +L feature (4)

[0075] Among them, L detail L featureThese represent the background detail preservation loss function and the target feature enhancement loss function, respectively. α is a trade-off parameter, experimentally set to 700. Since most of the background detail information in the fused image comes from the visible light image... detail The purpose is to preserve the detailed information and structural features in visible light images, which is defined as:

[0076] L detail =1-SSIM(O,I) vi (5)

[0077] Because infrared images contain more significant target features than visible light images, the loss function L feature Designed to limit the fusion of deep features in order to preserve salient target features. L feature Defined as:

[0078]

[0079] In equation (6), m is the number of fusion networks, set to M = 4. w1 is a weighted parameter vector used to balance the magnitude of the loss, set to {1, 10, 100, 1000} to balance the difference in magnitude at different scales. vi and w ir Control fusion feature map infrared characteristics and visible light characteristics The relative impacts were set to 6.0 and 3.0, respectively.

[0080] (II) Experimental Results and Analysis

[0081] This implementation describes the experimental setup for the training and testing phases, and then experimentally sets the α value in the loss function. Ablation experiments were conducted on the attention mechanism and the MCFN fusion network, fully demonstrating the effectiveness of the invention. Finally, this invention is compared with five other algorithms published in recent years, including: an end-to-end residual fusion network for infrared and visible light images (RFN-Nest), a generative adversarial network with multi-classification constraints for infrared and visible light image fusion (GANMCC), a fusion adversarial generative network (FusionGAN), a deep image decomposition for infrared and visible light image fusion (DIDFuse), and an unsupervised misaligned infrared and visible light image fusion based on cross-modal image generation and registration (UMF-CMGR). Eight metrics were used to objectively evaluate the quality of the fused image: information entropy (EN), spatial frequency (SF), average gradient (AG), standard deviation (SD), correlation coefficient (CC), sum of differential correlations (SCD), visual fidelity (VIF), and peak signal-to-noise ratio (PSNR).

[0082] (21) Experimental setup

[0083] The TNO dataset (TOET A. The TNO Multiband Image Data Collection[J]. Data in Brief, 2017, 15. 249-251.) contains rich military scenes, such as helicopters, houses, tanks, people, forests, and vehicles. Therefore, the TNO dataset perfectly meets the dataset requirements in the field of infrared and visible light image fusion and is the most authoritative dataset in this research field. 45,000 pairs of infrared and visible light images were obtained as the training set by expanding the dataset. The MSRS dataset, a commonly used dataset in the field of infrared and visible light image fusion, contains 1,444 pairs of high-quality aligned infrared and visible light images, including daytime and nighttime image pairs. To make the test results more authoritative, 42 pairs of images were selected from the TNO dataset as the TNO test set, and 10 pairs of images were selected from the MSRS dataset as the MSRS test set. The results were then averaged. Algorithm validation was conducted in an experimental environment built on Windows 10, trained on an NVIDIA RTX 3080 GPU, with an initial learning rate of 0.0001, batch size, and epochs both set to 4.

[0084] (22) Parameter setting experiment

[0085] This embodiment will analyze the experimental results of different α values ​​from both subjective and objective perspectives through experimental methods. Figure 7 The fusion results for different α values ​​are shown in the diagram.

[0086] from Figure 7 The results clearly show that the α value has a direct impact on the experimental results. When α is too large or too small, the infrared information of the fused image will be lost or even disappear. Figure 7 By comparing the details in the block diagram, it can be found that texture details are well preserved when α is set to 600, 700, and 800. To further determine the α value, Table 1 shows the index results of different α values ​​on the TNO test set.

[0087] Table 1 Results for different α values

[0088]

[0089] In Table 1, the top three values ​​are marked in bold. It can be clearly observed that when α = 700, the number of values ​​in the top three is 6, and when α = 800, the number of values ​​in the top three is 5. Therefore, in the experiment, α was set to 700.

[0090] (23) Ablation test

[0091] To verify the effectiveness of the ECA attention mechanism and modules introduced in this invention, ablation experiments were conducted on the ECA attention mechanism and the CSCA, GCB, and DSCB modules, fully demonstrating the effectiveness of the network designed in this invention. The ablation experiment results are shown in Table 2.

[0092] Table 2 Ablation Experiment Results

[0093]

[0094] Table 2 highlights the best-performing values ​​in bold, demonstrating that the CSCA block significantly improves all eight metrics. The ECA attention mechanism plays a crucial role in improving EN, SF, AG, and SD. While the GCB block has a significant negative impact on SD, its substantial improvement on other metrics cannot be ignored. Overall, the fusion block designed in this invention and the attention mechanism it employs are of great importance to the quality of the fused image.

[0095] (24) Comparative Experiment

[0096] To demonstrate the effectiveness of this invention, it is compared with five disclosed algorithms from both objective and subjective perspectives.

[0097] (241) Subjective evaluation

[0098] like Figure 8 As shown, a helicopter image from the TNO dataset was selected, and the fusion results were obtained using different algorithms. In the image, it is clear that the infrared image has no background information, but the outline is distinct, the target is prominent, and the propeller features are well displayed. The visible light image retains background texture information better, and the overall image visibility is also good, but the helicopter's detailed information is severely lost.

[0099] exist Figure 8The image fusion results of this invention and five existing algorithms are presented. The GANMCC algorithm results in a fusion image with severe loss of background texture information, barely visible tail rotor, and indistinct landing gear. The DIDFuse algorithm results in a fusion image with severe loss of tail and upper rotor information, while maintaining relatively good background texture details, but deviating from the background information in the visible light image. The FusionGAN algorithm results in a fusion image with good detail retention, but ghosting occurs in various parts of the helicopter, resulting in an overall blurry appearance. The UMF-CMGR algorithm results in a fusion image with good overall detail, but the landing gear is relatively unclear, some background information is lost, and the overall image is dark. The RFN-Nest algorithm results in a fusion image with almost complete loss of upper rotor information, only a rudimentary tail rotor shape, more visible light information and less infrared information in the landing gear, resulting in poor visual perception of the landing gear and severe loss of background texture information. The fusion result of this invention retains background texture details well, almost perfectly matching the background information of the visible light image. Details of the landing gear, tail rotor, and upper propeller are clearly visible, and the entire image is free of ghosting, resulting in a superior visual effect. Therefore, from a subjective perspective, this invention is superior to other algorithms in terms of target information preservation, background texture details, and overall image visual effect.

[0100] like Figure 9 As shown, a fusion image of soldiers in a jungle from the TNO dataset was selected, and the results were obtained using different algorithms. Among them, the fused images obtained using GANMCC, FusionGAN, and RFN-Nest algorithms showed significant loss of background texture details. The fused images obtained using DIDFuse and UMF-CMGR algorithms did not effectively highlight the soldier's target information, resulting in unclear character outlines. The fused image obtained by this invention retains the background texture information well, and the character outlines are also relatively clear.

[0101] To better illustrate the universality of this invention, a representative fusion result image is selected from the MSRS dataset, as shown below. Figure 10 As shown, this invention has certain advantages compared to the other five algorithms.

[0102] Figure 11 , Figure 12 Two more result images were selected from the TNO dataset for presentation. Subjective comparison reveals that this invention has certain advantages in highlighting target information, preserving details, and improving visual appeal.

[0103] (242) Objective evaluation

[0104] To better verify the effectiveness of this invention, five representative algorithms and eight objective evaluation metrics were selected for objective evaluation. For all eight selected evaluation metrics, higher values ​​generally indicate better image quality. To ensure fairness and reliability, the average results of the tests on 42 images from the TNO test set and 10 images from the MSRS test set were compared. This method, to some extent, eliminates subjective human factors, resulting in more objective evaluation results.

[0105] Table 3 Evaluation of the fusion effect of the TNO dataset

[0106]

[0107] To more clearly analyze the evaluation index data in Table 3, the top-ranked values ​​are highlighted in bold. Of the eight evaluation indicators used, this invention ranked first in four and second in three. The invention consistently ranked above the mean across all indicators, thus demonstrating a certain objective superiority over other algorithms on the TNO dataset. Specifically, SF improved by 25.9% compared to the second-place algorithm, and AG improved by 38.6%.

[0108] Table 4 Evaluation of the fusion effect of the MSRS dataset

[0109]

[0110] In Table 4, the top-ranked value is highlighted in bold. It is clear that this invention performs best in 6 out of the 8 evaluation metrics on the MSRS dataset. Therefore, this invention demonstrates a certain objective advantage over other algorithms on the MSRS dataset. Specifically, the SD score is 7.4% higher than the second-best performing algorithm. Overall, in terms of objective performance, this invention shows significant improvements over other algorithms on both the TNO and MSRS datasets, thus demonstrating good objective performance.

Claims

1. A method for fusing infrared and visible light images based on multi-layer convolution, characterized in that, The network structure includes an encoder, a decoder, and a multi-layer convolutional fusion network. The encoder consists of nested multi-layer convolutional blocks and an ECA attention mechanism. The decoder consists of decoding blocks, each of which comprises two convolutional layers. The multi-layer convolutional fusion network consists of gradient convolutional blocks, downsampling convolutional blocks, a convolutional spatial channel attention mechanism, and several convolutional layers. In the multi-layer convolutional fusion network, the downsampling convolutional blocks are formed by interleaving max pooling layers, 3×3 convolutional layers, and convolutional layers with activation functions. After the input image passes through the max pooling layer, the feature information is processed twice, through 3×3 convolutions and convolutional layers with activation functions. use Convolutional layers and those with LReLU activation function The convolutional blocks, composed of convolutional layers, directly extract features from the source image information; and integrate the features extracted from the source image information with the feature information extracted by the gradient convolutional blocks and downsampling convolutional blocks; The gradient convolutional block consists of convolutional layers with LReLU activation functions. Convolutional layers It consists of convolutional layers and gradient operators, with the main body employing dense connections and using two... Convolutional layers and those with LReLU activation function Feature extraction is performed on concatenated blocks of convolutional layers; the residual flow uses gradient operations to calculate the gradient magnitude of the features, and uses... Regularized convolutional layers are used to eliminate channel dimension differences; finally, the deep features extracted from the main dense flow and the fine-grained details obtained from the residual gradient flow are integrated. The steps include the following: S1, The registered infrared source image and visible light source image are sent to the encoder in pairs, and the encoder extracts the source image features; S2, the source image features are fused by a multi-layer convolutional fusion network to obtain the fused features; S3, the decoder reconstructs the fused features and outputs the image.

2. The infrared and visible light image fusion method based on multi-layer convolution according to claim 1, characterized in that, The encoder will extract features from infrared source images and visible light source images at four scales; C in the ECA attention mechanism W and H represent the channel dimension, kernel size, feature map width, and height, respectively; the kernel size is determined by the following formula: in, express Only odd numbers can be selected. and Set to 2 and 1.

3. The infrared and visible light image fusion method based on multi-layer convolution according to claim 1, characterized in that, In the decoder, each decoding block consists of two 3×3 convolutional layers; short connections are used to connect them in each row.

4. The infrared and visible light image fusion method based on multi-layer convolution according to claim 1, characterized in that, Using the autoencoder loss function To train an autoencoder network, the autoencoder loss function. The definition is as follows: In the formula, This represents the pixel loss between the input and output images. This represents the structural similarity loss between the input and output images; It is the Frobenius norm; This is a structural similarity measure that quantifies the structural similarity between two images.

5. The infrared and visible light image fusion method based on multi-layer convolution according to claim 1, characterized in that, Using a fusion strategy loss function To train a multi-layer convolutional fusion network, the fusion strategy loss function is... The definition is as follows: in, , These represent the background detail preservation loss function and the target feature enhancement loss function, respectively. It is a trade-off parameter; m is the number of fusion networks; It is a vector of tradeoff parameters used to balance the magnitude of loss, in order to balance the differences in magnitude at different scales; Control fusion feature map The relative influence of infrared features in the data. Control fusion feature map The relative influence of visible light characteristics in the light.

Images