Visible and thermal infrared fusion method based on double-layer feature extraction network

Abstract

The application discloses a visible light and thermal infrared fusion method based on a double-layer feature extraction network. The method comprises the following steps: collecting visible light and infrared images under different scenes and constructing the images into an image training set; constructing a double-layer feature extraction network, including a double-layer feature encoding module, a double-layer feature fusion module and a feature decoding module; inputting the image training set into the network for training to obtain a trained network; collecting visible light and infrared images to be fused under the same scene, converting the visible light image into a visible light grayscale image, inputting the visible light grayscale image and the infrared image into the network, and outputting a fused grayscale image after processing; performing color space conversion on the fused grayscale image to obtain a final color fusion image, and realizing the final fusion. The application trains the feature encoding and decoding modules in a self-supervised manner, designs a self-supervised loss function to effectively extract global brightness features and local detail features of the image, and guarantees the stability of network feature extraction and image reconstruction.

Description

Technical Field

[0001] This invention relates to a visible light and thermal infrared fusion method, which falls under the field of image processing technology, and specifically to a visible light and thermal infrared fusion method based on a two-layer feature extraction network. Background Technology

[0002] Visible light images can capture brightness and structural information of a scene well, but their image quality is poor in low-visibility scenes such as at night. In contrast, thermal infrared images are more sensitive to environmental temperature information and possess observational capabilities that visible light images lack, such as night vision, fog penetration, and smoke penetration. They also provide clear imaging of objects such as animals, people, and vehicles in a scene, making them an important data source for multispectral sensing. Therefore, multimodal image sensors are often used to capture cross-modal image information and fuse visible light and thermal infrared images, combining the advantages of both to obtain clearer and more sensory images.

[0003] Existing image fusion algorithms mainly include traditional algorithms and deep learning algorithms. Traditional image fusion algorithms include detail enhancement methods and gradient fusion methods, while deep learning-based fusion algorithms include DenseFuse and SwinFuse.

[0004] Detail enhancement methods extract additional structural information from infrared images using structural operators and then superimpose it onto visible light images. However, this approach only considers the complementarity of structural information and ignores the naturalness of brightness, resulting in a fused image that lacks naturalness.

[0005] Gradient fusion transforms the fusion problem into an optimization problem by transferring the gradient components of the source images into the fused image, achieving image fusion in the gradient domain. This method yields fusion results with high brightness fidelity for infrared images, but suffers significant loss of detail in visible light, making it less pleasing to the human eye.

[0006] DenseFuse uses dense autoencoders for parameter training to obtain better image feature encoders and decoders, and performs image fusion in the feature domain. However, its fusion strategy is relatively simple and does not effectively combine the complementary advantages of deep features from the source images.

[0007] SwinFuse uses the SwinTransformer structure to construct an autoencoder for images, which can effectively obtain global information about the image and uses a Softmax weighted fusion method to fuse the image information. However, it does not consider the fusion of local detail features of the image. Summary of the Invention

[0008] To address the problems existing in the background technology, the present invention provides a visible light and thermal infrared fusion method based on a two-layer feature extraction network.

[0009] The technical solution adopted in this invention is:

[0010] The visible light and thermal infrared fusion method based on a two-layer feature extraction network of the present invention includes:

[0011] 1) Collect several visible light and infrared images from different scenes and construct them into an image training set.

[0012] 2) Construct a two-layer feature extraction network, which includes a two-layer feature encoding module, a two-layer feature fusion module, and a feature decoding module connected in sequence; input the image training set into the two-layer feature extraction network for training to obtain the trained two-layer feature extraction network.

[0013] 3) Acquire visible light and infrared images to be fused from the same scene. Convert the visible light image to the L channel of the LAB color space to obtain a visible light grayscale image. Input the visible light grayscale image and the infrared image into a trained two-layer feature extraction network for processing. After processing, the trained two-layer feature extraction network outputs a fused grayscale image. Convert the fused grayscale image to the color space using the AB chromaticity channels of the original visible light image to obtain the final color fused image, thus achieving the final fusion of visible light and thermal infrared images.

[0014] In step 2), the dual-layer feature encoding module includes a local detail feature encoder and a global brightness feature encoder. The input image of the dual-layer feature encoding module is input into the local detail feature encoder and the global brightness feature encoder respectively, and after processing, the local detail features and global brightness features are output, and then jointly output to the dual-layer feature fusion module.

[0015] The local detail encoder comprises six fully convolutional residual layers connected in sequence. The input of the global brightness feature encoder is first processed in the first fully convolutional residual layer. The processed output is added to the input of the local detail encoder to obtain a first summed result. The first summed result is then processed in the second fully convolutional residual layer. The processed output is added to the first summed result to obtain a second summed result. The second summed result is then processed in the third fully convolutional residual layer. The processed output is added to the second summed result to obtain a third summed result. The third summed result is then processed in the fourth fully convolutional residual layer. The processed output is added to the third summed result to obtain a fourth summed result. The fourth summed result is then processed in the fifth fully convolutional residual layer. The processed output is added to the fourth summed result to obtain a fifth summed result. The fifth summed result is then processed in the sixth fully convolutional residual layer. The processed output is added to the fifth summed result to output the local detail feature. The local detail encoder is used to extract the structural features of the image while preserving the original high-frequency details of the image.

[0016] The global brightness feature encoder includes four convolutional layers and two autocorrelation coding layers connected in sequence. The input of the global brightness feature encoder is processed by the four convolutional layers and the two autocorrelation coding layers in sequence to output global brightness features. The global brightness feature encoder is used to establish the global correlation matrix of the image and perform autocorrelation enhancement on the global brightness features.

[0017] In step 2), the dual-layer feature fusion module uses local luminance entropy to fuse global luminance features and uses absolute value normalization weights to perform weighted fusion of local detail features.

[0018] In step 2), the feature encoding module includes four densely connected convolutional layers. The feature encoder first concatenates the input local detail features and global brightness features to obtain a first concatenation result. The first concatenation result is then input into the first convolutional layer for processing. The processed output is then concatenated with the first concatenation result to obtain a second concatenation result. The second concatenation result is then input into the second convolutional layer for processing. The processed output is then concatenated with the first and second concatenation results to obtain a third concatenation result. The third concatenation result is then input into the third convolutional layer for processing. The processed output is then concatenated with the first, second, and third concatenation results to obtain a fourth concatenation result. The fourth concatenation result is then input into the fourth convolutional layer for processing. The processed output is then concatenated with the first, second, third, and fourth concatenation results to finally decode and obtain the reconstructed image.

[0019] In step 2), when the image training set is input into the dual-layer feature extraction network for training, for each infrared image and visible light image in the image training set, a single infrared image or visible light image is input into the dual-layer feature encoding module of the dual-layer feature extraction network for processing. After processing, the global brightness features and local detail features of the single infrared image or visible light image are output. The global brightness features and local detail features of the single infrared image or visible light image are then input into the feature decoding module for processing. After processing, the reconstructed image is output.

[0020] During training, a self-supervised training method is adopted to train learnable parameters for the two-layer feature encoding and feature decoding modules of the two-layer feature extraction network. Specifically, the Adam trainer is used to optimize and iteratively update the network parameters of the two-layer feature extraction network. When the preset number of iterations is reached, the parameter update is stopped to obtain the optimal network parameters of the two-layer feature extraction network. The self-supervised loss function of the two-layer feature extraction network is then calculated, as follows:

[0021] L=λ1L pixel +λ2L SSIM +λ3L Feat

[0022] Where L represents the self-supervised loss value of the two-layer feature extraction network; λ1, λ2, and λ3 represent the first, second, and third weighting parameters of the two-layer feature extraction network, respectively. Preferably, λ1 = 1, λ2 = 10, and λ3 = 0.1. pixel L represents the pixel loss value between the input original image and the output reconstructed image of the two-layer feature extraction network. SSIM L represents the structural similarity loss value between the input original image and the output reconstructed image of the two-layer feature extraction network. Feat This represents the feature attribute loss value between the input original image and the output reconstructed image of the two-layer feature extraction network.

[0023] Finally, the self-supervised loss value of the two-layer feature extraction network gradually decreases and stabilizes, completing the self-supervised training and obtaining the trained two-layer feature extraction network.

[0024] The self-supervised loss function achieves two functions: first, it constrains the reconstructed image content to be as consistent as possible with the original image content and structure, ensuring high quality and stability of image reconstruction; second, it guarantees the extraction capabilities of the encoded global brightness features and local detail features, respectively. Therefore, the network loss function mainly includes two aspects: reconstruction loss and feature property loss.

[0025] This invention utilizes pixel loss to constrain the brightness consistency between the reconstructed image and the input image; it uses structural similarity loss to ensure the structural consistency between the reconstructed image and the input image, specifically using the Structure Similarity Index Measurement (SSIM) to calculate the structural similarity loss between the two images; and it uses feature attribute loss for constraint, so that the encoded global brightness features and local detail features can guarantee the smoothness of the global brightness features and the richness of the local detail features.

[0026] The feature attribute loss value L between the input original image and the output reconstructed image of the two-layer feature extraction network. Feat Specifically as follows:

[0027]

[0028] in, F represents the Laplace operator; D Represents local details; F B ε represents the global brightness characteristic; ε represents a minimal constant.

[0029] In step 3), the visible light grayscale image and the infrared image are jointly input into the trained two-layer feature extraction network for processing. First, the visible light grayscale image and the infrared image are input into the two-layer feature encoding module for processing. After processing, the visible light local detail features and visible light global brightness features of the visible light grayscale image, as well as the infrared local detail features and infrared global brightness features of the infrared image, are output to the two-layer feature fusion module. The two-layer feature fusion module performs weighted fusion of the visible light local detail features, infrared local detail features, and visible light global brightness features, respectively. The visible light local detail features are then weighted using absolute value Softmax to calculate the fusion weight and then combined with their own weights. The first dot product result is obtained by multiplying the infrared local detail features using absolute value Softmax to calculate the fusion weights and then multiplying them with the features themselves to obtain the second dot product result. The first and second dot product results are added together to obtain the fused local detail features. The third dot product result is obtained by multiplying the visible light global brightness features using local range to calculate the fusion weights and then multiplying them with the features themselves. The fourth dot product result is obtained by multiplying the infrared global brightness features using local range to calculate the fusion weights and then multiplying them with the features themselves. The third and fourth dot product results are added together to obtain the fused global brightness features. The feature decoding module performs feature decoding on the fused local detail features and the fused global brightness features to obtain the final reconstructed fused grayscale image.

[0030] The beneficial effects of this invention are:

[0031] The local detail encoder in the dual-layer feature encoding module of the network of the present invention is used to extract the structural features of the image, which can preserve the original high-frequency details of the image. The global brightness feature encoder is used to establish the global correlation matrix of the image, which can perform autocorrelation enhancement on the global brightness features.

[0032] The two-layer feature fusion module in the network of the present invention employs different fusion strategies for global brightness features and local detail features, which can effectively combine the complementary advantages of the depth features of the source image.

[0033] The network self-supervised loss function of the present invention can achieve two functions: first, it constrains the content and structure of the reconstructed image to be as consistent as possible with the content and structure of the original image, so as to ensure the high quality and stability of the image reconstruction; second, it ensures the extraction capabilities of the global brightness features and local detail features generated by encoding, respectively.

[0034] This invention utilizes a self-supervised approach to train the feature encoding and decoding modules, and designs a self-supervised loss function to effectively extract global brightness features and local detail features of the image, ensuring the stability of network feature extraction and image reconstruction. Attached Figure Description

[0035] Figure 1 This is a flowchart illustrating the training and testing phases of the method of the present invention;

[0036] Figure 2 This is a diagram of the overall network structure of the present invention;

[0037] Figure 3 This is a schematic diagram of a two-layer feature coding module;

[0038] Figure 4 This is a schematic diagram of a two-layer feature fusion module;

[0039] Figure 5 This is a schematic diagram of the feature decoder module;

[0040] Figure 6 This is a comparison diagram between the method of the present invention and existing advanced visible light-thermal infrared image fusion algorithms. Detailed Implementation

[0041] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0042] The visible light and thermal infrared fusion method based on a two-layer feature extraction network of the present invention includes:

[0043] 1) Collect several visible light and infrared images from different scenes and construct them into an image training set.

[0044] 2) Construct a two-layer feature extraction network, which includes a two-layer feature encoding module, a two-layer feature fusion module, and a feature decoding module connected in sequence; input the image training set into the two-layer feature extraction network for training to obtain the trained two-layer feature extraction network.

[0045] like Figure 3 As shown in step 2), the dual-layer feature encoding module includes a local detail feature encoder and a global brightness feature encoder. The input image of the dual-layer feature encoding module is input into the local detail feature encoder and the global brightness feature encoder respectively, and after processing, the local detail features and global brightness features are output, and then jointly output to the dual-layer feature fusion module.

[0046] The local detail encoder comprises six sequentially connected fully convolutional residual layers. The input to the global brightness feature encoder is first processed in the first fully convolutional residual layer. The processed output is added to the input of the local detail encoder to obtain the first summed result. The first summed result is then processed in the second fully convolutional residual layer. The processed output of the second summed result is added to the first summed result to obtain the second summed result. The second summed result is then processed in the third fully convolutional residual layer. The processed output of the third summed result is added to the second summed result to obtain the third summed result. The third summed result is then processed in the fourth fully convolutional residual layer. The processed output of the fourth summed result is added to the third summed result to obtain the fourth summed result. The fourth summed result is then processed in the fifth fully convolutional residual layer. The processed output of the fifth summed result is added to the sixth fully convolutional residual layer to obtain the local detail feature. The local detail encoder is used to extract the structural features of the image while preserving the original high-frequency details of the image.

[0047] In the two-layer feature encoding module, the global brightness feature encoder includes four convolutional layers and two autocorrelation coding layers connected in sequence. The input of the global brightness feature encoder is processed by the four convolutional layers and the two autocorrelation coding layers in sequence and outputs global brightness features. The global brightness feature encoder is used to establish the global correlation matrix of the image and perform autocorrelation enhancement on the global brightness features.

[0048] For the local detail feature encoder, six residual convolutional structures are used to aggregate local structural information, and the residual connection method ensures that high-frequency details are not lost. For the global brightness feature encoder, four convolutional structures are first used to expand the feature channels, and then two self-attention coding layers are used to enhance the global correlation of the feature space to obtain global image information.

[0049] like Figure 4As shown, in step 2), the dual-layer feature fusion module uses local brightness entropy to fuse global brightness features and uses absolute value normalization weights to perform weighted fusion of local detail features.

[0050] like Figure 5 As shown, in step 2), the feature encoding module includes four densely connected convolutional layers. The feature encoder first concatenates the input local detail features and global brightness features to obtain a first concatenation result. The first concatenation result is then input into the first convolutional layer for processing. The processed output is then concatenated with the first concatenation result to obtain a second concatenation result. The second concatenation result is then input into the second convolutional layer for processing. The processed output is then concatenated with the first and second concatenation results to obtain a third concatenation result. The third concatenation result is then input into the third convolutional layer for processing. The processed output is then concatenated with the first, second, and third concatenation results to obtain a fourth concatenation result. The fourth concatenation result is then input into the fourth convolutional layer for processing. The processed output is then concatenated with the first, second, third, and fourth concatenation results to finally decode and obtain the reconstructed image.

[0051] In the feature encoding module, local detail features and global brightness features are first concatenated and then input into a densely connected four-layer convolutional layer. The dense connection method ensures that feature information is not lost during transmission, so as to reconstruct the image with high accuracy.

[0052] like Figure 1 As shown in step 2), when the image training set is input into the dual-layer feature extraction network for training, for each infrared image and visible light image in the image training set, a single infrared image or visible light image is input into the dual-layer feature encoding module of the dual-layer feature extraction network for processing. After processing, the global brightness features and local detail features of the single infrared image or visible light image are output. The global brightness features and local detail features of the single infrared image or visible light image are then input into the feature decoding module for processing. After processing, the reconstructed image is output.

[0053] During training, a self-supervised training method is adopted to train learnable parameters for the two-layer feature encoding and feature decoding modules of the two-layer feature extraction network. Specifically, the Adam trainer is used to optimize and iteratively update the network parameters of the two-layer feature extraction network. When the preset number of iterations is reached, the parameter update is stopped to obtain the optimal network parameters of the two-layer feature extraction network. The self-supervised loss function of the two-layer feature extraction network is then calculated, as follows:

[0054] L=λ1L pixel +λ2L SSIM +λ3L Feat

[0055] Where L represents the self-supervised loss value of the two-layer feature extraction network; λ1, λ2, and λ3 represent the first, second, and third weighting parameters of the two-layer feature extraction network, respectively. Preferably, λ1 = 1, λ2 = 10, and λ3 = 0.1. pixel L represents the pixel loss value between the input original image and the output reconstructed image of the two-layer feature extraction network. SSIM L represents the structural similarity loss value between the input original image and the output reconstructed image of the two-layer feature extraction network. Feat This represents the feature attribute loss value between the input original image and the output reconstructed image of the two-layer feature extraction network.

[0056] Finally, the self-supervised loss value of the two-layer feature extraction network gradually decreases and stabilizes, completing the self-supervised training and obtaining the trained two-layer feature extraction network.

[0057] The self-supervised loss function achieves two functions: first, it constrains the reconstructed image content to be as consistent as possible with the original image content and structure, ensuring high quality and stability of image reconstruction; second, it guarantees the extraction capabilities of the encoded global brightness features and local detail features, respectively. Therefore, the network loss function mainly includes two aspects: reconstruction loss and feature property loss.

[0058] This invention utilizes pixel loss to constrain the brightness consistency between the reconstructed image and the input image; it uses structural similarity loss to ensure the structural consistency between the reconstructed image and the input image, specifically using the structural similarity metric SSIM to calculate the structural similarity loss between the two images; and it uses feature attribute loss for constraint, so that the encoded global brightness features and local detail features can guarantee the smoothness of the global brightness features and the richness of the local detail features.

[0059] The feature attribute loss value L between the input original image and the output reconstructed image of the two-layer feature extraction network Feat Specifically as follows:

[0060]

[0061] in, F represents the Laplace operator; D Represents local details; F B ε represents the global brightness characteristic; ε represents a minimal constant.

[0062] 3) Acquire visible light and infrared images to be fused from the same scene. Convert the visible light image to the L channel of the LAB color space to obtain a visible light grayscale image. Input both the visible light grayscale image and the infrared image into a trained two-layer feature extraction network for processing. The trained two-layer feature extraction network outputs a fused grayscale image. Then, perform color space conversion on the fused grayscale image using the AB chroma channels from the original visible light image to obtain the final color fused image, thus achieving the final fusion of visible light and thermal infrared images. Figure 2 As shown.

[0063] like Figure 1 As shown, in specific implementation, the same operation as in step 3) is performed on each visible light image and infrared image of the same scene in the image training set in step 1).

[0064] In step 3), the visible light grayscale image and the infrared image are jointly input into the trained two-layer feature extraction network for processing. First, the visible light grayscale image and the infrared image are input into the two-layer feature encoding module for processing. After processing, the visible light local detail features and visible light global brightness features of the visible light grayscale image, as well as the infrared local detail features and infrared global brightness features of the infrared image, are output to the two-layer feature fusion module. The two-layer feature fusion module performs weighted fusion of the visible light local detail features and infrared local detail features, as well as the visible light global brightness features and infrared global brightness features, respectively. The visible light local detail features are then weighted and fused. The first dot product result is obtained by multiplying the fusion weights calculated using Softmax on the absolute value of the infrared local detail features by itself. The second dot product result is obtained by multiplying the fusion weights calculated using Softmax on the absolute value of the infrared local detail features by itself. The fused local detail features are then summed. The third dot product result is obtained by multiplying the fusion weights calculated using the local range of the visible light global brightness features by itself. The fourth dot product result is obtained by multiplying the fusion weights calculated using the local range of the infrared global brightness features by itself. The fused global brightness features are then summed. Figure 4 As shown, the feature decoding module performs feature decoding by fusing local detail features and fusing global brightness features to obtain the final reconstructed fused grayscale image.

[0065] like Figure 6 As shown in the figure, the method of the present invention is compared with existing advanced visible light and infrared image fusion algorithms. It can be seen from the figure that the method proposed in this invention has better fusion visual effects in terms of improving the brightness of dark areas, highlighting hot objects, maintaining color vibrancy, and overall naturalness.

[0066] The above are merely specific embodiments of the present invention and should not be construed as limiting the scope of the present invention. Equivalent variations made by those skilled in the art based on this invention, as well as changes well-known to those skilled in the art, should still fall within the scope of the present invention.

Claims

1. A visible light and thermal infrared fusion method based on a two-layer feature extraction network, characterized in that, include: 1) Collect several visible light and infrared images from different scenes and construct them into an image training set; 2) Construct a two-layer feature extraction network, which includes a two-layer feature encoding module, a two-layer feature fusion module, and a feature decoding module connected in sequence; input the image training set into the two-layer feature extraction network for training to obtain the trained two-layer feature extraction network; 3) Acquire visible light and infrared images to be fused from the same scene. Convert the visible light image to the L channel of the LAB color space to obtain a visible light grayscale image. Input the visible light grayscale image and the infrared image into a trained two-layer feature extraction network for processing. After processing, the trained two-layer feature extraction network outputs a fused grayscale image. Convert the fused grayscale image to the color space using the AB chromaticity channels of the original visible light image to obtain the final color fused image, thus achieving the final fusion of visible light and thermal infrared images. In step 2), the dual-layer feature encoding module includes a local detail feature encoder and a global brightness feature encoder. The input image of the dual-layer feature encoding module is input into the local detail feature encoder and the global brightness feature encoder respectively, and after processing, the local detail features and global brightness features are output, and then jointly output to the dual-layer feature fusion module. The local detail feature encoder comprises six fully convolutional residual layers connected in sequence. The input of the global brightness feature encoder is first processed in the first fully convolutional residual layer. The processed output is added to the input of the local detail feature encoder to obtain a first sum result. The first sum result is then processed in the second fully convolutional residual layer. The processed output is added to the first sum result to obtain a second sum result. The second sum result is then processed in the third fully convolutional residual layer. The processed output is added to the second sum result to obtain a third sum result. The third sum result is then processed in the fourth fully convolutional residual layer. The processed output is added to the third sum result to obtain a fourth sum result. The fourth sum result is then processed in the fifth fully convolutional residual layer. The processed output is added to the fourth sum result to obtain a fifth sum result. The fifth sum result is then processed in the sixth fully convolutional residual layer. The processed output is added to the fifth sum result to output the local detail feature. The global luminance feature encoder includes four convolutional layers and two autocorrelation coding layers connected in sequence. The input of the global luminance feature encoder is processed by the four convolutional layers and the two autocorrelation coding layers in sequence to output global luminance features.

2. The visible light and thermal infrared fusion method based on a dual-layer feature extraction network according to claim 1, characterized in that: In step 2), the dual-layer feature fusion module uses local luminance entropy to fuse global luminance features and uses absolute value normalization weights to perform weighted fusion of local detail features.

3. The visible light and thermal infrared fusion method based on a dual-layer feature extraction network according to claim 1, characterized in that: In step 2), the feature encoding module includes four densely connected convolutional layers. The feature encoder first concatenates the input local detail features and global brightness features to obtain a first concatenation result. The first concatenation result is then input into the first convolutional layer for processing. The processed output is then concatenated with the first concatenation result to obtain a second concatenation result. The second concatenation result is then input into the second convolutional layer for processing. The processed output is then concatenated with the first and second concatenation results to obtain a third concatenation result. The third concatenation result is then input into the third convolutional layer for processing. The processed output is then concatenated with the first, second, and third concatenation results to obtain a fourth concatenation result. The fourth concatenation result is then input into the fourth convolutional layer for processing. The processed output is then concatenated with the first, second, third, and fourth concatenation results to finally decode and obtain the reconstructed image.

4. The visible light and thermal infrared fusion method based on a dual-layer feature extraction network according to claim 1, characterized in that: In step 2), when the image training set is input into the dual-layer feature extraction network for training, for each infrared image and visible light image in the image training set, a single infrared image or visible light image is input into the dual-layer feature encoding module of the dual-layer feature extraction network for processing. After processing, the global brightness features and local detail features of the single infrared image or visible light image are output. The global brightness features and local detail features of the single infrared image or visible light image are then input into the feature decoding module for processing. After processing, the reconstructed image is output. During training, a self-supervised training method is adopted to train learnable parameters for the two-layer feature encoding and feature decoding modules of the two-layer feature extraction network. Specifically, the Adam trainer is used to optimize and iteratively update the network parameters of the two-layer feature extraction network. When the preset number of iterations is reached, the parameter update is stopped to obtain the optimal network parameters of the two-layer feature extraction network. The self-supervised loss function of the two-layer feature extraction network is then calculated, as follows: L=λ1L pixel +λ2L SSIM +λ3L Feat Where L represents the self-supervised loss value of the two-layer feature extraction network; λ1, λ2, and λ3 represent the first, second, and third weighting parameters of the two-layer feature extraction network, respectively; L pixel L represents the pixel loss value between the input original image and the output reconstructed image of the two-layer feature extraction network. SSIM L represents the structural similarity loss value between the input original image and the output reconstructed image of the two-layer feature extraction network. Feat This represents the feature attribute loss value between the input original image and the output reconstructed image of the two-layer feature extraction network; Finally, the self-supervised loss value of the two-layer feature extraction network gradually decreases and stabilizes, completing the self-supervised training and obtaining the trained two-layer feature extraction network.

5. The visible light and thermal infrared fusion method based on a dual-layer feature extraction network according to claim 4, characterized in that: The feature attribute loss value L between the input original image and the output reconstructed image of the two-layer feature extraction network. Feat Specifically as follows: L Feat =(|F D |+|▽F B |) / (|F B |+|▽F D |+e) Where ▽ represents the Laplace operator; Indicates local details; Represents global brightness characteristics; Represents a constant.

6. The visible light and thermal infrared fusion method based on a dual-layer feature extraction network according to claim 1, characterized in that: In step 3), the visible light grayscale image and the infrared image are input into the trained dual-layer feature extraction network for processing. First, the visible light grayscale image and the infrared image are input into the dual-layer feature encoding module for processing. After processing, the visible light local detail features and visible light global brightness features of the visible light grayscale image, as well as the infrared local detail features and infrared global brightness features of the infrared image, are output to the dual-layer feature fusion module. The dual-layer feature fusion module performs weighted fusion of visible light local detail features, infrared local detail features, visible light global brightness features, and infrared global brightness features. The visible light local detail features are weighted using Softmax and multiplied by themselves to obtain the first multiplication result. The infrared local detail features are weighted using Softmax and multiplied by themselves to obtain the second multiplication result. The first and second multiplication results are added to obtain the fused local detail features. The visible light global brightness features are weighted using local range and multiplied by themselves to obtain the third multiplication result. The infrared global brightness features are weighted using local range and multiplied by themselves to obtain the fourth multiplication result. The third and fourth multiplication results are added to obtain the fused global brightness features. feature The decoding module performs feature decoding by fusing local detail features and global brightness features to obtain the final reconstructed fused grayscale image.

Images