Two-stage infrared and visible image fusion method based on mask prior

Abstract

The present invention proposes a two-stage infrared and visible light image fusion method based on mask prior, which belongs to the technical field of infrared and visible light image fusion and solves the problem that a single infrared or visible light image cannot fully express the scene information. The method includes: 1. Preprocessing the infrared and visible light source images in the image database and dividing them into a training set and a test set. 2. Training the salient target detection network to obtain a trained salient target detection network model. 3. Inputting the infrared source image into the salient target detection network to obtain a mask image. 4. Constructing an image fusion network, training the image fusion network, and obtaining a trained image fusion network model. 5. Using the image fusion network model to test the images in the test set to obtain a fused image. Based on mask prior knowledge, the present invention can effectively extract salient information from infrared images, so the obtained fused image can better highlight the target information and reduce the influence of strong light and other environments on the fused image.

Description

Technical Field

[0001] The present invention belongs to the field of infrared and visible light image fusion, and in particular relates to a method for fusing infrared and visible light images in a strong light environment. Background Art

[0002] Infrared images provide saliency information but lack texture detail. Conversely, visible light images offer rich texture detail, but their image quality is easily affected by harsh environments. Infrared and visible light image fusion technology integrates the complementary information from the two images and merges them into a single image. The fused image contains more salient object information and texture detail, thus facilitating subsequent visual tasks.

[0003] Traditional methods, due to their hand-crafted feature extraction rules, struggle to adapt to complex scenarios. While deep learning methods have made significant progress in recent years, most algorithms are designed for normal environments, resulting in poor fusion performance in adverse environments. Furthermore, the loss function is crucial to fusion performance, and effectively integrating the complementary information of images from different modalities is a challenge in current image fusion. Summary of the Invention

[0004] In view of this, the present invention provides an infrared and visible light image fusion method for strong light environment, which can solve the problem of degraded fusion performance caused by conditions such as excessive light during image fusion. In addition, a mask loss function is designed to promote the fused image to integrate more information from the source images.

[0005] To achieve the above, the technical solution adopted by the present invention is:

[0006] The present invention provides a two-stage infrared and visible light image fusion method based on mask prior, comprising the following steps:

[0007] Step 1: Obtain the training set of the salient object detection network;

[0008] Step 2: Introduce a salient object detection network and use the images in the training set for training to extract salient objects in infrared images and generate mask images;

[0009] Step 3: Obtain the training set of the fusion network;

[0010] Step 4: The obtained mask image and the training set are input into the fusion network, which is trained under the guidance of a loss function. The fusion network adopts an autoencoder structure. The encoder consists of three branches and a scene segmentation enhancement module, including one foreground branch and two background branches. Each branch contains five convolutional units, each consisting of a 3x3 convolutional layer and a Reinforced Luminaire (ReLU) activation layer. The scene segmentation enhancement module extracts high-frequency information from the image and uses it as compensation to supplement the foreground and background texture information. The fusion strategy comprises channel and spatial attention mechanisms, respectively fusing the extracted foreground and background features. The decoder consists of four decoding units and a mask prior branch. Each decoding unit consists of a 3x3 convolutional layer and a Reinforced Luminaire (ReLU) activation layer. The mask prior branch supplements the shallow mask features extracted by the encoder into the decoder as prior knowledge, guiding the decoder to reconstruct a fused image with richer saliency information.

[0011] Step 5: Input the infrared and visible light source images to be fused and the corresponding mask images in the test set into the trained fusion network to obtain the fused image;

[0012] Preferably, in step 3, the infrared and visible light source images are from the MSRS dataset, specifically, 1083 pairs of images in the MSRS dataset are selected as the training set for the fusion network;

[0013] Preferably, in step 5, the infrared and visible light source images are from the TNO, RoadScene, and FLIR datasets. Specifically, 20 pairs of images from the TNO dataset, 220 pairs of images from the RoadScene dataset, and 170 pairs of images from the FLIR dataset are selected as test sets for the fusion network.

[0014] Preferably, in step 4, the loss function includes pixel loss L mask , structural similarity loss L SSIM and mask loss L Mask The calculation formula is shown in formula (1):

[0015] L total =αL pixel +βL SSIM +γL Mask (1)

[0016] In formula (1), α, β, and γ are all hyperparameters;

[0017] In formula (1), the calculation formula for pixel loss is shown in formula (2):

[0018]

[0019] In formula (2), represents the L2 norm.

[0020] In formula (1), the calculation formula of structural similarity loss is shown in formula (3):

[0021] L ssim =1-SSIM(Output,Input) (3)

[0022] In formula (3), SSIM represents the structural similarity loss.

[0023] In formula (1), the calculation formula of mask loss is shown in formula (4):

[0024]

[0025] In formula (4), H and W are the height and width of the image, respectively. M and (1-M) represent the mask images of the salient target area and the background area, respectively. The segmentation is performed using the mask images in the MSRS dataset. ▽ represents the gradient operator. This paper uses the Sobel operator to calculate the gradient. Max(·) represents the maximum value of each element. represents element-wise multiplication. || ||1 refers to the L1 norm.

[0026] Compared with the prior art, the present invention has the following beneficial effects:

[0027] The present invention provides a two-stage infrared and visible light image fusion method based on mask prior, which aims to address the problems of unclear salient targets and insufficient detailed texture in the fused image due to strong light. By introducing a mask, the image fusion network is divided into two branches, foreground and background, for processing, thereby avoiding the influence of redundant information on saliency information and texture information. Through the designed loss function, the saliency information in the infrared source image and the texture information in the visible light source image can be fully integrated, so that the generated fused image can obtain rich texture details and salient target information, which is convenient for visual perception. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0029] Figure 1 This is the overall flow chart of this application;

[0030] Figure 2A schematic diagram of the overall model provided by the present invention;

[0031] Figure 3 A schematic diagram of the training phase provided by the present invention;

[0032] Figure 4 A schematic diagram of the fusion module provided by the present invention;

[0033] Figure 5 Comparison of original infrared images, original visible light images, and fused images in daytime and nighttime scenes;

[0034] Figure 6 This is a comparison of another set of original infrared images, original visible light images, and fused images in a strong light scene. DETAILED DESCRIPTION

[0035] To make the purpose, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0036] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed, but rather merely represents selected embodiments of the present invention. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without creative effort are also within the scope of protection of the present invention.

[0037] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not need to be further defined or explained in subsequent drawings.

[0038] The embodiments of the present invention are described in detail below with reference to the accompanying drawings and examples.

[0039] The present invention provides a two-stage infrared and visible light image fusion method based on mask prior, comprising the following steps:

[0040] Step 1: Select 800 infrared images from the public image fusion dataset MSRS to construct a training set for the salient object detection network;

[0041] Step 2: Inputting the infrared image into the salient object detection network to train the salient object detection network;

[0042] Step 3: Select 1600 pairs of infrared and visible light images from the public image fusion dataset MSRS to construct the training set of the image fusion network;

[0043] Step 4: Construct an image fusion network for feature extraction and image restoration. The fusion network includes an encoder module based on scene segmentation and enhancement, a foreground-background fusion strategy module, and a decoder module based on mask priors.

[0044] Step 4.1: The encoder module based on scene segmentation and enhancement first extracts shallow-scale features of the infrared and visible light images and the mask image, and then segments the infrared and visible light into foreground and background features through the mask. The foreground and background features are then Fourier transformed to extract high-frequency information and used as supplementary information to be input into the encoder of the next layer.

[0045] Among them, the feature definition of each layer of the encoder is supplemented as follows:

[0046]

[0047] In formula (1), n ​​represents different feature maps, n = 1 is infrared feature map, n = 2 is visible light feature map, n = 3 is mask feature map, m represents the features of different layers extracted by the encoder, m∈{1,2,3,4}, and represents the high-frequency information extracted from infrared and visible light images, GAP(·) is the global average pooling, and δ is the sigmoid function.

[0048] Step 4.2: The foreground-background fusion strategy module uses the enhanced fourth-scale features to obtain fused features. The fusion module uses a mask to segment the infrared and visible light images into foreground and background features. The foreground and background of the infrared and visible light images are then fed into the dual-branch feature fusion module for feature fusion. To ensure the generalization of the fusion network, the fused foreground and background information are weighted with the infrared foreground and visible light background, respectively, and then added together to obtain the final fused features.

[0049] Among them, in the above scheme, the definition of the fused image is as follows:

[0050]

[0051] In formula (4), Represents the fusion feature. In formula (2), and They represent the foreground information of the fusion feature and the infrared feature respectively, and the weight values ​​of the two are obtained through the softmax function and In formula (3) and Represents the background information of fusion features and visible light features respectively, and the weights of the two are obtained through the softmax function and

[0052] Step 4.3: The mask prior-based decoder module uses the shallow mask features extracted by the encoder to further extract mask features as prior knowledge through the mask prior module, and jointly guides the decoder to reconstruct the fused image together with the fusion features obtained by the fusion module.

[0053] Among them, in the above scheme, the mask features further extracted are defined as follows:

[0054]

[0055] In formula (5), represents the mask feature of the mth layer, CA(·) represents the channel attention mechanism, Represents pixel multiplication.

[0056] Step 5: Input the infrared and visible light image pairs into the image fusion network, and train the image fusion network under the guidance of the loss function;

[0057] In step 5, the loss function includes a pixel loss function, a structural similarity loss function, and a mask loss function.

[0058] Among them, the loss function is defined as follows:

[0059] L total =αL pixel +βL mask +γL ssim (6)

[0060] In formula (6), α, β, and γ are the loss weights of each loss function, which are defined as 0.5, 10, and 0.5 respectively.

[0061] The pixel loss function is defined as follows:

[0062]

[0063] In formula (7), Output is the fused image output by the network, and Input is the source image.

[0064] The structural similarity loss function is defined as follows:

[0065] L ssim =1-SSIM(Output,Input) (8)

[0066] In formula (8), SSIM(·) is the structural similarity function.

[0067] Among them, the definition of the mask loss function is as follows:

[0068]

[0069] In formula (9), H, W are the height and width of the image, and M is the mask image used to distinguish the foreground and background of the image. f and I r are fused image and infrared image respectively, ▽I f , ▽I ir , ▽I vi are the gradient information of the fused image, infrared image and visible light image respectively.

[0070] Step 6: Input the infrared image in the infrared and visible light image pair to be fused into the trained salient object detection network to obtain a mask image;

[0071] Step 7: Input the infrared and visible light image pairs to be fused and the mask image into the trained fusion network to obtain the fused image.

[0072] In summary, the foregoing is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A two-stage infrared and visible light image fusion method based on mask prior, characterized by include: Step 1: Obtain infrared and visible light paired datasets, preprocess and normalize the images, and further divide the processed paired images into training and test sets; Step 2: Build a salient object detection network to extract the mask in the infrared image to guide the network to extract more salient information; Step 3: Input the infrared images in the training set into the salient object detection network and train it under the guidance of the loss function; Step 4: Construct an encoder module based on scene segmentation and enhancement for the infrared and visible light image fusion network. This module uses masks to separate the infrared and visible light images into foreground and background, extracts high-frequency information of the foreground and background, and adds the high-frequency information to the features extracted by the encoder to enhance texture detail information. Step 5: Construct a foreground-background fusion strategy module of the infrared and visible light image fusion network, which fuses the multi-scale infrared and visible light image foreground and background features extracted by the encoder module respectively; Step 6: Construct a decoder module based on mask priors for the infrared and visible light image fusion network. During feature reconstruction, this module uses the multi-scale mask features extracted by the encoder as prior knowledge, and together with the fusion features obtained by the fusion strategy module, guides the decoder to reconstruct the image. Step 7: Input the infrared and visible light images into the above fusion network and train the encoder feature extraction ability and the decoder feature reconstruction ability under the guidance of the newly constructed loss function; Step 8: Input the infrared and visible light images to be fused into the trained fusion network and reconstruct them into a fused image through the encoder, fusion strategy, and decoder in sequence.

2. The two-stage infrared and visible light image fusion method based on mask prior according to claim 1, characterized in that: The salient object detection network extracts features at six scales, then concatenates the six feature maps, and then obtains the final mask image through convolution and Sigmoid activation function.

3. The two-stage infrared and visible light image fusion method based on mask prior according to claim 1, characterized in that: The newly constructed loss function includes intensity loss, structural similarity loss and mask loss.

4. The two-stage infrared and visible light image fusion method based on mask prior according to claim 1 is characterized in that The encoder module based on scene segmentation and enhancement in step 4 uses Fourier transform to extract high-frequency information in features and supplements the detailed texture of features of the same scale as supplementary information.

5. The two-stage infrared and visible light image fusion method based on mask prior according to claim 2 is characterized in that The salient object detection network in step 3 is a two-level nested U structure, which enables the network to capture richer local and global information from shallow and deep layers.

6. The two-stage infrared and visible light image fusion method based on mask prior according to claim 3 is characterized in that The mask loss calculation process is to segment the obtained fused image and the corresponding infrared and visible light images into foreground and background using the masks obtained by the salient object detection network and calculate the foreground and background losses respectively. The foreground uses pixel intensity loss to constrain the pixel intensity of the fused image to be close to the infrared image, and the background uses gradient loss to constrain the gradient of the fused image to be close to the image with the largest gradient between the infrared and visible light images.

Images