Two-stage infrared and visible light image fusion method based on mask prior

Abstract

The invention provides a mask prior-based two-stage infrared and visible light image fusion method, belongs to the technical field of infrared and visible light image fusion, and solves the problem that scene information cannot be comprehensively expressed by an infrared or visible light single image. The method comprises the following steps: 1, preprocessing infrared and visible light source images in an image database, and dividing the infrared and visible light source images into a training set and a test set; 2, training a salient target detection network to obtain a trained salient target detection network model; and 3, inputting the infrared source image into the salient target detection network to obtain a mask image. And 4, constructing an image fusion network, and training the image fusion network to obtain a trained image fusion network model. And 5, testing the images in the test set by using the image fusion network model to obtain a fused image. According to the invention, based on the mask priori knowledge, the saliency information in the infrared image can be effectively extracted, so that the obtained fusion image can better highlight the target information, and the influence of environments such as strong light on the fusion image is weakened.

Description

Technical Field

[0001] The present invention belongs to the field of infrared and visible light image fusion, and in particular, relates to an infrared and visible light image fusion method for use in a strong light environment. Background Art

[0002] Infrared images can provide saliency information but lack texture details; on the contrary, visible light images can provide rich texture details, but the image quality is easily affected by harsh environments. Infrared and visible light image fusion technology can integrate the complementary information in the two images and merge them into one image. The fused image contains more salient target information and texture details, thus facilitating the subsequent visual tasks.

[0003] Traditional methods are difficult to adapt to complex scene applications due to their manually designed feature extraction rules. Although deep learning methods have made significant progress in recent years, most algorithms are designed for normal environments, so the fusion effect in harsh environments is poor. In addition, the loss function is crucial to the fusion performance, and effectively integrating the complementary information of different modal images is a challenge facing current image fusion. Summary of the invention

[0004] In view of this, the present invention provides an infrared and visible light image fusion method for a strong light environment, which can solve the problem of decreased fusion performance caused by excessive light during image fusion. In addition, a mask loss function is designed to promote the fused image to integrate more information from the source images.

[0005] To achieve the above content, the technical solution adopted by the present invention is:

[0006] The present invention provides a two-stage infrared and visible light image fusion method based on mask prior, comprising the following steps:

[0007] Step 1: Obtain the training set of the salient object detection network;

[0008] Step 2: Introduce a salient object detection network and use the images in the training set for training to extract salient objects in infrared images and generate mask images;

[0009] Step 3: Obtain the training set of the fusion network;

[0010] Step 4: Input the obtained mask image and the above training set into the fusion network, and train under the guidance of the loss function; wherein the fusion network adopts an autoencoder structure, wherein the encoder Encoder comprises three branches and a scene segmentation enhancement module, including a foreground branch and two background branches. wherein each branch comprises five convolution units, each of which consists of a 3x3 convolution layer and a Relu activation layer; wherein the scene segmentation enhancement module is used to extract high-frequency information in the image and supplement the foreground and background texture information as compensation information. The fusion strategy consists of a channel and a spatial attention mechanism, which fuse the extracted foreground and background features respectively. The decoder Decoder consists of four decoding units and a mask prior branch, wherein each decoding unit consists of a 3x3 convolution and a Relu activation layer; wherein the mask prior branch supplements the shallow mask features extracted by the encoder into the decoder as prior knowledge, guiding the decoder to reconstruct a fused image with richer saliency information.

[0011] Step 5: Input the infrared and visible light source images to be fused in the test set and the corresponding mask images into the trained fusion network to obtain a fused image;

[0012] Preferably, in step 3, the infrared and visible light source images are from the MSRS dataset, specifically, 1083 pairs of images in the MSRS dataset are selected as training sets for the fusion network;

[0013] Preferably, in step 5, the infrared and visible light source images are from TNO, RoadScene and FLIR datasets, specifically, 20 pairs of images from the TNO dataset, 220 pairs of images from the RoadScene dataset and 170 pairs of images from the FLIR dataset are selected as test sets for the fusion network respectively;

[0014] Preferably, in step 4, the loss function includes the pixel loss L mask , structural similarity loss L SSIM and the mask loss L Mask The calculation formula is shown in formula (1):

[0015] L total =αL pixel +βL SSIM +γL Mask (1)

[0016] In formula (1), α, β, and γ are all hyperparameters;

[0017] In formula (1), the calculation formula of pixel loss is shown in formula (2):

[0018]

[0019] In formula (2), || ||2 represents the L2 norm.

[0020] In formula (1), the calculation formula of structural similarity loss is shown in formula (3):

[0021] L ssim =1-SSIM(Output,Input) (3)

[0022] In formula (3), SSIM represents the structural similarity loss.

[0023] In formula (1), the calculation formula of mask loss is shown in formula (4):

[0024]

[0025] In formula (4), H and W are the height and width of the image, respectively. M and (1-M) represent the mask images of the salient target area and the background area, respectively. The segmentation is performed using the mask images in the MSRS dataset. ▽ represents the gradient operator. This paper uses the Sobel operator to calculate the gradient. Max(·) represents the maximum value of each element. represents element-wise multiplication. || ||1 refers to the l1 norm.

[0026] Compared with the prior art, the present invention has the following beneficial effects:

[0027] The present invention provides a two-stage infrared and visible light image fusion method based on mask prior, which aims to deal with the problems of unclear salient targets and insufficient detailed texture in the fused image due to strong light. By introducing a mask, the image fusion network is divided into two branches, foreground and background, for processing, thereby avoiding the influence of redundant information on salient information and texture information. Through the designed loss function, the salient information in the infrared source image and the texture information in the visible light source image can be fully integrated, so that the generated fused image can obtain rich texture details and salient target information, which is convenient for visual perception. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0029] Figure 1 This is the overall flow chart of this application;

[0030] Figure 2 A schematic diagram of the overall model provided by the present invention;

[0031] Figure 3 A schematic diagram of the training phase provided by the present invention;

[0032] Figure 4 A schematic diagram of a fusion module provided by the present invention;

[0033] Figure 5 Comparison chart of original infrared image, original visible light image and fused image in day and night scenes;

[0034] Figure 6 This is a comparison chart of another set of original infrared images, original visible light images and fused images in a strong light scene. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0036] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0037] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0038] The embodiments of the present invention are described in detail below with reference to the accompanying drawings and examples.

[0039] The present invention provides a two-stage infrared and visible light image fusion method based on mask prior, comprising the following steps:

[0040] Step 1: Select 800 infrared images from the public image fusion dataset MSRS to construct a training set for the salient object detection network;

[0041] Step 2: inputting the infrared image into the salient object detection network to train the salient object detection network;

[0042] Step 3: Select 1600 pairs of infrared and visible light images from the public image fusion dataset MSRS to construct a training set for the image fusion network;

[0043] Step 4: construct an image fusion network for feature extraction and image restoration, wherein the fusion network includes: an encoder module based on scene segmentation and enhancement, a foreground-background fusion strategy module, and a decoder module based on mask prior;

[0044] Step 4.1: The encoder module based on scene segmentation and enhancement first extracts shallow-scale features of infrared and visible light images and mask images, and segments the infrared and visible light into foreground and background features through masks. The high-frequency information of the foreground and background features is extracted through Fourier transform and input into the encoder of the next layer as supplementary information.

[0045] Among them, the feature supplement of each layer of the encoder is defined as follows:

[0046]

[0047] In formula (1), n ​​represents different feature maps, n=1 is the infrared feature map, n=2 is the visible light feature map, and n=3 is the mask feature map. and represents the high-frequency information extracted from infrared and visible light images, GAP(·) is the global average pooling, and δ is the sigmoid function.

[0048] Step 4.2: The foreground-background fusion strategy module uses the enhanced fourth-scale features to obtain fusion features. The fusion module uses a mask to segment infrared and visible light into foreground and background features, and then sends the foreground and background of the infrared and visible light images to the dual-branch feature fusion module for feature fusion. Then, in order to ensure the generalization of the fusion network, the fused foreground and background information are respectively added to the infrared foreground and visible light background after calculating the weights to obtain the final fusion feature.

[0049] Among them, in the above scheme, the definition of the fused image is as follows:

[0050]

[0051] In formula (4), represents the fusion feature. In formula (2), and They represent the foreground information of the fusion feature and the infrared feature respectively, and the weight values ​​of the two are obtained through the softmax function and In formula (3) and Respectively represent the background information of the fusion feature and the visible light feature, and the weights of the two are obtained through the sofrmax function and

[0052] Step 4.3: The mask prior-based decoder module uses the shallow mask features extracted by the encoder to further extract mask features as prior knowledge through the mask prior module, which together with the fusion features obtained by the fusion module guide the decoder to reconstruct the fused image.

[0053] Among them, in the above scheme, the mask features further extracted are defined as follows:

[0054]

[0055] In formula (5), represents the mask feature of the mth layer, CA(·) represents the channel attention mechanism, Represents pixel multiplication.

[0056] Step 5: inputting the infrared and visible light image pairs into the image fusion network, and training the image fusion network under the guidance of the loss function;

[0057] Wherein, in step 5, the loss function includes a pixel loss function, a structural similarity loss function and a mask loss function.

[0058] The loss function is defined as follows:

[0059] L total =αL pixel +βL mask +γL ssim (6)

[0060] In formula (6), α, β, and γ are the loss weights of each loss function, which are defined as 0.5, 10, and 0.5 respectively.

[0061] The pixel loss function is defined as follows:

[0062]

[0063] In formula (7), Output is the fused image output by the network, and Input is the source image.

[0064] The definition of the structural similarity loss function is as follows:

[0065] L ssim =1-SSIM(Output,Input) (8)

[0066] In formula (8), SSIM(·) is the structural similarity function.

[0067] Among them, the definition of the mask loss function is as follows:

[0068]

[0069] In formula (9), H and W are the height and width of the image, and M is the mask image used to distinguish the foreground and background of the image. f and I r are fused image and infrared image respectively, They are the gradient information of the fused image, infrared image and visible light image respectively.

[0070] Step 6: Input the infrared image in the pair of infrared and visible light images to be fused into the trained salient object detection network to obtain a mask image;

[0071] Step 7: Input the infrared and visible light image pairs to be fused and the mask image into the trained fusion network to obtain a fused image.

[0072] In summary, the above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art who is familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A two-stage infrared and visible light image fusion method based on mask prior, characterized in that: include: Step 1: Obtain infrared and visible light paired data sets, and preprocess and normalize the images. The processed paired images are further divided into training sets and test sets; Step 2: Construct a salient object detection network to extract masks in infrared images to guide the network to extract more salient information; Step 3: Input the infrared images in the training set into the salient object detection network and perform training under the guidance of the loss function; Step 4: construct an infrared and visible light image fusion network for feature extraction, fusion and reconstruction. The fusion network includes: an encoder module based on scene segmentation and enhancement, a foreground and background fusion strategy module, and a decoder module based on mask priors; Step 5: Input the infrared and visible light images into the above fusion network, and train the encoder feature extraction ability and the decoder feature reconstruction ability under the guidance of the newly constructed loss function; Step 6: Input the infrared and visible light images to be fused into the trained fusion network, and reconstruct the fused image through the encoder, fusion strategy, and decoder in sequence.

2. The two-stage infrared and visible light image fusion method based on mask prior according to claim 1 is characterized in that: The salient object detection network extracts features at six scales, then concatenates the six feature maps, and then obtains the final mask image through convolution and Sigmoid activation function.

3. The two-stage infrared and visible light image fusion method based on mask prior according to claim 1, characterized in that: The encoder module based on scene segmentation and enhancement extracts shallow infrared and visible light images and mask features through convolution, and then uses the mask features to separate the infrared and visible light images into foreground and background respectively, and extracts high-frequency information of the foreground and background, and supplements the high-frequency information to the features extracted by the encoder of the same scale.

4. The two-stage infrared and visible light image fusion method based on mask prior according to claim 1, characterized in that: The foreground-background fusion strategy module splits the infrared and visible light images extracted by the encoder module into the foreground and background of the corresponding images, and fuses the foreground and background features respectively.

5. The two-stage infrared and visible light image fusion method based on mask prior according to claim 1, characterized in that: The mask prior-based decoder module uses the mask features extracted by the encoder of the corresponding level as prior knowledge as a supplementary feature during feature reconstruction, and cooperates with the features obtained by the fusion strategy to jointly guide the decoder to reconstruct the image.

6. The two-stage infrared and visible light image fusion method based on mask prior according to claim 1, characterized in that: The newly constructed loss functions include an intensity loss function, a structural similarity loss function, and a mask loss.

7. The two-stage infrared and visible light image fusion method based on mask prior according to claim 2, characterized in that: The encoder module based on scene segmentation and enhancement in step S40, wherein the scene segmentation and enhancement module uses Fourier transform to extract high-frequency information in features, and uses the information as supplementary information to supplement the detail texture of features of the same scale.

8. The two-stage infrared and visible light image fusion method based on mask prior according to claim 3, characterized in that: The salient object detection network in step S30 is a two-level nested U structure, which enables the network to capture richer local and global information from shallow and deep layers.

9. The two-stage infrared and visible light image fusion method based on mask prior according to claim 4, characterized in that: The mask loss calculation process is to segment the obtained fused image and the corresponding infrared and visible light images into foreground and background using the mask obtained by the salient object detection network, and calculate the foreground and background losses respectively. The foreground uses pixel intensity loss to constrain the pixel intensity of the fused image to be close to the infrared image, and the background uses gradient loss to constrain the gradient of the fused image to be close to the image with the largest gradient between the infrared and visible light images.

Images