A method of infrared and visible light image fusion for nighttime target enhancement

By using a multi-path network model and feature extraction module in night image fusion, combined with loss function optimization, the problems of low contrast and information loss in the existing technology are solved, and the significance target enhancement effect is significantly improved and the fusion image quality is improved.

CN115565035BActive Publication Date: 2024-11-19CHANGCHUN UNIV OF SCI & TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202211391319.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-11-08
Publication Date
2024-11-19
Estimated Expiration
2042-11-08

AI Technical Summary

Technical Problem

Existing night image fusion methods lead to low contrast of fusion images, loss of important information and poor enhancement effects of infrared saliency targets.

Method used

Using a network model including three input paths, infrared and visible images are fused through feature extraction and image reconstruction parts, using spatial attention modules and grouped convolution blocks, combining pixels and gradient loss functions to ensure outstanding and information of significance targets reserve.

Benefits of technology

The contrast of the fusion image is improved, more important information is retained, the significance objectives are more prominent, and the fusion image is more in line with the human visual system.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115565035B_ABST
    Figure CN115565035B_ABST
Patent Text Reader

Abstract

The invention discloses an infrared and visible light image fusion method for night target enhancement, which belongs to the field of image processing and machine vision technology. In order to solve the problems that the fused image obtained by the existing method has low contrast, information is lost in the process of wavelet transform, and the enhancement effect of infrared significant targets is poor, the method comprises the following steps: preparing training data; constructing a network model; designing a loss function; training the network model; determining a fusion model; adding an infrared foreground image as the input of the network to enhance the significant targets in the source image in the fused image; adding a spatial attention module after shallow feature extraction, wherein the spatial attention mechanism filters background details according to shallow features, pays more attention to the foreground area, and highlights the significant targets in the fused image; using grouped convolution in the network model to prevent information loss; using an adjusted image as an image label when constructing a loss function to avoid the information in the fused image from being biased towards a certain source image, thereby avoiding information loss.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to an infrared and visible light image fusion method for nighttime target enhancement, belonging to the technical field of image processing and machine vision. Background Art

[0002] Image fusion integrates source images from different sensors, and the fusion result makes up for the information limitations of a single sensor. Visible light images taken in night scenes have poor visual quality, low contrast, and serious detail loss. At the same time, unprocessed night visible light images are difficult to highlight salient targets and are difficult to use for subsequent advanced visual tasks. Thanks to the special acquisition principle of infrared images, they are not affected by lighting changes and bad weather, and can highlight salient targets. Therefore, infrared and visible light images are fused to obtain a fused image that is suitable for the human visual system and highlights salient targets. The fusion results can be applied to target recognition, security monitoring, target tracking and other fields. However, the existing image fusion methods for night scenes have the following problems: low contrast of the fused image, loss of important information and poor enhancement of salient targets.

[0003] The Chinese patent publication number is "CN105069769A", and its name is "A method for fusing low-light and infrared night vision". This method collects low-light images and infrared images for the same scene, then detects the type of noise in the collected source image, and performs corresponding denoising for different types of noise. The denoised low-light and infrared images are effectively registered using an image registration method based on edge features, and finally image fusion is performed using wavelet transform. The wavelet transform first performs wavelet decomposition on each source image, and then processes the decomposition layers with different fusion operators and rules to finally obtain a fused image. The fused image obtained by this method has low contrast, there is information loss in the wavelet transform process, and the enhancement effect of infrared salient targets is poor. Summary of the invention

[0004] In order to solve the problems of low contrast of fused images obtained by existing methods, information loss in the process of wavelet transform, and poor enhancement effect of infrared significant targets, the present invention provides an infrared and visible light image fusion method for nighttime target enhancement. The fused image obtained by the present invention has higher contrast, can retain more important information, and the significant targets are more prominent.

[0005] The solution to the technical problem of the present invention is:

[0006] A method for fusion of infrared and visible light images for nighttime target enhancement, the specific steps are as follows:

[0007] Step 1, prepare training data: perform image preprocessing on infrared and visible light datasets in night scenes and select training datasets;

[0008] Step 2, building a network model: The network model includes three input paths. The three input images are respectively passed through the feature extraction part to obtain feature maps. The feature maps obtained on the three paths are connected in the channel dimension and then input into the image reconstruction part to finally obtain a fused image.

[0009] Step 3, design loss function: use the fused image and image label output by the network to construct the loss function;

[0010] Step 4, training the network model: during the training process, the value of the loss function is minimized until the number of training times reaches a preset threshold or the value of the loss function is stable within a preset range, the network model training is considered complete, and the parameters of the network model are saved;

[0011] Step 5, determine the fusion model: After the network training is completed, fix the network model parameters and determine the network model as the final fusion model. When the infrared and visible light image fusion operation is required, the image can be directly input into the network to obtain the final fusion image.

[0012] The data set in step 1 uses the nighttime road scene data set MRSR, wherein the image preprocessing operations include pre-registration and foreground extraction, and the infrared source image is input into the existing instance segmentation network to obtain the infrared foreground image. Finally, the registered infrared source image, visible light source image and infrared foreground image constitute the training data set.

[0013] The feature extraction parts of the first and second input paths in step 2 include a convolution block, a spatial attention module and three grouped convolution blocks; the feature extraction part of the third input path includes a convolution block and three grouped convolution blocks; the image reconstruction part includes three convolution blocks. The convolution block includes a convolution layer, an activation function and a splicing operation, and the activation function uses a linear rectification function; the grouped convolution block includes a grouped convolution layer, an activation function and a splicing operation, and the activation function uses a parametric linear rectification function; the spatial attention module includes a maximum pooling layer, an average pooling layer, a convolution layer, an activation function and a splicing operation, and the activation function uses an S-type activation function; wherein the convolution kernel size of all convolution layers is n×n.

[0014] The loss function in step 3 includes a pixel loss function and a gradient loss function, and the adjusted image is used as the image label.

[0015] In the step 4, during the network model training process, the infrared source image and the visible light source image are used as inputs of the first and second paths, and the infrared foreground image is used as input of the third path.

[0016] The beneficial effects of the present invention are as follows:

[0017] 1. Adding infrared foreground images as network input to the network model can enhance the salient targets in the source image in the fused image.

[0018] 2. A spatial attention module is added after shallow feature extraction. The spatial attention mechanism can filter background details according to shallow features, pay more attention to the foreground area, and highlight the salient targets in the fused image.

[0019] 3. Use grouped convolution in the network model. Traditional convolution combines multi-channel features together for processing, which will cause the features of some channels to be submerged. Grouped convolution groups multi-channel features for processing, which can retain important information and prevent information loss.

[0020] 4. When constructing the loss function, use the adjusted image as the image label to avoid the information in the fused image being biased towards a certain source image and to avoid information loss. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 The present invention is a flow chart of a method for fusing infrared and visible light images for nighttime target enhancement.

[0022] Figure 2 This is a network structure diagram of an infrared and visible light image fusion method for nighttime target enhancement according to the present invention.

[0023] Figure 3 This is a diagram showing the specific structure of the convolution block described in the present invention.

[0024] Figure 4 This is a diagram showing the specific structure of the grouped convolution block described in the present invention.

[0025] Figure 5 This is a diagram showing the specific structure of the spatial attention module described in the present invention. DETAILED DESCRIPTION

[0026] The present invention is further described in detail below with reference to the accompanying drawings.

[0027] like Figure 1 As shown, a method for fusion of infrared and visible light images for nighttime target enhancement includes the following steps:

[0028] Step 1, prepare training data: perform image preprocessing on infrared and visible light datasets in night scenes and select training datasets. The dataset uses the night road scene dataset MRSR, where the image preprocessing operations include pre-registration and foreground extraction. The infrared source image is input into the existing instance segmentation network to obtain the infrared foreground image. Finally, the registered infrared source image, visible light source image and infrared foreground image constitute the training dataset.

[0029] Step 2, construct the network model: the network model includes three input paths, the three input images are respectively passed through the feature extraction part to obtain feature maps, the feature maps obtained on the three paths are connected in the channel dimension, and then input into the image reconstruction part, and finally a fused image is obtained; among them, the feature extraction part of the first and second input paths includes a convolution block, a spatial attention module and three grouped convolution blocks; the feature extraction part of the third input path includes a convolution block and three grouped convolution blocks; the image reconstruction part includes three convolution blocks. The convolution block includes a convolution layer, an activation function and a splicing operation, and the activation function uses a linear rectification function; the grouped convolution block includes a grouped convolution layer, an activation function and a splicing operation, and the activation function uses a parametric linear rectification function; the spatial attention module includes a maximum pooling layer, an average pooling layer, a convolution layer, an activation function and a splicing operation, and the activation function uses an S-type activation function; the convolution kernel size of all convolution layers is n×n.

[0030] Step 3, design the loss function: use the fused image and image label output by the network to construct the loss function; the loss function includes pixel loss function and gradient loss function, and uses the adjusted image as the image label.

[0031] Step 4, training the network model: during the training process, the value of the loss function is minimized until the number of training times reaches a preset threshold or the value of the loss function is stable within a preset range, the network model training is considered complete, and the parameters of the network model are saved. During the network model training process, the infrared source image and the visible light source image are used as inputs of the first and second paths, and the infrared foreground image is used as input of the third path.

[0032] Step 5, determine the fusion model: After the network training is completed, fix the network model parameters and determine the network model as the final fusion model. When the infrared and visible light image fusion operation is required, the image can be directly input into the network to obtain the final fusion image.

[0033] Example:

[0034] like Figure 1 As shown, a method for fusion of infrared and visible light images for nighttime target enhancement includes the following steps:

[0035] Step 1: Prepare training data. The dataset uses the nighttime road scene dataset MSRS. The dataset contains 742 sets of infrared and visible light images. Each image is pre-processed for registration. Each original image is cropped to obtain 18 sub-images of size 360×360, for a total of 13,356 sets of training images.

[0036] Step 2: Build a network model. The network model structure is shown in the figure below: Figure 2As shown, the entire network model includes three input paths. The feature extraction parts of the first and second input paths include a convolution block, a spatial attention module and three grouped convolution blocks. The feature extraction part of the third input path includes a convolution block and three grouped convolution blocks, and the image reconstruction part includes three convolution blocks.

[0037] Convolutional blocks 1, 2, 3, and 4 have the same structure. The specific structure of the convolutional block is shown in the figure below. Figure 3 As shown in the figure, it includes 2 convolutional layers and 2 activation functions. The convolution kernel sizes of the 2 convolutional layers are 3×3 and 3×3 respectively, and the step sizes are both 1. The 2 activation functions both use linear rectification functions. The input path is then connected to a 1×1 convolutional layer and concatenated with the result of the second layer activation function.

[0038] The composition structure of grouped convolution block 1, grouped convolution block 2 and grouped convolution block 3 is the same. The specific structure diagram of the grouped convolution block is as follows: Figure 4 As shown in the figure, it includes 2 layers of grouped convolutional layers and 2 activation functions. The convolution kernel sizes of the 2 layers of grouped convolutional layers are 1×1 and 3×3 respectively, the number of groups is 4, the step size is 1, and the 2 activation functions both use parametric linear rectification functions. The input path is then connected to a 1×1 convolutional layer and concatenated with the result of the second layer activation function.

[0039] The specific structure of the spatial attention module is shown in the figure Figure 5 As shown in the figure, the feature map passes through a maximum pooling layer and an average pooling layer to obtain two feature maps, which are concatenated in the channel dimension, and then pass through an activation function after a convolution to generate a spatial weight coefficient. Finally, the weight coefficient is multiplied by the input feature map to obtain the final feature map, where the size of the convolution kernel is 7×7, and the activation function uses the S-type activation function.

[0040] The linear rectification function, parameter linear rectification function, and sigmoid activation function are defined as follows:

[0041]

[0042]

[0043]

[0044] Step 3: Design the loss function. The loss function includes the pixel loss function and the gradient loss function. The adjusted image is used as the image label. The calculation formula of the pixel loss function is as follows:

[0045]

[0046] Where H and W represent the height and width of the image, respectively, ||·|| F represents the F-norm, If (x, y), I a (x, y) represents the pixel value of the fused image and the adjusted image at (x, y), respectively.

[0047] The calculation formula for adjusting the image is as follows:

[0048]

[0049] Among them I mask represents the mask of the infrared image obtained by instance segmentation, I ir and I vi Indicates infrared and visible light images, operations Represents element-wise multiplication.

[0050] The gradient loss function calculation formula is as follows:

[0051]

[0052] Represent the gradient values ​​of the fused image and the adjusted image at (x, y) respectively.

[0053] The total loss function is defined as: L = L pixel +L gradient

[0054] Step 4: Train the network model. During the network training process, the enhanced infrared and visible light images are used as the input of the first and second feature extraction paths, and the foreground of the infrared and visible light images is used as the input of the third feature extraction path. The number of training iterations is set to 100, and the learning rate is set to 10 -3 , the loss function threshold is set to 0.0002, and when the value of the loss function is less than 0.0002, the branch network training is considered to be completed.

[0055] Step 5: Determine the fusion model. After the network training is completed, the network model parameters are fixed and the network model is determined as the final fusion model. Then the registered infrared and visible light images are input into the fusion model, and the fused image is finally output. The fused image can not only highlight the salient target, but also contains rich texture details, which is more in line with the human visual system.

[0056] Among them, the implementation of convolution, activation function, splicing operation, maximum pooling and average pooling are algorithms well known to those skilled in the art, and the specific processes and methods can be found in corresponding textbooks or technical literature.

[0057] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for fusion of infrared and visible light images for nighttime target enhancement, characterized in that: The method comprises the following steps: Step 1, prepare training data: perform image preprocessing on infrared and visible light datasets in night scenes and select training datasets; Step 2, building a network model: The network model includes three input paths. The three input images are respectively passed through the feature extraction part to obtain feature maps. The feature maps obtained on the three paths are connected in the channel dimension and then input into the image reconstruction part to finally obtain a fused image. Step 3, design loss function: use the fused image and image label output by the network to construct the loss function; Step 4, training the network model: during the training process, the value of the loss function is minimized until the number of training times reaches a preset threshold or the value of the loss function is stable within a preset range, the network model training is considered complete, and the parameters of the network model are saved; Step 5, determine the fusion model: after the network training is completed, fix the network model parameters and determine the network model as the final fusion model; when the infrared and visible light image fusion operation is required, directly input the image into the network to obtain the final fusion image; The feature extraction parts of the first and second input paths in step 2 include a convolution block, a spatial attention module and three grouped convolution blocks; the feature extraction part of the third input path includes a convolution block and three grouped convolution blocks; the image reconstruction part includes three convolution blocks; the convolution block includes a convolution layer, an activation function and a splicing operation, and the activation function uses a linear rectification function; the grouped convolution block includes a grouped convolution layer, an activation function and a splicing operation, and the activation function uses a parameter linear rectification function; the spatial attention module includes a maximum pooling layer, an average pooling layer, a convolution layer, an activation function and a splicing operation, and the activation function uses an S-type activation function; wherein the convolution kernel size of all convolution layers is n×n; In the step 4, during the network model training process, the infrared source image and the visible light source image are used as inputs of the first and second paths, and the infrared foreground image is used as input of the third path.

2. The infrared and visible light image fusion method for nighttime target enhancement according to claim 1, characterized in that: The data set in step 1 uses the nighttime road scene data set MRSR, wherein the image preprocessing operations include pre-registration and foreground extraction, and the infrared source image is input into the existing instance segmentation network to obtain the infrared foreground image. Finally, the registered infrared source image, visible light source image and infrared foreground image constitute the training data set.

3. The infrared and visible light image fusion method for nighttime target enhancement according to claim 1, characterized in that: The loss function in step 3 includes a pixel loss function and a gradient loss function, and the adjusted image is used as the image label.