Infrared and visible light image fusion method based on meta learning and semantic perception

Abstract

The invention relates to the technical field of deep learning, image fusion, meta-learning, image semantic segmentation and the like, and provides an infrared and visible light image fusion method based on meta-learning and semantic perception in order to realize adaptive fusion and enhance semantic information of a fused image. The method comprises the following steps: 1) performing image registration on original infrared and visible light images acquired by a sensor through affine transformation; 2) inputting the registered infrared and visible light images into a fusion network and a meta learning network at the same time to generate a final fusion image; (3) the input infrared and visible light images are input into the fusion network and are processed by the meta-learning network at the same time; 4) inputting the image fused by the fusion network into a segmentation network; and 5) fusing the images by using the trained fusion network. The method is mainly applied to the occasion of enhancing the semantic information of the fused image.

Description

Technical Field

[0001] The present invention relates to the technical fields of deep learning, image fusion, meta-learning, image semantic segmentation, etc. and to an infrared and visible light image fusion method based on meta-learning and semantic perception. Background Art

[0002] Infrared sensors capture the thermal radiation information of objects and can effectively highlight salient targets even in extreme conditions, bad weather and partial occlusion. However, infrared images cannot provide sufficient environmental information, such as texture details, ambient lighting, etc. In contrast, visible light sensors collect reflected light from the surface of objects, so visible light images contain rich texture detail information and are more in line with human visual perception. Infrared and visible light image fusion aims to integrate the complementary information in the source images and generate high-contrast fused images that can highlight salient targets and contain rich texture details. Infrared and visible light fusion technology has a wide range of applications in many fields, such as military reconnaissance, night vision, security monitoring, etc.

[0003] At present, infrared and visible light image fusion includes traditional methods and methods based on deep learning. Traditional fusion methods mainly have three directions: methods based on sparse representation, methods based on multi-scale decomposition, and methods based on spatial transformation. In recent years, deep learning has dominated the development of computer vision with its powerful feature extraction and expression capabilities, and has shown significant performance advantages in visual tasks such as image classification, object detection, and semantic segmentation. In order to overcome the shortcomings of traditional algorithms, researchers in the field of image fusion have also explored a large number of image fusion algorithms based on deep learning. Deep learning-based infrared and visible light fusion algorithms mainly include methods based on autoencoders, CNNs (convolutional neural networks), and GANs (generative adversarial networks). The autoencoder-based method first trains an autoencoder on a large-scale natural image dataset, and then the pre-trained encoding network and decoding network are used to realize feature extraction and image reconstruction respectively. Finally, a manually designed fusion strategy is generally used to fuse the deep features extracted by the encoding network to realize image fusion. In order to enhance the ability of the encoding network to extract features, Wang Jianzhong et al. introduced dense connections, honeycomb connections, and residual dense blocks in the image fusion framework based on autoencoders. In addition, Liu et al., Jian et al., and Yu Lixin et al. introduced the attention mechanism in the autoencoder-based fusion framework, which enables the network to pay more attention to salient objects and texture detail information. However, it is very time-consuming to calculate pixel-level saliency and perform importance evaluation. Therefore, further research on real-time learnable fusion strategies will be one of the research hotspots in the future image fusion framework based on autoencoders. The end-to-end image fusion framework based on CNN is another technical route to avoid the drawbacks of manually designed fusion rules. This type of method usually relies on the designed network structure and loss function to implicitly realize feature extraction, feature fusion and image reconstruction. Generative adversarial networks can effectively model data distribution without supervised information, which is very suitable for the needs of infrared and visible light image fusion. Specifically, the generator is responsible for capturing the potential distribution in the source image samples and fully integrating these distribution characteristics into the fused image. The discriminator is responsible for distinguishing whether the input is the source image or the fused image from the distribution, thereby forcing the fusion result synthesized by the generator to contain as many distribution characteristics of the source image as possible. Although the GAN-based image fusion algorithm can generate better fusion results, how to maintain the balance between the generator and the discriminator during the training process is still worthy of further study.

[0004] However, both traditional methods and deep learning-based methods emphasize the improvement of fused image quality and evaluation indicators, while ignoring the needs of advanced visual tasks. In practical applications, fused images with good image quality may be suitable for human visual perception, but may not be suitable for advanced visual tasks. A robust image fusion algorithm should enhance the semantic information of the fused image while fully fusing the complementary information of the source images. Moreover, the existing deep learning-based image fusion methods rely on the same network model for feature expression and fusion tasks for infrared and visible light images in diverse scenes, without considering the characteristics of the current input samples of the model itself and ignoring the differences in image features in different scenes, resulting in poor fusion effects of the model on diverse scene images and loss of details in the fused image. Summary of the invention

[0005] In order to overcome the shortcomings of the prior art, the present invention aims to fully consider the characteristics of samples in diverse scenes, realize adaptive fusion, and enhance the semantic information of the fused image. To this end, the technical solution adopted by the present invention is a method for fusion of infrared and visible light images based on meta-learning and semantic perception, and the steps are as follows:

[0006] 1. Perform image registration on the original infrared and visible light images collected by the sensor through affine transformation;

[0007] 2. The registered infrared and visible light images are simultaneously input into the fusion network and the meta-learning network. The fusion network is divided into two pathways: the infrared pathway and the visible light pathway. The infrared and visible light images are respectively extracted through the input convolution layer, dense block, and deep convolution in the corresponding pathways. After the features are connected through channels, two 3×3 convolution layers and one 1×1 convolution layer are used to restore the features and generate the final fused image.

[0008] 3. The input infrared and visible light images are input to the fusion network and processed by the meta-learning network at the same time. The meta-learning network consists of a 3×3 input convolution, three identical residual blocks and two 1×1 convolutions, where each residual block consists of a 3×3 convolution and a 1×1 convolution. After multiple layers of convolution, the meta-learning network maps the infrared or visible light image to the corresponding convolution kernel parameters, which guide the feature extraction of the infrared and visible light pathways in the fusion network.

[0009] 4. Input the image obtained by fusion of the fusion network into the segmentation network to obtain the semantic loss, and combine this loss with the content loss to jointly guide the training of the fusion network and the meta-learning network: First, pre-train the fusion network, and then train the fusion network and the meta-learning network as a whole. The specific steps are: (1) Remove the meta-learning network, replace the deep convolution layer guided by the meta-learning network in the fusion network with a common convolution layer, and pre-train the fusion network and the meta-learning network; (2) Fix the shallow convolution parameters of the pre-trained fusion network, including the input convolution layer and the dense block, and do not participate in the subsequent network model training; (3) Add the meta-learning network module and only train the parts other than the above shallow convolution, namely, the meta-learning network, the deep convolution layer guided by the meta-learning network in the fusion network, and the fusion output part;

[0010] 5. Use the trained fusion network to fuse the images.

[0011] In the feature extraction part of step 2: the input convolution layer size is 5×5, which is used to extract shallow features of the image; the dense block is composed of three 3×3 convolution layers, where the input of the nth convolution layer is the cascade of all convolution outputs of the previous n-1 layers, and dense connection is achieved through layer-by-layer cascading; the dense block uses channel merging operations to extract as many deep features of the input image as possible while reducing the number of parameters, so that both shallow and deep features can be well utilized.

[0012] The detailed steps are as follows:

[0013] 1. Select the coordinates of three key points of the visible light image [x vi1 ,y vi1 ],[x vi2 ,y vi2 ],[x vi3 ,y vi3 ] corresponds to the unregistered infrared image coordinates [x ir1 ,y ir1 ],[x ir2 ,y ir2 ],[x ir3 ,y ir3 ], calculate the affine transformation matrix according to the coordinate points, generate the registered infrared image with the original infrared image as input, and convert all the original image pairs into the registered image pairs;

[0014] 2. The registered infrared and visible light images are simultaneously input into the fusion network and the meta-learning network. In the fusion network, the input image IR or VI is extracted through dense blocks to output the feature map x, x∈R N,H,W,C , where H, W, C represent the height, width and number of channels of the feature map respectively, and N represents the number of batches input to the network at the same time;

[0015] 3. The input samples of the meta-learning network undergo multiple layers of convolution, and the final output is the convolution kernel parameters required by the fusion network This means that for each pixel position of the feature map x, the meta-learning network generates an adaptive convolution kernel parameter of size 3×3, which corresponds to the three neighborhoods of the feature pixel position, thereby realizing adaptive deep feature extraction of each pixel position of the input feature map x; for a specific pixel point x of the feature map x [n,c,h,w] And the convolution kernel parameters adaptively generated by the meta-learning network The convolution operation is performed using the following formula:

[0016]

[0017] 4. Reconstruct the fused feature map to restore the fused image, input the fused image into the segmentation network, obtain the semantic loss, and design the corresponding loss function according to the information contribution requirements of infrared and visible light images in the fused image to drive the optimization update of the network model; the loss function consists of content loss and perceptual loss, and the total loss is: L = L content +θL S ; Using semantic loss L S To measure the semantic information contained in the fused image; the gap between the segmentation result R and the semantic label T is expressed as the semantic loss L S The training set is the TNO dataset. All images are normalized to [0, 1] before entering the network. The fusion network is first pre-trained, and then the fusion network and the meta-learning network are trained as a whole.

[0018] The fusion results are tested and the quality of the fused image is evaluated from the objective indicators: first, static indicators such as entropy EN, standard deviation SD, structural similarity SSIM and correlation coefficient CON are used, and the fusion results are evaluated using the average IoU of background, street lights, people and vehicles in combination with the segmentation task.

[0019] The characteristics and beneficial effects of the present invention are:

[0020] 1. The present invention can adaptively complete the image fusion task for infrared and visible light images in different application scenarios, and the fused image obtained by the fusion model is consistent with the current sample information.

[0021] 2. The semantic information of the fused image is enhanced. At the same time, the fusion network has a small number of parameters and can be processed in real time, which is convenient for subsequent advanced visual tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It is the overall structural diagram of the present invention;

[0023] Figure 2It is a diagram of the meta-learning network structure. DETAILED DESCRIPTION

[0024] The purpose of the present invention is to fully consider the characteristics of samples in diverse scenes, realize adaptive fusion, and enhance the semantic information of the fused image. The technical solution is as follows:

[0025] A method for fusion of infrared and visible light images based on meta-learning and semantic perception:

[0026] 1. Perform image registration on the original infrared and visible light images collected by the sensor through affine transformation;

[0027] 2. Input the registered infrared and visible light images into the fusion network and meta-learning network at the same time. The fusion network is divided into two paths: infrared path and visible light path. The infrared and visible light images are respectively subjected to the input convolution layer, dense block, and deep convolution to complete feature extraction and connect the features by channels, and then the final fused image is generated by the feature recovery module. In the feature extraction part, the input convolution layer size is 5×5, which is used to extract the shallow features of the image; the dense block is composed of three 3×3 convolution layers, where the input of the nth convolution layer is the cascade of all convolution outputs of the previous n-1 layers, and dense connection is achieved through layer-by-layer cascading; the dense block uses channel merging operations to extract as many deep features of the input image as possible while reducing the number of parameters, so that both shallow and deep features can be well utilized; the final fused image is then generated by the feature recovery module, which is composed of two 3×3 convolution layers and one 1×1 convolution layer;

[0028] 3. The input infrared and visible light images are processed by the meta-learning network at the same time. The meta-learning network structure can be found in Figure 2 , which consists of a 3×3 input convolution, three identical residual blocks and two 1×1 convolutions, where each residual block consists of a 3×3 convolution and a 1×1 convolution; after multiple layers of convolution, the meta-learning network maps the infrared or visible light image to the corresponding convolution kernel parameters, which guide the feature extraction of the infrared and visible light pathways in the fusion network;

[0029] 4. Input the image obtained by fusion of the fusion network into the segmentation network to obtain the semantic loss, and combine this loss with the content loss to guide the network training together. First, pre-train the fusion network, and then train the fusion network and the meta-learning network as a whole. The specific operations are as follows: (1) Remove the meta-learning network, replace the deep convolutional layer guided by the meta-learning network in the fusion network with a normal convolutional layer, and pre-train the above network; (2) Fix the shallow convolution parameters (input convolutional layer, dense block) of the pre-trained fusion network, and do not participate in the subsequent network model training; (3) Add the meta-learning network module to the above network model, and only train the modules other than the above shallow convolution (meta-learning network, deep convolutional layer guided by the meta-learning network in the fusion network, and fusion output module).

[0030] 5. Use the trained fusion network to fuse the images.

[0031] In the feature extraction part of step 2: the input convolution layer size is 5×5, which is used to extract shallow features of the image; the dense block is composed of three 3×3 convolution layers, where the input of the nth convolution layer is the cascade of all convolution outputs of the previous n-1 layers, and dense connection is achieved through layer-by-layer cascading; the dense block uses channel merging operations to extract as many deep features of the input image as possible while reducing the number of parameters, so that both shallow and deep features can be well utilized.

[0032] The detailed steps are as follows:

[0033] 4. Select the coordinates of three key points of the visible light image [x vi1 ,y vi1 ],[x vi2 ,y vi2 ],[x vi3 ,y vi3 ] corresponds to the unregistered infrared image coordinates [x ir1 ,y ir1 ],[x ir2 ,y ir2 ],[x ir3 ,y ir3 ], calculate the affine transformation matrix according to the coordinate points, generate the registered infrared image with the original infrared image as input, and convert all the original image pairs into the registered image pairs;

[0034] 5. The registered infrared and visible light images are simultaneously input into the fusion network and the meta-learning network. In the fusion network, the input image IR or VI is extracted through dense blocks to output the feature map x, x∈R N,H,W,C , where H, W, C represent the height, width and number of channels of the feature map respectively, and N represents the number of batches input to the network at the same time;

[0035] 6. The input samples of the meta-learning network undergo multiple layers of convolution, and the final output is the convolution kernel parameters required by the fusion network This means that for each pixel position of the feature map x, the meta-learning network generates an adaptive convolution kernel parameter of size 3×3, which corresponds to the three neighborhoods of the feature pixel position, thereby realizing adaptive deep feature extraction of each pixel position of the input feature map x; for a specific pixel point x of the feature map x [n,c,h,w] And the convolution kernel parameters adaptively generated by the meta-learning network The convolution operation is performed using the following formula:

[0036]

[0037] 4. Reconstruct the fused feature map to restore the fused image, input the fused image into the segmentation network, obtain the semantic loss, and design the corresponding loss function according to the information contribution requirements of infrared and visible light images in the fused image to drive the optimization update of the network model; the loss function consists of content loss and perceptual loss, and the total loss is: L = L content +θL S ; Using semantic loss L S To measure the semantic information contained in the fused image; the gap between the segmentation result R and the semantic label T is expressed as the semantic loss L S The training set is the TNO dataset. All images are normalized to [0, 1] before entering the network. The fusion network is first pre-trained, and then the fusion network and the meta-learning network are trained as a whole.

[0038] The fusion results are tested and the quality of the fused image is evaluated from the objective indicators: first, static indicators such as entropy EN, standard deviation SD, structural similarity SSIM and correlation coefficient CON are used, and the fusion results are evaluated using the average IoU of background, street lights, people and vehicles in combination with the segmentation task.

[0039] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0040] 1. Select the coordinates of three key points of the visible light image [x vi1 ,y vi1 ],[x vi2 ,y vi2 ],[x vi3 ,y vi3 ] corresponds to the unregistered infrared image coordinates [x ir1 ,y ir1 ],[x ir2 ,y ir2 ],[x ir3 ,y ir3], calculate the affine transformation matrix according to the coordinate points, generate the registered infrared image with the original infrared image as input, and convert all the original image pairs into the registered image pairs;

[0041] 2. The registered infrared and visible light images are simultaneously input into the fusion network and the meta-learning network. In the fusion network, the input image IR or VI is extracted through dense blocks to output the feature map x, x∈R N,H,W,C Where H, W, C represent the height, width and number of channels of the feature map respectively, and N represents the number of batches input into the network at the same time.

[0042] 3. The input samples of the meta-learning network undergo multiple layers of convolution, and the final output is the convolution kernel parameters required by the fusion network This means that for each pixel of the feature map x, the meta-learning network generates an adaptive convolution kernel parameter of size 3×3, which corresponds to the three neighborhoods of the feature pixel position, thereby achieving adaptive deep feature extraction of each pixel position of the input feature map x. [n,c,h,w] And the convolution kernel parameters adaptively generated by the meta-learning network The convolution operation is performed using the following formula:

[0043]

[0044] 4. Reconstruct the fused feature map to restore the fused image, and input the fused image into the segmentation network to obtain the semantic loss. In order to enable the network model to better complete the infrared and visible light fusion task, a corresponding loss function is designed to drive the optimization update of the network model based on the information contribution requirements of infrared and visible light images in the fused image. The loss function consists of content loss and perceptual loss, and the total loss is: L = L content +θL S Taking into full consideration the requirements of advanced visual tasks for fused images, the semantic loss L is used S To measure the semantic information contained in the fused image. The gap between the segmentation result R and the semantic label T is expressed as the semantic loss L S. The training set is the TNO dataset. All images are normalized to [0,1] before being input into the network. First, the fusion network is pre-trained, and then the fusion network and the meta-learning network are trained as a whole. The specific operations are as follows: (1) Remove the meta-learning network, replace the deep convolutional layer guided by the meta-learning network in the fusion network with the ordinary convolutional layer, and pre-train the above network; (2) Fix the shallow convolution parameters (input convolutional layer, dense block) of the pre-trained fusion network, and do not participate in the subsequent network model training; (3) Add the meta-learning network module to the above network model, and only train the modules other than the above shallow convolution (meta-learning network, deep convolutional layer guided by the meta-learning network in the fusion network, and fusion output module).

[0045] 5. Test the fusion results and evaluate the quality of the fused image from the objective indicators. First, static indicators such as entropy (EN), standard deviation (SD), structural similarity (SSIM) and correlation coefficient (CON) are used, and the fusion results are evaluated in combination with the segmentation task using mIoU (the average IoU of background, street lights, people, cars, etc.). The present invention has the highest CON and SSIM values, which means that it can adaptively fit the information of the input infrared and visible light images, so that the fused image is closest to the source image in terms of content relevance and structural information retention. At the same time, in the subsequent segmentation tasks, it also reflects the improvement of the segmentation task performance by the enhanced semantic information.

[0046] Table 1 Fusion solution performance comparison

[0047]

[0048] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed in the present invention should be covered within the protection scope of the present invention.

Claims

1. A method for fusion of infrared and visible light images based on meta-learning and semantic perception, characterized in that the steps are: as follows: 1) The original infrared and visible light images collected by the sensor are registered through affine transformation; 2) The registered infrared and visible light images are simultaneously input into the fusion network and the meta-learning network. The fusion network is divided into two pathways: the infrared pathway and the visible light pathway. The infrared and visible light images are respectively subjected to the input convolution layer, dense block, and deep convolution in the corresponding pathways to complete feature extraction and connect the features by channels. Then, two 3×3 convolution layers and one 1×1 convolution layer are used to restore the features to generate the final fused image. 3) The input infrared and visible light images are input to the fusion network and processed by the meta-learning network at the same time. The meta-learning network consists of a 3×3 input convolution, three identical residual blocks and two 1×1 convolutions, where each residual block consists of a 3×3 convolution and a 1×1 convolution. After multiple layers of convolution, the meta-learning network maps the infrared or visible light image to the corresponding convolution kernel parameters, which guide the feature extraction of the infrared and visible light pathways in the fusion network. 4) Input the image obtained by fusion of the fusion network into the segmentation network to obtain the semantic loss, and combine this loss with the content loss to jointly guide the training of the fusion network and the meta-learning network: first pre-train the fusion network, and then train the fusion network and the meta-learning network as a whole. The specific steps are: (1) Remove the meta-learning network, replace the deep convolution layer guided by the meta-learning network in the fusion network with a common convolution layer, and pre-train the fusion network and the meta-learning network; (2) Fix the shallow convolution parameters of the pre-trained fusion network, including the input convolution layer and the dense block, and do not participate in the subsequent network model training; (3) Add the meta-learning network module, and only train the parts other than the above shallow convolution, namely: the meta-learning network, the deep convolution layer guided by the meta-learning network in the fusion network, and the fusion output part; 5) Use the trained fusion network to fuse the images.

2. The infrared and visible light image fusion method based on meta-learning and semantic perception as claimed in claim 1, characterized in that: In step 2), the feature extraction part: the input convolution layer size is 5×5, which is used to extract shallow features of the image; The dense block consists of three 3×3 convolutional layers, where the input of the nth convolution layer is the cascade of all convolutional outputs of the previous n-1 layers. Through layer-by-layer cascading, dense connection is achieved; the dense block uses channel merging operations to extract as many deep features of the input image as possible while reducing the number of parameters, so that both shallow and deep features can be well utilized.

3. The infrared and visible light image fusion method based on meta-learning and semantic perception as claimed in claim 1, characterized in that: The detailed steps are as follows: 1) Select the coordinates of three key points of the visible light image [x vi1 ,y vi1 ],[x vi2 ,y vi2 ],[x vi3 ,y vi3 ] corresponds to the unregistered infrared image coordinates [x ir1 ,y ir1 ],[x ir2 ,y ir2 ],[x ir3 ,y ir3 ], calculate the affine transformation matrix according to the coordinate points, generate the registered infrared image with the original infrared image as input, and convert all the original image pairs into the registered image pairs; 2) The registered infrared and visible light images are simultaneously input into the fusion network and the meta-learning network. In the fusion network, the input image IR or VI is extracted through dense blocks to output the feature map x, x∈R N,H,W,C , where H, W, C represent the height, width and number of channels of the feature map respectively, and N represents the number of batches input to the network at the same time; 3) The input samples of the meta-learning network undergo multiple layers of convolution, and the final output is the convolution kernel parameters required by the fusion network. This means that for each pixel position of the feature map x, the meta-learning network generates an adaptive convolution kernel parameter of size 3×3, which corresponds to the three neighborhoods of the feature pixel position, thereby realizing adaptive deep feature extraction of each pixel position of the input feature map x; for a specific pixel point x of the feature map x [n,c, h ,w] And the convolution kernel parameters adaptively generated by the meta-learning network The convolution operation is performed using the following formula: 4) Reconstruct the fused feature map to restore the fused image, input the fused image into the segmentation network, obtain the semantic loss, and design the corresponding loss function according to the information contribution requirements of infrared and visible light images in the fused image to drive the optimization update of the network model; The loss function consists of content loss and perceptual loss, and the total loss is: L = L content +θL S ; Using semantic loss L S To measure the semantic information contained in the fused image; the gap between the segmentation result R and the semantic label T is expressed as the semantic loss L S The training set is the TNO dataset. All images are normalized to [0, 1] before entering the network. The fusion network is first pre-trained, and then the fusion network and the meta-learning network are trained as a whole.

4. The infrared and visible light image fusion method based on meta-learning and semantic perception as claimed in claim 1, characterized in that: The fusion results are tested and the quality of the fused image is evaluated from the objective indicators: first, static indicators such as entropy EN, standard deviation SD, structural similarity SSIM and correlation coefficient CON are used, and the fusion results are evaluated using the average IoU of background, street lights, people and vehicles in combination with the segmentation task.

Images