An unsupervised infrared image and visible light image fusion method

Abstract

The application discloses a kind of fusion methods of unsupervised infrared image and visible light image, based on image fusion model, image fusion model includes multiscale feature extraction module and feature reconstruction module, specifically includes: (1) infrared image and visible light image are input to feature extraction module and are extracted to multiscale feature;The feature extraction module includes two feature extraction paths of infrared image and visible light image, each path includes three parallel branches;Infrared image and visible light image are input to respective corresponding feature extraction path;(2) the feature of two feature extraction paths in feature extraction module is connected in series along channel dimension, then input to feature reconstruction module to obtain fusion image;(3) the model proposed is constantly optimized under the guidance of loss function, and finally ideal fusion image is obtained.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image processing, and particularly relates to the field of infrared image and visible light image fusion. Specifically, the present application provides an unsupervised infrared and visible light image fusion method based on multi-scale feature extraction. BACKGROUND

[0002] Image fusion is a research hotspot in the field of image processing, and infrared image and visible light image fusion is the most widely used. Infrared image has low resolution and lacks texture details, but it is sensitive to radiation information, can capture significant target information, and is not affected by working environment (such as rain, fog and other bad weather). Visible light image has detailed texture detail information and high image resolution, but it is easily affected by working environment [1-4] . The properties of these two images have certain complementarity. Fusing these two images can obtain a comprehensive fusion image with rich information and consistent with human visual perception, which is conducive to the interpretation of scene information and promotes downstream applications such as target recognition, detection and tracking.

[0003] The key to image fusion is to extract important features from source images and then fuse the extracted features. Existing image fusion methods mainly include two categories: traditional methods and deep learning-based methods. Although the traditional method has achieved good fusion effect, it excessively depends on the extraction of manual features and the design of fusion rules. For complex source images, the design of feature extraction and fusion rules will also become complex, resulting in time-consuming and difficult to implement. The emergence of deep learning brings new development opportunities for image fusion.

[0004] The powerful feature extraction and data representation capabilities of deep learning can overcome the problems existing in traditional methods, so it has emerged in the field of image fusion and has been widely used [5],[6] . At present, the method based on deep learning has achieved some good results in the field of image fusion. A representative method is the CNN-based image fusion method proposed by Liu et al. [7] It is well known that it is difficult to have ground truth for infrared image and visible light image fusion, so the method based on GAN is applied to this field. Ma et al. [8] proposed FusionGAN, which opened up the application of GAN in the field of infrared image and visible light image fusion. In order to solve the problem of detail loss and fuzzy target edge in FusionGAN, Ma et al. [9]ResNetFusion is proposed. This method designs two kinds of loss functions, namely detail loss and target edge enhancement, which improves the quality of detail information and sharpens the target edge. In order to solve the problem that the fused image is only similar to one of the source images, Ma et al.

[10] DDcGAN is proposed, which has a generator and two discriminators, and can make the generator fully trained.

[0005] However, there are still some shortcomings in the deep learning based method: 1) it is a challenge to maintain the balance between the generator and the discriminator in the GAN based fusion method; 2) the feature extraction is not comprehensive, ignoring the importance of extracting different scale features, which affects the fusion performance. 3) some existing unified frameworks can fuse multiple types of images, but this kind of method is not strong in pertinence, which will limit the fusion performance in the process of fixed type image fusion.

[0006] References:

[0007] [1] Mao Yaqi. An image fusion method based on improved DDcGAN model [P]. Shanxi Province: CN111833282A, 2020-10-27.

[0008] [2] Yang Y, Liu J, Huang S, et al. Infrared and visible image fusion based on modal feature fusion network and dual visual decision [C] / / 2021 IEEE International Conference on Multimedia and Expo (ICME). IEEE, 2021: 1-6.

[0009] [3] Li Yang, Yang Haitao, Kong Zhuo, Zhang Changgong, Wang Jinyu. Review of pixel-level infrared and visible image fusion methods [J / OL]. Computer Engineering and Applications: 1-12.

[0010] [4] Ma J, Ma Y, Li C. Infrared and visible image fusion methods and applications: A survey [J]. Information Fusion, 2019, 45: 153-178.

[0011] [5] Huang J, Le Z, Ma Y, et al. Mgmdcgan: Medical image fusion using multi-generator multi-discriminator conditional generative adversarial network [J]. IEEE Access, 2020, 8: 55145-55157.

[0012] [6] Zhang Y, Liu Y, Sun P, et al. IFCNN: A general image fusion framework based on convolutional neural network [J]. Information Fusion, 2020, 54: 99-118.

[0013] [7] Liu Y, Chen X, Cheng J, et al. A medical image fusion method based on convolutional neural networks [C] / / 2017 20th international conference on information fusion (Fusion). IEEE, 2017: 1-7.

[0014] [8] Ma J, Yu W, Liang P, et al. FusionGAN: A generative adversarial network for infrared and visible image fusion [J]. Information fusion, 2019, 48: 11-26.

[0015] [9] Ma J, Liang P, Yu W, et al. Infrared and visible image fusion via detail preserving adversarial learning [J]. Information Fusion, 2020, 54: 85-98.

[0016]

[10] Ma J, Xu H, Jiang J, et al. DDcGAN: A dual-discriminator conditional generative adversarial network for multi-resolution image fusion[J]. IEEE Transactions on Image Processing, 2020, 29: 4980-4995. SUMMARY

[0017] The purpose of the present application is to overcome the deficiencies in the prior art, provide an unsupervised infrared image and visible light image fusion method based on multi-scale feature extraction. This method can extract features of different scales, so that the information in the source image is retained to a greater extent, and the training balance problem existing in the fusion method based on GAN is overcome.

[0018] The purpose of the present application is realized by the following technical solutions:

[0019] An unsupervised infrared image and visible light image fusion method based on an image fusion model, the image fusion model comprising a multi-scale feature extraction module and a feature reconstruction module, the fusion method comprising:

[0020] (1) inputting the infrared image and the visible light image into the multi-scale feature extraction module for multi-scale feature extraction; the multi-scale feature extraction module comprises two feature extraction paths for the infrared image and the visible light image, and each path comprises three parallel branches; the infrared image and the visible light image are input into the corresponding feature extraction paths;

[0021] (2) concatenating the features extracted by the two feature extraction paths in the multi-scale feature extraction module along the channel dimension, and then inputting them into the feature reconstruction module to obtain a fusion image;

[0022] (3) continuously optimizing the image fusion model under the guidance of the loss function, and finally obtaining an ideal fusion image.

[0023] Further, in step (1), the infrared image and the visible light image are input into the corresponding feature extraction paths in the multi-scale feature extraction module, and the results obtained by the first layer convolution of the image fusion model are input into the three parallel branches corresponding to each feature extraction path to extract features of different scales; wherein the sizes of the convolution kernels used by the three parallel branches in each path are 1x1, 3x3 and 5x5 respectively; the outputs of the three parallel branches of each path are concatenated along the channel dimension and input into a 1x1 convolution layer A; the output of the 1x1 convolution layer A and the first layer convolution of the image fusion model are concatenated and input into a 3x3 convolution layer.

[0024] Further, the feature reconstruction module is composed of two parallel branches, and the convolution kernel size of the two branches is 1x1 and 3x3 respectively, so as to make full use of the extracted information.

[0025] Further, the loss function L in step (3) is composed of content loss L T and structural similarity loss L con , and is defined as follows: ssim

[0026] L T = L con + aL ssim

[0027] Wherein, a is a weight parameter for balancing the content loss and the structural similarity loss, a = 100;

[0028] The structural similarity loss is used to measure the similarity between two images, and is defined as follows:

[0029]

[0030] Wherein, I ir , I vi , and I f respectively represent the infrared image, the visible light image and the fusion image; the calculation formula of the structural similarity index SSIM is as follows:

[0031]

[0032] Wherein, I1 and I2 respectively represent two images to be calculated, μ represents the average value, σ represents the standard deviation, C1 and C2 are stability coefficients, is the cross-correlation between I1 and I2;

[0033] The content loss L con includes gradient loss L gra and intensity loss L int part, and is defined as follows:

[0034] L con = L int + bL gra

[0035] Wherein, b is a weight parameter, and b is set to 10; the gradient loss L gra and the intensity loss L int are defined as follows:

[0036]

[0037] ​

[0038] H and W represent the width and height of the input image respectively; ‖·‖ F represents the matrix F-norm, represents the gradient operation; η, ξ, η1 and ξ1 are weight parameters, which are set to 1, 0.5, 5 and 3 respectively.

[0039] The application further provides an electronic device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the fusion method of the unsupervised infrared image and the visible light image when executing the program.

[0040] The application further provides a computer readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of the fusion method of the unsupervised infrared image and the visible light image when executed by a processor.

[0041] The application further provides a computer readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of the fusion method of the unsupervised infrared image and the visible light image when executed by a processor.

[0042] Compared with the prior art, the technical scheme of the application has the beneficial effects that:

[0043] 1. The application extracts features of different sizes by using a multi-scale feature extraction module, and then connects the features of different scales to enrich the information of each layer. The features extracted by the feature extraction module are input into a feature reconstruction module, and the model is optimized under the guidance of a loss function, so that a fusion image with rich information is obtained.

[0044] 2. The application is an end-to-end image fusion framework, which can obtain a fusion image without manual feature extraction and subsequent operations, and is convenient to operate.

[0045] 3. The application applies the idea of multi-scale feature extraction to the fusion of infrared images and visible light images, thereby enriching the feature extraction of the source images to improve the image fusion performance.

[0046] 4. The application designs a specific loss function composed of structural similarity loss and content loss to guide model optimization. The fusion result obtained by the model contains rich texture information and conforms to human visual effects. BRIEF DESCRIPTION OF DRAWINGS

[0047] Figure 1 is a schematic diagram of the overall process of the method of the application.

[0048] Figures 2a to 2d Each of the images in the figure is the input infrared image, the input visible light image and the image output by the application method from top to bottom. DETAILED DESCRIPTION

[0049] The application will be further described below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely intended to explain the application and are not intended to limit the application.

[0050] The application provides a fusion method of unsupervised infrared images and visible light images, based on an image fusion model, which includes a multi-scale feature extraction module and a feature reconstruction module.

[0051] The application first extracts features of source images by two paths. Each path includes three different branches, which use different sizes of convolution kernels (equivalent to expanding the receptive field) to extract multi-level structural features. Then, the features are concatenated to enrich the information of each layer. Next, the features obtained by the two branches are concatenated and a series of convolution operations are performed to obtain the final fusion image.

[0052] Specifically, in order to obtain an information-rich fusion image, the process of the embodiment mainly includes the following steps, as shown in Figure 1

[0053] First, the infrared image and the visible light image are input into the corresponding feature extraction paths for feature extraction. In each feature extraction path, there are three parallel branches with different sizes of convolution kernels. The three branches can extract features of different sizes, and the features extracted by the three branches can be concatenated to enrich the information of each layer.

[0054] Specifically, the two feature extraction paths are the infrared image and visible light image feature extraction paths. The infrared image and the visible light image are input into the corresponding feature extraction paths in the multi-scale feature extraction module, and the results obtained by the first layer convolution of the image fusion model are input into the three parallel branches corresponding to each feature extraction path to extract features of different scales. The sizes of the convolution kernels used in the three parallel branches in each path are 1x1, 3x3 and 5x5, respectively. The outputs of the three parallel branches of each path are concatenated along the channel dimension and input into a 1x1 convolution layer A. The output of the 1x1 convolution layer A and the first layer convolution of the image fusion model are concatenated and input into a 3x3 convolution layer.

[0055] Secondly, the features extracted by the infrared image and visible light feature extraction paths are concatenated along the channel dimension and then input into the feature reconstruction module. The feature reconstruction module mainly has two parallel branches. The convolution kernels used in the two branches are different (1x1 and 3x3, respectively), and the purpose of this is to maximize the use of the extracted information. The output of the feature reconstruction module is the fusion image.

[0056] ​Finally, the proposed image fusion model is constantly optimized under the guidance of the loss function, and finally the ideal fusion image is obtained. Specifically:

[0057] A suitable loss function helps guide the optimization training of the model. The loss function L of the embodiment is as follows: T The content loss L con and the structural similarity loss L ssim are composed of two parts, which are defined as follows:

[0058] L T = I con + αL ssim ,

[0059] wherein, α is a weight parameter for balancing the content loss and the structural similarity loss, and in the embodiment, α = 100.

[0060] The structural similarity loss is used to measure the similarity between two images, and is defined as follows:

[0061]

[0062] wherein, I ir , I vi , and I f represent the infrared image, the visible light image, and the fusion image respectively. The calculation formula of the structural similarity index (SSIM) is as follows:

[0063]

[0064] wherein, I1 and I2 represent two images to be calculated respectively, μ represents the average value, σ represents the standard deviation, C1 and C2 are stability coefficients, is the cross-correlation between I1 and I2.

[0065] The content loss includes gradient loss L gra and intensity loss L int , which are defined as follows:

[0066] L con = L int + βL gra ,

[0067] wherein, β is a weight parameter, and in the patent, β is set to 10. The definitions of the gradient loss L gra and the intensity loss L int are as follows:

[0068]

[0069]

[0070] H and W represent the width and height of the input image, respectively. F denotes the matrix F-norm, denotes the gradient operation. η, ξ, η1 and ξ1 are weight parameters, which are set to 1, 0.5, 5 and 3, respectively.

[0071] In addition, the present application has verified the effectiveness of the multi-scale feature extraction module, the feature reconstruction module and the loss function through a large number of experiments. Feature extraction is an indispensable part in infrared image and visible image fusion, and has a crucial influence on the fusion performance. Of course, feature reconstruction and loss function are equally important, and the three complement each other and jointly act to achieve an ideal result. Figures 2a to 2d Four groups of fusion results of the present application are shown.

[0072] The present application is not limited to the above-described embodiments. The above description of specific embodiments is intended to describe and illustrate the technical solutions of the present application, and the specific embodiments described above are merely illustrative and not restrictive. Without departing from the purpose of the present application and the scope protected by the claims, those skilled in the art can make many forms of specific changes under the inspiration of the present application, and these all belong to the protection scope of the present application.

Claims

1. An unsupervised fusion method for infrared and visible light images, based on an image fusion model, the image fusion model including a multi-scale feature extraction module and a feature reconstruction module, characterized in that, Fusion methods include: (1) Input the infrared image and the visible light image into the multi-scale feature extraction module for multi-scale feature extraction; the multi-scale feature extraction module includes two feature extraction paths for the infrared image and the visible light image, and each path includes three parallel branches; the infrared image and the visible light image are input into their respective feature extraction paths; (2) The features extracted by the two feature extraction paths in the multi-scale feature extraction module are concatenated along the channel dimension and then input into the feature reconstruction module to obtain the fused image; the feature reconstruction module consists of two parallel branches, and the convolution kernel sizes used by the two branches are 1×1 and 3×3, respectively, to maximize the use of the extracted information; (3) The image fusion model is continuously optimized under the guidance of the loss function, and finally the ideal fused image is obtained.

2. The method for fusing unsupervised infrared and visible light images according to claim 1, characterized in that, In step (1), the infrared image and the visible light image are input into the corresponding feature extraction paths in the multi-scale feature extraction module. The results obtained by the first convolution of the image fusion model are input into the three parallel branches corresponding to each feature extraction path to extract features at different scales. The convolution kernel sizes used by the three parallel branches in each path are 1×1, 3×3 and 5×5, respectively. The outputs of the three parallel branches of each path are concatenated along the channel dimension and input into a 1×1 convolutional layer A. The output of the 1×1 convolutional layer A and the first convolution of the image fusion model are concatenated and input into a 3×3 convolutional layer.

3. The method for fusing unsupervised infrared and visible light images according to claim 1, characterized in that, Loss function in step (3) Content loss and structural similarity loss Composition, defined as follows: ； in, α It is a weighting parameter that balances content loss and structural similarity loss. α =100; Structural similarity loss is used to measure the similarity between two images, and is defined as follows: ； in, , ,and These represent infrared images, visible light images, and fused images, respectively; the formula for calculating the structural similarity index (SSIM) is as follows: ； in, and These represent the two images to be calculated. This represents the average value. Indicates standard deviation, and It is the stability coefficient. yes and The cross-correlation between them; Content loss Including gradient loss and intensity loss Part, defined as follows: ； in, β It is a weight parameter. β Set to 10; gradient loss and intensity loss The definition is as follows: ； ； H and W represent the width and height of the input image, respectively; Describing the F-norm of a matrix, Indicates gradient operation; , , and These are weight parameters, set to 1, 0.5, 5, and 3 respectively.

4. An electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the unsupervised infrared image and visible light image fusion method according to any one of claims 1 to 3.

5. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the unsupervised infrared image and visible light image fusion method according to any one of claims 1 to 3.

Images