Infrared and Visible Image Fusion Method Based on Adaptive Weight Learning

Abstract

The present invention relates to an infrared and visible light image fusion method based on adaptive weight learning. A depth feature adaptive extraction module is constructed based on a pixel-level attention mechanism and adaptive generation of fusion weights, and a generator network is built based on the depth feature adaptive extraction module in a cross-level cascading manner; based on the idea of a siamese network, a dual-channel discriminator network is built; after the infrared and visible light images are subjected to a preliminary connection operation and input into the generator, a fusion image is generated, and the fusion of the infrared image and the visible light image is completed through the game confrontation between the generator and the discriminator; the present invention incorporates a pixel-level attention mechanism and a hierarchical cascading idea, which can enhance the network's depth feature extraction ability while reducing the number of network parameters, improve the generation quality of the fusion image of the infrared and visible light images, and achieve the fusion enhancement of the infrared image and the visible light image.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular, to an infrared and visible light image fusion method based on adaptive weight learning. Background Art

[0002] Infrared images are obtained by imaging the thermal radiation characteristics of objects and are not affected by lighting conditions. However, the ability of infrared images to capture scene detail information is not strong, and the target edge information is blurred; visible light images are obtained by imaging the reflection characteristics of objects and can capture the detailed texture information of the target, and the scene restoration ability is relatively strong. The imaging information of infrared images and visible light images is highly complementary. Therefore, fusing infrared and visible light images can make up for the deficiency of the information acquisition ability of a single sensor, improve environmental adaptability, and obtain high-quality scene imaging information.

[0003] Currently, infrared and visible light image fusion enhancement methods are mainly applied in face recognition, object detection, object tracking, defect detection, etc. However, due to problems such as complex background environments in infrared and visible light images, and complex combination methods between targets and between targets and the environment, it brings huge challenges to image fusion. With the help of artificial intelligence technology, satisfactory image fusion effects have not been achieved; for example, existing fusion algorithms based on adversarial neural networks have problems such as low utilization efficiency of deep features and insufficient restoration of texture information in fused images.

[0004] In view of the above deficiencies, the present invention proposes an infrared and visible light image fusion method based on adaptive weight learning, which can enhance the network's deep feature extraction ability while reducing the number of network parameters, and improve the quality of the fused image generated from infrared and visible light images. Summary of the Invention

[0005] The technical problem to be solved by the present invention is that existing image fusion algorithms based on adversarial neural networks have low utilization efficiency of deep features and insufficient restoration of texture detail information in fused images; the infrared and visible light image fusion method based on adaptive weight learning provided by the present invention incorporates a pixel-level attention mechanism and a hierarchical cascade idea, which can enhance the network's deep feature extraction ability while reducing the number of network parameters, and improve the quality of the fused image generated from infrared and visible light images.

[0006] To solve the above technical problems, the present invention provides an infrared and visible light image fusion method based on adaptive weight learning, including the following steps: constructing a generator and a discriminator based on a generative adversarial mechanism; performing a connection operation on the source data and then entering the generator network, generating a fused image through deep feature extraction and hierarchical feature fusion; the fused image enters a two-channel discriminator for discrimination; through the adversarial learning between the generator and the discriminator, the result of the fused image is output.

[0007] Preferably, in the step of constructing the generator and the discriminator based on the generative adversarial mechanism, the following process is included: constructing the generator, constructing a deep feature adaptive extraction module by using a pixel-level attention mechanism and a fusion weight adaptive learning mechanism, and constructing a generator network by using a cross-level cascading method with multiple deep feature adaptive extraction modules.

[0008] Preferably, the deep feature adaptive extraction module includes an attention sub-network, a non-attention sub-network, and a fusion weight adaptive generation sub-network.

[0009] Preferably, the attention sub-network and the non-attention sub-network extract and compress the image features, and the fusion weight adaptive generation sub-network adaptively generates a weight factor, and performs channel restoration and adaptive feature reconstruction on the features output by the attention sub-network and the non-attention sub-network.

[0010] Preferably, in the step of inputting the source data into the generator network after connection operation and generating a fused image through deep feature extraction and hierarchical feature fusion, the following process is specifically included: performing a connection operation on the source data to obtain an input image; extracting the gradient of the input image and inputting it into the generator as a whole; obtaining a fused image through the generator network.

[0011] Preferably, the source data includes an infrared image and a visible light image.

[0012] Preferably, in the step of constructing the generator and the discriminator based on the generative adversarial mechanism, the following process is included: constructing the discriminator, and constructing a dual-path discriminator by using a siamese network.

[0013] Preferably, the dual-path discriminators are respectively a visible light discriminator and an infrared discriminator. The visible light discriminator is used for discriminating between the fused image and the visible light image, and the infrared discriminator is used for discriminating between the fused image and the infrared image.

[0014] Preferably, the loss function of the generator is L G , the adversarial loss function is L adv , the gradient loss function is L grad , the structural similarity loss function is L ssim , then there is

[0015] L G = L adv + L grad + L ssim ,

[0016]

[0017]

[0018] Lssim -1 - SSIM,

[0019] where N represents the number of fused images, represents the classification result of the fused image, represents the value of the fake data that the generator hopes the discriminator to believe, β 1 , β 2 are constant parameters, I f , I ir , I vis respectively represent the fused image, infrared image, visible light image, H and W are the height and width of the input image, respectively represent the gradient value of the fused image, the gradient value of the infrared image, the gradient value of the visible light image, and SSIM is the calculated value using the image multi - scale structural similarity method.

[0020] Preferably, the loss function of the infrared discriminator is L D_ir , and the loss function of the visible light discriminator is L D_vis , then there is:

[0021]

[0022]

[0023] where N represents the number of fused images, a, b, and c respectively represent the true value of the infrared image, the true value of the fused image, and the true value of the visible light image, D IR (I ir ) represents the discrimination result of the infrared discriminator for the infrared image, D IR (I f ) represents the discrimination result of the infrared discriminator for the fused image, D vis (I vis ) represents the discrimination result of the visible light discriminator for the visible light image, D vis (I f ) represents the discrimination result of the visible light discriminator for the fused image.

[0024] Implementing the infrared and visible light image fusion method based on adaptive weight learning of the present invention has the following beneficial effects: A depth feature adaptive extraction module is constructed based on pixel-level attention mechanism and adaptive generation of fusion weights, and a generator network is built based on the depth feature adaptive extraction module in a cross-level cascading manner; Based on the idea of a siamese network, a dual-channel discriminator network is built; After performing a preliminary connection operation on the infrared and visible light images and inputting them into the generator, a fused image is generated, and the fusion generation of the infrared image and the visible light image is completed through the game confrontation between the generator and the discriminator; The present invention incorporates the pixel-level attention mechanism and the hierarchical cascading idea, which can enhance the network's ability to extract depth features while reducing the number of network parameters, improve the quality of the fused image generated from the infrared and visible light images, and achieve the fusion enhancement of the infrared image and the visible light image, which has great application value in face recognition, object detection, object tracking, defect detection, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 is a flowchart of the infrared and visible light image fusion method based on adaptive weight learning according to an embodiment of the present invention;

[0026] Figure 2 is a network framework diagram of the infrared and visible light image fusion method based on adaptive weight learning according to an embodiment of the present invention;

[0027] Figure 3 is a structural framework diagram of the generator of the infrared and visible light image fusion method based on adaptive weight learning according to an embodiment of the present invention;

[0028] Figure 4 is a structural schematic diagram of the depth feature adaptive extraction module of the infrared and visible light image fusion method based on adaptive weight learning according to an embodiment of the present invention;

[0029] Figure 5 is a structural framework diagram of the discriminator of the infrared and visible light image fusion method based on adaptive weight learning according to an embodiment of the present invention;

[0030] In the figure, 1: source data; 11: visible light image; 12: infrared image; 2: generator; 21: depth feature adaptive extraction module; 211: attention sub-network; 212: non-attention sub-network; 213: fusion weight adaptive generation sub-network; 22: cascading operation module; 23: DCB module; 3: fused image; 41: visible light discriminator; 42: infrared discriminator. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0031] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Apparently, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0032] Figure 1 is a flowchart of the infrared and visible light image fusion method based on adaptive weight learning according to the embodiments of the present invention; as Figure 1 shown, the infrared and visible light image fusion method based on adaptive weight learning provided by the embodiments of the present invention includes the following steps; Step S01: Construct a generator and a discriminator based on the generative adversarial mechanism; Step S02: After the source data is subjected to a concatenation operation, it enters the generator network, and a fused image is generated through deep feature extraction and hierarchical feature fusion; Step S03: The fused image enters a dual-channel discriminator for discrimination; Step S04: Through the adversarial learning between the generator and the discriminator, the fused image result is output.

[0033] Figure 2 is a network framework diagram of the infrared and visible light image fusion method based on adaptive weight learning according to the embodiments of the present invention; as Figure 2 shown, the network structure of the infrared and visible light image fusion method based on adaptive weight learning is as follows: A generator 2 and a discriminator are constructed based on the generative adversarial mechanism. In the construction of the generator 2, a deep feature adaptive extraction module 21 (DFAFE module) is constructed by adopting a pixel-level attention mechanism and a fusion weight adaptive learning mechanism, and a generator network is constructed by using a cascading operation module 22 to cascade multiple deep feature adaptive extraction modules 21 in a cross-layer cascading manner. In the construction of the discriminator, a twin network is adopted to construct a dual-path discriminator. The source data 1 (including the infrared image 12 and the visible light image 11) first undergoes a concatenation operation and then enters the generator network, a fused image 3 is generated through deep feature extraction and hierarchical feature fusion, and then it enters the dual-channel discriminator for discrimination. Finally, through the adversarial learning between the generator 2 and the discriminator, the fusion generation of high-quality infrared and visible light images 11 is achieved.

[0034] In the infrared and visible light image fusion method based on adaptive weight learning provided by the embodiments of the present invention, the deep feature adaptive extraction module 21 includes three sub-networks: an attention sub-network 211, a non-attention sub-network 212, and a fusion weight adaptive generation sub-network 213. The attention sub-network 211 and the non-attention sub-network 212 can extract and compress image features in channels. The fusion weight adaptive generation sub-network 213 can adaptively generate weight factors to perform channel restoration and adaptive feature reconstruction on the features extracted by the attention sub-network 211 and the non-attention sub-network 212, improving the generalization ability of network representation with only a very small increase in the number of parameters. The infrared and visible light image fusion method based on adaptive weight learning provided by the embodiments of the present invention adopts a hierarchical cascading method, connecting multiple deep feature adaptive extraction modules 21 across levels to form a multi-level feature extraction network, and realizing the extraction and fusion operations of hierarchical features.

[0035] In the infrared and visible light image fusion method based on adaptive weight learning provided by the embodiments of the present invention, under the guidance of the adversarial neural network design framework, a deep feature adaptive extraction module 21 based on pixel-level attention mechanism and adaptive generation of fusion weights is designed to efficiently extract image features. Then, multiple deep feature adaptive extraction modules 21 are cascaded across levels in a hierarchical manner to further improve the deep feature extraction and hierarchical feature fusion capabilities, thereby constructing a generator network. Then, a discriminator group is constructed based on a dual-path neural network. Finally, the infrared image 12 and the visible light image 11 are connected and input into the network, and the fusion generation of the infrared and visible light images is completed through the game confrontation between the generator 2 and the discriminator. Experimental results show that the fusion image 3 generated by the method proposed in the present invention retains the target information of the infrared image 12 and also retains more detailed information of the visible light image 11, and obtains good performance in both subjective evaluation and objective evaluation.

[0036] Figure 3It is the structural framework diagram of Generator 2 of the infrared and visible light image fusion method based on adaptive weight learning according to the embodiments of the present invention. In the infrared and visible light image fusion method based on adaptive weight learning provided by the embodiments of the present invention, the generator network is constructed by cascading multiple levels with the deep feature adaptive extraction module 21 as the basic unit, which can effectively fuse hierarchical features and improve the overall feature representation ability of the network. The data flow of the generator network is as follows: The infrared image 12 and the visible light image 11 are first connected to obtain the input image, and then the gradient of the connected input image is extracted and input into Generator 2 as a whole, and then the fused image 3 is obtained through the feature extraction network constructed by the cross-level cascading of the deep feature adaptive extraction module 21. Among them, the DCB module 23 (dense connection module) in the network includes 2 convolutional layers, and the ReLU function is used for activation after each convolutional layer.

[0037] Figure 4 It is the structural schematic diagram of the deep feature adaptive extraction module 21 of the infrared and visible light image fusion method based on adaptive weight learning according to the embodiments of the present invention; in the infrared and visible light image fusion method based on adaptive weight learning provided by the embodiments of the present invention, the deep feature adaptive extraction module 21 is mainly used to efficiently extract feature information and is composed of an attention sub-network 211, a non-attention sub-network 212, and a fusion weight adaptive generation sub-network 213. The attention sub-network 211 is composed of two 3×3 convolutions and one 1×1 convolution, and a pixel-level attention mechanism is added to the first 3×3 convolution, and the pixel-level attention mechanism can assign corresponding weights to different channels, which can improve the ability of the network to extract important information. The non-attention sub-network 212 is used for feature mapping with a 3×3 convolution and channel reconstruction with a 1×1 convolution in order to maximize the preservation of the original information.

[0038] In order to filter redundant information in the feature network and reduce the number of parameters, a fusion weight adaptive generation sub-network 213 is set in the deep feature adaptive network. Through the self-learning process, this network can dynamically allocate weights to the attention sub-network 211 and the non-attention sub-network 212, thereby weakening the ineffective attention features and enabling the attention sub-network 211 and the non-attention sub-network 212 to achieve adaptive balance.

[0039] In addition, the input feature x n-1 After passing through the attention sub-network 211 and the non-attention sub-network 212 respectively, the number of channels of the features obtained at their respective output ends is reduced to half of the input channels, which are x' n , x'' n . And x' n , x'' n After entering the fusion weight adaptive generation sub-network 213 as input features respectively, the 1×1 convolution in it is used for x'n , x″ n Perform channel boosting so that x′ n , x″ n has the same number of channels as X n Then, x′ n , x″ n and γ 1 , γ 2 are multiplied element - by - element and added, and finally input after 1×1 convolution. Through the above - mentioned feature channel transformation and adaptive reconstruction operations, the generalization ability of the network representation is improved with only a very small increase in the number of parameters, the ability of adaptive extraction of deep features is improved, which is beneficial to capturing complex backgrounds and target detail information in infrared and visible light images, and high - quality fusion generation of efficient infrared and visible light images is realized.

[0040] Figure 5 is the structural framework diagram of the discriminator of the infrared and visible light image fusion method based on adaptive weight learning in the embodiment of the present invention; in the infrared and visible light image fusion method based on adaptive weight learning provided by the embodiment of the present invention, the discriminator is used to form an adversarial game with the generator 2. The present invention adopts a dual - discriminator structure, namely a visible - light discriminator 41 (Discriminator - VIS) and an infrared discriminator 42 (Discriminator - IR), which are respectively used to discriminate between the fused image and visible - light and infrared images. The discriminator adopts a siamese network structure and shares network parameters. Its single - branch network adopts a three - layer convolution structure, the stride is set to 2, and the output is a scalar, that is, a binary classification result.

[0041] In the infrared and visible light image fusion method based on adaptive weight learning provided by the embodiment of the present invention, the loss function of the generator 2 is L G , the adversarial loss function is L adv , the gradient loss function is L grad , and the structural similarity loss function is L ssim , then there is,

[0042] L G =L adv |L grad |L ssim ,

[0043]

[0044]

[0045] L ssim =1 - SSIM,

[0046] where N represents the number of fused images, represents the classification result of the fused image, The false data value that the generator hopes the discriminator to believe, β 1 , β 2 are constant parameters, I f , I ir , I vis respectively represent the fused image, the infrared image, and the visible light image. H and W are the height and width of the input image. They respectively represent the gradient value of the fused image, the gradient value of the infrared image, and the gradient value of the visible light image. SSIM is the calculated value using the multi-scale structural similarity method of the image. To make the adversarial game between the generator and the discriminator reach an equilibrium state and realize the mutual promotion and synchronous learning and evolution of the generator and the discriminator, the loss function L G of the generator 2 in the embodiments of the present invention consists of three parts. One is the adversarial loss function L adv , which is used to control the adversarial process between the generator and the discriminator, so that the fused image can obtain more infrared and visible light detail information; the second is the gradient loss function L grad used to control the learning ability of the generator for the thermal radiation information of the infrared image and the texture detail information of the visible light image; the third is the structural similarity loss function L ssim , which is used to control the structural similarity between the fused image and the infrared and visible light images.

[0047] In the infrared and visible light image fusion method based on adaptive weight learning provided by the embodiments of the present invention, the loss function of the infrared discriminator 42 is L D_ir , and the loss function of the visible light discriminator 41 is L D_vis , then there is:

[0048]

[0049]

[0050] In the formula, N represents the number of fused images, a, b, and c respectively represent the true value of the infrared image, the true value of the fused image, and the true value of the visible light image, D IR (I ir ) represents the discrimination result of the infrared discriminator for the infrared image, D IR (I f ) represents the discrimination result of the infrared discriminator for the fused image, D vis (I vis ) represents the discrimination result of the visible light discriminator for the visible light image, D vis (I f ) represents the discrimination result of the visible light discriminator for the fused image. The embodiments of the present invention adopt a dual discriminator design, which can more effectively capture the thermal radiation information of the infrared image and the texture detail information of the visible light image.

[0051] The infrared and visible light image fusion method based on adaptive weight learning in the embodiments of the present invention is used to experimentally verify the infrared and visible light image fusion performance of the method of the present invention. In the verification experiment, the publicly available image fusion dataset TNO is used. Approximately 20 pairs of infrared and visible light images are randomly selected from it, and the data is expanded to 10,200 pairs of images through data augmentation. Among them, the training dataset and the verification dataset are constructed according to a ratio of 7:3. In addition, 10 pairs of infrared and visible light images are randomly selected from the TNO dataset to construct the test dataset.

[0052] In this experiment, the comparison algorithms selected are also neural network image fusion algorithms based on the adversarial mechanism, namely FusionGAN (Fusion Generative Adversarial Network), DDcGAN (Conditional Generative Adversarial Network), and FusionDN (Improved Fusion Generative Adversarial Network).

[0053] In order to objectively evaluate the fused images obtained by the above different fusion methods, 4 indicators are selected for objective evaluation, namely information entropy (EN), standard deviation (SD), mutual information (MI), and multi-scale structural similarity (MS-SSIM). The larger the EN, the more information there is in the fused image and the more image details are retained. The larger the SD, the higher the quality of the image and the clearer it is. MI measures the similarity between images. The larger the MI, the more information of the source images is retained in the fused image and the better the quality. MS-SSIM measures the similarity between the image and the source image.

[0054] The experimental results are statistically shown in the following table. It can be seen from the table that the infrared and visible light image fusion algorithm proposed in the present invention has improvements in multiple indicators compared with the control algorithms, indicating the effectiveness of the algorithm proposed in the present invention.

[0055]

[0056] In summary, by implementing the infrared and visible light image fusion method based on adaptive weight learning of the present invention, the generated fused image retains the target information of the infrared image and also retains more detailed information of the visible light image, and obtains good performance in both subjective evaluation and objective evaluation.

[0057] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements for some of the technical features. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An infrared and visible light image fusion method based on adaptive weight learning, characterized in that, it includes the following steps: Construct a generator and a discriminator based on the generative adversarial mechanism; After the source data is subjected to a concatenation operation, it enters the generator network, and a fused image is generated through deep feature extraction and hierarchical feature fusion; The fused image enters a two-channel discriminator for discrimination; Through the adversarial learning between the generator and the discriminator, the fused image result is output; When constructing the generator, a deep feature adaptive extraction module is constructed based on a pixel-level attention mechanism and a fusion weight adaptive learning mechanism, and the generator network is constructed by using a cross-level cascading method with multiple such deep feature adaptive extraction modules; The deep feature adaptive extraction module includes an attention sub-network, a non-attention sub-network, and a fusion weight adaptive generation sub-network; the attention sub-network and the non-attention sub-network extract and channel compress image features, and the fusion weight adaptive generation sub-network adaptively generates a weight factor and performs channel restoration and adaptive feature reconstruction on the features output by the attention sub-network and the non-attention sub-network; The attention sub-network is composed of two 3×3 convolutions and one 1×1 convolution; among them, the 3×3 convolution is used for feature mapping, and the 1×1 convolution is used for channel reconstruction, and a pixel-level attention mechanism is added to the first 3×3 convolution, and the pixel-level attention mechanism is used to assign corresponding weights to different channels to improve the ability of the attention sub-network to extract important information; Input feature x n-1 After passing through the attention sub-network and the non-attention sub-network respectively, the number of channels of the features obtained at their respective output ends is reduced to half of the input channels, and x' is obtained n and x″ n After x' n and x″ n are respectively used as input features to enter the fusion weight adaptive generation sub-network, the channel is promoted through the 1×1 convolution pair therein, so that the number of channels of x' n and x″ n is consistent with that of X n The output x' n and x″ n are multiplied by the corresponding weight factors and added element by element, and finally output after 1×1 convolution.

2. The infrared and visible light image fusion method based on adaptive weight learning according to claim 1, characterized in that, in the step of generating a fused image through deep feature extraction and hierarchical feature fusion after the source data is subjected to a concatenation operation and enters the generator network, specifically it includes the following process: The source data is subjected to a concatenation operation to obtain an input image; The input image is extracted with gradients and then input into the generator as a whole; A fused image is obtained through the generator network.

3. The infrared and visible light image fusion method based on adaptive weight learning according to claim 1, characterized in that, the source data includes an infrared image and a visible light image.

4. The infrared and visible light image fusion method based on adaptive weight learning according to claim 1, characterized in that, in the step of constructing a generator and a discriminator based on the generative adversarial mechanism, it includes the following process: construct the discriminator, and use a siamese network to construct a two-way discriminator.

5. The infrared and visible light image fusion method based on adaptive weight learning according to claim 4, characterized in that, the two-way discriminator is respectively a visible light discriminator and an infrared discriminator, the visible light discriminator is used for the discrimination between the fused image and the visible light image, and the infrared discriminator is used for the discrimination between the fused image and the infrared image.

6. The infrared and visible light image fusion method based on adaptive weight learning according to claim 5, characterized in that, The loss function of the generator is L G , and the adversarial loss function is L adv , and the gradient loss function is L grad , and the structural similarity loss function is L ssim , then L G = L adv + L grad + L ssim , L ssim = 1 - SSIM, Where N represents the number of fused images, represents the classification result of the fused image, represents the value of the false data that the generator hopes the discriminator can believe, β 1 and β 2 are constant parameters, I f and I ir and I vis respectively represent the fused image, the infrared image, and the visible light image. H and W are the height and width of the input image, respectively represent the gradient value of the fused image, the gradient value of the infrared image, and the gradient value of the visible light image. SSIM is the calculated value using the image multi-scale structural similarity method.

7. The infrared and visible light image fusion method based on adaptive weight learning according to claim 6, characterized in that, The loss function of the infrared discriminator is L D_ir and the loss function of the visible light discriminator is L D_vis then we have: Wherein, N represents the number of fused images, a, b, and c respectively represent the true values of the infrared image, the true value of the fused image, and the true value of the visible light image, and D IR (I ir ) represents the discrimination result of the infrared discriminator for the infrared image, and D IR (I f ) represents the discrimination result of the infrared discriminator for the fused image, and D vis (I vis ) represents the discrimination result of the visible light discriminator for the visible light image, and D vis (I f ) represents the discrimination result of the visible light discriminator for the fused image.

Images