Infrared visible light image fusion method of multi-scale attention convolutional neural network

Abstract

The invention discloses an infrared visible light image fusion method of a multi-scale attention convolutional neural network, and the method comprises the steps: connecting a multi-scale global average pooling attention unit and a parallel structure channel attention unit in series to form a multi-scale attention module, and then connecting the multi-scale attention module to the convolutional neural network in parallel, and constructing a multi-scale attention convolutional neural network. Infrared and visible light images are processed in batches and input into a convolutional neural network for training, original features of the infrared image and original features of the visible light image are extracted, difference is carried out on the original features to obtain complementary features, and the complementary features are input into a multi-scale attention module for processing to obtain complementary features with attention weights; and superposing and feeding back to the convolutional neural network to obtain a total feature, and reconstructing the total feature into a fused image through an image reconstruction device. According to the method, different scale features can be highlighted, image fusion is realized, image redundant information in the fusion process is effectively inhibited, and the infrared and visible light image fusion quality and information amount are remarkably improved.

Description

Technical Field

[0001] The present invention belongs to the field of image fusion, and in particular relates to an infrared visible light image method based on multi-scale attention and convolutional neural network. Background Art

[0002] Infrared images are images obtained by infrared cameras in the infrared band, and visible light images are images obtained by cameras in the visible light band visible to the human eye. Due to theoretical and technical limitations, images taken by a single modality sensor cannot effectively and comprehensively describe the imaging scene. Infrared sensors can highlight salient targets by capturing the thermal radiation emitted by objects, but infrared images have poor description capabilities for scene texture details and are easily affected by noise. In contrast, visible light sensors capture reflected light information, and their images usually contain rich texture information, but are very sensitive to the environment. Therefore, by extracting the saliency information of infrared images and the detail information of visible light images, the useful information of the two images is concentrated in one image, and the final fusion result is a fused image with prominent targets and rich details, that is, the fusion result combines the advantages of the two types of images.

[0003] Infrared and visible light image fusion is a typical multi-source fusion task, which has a wide range of applications in computer vision, such as target detection, tracking, and recognition. Existing fusion methods usually use the same transformation or representation for different types of source images in the fusion process. However, this method is not suitable for infrared and visible light images, because the thermal radiation area in the infrared image and the detail content in the visible light image are two completely different representations. In addition, the fusion rules in the existing methods are mostly designed manually, and the fusion process is becoming more and more complicated, with problems such as high difficulty in implementation and high computational cost. This traditional method is very time-consuming and labor-intensive to model the extracted features and design fusion rules, so its performance is greatly limited.

[0004] Image fusion methods based on deep learning can solve the limitations of traditional methods to a certain extent, thereby achieving better fusion performance, but there are still some problems that need to be improved. The deep learning methods currently used are basically convolutional neural network methods. Although convolutional neural networks are very efficient, this method interrupts the correlation of images and cannot extract relevant information from many aspects of images well, such as the complementary features of infrared and visible light images, which has a certain impact on subsequent image fusion. Although the convolutional network extracts features quickly, it has poor feature recognition and is prone to merging redundant features into the fused image, which deteriorates the visual quality of the fused image. In addition, the inappropriate design of the loss function causes color distortion in the fused image. Summary of the invention

[0005] In order to overcome the shortcomings of the prior art, the present invention proposes an infrared-visible light image fusion method of a multi-scale attention convolutional neural network. Convolutional neural networks are used to extract features of infrared and visible light images from different sources, and preliminarily extract significant information and detail information in the image. However, if only convolutional neural networks are used, the problem of incomplete feature information will be caused. The information extracted from one image has no correlation, which has a great impact on subsequent results. Therefore, the present invention not only uses a convolutional neural network, but also connects a multi-scale attention module in parallel after this convolutional neural network to process the complementary features of infrared and visible light images, and remove redundant features, which makes up for this defect of the convolutional neural network, and the obtained fusion result is more natural, and finally an ideal fusion result is obtained. The loss function module is also used to perform multi-faceted loss constraints, so that the fusion image is successfully fused without color distortion and spectral pollution, which makes up for the defects of the convolutional neural network in fusing infrared and visible light images.

[0006] To achieve the above object, the present invention adopts the following technical solutions:

[0007] A method for fusion of infrared and visible light images using a multi-scale attention convolutional neural network, comprising:

[0008] S1. Construct a multi-scale attention convolutional neural network: connect a multi-scale global average pooling attention unit and a parallel structure channel attention unit in series to form a multi-scale attention module, connect the multi-scale attention module in parallel to the convolutional neural network, and construct a multi-scale attention convolutional neural network; the multi-scale attention convolutional neural network also includes an illumination discrimination auxiliary network and a loss function module, the illumination discrimination auxiliary network runs in parallel with the convolutional neural network and the multi-scale attention module, and the visible light image is input into the auxiliary network to calculate the illumination probability weight and assign it to the loss function module; the loss function module is used to perform loss calculation on the generated fusion image to guide the training and optimization of the multi-scale attention convolutional neural network;

[0009] S2. Training the illumination discrimination auxiliary network: Input the batch processed visible light images into the illumination discrimination auxiliary network to calculate the illumination probability weight and assign it to the loss function module;

[0010] S3. Training a multi-scale attention convolutional neural network: batch process infrared and visible light images and input them into a convolutional neural network to obtain the original features of the infrared image and the visible light image, perform a differential operation on the original features to obtain two complementary features, input the two complementary features into a multi-scale attention module for processing to obtain two complementary features with attention weights, then superimpose the two complementary features with attention weights and feed them back to the convolutional neural network to add the original features of the infrared image and the original features of the visible light image to obtain the total features of the infrared image and the total features of the visible light image, splice the total features of the infrared image and the total features of the visible light image in the channel dimension, and reconstruct them into a fused image through the image reconstructor of the convolutional neural network, use the loss function module to calculate the loss of the fused image, guide the multi-scale attention convolutional neural network to optimize the quality of the fused image, and obtain a multi-scale attention convolutional neural network that has completed training;

[0011] S4. Use the trained multi-scale attention convolutional neural network to fuse infrared and visible light images to obtain the optimal fused image.

[0012] Furthermore, the batched infrared and visible light images input to the convolutional neural network have a resolution of n×n, are paired images and have the same size.

[0013] Furthermore, the convolutional layer in the convolutional neural network is used to extract the original features of the infrared and visible light images. , They are the original feature representations of the input infrared image and the original feature representation of the visible light image, respectively. There are five convolutional layers in the convolutional neural network used to extract the original features of the infrared and visible light images. The first one uses a 1×1 convolutional layer, followed by four 3×3 convolutional layers. All convolutional layers use activation functions to process nonlinear relationships.

[0014] Furthermore, the convolutional neural network is connected in parallel with the multi-scale attention module to perform differential operations on the original features of the infrared image and the visible light image, and a feature subtraction method is adopted to obtain complementary features, which are and .

[0015] Furthermore, the multi-scale global average pooling attention unit of the multi-scale attention module processes complementary features as follows:

[0016] (1)

[0017] (2)

[0018] (3)

[0019] (4)

[0020] in, Represented as a 1×1 global average pooling layer, Represented as a 2×2 global average pooling layer, It is represented as a 4×4 global average pooling layer, which performs pooling operations of different scales on the two complementary features of the input. , , and , , They represent the infrared complementary features and visible light complementary features after being processed by the global average pooling layers of different scales in the multi-scale global average pooling attention unit, respectively. The superscripts 1, 2, and 4 represent the corresponding , , Global average pooling layer. represents the sigmoid function, Represents interpolation. and They respectively represent the infrared complementary features and visible light complementary features after the multi-scale global average pooling attention unit.

[0021] Furthermore, in the multi-scale attention module, the complementary features with attention weights after the multi-scale global average pooling attention unit are multiplied element by element with the original complementary features, and then passed through the parallel structure channel attention unit. The parallel structure channel attention unit is connected in series after the multi-scale global average pooling attention unit, which processes the output of the multi-scale global average pooling attention unit. The features processed by the parallel structure channel attention unit are then multiplied element by element with the original complementary features, which can be expressed as:

[0022] (5)

[0023] (6)

[0024] Among them, PCAU represents the parallel structure channel attention unit for processing, Represents element-wise multiplication of features.

[0025] After the complementary features with attention weights of the parallel structure channel attention unit are multiplied element by element with the original complementary features, the processed complementary features are added to the original features of the infrared image and the original features of the visible light image as supplementary information to obtain the total features of the infrared image and the total features of the visible light image, which are expressed as:

[0026] (7)

[0027] (8)

[0028] in, Represent the total features of infrared images and visible light images respectively, It represents the element-wise addition of features.

[0029] The fusion method of image features is to splice the infrared total features and the visible light total features in the channel dimension to achieve the fusion of the total features of the two, which is expressed as:

[0030] (9)

[0031] in, Represent the total features of infrared images and visible light images respectively, represents the fusion feature, and C(·) refers to the connection of the total features of the infrared image and the total features of the visible light image in the channel dimension.

[0032] Image Reconstructor It consists of 5 convolutional layers, which use activation functions to handle nonlinear relationships. Image Reconstructor Recover the fused image from the fused features , expressed as:

[0033] (10)

[0034] Multi-scale attention convolutional neural network This entire network calculates the intensity loss of the input infrared image and the input visible light image , auxiliary strength loss and gradient loss To construct the loss function module, the loss function module is used as an overall loss To guide the training of the entire multi-scale attention convolutional neural network, it is defined as:

[0035] (11)

[0036] in, , , are weight parameters that characterize the importance of strength loss, auxiliary strength loss, and gradient loss respectively.

[0037] Furthermore, the strength loss It is expressed as the sum of the infrared image intensity loss and the visible light image intensity loss with weight coefficients. It can be expressed as:

[0038] (12)

[0039] (13)

[0040] (14)

[0041] in, and are the height and width of the input image, respectively. for Norm. , , They are respectively represented as infrared image, visible light image and fused image. and represents the intensity loss of infrared image and visible light image, and The weight coefficients representing the intensity of infrared images and visible light images are provided by the output of the illumination discrimination auxiliary network.

[0042] Furthermore, the illumination discrimination auxiliary network consists of four convolutional layers and two fully connected layers. The convolutional layer uses an activation function to process nonlinear relationships. The two fully connected layers calculate the illumination probability based on the illumination information to generate the weight coefficients of the infrared image intensity and the visible light image intensity. and .

[0043] Furthermore, the auxiliary strength loss Maintaining the optimal intensity distribution for the fused image is expressed as:

[0044] (15)

[0045] in, Represents an element-wise maximal selection.

[0046] Furthermore, the gradient loss It is expressed as:

[0047] (16)

[0048] in, Represents the gradient operator that measures the texture information of an image. It refers to absolute operation.

[0049] Compared with the prior art, the present invention has the following advantages:

[0050] (1) Compared with traditional methods, the present invention does not require manual design of extremely complex fusion rules, thus reducing the feature extraction and fusion process.

[0051] (2) Compared with the convolutional neural network, the present invention adds a multi-scale attention module and an illumination discrimination auxiliary network, which makes the information of the fused image more comprehensive, realizes the full fusion of infrared feature information and visible light feature information, highlights features of different scales, and significantly increases the amount of information.

[0052] (3). The infrared and visible light image fusion network method proposed in this invention can eliminate redundant features in complex scenes and improve the image fusion quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 Schematic diagram of the multi-scale attention convolutional neural network structure;

[0054] Figure 2 Schematic diagram of the convolutional neural network structure;

[0055] Figure 3 This is a schematic diagram of the auxiliary network structure for illumination discrimination;

[0056] Figure 4 Schematic diagram of the multi-scale attention module structure;

[0057] Figure 5 Schematic diagram of parallel structure channel attention unit;

[0058] Figure 6 The infrared and visible light image fusion results are shown in Figure 1, where (a) is the infrared image, (b) is the visible light image, and (c) is the fused image.

[0059] Among them: infrared image 1, visible light image 2, convolutional neural network 3, multi-scale attention module 4, parallel structure channel attention unit 5, multi-scale global average pooling attention unit 6, loss function module 7, training illumination discrimination auxiliary network 8, channel dimension splicing module 9, image reconstructor 10, fused image 11, first convolution layer 12, second convolution layer 13, fully connected layer 14, horizontal and vertical dimension average pooling layer 15, 1×1 convolution layer 16, normalization layer 17, one-dimensional global average pooling layer 18, 3×3 convolution layer 19. DETAILED DESCRIPTION

[0060] The present invention is further described below with reference to the accompanying drawings and specific examples.

[0061] like Figure 1As shown, the multi-scale attention convolutional neural network includes: a convolutional neural network 3, a multi-scale attention module 4, a training illumination discrimination auxiliary network 8 and a loss function module 7. The multi-scale attention module 4 is composed of a multi-scale global average pooling attention unit 6 and a parallel structure channel attention unit 5 in series. Training illumination discrimination auxiliary network 8: Input the visible light image 2 into the illumination discrimination auxiliary network to generate illumination discrimination probabilities during the day and at night, and assign illumination probability weights to the loss function module 7. The multi-scale attention convolutional neural network is trained; the batch-processed infrared image 1 and the visible light image 2 are input into the convolutional neural network 3 to obtain the original features of the infrared image and the original features of the visible light image, the original features are differentially operated to obtain two complementary features, and the two complementary features are input into the multi-scale attention module 4, wherein the two complementary features are first input into the multi-scale global average pooling attention unit 6 in the multi-scale attention module 4 for processing, and then processed by the parallel structure channel attention unit 5, and two complementary features with attention weights are obtained after processing, and then the two complementary features with attention weights are superimposed and fed back to the convolutional neural network 3, and added to the original features of the infrared image and the original features of the visible light image to obtain the total features of the infrared image and the total features of the visible light image, and the total features of the infrared image and the total features of the visible light image are reconstructed into a fused image 11 by the channel dimension splicing module 9 and the image reconstructor 10, and in this training process, the loss function module 7 is used to calculate the loss of the fused image, and finally a multi-scale attention convolutional neural network that has completed training is obtained.

[0062] like Figure 2 As shown, the convolutional neural network 3 is mainly used to extract the original features of the infrared and visible light images, preliminarily extract the significant information and detail information in the images, and input the infrared and visible light images into the convolutional neural network 3 for training. The first convolutional layer 12 extracts the original features of the infrared image and the original features of the visible light image, and then the original features are differentially operated to obtain two complementary features, which are processed by the multi-scale attention module 4 to obtain complementary features with attention weights. The complementary features with attention weights are superimposed and fed back to the convolutional neural network 3 and added to the original features of the infrared image and the original features of the visible light, and then feature fusion is realized in the channel dimension stitching module 9, and then reconstructed into a fused image 11 by the image reconstructor 10, and finally a multi-scale attention convolutional neural network that has completed training is obtained.

[0063] like Figure 3 As shown, the illumination discrimination auxiliary network consists of four second convolutional layers 13 and two fully connected layers 14. The second convolutional layer 13 uses an activation function to process nonlinear relationships. The two fully connected layers 14 calculate the illumination probability based on the illumination information to generate weight coefficients for the infrared image intensity and the visible light image intensity.

[0064] like Figure 4As shown, the differential operation of the infrared image and visible light image features is performed to obtain complementary features, which are input into the multi-scale attention module 4. The global information of the image at different spatial scales is first captured by the multi-scale global average pooling attention unit 6. The pooling results of different scales are mapped to weight distributions through activation functions, and weighted processing is performed according to the different importance of complementary features to highlight the significant part of the complementary feature information. The channel weighted complementary features after the multi-scale global average pooling layer are multiplied element by element with the original complementary features, and then passed through the parallel structure channel attention unit 5. Through parallel grouping, local features can be learned in each group to further deepen the role of the attention mechanism. The features after the parallel structure channel attention unit 5 are then multiplied element by element with the original difference complementary features, and the results are added as supplementary information to the original features of the infrared image and the original features of the visible light image extracted by the convolution layer.

[0065] like Figure 5 As shown, in the parallel structure channel attention unit 5, the first branch of the original complementary features processed by the multi-scale global average pooling attention unit passes through the horizontal and vertical dimension average pooling layer 15, is cascaded, and is normalized in the normalization layer 17 after being element-wise multiplied by the 1×1 convolution layer 16 and the original complementary features. The second branch is the original complementary features passing through the 3×3 convolution layer 19 and the one-dimensional global average pooling layer 18. The two branches adopt a parallel processing strategy. The features of the first branch are normalized and then passed through the one-dimensional global average pooling layer 18. The second branch captures the multi-scale feature representation through the 3×3 convolution layer 19. The two channel attention feature maps are aggregated by element-wise multiplication, so that different cross-channel interaction features can be achieved between the two parallel routes of the two branches. The outputs of the parallel processing are multiplied by matrix dot product operations. Finally, the output feature map in each group is calculated as a set of two generated spatial attention weight values, and then multiplied element-by-element with the original complementary features for output.

[0066] like Figure 6 As shown, the fusion result of the infrared and visible light images of the present invention is given, where (a) is the infrared image, (b) is the visible light image, and (c) is the fused image. It can be seen that more detail information is retained and the target is highly significant.

[0067] It should be pointed out that Figure 2 and Figure 3 Only one possible convolution composition method is given. Based on the fusion method proposed in the present invention, there are multiple possible implementation methods of the convolution layer. As long as the basic features of the present invention are met, they should be included in the protection scope of the present invention. Although the present invention has been disclosed as above with preferred embodiments, it is not intended to limit the present invention.

[0068] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for infrared and visible light image fusion based on a multi-scale attention convolutional neural network, characterized in that: The following steps are involved: S1. Construct a multi-scale attention convolutional neural network: connect a multi-scale global average pooling attention unit and a parallel structure channel attention unit in series to form a multi-scale attention module, connect the multi-scale attention module in parallel to the convolutional neural network, and construct a multi-scale attention convolutional neural network; the multi-scale attention convolutional neural network also includes an illumination discrimination auxiliary network and a loss function module, and the illumination discrimination auxiliary network runs in parallel with the convolutional neural network and the multi-scale attention module; S2. Training the illumination discrimination auxiliary network: Input the batch processed visible light images into the illumination discrimination auxiliary network to calculate the illumination probability weight and assign it to the loss function module; S3. Training a multi-scale attention convolutional neural network: batch process infrared and visible light images and input them into a convolutional neural network to obtain the original features of the infrared image and the visible light image, perform a differential operation on the original features to obtain two complementary features, input the two complementary features into a multi-scale attention module for processing to obtain two complementary features with attention weights, then superimpose the two complementary features with attention weights and feed them back to the convolutional neural network to add the original features of the infrared image and the original features of the visible light image to obtain the total features of the infrared image and the total features of the visible light image, splice the total features of the infrared image and the total features of the visible light image in the channel dimension, and reconstruct them into a fused image through the image reconstructor of the convolutional neural network, use the loss function module to calculate the loss of the fused image, guide the multi-scale attention convolutional neural network to optimize the quality of the fused image, and obtain a multi-scale attention convolutional neural network that has completed training; S4. Use the trained multi-scale attention convolutional neural network to fuse infrared and visible light images to obtain the optimal fused image.

2. According to claim 1, the infrared-visible light image fusion method of a multi-scale attention convolutional neural network is characterized in that: The convolutional layer in the convolutional neural network is used to extract the original features of infrared and visible light images. , are the original feature representation of the input infrared image and the original feature representation of the visible light image, respectively. The multi-scale attention module extracts and processes the complementary features of the infrared image and the visible light image, respectively. and .

3. The infrared-visible light image fusion method of a multi-scale attention convolutional neural network according to claim 1, characterized in that: The multi-scale attention module is composed of a multi-scale global average pooling attention unit and a parallel structure channel attention unit connected in series.

4. The infrared-visible light image fusion method of a multi-scale attention convolutional neural network according to claim 3 is characterized in that: The multi-scale global average pooling attention unit of the multi-scale attention module processes complementary features as follows: (1) (2) (3) (4) in, Represented as a 1×1 global average pooling layer, Represented as a 2×2 global average pooling layer, It is represented as a 4×4 global average pooling layer, which performs pooling operations of different scales on the two complementary features of the input. , , and , , They represent the infrared complementary features and visible light complementary features after being processed by the global average pooling layers of different scales in the multi-scale global average pooling attention unit, respectively. The superscripts 1, 2, and 4 represent the corresponding , , Global average pooling layer, represents the sigmoid function, represents interpolation, and They respectively represent the infrared complementary features and visible light complementary features after the multi-scale global average pooling attention unit.

5. The infrared-visible light image fusion method of a multi-scale attention convolutional neural network according to claim 4 is characterized in that: In the multi-scale attention module, the complementary features with attention weights passed through the multi-scale global average pooling attention unit are multiplied element by element with the original complementary features, and then passed through the parallel structure channel attention unit. The parallel structure channel attention unit is connected in series after the multi-scale global average pooling attention unit, and the output of the multi-scale global average pooling attention unit is processed. The features processed by the parallel structure channel attention unit are then multiplied element by element with the original complementary features, which is expressed as: (5) (6) Among them, PCAU represents the parallel structure channel attention unit for processing, Represents element-wise multiplication of features.

6. The infrared-visible light image fusion method of a multi-scale attention convolutional neural network according to claim 1, characterized in that: In the multi-scale attention module, after the complementary features with attention weights of the parallel structure channel attention unit are multiplied element by element with the original complementary features, the processed complementary features are added as supplementary information to the original features of the infrared image and the original features of the visible light image to obtain the total features of the infrared image and the total features of the visible light image, which are expressed as: (7) (8) in, Represent the total features of infrared images and visible light images respectively, It represents the element-wise addition of features.

7. The infrared-visible light image fusion method of a multi-scale attention convolutional neural network according to claim 1, characterized in that: The infrared total feature and the visible light total feature are spliced ​​in the channel dimension to achieve the fusion of the total features, which is expressed as: (9) in, Represent the total features of infrared images and visible light images respectively, represents the fusion feature, and C(·) refers to the connection of the total features of the infrared image and the total features of the visible light image in the channel dimension.

8. The infrared-visible light image fusion method of a multi-scale attention convolutional neural network according to claim 1, characterized in that: Image Reconstructor Recover the fused image from the fused total features , expressed as: (10)。 9. The infrared-visible light image fusion method of a multi-scale attention convolutional neural network according to claim 8, characterized in that: The image reconstructor is a component of a convolutional neural network, which is composed of a series of convolutional layers.

10. The infrared-visible light image fusion method of a multi-scale attention convolutional neural network according to claim 1, characterized in that: The multi-scale attention convolutional neural network calculates the intensity loss of the input infrared image and the input visible light image. , auxiliary strength loss and gradient loss To construct the loss function module, the loss function module is used as an overall loss To guide the training of the multi-scale attention convolutional neural network, it is defined as: (11) in, , , are weight parameters that characterize the importance of strength loss, auxiliary strength loss, and gradient loss respectively.

11. The infrared-visible light image fusion method of a multi-scale attention convolutional neural network according to claim 10, characterized in that: The strength loss It is expressed as the sum of the infrared image intensity loss and the visible light image intensity loss with weight coefficients, expressed as: (12) (13) (14) in, and are the height and width of the input image, respectively. for norm, , , They are respectively represented as infrared image, visible light image and fused image. and represents the intensity loss of infrared image and visible light image, and The weight coefficients representing the intensity of infrared images and visible light images are provided by the output of the illumination discrimination auxiliary network.

12. The infrared-visible light image fusion method of a multi-scale attention convolutional neural network according to claim 10, characterized in that: The auxiliary strength loss Maintaining the optimal intensity distribution for the fused image is expressed as: (15) in, Represents an element-wise maximal selection.

13. The infrared-visible light image fusion method of a multi-scale attention convolutional neural network according to claim 10, characterized in that: The gradient loss It is expressed as: (16) in, represents the gradient operator that measures the texture information of the image, It refers to absolute operation.

Images