An infrared and visible light image fusion method of a multi-scale attention convolutional neural network

Abstract

The application discloses an infrared and visible light image fusion method based on a multi-scale attention convolutional neural network, which comprises the following steps: connecting a multi-scale global average pooling attention unit and a parallel structure channel attention unit in series to form a multi-scale attention module, and then connecting the multi-scale attention module in parallel on a convolutional neural network to construct a multi-scale attention convolutional neural network; batch processing infrared and visible light images and inputting the images into the convolutional neural network for training; extracting original features of the infrared and visible light images, and differentiating the original features to obtain complementary features; inputting the complementary features into the multi-scale attention module for processing to obtain the complementary features with attention weights; and superimposing and feeding back the complementary features with the attention weights to the convolutional neural network to obtain total features, and reconstructing the total features into a fusion image by using an image reconstructor. The application can highlight different scale features, realize image fusion, effectively suppress image redundant information in the fusion process, and significantly improve the infrared and visible light image fusion quality and information quantity.

Description

Technical Field

[0001] This invention belongs to the field of image fusion, and particularly relates to an infrared and visible light image method based on multi-scale attention and convolutional neural networks. Background Technology

[0002] Infrared images are obtained by infrared cameras capturing radiation in the infrared band, while visible light images are obtained by cameras capturing visible light bands visible to the human eye. Due to theoretical and technological limitations, images captured by a single modal sensor cannot effectively and comprehensively describe the imaging scene. Infrared sensors can highlight salient targets by capturing thermal radiation emitted by objects, but infrared images have poor ability to describe scene texture details and are easily affected by noise. Conversely, visible light sensors capture reflected light information, and their images usually contain rich texture information, but are very sensitive to the environment. Therefore, by extracting salient information from infrared images and detailed information from visible light images, the useful information from both images is combined into a single image. The final fusion result is a fused image with prominent targets and rich details, combining the advantages of both types of images.

[0003] Infrared and visible light image fusion is a typical multi-source fusion task with wide applications in computer vision, such as object detection, tracking, and recognition. Existing fusion methods typically use the same transformation or representation for different types of source images during the fusion process. However, this approach is not suitable for infrared and visible light images because the thermal radiation regions in infrared images and the details in visible light images are two distinct representations. Furthermore, most fusion rules in existing methods are manually designed, leading to increasingly complex fusion processes, high implementation difficulty, and high computational costs. This traditional approach is very time-consuming and labor-intensive in modeling extracted features and designing fusion rules, thus significantly limiting its performance.

[0004] Deep learning-based image fusion methods can overcome the limitations of traditional methods to some extent, thus achieving better fusion performance. However, some issues still require improvement. Currently used deep learning methods are primarily convolutional neural networks (CNNs). While CNNs are highly efficient, they disrupt image correlations and fail to effectively extract multifaceted information, such as complementary features between infrared and visible light images, which negatively impacts subsequent image fusion. Although CNNs extract features quickly, they are poor at feature recognition, easily incorporating redundant features into the fused image, thus degrading its visual quality. Furthermore, inappropriately designed loss functions can cause color distortion in the fused image. Summary of the Invention

[0005] To overcome the shortcomings of existing technologies, this invention proposes a multi-scale attention convolutional neural network (CNN) method for infrared and visible light image fusion. While CNNs are used to extract features from infrared and visible light images from different sources, initially extracting salient and detailed information, using only CNNs results in incomplete feature information; information extracted from one image lacks relevance, significantly impacting subsequent results. Therefore, this invention not only uses a CNN but also connects a multi-scale attention module in parallel to it to process complementary features of infrared and visible light images and remove redundant features, overcoming this deficiency of CNNs. The resulting fusion is more natural, ultimately achieving an ideal fusion result. Furthermore, a loss function module is employed for multi-faceted loss constraints, ensuring successful image fusion without color distortion or spectral contamination, thus compensating for the shortcomings of CNNs in fusing infrared and visible light images.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] An infrared-visible light image fusion method based on a multi-scale attention convolutional neural network includes:

[0008] S1. Constructing a multi-scale attention convolutional neural network: A multi-scale attention module is formed by concatenating multi-scale global average pooling attention units and parallel channel attention units. This multi-scale attention module is then connected in parallel to a convolutional neural network to construct the multi-scale attention convolutional neural network. The multi-scale attention convolutional neural network also includes an illumination discrimination auxiliary network and a loss function module. The illumination discrimination auxiliary network runs in parallel with the convolutional neural network and the multi-scale attention module. Visible light images are input into this auxiliary network to calculate illumination probability weights, which are then assigned to the loss function module. The loss function module is used to perform loss calculations on the generated fused image to guide the training and optimization of the multi-scale attention convolutional neural network.

[0009] S2. Training the illumination discrimination auxiliary network: Input the batched visible light images into the illumination discrimination auxiliary network to calculate the illumination probability weights and assign them to the loss function module;

[0010] S3. Training a multi-scale attention convolutional neural network: Batch processing of infrared and visible light images and inputting them into the convolutional neural network to obtain the original features of the infrared and visible light images. The original features are then differentially processed to obtain two complementary features. These two complementary features are then input into the multi-scale attention module for processing to obtain two complementary features with attention weights. The two complementary features with attention weights are then superimposed and fed back into the convolutional neural network and added to the original features of the infrared and visible light images to obtain the total features of the infrared and visible light images. The total features of the infrared and visible light images are then concatenated along the channel dimension and reconstructed into a fused image by the image reconstructor of the convolutional neural network. The loss function module is used to calculate the loss of the fused image to guide the multi-scale attention convolutional neural network to optimize the quality of the fused image, resulting in a fully trained multi-scale attention convolutional neural network.

[0011] S4. Use the trained multi-scale attention convolutional neural network to fuse infrared and visible light images to obtain the optimal fused image.

[0012] Furthermore, the batch of infrared and visible light images input into the convolutional neural network has a resolution of n×n, and consists of paired images of the same size.

[0013] Furthermore, the convolutional layers in the convolutional neural network are used to extract the original features of infrared and visible light images, denoted as... , The original feature representations of the input infrared image and the original feature representation of the visible light image are respectively. The convolutional neural network has five convolutional layers used to extract the original features of the infrared and visible light images respectively. The first layer is a 1×1 convolutional layer, and the following four layers are 3×3 convolutional layers. The convolutional layers all use activation functions to handle nonlinear relationships.

[0014] Furthermore, a multi-scale attention module is connected in parallel to the convolutional neural network to perform a difference operation on the original features of the infrared image and the visible light image, and complementary features are obtained by feature subtraction. These complementary features are... and .

[0015] Furthermore, the multi-scale global average pooling attention unit of the multi-scale attention module represents the complementary feature processing as follows:

[0016] (1)

[0017] (2)

[0018] (3)

[0019] (4)

[0020] in, Represented as a 1×1 global average pooling layer, Represented as a 2×2 global average pooling layer, This is represented as a 4×4 global average pooling layer, which performs pooling operations at different scales on the two complementary input features. , , and , , These represent the infrared complementary features and visible light complementary features after processing by global average pooling layers of different scales in the multi-scale global average pooling attention unit, respectively. The superscripts 1, 2, and 4 indicate the corresponding... , , Global average pooling layer. This represents the sigmoid function. This indicates interpolation. and These represent the infrared complementary features and visible light complementary features, respectively, after passing through the multi-scale global average pooling attention unit.

[0021] Furthermore, in the multi-scale attention module, the complementary features with attention weights, after passing through the multi-scale global average pooling attention unit, are multiplied element-wise with the original complementary features. This multiplication then passes through a parallel channel attention unit, which is concatenated after the multi-scale global average pooling attention unit. The parallel channel attention unit processes the output of the multi-scale global average pooling attention unit. The processed features are then multiplied element-wise with the original complementary features, as shown below:

[0022] (5)

[0023] (6)

[0024] Here, PCAU represents parallel channel attention unit processing. This indicates that the features are multiplied element by element.

[0025] After the complementary features with attention weights in the parallel channel attention unit are multiplied element-wise with the original complementary features, the resulting complementary features are added as supplementary information to the original features of the infrared image and the original features of the visible light image, resulting in the total features of the infrared image and the total features of the visible light image, represented as:

[0026] (7)

[0027] (8)

[0028] in, These represent the total features of an infrared image and the total features of a visible light image, respectively. This indicates that the features are added element by element.

[0029] The image feature fusion method involves concatenating the total infrared features and the total visible light features along the channel dimension to achieve the fusion of their total features, as expressed as:

[0030] (9)

[0031] in, These represent the total features of an infrared image and the total features of a visible light image, respectively. The fusion feature, C(·), refers to the connection between the total features of the infrared image and the total features of the visible light image in the channel dimension.

[0032] Image Reconstructor It consists of 5 convolutional layers, each employing an activation function to handle non-linear relationships. Image Reconstructor Recover the fused image from the fused features. , is represented as:

[0033] (10)

[0034] The multi-scale attention convolutional neural network calculates intensity loss from both the input infrared image and the input visible light image. auxiliary strength loss and gradient loss To construct the loss function module, the loss function module serves as a total loss. To guide the training of the entire multi-scale attention convolutional neural network, it is defined as:

[0035] (11)

[0036] in, , , These are weight parameters that respectively characterize the importance of intensity loss, auxiliary intensity loss, and gradient loss.

[0037] Furthermore, the strength loss This is expressed as the sum of the weighted intensity loss of the infrared image and the intensity loss of the visible light image. It is represented as:

[0038] (12)

[0039] (13)

[0040] (14)

[0041] in, and These are the height and width of the input image, respectively. for Norm. , , These are represented as infrared image, visible light image, and fused image, respectively. and This represents the intensity loss in infrared images and the intensity loss in visible light images. and The weighting coefficients representing the intensity of the infrared image and the intensity of the visible light image are provided by the output of the illumination discrimination auxiliary network.

[0042] Furthermore, the illumination discrimination auxiliary network consists of four convolutional layers and two fully connected layers. The convolutional layers use activation functions to handle non-linear relationships. The two fully connected layers calculate the illumination probability based on the illumination information to generate weight coefficients for the infrared image intensity and the visible light image intensity. and .

[0043] Furthermore, the aforementioned auxiliary strength loss To maintain the optimal intensity distribution in the fused image, it is expressed as:

[0044] (15)

[0045] in, This represents the maximum selection element by element.

[0046] Furthermore, the gradient loss Represented as:

[0047] (16)

[0048] in, This represents the gradient operator for measuring image texture information. This refers to absolute operations.

[0049] Compared with the prior art, the present invention has the following advantages:

[0050] (1). Compared with traditional methods, the present invention does not require the manual design of extremely complex fusion rules, thus reducing the feature extraction and fusion process.

[0051] (2). Compared with convolutional neural networks, this invention adds a multi-scale attention module and an illumination discrimination auxiliary network, which makes the information of the fused image more comprehensive, realizes the full fusion of infrared feature information and visible light feature information, highlights features at different scales, and significantly improves the amount of information.

[0052] (3). The infrared and visible light image fusion network method proposed in this invention can eliminate redundant features in complex scenes and improve the image fusion quality. Attached Figure Description

[0053] Figure 1 This is a schematic diagram of a multi-scale attention convolutional neural network structure.

[0054] Figure 2 This is a schematic diagram of a convolutional neural network structure;

[0055] Figure 3 A schematic diagram of the illumination discrimination auxiliary network structure;

[0056] Figure 4 This is a schematic diagram of the multi-scale attention module structure;

[0057] Figure 5 This is a schematic diagram of a parallel channel attention unit;

[0058] Figure 6 The images shown are the result of infrared and visible light image fusion, where (a) is the infrared image, (b) is the visible light image, and (c) is the fused image.

[0059] The components are: 1. Infrared image; 2. Visible light image; 3. Convolutional neural network; 4. Multi-scale attention module; 5. Parallel structure channel attention unit; 6. Multi-scale global average pooling attention unit; 7. Loss function module; 8. Training illumination discrimination auxiliary network; 9. Channel dimension stitching module; 10. Image reconstructor; 11. Fusion image; 12. First convolutional layer; 13. Second convolutional layer; 14. Fully connected layer; 15. Horizontal and vertical dimension average pooling layer; 16. 1×1 convolutional layer; 17. Normalization layer; 18. One-dimensional global average pooling layer; 19. 3×3 convolutional layer. Detailed Implementation

[0060] The present invention will be further described below with reference to the accompanying drawings and specific examples.

[0061] like Figure 1As shown, the multi-scale attention convolutional neural network includes: a convolutional neural network 3, a multi-scale attention module 4, a training illumination discrimination auxiliary network 8, and a loss function module 7. The multi-scale attention module 4 is composed of a multi-scale global average pooling attention unit 6 and a parallel channel attention unit 5 connected in series. The training illumination discrimination auxiliary network 8: The visible light image 2 is input into the illumination discrimination auxiliary network to generate illumination discrimination probabilities for daytime and nighttime, which are then assigned illumination probability weights to the loss function module 7. The multi-scale attention convolutional neural network is trained. Batch-processed infrared image 1 and visible light image 2 are input into convolutional neural network 3 to obtain the original features of the infrared image and the original features of the visible light image. The original features are then differentially processed to obtain two complementary features. These two complementary features are input into multi-scale attention module 4. The two complementary features are first processed by multi-scale global average pooling attention unit 6 in multi-scale attention module 4, and then by parallel channel attention unit 5. After processing, two complementary features with attention weights are obtained. These two complementary features with attention weights are then superimposed and fed back into convolutional neural network 3, added to the original features of the infrared image and the original features of the visible light image to obtain the total features of the infrared image and the total features of the visible light image. The total features of the infrared image and the total features of the visible light image are then processed by channel dimension stitching module 9 and reconstructed into a fused image 11 by image reconstructor 10. During this training process, loss function module 7 is used to calculate the loss on the fused image, and finally, the trained multi-scale attention convolutional neural network is obtained.

[0062] like Figure 2 As shown, the convolutional neural network 3 is mainly used to extract the original features of infrared and visible light images, initially extracting salient and detailed information from the images. The infrared and visible light images are input into the convolutional neural network 3 for training. The first convolutional layer 12 extracts the original features of the infrared image and the original features of the visible light image. Then, the original features are differentially processed to obtain two complementary features. After being processed by the multi-scale attention module 4, complementary features with attention weights are obtained. The complementary features with attention weights are superimposed and fed back to the convolutional neural network 3. After being added to the original features of the infrared image and the original features of the visible light image, feature fusion is achieved in the channel dimension stitching module 9. Then, the image is reconstructed into a fused image 11 by the image reconstructor 10. Finally, the multi-scale attention convolutional neural network that has been trained is obtained.

[0063] like Figure 3 As shown, the illumination discrimination auxiliary network consists of four second convolutional layers 13 and two fully connected layers 14. The second convolutional layers 13 use activation functions to handle nonlinear relationships. The two fully connected layers 14 calculate the illumination probability based on the illumination information to generate weight coefficients for the infrared image intensity and the visible light image intensity.

[0064] like Figure 4As shown, complementary features are obtained by performing a difference operation on the features of infrared and visible light images. These complementary features are then input into the multi-scale attention module 4, where they first pass through a multi-scale global average pooling attention unit 6 to capture global information of the image at different spatial scales. The pooling results at different scales are mapped to weight distributions through activation functions, and weighted according to the different importance of the complementary features to highlight the significant parts of the complementary feature information. The channel-weighted complementary features after the multi-scale global average pooling layer are multiplied element-wise with the original complementary features, and then pass through a parallel channel attention unit 5. Through parallel grouping, local features can be learned within each group to further deepen the effect of the attention mechanism. The features after passing through the parallel channel attention unit 5 are then multiplied element-wise with the original differential complementary features, and the result is added as supplementary information to the original infrared and visible light image features extracted by the convolutional layer.

[0065] like Figure 5 As shown, in the parallel channel attention unit 5, the first branch of the original complementary features processed by the multi-scale global average pooling attention unit is cascaded through horizontal and vertical dimensional average pooling layers 15, then multiplied element-wise by the original complementary features through a 1×1 convolutional layer 16, and normalized in a normalization layer 17. The second branch is the original complementary features processed through a 3×3 convolutional layer 19 and a one-dimensional global average pooling layer 18. The two branches employ a parallel processing strategy. The features of the first branch, after normalization, pass through a one-dimensional global average pooling layer 18. The second branch captures multi-scale feature representations through a 3×3 convolutional layer 19. Element-wise multiplication aggregates the attention feature maps of the two channels, enabling different cross-channel interactive features between the two parallel paths of the two branches. The outputs of the parallel processing are multiplied by matrix dot product operations. Finally, the output feature map within each group is calculated as a set of two generated spatial attention weight values, which are then multiplied element-wise by the original complementary features for output.

[0066] like Figure 6 As shown, the fusion results of infrared and visible light images of the present invention are presented, where (a) is the infrared image, (b) is the visible light image, and (c) is the fused image. It can be seen that more detailed information is retained and the target is highly salient.

[0067] It should be pointed out that, Figure 2 and Figure 3 Only one possible convolutional composition method is given. Based on the fusion method proposed in this invention, there are many possible implementations of convolutional layers. As long as they meet the basic characteristics of this invention, they should all be covered within the protection scope of this invention. Although this invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the invention.

[0068] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for infrared and visible light image fusion using a multi-scale attention convolutional neural network, characterized in that, Includes the following steps: S1. Construct a multi-scale attention convolutional neural network: Connect multi-scale global average pooling attention units and parallel structure channel attention units in series to form a multi-scale attention module, and connect the multi-scale attention module in parallel to the convolutional neural network to construct a multi-scale attention convolutional neural network; the multi-scale attention convolutional neural network also includes an illumination discrimination auxiliary network and a loss function module, and the illumination discrimination auxiliary network runs in parallel with the convolutional neural network and the multi-scale attention module; S2. Training the illumination discrimination auxiliary network: Input the batched visible light images into the illumination discrimination auxiliary network to calculate the illumination probability weights and assign them to the loss function module; S3. Training a multi-scale attention convolutional neural network: Batch processing of infrared and visible light images and inputting them into the convolutional neural network to obtain the original features of the infrared and visible light images. The original features are then differentially processed to obtain two complementary features. These two complementary features are then input into the multi-scale attention module for processing to obtain two complementary features with attention weights. The two complementary features with attention weights are then superimposed and fed back into the convolutional neural network and added to the original features of the infrared and visible light images to obtain the total features of the infrared and visible light images. The total features of the infrared and visible light images are then concatenated along the channel dimension and reconstructed into a fused image by the image reconstructor of the convolutional neural network. The loss function module is used to calculate the loss of the fused image to guide the multi-scale attention convolutional neural network to optimize the quality of the fused image, resulting in a fully trained multi-scale attention convolutional neural network. S4. Use the trained multi-scale attention convolutional neural network to fuse infrared and visible light images to obtain the optimal fused image.

2. The infrared-visible image fusion method using a multi-scale attention convolutional neural network according to claim 1, characterized in that, The convolutional layers in the convolutional neural network are used to extract the original features of infrared and visible light images, denoted as... , Let be the original feature representations of the input infrared image and the original feature representation of the visible light image, respectively. Let be the complementary features extracted and processed by the multi-scale attention module from the infrared and visible light images, respectively. and .

3. The infrared-visible light image fusion method using a multi-scale attention convolutional neural network according to claim 1, characterized in that, The multi-scale attention module is composed of multi-scale global average pooling attention units and parallel structure channel attention units connected in series.

4. The infrared-visible image fusion method using a multi-scale attention convolutional neural network according to claim 3, characterized in that, The multi-scale global average pooling attention unit of the multi-scale attention module represents the complementary feature processing as follows: ， (1) ， (2) ， (3) ， (4) in, Represented as a 1×1 global average pooling layer, Represented as a 2×2 global average pooling layer, This is represented as a 4×4 global average pooling layer, which performs pooling operations at different scales on the two complementary input features. , , and , , These represent the infrared complementary features and visible light complementary features after processing by global average pooling layers of different scales in the multi-scale global average pooling attention unit, respectively. The superscripts 1, 2, and 4 indicate the corresponding... , , Global average pooling layer This represents the sigmoid function. Indicates interpolation. and These represent the infrared complementary features and visible light complementary features, respectively, after passing through the multi-scale global average pooling attention unit.

5. The infrared-visible image fusion method using a multi-scale attention convolutional neural network according to claim 4, characterized in that, In the multi-scale attention module, the complementary features with attention weights, after passing through the multi-scale global average pooling attention unit, are multiplied element-wise with the original complementary features. This is then passed through a parallel channel attention unit, which is connected in series after the multi-scale global average pooling attention unit. The parallel channel attention unit processes the output of the multi-scale global average pooling attention unit. The processed features are then multiplied element-wise with the original complementary features, as shown below: (5) (6) Here, PCAU represents parallel channel attention unit processing. This indicates that the features are multiplied element by element.

6. The infrared-visible image fusion method using a multi-scale attention convolutional neural network according to claim 1, characterized in that, In the multi-scale attention module, after the complementary features with attention weights from the parallel structure channel attention unit are multiplied element-wise with the original complementary features, the processed complementary features are added as supplementary information to the original features of the infrared image and the original features of the visible light image, resulting in the total features of the infrared image and the total features of the visible light image, expressed as: (7) (8) in, These represent the total features of an infrared image and the total features of a visible light image, respectively. This indicates that the features are added element by element.

7. The infrared-visible image fusion method using a multi-scale attention convolutional neural network according to claim 1, characterized in that, The infrared and visible light total features are spliced ​​together along the channel dimension to achieve feature fusion, represented as: (9) in, These represent the total features of an infrared image and the total features of a visible light image, respectively. The fusion feature, C(·), refers to the connection between the total features of the infrared image and the total features of the visible light image in the channel dimension.

8. The infrared-visible light image fusion method using a multi-scale attention convolutional neural network according to claim 1, characterized in that, Image Reconstructor From fusion characteristics Mid-restore fused image , is represented as: (10)。 9. The infrared-visible image fusion method using a multi-scale attention convolutional neural network according to claim 8, characterized in that, Image reconstructors are components of convolutional neural networks, which consist of interconnected convolutional layers.

10. The infrared-visible image fusion method using a multi-scale attention convolutional neural network according to claim 1, characterized in that, The multi-scale attention convolutional neural network calculates intensity loss from the input infrared image and the input visible light image. auxiliary strength loss and gradient loss To construct the loss function module, the loss function module serves as a total loss. To guide the training of the multi-scale attention convolutional neural network, it is defined as: (11) in, , , These are weight parameters that respectively characterize the importance of intensity loss, auxiliary intensity loss, and gradient loss.

11. The infrared-visible image fusion method using a multi-scale attention convolutional neural network according to claim 10, characterized in that, The strength loss The sum of the weighted infrared image intensity loss and the visible light image intensity loss is expressed as: (12) (13) (14) in, H and W These represent the height and width of the input image, respectively, and ||·|1 represents... Norm, , , These are represented as infrared image, visible light image, and fused image, respectively. and This represents the intensity loss in infrared images and the intensity loss in visible light images. and The weighting coefficients representing the intensity of the infrared image and the intensity of the visible light image are provided by the output of the illumination discrimination auxiliary network.

12. The infrared-visible image fusion method using a multi-scale attention convolutional neural network according to claim 10, characterized in that, The auxiliary strength loss To maintain the optimal intensity distribution in the fused image, it is expressed as: (15) Here, max(·) represents the maximum selection element by element.

13. The infrared-visible image fusion method using a multi-scale attention convolutional neural network according to claim 10, characterized in that, The gradient loss Represented as: (16) Where ▽ represents the gradient operator for measuring image texture information, and |·| refers to the absolute operation.

Images