Tranform-based visible light image and infrared image fusion method and system

Through the end-to-end method based on Transformer, the problems of poor accuracy and high network complexity of visible light images and infrared images in the prior art in unknown scenarios are solved, and higher quality image fusion and task effects are achieved.

CN118968244APending Publication Date: 2024-11-15WU HAN XUAN YUAN ZHI JIA KE JI YOU XIAN GONG SI
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411062188.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-05
Publication Date
2024-11-15

AI Technical Summary

Technical Problem

The existing methods of fusion of visible light images and infrared images are difficult to obtain accurate results in unknown scenarios, and the generalization ability is poor, resulting in high network complexity.

Method used

The end-to-end method based on Transformer is adopted to extract the original feature map of the image, input the feature pyramid for feature fusion, and use the Transformer encoding and decoding module and cross-attention mechanism for information fusion to reduce network complexity.

Benefits of technology

Improve the quality of image fusion results and subsequent task effects, such as object detection and tracking, and enhance the generalization ability of image fusion.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118968244A_ABST
    Figure CN118968244A_ABST
Patent Text Reader

Abstract

The invention discloses a fusion method and system of a visible light image and an infrared image based on Transform. The method comprises the following steps: respectively extracting original feature maps of the visible light image and the infrared image; inputting the original feature map of the visible light image into the feature pyramid for feature fusion to obtain a first feature map; inputting the original feature map of the infrared image into the feature pyramid for feature fusion to obtain a second feature map; inputting the first feature map into a Transform coding module to obtain a third feature map; inputting the second feature map into a Transform decoding module to obtain a fourth feature map; and inputting the third feature map and the fourth feature map into a cross attention mechanism module for fusion to obtain a fifth feature map. In order to solve the problems of poor generalization ability and complex network in the existing visible light image and infrared image fusion method, a Transform module is introduced, and associated points between a visible light image and an infrared image are automatically learned and inquired by using an end-to-end attention mechanism as a core method to solve the problems.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of image fusion, and in particular relates to a method and system for fusing a visible light image and an infrared image based on a Transformer. Background Art

[0002] Existing visible light image and infrared image fusion methods that rely on manual design are difficult to obtain accurate results in unknown scenes and have poor generalization ability. Two-stage or multi-stage fusion methods are not end-to-end solutions, which makes the network more complex. Summary of the invention

[0003] The purpose of the present invention is to overcome the defects of the prior art. The present invention provides a method and system for fusing visible light images and infrared images based on Transformer. By introducing Transformer technology, the present invention eliminates the need for artificially designed fusion methods and can reduce network complexity through an end-to-end model.

[0004] In order to achieve the expected effect, the present invention adopts the following technical solutions:

[0005] The present invention discloses a method for fusing a visible light image and an infrared image based on a Transformer, comprising:

[0006] Extract the original feature maps of the visible light image and the infrared image respectively;

[0007] The original feature map of the visible light image is input into the feature pyramid for feature fusion to obtain a first feature map;

[0008] The original feature map of the infrared image is input into the feature pyramid for feature fusion to obtain a second feature map;

[0009] Input the first feature map into the Transformer encoding module to obtain a third feature map;

[0010] Input the second feature map into the Transformer decoding module to obtain the fourth feature map;

[0011] The third feature map and the fourth feature map are input into the cross attention mechanism module for fusion to obtain the fifth feature map.

[0012] Furthermore, a deep convolutional neural network is used to extract the original feature maps of visible light images and infrared images respectively.

[0013] Furthermore, the original feature map of the visible light image is input into the FPN feature pyramid for multi-scale feature fusion to obtain the first feature map.

[0014] Furthermore, the top-down path of the FPN feature pyramid gradually upsamples the original feature map of the visible light image extracted from the deep convolutional neural network to obtain the first feature map.

[0015] Furthermore, the step of inputting the first feature map into the Transformer encoding module to obtain the third feature map specifically includes: performing Q, K, and V calculations on each feature point in the first feature map to obtain a corresponding matrix, inputting the matrix into a multi-head self-attention mechanism module to obtain global information of the first image, inputting the first image global information into an Add&Norm layer for residual connection and normalization, and then inputting the first image global information into the FFN network to obtain the third feature map.

[0016] Furthermore, Q, K, and V satisfy the following relationship:

[0017]

[0018] Among them, Q is the query, K is the key value, and V is the value.

[0019] Furthermore, the step of inputting the second feature map into the Transformer decoding module to obtain the fourth feature map specifically includes: performing Q, K, and V calculations on each feature point in the second feature map to obtain a corresponding matrix, inputting the matrix into the multi-head self-attention mechanism module to obtain global information of the second image, and inputting the second image global information into the Add&Norm layer for residual connection and normalization to obtain the fourth feature map.

[0020] Furthermore, the step of inputting the third feature map and the fourth feature map into the cross-attention mechanism module for fusion to obtain the fifth feature map specifically includes: taking the fourth feature map as a Q value, taking the third feature map as a K value and a V value, and then inputting the Q value, K value, and V value into the cross-attention mechanism module for fusion and then decoding through a deconvolution layer to obtain the fifth feature map.

[0021] Furthermore, the method also includes: performing SSIM loss calculation on the fifth feature map, the original visible light image, and the original infrared image respectively to evaluate the fusion effect.

[0022] The present invention also discloses a visible light image and infrared image fusion system based on Transformer, comprising:

[0023] An acquisition module, used for acquiring visible light images and infrared images to be fused;

[0024] A fusion module is used to fuse the visible light image and the infrared image according to any of the methods described above.

[0025] Compared with the prior art, the beneficial effects of the present invention are as follows: the present invention provides a method and system for fusing visible light images and infrared images based on Transformer. The present invention aims at the problems of poor generalization ability and complex network in the existing methods for fusing visible light images and infrared images. By introducing the Transformer module, the above problems are solved by automatically learning and querying the related points between visible light images and infrared images in an end-to-end method with attention mechanism as the core. Compared with other methods, the present invention focuses on the global understanding of the image, captures the global information in the image through the attention mechanism, and comprehensively utilizes all the information of the visible light image and the infrared image in the fusion process. The global context understanding of the present invention helps to improve the quality of the image fusion results, and at the same time improves the effects of subsequent target detection, tracking and other tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0027] Figure 1 It is a schematic diagram of Q, K, and V calculation provided by an embodiment of the present invention.

[0028] Figure 2 It is a calculation flow chart of a Transformer encoding module provided by an embodiment of the present invention.

[0029] Figure 3 It is a calculation flow chart of a Transformer decoding module provided by an embodiment of the present invention.

[0030] Figure 4 This is a loss calculation flow chart provided by an embodiment of the present invention.

[0031] Figure 5 This is a flow chart of a method for fusing visible light images and infrared images based on Transformer provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0032] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0033] Existing visible light image and infrared image fusion methods can be roughly divided into traditional methods and deep learning methods. Traditional methods include multi-scale transformation method, sparse representation method, wavelet transformation method, etc. Traditional methods perform multi-scale decomposition of images, divide images into regions of different frequencies, and then distinguish and filter and fuse the regions. Deep learning methods can be divided into autoencoder (AE), convolutional neural network (CNN) and generative adversarial network (GAN) methods. The architecture of the AE-based method is generally divided into three parts: encoder, fusion layer and decoder, which respectively complete feature extraction, feature fusion and image reconstruction; the CNN-based method integrates the three processes of feature extraction, feature fusion and image reconstruction; the adversarial game between the generator and the discriminator in the GAN network can force the network to fit the distribution of the original image data.

[0034] See also Figures 1 to 5 The present invention discloses a method for fusing a visible light image and an infrared image based on a Transformer, comprising:

[0035] Extract the original feature maps of the visible light image and the infrared image respectively;

[0036] In one embodiment, a deep convolutional neural network is used to extract original feature maps of the visible light image and the infrared image respectively.

[0037] The original feature map of the visible light image is input into the feature pyramid for feature fusion to obtain a first feature map;

[0038] In one embodiment, the original feature map of the visible light image is input into the FPN feature pyramid for multi-scale feature fusion to obtain the first feature map.

[0039] FPN (Feature Pyramid Network) is a network architecture used for target detection and semantic segmentation, which aims to solve the detection and segmentation problems of objects of different scales.

[0040] Principle: It processes objects of different scales by constructing a multi-scale feature pyramid. It mainly includes two key steps: bottom-up feature extraction and top-down feature fusion.

[0041] In one embodiment, the top-down path of the FPN feature pyramid gradually upsamples the original feature map of the visible light image extracted from the deep convolutional neural network to obtain a first feature map.

[0042] The original feature map of the infrared image is input into the feature pyramid for feature fusion to obtain a second feature map;

[0043] In one embodiment, the original feature map of the infrared image is input into the FPN feature pyramid for multi-scale feature fusion to obtain the second feature map.

[0044] In one embodiment, the top-down path of the FPN feature pyramid gradually upsamples the original feature map of the infrared image extracted from the deep convolutional neural network to obtain a second feature map.

[0045] Specifically, the method uses a deep convolutional neural network to extract feature maps of visible light images and infrared images, and inputs the extracted feature maps into FPN for multi-scale feature fusion. Preferably, VGG-16 is used as a feature extractor. Preferably, the first few convolutional layers of VGG-16 can be used as feature extractors, that is, the input visible light image and infrared image are passed through the convolutional layer and pooling layer of VGG-16 to obtain a series of feature maps. For example, you can choose to extract features after the third, fourth, and fifth convolutional blocks of VGG-16, so that feature representations at different levels can be obtained. Then the feature maps extracted by VGG-16 are used as input and input into FPN respectively. The top-down path of FPN gradually upsamples the high-resolution feature maps extracted from VGG-16 to obtain coarser but lower resolution feature maps. The lateral connection of FPN fuses low-level features with high-level features to retain more detailed information.

[0046] Input the first feature map into the Transformer encoding module to obtain a third feature map;

[0047] In one embodiment, the step of inputting the first feature map into the Transformer encoding module to obtain the third feature map specifically includes: performing Q, K, and V calculations on each feature point in the first feature map to obtain a corresponding matrix, inputting the matrix into a multi-head self-attention mechanism module to obtain global information of the first image, inputting the first image global information into the Add&Norm layer for residual connection and normalization, and then inputting the FFN network to obtain the third feature map.

[0048] FFN network, feedforward neural network is a mathematical model based on the working principle of biological neurons, composed of multiple layers of neurons, divided into input layer, hidden layer (optional) and output layer. Each neuron performs weighted summation of the input through weights and activation functions, and outputs it to the next layer.

[0049] In one embodiment, Q, K, and V satisfy the following relationship:

[0050]

[0051] Among them, Q is the query, K is the key value, and V is the value.

[0052] Specifically, the feature map of the visible light image is first input into the transformer encoder encoding module, specifically: each feature point (vector) of the feature map of the visible light image is calculated by Q, K, and V to obtain the matrices of the three, such as Figure 1 shown.

[0053] The matrix is ​​then input into the multi-head self-attention mechanism module to update each feature point to a feature point containing all feature point information, thereby obtaining the global information of the image.

[0054] The updated feature point vectors are input into the Add&Norm layer for residual connection and normalization. This can solve the difficulties in training multi-layer neural networks and normalize the activation values ​​of each layer, which can accelerate the training process of the model and make it converge faster.

[0055] Finally, it is input into the FFN network to obtain the final encoded feature map

[0056] Input the second feature map into the Transformer decoding module to obtain the fourth feature map;

[0057] In one embodiment, the step of inputting the second feature map into the Transformer decoding module to obtain the fourth feature map specifically includes: performing Q, K, and V calculations on each feature point in the second feature map to obtain a corresponding matrix, inputting the matrix into the multi-head self-attention mechanism module to obtain the second image global information, and inputting the second image global information into the Add&Norm layer for residual connection and normalization to obtain the fourth feature map.

[0058] The third feature map and the fourth feature map are input into the cross attention mechanism module for fusion to obtain the fifth feature map.

[0059] In one embodiment, the inputting the third feature map and the fourth feature map into the cross-attention mechanism module for fusion to obtain the fifth feature map specifically includes: treating the fourth feature map as a Q value, treating the third feature map as a K value and a V value, and then inputting the Q value, K value, and V value into the cross-attention mechanism module for fusion and then decoding through a deconvolution layer to obtain the fifth feature map.

[0060] Specifically, the feature map of the infrared image is first used as an image query to calculate Q, K, and V, and then input into the multi-head self-attention mechanism module to learn the global information of the image; after passing through the Add&Norm layer, the output is used as q, and the output of the encoder is calculated for k and v, and then these three q, k, and v are input into the cross-attention mechanism module to fuse the infrared image features with the visible light image features. Finally, it is decoded through the deconvolution layer and the fused image is finally output.

[0061] In one embodiment, the method further includes: performing SSIM loss calculation on the fifth feature map, the original visible light image, and the original infrared image respectively to evaluate the fusion effect.

[0062] SSIM (Structural Similarity) is an indicator to measure the similarity between two images. This indicator was first proposed by the Laboratory for Image and Video Engineering at the University of Texas at Austin. SSIM uses two images, one is an uncompressed, undistorted image, and the other is a distorted image.

[0063] Specifically, the loss model of the present invention is a self-supervised model and does not require a true value. Therefore, the SSIM loss is performed on the two fused images and the original image. SSIM is an indicator for measuring the structural similarity of two images.

[0064] In one embodiment, SSIM is expressed as:

[0065]

[0066] in,

[0067]

[0068] Represents the brightness component, comparing the average brightness of the two images, μ y and are images y and The average brightness of , C1 is a small constant used to avoid the denominator being zero.

[0069]

[0070] represents the contrast component, σ y and They are image y and The standard deviation of C2 is a larger constant to ensure numerical stability.

[0071]

[0072] represents the structural component and compares the covariance of two images; is the covariance of the two images, and C2 is a constant to ensure numerical stability.

[0073] In one embodiment, the loss function is:

[0074]

[0075] The present invention aims to solve the problems of poor generalization ability and complex network in existing visible light image and infrared image fusion methods by introducing the Transformer module, using an end-to-end method with an attention mechanism as the core to automatically learn and query the related points between visible light images and infrared images. Compared with other methods, the present invention focuses on the global understanding of the image, captures the global information in the image through the attention mechanism, and comprehensively utilizes all the information of the visible light image and the infrared image in the fusion process. The global context understanding of the present invention helps to improve the quality of image fusion results, and at the same time improves the effects of subsequent target detection, tracking and other tasks.

[0076] The present invention also discloses a visible light image and infrared image fusion system based on Transformer, comprising:

[0077] An acquisition module, used for acquiring visible light images and infrared images to be fused;

[0078] A fusion module is used to fuse the visible light image and the infrared image according to any of the methods described above.

[0079] The system embodiments can be implemented one-to-one with the aforementioned method embodiments, and will not be described in detail here.

[0080] Based on the same inventive concept, the present invention also discloses an electronic device, which may include: a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus. The processor may call the logic instructions in the memory to execute a Transformer-based visible light image and infrared image fusion method, including:

[0081] Extract the original feature maps of the visible light image and the infrared image respectively;

[0082] The original feature map of the visible light image is input into the feature pyramid for feature fusion to obtain a first feature map;

[0083] The original feature map of the infrared image is input into the feature pyramid for feature fusion to obtain a second feature map;

[0084] Input the first feature map into the Transformer encoding module to obtain a third feature map;

[0085] Input the second feature map into the Transformer decoding module to obtain the fourth feature map;

[0086] The third feature map and the fourth feature map are input into the cross attention mechanism module for fusion to obtain the fifth feature map.

[0087] In addition, the logic instructions in the above-mentioned memory can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0088] On the other hand, an embodiment of the present invention further provides a computer program product, the computer program product including a computer program stored on a non-transitory computer-readable storage medium, the computer program including program instructions, and when the program instructions are executed by a computer, the computer can execute a Transformer-based visible light image and infrared image fusion method provided in the above-mentioned method embodiments, including:

[0089] Extract the original feature maps of the visible light image and the infrared image respectively;

[0090] The original feature map of the visible light image is input into the feature pyramid for feature fusion to obtain a first feature map;

[0091] The original feature map of the infrared image is input into the feature pyramid for feature fusion to obtain a second feature map;

[0092] Input the first feature map into the Transformer encoding module to obtain a third feature map;

[0093] Input the second feature map into the Transformer decoding module to obtain the fourth feature map;

[0094] The third feature map and the fourth feature map are input into the cross attention mechanism module for fusion to obtain the fifth feature map.

[0095] In another aspect, an embodiment of the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method for fusing a visible light image and an infrared image based on a Transformer provided in the above embodiments is implemented, including:

[0096] Extract the original feature maps of the visible light image and the infrared image respectively;

[0097] The original feature map of the visible light image is input into the feature pyramid for feature fusion to obtain a first feature map;

[0098] The original feature map of the infrared image is input into the feature pyramid for feature fusion to obtain a second feature map;

[0099] Input the first feature map into the Transformer encoding module to obtain a third feature map;

[0100] Input the second feature map into the Transformer decoding module to obtain the fourth feature map;

[0101] The third feature map and the fourth feature map are input into the cross attention mechanism module for fusion to obtain the fifth feature map.

[0102] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0103] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A visible light image and infrared image fusion method based on Transformer, characterized in that: include: Extract the original feature maps of the visible light image and the infrared image respectively; The original feature map of the visible light image is input into the feature pyramid for feature fusion to obtain the first feature map; The original feature map of the infrared image is input into the feature pyramid for feature fusion to obtain a second feature map; Input the first feature map into the Transformer encoding module to obtain a third feature map; Input the second feature map into the Transformer decoding module to obtain the fourth feature map; The third feature map and the fourth feature map are input into the cross attention mechanism module for fusion to obtain the fifth feature map.

2. The method according to claim 1, characterized in that: A deep convolutional neural network is used to extract the original feature maps of visible light images and infrared images respectively.

3. The method according to claim 2, characterized in that: The original feature map of the visible light image is input into the FPN feature pyramid for multi-scale feature fusion to obtain the first feature map.

4. The method according to claim 3, characterized in that: The top-down path of the FPN feature pyramid gradually upsamples the original feature map of the visible light image extracted from the deep convolutional neural network to obtain the first feature map.

5. The method according to claim 1, characterized in that: The step of inputting the first feature map into the Transformer encoding module to obtain the third feature map specifically includes: performing Q, K, and V calculations on each feature point in the first feature map to obtain a corresponding matrix, inputting the matrix into the multi-head self-attention mechanism module to obtain the global information of the first image, inputting the first image global information into the Add&Norm layer for residual connection and normalization, and then inputting the FFN network to obtain the third feature map.

6. The method according to claim 5, characterized in that The Q, K, and V satisfy the following relationship: Among them, Q is the query, K is the key value, and V is the value.

7. The method according to claim 6, characterized in that The step of inputting the second feature map into the Transformer decoding module to obtain the fourth feature map specifically includes: performing Q, K, and V calculations on each feature point in the second feature map to obtain a corresponding matrix, inputting the matrix into the multi-head self-attention mechanism module to obtain the second image global information, and inputting the second image global information into the Add&Norm layer for residual connection and normalization to obtain the fourth feature map.

8. The method according to claim 7, characterized in that: The step of inputting the third feature map and the fourth feature map into the cross-attention mechanism module for fusion to obtain the fifth feature map specifically includes: taking the fourth feature map as the Q value, taking the third feature map as the K value and the V value, and then inputting the Q value, the K value, and the V value into the cross-attention mechanism module for fusion and then decoding through the inverse convolution layer to obtain the fifth feature map.

9. The method according to any one of claims 1 to 8, characterized in that: The method also includes: performing SSIM loss calculation on the fifth feature map, the original visible light image, and the original infrared image respectively to evaluate the fusion effect.

10. A visible light image and infrared image fusion system based on Transformer, characterized in that: include: An acquisition module, used for acquiring visible light images and infrared images to be fused; A fusion module, used for fusing a visible light image and an infrared image according to the method according to any one of claims 1 to 9.