Infrared and visible light image fusion method, system and device and storage medium

Abstract

The invention relates to the technical field of image processing, in particular to an infrared and visible light image fusion method, system and device and a storage medium, and the method comprises the steps: obtaining an infrared image and a visible light image, and converting the visible light image into a YUV color space through a color space conversion method; obtaining original brightness channel data and original chrominance channel data of the visible light image; performing feature extraction on the original brightness channel data of the infrared image and the visible light image to obtain different modal image features; a gradient attention mechanism is introduced, gradient-guided feature fusion is carried out on different modal image features, and modal fusion feature information is obtained; and performing feature reconstruction according to the modal fusion feature information and the original chrominance channel data of the visible light image to obtain a color fusion image. According to the method, the feature extraction network and gradient weights of different modal image features are combined, so that texture details are highlighted and reserved, and the fusion quality of the infrared image and the visible light image is improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an infrared and visible light image fusion method, system, device and storage medium. Background Art

[0002] Since single-modality images often cannot provide sufficiently comprehensive scene information, and multimodal image fusion can fuse scene information of different modalities together to generate a fused image containing more information, infrared and visible light image fusion technology has received extensive attention and research in recent years. Fusion of these two modal images can provide complementary information, thereby improving the performance of applications such as target detection, tracking, and scene segmentation. However, traditional image fusion techniques are mostly based on spatial domain and transform domain methods, such as Laplace pyramid, discrete wavelet transform, and downsampled contourlet transform. These methods have some limitations in practical applications, such as insufficient robustness and difficulty in coping with complex and changing scenes; limited generalization ability, difficulty in adapting to different types of images, and high consumption of computing resources. These methods will lead to the loss of detail information during the fusion process, thereby affecting the quality of the final image.

[0003] With the development of deep learning, infrared and visible light image fusion networks based on convolutional neural networks have made significant progress. Through key components such as feature extraction, feature fusion, and image reconstruction, this type of network can automatically extract and fuse source image features, and reconstruct the fused features into images of original resolution. However, existing networks still have challenges in feature extraction, especially how to balance local features and global features. Local features are crucial for capturing image details, while global features help understand the overall structure and relationships of the image.

[0004] In addition, the existing fusion algorithms also have shortcomings in feature fusion strategies. Traditional fusion strategies often use simple channel concatenation or fixed weight ratios for fusion, and lack adaptive consideration of the importance of different features. This leads to problems such as brightness inconsistency in the fused image, fuzzy existing fusion rules for texture information, or even loss. Especially when balancing the differences between infrared and visible light images, traditional fusion strategies often find it difficult to achieve ideal results. Therefore, how to improve feature extraction and fusion strategies to achieve more efficient and accurate fusion of infrared and visible light images is an important direction of current research. Summary of the invention

[0005] In order to solve the above technical problems, the present invention provides an infrared and visible light image fusion method, system, device and storage medium.

[0006] In a first aspect, the present invention provides a method for fusing infrared and visible light images, the method comprising the following steps:

[0007] Acquire an infrared image and a visible light image, and convert the visible light image into a YUV color space using a color space conversion method, and separate and obtain original brightness channel data and original chrominance channel data of the visible light image;

[0008] Using a feature extraction network to extract features from the original brightness channel data of the infrared image and the visible light image, respectively, to obtain different modality image features; wherein the different modality image features include infrared image thermal radiation features and visible light image texture features;

[0009] The gradient attention mechanism is introduced, and the multimodal feature fusion network is used to perform gradient-guided feature fusion on the features of different modal images to obtain modal fusion feature information;

[0010] Feature reconstruction is performed based on the modality fusion feature information and the original chromaticity channel data of the visible light image to obtain a color fused image.

[0011] In a further embodiment, the feature extraction network includes a first convolutional layer and at least three spatiotemporal graph convolutional modules connected in cascade; the output of the first convolutional layer is connected to the input of each spatiotemporal graph convolutional module using a global residual connection mechanism, and directly forms a residual connection with the output of the last spatiotemporal graph convolutional module;

[0012] The output of each spatiotemporal graph convolution module skips the next spatiotemporal graph convolution module connected to it and is connected to the input of the third spatiotemporal graph convolution module through a local residual connection;

[0013] The spatiotemporal graph convolution module includes a global-local visual attention model and a texture detail gradient enhancement model connected in parallel, and the output of the global-local visual attention model forms a residual connection with the output of the texture detail gradient enhancement model; wherein the global-local visual attention model includes a serial number of Slide-Transformer modules and a second convolutional layer; the texture detail gradient enhancement model includes a serial gradient operator module and a third convolutional layer.

[0014] In a further embodiment, the step of using a feature extraction network to extract features from the original brightness channel data of the infrared image and the visible light image to obtain features of images of different modalities comprises:

[0015] Performing feature projection on the original brightness channel data of the infrared image and the visible light image respectively through the first convolution layer to obtain corresponding infrared image projection features and visible light brightness channel projection features;

[0016] The infrared image projection feature and the visible light brightness channel projection feature are connected to the input of each spatiotemporal graph convolution module through a global residual connection, and a plurality of serial spatiotemporal graph convolution modules are used to sequentially learn to balance the hierarchical global-local feature representations of different scales, and a gradient operator is introduced to enhance the capture of texture detail features to obtain a modal high-dimensional feature representation;

[0017] According to the infrared image projection features and the visible light brightness channel projection features, the modal high-dimensional feature representation is enhanced using a global residual connection mechanism to extract the corresponding infrared image thermal radiation features and visible light image texture features, respectively.

[0018] In a further embodiment, the multimodal feature fusion network includes a modal gradient attention extraction module and a modal feature fusion module connected in cascade, the modal gradient attention extraction module includes a plurality of modal gradient attention extraction submodules; each of the modal gradient attention extraction submodules includes a gradient channel attention enhancement unit and a gradient space attention regulation unit connected in parallel;

[0019] The gradient channel attention enhancement unit includes a gradient attention unit and a channel attention unit connected in parallel, and an input of the gradient channel attention enhancement unit forms a residual connection with an output of the gradient channel attention enhancement unit;

[0020] The gradient spatial attention control unit includes a gradient attention unit and a spatial attention unit connected in parallel, and the input of the gradient spatial attention control unit forms a residual connection with the output of the gradient spatial attention control unit.

[0021] In a further embodiment, the step of introducing a gradient attention mechanism and using a multimodal feature fusion network to perform gradient-guided feature fusion on different modal image features to obtain modal fusion feature information includes:

[0022] The gradient channel attention enhancement unit performs gradient-guided channel feature enhancement on the features of different modal images to obtain the gradient channel attention weights; the gradient channel attention weights include the infrared gradient channel attention weights and the visible light gradient channel attention weights;

[0023] The gradient channel attention weight is used to perform channel texture enhancement on image features of different modalities to obtain channel enhancement features of different modalities; the channel enhancement features of different modalities include infrared channel enhancement features and visible light channel enhancement features;

[0024] The gradient spatial attention control unit is used to perform gradient-guided spatial feature enhancement on the features of different modal images, and the gradient spatial attention weight is obtained; the gradient spatial attention weight includes the infrared gradient spatial attention weight and the visible light gradient spatial attention weight;

[0025] The gradient spatial attention weight is used to perform spatial texture enhancement on image features of different modalities to obtain spatial enhancement features of different modalities; the spatial enhancement features of different modalities include infrared spatial enhancement features and visible light spatial enhancement features;

[0026] The channel enhancement features of different modalities are merged through the modal feature fusion module to obtain the channel attention weighted features, and the spatial enhancement features of different modalities are merged to obtain the spatial attention weighted features;

[0027] The channel attention weighted features and the spatial attention weighted features are merged to obtain modal fusion feature information.

[0028] In a further embodiment, the step of performing gradient-guided channel feature enhancement on different modal image features by the gradient channel attention enhancement unit to obtain the gradient channel attention weight comprises:

[0029] The gradient attention unit and the channel attention unit in the gradient channel attention enhancement unit are used to respectively calculate the gradient attention and channel attention of different modality image features on each channel to obtain channel gradient weights and channel feature weights;

[0030] According to the channel gradient weights of different modal image features on each channel, the channel feature weights are guided by texture detail gradients to obtain gradient channel attention weights.

[0031] In a further embodiment, the step of performing gradient-guided spatial feature enhancement on different modal image features by the gradient spatial attention control unit to obtain the gradient spatial attention weight comprises:

[0032] The gradient attention unit and the spatial attention unit in the gradient spatial attention control unit are used to respectively calculate the gradient attention and the spatial attention of different modal image features at each spatial position to obtain the spatial gradient weight and the spatial feature weight;

[0033] According to the spatial gradient weights of different modal image features at each spatial position, the spatial feature weights are guided by texture detail gradients to obtain the gradient channel attention weights.

[0034] In a second aspect, the present invention provides an infrared and visible light image fusion system, the system comprising:

[0035] A space conversion module is used to obtain an infrared image and a visible light image, and convert the visible light image into a YUV color space using a color space conversion method, so as to separate and obtain original brightness channel data and original chrominance channel data of the visible light image;

[0036] A feature extraction module, used to use a feature extraction network to perform feature extraction on the original brightness channel data of the infrared image and the visible light image respectively, to obtain different modality image features; wherein the different modality image features include infrared image thermal radiation features and visible light image texture features;

[0037] The feature fusion module is used to introduce the gradient attention mechanism and use the multimodal feature fusion network to perform gradient-guided feature fusion on the features of different modal images to obtain modal fusion feature information;

[0038] The feature reconstruction module is used to perform feature reconstruction based on the modal fusion feature information and the original chromaticity channel data of the visible light image to obtain a color fused image.

[0039] In a third aspect, the present invention further provides a computer device, comprising a processor and a memory, wherein the processor is connected to the memory, the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory so that the computer device performs the steps of implementing the above method.

[0040] In a fourth aspect, the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program implements the steps of the above method when executed by a processor.

[0041] The present invention provides an infrared and visible light image fusion method, system, device and storage medium. The method acquires an infrared image and a visible light image, and converts the visible light image into a YUV color space by using a color space conversion method, so as to separate and obtain original brightness channel data and original chromaticity channel data of the visible light image; uses a feature extraction network to perform feature extraction on the original brightness channel data of the infrared image and the visible light image, respectively, to obtain different modal image features; wherein the different modal image features include infrared image thermal radiation features and visible light image texture features; introduces a gradient attention mechanism, and uses a multimodal feature fusion network to perform gradient-guided feature fusion on different modal image features, so as to obtain modal fusion feature information; performs feature reconstruction based on the modal fusion feature information and the original chromaticity channel data of the visible light image, so as to obtain a color fused image. Compared with the existing technology, this method combines the feature extraction network of Slide Transformer and gradient operator modules, calculates the gradient weights of the two image modal features of infrared and visible light, and locally affects the weights of channel attention and spatial attention, thereby highlighting the texture details and avoiding the problem of blurred texture details in traditional fusion methods, thereby achieving accurate extraction of infrared image and visible light image features. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a schematic diagram of the flow of the infrared and visible light image fusion method provided by an embodiment of the present invention;

[0043] Figure 2 It is a general framework diagram of the infrared and visible light image fusion process provided by an embodiment of the present invention;

[0044] Figure 3 is a schematic diagram of a feature extraction network structure provided by an embodiment of the present invention;

[0045] Figure 4 It is a schematic diagram of the structure of a spatiotemporal graph convolution module provided by an embodiment of the present invention;

[0046] Figure 5 is a schematic diagram of a multimodal feature fusion network structure provided by an embodiment of the present invention;

[0047] Figure 6 is an example diagram of a fused image provided by an embodiment of the present invention;

[0048] Figure 7 is a block diagram of an infrared and visible light image fusion system provided by an embodiment of the present invention;

[0049] Figure 8 It is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0050] The following specifically illustrates the implementation mode of the present invention in conjunction with the accompanying drawings. The embodiments are provided for illustrative purposes only and are not to be construed as limitations of the present invention. The accompanying drawings are provided for reference and illustration only and do not constitute limitations on the scope of patent protection of the present invention, because many changes may be made to the present invention without departing from the spirit and scope of the present invention.

[0051] refer to Figure 1 , the embodiment of the present invention provides a method for fusing infrared and visible light images, such as Figure 1 As shown, the method comprises the following steps:

[0052] S1. Acquire an infrared image and a visible light image, and convert the visible light image into a YUV color space using a color space conversion method, and separate and obtain original brightness channel data and original chrominance channel data of the visible light image.

[0053] like Figure 2As shown, when traditional methods process the fusion of infrared and visible light images, it is often difficult to accurately extract the features of the two images, resulting in incomplete or distorted image information after fusion. In order to ensure the fusion quality of color images, before feature extraction, this embodiment uses a color space conversion method to convert the input visible light image from RGB format to YUV format. This conversion process separates and encodes the brightness channel data (Y) and chrominance channel data (U and V) of the visible light image, so that the subsequent feature extraction stage can more accurately control the brightness information and color information of the image.

[0054] S2. Use a feature extraction network to perform feature extraction on the original brightness channel data of the infrared image and the visible light image respectively to obtain different modal image features; wherein the different modal image features include infrared image thermal radiation features and visible light image texture features.

[0055] In order to solve the problem of insufficient local and global feature extraction, the feature extraction network in this embodiment introduces a spatiotemporal graph convolution module (STG-Block) that combines Slide-Transformer and a gradient operator module (Gradient Operator). Specifically, the Sobel operator is used as the Gradient Operator in STG-Block, which significantly enhances the network's ability to capture fine-grained details of the image. At the same time, Slide-Transformer effectively captures local features by limiting the receptive field of the query to cover only its adjacent pixels. Compared with other attention modes, Slide-Transformer not only inherits the advantages of translation equivariance and local inductive bias of convolution, but also retains the flexibility and data dependence of the self-attention mechanism. Therefore, STG-Block can take into account both global and local features during the feature extraction process, thereby improving the adequacy and accuracy of feature extraction. In this embodiment, the feature extraction network includes a cascade-connected first convolution layer and at least three spatiotemporal graph convolution modules. For ease of understanding, as shown in FIG. Figure 3 As shown, this embodiment is illustrated by an example in which the feature extraction network includes a cascade-connected first convolution layer (Conv-in) and three spatiotemporal graph convolution modules. In this embodiment, the output of the first convolution layer is connected to the input of each spatiotemporal graph convolution module using a global residual connection mechanism, and directly forms a residual connection with the output of the last spatiotemporal graph convolution module; the output of each spatiotemporal graph convolution module skips the next spatiotemporal graph convolution module connected to it and is connected to the input of the third spatiotemporal graph convolution module through a local residual connection.

[0056] Specifically, the output of the first convolutional layer will be used as one of the inputs of the subsequent spatiotemporal graph convolution modules, that is, the output of the first convolutional layer is not only directly input into the first spatiotemporal graph convolution module, but also connected to the input of each subsequent spatiotemporal graph convolution module through a global residual connection; in addition, the input of each spatiotemporal graph convolution module (except the first one) includes not only the output of the previous spatiotemporal graph convolution module, but also the output of the first convolutional layer (through a global residual connection) and the output of the previous skipped spatiotemporal graph convolution module (through a local residual connection). The output of the first spatiotemporal graph convolution module skips the adjacent second spatiotemporal graph convolution module and is directly used as the third module. As well as additional inputs to subsequent modules (through local residual connections), this embodiment utilizes multiple serial spatiotemporal graph convolution modules to extract features at different levels, from low-level edges and textures to high-level global semantic information, and can sequentially learn and balance global and local feature representations. These modules not only effectively solve the problem of imbalance between global and local features in traditional feature extraction, but also enable the network to capture dynamic changes and spatial structural information in the image through its inherent spatiotemporal modeling capabilities. In order to further optimize the feature extraction process, this embodiment introduces the Sobel operator, which can highlight the texture details in the image, thereby enhancing the network's ability to capture texture features.

[0057] In this embodiment, the spatiotemporal graph convolution module includes a global-local visual attention model and a texture detail gradient enhancement model connected in parallel, and the output of the global-local visual attention model forms a residual connection with the output of the texture detail gradient enhancement model; wherein the global-local visual attention model includes a serial number of Slide-Transformer modules and a second convolutional layer; the texture detail gradient enhancement model includes a serial gradient operator module and a third convolutional layer. For ease of understanding, Figure 4 The present invention exemplarily demonstrates a spatiotemporal graph convolution module including two serial Slide-Transformer modules and a second convolution layer based on a global-local visual attention model. Those skilled in the art can adjust the number of Slide-Transformer modules according to the specific implementation situation. The present invention is not limited to this invention. In this embodiment, STG-Block is formed by residual connection of Slide-Transformer and Gradient Operator. This embodiment realizes accurate extraction of features of infrared images and visible light images by combining the feature extraction network of SlideTransformer and Gradient Operator. Slide-Transformer solves the problem of unbalanced feature extraction of local features and global features, while Gradient Operator can further enhance the extraction of texture details.

[0058] It should be noted that in order to overcome the inefficient feature shift operation in the traditional method, in the local attention mechanism of Slide-Transformer, the Slide-Transformer module in this embodiment uses the depth convolution of the convolution kernel to replace the inefficient feature shift. Specifically, the Slide-Transformer module limits the Key and Value to fixed adjacent positions through the kernel weight. In order to further enhance the flexibility of local attention, the Slide-Transformer module introduces parallel convolution paths. The convolution kernel parameters of these parallel paths are randomly initialized and optimized by the back propagation algorithm during the training phase, so that they can learn to capture diverse features. In this way, the Slide-Transformer module not only maintains the advantages of local attention, such as translation equivariance and local inductive bias, but also enhances its flexibility, enabling it to more effectively extract local features in the image. At the same time, combined with other structural modules such as the gradient operator module (GradientOperator) in STG-Block, the feature extraction network can take into account both global and local features, providing rich feature representations for subsequent image processing tasks.

[0059] Based on the feature extraction network, the step of using the feature extraction network to extract features from the original brightness channel data of the infrared image and the visible light image to obtain features of different modal images includes:

[0060] Performing feature projection on the original brightness channel data of the infrared image and the visible light image respectively through the first convolution layer to obtain corresponding infrared image projection features and visible light brightness channel projection features;

[0061] The infrared image projection feature and the visible light brightness channel projection feature are connected to the input of each spatiotemporal graph convolution module through a global residual connection, and a plurality of serial spatiotemporal graph convolution modules are used to sequentially learn to balance the hierarchical global-local feature representations of different scales, and a gradient operator is introduced to enhance the capture of texture detail features to obtain a modal high-dimensional feature representation;

[0062] According to the infrared image projection features and the visible light brightness channel projection features, the modal high-dimensional feature representation is enhanced using a global residual connection mechanism to extract the corresponding infrared image thermal radiation features and visible light image texture features, respectively.

[0063] In the feature extraction network proposed in this embodiment, the first convolution layer (Conv-in) is a 1x1 convolution layer. The first convolution layer is used to perform feature projection on the input image to obtain corresponding infrared image projection features and visible light brightness channel projection features, and provide input for the subsequent STG-Block. STG-Block combines modules such as Slide-Transformer for extracting local features. Specifically, the feature extraction network combining Slide Transformer and GradientOperator proposed in this embodiment uses the Slide Transformer module in the global-local visual attention model to sequentially learn to balance hierarchical global-local feature representations of different scales, and realizes the extraction of local features and global features through the parallel convolution of Slide Transformer. At the same time, the gradient operator module (Gradient Operator) uses the Sobel operator to further capture and enhance the texture detail features to obtain the modal high-dimensional feature representation.

[0064] S3. The gradient attention mechanism is introduced, and the multimodal feature fusion network is used to perform gradient-guided feature fusion on the features of different modal images to obtain modal fusion feature information.

[0065] In the image fusion process, due to the differences and interference between different images, texture details are often blurred. This embodiment achieves the highlighting of texture details and avoids the problem of blurred texture details by calculating the gradient weight of the feature and locally affecting the weights of the channel attention and the spatial attention. In this embodiment, the multimodal feature fusion network includes a cascade-connected modal gradient attention extraction module and a modal feature fusion module, and the modal gradient attention extraction module includes a plurality of modal gradient attention extraction submodules; each of the modal gradient attention extraction submodules includes a gradient channel attention enhancement unit and a gradient spatial attention control unit connected in parallel; wherein the gradient channel attention enhancement unit includes a gradient attention unit and a channel attention unit connected in parallel, and the input of the gradient channel attention enhancement unit forms a residual connection with the output of the gradient channel attention enhancement unit; the gradient spatial attention control unit includes a gradient attention unit and a spatial attention unit connected in parallel, and the input of the gradient spatial attention control unit forms a residual connection with the output of the gradient spatial attention control unit. Figure 5 Schematic diagram of the multimodal feature fusion network structure.

[0066] In this embodiment, the step of introducing the gradient attention mechanism and using the multimodal feature fusion network to perform gradient-guided feature fusion on different modal image features to obtain modal fusion feature information includes:

[0067] The gradient channel attention enhancement unit performs gradient-guided channel feature enhancement on the features of different modal images to obtain the gradient channel attention weights; the gradient channel attention weights include the infrared gradient channel attention weights and the visible light gradient channel attention weights;

[0068] The gradient channel attention weight is used to perform channel texture enhancement on image features of different modalities to obtain channel enhancement features of different modalities; the channel enhancement features of different modalities include infrared channel enhancement features and visible light channel enhancement features;

[0069] The gradient spatial attention control unit is used to perform gradient-guided spatial feature enhancement on the features of different modal images, and the gradient spatial attention weight is obtained; the gradient spatial attention weight includes the infrared gradient spatial attention weight and the visible light gradient spatial attention weight;

[0070] The gradient spatial attention weight is used to perform spatial texture enhancement on image features of different modalities to obtain spatial enhancement features of different modalities; the spatial enhancement features of different modalities include infrared spatial enhancement features and visible light spatial enhancement features;

[0071] The channel enhancement features of different modalities are merged through the modal feature fusion module to obtain the channel attention weighted features, and the spatial enhancement features of different modalities are merged to obtain the spatial attention weighted features;

[0072] The channel attention weighted features and the spatial attention weighted features are merged to obtain modal fusion feature information.

[0073] The step of performing gradient-guided channel feature enhancement on different modal image features by the gradient channel attention enhancement unit to obtain the gradient channel attention weight comprises:

[0074] The gradient attention unit and the channel attention unit in the gradient channel attention enhancement unit are used to respectively calculate the gradient attention and channel attention of different modality image features on each channel, and obtain the channel gradient weight and channel feature weight of the corresponding modality image feature;

[0075] According to the channel gradient weights of different modal image features on each channel, the channel feature weights are guided by texture detail gradients to obtain gradient channel attention weights.

[0076] At the same time, the step of performing gradient-guided spatial feature enhancement on different modal image features by the gradient spatial attention control unit to obtain the gradient spatial attention weight includes:

[0077] The gradient attention unit and the spatial attention unit in the gradient spatial attention control unit are used to respectively calculate the gradient attention and the spatial attention of different modal image features at each spatial position, and obtain the spatial gradient weight and the spatial feature weight of the corresponding modal image feature;

[0078] According to the spatial gradient weights of different modal image features at each spatial position, the spatial feature weights are guided by texture detail gradients to obtain the gradient channel attention weights.

[0079] Specifically, in order to solve the problem of brightness inconsistency in the fused image, this embodiment introduces channel attention and spatial attention mechanisms. These two mechanisms can dynamically calculate weights based on specific image information from two dimensions: pixels and channels, so as to more effectively balance the brightness of different areas and solve the problem of brightness inconsistency. At the same time, in order to avoid the texture details of the visible light image being obscured by the thermal radiation information of the infrared image during the fusion process, the multimodal feature fusion network adopted in this embodiment incorporates the gradient attention mechanism on the basis of channel attention and spatial attention. The multimodal feature fusion network uses the Sobel operator to calculate the gradient of the features, and calculates the gradient attention weights of the two modes accordingly. On this basis, the original channel attention weights and spatial attention weights are weakened. This strategy not only helps to retain and highlight the texture details of the visible light image, but also solves the problem that the texture information of the visible light image is covered by the thermal radiation information of the infrared image, and significantly improves the overall quality of the fused image.

[0080] In this embodiment, channel attention is mainly used to evaluate the information of important channels in the fusion process, while spatial attention is used to determine which areas in the image should be emphasized during fusion. By combining these two attention mechanisms, a more accurate and efficient feature fusion strategy can be implemented, thereby improving the quality and information content of the fused image. In addition, based on channel attention and spatial attention, this embodiment further introduces a gradient attention mechanism to highlight the texture features of the image. Specifically, this embodiment uses the Sobel operator to calculate the gradient of each pixel. The Sobel operator consists of two groups of 3x3 matrices, which are used for horizontal and vertical edge detection respectively. For example, for a given modal image feature F, the image of its horizontal edge detection is calculated by the following formula:

[0081]

[0082] Similarly, the image of vertical edge detection is calculated by the following formula:

[0083]

[0084] In the formula, G x and G y They represent the lateral (horizontal) gradient and longitudinal (vertical) gradient of each pixel in the modal image feature F, respectively. These gradient values ​​reflect the edge strength of the image in the horizontal and vertical directions.

[0085] Next, this embodiment combines the lateral gradient G of each pixel x and the longitudinal gradient G y The approximate value of is used to calculate the size of the gradient value G:

[0086]

[0087] After obtaining the gradient value G, this embodiment uses the softmax function to calculate the gradient weight. Specifically, for the infrared (IR) image thermal radiation feature and the visible light (VI) image texture feature, the gradient weights are calculated as follows:

[0088]

[0089] In the formula, Γ IR (x, y) is the gradient weight of the thermal radiation feature of the infrared image at the position (x, y); G IR is the gradient value of the thermal radiation feature of the infrared image; G VI is the gradient value of the texture feature of the visible light image; Γ VI (x, y) is the gradient weight of the visible light image texture feature at position (x, y).

[0090] This embodiment introduces a gradient attention mechanism and adopts the above-mentioned gradient weight calculation method to obtain the channel gradient weight and spatial gradient weight of infrared (IR) and visible light (VI) image modalities respectively, so as to more effectively emphasize and utilize the texture features of the image.

[0091] In this embodiment, the channel attention mechanism focuses on the channel dimension of the image, and assigns corresponding weights to different channels by learning the importance of different channels. In the feature fusion process of infrared and visible light images, the channel attention unit performs global average pooling on the infrared image features and the visible light image features, and calculates the global average value of each channel. For example, for an image feature F with n channels, a channel global pooling vector is obtained by a global average pooling function P(F) (taking into account the height H and width W of the feature). Subsequently, the channel global pooling vector is calculated using a soft-max function to obtain preliminary channel weights of infrared image features (infrared image thermal radiation features) and visible light image features (visible light image texture features). The preliminary channel weights reflect the relative importance of infrared and visible light features in each channel. Specifically, the calculation formula for the preliminary channel weights of infrared image thermal radiation features and visible light image texture features is:

[0092]

[0093] In the formula, α IR is the preliminary channel weight of the thermal radiation feature of the infrared image; is the channel global pooling vector of the thermal radiation feature of the infrared image, which is calculated by global average pooling; is the channel global pooling vector of visible light image texture features; α VI is the preliminary channel weight of the texture feature of the visible light image.

[0094] After calculating the preliminary channel weight, this embodiment compares the preliminary channel weight with the channel gradient weight. Specifically, for each position (x, y) of the infrared image feature and the visible light image feature, the value of the final channel weight (gradient channel attention weight) depends on the corresponding channel gradient weight. If the channel gradient weight is not equal to 0.5, the channel gradient weight is used as the gradient channel attention weight; otherwise, the preliminary channel weight is kept unchanged and used as the gradient channel attention weight. The specific formula is expressed as follows:

[0095]

[0096] In the formula, δ IR (x, y) is the infrared gradient channel attention weight of the infrared image thermal radiation feature at position (x, y); α IR (x, y) is the initial channel weight of the thermal radiation feature of the infrared image at the position (x, y); δ VI (x, y) is the visible light gradient channel attention weight of the visible light image texture feature at position (x, y); α VI (x, y) is the initial channel weight of the visible light image texture feature at position (x, y); is the channel gradient weight of the thermal radiation feature of the infrared image at position (x, y); is the channel gradient weight of the visible light image texture feature at position (x, y).

[0097] Finally, this embodiment uses the gradient channel attention weight to perform weighted summation on the infrared image thermal radiation features and the visible light image texture features to obtain the channel attention weighted features:

[0098] F CA =δ IR (x,y)*F IR (x,y)+δ VI (x,y)*F VI (x,y)

[0099] In the formula, F CA is the channel attention weighted feature; δ IR (x, y) is the infrared gradient channel attention weight; δ VI (x, y) is the attention weight of the visible light gradient channel; F IR (x, y) is the thermal radiation feature of the infrared image; F VI (x, y) is the texture feature of the visible light image; δ IR (x,y)*F IR (x, y) is the infrared channel enhancement feature; δ VI (x,y)*F VI (x, y) is the visible light channel enhancement feature.

[0100] In this embodiment, the spatial attention mechanism focuses on the spatial dimension of the image. It learns the importance of different positions of the image, pays attention to the important areas in the image and assigns corresponding weights to these important areas. In the feature fusion strategy of infrared and visible light images, the spatial attention unit uses different convolution kernels to calculate the feature response of each position. For the thermal radiation features of infrared images and the texture features of visible light images, the spatial attention weight is calculated by the following formula:

[0101]

[0102] In the formula, ‖·‖ represents the L1 norm; β IR (x, y) is the spatial attention weight of the thermal radiation feature of the infrared image; β VI (x,y) is the spatial attention weight of the texture feature of the visible light image.

[0103] Next, this embodiment compares the calculated spatial attention weight with the spatial gradient weight. If the spatial gradient weight is not equal to 0.5, the spatial gradient weight is used as the gradient spatial attention weight; if the spatial gradient weight is equal to 0.5, the spatial attention weight is used as the gradient spatial attention weight. The specific formula is expressed as follows:

[0104]

[0105] In the formula, is the infrared gradient spatial attention weight of the thermal radiation feature of the infrared image at position (x, y); is the attention weight of the visible light gradient channel of the visible light image texture feature at position (x, y); is the spatial gradient weight of the thermal radiation feature of the infrared image at the position (x, y); is the spatial gradient weight of the visible light image texture feature at position (x, y).

[0106] Then, the gradient spatial attention weight is used to perform weighted summation on the infrared and visible light image feature maps to obtain the spatial attention weighted feature:

[0107]

[0108] In the formula, F SA is the spatial attention weighted feature; F IR (x, y) is the thermal radiation feature of the infrared image; F VI (x, y) is the texture feature of the visible light image; Enhance features for infrared space; Enhance features for visible light space.

[0109] Through the above steps, the spatial attention unit can effectively fuse the features of infrared and visible light images, highlight the key information, and finally add the channel attention weighted features and the spatial attention weighted features in equal proportion to obtain the modal fusion feature information, whose mathematical expression is:

[0110]

[0111] Where D is the modality fusion feature information.

[0112] S4. Perform feature reconstruction based on the modal fusion feature information and the original chromaticity channel data of the visible light image to obtain a color fused image.

[0113] In the image reconstruction stage, the main goal of this embodiment is to use the modal fusion feature information to generate a clear and realistic visual image. In order to solve the problem that the traditional fusion method cannot effectively reconstruct a high-quality image from a high-dimensional and non-intuitive feature representation, the image reconstruction operation of this embodiment is completed by a convolution layer with a 1×1 kernel, which is padded with 0, and a Tanh activation layer is added to further enhance the output effect of the image. Specifically, this embodiment performs feature reconstruction according to the modal fusion feature information to obtain reconstructed brightness channel data, and performs color space conversion in RGB format based on the reconstructed brightness channel data and the visible light image chromaticity channel data, converting it into RGB format to obtain a color fused image I F , Figure 6 This is an example of the final color fusion image. Figure 6 The images from top to bottom are visible light images, infrared images, and color fused images. The fused image not only retains the thermal radiation information in the infrared image, but also incorporates the rich texture and color details in the visible light image, making the image information richer and more comprehensive. This embodiment accurately combines the fused brightness channel data with the original chrominance channel data through color space conversion technology, and converts the YUV color space back to the RGB color space, thereby ensuring the color restoration accuracy of the fused image.

[0114] In the infrared and visible light image fusion method provided in this embodiment, this embodiment adopts a composite loss function to optimize the model. The composite loss function includes three parts: MS-SSIM loss function, L1 loss function and visual perception loss function. Among them, MS-SSIM is an extended version of SSIM, which is used to evaluate the similarity of two images. It can well preserve the edge and detail information of the image. However, the use of MS-SSIM loss function is prone to brightness and color deviation. In order to solve this problem, this embodiment introduces L1 loss function. L1 loss function calculates the absolute value of the difference between the model prediction value and the true value. To keep the brightness and color of the image unchanged, in addition to the MS-SSIM loss function and the L1 loss function, this embodiment also introduces a visual perception loss function, which measures the difference between images in the feature space instead of directly comparing them in the pixel space. The visual perception loss function can help the model retain the visual quality and content information of the image, and ensure the perceptual quality and content consistency of the image. The calculation of the visual perception loss function is based on the features extracted from each layer of the pre-trained vgg19 network, and the mean square error function MSE is used to measure the difference between the features. Therefore, the mathematical expression of the composite loss function used in this embodiment is:

[0115]

[0116] In the formula, is the composite loss function; It is the first part of the composite loss function, which combines the MS-SSIM loss function and the L1 loss function to measure the difference between the model-predicted image and the actual image; σ is the standard deviation, which is used to calculate the standard deviation term and covariance term in the MS-SSIM loss; is the visual perception loss function; α is a balance parameter used to adjust the weights of the MS-SSIM loss function and the L1 loss function in the composite loss function; The MS-SSIM loss function uses the Multi-Scale Structural Similarity Index to measure the similarity between two images. is the Gaussian distribution parameter; is the L1 loss function, which is used to calculate the absolute value of the difference between the model predicted image and the actual image, and find the average; n p is the number of pixels in the feature map, used to average the differences of all pixels when calculating the L1 loss function; i is the i-th input image; i is the i-th output image; M is the number of different scales, which is used for multi-scale analysis in MS-SSIM loss calculation; μ I is the mean of the input image; μ O is the mean of the output image; c1 and c2 are constant terms in the MS-SSIM loss function; σ IO is the covariance between the input image and the output image, used to calculate the MS-SSIM loss; σ I is the standard deviation of the input image; σ O is the standard deviation of the output image; Φ l (I) is the input image feature function extracted by the first layer of the pre-trained vgg19 network; Φ l (Ο) is the output image feature function extracted from the first layer of the pre-trained vgg19 network; β m and γ m They are all weight parameters in the MS-SSIM loss function, which are used to adjust the impact of the mean and covariance terms at different scales on the final loss; MSE is the mean square error function.

[0117] The embodiment of the present invention provides an infrared and visible light image fusion method, the method acquires an infrared image and a visible light image, and converts the visible light image into a YUV color space by using a color space conversion method, and separates the original brightness channel data and the original chromaticity channel data of the visible light image; uses a feature extraction network to extract features of the original brightness channel data of the infrared image and the visible light image, respectively, to obtain different modal image features; wherein the different modal image features include infrared image thermal radiation features and visible light image texture features; introduces a gradient attention mechanism, and uses a multimodal feature fusion network to perform gradient-guided feature fusion on the features of different modal images to obtain modal fusion feature information; and performs feature reconstruction based on the modal fusion feature information and the original chromaticity channel data of the visible light image to obtain a color fused image. Compared with the prior art, the method combines the feature extraction network of the Slide Transformer and the gradient operator module, calculates the gradient weights of the two image modal features of infrared and visible light, and locally affects the weights of channel attention and spatial attention, thereby achieving the highlighting of texture details, avoiding the texture detail blurring problem that occurs in the traditional fusion method, and thus achieving accurate extraction of infrared image and visible light image features.

[0118] It should be noted that the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0119] In one embodiment, Figure 7 As shown, an embodiment of the present invention provides an infrared and visible light image fusion system, the system comprising:

[0120] A space conversion module is used to obtain an infrared image and a visible light image, and convert the visible light image into a YUV color space using a color space conversion method, so as to separate and obtain original brightness channel data and original chrominance channel data of the visible light image;

[0121] A feature extraction module, used to use a feature extraction network to perform feature extraction on the original brightness channel data of the infrared image and the visible light image respectively, to obtain different modality image features; wherein the different modality image features include infrared image thermal radiation features and visible light image texture features;

[0122] The feature fusion module is used to introduce the gradient attention mechanism and use the multimodal feature fusion network to perform gradient-guided feature fusion on the features of different modal images to obtain modal fusion feature information;

[0123] The feature reconstruction module is used to perform feature reconstruction based on the modal fusion feature information and the original chromaticity channel data of the visible light image to obtain a color fused image.

[0124] For the specific definition of an infrared and visible light image fusion system, please refer to the above-mentioned definition of an infrared and visible light image fusion method, which will not be repeated here. A person of ordinary skill in the art will appreciate that the various modules and steps described in conjunction with the embodiments disclosed in this application can be implemented in hardware, software, or a combination of both. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0125] An embodiment of the present invention provides an infrared and visible light image fusion system, wherein the system acquires an infrared image and a visible light image through a space conversion module, and converts the visible light image into a YUV color space using a color space conversion method, and separates the original brightness channel data and the original chromaticity channel data of the visible light image; a feature extraction module uses a feature extraction network to perform feature extraction on the original brightness channel data of the infrared image and the visible light image, respectively, to obtain different modal image features; wherein the different modal image features include infrared image thermal radiation features and visible light image texture features; a feature fusion module introduces a gradient attention mechanism, and uses a multimodal feature fusion network to perform gradient-guided feature fusion on different modal image features, to obtain modal fusion feature information; a feature reconstruction module performs feature reconstruction based on the modal fusion feature information and the original chromaticity channel data of the visible light image, to obtain a color fused image. Compared with the existing technology, this system combines the feature extraction network of Slide Transformer and gradient operator modules, calculates the gradient weights of the two image modal features of infrared and visible light, and locally affects the weights of channel attention and spatial attention, thereby highlighting the texture details and avoiding the problem of blurred texture details in traditional fusion methods, thereby achieving accurate extraction of infrared image and visible light image features.

[0126] Figure 8 A computer device provided by an embodiment of the present invention includes a memory, a processor and a transceiver, which are connected via a bus; the memory is used to store a set of computer program instructions and data, and can transmit the stored data to the processor, and the processor can execute the program instructions stored in the memory to perform the steps of the above method.

[0127] The memory may include a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories; the processor may be a central processing unit, a microprocessor, an application-specific integrated circuit, a programmable logic device, or a combination thereof. By way of example but not limitation, the programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic, or any combination thereof.

[0128] Additionally, the memory may be a physically separate unit or may be integrated with the processor.

[0129] It can be understood by those skilled in the art that Figure 8 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have the same component arrangement.

[0130] In one embodiment, the present invention provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the above method are implemented.

[0131] An infrared and visible light image fusion method, system, device and storage medium provided by an embodiment of the present invention, wherein the infrared and visible light image fusion method effectively extracts thermal radiation information of infrared images and texture information of visible light images by combining a feature extraction network of a Slide Transformer and a Gradient Operator, thereby improving the accuracy and efficiency of feature extraction; at the same time, during the fusion process, by calculating the gradient weights of the two image modal features of infrared and visible light, and locally affecting the weights of channel attention and spatial attention, the texture details are highlighted, and the problem of blurred texture details in traditional fusion methods is avoided, thereby achieving accurate extraction of infrared image and visible light image features.

[0132] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., an SSD), etc.

[0133] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing related hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods.

[0134] The above-mentioned embodiments only express several preferred implementation modes of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in the technical field, several improvements and substitutions can be made without departing from the technical principles of the present invention, and these improvements and substitutions should also be regarded as the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be based on the protection scope of the claims.

Claims

1. A method for fusing infrared and visible light images, characterized in that: The following steps are involved: Acquire an infrared image and a visible light image, and convert the visible light image into a YUV color space using a color space conversion method, and separate and obtain original brightness channel data and original chrominance channel data of the visible light image; Using a feature extraction network to extract features from the original brightness channel data of the infrared image and the visible light image, respectively, to obtain different modality image features; wherein the different modality image features include infrared image thermal radiation features and visible light image texture features; The gradient attention mechanism is introduced, and the multimodal feature fusion network is used to perform gradient-guided feature fusion on the features of different modal images to obtain modal fusion feature information; Feature reconstruction is performed based on the modality fusion feature information and the original chromaticity channel data of the visible light image to obtain a color fused image.

2. The infrared and visible light image fusion method according to claim 1, characterized in that: The feature extraction network includes a first convolutional layer and at least three spatiotemporal graph convolutional modules connected in cascade; the output of the first convolutional layer is connected to the input of each spatiotemporal graph convolutional module using a global residual connection mechanism, and directly forms a residual connection with the output of the last spatiotemporal graph convolutional module; The output of each spatiotemporal graph convolution module skips the next spatiotemporal graph convolution module connected to it and is connected to the input of the third spatiotemporal graph convolution module through a local residual connection; The spatiotemporal graph convolution module includes a global-local visual attention model and a texture detail gradient enhancement model connected in parallel, and the output of the global-local visual attention model forms a residual connection with the output of the texture detail gradient enhancement model; wherein the global-local visual attention model includes a serial number of Slide-Transformer modules and a second convolutional layer; the texture detail gradient enhancement model includes a serial gradient operator module and a third convolutional layer.

3. The infrared and visible light image fusion method according to claim 2, characterized in that: The step of using a feature extraction network to extract features from the original brightness channel data of the infrared image and the visible light image to obtain features of images of different modalities comprises: Performing feature projection on the original brightness channel data of the infrared image and the visible light image respectively through the first convolution layer to obtain corresponding infrared image projection features and visible light brightness channel projection features; The infrared image projection feature and the visible light brightness channel projection feature are connected to the input of each spatiotemporal graph convolution module through a global residual connection, and a plurality of serial spatiotemporal graph convolution modules are used to sequentially learn to balance the hierarchical global-local feature representations of different scales, and a gradient operator is introduced to enhance the capture of texture detail features to obtain a modal high-dimensional feature representation; According to the infrared image projection features and the visible light brightness channel projection features, the modal high-dimensional feature representation is enhanced using a global residual connection mechanism to extract the corresponding infrared image thermal radiation features and visible light image texture features, respectively.

4. The infrared and visible light image fusion method according to claim 1, characterized in that: The multimodal feature fusion network includes a cascade-connected modal gradient attention extraction module and a modal feature fusion module, wherein the modal gradient attention extraction module includes a plurality of modal gradient attention extraction submodules; each of the modal gradient attention extraction submodules includes a gradient channel attention enhancement unit and a gradient space attention regulation unit connected in parallel; The gradient channel attention enhancement unit includes a gradient attention unit and a channel attention unit connected in parallel, and an input of the gradient channel attention enhancement unit forms a residual connection with an output of the gradient channel attention enhancement unit; The gradient spatial attention control unit includes a gradient attention unit and a spatial attention unit connected in parallel, and the input of the gradient spatial attention control unit forms a residual connection with the output of the gradient spatial attention control unit.

5. The infrared and visible light image fusion method according to claim 1, characterized in that: The step of introducing the gradient attention mechanism and using the multimodal feature fusion network to perform gradient-guided feature fusion on different modal image features to obtain modal fusion feature information includes: The gradient channel attention enhancement unit performs gradient-guided channel feature enhancement on the features of different modal images to obtain the gradient channel attention weights; the gradient channel attention weights include the infrared gradient channel attention weights and the visible light gradient channel attention weights; The gradient channel attention weight is used to perform channel texture enhancement on image features of different modalities to obtain channel enhancement features of different modalities; the channel enhancement features of different modalities include infrared channel enhancement features and visible light channel enhancement features; The gradient spatial attention control unit is used to perform gradient-guided spatial feature enhancement on the features of different modal images, and the gradient spatial attention weight is obtained; the gradient spatial attention weight includes the infrared gradient spatial attention weight and the visible light gradient spatial attention weight; The gradient spatial attention weight is used to perform spatial texture enhancement on image features of different modalities to obtain spatial enhancement features of different modalities; the spatial enhancement features of different modalities include infrared spatial enhancement features and visible light spatial enhancement features; The channel enhancement features of different modalities are merged through the modal feature fusion module to obtain the channel attention weighted features, and the spatial enhancement features of different modalities are merged to obtain the spatial attention weighted features; The channel attention weighted features and the spatial attention weighted features are merged to obtain modal fusion feature information.

6. The infrared and visible light image fusion method according to claim 5, characterized in that: The step of performing gradient-guided channel feature enhancement on different modal image features by the gradient channel attention enhancement unit to obtain the gradient channel attention weight comprises: The gradient attention unit and the channel attention unit in the gradient channel attention enhancement unit are used to respectively calculate the gradient attention and channel attention of different modality image features on each channel to obtain channel gradient weights and channel feature weights; According to the channel gradient weights of different modal image features on each channel, the channel feature weights are guided by texture detail gradients to obtain gradient channel attention weights.

7. The infrared and visible light image fusion method according to claim 5, characterized in that: The step of performing gradient-guided spatial feature enhancement on different modal image features by the gradient spatial attention control unit to obtain the gradient spatial attention weight comprises: The gradient attention unit and the spatial attention unit in the gradient spatial attention control unit are used to respectively calculate the gradient attention and the spatial attention of different modal image features at each spatial position to obtain the spatial gradient weight and the spatial feature weight; According to the spatial gradient weights of different modal image features at each spatial position, the spatial feature weights are guided by texture detail gradients to obtain the gradient channel attention weights.

8. An infrared and visible light image fusion system, characterized in that: The system comprises: A space conversion module is used to obtain an infrared image and a visible light image, and convert the visible light image into a YUV color space using a color space conversion method, so as to separate and obtain original brightness channel data and original chrominance channel data of the visible light image; A feature extraction module, used to use a feature extraction network to perform feature extraction on the original brightness channel data of the infrared image and the visible light image respectively, to obtain different modality image features; wherein the different modality image features include infrared image thermal radiation features and visible light image texture features; The feature fusion module is used to introduce the gradient attention mechanism and use the multimodal feature fusion network to perform gradient-guided feature fusion on the features of different modal images to obtain modal fusion feature information; The feature reconstruction module is used to perform feature reconstruction based on the modal fusion feature information and the original chromaticity channel data of the visible light image to obtain a color fused image.

9. A computer device, characterized in that: The computer device comprises a processor and a memory, wherein the processor is connected to the memory, the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory, so that the computer device executes the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed, the method according to any one of claims 1 to 7 is implemented.

Images