An infrared and visible light image fusion method, system, device and storage medium

Abstract

The present application relates to the technical field of image processing, and particularly relates to an infrared and visible light image fusion method, system, device and storage medium, comprising obtaining an infrared image and a visible light image, and converting the visible light image to a YUV color space by using a color space conversion method to obtain original luminance channel data and original chrominance channel data of the visible light image; performing feature extraction on the original luminance channel data of the infrared image and the visible light image respectively to obtain different modal image features; introducing a gradient attention mechanism to perform gradient-guided feature fusion on the different modal image features to obtain modal fusion feature information; and performing feature reconstruction according to the modal fusion feature information and the original chrominance channel data of the visible light image to obtain a color fusion image. The present application combines a feature extraction network and gradient weights of different modal image features to highlight and retain texture details, thereby improving the quality of infrared image and visible light image fusion.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to an infrared and visible light image fusion method, system, device and storage medium. BACKGROUND

[0002] Since a single modality image often cannot provide comprehensive scene information, multi-modal image fusion can fuse scene information of different modalities together to generate a fusion image containing more information. Therefore, infrared and visible light image fusion technology has been widely concerned and researched in recent years. Fusion of images of these two modalities can provide complementary information, thereby improving the performance of target detection, tracking, scene segmentation and other applications. However, traditional image fusion technologies are mostly based on spatial domain and transform domain methods, such as Laplacian pyramid, discrete wavelet transform and down-sampling contourlet transform, etc. These methods have some limitations in practical applications, such as insufficient robustness, difficulty in coping with complex and variable scenes, limited generalization ability, difficulty in adapting to different types of images, and large consumption of computing resources. These methods can cause loss of detail information in the fusion process, thereby affecting the quality of the final image.

[0003] With the development of deep learning, infrared and visible light image fusion networks based on convolutional neural networks have made significant progress. Such networks realize automatic extraction and fusion of source image features through key components such as feature extraction, feature fusion and image reconstruction, and reconstruct the fused features into images of original resolution. However, existing networks still have challenges in feature extraction, especially the problem of balancing local features and global features. Local features are crucial for capturing image details, while global features help understand the overall structure and relationship of the image.

[0004] In addition, existing fusion algorithms also have deficiencies in feature fusion strategies. Traditional fusion strategies often use simple channel concatenation or fixed weight ratio for fusion, lacking adaptive consideration of the importance of different features, which leads to problems such as inconsistent brightness, blurred or even lost texture information in the fused image. Especially when balancing the differences between infrared and visible light images, traditional fusion strategies often fail to achieve ideal results. Therefore, how to improve feature extraction and fusion strategies to achieve more efficient and accurate infrared and visible light image fusion is an important direction of current research. SUMMARY

[0005] To solve the above technical problems, the present application provides an infrared and visible light image fusion method, system, device and storage medium.

[0006] In a first aspect, the present application provides an infrared and visible light image fusion method, the method comprising the following steps:

[0007] Infrared and visible light images are acquired, and the visible light image is converted to the YUV color space using a color space conversion method to separate the original luminance channel data and the original chrominance channel data of the visible light image.

[0008] Feature extraction networks are used to extract features from the raw brightness channel data of the infrared image and the visible light image respectively, to obtain image features of different modalities; wherein, the image features of different modalities include thermal radiation features of the infrared image and texture features of the visible light image;

[0009] A gradient attention mechanism is introduced, and a multimodal feature fusion network is used to perform gradient-guided feature fusion of image features from different modalities to obtain modal fusion feature information;

[0010] Based on the modal fusion feature information and the original chroma channel data of the visible light image, feature reconstruction is performed to obtain a color fusion image.

[0011] In a further embodiment, the feature extraction network includes a cascaded first convolutional layer and at least three spatiotemporal graph convolutional modules; the output of the first convolutional layer is connected to the input of each spatiotemporal graph convolutional module using a global residual connection mechanism, and forms a residual connection directly with the output of the last spatiotemporal graph convolutional module.

[0012] The output of each spatiotemporal graph convolutional module skips the next spatiotemporal graph convolutional module connected to it and is connected to the input of the third spatiotemporal graph convolutional module through a local residual connection method;

[0013] The spatiotemporal graph convolution module includes a global-local visual attention model and a texture detail gradient enhancement model connected in parallel, and the output of the global-local visual attention model and the output of the texture detail gradient enhancement model form a residual connection; wherein, the global-local visual attention model includes a series of Slide-Transformer modules and a second convolutional layer; the texture detail gradient enhancement model includes a series of gradient operator modules and a third convolutional layer.

[0014] In a further embodiment, the step of using a feature extraction network to extract features from the original brightness channel data of the infrared image and the visible light image respectively to obtain image features of different modalities includes:

[0015] The first convolutional layer performs feature projection on the original brightness channel data of the infrared image and the visible light image respectively to obtain the corresponding infrared image projection features and visible light brightness channel projection features.

[0016] The infrared image projection feature and the visible light brightness channel projection feature are connected with the input of each spatio-temporal graph convolution module through a global residual connection, a plurality of serial spatio-temporal graph convolution modules are used to sequentially learn balanced hierarchical global-local feature representations of different scales, a gradient operator is introduced to capture and enhance texture detail features, and a modal high-dimensional feature representation is obtained;

[0017] According to the infrared image projection feature and the visible light brightness channel projection feature, the modal high-dimensional feature representation is enhanced by using a global residual connection mechanism, and corresponding infrared image thermal radiation features and visible light image texture features are extracted.

[0018] In further embodiments, the multi-modal feature fusion network comprises a modal gradient attention extraction module and a modal feature fusion module connected in cascade, the modal gradient attention extraction module comprises a plurality of modal gradient attention extraction sub-modules; each modal gradient attention extraction sub-module comprises a gradient channel attention enhancement unit and a gradient spatial attention regulation unit connected in parallel;

[0019] The gradient channel attention enhancement unit comprises a gradient attention unit and a channel attention unit connected in parallel, and the input of the gradient channel attention enhancement unit and the output of the gradient channel attention enhancement unit form a residual connection;

[0020] The gradient spatial attention regulation unit comprises a gradient attention unit and a spatial attention unit connected in parallel, and the input of the gradient spatial attention regulation unit and the output of the gradient spatial attention regulation unit form a residual connection.

[0021] In further embodiments, the step of introducing a gradient attention mechanism and using a multi-modal feature fusion network to perform gradient-guided feature fusion on different modal image features to obtain modal fusion feature information comprises:

[0022] The gradient channel attention enhancement unit is used to perform gradient-guided channel feature enhancement on different modal image features respectively to obtain gradient channel attention weights; the gradient channel attention weights comprise infrared gradient channel attention weights and visible light gradient channel attention weights;

[0023] The gradient channel attention weights are used to perform channel texture enhancement on different modal image features to obtain channel enhanced features of different modalities; the channel enhanced features of different modalities comprise infrared channel enhanced features and visible light channel enhanced features;

[0024] The gradient spatial attention regulation unit is used to perform gradient-guided spatial feature enhancement on different modal image features respectively to obtain gradient spatial attention weights; the gradient spatial attention weights comprise infrared gradient spatial attention weights and visible light gradient spatial attention weights;

[0025] The gradient spatial attention weight is used to perform spatial texture enhancement on different modal image features to obtain spatial enhanced features of different modalities, and the spatial enhanced features of different modalities include infrared spatial enhanced features and visible light spatial enhanced features;

[0026] The channel enhanced features of different modalities are merged by the modal feature fusion module to obtain channel attention weighted features, and the spatial enhanced features of different modalities are merged to obtain spatial attention weighted features;

[0027] The channel attention weighted features and the spatial attention weighted features are merged to obtain modal fusion feature information.

[0028] In a further embodiment, the step of performing gradient-guided channel feature enhancement on different modal image features by the gradient channel attention enhancement unit to obtain gradient channel attention weights comprises:

[0029] The gradient attention unit and the channel attention unit in the gradient channel attention enhancement unit are used to respectively calculate gradient attention and channel attention of different modal image features on each channel to obtain channel gradient weights and channel feature weights;

[0030] The channel feature weights are gradient-guided by the channel gradient weights of different modal image features on each channel to obtain gradient channel attention weights.

[0031] In a further embodiment, the step of performing gradient-guided spatial feature enhancement on different modal image features by the gradient spatial attention regulation unit to obtain gradient spatial attention weights comprises:

[0032] The gradient attention unit and the spatial attention unit in the gradient spatial attention regulation unit are used to respectively calculate gradient attention and spatial attention of different modal image features on each spatial position to obtain spatial gradient weights and spatial feature weights;

[0033] The spatial feature weights are gradient-guided by the spatial gradient weights of different modal image features on each spatial position to obtain gradient channel attention weights.

[0034] In a second aspect, the present application provides an infrared and visible light image fusion system, which comprises:

[0035] A spatial conversion module is configured to obtain an infrared image and a visible light image, and convert the visible light image to a YUV color space by using a color space conversion method to separate original luminance channel data and original chrominance channel data of the visible light image;

[0036] a feature extraction module configured to perform feature extraction on original luminance channel data of the infrared image and the visible light image respectively by using a feature extraction network to obtain different modal image features, wherein the different modal image features include infrared image thermal radiation features and visible light image texture features;

[0037] a feature fusion module configured to introduce a gradient attention mechanism and perform gradient-guided feature fusion on the different modal image features by using a multi-modal feature fusion network to obtain modal fusion feature information;

[0038] a feature reconstruction module configured to perform feature reconstruction according to the modal fusion feature information and original chrominance channel data of the visible light image to obtain a color fusion image.

[0039] In a third aspect, the present application further provides a computer device comprising a processor and a memory, wherein the processor is connected to the memory, the memory is configured to store a computer program, and the processor is configured to execute the computer program stored in the memory to enable the computer device to perform the steps of the above method.

[0040] In a fourth aspect, the present application further provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is executed by a processor to perform the steps of the above method.

[0041] The present application provides an infrared and visible light image fusion method, system, device and storage medium. The method comprises the following steps: acquiring an infrared image and a visible light image, and converting the visible light image to a YUV color space by using a color space conversion method to separate original luminance channel data and original chrominance channel data of the visible light image; performing feature extraction on the original luminance channel data of the infrared image and the visible light image respectively by using a feature extraction network to obtain different modal image features, wherein the different modal image features include infrared image thermal radiation features and visible light image texture features; introducing a gradient attention mechanism and performing gradient-guided feature fusion on the different modal image features by using a multi-modal feature fusion network to obtain modal fusion feature information; and performing feature reconstruction according to the modal fusion feature information and the original chrominance channel data of the visible light image to obtain a color fusion image. Compared with the prior art, the method combines a feature extraction network of a Slide Transformer and a gradient operator module, calculates gradient weights of infrared and visible light image modal features, and locally affects weights of channel attention and spatial attention, highlights texture details, avoids the problem of blurred texture details in traditional fusion methods, and thus realizes accurate extraction of infrared image and visible light image features. BRIEF DESCRIPTION OF DRAWINGS

[0042] Figure 1 is a flowchart of an infrared and visible light image fusion method provided by an embodiment of the present application;

[0043] Figure 2 is a general framework diagram of an infrared and visible light image fusion process provided by an embodiment of the present application;

[0044] Figure 3 is a network structure diagram of feature extraction provided by an embodiment of the present application;

[0045] Figure 4 is a structure diagram of a spatio-temporal graph convolution module provided by an embodiment of the present application;

[0046] Figure 5 is a structure diagram of a multi-modal feature fusion network provided by an embodiment of the present application;

[0047] Figure 6 is an example diagram of a fused image provided by an embodiment of the present application;

[0048] Figure 7 is a block diagram of an infrared and visible light image fusion system provided by an embodiment of the present application;

[0049] Figure 8 is a structure diagram of a computer device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0050] The embodiments of the present application will be described in detail below with reference to the accompanying drawings. The embodiments are presented only for the purpose of illustration and should not be understood as limiting the present application. The accompanying drawings are only for reference and illustration, and do not limit the scope of patent protection of the present application, because many changes can be made to the present application without departing from the spirit and scope of the present application.

[0051] Reference Figure 1 , an embodiment of the present application provides an infrared and visible light image fusion method, as shown in Figure 1 , the method comprises the following steps:

[0052] S1. Obtain an infrared image and a visible light image, and convert the visible light image to a YUV color space by using a color space conversion method to separate original luminance channel data and original chrominance channel data of the visible light image.

[0053] As Figure 2As shown, the traditional method is often difficult to accurately extract the features of the two images when processing infrared and visible light image fusion, resulting in incomplete or distorted image information after fusion. In order to ensure the fusion quality of the color image, the color space conversion method is used to convert the input visible light image from RGB format to YUV format before feature extraction. This conversion process separates and encodes the luminance channel data (Y) and chrominance channel data (U and V) of the visible light image, so as to more accurately control the brightness information and color information of the image in the subsequent feature extraction stage.

[0054] S2. Feature extraction is performed on the original luminance channel data of the infrared image and the visible light image respectively using the feature extraction network to obtain different modal image features; wherein the different modal image features include infrared image thermal radiation features and visible light image texture features.

[0055] In order to solve the problem of insufficient local and global feature extraction, the feature extraction network in this embodiment introduces a spatio-temporal graph convolution module (STG-Block) combining Slide-Transformer and Gradient Operator. Specifically, the Sobel operator is used as the Gradient Operator in the STG-Block, which significantly enhances the network's ability to capture fine-grained details of the image. At the same time, Slide-Transformer effectively captures local features by limiting the receptive field of the query to only cover its adjacent pixels. Compared with other attention modes, Slide-Transformer not only inherits the advantages of translation equivariance and local inductive bias of convolution, but also retains the flexibility and data dependency of self-attention mechanism. Therefore, the STG-Block can consider both global and local features during feature extraction, thereby improving the sufficiency and accuracy of feature extraction. In this embodiment, the feature extraction network includes a first convolution layer and at least three spatio-temporal graph convolution modules connected in cascade. For ease of understanding, as shown in the figure, this embodiment takes the feature extraction network including a first convolution layer (Conv-in) and three spatio-temporal graph convolution modules connected in cascade as an example. In this embodiment, the output of the first convolution layer is connected to the input of each spatio-temporal graph convolution module using a global residual connection mechanism, and is directly connected to the output of the last spatio-temporal graph convolution module to form a residual connection. The output of each spatio-temporal graph convolution module is connected to the input of the third spatio-temporal graph convolution module through a local residual connection mechanism, skipping the next spatio-temporal graph convolution module connected thereto. Figure 3

[0056] ​Specifically, the output of the first convolutional layer will be one of the inputs of the subsequent spatio-temporal graph convolution module, that is, the output of the first convolutional layer is not only directly input to the first spatio-temporal graph convolution module, but also connected with the input of each subsequent spatio-temporal graph convolution module through global residual connection; in addition, the input of each spatio-temporal graph convolution module (except the first one) not only includes the output of the previous spatio-temporal graph convolution module, but also includes the output of the first convolutional layer (through global residual connection) and the output of the previous skipped spatio-temporal graph convolution module (through local residual connection), the output of the first spatio-temporal graph convolution module skips the adjacent second spatio-temporal graph convolution module and directly serves as an additional input (through local residual connection) of the third module and subsequent modules, the embodiment uses multiple serial spatio-temporal graph convolution modules to extract different levels of features, from low-level edges and textures to high-level global semantic information, which can sequentially learn and balance global and local feature representations, these modules not only effectively solve the problem of imbalance between global and local features in traditional feature extraction, but also enable the network to capture dynamic changes and spatial structure information in the image through its inherent spatio-temporal modeling capability, in order to further optimize the feature extraction process, the embodiment introduces a Sobel operator, which can highlight the texture details in the image, thereby enhancing the network's ability to capture texture features.

[0057] In the embodiment, the spatio-temporal graph convolution module includes a global-local visual attention model and a texture detail gradient enhancement model connected in parallel, and the output of the global-local visual attention model and the output of the texture detail gradient enhancement model form a residual connection; wherein the global-local visual attention model includes a plurality of serial Slide-Transformer modules and a second convolutional layer; the texture detail gradient enhancement model includes a gradient operator module and a third convolutional layer connected in series, for the convenience of understanding, Figure 4 Exemplarily, a spatio-temporal graph convolution module is shown in which the global-local visual attention model includes two serial Slide-Transformer modules and a second convolutional layer, those skilled in the art can adjust the number of Slide-Transformer modules according to specific implementation, which is not limited in the present application, in the embodiment, the STG-Block is formed by the residual connection of Slide-Transformer and Gradient Operator, the embodiment realizes accurate extraction of infrared image and visible light image features by combining the feature extraction network of Slide-Transformer and Gradient Operator, Slide-Transformer solves the problem of imbalance between local features and global features, and Gradient Operator can further enhance the extraction of texture details.

[0058] It should be noted that in order to overcome the low efficiency of the feature shift operation in the traditional method, in the local attention mechanism of the Slide-Transformer, the Slide-Transformer module in the embodiment adopts a deep convolution of a convolution kernel to replace the low efficiency of the feature shift. Specifically, the Slide-Transformer module limits the Key and Value to fixed adjacent positions through kernel weights, and in order to further enhance the flexibility of local attention, the Slide-Transformer module introduces parallel convolution paths, the convolution kernel parameters of these parallel paths are randomly initialized and optimized through the back propagation algorithm in the training stage, so as to be able to learn to capture diversified features. In this way, the Slide-Transformer module not only maintains the advantages of local attention, such as translation equivariance and local inductive bias, but also enhances its flexibility, so that it can more effectively extract local features in the image. At the same time, combined with other structural modules such as the GradientOperator module in the STG-Block, the feature extraction network can balance global and local features, providing rich feature representations for subsequent image processing tasks.

[0059] Based on the above feature extraction network, the step of extracting different modal image features by using the feature extraction network to respectively extract features from the original luminance channel data of the infrared image and the visible light image includes:

[0060] The original luminance channel data of the infrared image and the visible light image are respectively projected through the first convolution layer to obtain corresponding infrared image projection features and visible light luminance channel projection features;

[0061] The infrared image projection features and the visible light luminance channel projection features are connected to the input of each spatio-temporal graph convolution module through global residual connection, and a plurality of serial spatio-temporal graph convolution modules are used to sequentially learn balanced hierarchical global-local feature representations of different scales, and a gradient operator is introduced to enhance texture detail feature capture, to obtain modal high-dimensional feature representations;

[0062] According to the infrared image projection features and the visible light luminance channel projection features, the modal high-dimensional feature representations are enhanced by using a global residual connection mechanism, and corresponding infrared image thermal radiation features and visible light image texture features are extracted.

[0063] In the feature extraction network proposed in the embodiment, the first convolutional layer (Conv-in) is a 1x1 convolutional layer, which is used to project the input image to obtain corresponding infrared image projection features and visible light brightness channel projection features, providing input for the subsequent STG-Block. The STG-Block combines modules such as Slide-Transformer and is used for local feature extraction. Specifically, the feature extraction network combining Slide Transformer and Gradient Operator proposed in the embodiment uses the Slide Transformer module in the global-local visual attention model to sequentially learn hierarchical global-local feature representations of different scales. The parallel convolution of the Slide Transformer realizes the extraction of local features and global features, and the Gradient Operator module adopts the Sobel operator to further capture and enhance the texture detail features, obtaining modal high-dimensional feature representations.

[0064] S3. Introduce gradient attention mechanism, use multi-modal feature fusion network to guide the feature fusion of different modal image features by gradient, and obtain modal fusion feature information.

[0065] In the image fusion process, due to the differences and interference between different images, the texture details are often blurred. In the embodiment, the gradient weight of the feature is calculated, and the weight of the local channel attention and spatial attention is influenced, so as to highlight the texture details and avoid the problem of blurred texture details. In the embodiment, the multi-modal feature fusion network includes a modal gradient attention extraction module and a modal feature fusion module connected in cascade, the modal gradient attention extraction module includes a plurality of modal gradient attention extraction sub-modules, each modal gradient attention extraction sub-module includes a gradient channel attention enhancement unit and a gradient spatial attention regulation unit connected in parallel, the gradient channel attention enhancement unit includes a gradient attention unit and a channel attention unit connected in parallel, the input of the gradient channel attention enhancement unit and the output of the gradient channel attention enhancement unit form a residual connection, the gradient spatial attention regulation unit includes a gradient attention unit and a spatial attention unit connected in parallel, the input of the gradient spatial attention regulation unit and the output of the gradient spatial attention regulation unit form a residual connection, Figure 5 is a structural diagram of the multi-modal feature fusion network.

[0066] In the embodiment, the step of introducing the gradient attention mechanism, using the multi-modal feature fusion network to guide the feature fusion of different modal image features by gradient, and obtaining modal fusion feature information includes:

[0067] The gradient channel attention enhancement unit respectively performs gradient-guided channel feature enhancement on different modal image features to obtain gradient channel attention weights; the gradient channel attention weights include infrared gradient channel attention weights and visible light gradient channel attention weights;

[0068] The gradient channel attention weights are used for channel texture enhancement on different modal image features to obtain channel enhanced features of different modalities; the channel enhanced features of different modalities include infrared channel enhanced features and visible light channel enhanced features;

[0069] The gradient spatial attention regulation unit respectively performs gradient-guided spatial feature enhancement on different modal image features to obtain gradient spatial attention weights; the gradient spatial attention weights include infrared gradient spatial attention weights and visible light gradient spatial attention weights;

[0070] The gradient spatial attention weights are used for spatial texture enhancement on different modal image features to obtain spatial enhanced features of different modalities; the spatial enhanced features of different modalities include infrared spatial enhanced features and visible light spatial enhanced features;

[0071] The modal feature fusion module combines the channel enhanced features of different modalities to obtain channel attention weighted features, and combines the spatial enhanced features of different modalities to obtain spatial attention weighted features;

[0072] The channel attention weighted features and the spatial attention weighted features are combined to obtain modal fusion feature information.

[0073] The gradient channel attention enhancement unit respectively performs gradient-guided channel feature enhancement on different modal image features to obtain gradient channel attention weights, including:

[0074] The gradient attention unit and the channel attention unit in the gradient channel attention enhancement unit are used to respectively calculate the gradient attention and the channel attention of different modal image features on each channel to obtain the channel gradient weight and the channel feature weight of the corresponding modal image feature;

[0075] The channel gradient weight of different modal image features on each channel is used to guide the texture detail gradient of the channel feature weight to obtain the gradient channel attention weight.

[0076] Meanwhile, the gradient spatial attention regulation unit respectively performs gradient-guided spatial feature enhancement on different modal image features to obtain gradient spatial attention weights, including:

[0077] The gradient attention unit and the spatial attention unit in the gradient spatial attention regulation unit are used to respectively calculate gradient attention and spatial attention of different modal image features at each spatial position, so as to obtain spatial gradient weights and spatial feature weights of the corresponding modal image features;

[0078] The spatial gradient weights of different modal image features at each spatial position are used to guide the texture detail gradient of the spatial feature weights, so as to obtain gradient channel attention weights.

[0079] Specifically, in order to solve the problem of incoordination of brightness in the fused image, the embodiment introduces channel attention and spatial attention mechanisms. These two mechanisms can dynamically calculate weights according to specific image information from the dimensions of pixels and channels, so as to more effectively balance the brightness of different regions and solve the problem of incoordination of brightness. In order to avoid the texture details of the visible light image being covered by the thermal radiation information of the infrared image in the fusion process, the multi-modal feature fusion network used in the embodiment integrates gradient attention mechanism on the basis of channel attention and spatial attention. The multi-modal feature fusion network uses a Sobel operator to calculate the gradient of the feature, and calculates the gradient attention weights of the two modalities according to the gradient. On this basis, the original channel attention weights and spatial attention weights are weakened. This strategy not only helps to retain and highlight the texture details of the visible light image, but also solves the problem that the texture information of the visible light image is covered by the thermal radiation information of the infrared image, and significantly improves the overall quality of the fused image.

[0080] In the embodiment, channel attention is mainly used to evaluate the information of important channels in the fusion process, and spatial attention is used to determine which regions in the image should be emphasized in the fusion. Through the combination of the two attention mechanisms, a more accurate and efficient feature fusion strategy can be achieved, so as to improve the quality and information content of the fused image. In addition, on the basis of channel attention and spatial attention, the embodiment further introduces gradient attention mechanism to highlight the texture features of the image. Specifically, the embodiment uses a Sobel operator to calculate the gradient of each pixel. The Sobel operator is composed of two groups of 3x3 matrices, which are used for horizontal and vertical edge detection, respectively. For example, for a given modal image feature F, the horizontal edge detection image is calculated by the following formula:

[0081]

[0082] Similarly, the vertical edge detection image is calculated by the following formula:

[0083]

[0084] where G x and G y respectively represent the horizontal and vertical gradients of each pixel in the modal image feature F, which reflect the edge strength of the image in the horizontal and vertical directions.

[0085] Then, the embodiment calculates the size of the gradient value G by combining the approximate values of the horizontal gradient G x and the vertical gradient G y of each pixel:

[0086]

[0087] After obtaining the gradient value G, the embodiment uses the softmax function to calculate the gradient weight, specifically, for the infrared (IR) image thermal radiation feature and the visible light (VI) image texture feature, the gradient weights are calculated as follows:

[0088]

[0089] where Γ IR (x, y) is the gradient weight of the infrared image thermal radiation feature at position (x, y); G IR is the gradient value of the infrared image thermal radiation feature; G VI is the gradient value of the visible light image texture feature; Γ VI (x, y) is the gradient weight of the visible light image texture feature at position (x, y).

[0090] By introducing the gradient attention mechanism, the embodiment uses the above gradient weight calculation method to obtain the channel gradient weight and the spatial gradient weight of the two modal images of infrared (IR) and visible light (VI) respectively, so as to more effectively emphasize and utilize the texture features of the image.

[0091] In the embodiment, the channel attention mechanism focuses on the channel dimension of the image, and assigns corresponding weights to different channels by learning the importance of the channels. In the feature fusion process of the infrared and visible light images, the channel attention unit performs global average pooling on the infrared image features and the visible light image features to calculate the global average value of each channel. For example, for an image feature F with n channels, the channel global pooling vector is obtained by the global average pooling function P(F) (considering the height H and width W of the feature). Then, the channel global pooling vector is calculated by using the soft-max function to obtain the preliminary channel weight of the infrared image features (infrared image thermal radiation features) and the visible light image features (visible light image texture features). The preliminary channel weight reflects the relative importance of the infrared and visible light features in each channel. Specifically, the calculation formula of the preliminary channel weight of the infrared image thermal radiation features and the visible light image texture features is:

[0092]

[0093] wherein, α IR is the preliminary channel weight of the infrared image thermal radiation features; is the channel global pooling vector of the infrared image thermal radiation features, which is calculated by the global average pooling; is the channel global pooling vector of the visible light image texture features; α VI is the preliminary channel weight of the visible light image texture features.

[0094] After calculating the preliminary channel weight, the embodiment compares the preliminary channel weight with the channel gradient weight. Specifically, for each position (x, y) of the infrared image features and the visible light image features, the value of the final channel weight (gradient channel attention weight) depends on the corresponding channel gradient weight. If the channel gradient weight is not equal to 0.5, the channel gradient weight is used as the gradient channel attention weight. Otherwise, the preliminary channel weight is kept unchanged and used as the gradient channel attention weight. The specific formula is:

[0095]

[0096] wherein, δ IR is the infrared gradient channel attention weight of the infrared image thermal radiation features at the position (x, y); α IR is the preliminary channel weight of the infrared image thermal radiation features at the position (x, y); δ VI is the visible light gradient channel attention weight of the visible light image texture features at the position (x, y); α VI is the preliminary channel weight of the visible light image texture features at the position (x, y). is the channel gradient weight of the infrared image thermal radiation feature at position (x, y); is the channel gradient weight of the visible light image texture feature at position (x, y).

[0097] Finally, the embodiment utilizes the gradient channel attention weight to weight and sum the infrared image thermal radiation feature and the visible light image texture feature to obtain a channel attention weighted feature:

[0098] F CA = δ IR (x,y)*F IR (x,y)+ δ VI (x,y)*F VI (x,y)

[0099] In the formula, F CA is the channel attention weighted feature; δ IR (x,y) is the infrared gradient channel attention weight; δ VI (x,y) is the visible light gradient channel attention weight; F IR (x,y) is the infrared image thermal radiation feature; F VI (x,y) is the visible light image texture feature; δ IR (x,y)*F IR (x,y) is the infrared channel enhanced feature; δ VI (x,y)*F VI (x,y) is the visible light channel enhanced feature.

[0100] In the embodiment, the spatial attention mechanism focuses on the spatial dimension of the image, and by learning the importance of different positions of the image, it focuses on important regions in the image and assigns corresponding weights to these important regions. In the feature fusion strategy of the infrared and visible light images, the spatial attention unit utilizes different convolution kernels to calculate the feature response of each position. For the infrared image thermal radiation feature and the visible light image texture feature, the spatial attention weight is calculated by the following formula:

[0101]

[0102] In the formula, ‖·‖ represents the L1 norm; β IR (x,y) is the spatial attention weight of the infrared image thermal radiation feature; β VI (x,y) is the spatial attention weight of the visible light image texture feature.

[0103] Then, the embodiment compares the calculated spatial attention weight with the spatial gradient weight, and if the spatial gradient weight is not equal to 0.5, the spatial gradient weight is used as the gradient spatial attention weight; if the spatial gradient weight is equal to 0.5, the spatial attention weight is used as the gradient spatial attention weight, and the specific formula is as follows:

[0104]

[0105] In the formula, is the infrared gradient spatial attention weight of the infrared image thermal radiation feature at position (x, y); is the visible light gradient channel attention weight of the visible light image texture feature at position (x, y); is the spatial gradient weight of the infrared image thermal radiation feature at position (x, y); is the spatial gradient weight of the visible light image texture feature at position (x, y).

[0106] Then, the gradient spatial attention weight is used to weight and sum the infrared and visible light image feature maps to obtain a spatial attention weighted feature:

[0107]

[0108] In the formula, F SA is the spatial attention weighted feature; F IR (x, y) is the infrared image thermal radiation feature; F VI (x, y) is the visible light image texture feature; is the infrared spatial enhanced feature; is the visible light spatial enhanced feature.

[0109] Through the above steps, the spatial attention unit can effectively fuse the features of the infrared and visible light images, highlight the key information, and finally add the channel attention weighted feature and the spatial attention weighted feature in equal proportion to obtain the modal fusion feature information, and the mathematical expression is as follows:

[0110]

[0111] In the formula, D is the modal fusion feature information.

[0112] S4. Feature reconstruction is performed according to the modal fusion feature information and the original chrominance channel data of the visible light image to obtain a color fusion image.

[0113] In the image reconstruction stage, the main goal of the embodiment is to generate a clear and real visual image by using the modal fusion feature information. In order to solve the problem that the existing fusion methods cannot effectively reconstruct high-quality images from high-dimensional and non-intuitive feature representations, the image reconstruction operation of the embodiment is completed by a 1x1 kernel convolution layer, which adopts 0 padding and is additionally provided with a Tanh activation layer to further enhance the output effect of the image. Specifically, the embodiment reconstructs features according to the modal fusion feature information, obtains reconstructed luminance channel data, and performs color space conversion in RGB format according to the reconstructed luminance channel data and the visible light image chrominance channel data, converts it into RGB format, and obtains a color fusion image I F , Figure 6 is an example of the final color fusion image, Figure 6 The images from top to bottom are the visible light image, the infrared image and the color fusion image. The fused image not only retains the thermal radiation information in the infrared image, but also integrates the rich texture and color details in the visible light image, making the image information more rich and comprehensive. The color space conversion technology in the embodiment accurately combines the fused luminance channel data and the original chrominance channel data to convert from YUV color space to RGB color space, ensuring the accuracy of color restoration of the fusion image.

[0114] In the infrared and visible light image fusion method provided by the embodiment, a composite loss function is used to optimize the model. The composite loss function includes three parts: MS-SSIM loss function, L1 loss function and visual perception loss function. MS-SSIM is an extended version of SSIM, which is used to evaluate the similarity between two images. It can well preserve the edge and detail information of the image. However, using the MS-SSIM loss function can easily cause deviations in brightness and color. To solve this problem, the embodiment introduces the L1 loss function, which maintains the brightness and color of the image by calculating the absolute value of the difference between the model prediction value and the true value. In addition, in addition to the MS-SSIM loss function and the L1 loss function, the embodiment also introduces the visual perception loss function. The visual perception loss function measures the difference between images in the feature space, rather than directly comparing in the pixel space. The visual perception loss function can help the model to preserve the visual quality and content information of the image, and ensure the perceptual quality and content consistency of the image. The calculation of the visual perception loss function is based on the features extracted by each layer of the pre-trained vgg19 network, and the mean square error function MSE is used to measure the difference between the features. Therefore, the mathematical expression of the composite loss function used in the embodiment is:

[0115]

[0116] wherein, is a composite loss function; is the first part of the composite loss function, combining MS-SSIM loss function and L1 loss function, used to measure the difference between the model predicted image and the actual image; σ is the standard deviation, used to calculate the standard deviation term and the covariance term in the MS-SSIM loss; is the visual perception loss function; α is the balance parameter, used to adjust the weight of MS-SSIM loss function and L1 loss function in the composite loss function; is the MS-SSIM loss function, using the Multi-Scale Structural Similarity Index to measure the similarity between two images; is the Gaussian distribution parameter; is the L1 loss function, used to calculate the absolute value of the difference between the model predicted image and the actual image, and take the average; p is the number of pixels in the feature map, used to calculate the average of the difference of all pixels when calculating the L1 loss function; i is the i-th input image; i is the i-th output image; M is the number of different scales, used for multi-scale analysis in the MS-SSIM loss calculation; I is the mean of the input image; O is the mean of the output image; c1 and c2 are both constant terms in the MS-SSIM loss function; IO is the covariance between the input image and the output image, used to calculate the MS-SSIM loss; I is the standard deviation of the input image; O is the standard deviation of the output image; l is the input image feature function extracted by the pre-trained vgg19 network at the l-th layer; l is the output image feature function extracted by the pre-trained vgg19 network at the l-th layer; m and γ m are both weight parameters in the MS-SSIM loss function, used to adjust the influence of the mean and covariance terms at different scales on the final loss; MSE is the mean square error function.

[0117] The embodiment of the present application provides an infrared and visible light image fusion method, the method obtains an infrared image and a visible light image, and converts the visible light image to a YUV color space by using a color space conversion method to separate original luminance channel data and original chrominance channel data of the visible light image; a feature extraction network is used to perform feature extraction on the original luminance channel data of the infrared image and the visible light image respectively, to obtain different modal image features; wherein the different modal image features include infrared image thermal radiation features and visible light image texture features; a gradient attention mechanism is introduced, and a multi-modal feature fusion network is used to perform gradient-guided feature fusion on the different modal image features, to obtain modal fusion feature information; feature reconstruction is performed according to the modal fusion feature information and the original chrominance channel data of the visible light image, to obtain a color fusion image. Compared with the prior art, the method combines the feature extraction network of the Slide Transformer and the gradient operator module, calculates the gradient weight of the infrared and visible light image modal features, and locally influences the weight of the channel attention and the spatial attention, realizes highlighting of texture details, avoids the texture detail blur problem of the traditional fusion method, and thus realizes accurate extraction of the infrared image and visible light image features.

[0118] It should be noted that the size of the serial number of each process does not mean the order of execution, and the execution order of each process should be determined according to its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application.

[0119] In one embodiment, as shown in FIG. 1, Figure 7 The embodiment of the present application provides an infrared and visible light image fusion system, the system comprises:

[0120] A color space conversion module is configured to obtain an infrared image and a visible light image, and convert the visible light image to a YUV color space by using a color space conversion method to separate original luminance channel data and original chrominance channel data of the visible light image.

[0121] A feature extraction module is configured to use a feature extraction network to perform feature extraction on the original luminance channel data of the infrared image and the visible light image respectively, to obtain different modal image features; wherein the different modal image features include infrared image thermal radiation features and visible light image texture features.

[0122] A feature fusion module is configured to introduce a gradient attention mechanism, and use a multi-modal feature fusion network to perform gradient-guided feature fusion on the different modal image features, to obtain modal fusion feature information.

[0123] The feature reconstruction module is configured to reconstruct features according to the modality fusion feature information and original chrominance channel data of the visible light image to obtain a color fusion image.

[0124] The specific limitations of the infrared and visible light image fusion system can refer to the limitations of the infrared and visible light image fusion method described above, which will not be repeated here. Those skilled in the art can realize that the various modules and steps described in combination with the embodiments disclosed in the present application can be realized in hardware, software or a combination of both. Whether the functions are executed in hardware or software depends on the specific application of the technical solution and the design constraints. The skilled person can use different methods for each specific application to implement the described functions, but such implementation should not be considered beyond the scope of the present application.

[0125] The embodiment of the present application provides an infrared and visible light image fusion system. The system obtains an infrared image and a visible light image through a spatial conversion module, and converts the visible light image to a YUV color space by using a color space conversion method to separate original luminance channel data and original chrominance channel data of the visible light image. A feature extraction module uses a feature extraction network to extract features from the infrared image and the original luminance channel data of the visible light image, respectively, to obtain different modal image features. The different modal image features include infrared image thermal radiation features and visible light image texture features. A feature fusion module introduces a gradient attention mechanism and uses a multi-modal feature fusion network to perform gradient-guided feature fusion on the different modal image features to obtain modality fusion feature information. A feature reconstruction module reconstructs features according to the modality fusion feature information and the original chrominance channel data of the visible light image to obtain a color fusion image. Compared with the prior art, the system combines the feature extraction network of the Slide Transformer and the gradient operator module, calculates the gradient weight of the infrared and visible light image modal features, and locally affects the weight of the channel attention and the spatial attention, highlights the texture details, avoids the texture detail blur problem of the traditional fusion method, and thus realizes accurate extraction of the infrared image and visible light image features.

[0126] Figure 8 The computer device provided by the embodiment of the present application includes a memory, a processor and a transceiver, which are connected through a bus. The memory is used to store a set of computer program instructions and data, and can transmit the stored data to the processor. The processor can execute the program instructions stored in the memory to execute the steps of the above method.

[0127] The memory can include a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memory; the processor can be a central processing unit, a microprocessor, an application-specific integrated circuit, a programmable logic device, or a combination thereof. By way of example and not limitation, the programmable logic device can be a complex programmable logic device, a field programmable logic gate array, a general array logic, or any combination thereof.

[0128] In addition, the memory can be a physically independent unit, or can be integrated with the processor.

[0129] Those skilled in the art can understand that, Figure 8 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have the same component arrangement.

[0130] In one embodiment, the present embodiment provides a computer readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the steps of the above method.

[0131] The infrared and visible light image fusion method, system and device provided by the embodiment of the present application effectively extracts the thermal radiation information of the infrared image and the texture information of the visible light image by combining the feature extraction network of the Slide Transformer and the Gradient Operator, improves the accuracy and efficiency of feature extraction, and at the same time, in the fusion process, the gradient weight of the infrared and visible light image modal features is calculated, and the weight of the local influence channel attention and spatial attention is calculated, the texture details are highlighted, the texture detail blur problem of the traditional fusion method is avoided, and the accurate extraction of the infrared image and visible light image features is realized.

[0132] In the above embodiments, all or part of the methods can be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the methods can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transferred from one website, computer, server or data center to another website, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line) or wireless (such as infrared, wireless, microwave, etc.) manner. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. integrated with one or more available media. The available media can be magnetic media (for example, floppy disk, hard disk, magnetic tape), optical media (for example, DVD), or semiconductor media (for example, SSD) and the like.

[0133] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by instructing related hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments.

[0134] The above-mentioned embodiments only express several preferred embodiments of the present application, and the description is more specific and detailed, but it cannot be understood as a limitation on the scope of the patent. It should be noted that for ordinary skilled in the art, without departing from the technical principles of the present application, a number of improvements and replacements can be made, and these improvements and replacements should be regarded as the protection scope of the present application. Therefore, the protection scope of the patent of the present application should be subject to the protection scope of the claims.

Claims

1. A method for fusing infrared and visible light images, characterized in that, Includes the following steps: Infrared and visible light images are acquired, and the visible light image is converted to the YUV color space using a color space conversion method to separate the original luminance channel data and the original chrominance channel data of the visible light image. Feature extraction networks are used to extract features from the raw brightness channel data of the infrared image and the visible light image respectively, to obtain image features of different modalities; wherein, the image features of different modalities include thermal radiation features of the infrared image and texture features of the visible light image; A gradient attention mechanism is introduced, and a multimodal feature fusion network is used to perform gradient-guided feature fusion of image features from different modalities to obtain modal fusion feature information; Based on the modal fusion feature information and the original chroma channel data of the visible light image, feature reconstruction is performed to obtain a color fusion image; The feature extraction network includes a cascaded first convolutional layer and at least three spatiotemporal graph convolutional modules; the output of the first convolutional layer is connected to the input of each spatiotemporal graph convolutional module using a global residual connection mechanism, and forms a residual connection directly with the output of the last spatiotemporal graph convolutional module. The output of each spatiotemporal graph convolutional module skips the next spatiotemporal graph convolutional module connected to it and is connected to the input of the third spatiotemporal graph convolutional module through a local residual connection method; The spatiotemporal graph convolution module includes a global-local visual attention model and a texture detail gradient enhancement model connected in parallel, and the output of the global-local visual attention model and the output of the texture detail gradient enhancement model form a residual connection; wherein, the global-local visual attention model includes a series of Slide-Transformer modules and a second convolutional layer; the texture detail gradient enhancement model includes a series of gradient operator modules and a third convolutional layer.

2. The infrared and visible light image fusion method as described in claim 1, characterized in that, The step of using a feature extraction network to extract features from the original brightness channel data of the infrared image and the visible light image respectively to obtain image features of different modalities includes: The first convolutional layer performs feature projection on the original brightness channel data of the infrared image and the visible light image respectively to obtain the corresponding infrared image projection features and visible light brightness channel projection features. The infrared image projection features and the visible light brightness channel projection features are connected to the input of each spatiotemporal graph convolution module through a global residual connection. Multiple serial spatiotemporal graph convolution modules are used to learn and balance hierarchical global-local feature representations at different scales in sequence. Gradient operators are introduced to enhance the capture of texture detail features, resulting in a high-dimensional feature representation of the modality. Based on the projection features of the infrared image and the projection features of the visible light brightness channel, the high-dimensional feature representation of the modality is enhanced using a global residual connection mechanism, and the corresponding infrared image thermal radiation features and visible light image texture features are extracted respectively.

3. The infrared and visible light image fusion method as described in claim 1, characterized in that: The multimodal feature fusion network includes a cascaded modal gradient attention extraction module and a modal feature fusion module. The modal gradient attention extraction module includes multiple modal gradient attention extraction sub-modules. Each modal gradient attention extraction sub-module includes a gradient channel attention enhancement unit and a gradient space attention control unit connected in parallel. The gradient channel attention enhancement unit includes a gradient attention unit and a channel attention unit connected in parallel, and the input and output of the gradient channel attention enhancement unit form a residual connection. The gradient spatial attention control unit includes a gradient attention unit and a spatial attention unit connected in parallel, and the input and output of the gradient spatial attention control unit form a residual connection.

4. The infrared and visible light image fusion method as described in claim 1, characterized in that, The steps of introducing a gradient attention mechanism and using a multimodal feature fusion network to perform gradient-guided feature fusion of image features from different modalities to obtain modal fusion feature information include: Gradient channel attention weights are obtained by performing gradient-guided channel feature enhancement on image features of different modalities through gradient channel attention enhancement units; the gradient channel attention weights include infrared gradient channel attention weights and visible light gradient channel attention weights. Gradient channel attention weights are used to enhance the channel texture of image features of different modalities, resulting in channel enhancement features of different modalities. The channel enhancement features of different modalities include infrared channel enhancement features and visible light channel enhancement features. Gradient spatial attention weights are obtained by performing gradient-guided spatial feature enhancement on image features of different modalities through a gradient spatial attention modulation unit. The gradient spatial attention weights include infrared gradient spatial attention weights and visible light gradient spatial attention weights. Spatial texture enhancement is performed on image features of different modalities using gradient spatial attention weights, resulting in spatial enhancement features of different modalities; the spatial enhancement features of different modalities include infrared spatial enhancement features and visible light spatial enhancement features; The modal feature fusion module merges the channel enhancement features of different modalities to obtain channel attention-weighted features, and merges the spatial enhancement features of different modalities to obtain spatial attention-weighted features; The channel attention weighted features and the spatial attention weighted features are merged to obtain modality fusion feature information.

5. The infrared and visible light image fusion method as described in claim 4, characterized in that, The step of performing gradient-guided channel feature enhancement on image features of different modalities through the gradient channel attention enhancement unit to obtain gradient channel attention weights includes: The gradient attention unit and channel attention unit in the gradient channel attention enhancement unit are used to calculate the gradient attention and channel attention of different modal image features on each channel, respectively, to obtain the channel gradient weights and channel feature weights; Based on the channel gradient weights of different modal image features in each channel, the channel feature weights are guided by texture detail gradients to obtain gradient channel attention weights.

6. The infrared and visible light image fusion method as described in claim 4, characterized in that, The step of performing gradient-guided spatial feature enhancement on image features of different modalities through a gradient spatial attention control unit to obtain gradient spatial attention weights includes: The gradient attention unit and spatial attention unit in the gradient spatial attention control unit are used to calculate the gradient attention and spatial attention of different modal image features at each spatial location, respectively, to obtain spatial gradient weights and spatial feature weights; Based on the spatial gradient weights of different modal image features at each spatial location, texture detail gradient guidance is applied to the spatial feature weights to obtain gradient channel attention weights.

7. An infrared and visible light image fusion system, characterized in that, The system includes: The space conversion module is used to acquire infrared and visible light images, and to convert the visible light image to the YUV color space using a color space conversion method, thereby separating the original luminance channel data and the original chrominance channel data of the visible light image. The feature extraction module is used to extract features from the original brightness channel data of the infrared image and the visible light image using a feature extraction network to obtain different modal image features; wherein, the different modal image features include the thermal radiation features of the infrared image and the texture features of the visible light image; The feature fusion module is used to introduce a gradient attention mechanism and use a multimodal feature fusion network to perform gradient-guided feature fusion of image features from different modalities to obtain modal fusion feature information. The feature reconstruction module is used to perform feature reconstruction based on the modal fusion feature information and the original chroma channel data of the visible light image to obtain a color fusion image; The feature extraction network includes a cascaded first convolutional layer and at least three spatiotemporal graph convolutional modules; the output of the first convolutional layer is connected to the input of each spatiotemporal graph convolutional module using a global residual connection mechanism, and forms a residual connection directly with the output of the last spatiotemporal graph convolutional module. The output of each spatiotemporal graph convolutional module skips the next spatiotemporal graph convolutional module connected to it and is connected to the input of the third spatiotemporal graph convolutional module through a local residual connection method; The spatiotemporal graph convolution module includes a global-local visual attention model and a texture detail gradient enhancement model connected in parallel, and the output of the global-local visual attention model and the output of the texture detail gradient enhancement model form a residual connection; wherein, the global-local visual attention model includes a series of Slide-Transformer modules and a second convolutional layer; the texture detail gradient enhancement model includes a series of gradient operator modules and a third convolutional layer.

8. A computer device, characterized in that: The device includes a processor and a memory, the processor being connected to the memory for storing computer programs, and the processor for executing the computer programs stored in the memory to cause the computer device to perform the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method as described in any one of claims 1 to 6.

Images