Infrared image and visible light image fusion method, system and device and storage medium

Abstract

The invention discloses an infrared image and visible light image fusion method, system and device and a storage medium, belongs to the technical field of image information processing and computer world, and solves the problem that an image fusion method based on deep learning has some limitations. Outputting the bimodal mixed features of the first scale; outputting a bimodal mixed feature of a second scale; outputting a bimodal mixed feature of a third scale; inputting the bimodal mixed features of the third scale into an image reconstruction module, and outputting a reconstructed feature map; carrying out feature fusion on the reconstructed feature map and the bimodal mixed features of the second scale, and outputting the reconstructed feature map; carrying out feature fusion on the reconstructed feature map and the bimodal mixed features of the first scale, and outputting the reconstructed feature map; carrying out feature fusion on the reconstructed feature map and the bimodal mixed features of the initial scale, and outputting the reconstructed feature map; and after the reconstructed feature map, outputting a fused image.

Description

Technical Field

[0001] The present invention relates to image information processing technology and computer world technology field, and in particular to an infrared image and visible light image fusion method, system, device and storage medium. Background Art

[0002] With the continuous development of science and technology, image processing technology has been widely used in various fields. However, due to the limitations of its hardware design and imaging method, traditional single-type image sensors cannot fully capture all the information of the scene. In order to solve this problem, image fusion technology came into being. By combining multiple imaging modes, image fusion technology can extract complementary information from the source image and generate a composite image containing rich information.

[0003] Among them, the fusion of infrared images and visible light images is the most common application of image fusion technology. Infrared sensors can capture the thermal radiation information of the target and generate infrared images. This imaging method is not easily affected by weather and lighting conditions. However, infrared images have some shortcomings in terms of clarity, edge structure and noise. Relatively speaking, visible light images are imaged by reflecting visible light, which conforms to human visual perception and can provide higher resolution and rich detail textures. However, the quality of visible light images is greatly affected by weather conditions, especially in insufficient light or foggy weather, and their recognition is often low.

[0004] The fusion of infrared and visible light images can absorb the complementary features of the two modal images, overcome the limitations of a single sensor, provide richer spatial details and information, and assist in subsequent image processing tasks. In recent years, the infrared and visible light image fusion algorithm has become a key technology in the field of computer vision and is widely used in downstream tasks such as image segmentation, target detection, and target tracking.

[0005] At present, infrared and visible light image fusion algorithms can be mainly divided into two categories: traditional image fusion methods and image fusion algorithms based on deep learning. Traditional image fusion methods include multi-scale transform, wavelet transform, sparse representation, subspace analysis and saliency transform. These methods mainly rely on manually designed decomposition rules, which can only capture limited features in the source image and are difficult to fully display the rich information of the input image. For example, the multi-scale decomposition method mainly focuses on local high-frequency features and often finds it difficult to capture large-scale low-frequency information, resulting in insufficient overall coherence of the fused image. In addition, wavelet transform may cause certain data loss during the preprocessing process, making it impossible to fully utilize global information.

[0006] Image fusion methods based on deep learning have developed rapidly in recent years, mainly divided into methods based on CNN (convolutional neural network), AE (autoencoder), GAN (generative adversarial network) and Transformer (deep learning model). These methods can perform well in feature extraction and modeling by taking advantage of the deep learning framework. Although these frameworks can generate satisfactory fusion results, they still have some limitations. For example, CNN is difficult to capture information interactions between non-adjacent pixels due to the limitation of receptive field; and although GAN performs well in improving the quality of fused images, it often focuses on retaining the texture details of visible light images, which may lead to the loss of temperature significance information of infrared images.

[0007] In summary, although the deep learning-based image fusion method can generate satisfactory fusion results, there are still some limitations. Summary of the invention

[0008] The present invention solves the problem that the image fusion method based on deep learning has some limitations.

[0009] The infrared image and visible light image fusion method described in the present invention is specifically as follows: The infrared image and the visible light image are respectively input into the convolution embedding module for feature extraction to obtain the infrared stitching features. and visible light stitching features , among which, one infrared splicing feature and visible light stitching features The two-modal interaction residual fusion module is input together for feature fusion, and the initial scale bimodal mixed features are output. ; Another infrared stitching feature and visible light stitching features The multi-cascade hybrid global and local feature extraction modules are input into the feature extraction module, and the infrared stitching features of the first scale are output respectively. , infrared stitching features of the second scale , infrared stitching features at the third scale , the first scale visible light stitching features , visible light stitching features at the second scale and the third scale visible light stitching features ; Infrared stitching features at the first scale Visible light stitching features with the first scale The two models are jointly input into the bimodal interactive residual fusion module for feature fusion, and the bimodal mixed features of the first scale are output. ; Second scale infrared stitching features Visible light stitching features with the second scale The two models are jointly input into the bimodal interactive residual fusion module for feature fusion, and the bimodal mixed features of the second scale are output. ; Infrared stitching features at the third scale Visible light stitching features with the third scale The two models are jointly input into the bimodal interactive residual fusion module for feature fusion, and the bimodal mixed features of the third scale are output. ; Bimodal hybrid features at the third scale Input image reconstruction module, output reconstructed feature map ; Reconstruct feature map Bimodal hybrid features with the second scale The two modules are jointly input into the dual-modal interactive residual fusion module for feature fusion, and then input into the image reconstruction module to output the reconstructed feature map. ; Reconstruct feature map Bimodal hybrid features with the first scale The two modules are jointly input into the dual-modal interactive residual fusion module for feature fusion, and then input into the image reconstruction module to output the reconstructed feature map. ; Reconstruct feature map Bimodal hybrid features with initial scale The two modules are jointly input into the dual-modal interactive residual fusion module for feature fusion, and then input into the image reconstruction module to output the reconstructed feature map. ; Reconstructing feature maps After upsampling, multiple convolutional layers, activation functions, and splicing operations, the fused image is output.

[0010] Furthermore, in one embodiment of the present invention, the convolution embedding module is specifically: The image is divided into two paths. One path is sequentially input into the convolution layer, normalized, activated, and flattened. The channel dimension is swapped with the flattened spatial dimension to output a local feature map. Another image is input into Patch Embedding, and after the channel dimension is exchanged with the flattened spatial dimension, the position encoding is input and the global feature map is output; The local feature map and the global feature map are added element by element to output the feature map.

[0011] Furthermore, in one embodiment of the present invention, the hybrid global-local feature extraction module is specifically: The image is divided into two paths. One path is reshaped in turn to output a feature map. After feature extraction, the feature map is input into a multi-cascade local feature extraction module for feature extraction, and then reshaped again to output a local feature map. The other image is reconstructed and input into the one-dimensional convolution layer, and then reshaped again to output the feature map. The feature map is input into the multi-head attention and the global feature map is output. The local feature map and the global feature map are added element by element, and a mixed global-local feature map is output.

[0012] Furthermore, in one embodiment of the present invention, the feature map is subjected to feature extraction, specifically: The feature map is divided into two paths, one feature map is input into the residual block, and the other feature map is input into the two-dimensional convolution layer. The feature map passing through the residual block and the other feature map passing through the two-dimensional convolution layer are added element by element.

[0013] Further, in one embodiment of the present invention, the local feature extraction module performs feature extraction, specifically: The image is sequentially input into multi-cascade convolution and multi-cascade bottleneck layers for feature extraction.

[0014] Furthermore, in one embodiment of the present invention, the dual-modal interactive residual fusion module is specifically: The images input into the dual-modal interactive residual fusion module include at least two types, namely a first image and a second image; After global average pooling is performed on the first image and the second image respectively, the first feature map and the second feature map are output respectively, attention is allocated to the first feature map and the second feature map respectively, and the first weight and the second weight are output respectively. The first weight is element-wise multiplied by the first feature map, and the first feature map after element-wise multiplication is output. The second weight is element-wise multiplied by the second feature map, and the second feature map after element-wise multiplication is output. The first feature map after element-wise multiplication and the second feature map after element-wise multiplication are added element-by-element, and then input into a multi-stage convolutional layer, and a feature map is output. The feature map is added element-by-element with the first image and the second image, and a feature fused image is output.

[0015] Furthermore, in one embodiment of the present invention, the image reconstruction module is specifically: After the image is input to the convolutional layer for feature extraction, a feature map is output. The feature map undergoes layer normalization and multi-head attention operations in sequence, and the feature map after the multi-head attention operation is output. The feature map after the multi-head attention operation is added element by element with the feature map. Then, the layer normalization and multi-layer perception mechanism operations are performed in sequence, and the feature map after the multi-head perception mechanism operation is output. The feature map after the multi-head perception mechanism operation is added element by element with the feature map after the head attention operation.

[0016] The infrared image and visible light image fusion system described in the present invention is specifically: The infrared image and the visible light image are respectively input into the convolution embedding module for feature extraction to obtain the infrared stitching features. and visible light stitching features , among which, one infrared splicing feature and visible light stitching features The two-modal interaction residual fusion module is input together for feature fusion, and the initial scale bimodal mixed features are output. ; Another infrared stitching feature and visible light stitching features The multi-cascade hybrid global and local feature extraction modules are input into the feature extraction module, and the infrared stitching features of the first scale are output respectively. , infrared stitching features of the second scale , infrared stitching features at the third scale , the first scale visible light stitching features , visible light stitching features at the second scale and the third scale visible light stitching features ; Infrared stitching features at the first scale Visible light stitching features with the first scale The two models are jointly input into the bimodal interactive residual fusion module for feature fusion, and the bimodal mixed features of the first scale are output. ; Second scale infrared stitching features Visible light stitching features with the second scale The two models are jointly input into the bimodal interactive residual fusion module for feature fusion, and the bimodal mixed features of the second scale are output. ; Infrared stitching features at the third scale Visible light stitching features with the third scale The two models are jointly input into the bimodal interactive residual fusion module for feature fusion, and the bimodal mixed features of the third scale are output. ; Bimodal hybrid features at the third scale Input image reconstruction module, output reconstructed feature map ; Reconstruct feature map Bimodal hybrid features with the second scale The two modules are jointly input into the dual-modal interactive residual fusion module for feature fusion, and then input into the image reconstruction module to output the reconstructed feature map. ; Reconstruct feature map Bimodal hybrid features with the first scale The two modules are jointly input into the dual-modal interactive residual fusion module for feature fusion, and then input into the image reconstruction module to output the reconstructed feature map. ; Reconstruct feature map Bimodal hybrid features with initial scale The two modules are jointly input into the dual-modal interactive residual fusion module for feature fusion, and then input into the image reconstruction module to output the reconstructed feature map. ; Reconstructing feature maps After upsampling, multiple convolutional layers, activation functions, and splicing operations, the fused image is output.

[0017] An electronic device described in the present invention comprises a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; Memory, used to store computer programs; The processor is used to implement the method steps described in the above method when executing the program stored in the memory.

[0018] The present invention provides a computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the method steps described in the above method are implemented.

[0019] The present invention solves the problem that the image fusion method based on deep learning has some limitations. Specific beneficial effects include: 1. The infrared image and visible light image fusion method described in the present invention has some limitations in the existing deep learning image fusion method, and aims to solve the problems of edge information loss, artifact introduction, and global semantic information loss in the current infrared and visible light fusion process. In order to solve the above technical problems, the present invention designs a hybrid local-global feature multi-scale dual-stream encoder network for infrared and visible light image fusion, fully considering the specificity and common characteristics of infrared and visible light modalities, and designs a feature interaction module to realize the extraction and fusion of multi-scale features; 2. The infrared image and visible light image fusion method of the present invention is based on the network architecture of the multi-scale dual-stream encoder, and adopts a method of fusing mixed local detail information, global semantic bimodal information and multi-scale feature information, thereby effectively retaining the different scale feature information extracted by the multi-scale dual-stream encoder; The infrared image and visible light image fusion method described in the present invention improves the overall scene perception capability by fusing the image information of two different modalities, and is widely used in target detection, monitoring, autonomous driving and other fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which: Figure 1This is a diagram of the method for fusing infrared images and visible light images described in Implementation Method 1; Figure 2 is a structural diagram of a convolutional embedding module described in Implementation Method 2; Figure 3 is a structural diagram of a hybrid global-local feature extraction module described in Implementation Method 3; Figure 4 is a structural diagram of a local feature extraction module described in Implementation Method 3; Figure 5 4 is a structural diagram of a dual-modal interactive residual fusion module described in Embodiment 4; Figure 6 is a structural diagram of the image reconstruction module described in Embodiment 5; Figure 7 is a quantitative comparison diagram of 21 pairs of images of the TNO test set described in Embodiment 5; Figure 8 This is a visual quality comparison chart of two typical images in the TNO test set described in Implementation Example 5. DETAILED DESCRIPTION

[0021] The following will clearly and completely describe various embodiments of the present invention in conjunction with the accompanying drawings. The embodiments described with reference to the accompanying drawings are exemplary and intended to be used to explain the present invention, but should not be understood as limiting the present invention.

[0022] Embodiment 1: A method for fusing an infrared image and a visible light image described in this embodiment is characterized by: The infrared image and the visible light image are respectively input into the convolution embedding module for feature extraction to obtain the infrared stitching features. and visible light stitching features , among which, one infrared splicing feature and visible light stitching features The two-modal interaction residual fusion module is input together for feature fusion, and the initial scale bimodal mixed features are output. ; Another infrared stitching feature and visible light stitching features The multi-cascade hybrid global and local feature extraction modules are input into the feature extraction module, and the infrared stitching features of the first scale are output respectively. , infrared stitching features of the second scale , infrared stitching features at the third scale , the first scale visible light stitching features , visible light stitching features at the second scale and the third scale visible light stitching features ; Infrared stitching features at the first scale Visible light stitching features with the first scale The two models are jointly input into the bimodal interactive residual fusion module for feature fusion, and the bimodal mixed features of the first scale are output. ; Second scale infrared stitching features Visible light stitching features with the second scale The two models are jointly input into the bimodal interactive residual fusion module for feature fusion, and the bimodal mixed features of the second scale are output. ; Infrared stitching features at the third scale Visible light stitching features with the third scale The two models are jointly input into the bimodal interactive residual fusion module for feature fusion, and the bimodal mixed features of the third scale are output. ; Bimodal hybrid features at the third scale Input image reconstruction module, output reconstructed feature map ; Reconstruct feature map Bimodal hybrid features with the second scale The two modules are jointly input into the dual-modal interactive residual fusion module for feature fusion, and then input into the image reconstruction module to output the reconstructed feature map. ; Reconstruct feature map Bimodal hybrid features with the first scale The two modules are jointly input into the dual-modal interactive residual fusion module for feature fusion, and then input into the image reconstruction module to output the reconstructed feature map. ; Reconstruct feature map Bimodal hybrid features with initial scale The two modules are jointly input into the dual-modal interactive residual fusion module for feature fusion, and then input into the image reconstruction module to output the reconstructed feature map. ; Reconstructing feature maps After upsampling, multiple convolutional layers, activation functions, and splicing operations, the fused image is output.

[0023] In the existing technology, although deep learning-based fusion methods, such as CNN and GAN, have made certain progress in feature extraction and fusion accuracy, they still have problems such as edge information loss and artifact introduction in the fusion process, and cannot fully integrate the complementary features of infrared and visible light.

[0024] In order to solve the above technical problems, Figure 1As shown, this embodiment designs a method for fusing infrared images and visible light images. The image to be fused includes a visible light image and its corresponding infrared image. The RGB (red-green-blue) color space of the image to be fused is converted to the YCrCb (brightness-blue difference-red difference) color space for preprocessing. The preprocessed image to be fused is input into a multi-scale dual-stream encoder for modeling and processing to obtain infrared-visible light fusion features. The infrared-visible light fusion features are then input into a multi-scale feature decoder for image reconstruction to obtain a fused image. The multi-scale dual-stream encoder includes a convolutional embedding module, a mixed global-local feature module, and a dual-modal interactive residual fusion module connected in sequence. The multi-scale feature decoder includes an upsampling and image reconstruction module.

[0025] The infrared image and visible light image fusion method described in this embodiment is specifically as follows: First, both the infrared image and the visible light image are converted to color space, and only the Y (brightness) channel is retained for processing. This not only allows us to focus on the details of the infrared and visible light images, such as edges, contrast, and contours, but also reduces the interference of color noise and avoids processing complex color information at the same time, thereby reducing the amount of calculation and memory requirements. Subsequently, the infrared image and the visible light image are randomly cropped to generate The image patches are processed by applying image enhancement techniques. Finally, the output infrared image and visible light image are both Next, the infrared image and visible light image are input into the convolutional embedding module in the multi-scale dual-stream encoder for feature extraction.

[0026] The infrared splicing features are extracted through the convolution embedding module. and visible light stitching features , among which, one infrared splicing feature and visible light stitching features The mixed global and local features are input into three cascades to further extract the mixed features, and the infrared stitching features of the first scale are output respectively. , infrared stitching features of the second scale , infrared stitching features at the third scale , the first scale visible light stitching features , visible light stitching features at the second scale and the third scale visible light stitching features , and the corresponding scale sizes are , and Another infrared stitching feature and visible light stitching features All are input into the bimodal interactive residual fusion module for feature fusion to obtain the bimodal mixed features of the initial scale. , the scale size is .

[0027] The three scales are , , The first scale infrared stitching feature , infrared stitching features of the second scale , infrared stitching features at the third scale and the three scale sizes are , , The first scale visible light stitching feature , visible light stitching features at the second scale , the third scale visible light stitching features They are input into the dual-modal interactive residual fusion module respectively, and the scales are The first scale of bimodal mixed features , scale is The second scale of bimodal mixed features and the scale is The third scale of bimodal mixed features .

[0028] The scales are The first scale of bimodal mixed features , scale is The second scale of bimodal mixed features and the scale is The third scale of bimodal mixed features Input into the multi-scale feature decoder to reconstruct the image. Specifically: the scale is The third scale of bimodal mixed features Input image reconstruction module, output reconstructed feature map , whose size is . The feature map will be reconstructed With the scale The second scale of bimodal mixed features They are input into the dual-modal interactive residual fusion module for further feature fusion, and then input into the image reconstruction module to output the reconstructed feature map. , whose size is . Then reconstruct the feature map of the same scale With the scale The first scale of bimodal mixed features The two images are input into the dual-modal interactive residual fusion module for further feature fusion, and then input into the image reconstruction module to obtain the reconstructed feature map. , whose size is Then we will reconstruct the feature map Bimodal hybrid features with initial scale The image is input into the dual-modal interactive fusion module for further feature fusion, and then input into the image reconstruction module to output the final reconstructed feature map. , the size of which is Finally, the fusion reconstructed feature map Convert to a two-dimensional format to adapt it to the subsequent two-dimensional convolution layer operation. Exchange the channel and sequence dimensions, and reshape the reconstructed feature map Reshape , then reconstruct the feature map by upsampling The spatial size of , that is, the height and width are doubled, and the feature map is reconstructed at this time The dimension becomes , then enter into a In the convolution layer of the convolution kernel, the feature dimension becomes Finally, it is input into the second convolution layer, and the convolution kernel size is , reconstruct the feature map The dimension becomes Finally, it is activated by Sigmod (activation function), and the output scale is Finally, all the graphic blocks are spliced ​​together to obtain the original resolution fused image.

[0029] In order to make the fused image better, a loss function is needed to constrain the fused image to contain more source image information. Therefore, the loss function is designed in this embodiment as follows: the loss function is composed of pixel loss, image gradient loss and total variation loss. The formula is: ; in, represents the loss function, , and They represent pixel loss, gradient loss and total variational loss respectively. , and Respectively represent weight factors. This embodiment compares the weight factors through multiple experiments. , and Set to 10, 4, and 1 respectively.

[0030] Pixel loss calculates the absolute error between each pixel in the fused image and the pixel in the target image and averages them. The main purpose is to ensure that the pixel value of the fused image reproduces the higher quality part of the target image as much as possible. To maintain the edge and clarity of the fused image, the edge structure of the image is enhanced or maintained by constraining the gradient information of the fused image. The loss function is: ); in, represents the mean square error calculation of each pixel, represents the fused image, represents an infrared image, represents a visible light image, Represents the loss function.

[0031] Gradient loss uses the gradient information extracted from the fused image by the Sobel operator (edge ​​detection algorithm), that is, the edge contour strength. This makes the fused image contain rich texture details. When the image is fused, reconstructed, or enhanced, directly processing the pixel value will ignore the detail information, while processing the edge information can maintain or enhance the structural details of the image and ensure the clarity and contour of the generated image. The formula for gradient loss is as follows: ; in, Represents the fused image The gradient of and Represent the gradient information of infrared image and visible light image respectively, represents the gradient loss. The goal is to take the larger value of their gradients.

[0032] The total variation loss is used to evaluate the smoothness of the image. It encourages the model to generate images with smooth transitions in space, rather than sudden noise between adjacent pixels. The total variation loss helps to remove artifacts or noise caused by over-sharpening. Therefore, it is suitable for use in image enhancement and resolution improvement tasks to prevent over-sharpening. By limiting the difference between adjacent pixels, it ensures that the image does not have too much fine-grained noise and the output image is more natural. The total variation loss function is shown in the formula: ; in, Indicates that the output image is at position The pixel value of Represents the difference between adjacent points in the vertical direction of the pixel. Represents the difference between adjacent points in the horizontal direction, represents the total variational loss function.

[0033] In summary, this implementation is based on a multi-scale dual-stream encoder and combines the fusion strategy of global and local features to improve fusion accuracy while reducing computational complexity. The low-frequency shared information and high-frequency depth information of infrared images and visible light images are extracted through a shared encoder, and then these features are decomposed into multiple scales through the designed fusion strategy to achieve more effective feature fusion and reconstruction.

[0034] Embodiment 2: This embodiment further limits the infrared image and visible light image fusion method described in Embodiment 1. The convolution embedding module is specifically: The image is divided into two paths. One path is sequentially input into the convolution layer, normalized, activated, and flattened. The channel dimension is swapped with the flattened spatial dimension to output a local feature map. Another image is input into Patch Embedding, and after the channel dimension is exchanged with the flattened spatial dimension, the position encoding is input and the global feature map is output; The local feature map and the global feature map are added element by element to output the feature map.

[0035] The convolution embedding module described in this embodiment is as follows: Figure 2 As shown, specifically: The image is divided into two paths, one of which uses The two-dimensional convolutional layer extracts the local detail feature vector, and the output feature map shape and size is , normalize each channel and apply SiLu (activation function), and the output shape remains unchanged. The extracted feature map is flattened to flatten the height and width into a one-dimensional vector, so that the shape of the feature map changes from becomes Next, swap the channel dimensions With the flattened spatial dimensions , adjust the shape to , output the final local feature tensor, size is , where each element corresponds to a local detail feature. At the same time, the other image is embedded with features through PatchEmbedding to achieve global feature extraction. To prevent the loss of spatial position information, position encoding is also added to each image block, so that the model can capture global context information. Specifically, the image is divided into Each patch is embedded into a Then, through The two-dimensional convolution of the output feature map is Then, the feature map is flattened and dimensionally swapped to obtain The feature map is normalized (shape unchanged) and then embedded in the position code. Through the Dropout (regularization technique) operation, the shape of the global feature map embedded in the position code is Finally, the extracted local feature map and the global feature map are added element by element to obtain the concatenated feature shape size: .

[0036] Implementation method 3: This implementation method further limits the infrared image and visible light image fusion method described in implementation method 1. The hybrid global and local feature extraction module is specifically: The image is divided into two paths. One path is reshaped in turn to output a feature map. After feature extraction, the feature map is input into a multi-cascade local feature extraction module for feature extraction, and then reshaped again to output a local feature map. The other image is reconstructed and input into the one-dimensional convolution layer, and then reshaped again to output the feature map. The feature map is input into the multi-head attention and the global feature map is output. The local feature map and the global feature map are added element by element, and a mixed global-local feature map is output.

[0037] In this implementation manner, the feature map is used for feature extraction, specifically: The feature map is divided into two paths, one feature map is input into the residual block, and the other feature map is input into the two-dimensional convolution layer. The feature map passing through the residual block and the other feature map passing through the two-dimensional convolution layer are added element by element.

[0038] In this implementation manner, the local feature extraction module performs feature extraction, specifically: The image is sequentially input into multi-cascade convolution and multi-cascade bottleneck layers for feature extraction.

[0039] The hybrid global-local feature extraction module described in this embodiment is as follows: Figure 3 As shown, specifically: The hybrid global-local feature extraction module mainly includes two feature extraction paths, the local feature extraction path and the global feature extraction path. It aims to extract the hybrid local and global features of infrared images and visible light images. Specifically, the size of the input implementation method 1 is Infrared stitching features and visible light stitching features First, in the local feature extraction path, the infrared stitching feature and visible light stitching features Reshape the input tensor to the form , in order to adapt to the input format of the two-dimensional convolution layer. Then, after the residual block and the two-dimensional convolution layer, the residual block avoids the gradient disappearance and ensures that the information is not lost when it is transmitted in the deep network. After the residual block dimensionality reduction processing, we get The two-dimensional convolutional layer operation further extracts local features such as edges and textures in the image and also outputs The feature map of the residual block is added element by element to the feature map of the two-dimensional convolutional layer to obtain The feature map is then input into Figure 4 The two cascaded local feature extraction modules shown in Figure 1 perform more in-depth feature extraction. The local feature extraction module effectively captures complex local features in the image, such as edges and shapes, through multiple convolutional layers and bottleneck structures. By introducing multiple convolutional layers and nonlinear activation functions, more complex feature representations can be learned. The output local feature map is Finally, the local feature map is reshaped again, that is, the shape is adjusted, and the local feature map is output The shape and size are .

[0040] The global feature extraction path is first reshaped, that is, converted to Then, the feature map is downsampled using a one-dimensional convolutional layer to generate a feature map with a shape and size of The one-dimensional convolutional layer operation not only effectively captures local information, but also reduces the dimension of the feature map by downsampling, which facilitates the subsequent global feature extraction. Then, reshape again, that is, adjust the tensor dimension through transposition operation to ensure the correct arrangement of features in attention calculation, and convert it into . Subsequently, multi-head attention is used to extract global information. This module generates Q (query), K (key), and V (value) vectors through linear transformation, enabling the model to learn the correlation between features in multiple subspaces. Since the self-attention module has the ability to capture long-range dependencies, it can enhance the expression of global features. In addition, through the nonlinear transformation of MLP (multi-layer perception model), the model further improves its ability to learn complex patterns. Finally, after shape adjustment, the global feature map Output and local feature map The scale matches , output tensor. The design idea of ​​this architecture is that after the local and global feature extraction is completed, the feature maps of the two channels are fused by element-by-element addition to generate a mixed feature map , the scale size is .

[0041] In the prior art, Chinese patent CN117745560A discloses "a method and system for fusion of infrared and visible light images based on channel attention". The image fusion network model includes a coding layer, a fusion layer and a decoding layer. In the fusion layer, the two feature maps output by the coding layer are each subjected to a convolution, and a preliminary fusion result is obtained by a fusion method of element-by-element addition. Based on the preliminary fusion result, the spatial scale of the channel feature is compressed to obtain the information of each channel, and then a dimensionality reduction and then dimensionality increase operation is implemented through a two-layer fully connected layer network. First, a compact feature vector is obtained to provide guidance for the adaptive selection of the two branches of infrared light and visible light. Then, the feature vectors of the two branches are obtained by dimensionality increase. By reducing the dimensionality and then increasing the dimensionality of the channel features, features with more texture information can be captured for channel selection, thereby strengthening the salient targets of infrared and visible light and achieving a better fusion effect.

[0042] Its feature extraction module uses conventional 3×3 and 1×1 convolutional layers, as well as maximum pooling. This is currently a more common feature extraction framework that belongs to local information extraction. The convolution kernel can capture local features (such as edges and textures in pixel neighborhoods). Although the 3×3 convolutional layer is good at extracting local spatial features, and the 1×1 convolutional layer assists in cross-channel information fusion, this convolution method has certain limitations when dealing with larger feature areas, long-distance dependencies, and more complex contextual information, and there is a problem of a small receptive field of the convolutional layer.

[0043] Although convolutional layers can be stacked layer by layer, the ability of a simple convolutional layer to generalize to the global context of the entire image is limited. The offset of the convolutional layer is inflexible, and it only focuses on local area information, losing the ability to perceive the global structure. It is difficult for the convolutional layer to handle the connection between different areas in the image, especially the dependency between pixels at a longer distance. This makes the model lack sufficient global information when dealing with complex background interference and global relationship judgment. In addition, maximum pooling provides a simple but effective downsampling method that can compress the size of feature maps, but one limitation of maximum pooling is that it only retains the maximum response of the local area and ignores other detailed information. In complex scenes, over-reliance on pooling will lead to the loss of global background information, and it is difficult to capture long-distance dependencies and complex contextual relationships. Therefore, the overall grasp of complex scenes is weak.

[0044] In order to solve the above technical problems, this embodiment designs a hybrid global local feature extraction module. Compared with the 3×3 and 1×1 convolution layers in the prior art, this embodiment uses a local feature extraction module in the local feature extraction path. The input features are transmitted and processed along the path through a branch structure, which can better reuse features. Specifically, the local feature extraction module divides the input feature map into two parts, one part enters the standard convolution network for deep feature extraction, and the other part is used as a jump connection to fuse with the features extracted by the subsequent convolution layer. In this way, some shallower information (low-order features) can be retained during subsequent feature fusion. Alleviate feature redundancy: Compared with conventional convolution stacking without jump connections, the local feature extraction module processes features of different parts through branches, avoiding multiple extractions and redundant calculations of the same information. While reusing features, more fine information can be extracted, and local detail information can be extracted and enhanced from multi-scale levels, which is particularly helpful for scenes with large scale changes or complex backgrounds, especially subtle structures such as edges and corners. The local feature extraction module can fully extract deeper and shallower features while the parameter quantity remains basically unchanged.

[0045] In the global extraction path, this embodiment adopts multi-head attention. It can capture the relationship between all pixels at one time. It realizes the mutual relationship modeling of global features by constructing three matrices, namely Q query, K (key) and V (value). Each pixel can pay attention to other pixels in the whole picture, thereby capturing local and global contextual information at the same time. Compared with the convolution operation in the prior art that can only capture the information of the local neighborhood, the multi-head attention of this embodiment can capture the long-distance dependency between pixels, which is particularly suitable for processing large-scale changes and non-continuous objects, and solves the problem of insufficient global feature extraction capability in the prior art.

[0046] In order to realize the effectiveness of the local feature extraction module in the local feature extraction path and the multi-head attention in the global extraction path, when the two are combined, Figure 3 As shown, the relationship between each module needs to be redesigned so that the local feature extraction module in the local feature extraction path and the multi-head attention in the global extraction path can fully play their respective roles.

[0047] Embodiment 4: This embodiment further limits the infrared image and visible light image fusion method described in embodiment 1. The dual-modal interactive residual fusion module is specifically: The images input into the dual-modal interactive residual fusion module include at least two types, namely a first image and a second image; After global average pooling is performed on the first image and the second image respectively, the first feature map and the second feature map are output respectively, attention is allocated to the first feature map and the second feature map respectively, and the first weight and the second weight are output respectively. The first weight is element-wise multiplied by the first feature map, and the first feature map after element-wise multiplication is output. The second weight is element-wise multiplied by the second feature map, and the second feature map after element-wise multiplication is output. The first feature map after element-wise multiplication and the second feature map after element-wise multiplication are added element-by-element, and then input into a multi-stage convolutional layer, and a feature map is output. The feature map is added element-by-element with the first image and the second image, and a feature fused image is output.

[0048] The dual-modal interactive residual fusion module described in this embodiment is as follows: Figure 5 As shown, specifically: For each input feature map, the dimension size is first reshaped and transposed, that is, the infrared stitching feature of the first scale described in the first embodiment is , infrared stitching features of the second scale , infrared stitching features at the third scale , the first scale visible light stitching features , visible light stitching features at the second scale and the third scale visible light stitching features , and the corresponding scale sizes are , and , transposed to the scale sizes are: , and Then, the infrared characteristic images and visible light characteristics Perform global average pooling. The global average pooling operation can extract global features from each channel, reduce feature dimensions, and help the model aggregate the overall semantic information of the image, such as the overall shape and position of objects. The statistical information of each channel facilitates the subsequent fully connected layer to learn the global representation of each modality. Next, through the fully connected layer, activation function, and fully connected layer, special weight information can be extracted for the features of each modality. This process is equivalent to learning the complementary relationship between infrared and visible light modalities, so that each modality can contribute different weights according to different information in the scene. Then use the activation function to limit the weight to interval, which helps to allocate attention to infrared and visible light features. The weights obtained through this process and weight They are: ; ; in, Represents the Sigmoid function (activation function), FC represents the fully connected layer, and AvgPooling represents the global average pooling.

[0049] Finally, the weight and weight By performing element-by-element multiplication with the preliminary fusion feature map, the importance of each modality in a specific scenario can be dynamically adjusted, thus ensuring that the fused features can retain the advantages of infrared and visible light and eliminate redundant information. Finally, the residual connection structure is adopted. By directly adding the preliminary fused feature map to the weighted infrared and visible light features, this operation can effectively retain the key information in the original feature map while avoiding the loss of semantic information caused by over-fusion. At the same time, the residual connection can also help the model maintain stable gradient propagation during semantic fusion, making it easier to optimize the deep network. The formula is as follows: ; in, represents element-wise multiplication, represents element-by-element addition, Represents the fused feature group.

[0050] Finally, the fused feature group is input into two cascades The first convolution layer is used to reduce the number of channels of the feature, reducing the number of original channels by half. The second convolution layer is used to restore the original number of channels. Finally, the output feature map is flattened and transposed to make it consistent with the input shape. Feature map and infrared feature map , Visible light characteristic diagram Add element by element and output The first scale of bimodal mixed features , scale is The second scale of bimodal mixed features and the scale is The third scale of bimodal mixed features .

[0051] Therefore, the dual-modal interaction residual fusion module extracts global semantic information through global average pooling, learns adaptive fusion of different modal weights, retains original information through residual connection, and refines semantic information through element-by-element operation, and finally achieves deep fusion of infrared feature maps and visible light feature maps. It can establish effective semantic interaction between different modalities and make full use of the complementarity of multimodal information, thereby enhancing the model's ability to understand and process complex scenes.

[0052] Embodiment 5: This embodiment further limits the infrared image and visible light image fusion method described in Embodiment 1. The image reconstruction module is specifically: After the image is input to the convolutional layer for feature extraction, a feature map is output. The feature map undergoes layer normalization and multi-head attention operations in sequence, and the feature map after the multi-head attention operation is output. The feature map after the multi-head attention operation is added element by element with the feature map. Then, the layer normalization and multi-layer perception mechanism operations are performed in sequence, and the feature map after the multi-head perception mechanism operation is output. The feature map after the multi-head perception mechanism operation is added element by element with the feature map after the head attention operation.

[0053] The image reconstruction module described in this embodiment is as follows: Figure 6 As shown, specifically: First, the minimum scale is The third scale of bimodal mixed features The number of channels is halved, mainly through transposition operations and one-dimensional convolutional layers, and the size of the feature map becomes , transpose again, and the feature map is restored to Next, we perform layer normalization without changing the shape of the feature map. Then, we send the layer-normalized feature map to the multi-head attention, which will perform calculations on the feature map to obtain a new feature map, the size of which is still the same. After the feature map and feature map of the multi-head attention operation are added element by element, the output remains , after layer normalization and multi-layer perception mechanism operations in turn, the feature map operated by the multi-layer perception mechanism is output, and the feature map operated by the multi-layer perception mechanism is added element by element with the feature map operated by the multi-head attention mechanism.

[0054] In order to better illustrate the infrared image and visible light image fusion method described in this embodiment, the following embodiments are described in detail: Experimental data set: The experimental model of this embodiment adopts the Pytorch (open source machine learning library) framework, in which the torch (a function or method used to perform certain operations) version is 2.0.1+cu12, and the training data uses the MSRS dataset (a dataset for speech recognition research), with a total of 1083 sets of aligned infrared images and visible light images. During training, the RGB color space of the visible light image is converted into a grayscale image, the training number is 100 epochs (rounds), the learning rate is 0.0001, and the term learning rate decay strategy is adopted. The Adam (optimization algorithm) optimizer is used to update the training model. The test dataset is the commonly used infrared and visible light image fusion dataset TNO (dataset) and the test set of MSRS.

[0055] Figure 8The visual quality comparison of two typical images in the TNO test set is shown. In order to more intuitively demonstrate the effectiveness of each fusion algorithm in merging complementary information and improving the quality of the fused image, this implementation method selects night scenes and scenes blocked by objects such as tree branches. In order to highlight the visual effect, representative targets and details are marked with orange-red and blue frames, and these areas are enlarged for more detailed observation.

[0056] like Figure 8 As shown in (a), U2Fusion and DIVFusion have obvious ghosting in the image (such as the white area of ​​the person's head), and the edges are too bright. GANMcC and FusionGAN perform poorly in preserving image details, resulting in a darker overall image and blurry characters and branches. The images fused by LRRNet, TarDAL, and DenseFuse do not perform well in preserving the illumination edge gradient information. The image fused by PIAFusio loses the details of the outline of the character's clothes, and the texture of some small branches is also blurred. The outlines of the characters and trees in the IR (infrared) infrared image are relatively clear, but the resolution and texture information are still relatively small. The color information of the VIS (visible light) visible light image is rich, but at night, the character target is blurred and lost, the shadow of the tree is missing, the detail information is not clear enough, and the brightness of some areas is too high. In contrast, Ours and CDDFuse can effectively fuse infrared and visible light images while retaining image detail information well, such as the face of the person, the clothes, and the edges of the branches. Compared with CDDFuse, Ours better balances the information extraction of the two modalities, while CDDFuse focuses more on the infrared modality and is insufficient in extracting features from the visible light modality, such as blurring or losing details of small branches.

[0057] like Figure 8As shown in (b), the IR image highlights the person, but the details of the branches are lost and the outline is blurred. There is less brightness information in the image. The details of the VIS people and scenery become blurred or difficult to recognize due to insufficient lighting. The branches have more details and the color information in the image is rich. The details of the person in DIVFusion are rich, such as the belt part, but the overall brightness is brighter and artifacts appear in the outline of the branches. Although LRRNet and U2Fusion perform well in preserving the outline of pedestrians, the overall contrast is slightly insufficient. FusionGAN, DenseFuse and GANMcC focus more on the prominent performance of infrared targets, so the environmental details in the visible light image are not well preserved, resulting in a relatively simple overall image texture. Although TarDAL, PIAFusion and CDDFuse perform well in character processing, the preservation of environmental details (such as trees and ground) is not as good as Ours, and the contrast adjustment is slightly insufficient.

[0058] In general, this implementation can well preserve environmental details, such as tree texture and ground texture, while preserving the clarity of the characters. This makes the fused image more natural and has the best visual effect. Compared with several other fusion algorithms, the network architecture in this solution shows better fusion effects in terms of visual evaluation and objective evaluation.

[0059] like Figure 7 As shown, this embodiment selects nine state-of-the-art methods, including CNN-based methods, namely LRRNet (Low Rank and Recursive Residual Network), U2Fusion (U2 Fusion), GAN-based methods, namely FusionGAN (Fusion Generative Adversarial Network), TarDal (Tardal) and GANMcC (Specific Generative Adversarial Network Model), autoencoder-based methods DenseFuse (Dense Fusion) and DIVFusion (DIV Fusion), Transformer-based methods, CDDFuse (CDD Fusion), PIAFusion (Progressive Fusion of Illumination Perception), and a method for fusion of infrared images and visible light images described in this embodiment, denoted as Ours. At the same time, this embodiment uses eight indicators to quantitatively evaluate EN (entropy), SD (standard deviation), SF (spatial frequency), MI (mutual information), SCD (sum of correlated differences), VIF (visual information fidelity), Qab / f (gradient-based fusion performance) and SSIM (structural similarity index measure).

[0060] Figure 7 A quantitative comparison of 21 pairs of images from the TNO test set is shown. It is worth noting that each value in the figure is the average based on the number of images. Figure 7As shown in the figure, the implementation method has achieved satisfactory results in all eight indicators, including EN, SD, SF, MI, SCD, VIF, Qab / f and SSIM. Ours performs well in multiple key indicators, especially in EN, MI and SF. It can capture and fuse more source image information, and the generated image has high clarity and good visual information retention. These results show that the proposed Ours can effectively fuse the rich information in the source images into a fused image. The fused image contains both high-brightness infrared information and texture details of visible light images. In addition, the EN index, MI index and SSIM index of Ours are 7.15, 2.21 and 0.99 respectively, which are the highest among all algorithms, indicating that the algorithm can capture more image information and details and retain the most source image information in the fused image. In terms of indicators such as SD, SF, SCD and Qab / f, it is still at a high level. The algorithm is also excellent in the comprehensive performance of brightness, contrast and clarity.

[0061] According to the results of quantitative comparison, the present implementation outperforms the state-of-the-art methods in terms of fusion performance, and the results reach the same conclusion as the previous comparison on visual quality.

[0062] Embodiment 6: The infrared image and visible light image fusion system described in this embodiment is specifically: The infrared image and the visible light image are respectively input into the convolution embedding module for feature extraction to obtain the infrared stitching features. and visible light stitching features , among which, one infrared splicing feature and visible light stitching features The two-modal interaction residual fusion module is input together for feature fusion, and the initial scale bimodal mixed features are output. ; Another infrared stitching feature and visible light stitching features The multi-cascade hybrid global and local feature extraction modules are input into the feature extraction module, and the infrared stitching features of the first scale are output respectively. , infrared stitching features of the second scale , infrared stitching features at the third scale , the first scale visible light stitching features , visible light stitching features at the second scale and the third scale visible light stitching features ; Infrared stitching features at the first scale Visible light stitching features with the first scale The two models are jointly input into the bimodal interactive residual fusion module for feature fusion, and the bimodal mixed features of the first scale are output. ; Second scale infrared stitching features Visible light stitching features with the second scale The two models are jointly input into the bimodal interactive residual fusion module for feature fusion, and the bimodal mixed features of the second scale are output. ; Infrared stitching features at the third scale Visible light stitching features with the third scale The two models are jointly input into the bimodal interactive residual fusion module for feature fusion, and the bimodal mixed features of the third scale are output. ; Bimodal hybrid features at the third scale Input image reconstruction module, output reconstructed feature map ; Reconstruct feature map Bimodal hybrid features with the second scale The two modules are jointly input into the dual-modal interactive residual fusion module for feature fusion, and then input into the image reconstruction module to output the reconstructed feature map. ; Reconstruct feature map Bimodal hybrid features with the first scale The two modules are jointly input into the dual-modal interactive residual fusion module for feature fusion, and then input into the image reconstruction module to output the reconstructed feature map. ; Reconstruct feature map Bimodal hybrid features with initial scale The two modules are jointly input into the dual-modal interactive residual fusion module for feature fusion, and then input into the image reconstruction module to output the reconstructed feature map. ; Reconstructing feature maps After upsampling, multiple convolutional layers, activation functions, and splicing operations, the fused image is output.

[0063] Embodiment 7: An electronic device, characterized in that it includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; Memory, used to store computer programs; The processor is used to implement the method steps described in the first embodiment when executing the program stored in the memory.

[0064] Embodiment 8: A computer-readable storage medium of this embodiment stores a computer program, and when the computer program is executed by a processor, the method steps described in embodiment 1 are implemented.

[0065] The infrared image and visible light image fusion method, system, device and storage medium proposed in the present invention are introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for general technicians in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A method for fusing infrared images and visible light images, characterized in that: Specifically: The infrared image and the visible light image are respectively input into the convolution embedding module for feature extraction to obtain the infrared stitching features. and visible light stitching features , among which, one infrared splicing feature and visible light stitching features The two-modal interaction residual fusion module is input together for feature fusion, and the initial scale bimodal mixed features are output. ; Another infrared stitching feature and visible light stitching features The multi-cascade hybrid global and local feature extraction modules are input into the feature extraction module, and the infrared stitching features of the first scale are output respectively. , infrared stitching features of the second scale , infrared stitching features at the third scale , the first scale visible light stitching features , visible light stitching features at the second scale and the third scale visible light stitching features ; Infrared stitching features at the first scale Visible light stitching features with the first scale The two models are jointly input into the bimodal interactive residual fusion module for feature fusion, and the bimodal mixed features of the first scale are output. ; Second scale infrared stitching features Visible light stitching features with the second scale The two models are jointly input into the bimodal interactive residual fusion module for feature fusion, and the bimodal mixed features of the second scale are output. ; Infrared stitching features at the third scale Visible light stitching features with the third scale The two models are jointly input into the bimodal interactive residual fusion module for feature fusion, and the bimodal mixed features of the third scale are output. ; Bimodal hybrid features at the third scale Input image reconstruction module, output reconstructed feature map ; Reconstruct feature map Bimodal hybrid features with the second scale The two modules are jointly input into the dual-modal interactive residual fusion module for feature fusion, and then input into the image reconstruction module to output the reconstructed feature map. ; Reconstruct feature map Bimodal hybrid features with the first scale The two modules are jointly input into the dual-modal interactive residual fusion module for feature fusion, and then input into the image reconstruction module to output the reconstructed feature map. ; Reconstruct feature map Bimodal hybrid features with initial scale The two modules are jointly input into the dual-modal interactive residual fusion module for feature fusion, and then input into the image reconstruction module to output the reconstructed feature map. ; Reconstructing feature maps After upsampling, multiple convolutional layers, activation functions, and splicing operations, the fused image is output.

2. The method for fusing infrared images and visible light images according to claim 1, characterized in that: The convolution embedding module is specifically: The image is divided into two paths. One path is sequentially input into the convolution layer, normalized, activated, and flattened. The channel dimension is swapped with the flattened spatial dimension to output a local feature map. Another image is input into Patch Embedding, and after the channel dimension is exchanged with the flattened spatial dimension, the position encoding is input and the global feature map is output; The local feature map and the global feature map are added element by element to output the feature map.

3. The method for fusing infrared images and visible light images according to claim 1, characterized in that: The hybrid global-local feature extraction module is specifically: The image is divided into two paths. One path is reshaped in turn to output a feature map. After feature extraction, the feature map is input into a multi-cascade local feature extraction module for feature extraction, and then reshaped again to output a local feature map. The other image is reconstructed and input into the one-dimensional convolution layer, and then reshaped again to output the feature map. The feature map is input into the multi-head attention and the global feature map is output. The local feature map and the global feature map are added element by element, and a mixed global-local feature map is output.

4. The method for fusing infrared images and visible light images according to claim 3, characterized in that: The feature map is used for feature extraction, specifically: The feature map is divided into two paths, one feature map is input into the residual block, and the other feature map is input into the two-dimensional convolution layer. The feature map passing through the residual block and the other feature map passing through the two-dimensional convolution layer are added element by element.

5. The method for fusing infrared images and visible light images according to claim 3, characterized in that: The local feature extraction module performs feature extraction, specifically: The image is sequentially input into multi-cascade convolution and multi-cascade bottleneck layers for feature extraction.

6. The method for fusing infrared images and visible light images according to claim 1, characterized in that: The dual-modal interactive residual fusion module is specifically: The images input into the dual-modal interactive residual fusion module include at least two types, namely a first image and a second image; After global average pooling is performed on the first image and the second image respectively, the first feature map and the second feature map are output respectively, attention is allocated to the first feature map and the second feature map respectively, and the first weight and the second weight are output respectively. The first weight is element-wise multiplied by the first feature map, and the first feature map after element-wise multiplication is output. The second weight is element-wise multiplied by the second feature map, and the second feature map after element-wise multiplication is output. The first feature map after element-wise multiplication and the second feature map after element-wise multiplication are added element-by-element, and then input into a multi-stage convolutional layer, and a feature map is output. The feature map is added element-by-element with the first image and the second image, and a feature fused image is output.

7. The method for fusing infrared images and visible light images according to claim 1, characterized in that: The image reconstruction module is specifically: After the image is input to the convolutional layer for feature extraction, a feature map is output. The feature map undergoes layer normalization and multi-head attention operations in sequence, and the feature map after the multi-head attention operation is output. The feature map after the multi-head attention operation is added element by element with the feature map. Then, the layer normalization and multi-layer perception mechanism operations are performed in sequence, and the feature map after the multi-head perception mechanism operation is output. The feature map after the multi-head perception mechanism operation is added element by element with the feature map after the head attention operation.

8. An infrared image and visible light image fusion system, characterized in that: Specifically: The infrared image and the visible light image are respectively input into the convolution embedding module for feature extraction to obtain the infrared stitching features. and visible light stitching features , among which, one infrared splicing feature and visible light stitching features The two-modal interaction residual fusion module is input together for feature fusion, and the initial scale bimodal mixed features are output. ; Another infrared stitching feature and visible light stitching features The multi-cascade hybrid global and local feature extraction modules are input into the feature extraction module, and the infrared stitching features of the first scale are output respectively. , infrared stitching features of the second scale , infrared stitching features at the third scale , the first scale visible light stitching features , visible light stitching features at the second scale and the third scale visible light stitching features ; Infrared stitching features at the first scale Visible light stitching features with the first scale The two models are jointly input into the bimodal interactive residual fusion module for feature fusion, and the bimodal mixed features of the first scale are output. ; Second scale infrared stitching features Visible light stitching features with the second scale The two models are jointly input into the bimodal interactive residual fusion module for feature fusion, and the bimodal mixed features of the second scale are output. ; Infrared stitching features at the third scale Visible light stitching features with the third scale The two models are jointly input into the bimodal interactive residual fusion module for feature fusion, and the bimodal mixed features of the third scale are output. ; Bimodal hybrid features at the third scale Input image reconstruction module, output reconstructed feature map ; Reconstruct feature map Bimodal hybrid features with the second scale The two modules are jointly input into the dual-modal interactive residual fusion module for feature fusion, and then input into the image reconstruction module to output the reconstructed feature map. ; Reconstruct feature map Bimodal hybrid features with the first scale The two modules are jointly input into the dual-modal interactive residual fusion module for feature fusion, and then input into the image reconstruction module to output the reconstructed feature map. ; Reconstruct feature map Bimodal hybrid features with initial scale The two modules are jointly input into the dual-modal interactive residual fusion module for feature fusion, and then input into the image reconstruction module to output the reconstructed feature map. ; Reconstructing feature maps After upsampling, multiple convolutional layers, activation functions, and splicing operations, the fused image is output.

9. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, for implementing the method steps described in any one of claims 1 to 7 when executing a program stored in a memory.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps described in any one of claims 1 to 7 are implemented.

Images