Differential feature guided spatial channel multi-modal image fusion method and system

Abstract

The invention discloses a difference feature guided spatial channel multi-modal image fusion method and system, and the method specifically comprises the steps: 1, carrying out the fusion of a visible light multi-modal image and an infrared light multi-modal image at a plurality of layers, and carrying out the complementation of the missing information between two modals; obtaining image features of the visible light multi-modal image and image features of the infrared light multi-modal image; step 2, performing interaction and fusion on the two image features in the step 1 on a channel level to obtain features corresponding to the fused visible light multi-modal image and features corresponding to the fused infrared light multi-modal image; 3, performing spatial fusion on the features obtained in the step 2 to obtain fused spatial features; according to the invention, interactive fusion can be realized between the visible light mode and the infrared light mode, so that the quality of the fused image is improved. According to the method, the disadvantage of a single-mode image in a downstream task can be effectively solved, and complementary information of two modes can be better utilized.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image processing, and particularly relates to a difference feature guided spatial channel multi-modal image fusion method and system. BACKGROUND

[0002] Image fusion combines basic information representation from multiple source images to generate high-quality, content-rich fused images. According to the differences in imaging devices or imaging settings, image fusion can be divided into various types, including multi-modal image fusion (MMIF), digital photography image fusion, and remote sensing image fusion. Visible-infrared multi-modal image fusion (VIF) is a typical task of MMIF, aiming to integrate complementary information from different modalities into a single fused image according to visual image information captured by visible light image sensors and infrared light image sensors under the same viewing angle, to comprehensively represent the imaging scene and facilitate the completion of downstream visual tasks. These fused images provide enhanced scene representation and visual perception, which is helpful for subsequent practical visual applications such as multi-modal saliency detection, object detection, and semantic segmentation. As an important research direction in the field of computer vision, it has key application value in night market monitoring, autonomous driving, and unmanned aerial vehicle navigation. Compared with methods using lidar and depth cameras as sensors, visible-infrared image sensors are low in price, easy to use, and have high cost performance.

[0003] Visible light vision images provide rich details with high spatial resolution under well-defined lighting conditions, while infrared sensors capture changes in environmental temperature emitted by objects, highlighting heat target structures that are insensitive to lighting changes. However, infrared images are usually accompanied by blurred details with low spatial resolution, so multi-modal image fusion can better combine visible light and infrared light information, making the information between the two modalities complementary to each other, and fusing into an image with more complete information, so that the image can better facilitate the performance of downstream visual tasks. In recent years, visible-infrared multi-modal image fusion methods include traditional CNN convolutional neural network methods, diffusion model-based fusion methods, and Transformer-based fusion network methods.

[0004] Although existing methods have made great progress in visible-infrared multi-modal image fusion, there are still some deficiencies. The local receptive field of CNN is difficult to capture global information, and although the fusion network based on transformer can process global information, the computational complexity will increase seriously when the image resolution is high, which will affect the inference efficiency. The linear state space model of Mamba can maintain a low computational complexity while preserving global information, which has a great advantage over transformer model when facing high-resolution images. However, the current visible-infrared multi-modal image fusion method based on Mamba cannot fully fuse the complementary information of the two modalities, and is not perfect enough in the face of glare, overexposure, smoke interference and other environments. The fusion method needs to be further improved. SUMMARY

[0005] The purpose of the application is to solve the problems existing in the prior art, and provide a difference feature guided spatial channel multi-modal image fusion method.

[0006] Technical scheme: The application discloses a difference feature guided spatial channel multi-modal image fusion method, comprising the following steps:

[0007] Step 1: fuse the visible light multi-modal image and the infrared light multi-modal image at multiple levels, and complement the missing information between the two modalities; obtain the image features of the visible light multi-modal image and the image features of the infrared light multi-modal image

[0008] Step 2: interact and fuse and at the channel level to obtain the features corresponding to the fused visible light multi-modal image and the fused infrared light multi-modal image

[0009] Step 3: spatially fuse and to obtain the fused spatial feature F S ;

[0010] Step 4: input the fused spatial feature F S to the image decoder to obtain the final fused image.

[0011] Further, in step 1, N difference guided feature extraction modules are used to fuse and complement the missing information of the visible light multi-modal image and the infrared light multi-modal image.

[0012] Further, the difference guided feature extraction module comprises a first convolution module, a second convolution module and a visual feature extraction encoder; for the nth difference guided feature extraction module, the first convolution module performs convolution processing on the input feature , the second convolution module performs convolution processing on the input feature , and the output features of the two convolution modules are input into the visual feature extraction encoder to obtain the image features of the visible light multi-modal image and the image features of the infrared light multi-modal image output by the nth difference guided feature extraction module and respectively.

[0013] Further, the visual feature extraction encoder inputs the input feature into an infrared light VSS module, inputs the input feature into a visible light VSS module, inputs and into a third convolution module respectively, inputs the output of the third convolution module into a shared VSS module, performs difference calculation on the output of the shared VSS module, then inputs the difference feature into a difference mask module to obtain a difference mask Diff, inputs Diff, the infrared light VSS module and the visible light VSS module into a difference guided module, and the difference guided module processes the input feature as follows:

[0014]

[0015]

[0016] wherein, represents the output feature of the infrared light VSS module after is input into the infrared light VSS module; represents the output feature of the visible light VSS module after is input into the visible light VSS module.

[0017] Further, in step 2, the channel fusion module is used to interact and fuse and in the channel level; first, expand and into one-dimensional feature sequences T ir,j and T vi,j according to four directions respectively, where j represents the jth direction, j = 1, 2, 3, 4; the four directions are: from top to bottom, from left to right, from bottom to top, and from right to left.

[0018] T ir,j ​and T vi,j are mapped into the corresponding matrices in the Mamba state space respectively:

[0019] x vi,j ,B vi,j ,C vi,j =split[Linear(Flatten(T vi,j ))]

[0020] x ir,j ,B ir,j ,C ir,j =split[Linear(Flatten(T ir,j ))]

[0021] wherein, Flatten represents a flattening operation, Linear represents a linear transformation, split represents a split, the C matrix corresponds to the Q value of the attention mechanism, the B matrix corresponds to the K value of the attention mechanism, and the x matrix corresponds to the V value of the attention mechanism; the B matrix and the x matrix corresponding to the infrared light are exchanged with the B matrix and the x matrix corresponding to the visible light; based on the exchanged matrices, infrared light two-dimensional feature maps and visible light two-dimensional feature maps

[0022]

[0023]

[0024] wherein, Projection represents projecting an input matrix into a three-dimensional matrix, CrossMerge is a cross-merge operation, SSM1 is a state space model corresponding to a one-dimensional feature sequence in a first direction; SSM4 is a state space model corresponding to a one-dimensional feature sequence in a fourth direction;

[0025] are input into the infrared light residual module to obtain features are input into the visible light residual module to obtain features The channel fusion module also fuses and after being spliced in the channel dimension, performs a convolution operation, and compresses the feature after the convolution operation into a one-dimensional channel vector ω; based on ω, the features and are respectively weighted and fused in the channel dimension:

[0026]

[0027]

[0028] ​​Further, the infrared light residual module is used to perform the following operations:

[0029]

[0030] The visible light residual module is used to perform the following operations:

[0031]

[0032] Wherein, alpha and beta are weight matrices, and projection(.) represents projection.

[0033] Further, the multi-scale cross-modal state space in step 3 is used on the pixel-by-pixel scanning sequence to constantly learn the information of each pixel on the two input features, and the feature F S is obtained.

[0034] A difference feature guided spatial channel multi-modal image fusion method includes a difference feature guided feature fusion network, a channel level feature fusion network, and a spatial level pixel level fusion network.

[0035] The difference feature guided feature fusion network is used to extract the features of the visible light image and the infrared light image and combine the information preliminarily, and fully mix the complementary information of the two modalities.

[0036] The channel fusion module is used to mix the features output by the difference feature guided feature fusion network on the channel.

[0037] The spatial level pixel level fusion network is used to learn the two modalities of the features output by the channel level feature fusion network pixel by pixel and finally fuse them into an image.

[0038] A computer device includes a memory, a processor, and a computer program stored in the memory and capable of running on the processor, and the processor implements the steps of the difference feature guided spatial channel multi-modal image fusion method when executing the computer program.

[0039] A computer readable storage medium stores a computer program, and the computer program implements the steps of the difference feature guided spatial channel multi-modal image fusion method when executed by a processor.

[0040] Advantages:

[0041] 1、The application constructs a state space model feature interaction network based on visual Mamba, and fuses the visible light-infrared light multi-modal image at multiple levels from the perspective of the state space model, and enhances the anti-interference ability to glare, overexposure, smoke and the like.

[0042] 2、The application can realize interactive fusion between the visible light and infrared light two modes, thereby improving the quality of the fused image.

[0043] 3、The application designs a multi-modal interaction mechanism based on visual Mamba, utilizes the simple modular design and the self parameters of the network structure, and gives more robust and flexible feature extraction, so that the visible light and infrared light two modes are guided to interact and fuse, the contrast of the fused image is enhanced, and the quality of the image is improved, which is beneficial to the downstream task of the image.

[0044] 4、The difference feature guided fusion module, the gating attention mechanism of visual Mamba and the multi-scale pixel fusion module can effectively combine the information of the visible light and infrared light two modes, and still reliable results can be obtained in some environments with greater interference. BRIEF DESCRIPTION OF DRAWINGS

[0045] Figure 1 The method flowchart of the application.

[0046] Figure 2 The overall network structure schematic diagram of the application.

[0047] Figure 3 The four direction diagrams when the visible light and infrared light two modes are subjected to feature expansion. DETAILED DESCRIPTION

[0048] The accompanying drawings, which form a part of this application, are included to provide a further understanding of the application, and are incorporated in and constitute a part of this application. The embodiments of these drawings illustrate the preferred embodiment of the present application and, together with the description, serve to explain the principles of the application.

[0049] The method provided by the application is as shown in Figure 1 Specifically, the method comprises the following steps.

[0050] Step 1: The feature fusion network based on the state space scanning mechanism fuses the visible light-infrared light multi-modal image at multiple levels, complements the missing information between the two modes, and improves the quality of the fused image.

[0051] Step 2: The module for fusion guided by the feature difference enhances the extraction and utilization of the difference information by the model.

[0052] Step 3: Based on the visual Mamba attention gate mechanism, the two modal features are interacted to enhance the contrast information of the fused image.

[0053] As shown in Figure 2 , the feature fusion network of the state space scanning mechanism in step 1 is the difference guided feature extraction part in Figure 2 , specifically:

[0054] Feature extraction is performed on the original infrared light I i and visible light image I v of the two modalities. First, a two-layer simple CNN convolutional neural network is used for feature extraction:

[0055] F ir = Leakyrelu(Conv(I i ))

[0056] F vi = Leakyrelu(Conv(I v ))

[0057] Where F ir is the infrared feature map after feature extraction, and F vi is the visible light feature map after feature extraction.

[0058] Then, the visual feature extraction encoder based on the state space scanning mechanism of visual Mamba (corresponding to the difference guided feature extraction part of Figure 2 ) is used to extract the features of the visible light and infrared light image information:

[0059]

[0060]

[0061] In this embodiment, there are N MambaEncoders (visual Mamba state space scanning mechanism visual feature extraction encoder) connected in turn, is the visible light feature map output by the n-1th difference guided feature, is the infrared light feature map output by the n-1th difference guided feature. Each layer of MambaEncoder interacts and learns the features of the two modalities at the feature level

[0062] After fully modeling the image features of the two modalities, the features of the two modalities are interacted and fused at the channel level:

[0063]

[0064]

[0065] the feature of visible light output for the Nth difference guided feature, the feature of infrared light output for the Nth difference guided feature, ChannelFuse represents the channel fusion module.

[0066] After the features are further fused at the spatial level (corresponding to Figure 1 spatial fusion of the spatial fusion), a multi-scale multi-modal state space model (Mmtss) is used to constantly learn the information of each pixel on the two modal features on the pixel-by-pixel scanning sequence:

[0067]

[0068] Finally, the obtained fused image is output through the reconstruction layer:

[0069] Output=img_Dec(F S )

[0070] wherein img_Dec is an image decoder.

[0071] wherein the feature difference guided fusion module comprises:

[0072] For each feature difference guided fusion module, independent feature encoders of visible light and infrared light are set up for extracting the respective features possessed by the output of one layer of the two modalities. ir ,F vi represent the features extracted through the convolutional layer. Then, the features of the visible light and infrared light image information are extracted through the visual feature extraction encoder based on the state space scanning mechanism of the visual Mamba:

[0073]

[0074] wherein is the feature output by the independent visible light feature encoder in the nth feature difference guided fusion module, is the feature output by the independent infrared light feature encoder in the nth feature difference guided fusion module; VSS represents the VSS module of the independent branch in Figure 2 Meanwhile, the VSS module of the shared branch performs shared feature learning on the original features:

[0075]

[0076] The superscript share represents a shared branch. The difference mask Diff is obtained by differencing the shared features:

[0077]

[0078] Finally, the feature fusion is guided by the difference mask:

[0079]

[0080]

[0081] The features obtained by the difference feature weight mask guidance are sent to the next layer of feature difference guidance fusion module for further feature extraction, so as to fully learn the complementary information between the two modalities.

[0082] The channel level is based on the gating attention mechanism of visual Mamba Figure 2 The channel fusion part is specifically:

[0083] First, a cross-modal channel exchange similar to the channel attention mechanism. The visible light feature output by the Nth feature difference guidance fusion module and the infrared light feature output by the Nth feature difference guidance fusion module are input into the channel fusion part, and and are respectively unfolded into one-dimensional feature sequences in four directions of the respective 2D image features (how the four directions are shown in Figure 3 ):

[0084]

[0085]

[0086] where T vi is a one-dimensional sequence, j represents the jth direction, and j = 1, 2, 3, 4.

[0087] Then the feature sequence is mapped into the corresponding matrix in the Mamba state space. The linear attention mechanism of the state space can map the one-dimensional feature sequence corresponding to the two modalities in one direction into a matrix. Take one direction as an example:

[0088] x vi,j ,B vi,j ,C vi,j = split[Linear(Flatten(T vi,j ))]

[0089] x ir,j ,B ir,j ,C ir,j= split [ Linear ( Flatten ( T ir,j ))

[0090] Similar to the cross attention mechanism of the Transformer, x (corresponding to the Value of the attention mechanism), C corresponds to the Query of the attention mechanism, and B (corresponding to the Key of the attention mechanism) and C of the two modalities are exchanged, and then input into the SSM module to learn the features from the perspective of the state space, and then restore the four-direction sequence to the original two-dimensional feature map:

[0091]

[0092]

[0093] Wherein, Projection represents projecting the input matrix into a three-dimensional matrix, CrossMerge is a cross-merge operation, SSM1 is a state space model corresponding to the one-dimensional feature sequence of the first direction; SSM4 is a state space model corresponding to the one-dimensional feature sequence of the fourth direction.

[0094] Then a residual connection is introduced to retain useful information of the original input features, and a learnable parameter is added before the residual connection to limit the original input features, so as to ensure that the next layer of feature learning is not intervened too much. And introduce a learnable weight matrix α, β to further weight the output features added by the residual connection:

[0095]

[0096]

[0097] Meanwhile, the application also adds a channel gating mechanism to further fuse the channel features, and the original input of the two modalities is spliced in the channel dimension, and then the features are learned through a neural network, and then the learned features are compressed into a one-dimensional channel vector ω to screen and combine the features after channel exchange:

[0098]

[0099] The channel attention weight matrix generated in this way is multiplied by the features of the two modalities, which is beneficial to the fused channel features, and inhibits the channel features with small fusion effect, and finally obtains the output of the channel fusion

[0100]

[0101]

[0102] The key scheme of the application is further introduced below in combination with specific examples and calculation formulas, and details are described below:

[0103] I. Data preparation

[0104] The application verifies the effectiveness of the proposed method in the visible light-infrared light multi-modal image fusion data set M3FD.

[0105] The M3FD data set is obtained by a synchronous system sensor composed of a binocular optical camera and a binocular infrared sensor. The baseline (the distance between the focal points of the binocular lens) of the visible light binocular camera is 12 cm, and the baseline of the infrared binocular camera is 20 cm. The optical center distance between the visible light and the infrared sensor is 4 cm. The main scenes include the campus of Dalian University of Technology, the Golden Beach of China National Tourist Resort in Dalian, and the main roads in Jinzhou District of Dalian City. The visible light image has a high resolution of 1024x768 and a wide imaging range, while the standard resolution of the infrared image is 640x512, and finally the two modalities are mapped to a resolution of 1024x768. At the same time, the wavelength range is 8-14 microns. The 4200 groups of registered image pairs in the M3FD data set are divided into four typical types (day, cloudy, night, challenging scene), containing a variety of sub-scenes. At the same time, 33603 targets of six categories are labeled, including pedestrians, cars, buses, motorcycles, trucks and street lamps - these categories are typical in monitoring and autonomous driving scenes. The size and diversity of the M3FD data set provide a solid foundation for learning and evaluation of multi-modal image fusion target detection.

[0106] In the feature extraction stage, the preliminary features are realized using a conventional convolutional neural network, and then the VSS module guided by the difference features is used to extract the features of the two modalities, so as to obtain the modality feature maps after information complementation. Then, at the channel level, the features of the two modalities are fused, and the attention mechanism similar to Transformer is used to cross-attention the features of each scanning direction of the two modalities based on the visual Mamba state space scanning mechanism, and then residual connection and weight matrix activation are added, and channel gating weighting is used to obtain channel fusion features. Finally, the spatial fusion module is used to fuse the features of the two modalities into one picture at the pixel level, and the final picture output is obtained through picture reconstruction.

[0107] The image fusion evaluation indicators are used to evaluate the fusion image results, including information entropy (Entropy, EN), standard deviation (Standard Deviation, SD), spatial frequency (Spatial Frequency, SF), mean squared error (Mean Squared Error, MSE), and average gradient (Average Gradient, AG).

[0108] Information entropy measures the amount of information contained in the fusion image. The larger the value, the richer the image information, and the more information of the source image is retained. The calculation formula is:

[0109]

[0110] Where p(i) is the probability of gray level i appearing in the image, and L is the total number of gray levels (usually 256).

[0111] Standard deviation reflects the dispersion degree of image pixel value, and measures the contrast and detail richness. High SD value indicates that the image contrast is strong and the details are clear. The calculation formula is:

[0112]

[0113] Spatial frequency measures the spatial activity (richness of details and edges) of the image. High SF value indicates that the image edge is clear and the texture details are rich. It is divided into row frequency (RF) and column frequency (CF), and is integrated into total spatial frequency:

[0114]

[0115] Mean square error measures the difference between the fusion image and the reference image (Ground Truth). The smaller the MSE, the closer the fusion image is to the reference image. The formula is:

[0116]

[0117] Average gradient reflects the image sharpness and texture change. The larger the value, the sharper the edge. High AG value indicates that the image edge is clear and the texture is rich. The formula is:

[0118]

[0119] The embodiment is compared on the M3FD dataset to prove that the present application has better fusion results.

[0120] The input image resolution of the present application is the original resolution of each dataset image. In the training stage, the image is randomly cropped into 128x128 image blocks for training, and 100 epochs are trained. The optimizer uses Adam optimizer, the learning rate is 2x10-5, the weight decay rate is 0, the batch size is set to 4, and the training is performed on two V32 GPUs of auto dl, and the usual training time is 24 hours.

[0121] The multi-modal image fusion method proposed in the present application is compared with five relatively advanced fusion methods in five aspects. The comparison results are shown in Table 1:

[0122] Table 1

[0123] Method EN SD SF MSE AG TarD 6.99 39.1 12.649 2934.172 4.256 CDD 6.9 37.238 14.776 2.709 1.647 FusionMamba 6.85 37.314 9.278 3615.307 3.393 MambaDFuse 6.8 35.194 14.112 3778.498 4.685 EMMA 6.92 38.255 15.227 3682.692 5.328 ours 7.01 42.552 16.495 3191.093 5.66

[0124] From Table 1, it can be seen that the application is superior to other methods in the four indexes of EN, SD, SF and AG.

[0125] In addition, it should be noted that various specific technical features described in the foregoing embodiments can be combined in any appropriate manner without contradiction. In order to avoid unnecessary repetition, various possible combinations are not described again in the application.

Claims

1. A spatial channel multi-modal image fusion method guided by differential features, characterized in that, The method comprises the following steps: Step 1: fuse the visible light multi-modal image and the infrared light multi-modal image at multiple levels, complement the missing information between the two modalities; obtain image features of the visible light multi-modal image and image features of the infrared light multi-modal image Step 2: the following steps are performed and interacting and fusing at the channel level to obtain features corresponding to the fused visible light multi-modal image pair features corresponding to the fused infrared light multi-modal image Step 3: perform spatial fusion on the spatial features F and to obtain fused spatial features F S ; Step 4: The fused spatial features F S are input to the image decoder to obtain the final fused image.

2. The spatial channel multimodal image fusion method with difference feature guidance according to claim 1, characterized in that, The step 1 uses N sequentially connected difference guidance feature extraction modules to fuse and complement missing information of the visible light multi-modal image and the infrared light multi-modal image.

3. The spatial channel multimodal image fusion method with difference feature guidance according to claim 2, characterized in that, The difference guided feature extraction module comprises a first convolution module, a second convolution module and a visual feature extraction encoder; for the nth difference guided feature extraction module, the first convolution module performs convolution processing on the input feature , the second convolution module performs convolution processing on the input feature , and the output features of the two convolution modules are input into the visual feature extraction encoder to obtain the image features of the visible light multi-modal image and the image features of the infrared light multi-modal image output by the nth difference guided feature extraction module and respectively.

4. The spatial channel multimodal image fusion method with difference feature guidance according to claim 3, characterized in that, The visual feature extraction encoder will input features Input to the infrared VSS module, input features To the visible light VSS module, respectively and The input is fed into the third convolutional module; the output of the third convolutional module is fed into the shared VSS module, and the difference is calculated on the output of the shared VSS module. Then, the difference features are fed into the difference mask module to obtain the difference mask Diff; the Diff, the infrared VSS module, and the visible light VSS module are fed into the difference guidance module; the difference guidance module processes the input features as follows: wherein, represents the output characteristics of the infrared light VSS module after inputting to the infrared light VSS module. represents the output characteristics of the visible light VSS module after inputting to the visible light VSS module.

5. The spatial channel multimodal image fusion method with difference feature guidance according to claim 1, characterized in that, In step 2, the channel fusion module is used to and Interact and fuse at the channel level; first, the and are respectively expanded into one-dimensional feature sequences T ir,j and T vi,j , j represents the jth direction, j = 1, 2, 3, 4; the four directions are: from top to bottom, from left to right, from bottom to top, from right to left; T ir,j and T vi,j are mapped into the corresponding matrices in the Mamba state space, respectively: x vi,j ,B vi,j ,C vi,j = split[Linear(Flatten(T vi,j ))] x ir,j ,B ir,j ,C ir,j = split[Linear(Flatten(T ir,j ))] Wherein, Flatten represents a flattening operation, Linear represents a linear transformation, split represents a split, the C matrix corresponds to the Q value of the attention mechanism, the B matrix corresponds to the K value of the attention mechanism, and the x matrix corresponds to the V value of the attention mechanism; the B matrix and the C matrix corresponding to the infrared light are exchanged with the B matrix and the x matrix corresponding to the visible light; and the infrared light two-dimensional feature map and the visible light two-dimensional feature map are obtained based on the exchanged matrices and visible light two-dimensional feature map Wherein, Projection represents projecting the input matrix into a three-dimensional matrix, CrossMerge is a cross-merge operation, SSM1 is a state space model corresponding to a one-dimensional feature sequence in the first direction; SSM4 is a state space model corresponding to a one-dimensional feature sequence in the fourth direction; Input to the infrared light residual module to obtain features Input to the infrared light residual module to obtain features Input to the visible light residual module to obtain features Input to the visible light residual module to obtain features The channel fusion module also fuses And After splicing in the channel dimension, a convolution operation is performed, and the features after the convolution operation are compressed into a one-dimensional channel vector ω; based on ω, the features are fused And Weighted fusion is respectively performed in the channel dimension:

6. The spatial channel multimodal image fusion method with difference feature guidance according to claim 5, characterized in that, Infrared light residual module pair performs the following operations: Visible light residual module performs the following operations: Wherein, α and β are both weight matrices, and Projection(.) represents projection.

7. The spatial channel multimodal image fusion method with difference feature guidance according to claim 1, characterized in that, In step 3, the multi-scale cross-modal state space is used on the pixel-by-pixel scan sequence to constantly learn the information of each pixel on the two input features to obtain the feature F S .

8. The system for applying the spatial channel multimodal image fusion method with difference feature guidance according to claim 1, characterized in that, The method comprises the following steps: The difference feature guidance feature fusion network is used to extract features of the visible light image and the infrared light image and combine the features preliminarily, and fully mix complementary information of the two modalities. The channel fusion module is used to mix the features output by the difference feature guidance feature fusion network on the channel. The spatial level pixel level fusion network is used to learn and finally fuse the features of the two modalities output by the channel level feature fusion network into an image pixel by pixel.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that, The processor executes the computer program to realize the steps of the difference feature guidance spatial channel multi-modal image fusion method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, the computer-readable storage medium comprising: The computer program is executed by the processor to realize the steps of the difference feature guidance spatial channel multi-modal image fusion method according to any one of claims 1 to 7.

Images