Multi-level cross-modal collaborative feature encoder structure
By combining Transformer and CNN and introducing CBAM attention module for multi-level cross-modal feature encoding and fusion, the problem of insufficient feature extraction and poor fusion effect in traditional multi-modal image fusion is solved, and the problem of inadequate feature extraction and poor fusion effect is realized in complex environments high-quality image generation.
Patent Information
- Application Number
- CN202410964160.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-17
- Publication Date
- 2024-11-15
AI Technical Summary
The traditional multimodal image fusion method has problems such as insufficient feature extraction and poor fusion effect, and it is difficult to generate high-quality and diverse images in complex environments.
Combining the advantages of Transformer and CNN, the CBAM attention module is introduced to perform multi-level cross-modal collaborative feature encoding and fusion of infrared images and visible light images, and channel and spatial attention weighting are performed through the CBAM module to enhance the expression ability of features.
The extraction and fusion effect of multimodal images is significantly improved, and the generated image quality and resolution are improved, so that more reliable multimodal image fusion can be achieved in complex environments.
Smart Images

Figure CN118967468A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of multimodal image fusion, and in particular to a multimodal image fusion method that adds a feature fusion part to a Transformer and CNN structure and performs feature fusion through a CBAM attention module to enhance the ability of fusing multimodal features. Background Art
[0002] In the contemporary field of multimodal image fusion, improving the ability to extract multimodal image features has always been a hot topic. However, traditional multimodal image fusion methods often face problems such as insufficient feature extraction and poor feature fusion effects. In order to overcome these problems, the multi-level cross-modal collaborative feature encoder structure combines the advantages of Transformer, which is good at processing long-distance dependencies and global information, and can capture the global features and complex contextual relationships in the image, and CNN, which is good at capturing local features and details. The CBAM attention mechanism is introduced to improve the robustness and quality of generated images in complex environments, making the generated images more diverse and realistic.
[0003] In the field of multimodal fusion, CNN has become the mainstream with its excellent feature extraction and expression capabilities, but its limitation lies in its limited ability to model global information; Transformer has rich global detail capture capabilities. Combining Transformer and CNN to process infrared images and visible light images respectively can retain the details and advantages of each modality to a great extent. In the multi-level cross-modal collaborative feature encoding and weight fusion network, infrared images and visible light images are size-normalized and then input into Transformer and CNN for feature extraction. By dividing the image into small patches and performing linear dimensionality reduction, these patches are then input into the Transformer encoder to extract feature maps at different levels. These feature maps are deeply fused through convolutional layers, and the number of channels is adjusted after upsampling to adapt them to subsequent processing. At the same time, infrared images and visible light images are also input into CNN for feature extraction. Through multi-layer convolution and pooling operations, local features in the image are extracted, and further fusion of the two modal features is achieved through element-by-element addition and convolutional layer fusion.
[0004] In order to further improve the feature fusion effect, this paper introduces the CBAM module, which uses channel and spatial attention machines to
[0005] The CBAM module can adaptively emphasize important features and suppress irrelevant information, thereby enhancing the expressiveness of features. Finally, the feature maps extracted and fused from Transformer and CNN are further fused through the CBAM module to generate high-quality image features. Through this multi-level cross-modal collaborative feature encoding and fusion network, we can obtain higher-quality multimodal fused images in complex environments and improve the resolution and detail retention of images. Summary of the invention
[0006] In view of the problems of existing multimodal fusion technology, the present invention proposes a hierarchical cross-modal collaborative feature encoder structure. The network combines the advantages of Transformer and CNN, and introduces the CBAM attention mechanism to achieve efficient multimodal feature extraction and fusion. The present invention extracts features from infrared images and visible light images respectively, and deeply fuses the features of the two modalities, thereby retaining global and local information. During the fusion process, the CBAM module is used to perform channel and spatial attention weighting on the features, further enhancing the expressive power of the features. Ultimately, the fused feature map has higher quality and resolution, and can achieve more reliable multimodal image fusion in complex environments.
[0007] To achieve the above object, the present invention provides the following technical solutions:
[0008] A hierarchical cross-modal collaborative feature encoder structure includes the following implementation steps:
[0009] Step 1: training set preprocessing;
[0010] Step 101: Normalize the size of the infrared image and visible light image datasets and adjust them to a uniform resolution of 256×256×3.
[0011] Step 2: Input the infrared image and visible light image into Transformer for preliminary fusion and extraction;
[0012] Step 201: Transformer feature extraction: The visible light image Input Stream 1 and the infrared image Input Stream 2 are divided into 16×16 patches respectively, and then the patches of the two modalities are flattened in pixel value order and linearly reduced to 512 dimensions. Next, the patches are input into the Transformer encoder.
[0013] Step 202: Extract feature maps f1 (infrared) and f1 (visible light) from the first layer for infrared images and visible light images from the encoder; feature maps f2 (infrared) and f2 (visible light) from the third layer; feature maps f3 (infrared) and f3 (visible light) from the sixth layer; and feature maps f4 (infrared) and f4 (visible light) from the twelfth layer. The first layer contains the most primitive and detailed feature information, such as edges and simple textures; the third layer begins to capture some local patterns and structures while retaining certain details; the sixth layer captures more complex patterns and structures, which is the balance point between low-level and high-level features; the twelfth layer contains rich semantic information, such as the overall shape and category of the object.
[0014] Step 203: Add f1 (infrared) and f1 (visible light) element by element, and then perform feature fusion through the convolution layer to obtain f1. Repeat the above operation to obtain f2, f3, and f4.
[0015] Step 204: f1, f2, f3, and f4 are upsampled by bilinear interpolation, and then linear dimension reduction is performed to obtain a suitable number of channels, and the output is f1: 256×256×64, f2: 192×192×128, f3: 128×128×128, and f4: 64×64×512.
[0016] Step 3: Input the infrared image and visible light image into CNN for preliminary fusion and extraction;
[0017] Step 301: Input the visible light image Input Stream 1 into the convolution layer 1 and the pooling layer, and then pass through the convolution layer 2, the pooling layer, and finally the convolution layer 4 and the pooling layer. The output of each pooling layer is marked as follows: f1(1): 128×128×64, f2(1): 64×64×128, f3(1): 32×32×256, f4(1): 16×16×512.
[0018] Step 302: Repeat the above operation with the infrared image Input Stream 2 to obtain f1(2): 128×128×64, f2(2): 64×64×128, f3(2): 32×32×256, and f4(2): 16×16×512.
[0019] Step 303: Add f1(1), f2(1), f3(1), f4(1) from the visible light image and f1(2), f2(2), f3(2), f4(2) from the infrared image element by element, and then deeply fuse the features of the two modalities through a convolutional layer; obtain f1': 128×128×64, f2': 64×64×128, f3': 32×32×256, f4': 16×16×512.
[0020] Step 304: Perform bilinear interpolation to upsample f1', f2', f3', and f4' to f1: 256×256×64, f2: 192×192×128, f3: 128×128×128, and f4: 64×64×512, matching the sizes of feature maps f1, f2, f3, and f4 obtained by Transformer.
[0021] Step 4: The features extracted by the two networks are fused through the CBAM module;
[0022] Step 401: f1', f2', f3', f4' and f1, f2, f3, f4 are respectively subjected to channel weighting adjustment and spatial weighting adjustment by the CBAM module.
[0023] Step 402: Add f4 from Transformer and f4' from CNN element by element, and mark the fused feature map as F(2)4: 64×64×512.
[0024] Step 403: Pass F(2)4: 64×64×512 through convolution layer 1 and 2×2 pooling layer to obtain F(2)3: 32×32×256.
[0025] Step 404: perform 4×4 average pooling on f3: 128×128×256 from Transformer and f3': 128×128×256 from CNN respectively, and downsample to f3": 32×32×256.
[0026] Step 405: Add F(2)3: 32×32×256 and the downsampled f3″: 32×32×256 element by element, and then pass through convolution layer 2 and 2×2 pooling layer to obtain F(2)2: 16×16×128.
[0027] Step 406: Perform 12×12 average pooling on f2: 192×192×128 from Transformer and f2': 192×192×128 from CNN and downsample to f2": 16×16×128.
[0028] Step 407: F(2)2: 16×16×128 is added element-by-element to the downsampled f2″: 16×16×128, and then passed through the deconvolution layer 1 to obtain H(2)1′: 256×256×64.
[0029] Step 408: The upsampled H(2)1' is fused element by element with f1: 256×256×64 from Transformer and f1': 256×256×64 from CNN, and finally H1: 256×256×64 is obtained.
[0030] Compared with the prior art, the present invention provides a hierarchical cross-modal collaborative feature encoder structure, which has the following beneficial effects:
[0031] The present invention provides a hierarchical cross-modal collaborative feature encoder structure, which aims to optimize the multimodal fusion process of infrared images and visible light images. By combining the characteristics of Transformer and CNN and introducing the CBAM attention mechanism, efficient feature extraction and fusion of multimodal images are achieved. This method brings very little increase in the amount of computation in the feature fusion part, but significantly improves the adequacy of feature extraction and the fusion effect. By retaining global and local information, the present invention effectively solves the problems of insufficient feature extraction and poor fusion effect in traditional methods, and improves the quality and diversity of generated images. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] Figure 1 It is the Tranfoemer feature extraction graph;
[0033] Figure 2 It is the CNN feature extraction map;
[0034] Figure 3 This is the schematic diagram of the CBAM module;
[0035] Figure 4 It is the feature fusion data graph;
[0036] Figure 5 It is the overall data flow diagram; DETAILED DESCRIPTION
[0037] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0038] Combination Figures 1 to 5As shown, a multi-level cross-modal collaborative feature encoder structure under this embodiment includes the following steps:
[0039] Step 1: Preprocessing of training set.
[0040] Step 101: Read the visible light image and infrared image from the dataset, and resize the image to 256x256 pixels by bilinear interpolation. Next, check the number of channels of the image. If it is a grayscale image (one channel), copy the grayscale value to the red, green, and blue channels, and keep the image size 256×256×3.
[0041] Step 2: Input the infrared image and visible light image into Transformer for preliminary fusion and extraction.
[0042] Step 201: A double loop is used to traverse the visible light image Input Stream 1 and the infrared image Input Stream 2. The outer loop controls the rows and the inner loop controls the columns. A patch is cut out each time. The patches are divided into 16×16 patches. The size of each patch is 16×16×3.
[0043] Step 202: Flatten the patches of the two modalities into 768 (16×16×3) in the order of pixel values and reduce them to 512 dimensions through linear dimensionality reduction.
[0044] Step 203: The patch is input into the Transformer encoder and goes through twelve layers of extraction in sequence. Each layer includes a multi-head self-attention mechanism, residual links and layer normalization (first time), a feedforward network layer, residual links and layer normalization (second time) and then outputs the features to the next layer.
[0045] Step 204: extract from the encoder the feature map f1 (infrared) from the first layer of the infrared image; the feature map f2 (infrared) from the third layer; the feature map f3 (infrared) from the sixth layer; and the feature map f4 (infrared) from the twelfth layer.
[0046] Step 205: Add f1 (infrared) and f1 (visible light) element by element to get f1: 16×16×512. Pass f1 through convolution layer 1 with a kernel size of 1×1×512×512, a stride of 1, and a padding of 0. The calculation formula for the convolution layer size is:
[0047]
[0048] Among them, H out is the output size, H in is the input, P is the padding, s is the stride, and K is the convolution kernel size, which can be calculated as It can be seen that the size has not changed, and the output f1 is: 16×16×512.
[0049] Step 206: Repeat the above operation, and f2, f3, and f4 are respectively output through convolution layer 2, convolution layer 3, and convolution layer 4: f2: 16×16×512; f3: 16×16×512, and f4: 16×16×512.
[0050] Step 207: upsample f1: 16×16×512 by bilinear interpolation with a scaling ratio of 16 to obtain f1: 256×256×512; the calculation process is as follows: first, the original image coordinates (x, y) of each target pixel are calculated by the scaling ratio:
[0051]
[0052] Where x' and y' are the target positions. Next, find the four adjacent points of (x, y), with coordinates as follows: upper left corner ([x], [y]); upper right corner ([x]+1, [y]); lower left corner ([x], [y]+1); lower right corner ([x]+1, [y]+1). Next, calculate the weights Δx=x-[x]; Δy=y-[y]; finally, input the four adjacent points and weights into the bilinear interpolation formula:
[0053] I'(x',y',c)=(1-Δx)(1-Δy)I([x],[y],c)+Δx(1-Δy)I([x]+1,[y],c)
[0054] +(1-Δx)ΔyI([x],[y]+1,c)+ΔxΔyI([x]+1,[y]+1,c)
[0055] Each pixel position can be calculated, and the interpolation calculation is repeated to obtain the upsampled f1: 256×256×512. Similarly, f2: 16×16×512 is upsampled by bilinear interpolation with a scaling ratio of 12 to obtain f2: 192×192×512; f3: 16×16×512 is upsampled by bilinear interpolation with a scaling ratio of 8 to obtain f3: 128×128×512; f4: 16×16×512 is upsampled by bilinear interpolation with a scaling ratio of 4 to obtain f4: 64×64×512.
[0056] Step 208: The upsampled feature map f1: 256×256×512 is passed through the convolution layer 5 with a convolution kernel of 1×1×512×64, a stride of 1, and a padding of 0, and the output is f1: 256×256×64; similarly, f2: 192×192×512 is passed through the convolution layer 6 with a convolution kernel of 1×1×512×128, a stride of 1, and a padding of 0, and the output is 192×192×128; f3: 128×128×512 is passed through the convolution layer 7 with a convolution kernel of 1×1×512×256, a stride of 1, and a padding of 0, and f3: 128×128×128 is obtained.
[0057] Step 3: Input the infrared image and visible light image into CNN for preliminary fusion and extraction.
[0058] Step 301: Input the visible light image Input Stream 1 into the convolution layer 1 with a convolution kernel of 1×1×3×64, a stride s of 1, and a padding P of 0. From step 205, we can see that when the convolution kernel size is 1×1, the stride s is 1, and the padding P is 0, the size remains unchanged, the number of channels changes from 3 to 64, and the output is 256×256×64. Next, through the formula:
[0059]
[0060] Perform 2×2 average pooling, the size becomes 1 / 2 of the previous one, and the output f1(1): 128×128×64.
[0061] Step 302: Pass f1(1) through convolution layer 2 with a convolution kernel of 1×1×64×128 and a stride s of 1, and padding P to 0, to obtain 128×128×128, and then pass it through a 2×2 pooling layer to obtain f2(1): 64×64×128; pass it through convolution layer 3 with a convolution kernel of 1×1×128×256 and a stride s of 1, and padding P to 0 and a 2×2 pooling layer to obtain f3(1): 32×32×256; pass it through convolution layer 4 with a convolution kernel of 1×1×256×512 and a stride s of 1, and padding P to 0 and a 2×2 pooling layer to obtain f4(1): 16×16×512.
[0062] Step 303: The results after each pooling layer are numbered as f1(1), f2(1), f3(1), and f4(1) in sequence.
[0063] Step 304: Repeat the convolution layer and pooling layer of step 208 above for the infrared image Input Stream 2 to obtain f1(2), f2(2), f3(2), and f4(2).
[0064] Step 305: Add f1(1), f2(1), f3(1), f4(1) from the visible light image and f1(2), f2(2), f3(2), f4(2) from the infrared image element by element, and then deeply fuse the features of the two modalities through convolution layers with a convolution kernel of 1×1, a stride s of 1, and a padding P of 0. Obtain f1': 256×256×64, f2': 192×192×128, f3': 128×128×256, f4': 64×64×512.
[0065] Step 4: Feature fusion.
[0066] Step 401: f4: 64×64×512 from Transformer is passed through the CBAM module. First, the input feature map f4: 64×64×512 is subjected to global maximum pooling and global average pooling based on height and width respectively to obtain two 1×1×512 feature maps, and then they are respectively sent to a neural network with two shared MLP layers. The number of neurons in the first layer is C / r (r is the reduction rate), the activation function is Relu, and the output is 1×1×256. The number of neurons in the second layer is C, and the output is 1×1×512. This two-layer neural network is shared. Then, the output features are added element by element, and then activated by sigmoid to generate the final channel weight feature Wc. Finally, the channel weight feature and the input feature map F are multiplied element by element to generate F'.
[0067] Step 402: Take F' as the input of spatial weights, first perform global maximum pooling and global average pooling based on channels to obtain two 64×64×1 feature maps, then concatenate the two feature maps through channels to obtain 64×64×2, and then perform a 7×7 convolution operation to reduce the dimension to H×W×1. Then generate the spatial weight feature Ws through sigmoid. Finally, multiply Ws by the input feature map to obtain the final generated feature.
[0068] Step 403: Similarly, f1', f2', f3', f4' and f1, f2, f3 are also subjected to spatial weighting and channel weighting by the CBAM module respectively.
[0069] Step 404: Add f4: 64×64×512 and f4': 64×64×512 after the CBAM module element by element, and mark the fused feature map as F(2)4. Combine the information of the two modalities. Output F(2)4: 64×64×512.
[0070] Step 405: F(2)4: 64×64×512 is fused through the convolution layer 1 with a convolution kernel W of size 1×1×512×256, a stride of 1, and a padding of 0 to generate a feature map H(2)3 of size 64×64×256. A 2x2 average pooling operation is performed on H(2)3: 64×64×256 to output F(2)3: 32×32×256.
[0071] Step 406: perform 4×4 average pooling on f3: 128×128×256 and f3': 128×128×256 respectively, and downsample to f3": 32×32×256.
[0072]
[0073] Among them, f3 is the input feature map, and f3' is the downsampled feature map. 4×4 average pooling will reduce each 4×4 area in the input feature map to one pixel, and the spatial size (height and width) of the output feature map becomes 1 / 4 of the original, while the number of channels remains unchanged.
[0074] Step 407: Add F(2)3: 32×32×256 and the downsampled f3″: 32×32×256 element by element, and output H(2)2: 32×32×256.
[0075] Step 408: Pass H(2)2: 32×32×256 through convolution layer 2 with a convolution kernel W of size 1×1×256×128, stride 1, and padding 0 to obtain H(2)2': 32×32×128, and then perform 2×2 average pooling on H(2)2': 32×32×128 to output F(2)2: 16×16×128.
[0076] Step 409: Perform 12×12 average pooling on f2: 192×192×128 and f2': 192×192×128, and downsample to output f2": 16×16×128.
[0077]
[0078] Step 410: Add F(2)2: 16×16×128 and the downsampled f2”: 16×16×128 element by element to obtain H(2)1: 16×16×128. Then pass the deconvolution layer 1 with a convolution kernel size of 16×16×128×64, a stride s of 16, and a padding P of 0, and pass the deconvolution layer formula:
[0079] H out =(H in -1)×s+K-2P
[0080] It can be calculated that Hout =(16-1)×16+16-2×0=256, the number of channels changes from 128 to 64, and the output H(2)1': 256×256×64.
[0081] Step 411: perform element-by-element addition and fusion of H(2)1′, f1: 256×256×64 and f1′: 256×256×64, and output H1: 256×256×64.
Claims
1. A multi-level cross-modal collaborative feature encoder structure, comprising the following steps: Step 1: training set preprocessing; Step 101: Normalize the infrared image and visible light image datasets to a uniform size of 256×256×3. Step 2: Input the infrared image and visible light image into Transformer for preliminary fusion and extraction; Step 201: divide the visible light image Input Stream 1 and the infrared image Input Stream 2 into 16×16 patches respectively. Step 202: Flatten the patches of the two modalities in the order of pixel values and reduce the dimensions to 512 dimensions through linear dimensionality reduction. Step 203: Input the patch into the Transformer encoder. Step 204: The Transformer encoder extracts the feature map f1 (infrared) from the first layer; the feature map f2 (infrared) from the third layer; the feature map f3 (infrared) from the sixth layer; and the feature map f4 (infrared) from the twelfth layer. Step 205: The Transformer encoder extracts the feature map f1 (visible light) from the first layer; the feature map f2 (visible light) from the third layer; the feature map f3 (visible light) from the sixth layer; and the feature map f4 (visible light) from the twelfth layer. Step 206: Add f1(infrared) and f1(visible light) element by element, and then deeply fuse them through convolution layer 1 to obtain f1. Step 207: Repeat the above operation to obtain f2, f3, and f4. Step 208: up-sample f1, f2, f3, and f4 respectively by bilinear interpolation. Step 209: The feature maps f1, f2, f3, and f4 obtained after upsampling are respectively changed through convolutional layers to obtain a suitable number of channels. Step 3: Input the infrared image and visible light image into CNN for preliminary fusion and extraction; Step 301: The visible light image Input Stream 1 is sequentially passed through convolution layer 1 and pooling layer, convolution layer 2 and pooling layer, convolution layer 3 and pooling layer and convolution layer 4, pooling layer. Step 302: The results after each pooling layer are numbered as f1(1), f2(1), f3(1), and f4(1) in sequence. Step 303: Repeat the above operation on the infrared image Input Stream 2 to obtain f1(2), f2(2), f3(2), and f4(2). Step 304: Combine f1(1), f2(1), f3(1), f4(1) from the visible light image and f1(2), f2(2), f3(2) and f4(2) are added element by element in turn, and then the features of the two modes are fused through the convolutional layer to obtain f1', f2', f3', and f4'. Step 305: Perform bilinear interpolation upsampling on f1', f2', f3', and f4' to match the size of the feature maps f1, f2, f3, and f4 obtained by the Transformer. Step 4: The features extracted by the two networks are fused through the CBAM module; Step 401: f4 (64×64×512) from Transformer is subjected to channel weighting adjustment by the CBAM module to obtain f4(1). Step 402: f4(1) is used as input and adjusted by the spatial attention weighting of the CBAM module, and output is f4. Step 403: Similarly, f1', f2', f3', f4' and f1, f2, f3 are respectively subjected to spatial weighting and channel weighting by the CBAM module. Step 404: Add f4 and f4' element by element and mark the fused feature map F(2)4. Step 405: F(2)4 is convolved to obtain H(2)3, which is then subjected to 2×2 average pooling to output F(2)3. Step 406: Perform 4×4 average pooling on f3 and f3′, respectively, and downsample to obtain f3(1) and f3′(1). Step 407: Add F(2)3 to the downsampled f3(1) and f3'(1) element by element to obtain H(2)2. Step 408: Perform 1×1 convolution on the obtained H(2)2 to obtain H(2)2', and then perform 2×2 average pooling to obtain F(2)2. Step 409: Perform 12×12 average pooling on f2 and f2′, respectively, to obtain f2(1) and f2′(1). Step 410: Add F(2)2 to the downsampled f2(1) and f2'(1) element by element to obtain H(2)1, and pass H(2)1 through deconvolution layer 1 to obtain H(2)1'. Step 411: perform element-by-element addition and fusion of H(2)1', f1 and f1', and output H1.
2. A multi-level cross-modal collaborative feature encoder structure according to claim 1, characterized in that: In the step 204, after the ptach from the two modal slices is input into the Transformer encoder, the first layer, the third layer, the sixth layer and the twelfth layer are extracted respectively. Such feature selection can enhance the fusion of multimodal features. The captured information has different scales and levels of abstraction, and enhances the multi-scale feature fusion capability.
3. A multi-level cross-modal collaborative feature encoder structure according to claim 1, characterized in that: In the step 205, the two modalities f1, f2, f3, and f4 output by the Transformer encoder are sequentially fused through element-by-element addition and convolutional layers. The fused part of the features from the two modal images is added to the Transformer feature extraction. The fused feature map contains richer information, which can improve the feature representation capability and capture more details.
4. The multi-level cross-modal collaborative feature encoder structure according to claim 1, characterized in that: In step 304, f1(1), f2(1), f3(1), f4(1) from the visible light image and f1(2), f2(2), f3(2), f4(2) from the infrared image are added element by element in sequence, and then the features of the two modalities are fused through convolutional layers to obtain f1', f2', f3', f4', and the features of the infrared image and the visible light image are initially fused during the CNN extraction process.
5. The multi-level cross-modal collaborative feature encoder structure according to claim 1, characterized in that: In step 401, the features of the two modalities extracted from Transformer and CNN are subjected to spatial attention weighting and channel attention weighting by the CBAM module, thereby enhancing the model's attention to different channels and spatial positions in the feature map, and paying more attention to features that are important to the task. Feature fusion after CBAM has the advantage of being more comprehensive and accurate.
6. A multi-level cross-modal collaborative feature encoder structure according to claim 1, characterized in that: In the steps 404 to 411, the features of the two modalities extracted by Transformer and the features of the two modalities extracted by CNN are deeply fused, and the features of the infrared image and the visible light image are fused again by taking advantage of the fact that Transformer is good at processing long-distance dependencies and global information and can capture global features and complex contextual relationships in the image, and CNN is good at capturing local features and patterns and can effectively extract local information and details in the image.
Citation Information
Patent Citations
Infrared and visible light image fusion method combining Transform and CNN double encoders
CN117314808A
Infrared and visible light image fusion method and device
CN117830118A
RGB-D semantic segmentation method based on double-channel transformer cross fusion
CN118038060A
