A collaborative optimization method for image fusion and semantic segmentation

Abstract

The present application belongs to the technical field of image processing, and particularly relates to a collaborative optimization method for image fusion and semantic segmentation, comprising constructing an encoder, a decoder for an image fusion task, and a decoder for a semantic segmentation task, all the decoders sharing data output by the encoder, in a training process, output of the decoder for the image fusion task being spliced with output of the decoder for the semantic segmentation task and a label corresponding to the output, and consistency of the two spliced vectors being used as one of loss functions to optimize network parameters of the decoders and all the encoders. The present application utilizes a deep learning-based framework, can effectively learn multi-dimensional features from different modal data, and through mining of inherent consistency between tasks, realizes collaborative optimization of multi-task, so that better image fusion and semantic segmentation effects are achieved.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, specifically relating to a collaborative optimization method for image fusion and semantic segmentation. Background Technology

[0002] In the field of autonomous driving, accurate and robust scene analysis is crucial. However, in complex environments (such as inclement weather or nighttime driving), relying solely on visible light sensors may struggle to accurately identify targets. In contrast, infrared sensors are unaffected by these factors and can provide clear target information in low-light and inclement weather conditions, although their spatial resolution is lower. Therefore, infrared and visible light multimodal fusion has become an effective method to enhance scene understanding and target detection performance.

[0003] In recent years, advancements in deep learning have significantly propelled the development of the multimodal field, particularly with Convolutional Neural Networks (CNNs) and Transformers becoming mainstream. However, due to the limitations of the local receptive fields in CNNs and the high computational complexity introduced by Transformers, building a balanced model remains challenging. This makes it difficult to fully exploit the potential of CNNs and Transformers in practical applications, resulting in less than satisfactory performance.

[0004] Meanwhile, the introduction of the Mamba architecture has brought innovation to the field. Mamba excels in efficiently capturing global contextual information and is renowned for its computational efficiency based on state-space models (SSM). Vision Mamba and VMamba extend this architecture to computer vision tasks, further improving performance by enhancing the unidirectional scanning mechanism. Given Mamba's strength in global context modeling, it is well-suited for image processing tasks. However, most researchers typically train Mamba modules from scratch or directly apply pre-trained Vision Mamba modules. These methods rarely consider the efficient fusion of local and global information in images, limiting their full utilization of the features provided by pre-trained CNN models. Finally, most existing methods treat image fusion and semantic segmentation as two separate problems, ignoring their inherent connection. Although some studies have attempted to optimize image fusion and semantic segmentation tasks through cascading optimization, a lack of effective strategies to fully leverage the inherent consistency between the two has prevented maximizing their potential. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention provides a collaborative optimization method for image fusion and semantic segmentation. This method constructs an encoder, a decoder for image fusion, and a decoder for semantic segmentation. All decoders share the data output by the encoder. During training, the output of the decoder for image fusion is concatenated with the output of the decoder for semantic segmentation and the corresponding label. The consistency between the two concatenated vectors is used as one of the loss functions to optimize the network parameters of the decoder and all encoders.

[0006] Furthermore, the encoder includes a basic feature encoder, a global feature enhancement module, a global feature fusion module, a local feature enhancement module, a local feature fusion module, and a spatial recalibration and alignment module.

[0007] Furthermore, the basic feature encoder consists of multiple cascaded feature extraction layers, and the image input to the basic feature encoder is residually connected to the output of the last feature extraction layer. The feature extraction layer sequentially processes the input image using depthwise separable convolution, fully connected layers, fully connected layers, and GELU activation function layers to obtain the basic feature map.

[0008] Furthermore, the global feature enhancement module processes the basic feature map to obtain a global enhanced feature map. Specifically, this includes processing the input features sequentially using a linear layer, a depthwise separable convolution module, a selective state space module, a normalization module, and a linear layer, and then performing a residual connection between the processed features and the input features to obtain the global enhanced feature map.

[0009] Furthermore, the RGB image and infrared image are sequentially processed through a basic feature encoder and a global feature enhancement module to obtain corresponding global enhanced feature maps. The two global enhanced feature maps are then fused through a global feature fusion module to obtain a global fused feature map, specifically including:

[0010] The global enhancement feature maps of RGB images and infrared images are processed using linear layers with shared parameters and depthwise separable convolutions, respectively.

[0011] The selective state space module processes the observation matrices of RGB and infrared images, and then performs feature selection corresponding to the RGB images using the observation matrix of the infrared images, and feature selection corresponding to the observation matrix using the observation matrix of the RGB images.

[0012] After feature selection, the feature map is processed by the normalization module and the linear layer in sequence, and the processed features are residually connected with the corresponding global augmentation feature map.

[0013] The feature map obtained by residual concatenation is processed by global average pooling and sigmoid activation function to generate an attention map;

[0014] The attention map and the feature map after concatenation of residuals are multiplied element-wise. The resulting feature map is then added to the feature map after concatenation of residuals to obtain the global fused feature map.

[0015] Furthermore, the local feature enhancement module processes the basic feature map to obtain local enhanced features. Specifically, this includes: adjusting the number of channels in the basic feature map using a 1×1 convolution kernel, then using 3×3, 5×5, and 7×7 convolution kernels for multi-scale feature extraction, concatenating the multi-scale features together, performing global average pooling, adjusting the number of channels through convolution, generating spatial information weights using the Sigmoid activation function, and then multiplying these spatial information weights element-wise with the feature map input to the local feature enhancement module to obtain the local enhanced features.

[0016] Furthermore, the RGB image and infrared image are sequentially processed by a basic feature encoder and a local feature enhancement module to obtain corresponding local enhancement features. The local enhancement features of the two images are then fused by a local feature fusion module to obtain local fused features, specifically including:

[0017] The local enhancement features of RGB and infrared images are stitched together and then processed using depthwise separable convolution.

[0018] The feature map is divided into two parts along the channel dimension using the chunk operation. One part is processed by the Sigmoid activation function to generate a weight map, which is then multiplied element-wise with the other part of the feature map to finally obtain the local fused features.

[0019] Furthermore, the spatial recalibration alignment module is used to align the global enhanced feature map output by the global feature fusion module with the local enhanced feature map output by the local feature fusion module to obtain fused features. The alignment process includes:

[0020] The global enhanced feature map and the local enhanced feature map are added to obtain the global-local features, which are then integrated and the channels are adjusted through convolutional layers and normalization layers.

[0021] The feature map is pooled along the height and width directions using two pooling operations of different sizes, one of the pooled feature maps is shaped, and then the two pooled feature maps are concatenated together.

[0022] The concatenated feature maps are processed by 1×1 convolution and Sigmoid activation function, and the resulting feature maps are divided into two feature vectors based on the height and width of global-local features.

[0023] The two feature vectors are processed by 1×1 convolution and sigmoid activation function respectively, and then multiplied element-wise and fused to obtain the attention map;

[0024] The resulting attention map is multiplied element-wise with the global-local features to obtain the fused features.

[0025] Furthermore, the decoder includes a semantic segmentation head and an image fusion head, wherein:

[0026] The semantic segmentation head consists of multiple cascaded MAVSSBlocks. Each MAVSSBlock processes the input feature map by: normalizing the feature map; dividing the feature map into two parts through a chunking operation; the first part is processed sequentially through a linear layer, a depthwise separable convolution, a 2D selective scanning module, and a normalization layer to obtain an enhanced feature map; the second part is processed through channel attention and spatial attention modules to generate corresponding attention maps; and the enhanced feature map and the attention map are multiplied element-wise to obtain the output of the MAVSSBlock. In practice, the output of the cascaded MAVSSBlocks is passed through an upsampling layer and a multilayer perceptron to output the segmentation result.

[0027] The image fusion head processes data as follows: RGB and infrared images are stitched together, and the original features are extracted through the feature correction and calibration module. Images at multiple scales output by the shared encoder are processed by the dynamic region feature extraction module to obtain multiple feature maps. The multiple feature maps are upsampled to the size of the original feature map and then added and fused. After that, they are stitched together with the original features. The stitching result is input into the image reconstruction layer to obtain the fusion result. The multiple scale features are feature maps of different sizes obtained during the downsampling process of the basic encoder. In this invention, the image reconstruction layer consists of multiple convolutional layers and the ReLU activation function.

[0028] Preferably, the feature correction and calibration module of the present invention is used to correct abstract features in feature maps caused by excessive image processing. Specifically, it obtains the original image features by stitching together RGB and infrared images and inputting them into the module. As a preferred implementation, the dynamic region feature extraction module of the present invention processes images of multiple scales output by the shared encoder. In this module, the number of regions of interest is dynamically assigned to each feature map. The feature map input to the feature correction and calibration module is subjected to convolution operations at different scales based on the number of regions, and then sequentially passed through a 3×3 convolution, a normalization layer, a ReLU activation function, a 1×1 convolution, and a sigmoid activation function to obtain the total weighted fusion feature. The feature map input to the dynamic region feature extraction module is subjected to convolution operations at different scales based on the number of regions, and then the weight coefficients of different convolutions are calculated through two linear layers and a ReLU activation function, thereby dynamically determining the weight coefficients that each region needs to be focused on. A Softmax function is then used to normalize the coefficients. After obtaining the normalized coefficients, the extracted features corresponding to each region are multiplied by the corresponding feature weights, and all regions are summed to obtain the total weighted fusion feature.

[0029] Furthermore, during training, the predicted labels and ground truth labels of semantic segmentation are concatenated with the image fusion results. The concatenated results are then upscaled through multiple cascaded convolutional layers to obtain high-dimensional vectors. The distance between the two high-dimensional vectors is calculated as the consistency loss.

[0030] This invention addresses the challenges of extracting effective features through multimodal fusion and the difficulties in joint task optimization training. It proposes an image fusion and semantic segmentation method based on multimodal fusion and collaborative task optimization. By designing a multimodal fusion network to aggregate feature information from different modalities, and by improving the intrinsic consistency between image fusion and semantic segmentation tasks through collaborative optimization, this invention achieves the best of both worlds in the optimization of multiple tasks. Attached Figure Description

[0031] Figure 1 This is a schematic diagram of a collaborative optimization method for image fusion and semantic segmentation according to the present invention.

[0032] Figure 2 This is a schematic diagram of the global feature enhancement module structure used in this invention;

[0033] Figure 3 This is a schematic diagram of the local feature enhancement module structure used in this invention;

[0034] Figure 4 This is a schematic diagram of the local feature fusion module structure used in this invention;

[0035] Figure 5This is a schematic diagram of the global feature fusion module structure used in this invention;

[0036] Figure 6 This is a schematic diagram of the spatial recalibration and alignment module structure used in this invention;

[0037] Figure 7 This is a schematic diagram of the MAVSSBlock structure used in this invention. Detailed Implementation

[0038] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0039] This invention provides a collaborative optimization method for image fusion and semantic segmentation, including constructing an encoder, a decoder for image fusion, and a decoder for semantic segmentation. All decoders share the data output by the encoder. During training, the output of the decoder for image fusion is concatenated with the output of the decoder for semantic segmentation and the corresponding label. The consistency of the two concatenated vectors is used as one of the loss functions to optimize the network parameters of the decoder and all encoders.

[0040] This embodiment provides a specific implementation of a collaborative optimization method for image fusion and semantic segmentation, which includes the following steps:

[0041] Step 1: Prepare a multimodal image dataset and preprocess the image dataset. The dataset includes infrared-visible light images, that is, the same scene is captured by visible light sensors and infrared sensors.

[0042] Step 2: Construct a shared encoder for extracting multimodal features. The shared encoder consists of a basic feature encoder based on ConvNext, a global feature enhancement module based on Mamba self-attention mechanism, a global feature fusion module based on Mamba cross-attention mechanism, a local feature enhancement module based on multi-kernel convolution, a local feature fusion module based on pooling and depthwise separable convolution, and a fusion module based on coordinated attention mechanism.

[0043] Step 3: Construct specific decoders based on different task requirements. The specific decoder processes the data output by the weight-sharing encoder according to the constructed decoder. The specific decoder consists of a semantic segmentation head based on Mamba and a hybrid attention mechanism and an image fusion head based on pixel feature extraction and CNN.

[0044] Step 4: Achieve collaborative optimization by mining the inherent consistency of different tasks. The method of achieving collaborative optimization is to concatenate the corresponding prediction results and labels and map them to a high-dimensional space, and finally calculate the distance loss between different tasks through an appropriate distance loss function.

[0045] like Figure 1 In this embodiment, visible light images (RGB images) and infrared images are simultaneously input into the basic encoder to obtain basic features of the visible light image and basic features of the infrared image, respectively. The basic features of the visible light image and the basic features of the infrared image are then subjected to local feature enhancement and global feature enhancement, respectively. Then, the two local feature enhancements are fused into local features, and the two global feature enhancements are fused into global features. The fused results are spatially recalibrated and aligned, and finally input into the semantic segmentation head and the image fusion head for downstream tasks. Based on the prediction results of the downstream tasks, the network parameters are optimized during backpropagation using task consistency regularization as one of the loss functions.

[0046] As an optional implementation, this embodiment provides a specific implementation for preprocessing an image dataset, including:

[0047] First, load the RGB image and the infrared image, perform channel order conversion on the RGB image, and expand the infrared image into a pseudo-RGB image;

[0048] Next, during the training phase, data augmentation methods are used, including random horizontal flipping and random scaling, and the images are normalized.

[0049] Then, the system randomly crops the image according to the target size, fills the excess part and records the edge information. Finally, the image format is converted to CHW format and a binary mask is generated to mark the effective area and the filled area.

[0050] Finally, the verification phase is simplified, involving only normalization, channel conversion, and tensor conversion.

[0051] In this embodiment, the basic feature encoder is used to extract the corresponding basic feature map from the RGB image or infrared image, specifically including the following steps:

[0052] The input image is first subjected to preliminary size compression through a shared downsampling layer. This stage includes convolution operations, which aim to initially map the number of channels of the input image and reduce spatial dimensions.

[0053] Next, layer normalization is used to ensure the stability of subsequent calculations;

[0054] Then, the RGB image and the infrared image are respectively input into the same convolution and normalization operations. Each stage contains multiple residual blocks, where the image first undergoes depthwise separable convolution to extract features from each channel independently and then normalizes them.

[0055] Subsequently, two fully connected layers are used to further capture the relationships between features, and a nonlinear transformation is introduced through the GELU activation function;

[0056] Finally, the residual blocks use residual connections to add the input and output, and introduce a dropout strategy to discard some paths, thereby obtaining richer basic features;

[0057] In the above operations, specifically through the design of convolutional neural network modules and Transformer framework, preliminary features can be extracted better while reducing computational complexity.

[0058] In this embodiment, the global feature enhancement module based on the Mamba self-attention mechanism takes the output of the ConvNext-based basic encoder as input, such as... Figure 2 The specific processing procedure includes the following steps:

[0059] First, image features of different modalities are extracted by passing them through linear layers and depthwise separable convolutional modules respectively.

[0060] Subsequently, the corresponding A, B, C, D, and Δ parameters are obtained in the Selective Scan Space State Sequential Model (S6 module) (these parameters are generated by the linear layer, embedding layer, etc. of the S6 module. Those skilled in the art can obtain these parameters according to any existing S6 module. The Selective Scan Space State Sequential Model used in this embodiment is a mature feature processing module in the prior art, and this embodiment will not elaborate on it). After feature extraction by the Selective Scan Space Model, normalization processing is performed by the normalization module and the linear layer.

[0061] Finally, residual connections are used to obtain self-modal enhancement features.

[0062] In the above operations, the self-attention mechanism can effectively realize the global receptive field. However, since the traditional dot product operation has too high computational complexity requirements, this invention adopts the selection of state space to optimize the implementation of the self-attention mechanism.

[0063] In this embodiment, the global feature fusion module based on the Mamba cross-attention mechanism receives the output of the global feature enhancement module based on the Mamba self-attention mechanism as its output, such as... Figure 5The specific processing procedure includes the following steps:

[0064] First, the self-modal enhancement features of different modalities are processed through a shared linear layer and depthwise separable convolution (DWConv) and then fed into the S6 module (Selective Scan Space State Sequential Model, or S6 Blocks for short) to generate different A, B, C, D and Δ.

[0065] Next, the C values ​​of the two features are swapped, and further feature extraction is performed through the 2D Selective ScanModule (SS2D module).

[0066] Subsequently, after processing by Layer Normalization (LN) and linear layers, residual feature enhancement is performed by combining the initial self-modal enhancement features, that is, the input feature map is summed with the output of the last linear layer at the element-wise level.

[0067] Finally, the features of different modalities are concatenated, and an attention map is generated by global average pooling and sigmoid activation function. This attention map is multiplied element-wise with the original feature map, and the concatenated features are added to obtain the final global feature fusion output. In this implementation, the cross-attention mechanism has a significant effect on the extraction of global context features of different modalities. Similarly, this embodiment also uses the selection of state space sequence for optimization, thereby reducing memory consumption.

[0068] In this embodiment, the local feature enhancement module based on multi-kernel convolution receives the output of the basic encoder based on ConvNext as input, such as... Figure 3 The specific processing procedure includes the following steps:

[0069] First, the number of channels in the input feature maps of different modalities is adjusted by 1×1 convolution (Conv 1×1);

[0070] Subsequently, the feature maps were subjected to multi-scale feature extraction through 3×3 convolution (Conv 3×3), 5×5 convolution (Conv 5×5), and 7×7 convolution (Conv 7×7), respectively. The feature maps at different scales were summed to obtain preliminary local features.

[0071] Next, spatial scale feature maps are obtained through global average pooling, and their number of channels is adjusted through depthwise separable convolution.

[0072] Finally, the sigmoid activation function is used to generate spatial information weights, and these weights are fused with the initial feature map using matrix multiplication to enhance the spatial information.

[0073] In the above operations, convolution is used to extract the local receptive field of features, while multi-kernel convolution helps to obtain information from scale rather than scale, thus obtaining richer feature effects.

[0074] In this embodiment, the local feature fusion module based on pooling and depthwise separable convolution receives the output of the local feature enhancement module based on multi-kernel convolution as input, such as... Figure 4 The specific processing procedure includes the following steps:

[0075] First, the local feature enhancement maps from different modalities are tensor concatenated, and then the features are further fused through depthwise separable convolution.

[0076] Next, the concatenated feature map is divided into two parts along the channel dimension using a chunk operation. One part is processed by the Sigmoid activation function to generate a weight map, which is then multiplied with the other part of the feature map by the Hadamard product to finally obtain the fused local feature map.

[0077] In the above operations, depthwise separable convolution can efficiently aggregate and fuse features, and integrate features from different channels through attention maps, thereby improving the model's channel modeling capabilities.

[0078] In this embodiment, the spatial recalibration alignment module based on the coordinated attention mechanism receives the outputs of the global feature fusion module based on the Mamba cross-attention mechanism and the local feature fusion module based on pooling and depthwise separable convolution as input, such as... Figure 6 The specific processing procedure includes the following steps:

[0079] First, the feature maps from different fusion modules are summed, and then further feature integration and channel number adjustment are performed through convolutional layers and normalization layers;

[0080] Next, two pooling operations of different sizes are applied to pool the feature maps along the height and width directions respectively, generating two pooled feature maps;

[0081] Subsequently, the corresponding attention map is generated through 1×1 convolution and the Sigmoid activation function;

[0082] Finally, these attention maps are multiplied element-wise with the initial fused features to obtain the fused feature map.

[0083] In this embodiment, based on the image fusion and semantic segmentation method optimized by multimodal fusion and collaborative tasks, a decoder based on different task requirements is constructed. The constructed decoder then processes the data output by the shared encoder, including:

[0084] A semantic segmentation head based on Mamba and a hybrid attention mechanism is constructed. This segmentation head receives multi-scale fused feature maps from the aforementioned spatial recalibration and alignment module and generates the final semantic segmentation result. The data processing of this segmentation head includes: first, the input feature maps are processed through multiple MAVSSBlocks in ascending order, such as... Figure 7 In each block, the input feature map is first processed by a normalization layer, and then divided into two parts by a chunk operation. First, it is further processed by a fully connected layer and a depthwise separable convolution to extract features, and a global context is established by a 2D selective scanning module. Finally, the feature enhancement result is obtained by normalization. Second, the corresponding attention map is generated by the channel attention and spatial attention modules (in the process of generating the attention map, the input feature map is processed by average pooling and max pooling respectively. The features after the two pooling processes are added and processed by the activation function. Then, it is divided into two branches and processed by average pooling and max pooling again. The two feature maps after the second pooling process are passed). The two feature maps are then multiplied element-wise to obtain the enhanced result. After each feature map is processed by the corresponding block, it is upsampled to the same size as the previous level feature map. Then, it is summed and sent to the next level block for processing. Finally, after a total upsampling, the final semantic segmentation result is obtained by a multilayer perceptron.

[0085] Since CNN encoders offer more channels, this invention adds another branch to the original VSSBlock, improving its multi-channel integration capabilities through channel attention and spatial attention modules, and using a design similar to FPN to integrate features. Figure 1 By upsampling and fusing step by step, the final result is obtained. Therefore, an image fusion head based on pixel feature extraction and CNN is constructed. It receives multi-scale fusion feature maps from the above-mentioned spatial recalibration and alignment module to obtain the final image fusion result. Specifically, the input feature map is first processed by a convolutional neural network for preliminary feature extraction and upsampled to the initial image size. Then, the original multimodal image is processed by a pixel feature extraction module for feature extraction and then concatenated with the corresponding feature map. Finally, the concatenated result is processed by multiple convolutions and the ReLU activation function to reconstruct the image and obtain the image fusion result.

[0086] Since image fusion is highly dependent on the features of the original image, this invention uses the initial input as a reference, obtains the rich visual features of the original image through a pixel feature extraction module, and utilizes the effectiveness of CNN in low-level tasks to obtain the corresponding fused image result.

[0087] This image fusion and semantic segmentation method, based on multimodal fusion and collaborative task optimization, achieves collaborative optimization by mining the inherent consistency of different tasks. Specifically, it involves concatenating the segmentation result and the ground truth label output by the semantic segmentation head with the image fusion result along the channel dimension. In this embodiment, since the image fusion lacks corresponding ground truth labels, the image fusion result is directly concatenated with the ground truth label of the image segmentation result. Then, features of the concatenated image are extracted through multiple convolutional layers to obtain the corresponding high-dimensional latent representation. Subsequently, the similarity between the two high-dimensional latent representation vectors is calculated along the channel dimension using a cosine distance function. A new distance loss function is constructed in the above manner and added to the original loss function to adjust the optimization direction of the training.

[0088] This implementation considers the potential inherent consistency between different image processing tasks. Therefore, the final results are concatenated and their high-dimensional features are extracted. The distance loss between these high-dimensional features is used as an aspect of optimization in this invention, thereby enabling implicit alignment between multiple tasks and achieving joint optimization.

[0089] It should also be noted that the encoder and decoder of this invention adopt a U-shaped network structure, and the corresponding upsampling and downsampling adopt skip connections.

[0090] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for collaborative optimization for image fusion and semantic segmentation, constructing an encoder, a decoder for an image fusion task, and a decoder for a semantic segmentation task, characterized in that, All decoders share the data output by the encoder, and in the training process, the output of the decoder for the image fusion task is spliced with the output of the decoder for the semantic segmentation task and the label corresponding to the output, respectively, and the consistency of the two spliced vectors is used as one of the loss functions to optimize the network parameters of the decoder and all encoders; The encoder comprises a basic feature encoder, a global feature enhancement module, a global feature fusion module, a local feature enhancement module, a local feature fusion module and a spatial recalibration alignment module. The basic feature encoder is composed of a plurality of cascaded feature extraction layers, and the image input into the basic feature encoder is connected in residual with the output of the last feature extraction layer. The feature extraction layers sequentially process the input image by using a depth separable convolution, a full connection layer, a full connection layer and a GELU activation function layer to obtain a basic feature map. The global feature enhancement module processes the basic feature map to obtain a global enhanced feature map. The RGB image and the infrared image are sequentially input into the basic feature encoder and the global feature enhancement module to obtain corresponding global enhanced feature maps. The two global enhanced feature maps are fused by the global feature fusion module to obtain a global fusion feature map. The local feature enhancement module processes the basic feature map to obtain a local enhanced feature. The RGB image and the infrared image are sequentially input into the basic feature encoder and the local feature enhancement module to obtain corresponding local enhanced features. The local enhanced features of the two images are fused by the local feature fusion module to obtain a local fusion feature. The spatial recalibration alignment module is used to align the global enhanced feature map output by the global feature fusion module and the local enhanced feature map output by the local feature fusion module to obtain a fusion feature. The decoder comprises a semantic segmentation head and an image fusion head. The semantic segmentation head comprises a plurality of MAVSSBlocks connected in cascade. The processing of each MAVSSBlock on the input feature map comprises: normalizing the feature map, dividing the feature map into two parts by chunk operation, processing the first part by a linear layer, a depth separable convolution, a two-dimensional selective scanning module and a normalization layer in sequence to obtain an enhanced feature map, and processing the second part by a channel attention and a spatial attention module to generate a corresponding attention map. The enhanced feature map and the attention map are multiplied element by element to obtain the output of the MAVSSBlock. The processing of the image fusion head on data is: splicing the RGB image and the infrared image together, then extracting the original feature by a feature correction calibration module, processing a plurality of scales of images output by the shared encoder by a dynamic region feature extraction module to obtain a plurality of feature maps, upsampling the plurality of feature maps to the size of the original feature map and then adding and fusing, then splicing the fusion result with the original feature, and inputting the spliced result into an image reconstruction layer to obtain a fusion result.

2. The method of claim 1, wherein, The global feature enhancement module processes the basic feature map to obtain a global enhanced feature map, specifically including: sequentially using a linear layer, a depth separable convolution module, a selective state space module, a normalization module, and a linear layer to process the input feature, and then performing residual connection on the processed feature and the input feature to obtain the global enhanced feature map.

3. The method of claim 1 or 2, wherein, The RGB image and the infrared image are sequentially input into the basic feature encoder and the global feature enhancement module respectively to obtain corresponding global enhanced feature maps, and the two global enhanced feature maps are fused by the global feature fusion module to obtain a global fusion feature map, specifically including: The global enhanced feature map of the RGB image and the global enhanced feature map of the infrared image are processed by using a linear layer and a depth separable convolution with shared parameters respectively; The observation matrix of the RGB image and the observation matrix of the infrared image are exchanged and then processed by the selective state space module, and then the feature selection of the RGB image corresponding to the observation matrix of the infrared image is performed through the observation matrix of the infrared image, and the feature selection of the observation matrix corresponding to the RGB image is performed through the observation matrix of the RGB image; The feature maps after feature selection are sequentially processed by a normalization module and a linear layer, and then the processed feature is connected in residual with the corresponding global enhanced feature map; The feature map obtained by residual connection is input into a global average pooling layer and a sigmoid activation function to generate an attention map; The attention map is multiplied element by element with the feature map obtained by residual connection, and then the feature map obtained by multiplication is added to the feature map obtained by residual connection to obtain a global fusion feature map.

4. The method of claim 1, wherein, The local feature enhancement module processes the basic feature map to obtain a local enhanced feature, specifically including: adjusting the channel number of the basic feature map by using a 1×1 convolution kernel, then performing multi-scale feature extraction by using 3×3, 5×5 and 7×7 convolution kernels, concatenating the multi-scale features, adjusting the channel number by convolution after global average pooling, then using a sigmoid activation function to generate a spatial information weight, and multiplying the spatial information weight with the feature map input into the local feature enhancement module element by element to obtain the local enhanced feature.

5. The method of claim 1 or 4, wherein, The RGB image and the infrared image are sequentially input into the basic feature encoder and the local feature enhancement module respectively to obtain corresponding local enhanced features, and the local enhanced features of the two images are fused by the local feature fusion module to obtain a local fusion feature, specifically including: The local enhanced features of the RGB image and the infrared image are concatenated and then processed by a depth separable convolution; The chunk operation is used to divide the feature map into two parts along the channel dimension, one part is used to generate a weight map by using a sigmoid activation function, and then the weight map is multiplied element by element with the other part of the feature map to finally obtain the local fusion feature.

6. The method of claim 1, wherein, The spatial recalibration alignment module is used to align the global enhanced feature map output by the global feature fusion module and the local enhanced feature map output by the local feature fusion module to obtain a fusion feature, and the alignment process includes: The global enhanced feature map and the local enhanced feature map are added to obtain a global-local feature, and then the global-local feature is integrated and the channel is adjusted by a convolution layer and a normalization layer. The feature map is pooled along the height and width directions respectively by using two different size pooling operations, one of the pooled feature maps is reshaped, and then the two pooled feature maps are spliced together; After the spliced feature map is processed by 1*1 convolution and Sigmoid activation function, the obtained feature map is divided into two feature vectors according to the height and width of the global-local feature; After the two feature vectors are processed by 1*1 convolution and Sigmoid activation function respectively, the obtained attention map is obtained by element-wise multiplication fusion; The obtained attention map is multiplied with the global-local feature element by element to obtain the fused feature.

Images