Collaborative optimization method for image fusion and semantic segmentation

Abstract

The invention belongs to the technical field of image processing, and particularly relates to a collaborative optimization method for image fusion and semantic segmentation, which comprises the steps that an encoder, a decoder for an image fusion task and a decoder for a semantic segmentation task are constructed, all the decoders share data output by the encoder, and in the training process, all the decoders share the data output by the encoder. And splicing the output of the decoder for the image fusion task with the output of the decoder for the semantic segmentation task and the label corresponding to the output, and optimizing the network parameters of the decoder and all encoders by taking the consistency of two spliced vectors as one of loss functions. According to the method, a deep learning-based framework is utilized, multi-dimensional features from different modal data can be effectively learned, and meanwhile, collaborative optimization of multiple tasks is realized through mining of internal consistency between the tasks, so that a better image fusion and semantic segmentation effect is achieved.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and particularly relates to a collaborative optimization method for image fusion and semantic segmentation. Background Art

[0002] In the field of autonomous driving, accurate and robust scene analysis is crucial. However, in complex environments (such as bad weather or night driving), relying solely on visible light sensors may make it difficult to accurately identify targets. In contrast, infrared sensors are not affected by these factors and can provide clear target information under low light and bad weather conditions, but their spatial resolution is relatively low. Therefore, infrared and visible light multimodal fusion has become an effective method to enhance scene understanding and target detection performance.

[0003] In recent years, the progress of deep learning has significantly promoted the development of the multimodal field, especially with convolutional neural network (CNN) and Transformer technologies as the mainstream. However, due to the limitations of the local receptive field of the convolutional neural network (CNN) and the high computational complexity introduced by the Transformer model, it is still difficult to build a balanced model. This makes it difficult to fully exploit the potential between CNN and Transformer in practical applications, resulting in unsatisfactory performance.

[0004] At the same time, the introduction of the Mamba architecture has brought innovation to this field. Mamba performs excellently in efficiently capturing global context information and is well-known for its computational efficiency based on the state space model (SSM). Vision Mamba and VMamba extend this architecture to computer vision tasks and further improve performance by strengthening the unidirectional scanning mechanism. Given the advantages of Mamba in global context modeling, it is very suitable for application in image processing tasks. However, most researchers usually train the Mamba module from scratch or directly apply the pre-trained Vision Mamba module. These methods rarely consider the effective fusion of local and global information in images, limiting their full utilization of the features provided by pre-trained CNN models. Finally, most existing methods regard image fusion and semantic segmentation as two independent problems and ignore their internal associations. Although some studies attempt to optimize the image fusion and semantic segmentation tasks through cascading, there is a lack of effective strategies to fully utilize the internal consistency between the two, thus failing to maximize their potential. Summary of the Invention

[0005] To solve the above technical problems, the present invention provides a collaborative optimization method for image fusion and semantic segmentation, which constructs an encoder, a decoder for the image fusion task, and a decoder for the semantic segmentation task. All decoders share the data output by the encoder. During the training process, the output of the decoder for the image fusion task is respectively concatenated with the output of the decoder for the semantic segmentation task and the label corresponding to the output, and the consistency of the two concatenated vectors is used as one of the loss functions to optimize the network parameters of the decoder and all encoders.

[0006] Further, the encoder includes a basic feature encoder, a global feature enhancement module, a global feature fusion module, a local feature enhancement module, a local feature fusion module, and a spatial recalibration alignment module.

[0007] Further, the basic feature encoder is composed of multiple cascaded feature extraction layers, and the image input to the basic feature encoder is residually connected to the output of the last feature extraction layer. The feature extraction layer sequentially processes the input image using depthwise separable convolution, fully connected layer, fully connected layer, and GELU activation function layer to obtain the basic feature map.

[0008] Further, the global feature enhancement module processes the basic feature map to obtain the global enhanced feature map, specifically including: sequentially processing the input feature using a linear layer, a depthwise separable convolution module, a selective state space module, a normalization module, and a linear layer, and then residually connecting the processed feature with the input feature to obtain the global enhanced feature map.

[0009] Further, the RGB image and the infrared image are respectively passed through the basic feature encoder and the global feature enhancement module in sequence to obtain the corresponding global enhanced feature maps. The two global enhanced feature maps are fused through the global feature fusion module to obtain the global fusion feature map, specifically including:

[0010] Processing the global enhanced feature map of the RGB image and the global enhanced feature map of the infrared image respectively using a linear layer and a depthwise separable convolution with shared parameters;

[0011] Exchanging the observation matrices for processing the RGB image and the infrared image in the selective state space module, and then performing feature selection on the RGB image corresponding to the infrared image's observation matrix and feature selection on the infrared image corresponding to the RGB image's observation matrix;

[0012] The feature maps after feature selection are sequentially processed by a normalization module and a linear layer, and the processed features are residually connected to the corresponding global enhanced feature maps;

[0013] The feature maps obtained by the residual connection are passed through global average pooling and sigmoid activation function to generate the attention map;

[0014] The feature map obtained by element-wise multiplication of the attention map and the feature map after residual concatenation is added to the feature map after residual concatenation to obtain the global fusion feature map.

[0015] Furthermore, the local feature enhancement module processes the basic feature map to obtain local enhanced features, specifically including: adjusting the number of channels of the basic feature map through a 1×1 convolutional kernel, then performing multi-scale feature extraction using 3×3, 5×5, and 7×7 convolutional kernels, concatenating the multi-scale features together, performing global average pooling, then adjusting the number of channels through convolution, and then using the Sigmoid activation function to generate spatial information weights, and multiplying the spatial information weights with the feature map input to the local feature enhancement module element-wise to obtain local enhanced features.

[0016] Furthermore, the RGB image and the infrared image are respectively processed by the basic feature encoder and the local feature enhancement module in sequence to obtain corresponding local enhanced features, and the local enhanced features of the two images are fused by the local feature fusion module to obtain local fusion features, specifically including:

[0017] The local enhanced features of the RGB image and the infrared image are concatenated together and then processed using depthwise separable convolution;

[0018] The feature map is divided into two parts along the channel dimension using the chunk operation, one of which passes through the Sigmoid activation function to generate a weight map, and then is multiplied element-wise with the other part of the feature map, and finally the local fusion feature is obtained.

[0019] Furthermore, the spatial recalibration alignment module is used to align the global enhanced feature map output by the global feature fusion module and the local enhanced feature map output by the local feature fusion module to obtain the fusion feature, and the alignment process includes:

[0020] The global enhanced feature map and the local enhanced feature map are added together to obtain the global-local feature, and then integrated and channel-adjusted through a convolutional layer and a normalization layer;

[0021] The feature map is pooled along the height and width directions using two pooling operations with different sizes, one of the pooled feature maps is reshaped, and then the two pooled feature maps are concatenated together;

[0022] The concatenated feature map is processed by 1×1 convolution and the Sigmoid activation function, and the obtained feature map is divided into two feature vectors according to the height and width of the global-local feature;

[0023] The two feature vectors are respectively processed by 1×1 convolution and the Sigmoid activation function, and then multiplied element-wise and fused to obtain the attention map;

[0024] The obtained attention map is multiplied element-wise with the global-local features to obtain the fused features.

[0025] Further, the decoder includes a semantic segmentation head and an image fusion head, where:

[0026] The semantic segmentation head includes a cascade of multiple MAVSSBlocks. The processing process of each MAVSSBlock for the input feature map includes: normalizing the feature map, dividing the feature map into two parts through a chunk operation. The first part is processed sequentially through a linear layer, a depthwise separable convolution, a two-dimensional selective scanning module, and a normalization layer to obtain an enhanced feature map. The second part generates a corresponding attention map through a channel attention and a spatial attention module. The enhanced feature map is multiplied element-wise with the attention map to obtain the output of the MAVSSBlock. In specific implementation, the result output by the cascaded MAVSSBlocks passes through an upsampling and a multi-layer perceptron to output the segmentation result.

[0027] The processing process of the image fusion head for the data is as follows: The RGB image and the infrared image are spliced together, and then the original features are extracted through a feature correction and calibration module. The multiple-scale images output by the shared encoder are respectively processed through a dynamic region feature extraction module to obtain multiple feature maps. The multiple feature maps are upsampled to the size of the original feature map and then added and fused. After that, they are spliced together with the original features. The splicing result is input into an image reconstruction layer to obtain the fusion result. The multiple-scale features are the feature maps of different sizes obtained during the downsampling process of the basic encoder. In the present invention, the image reconstruction layer is composed of multiple convolutional layers and a RELU activation function.

[0028] Preferably, the feature correction and calibration module of the present invention is used to correct the abstract features of the feature map caused by excessive image processing, that is, the original image features of the acquired image are obtained by splicing the RGB image and the infrared image and inputting them into this module. As a preferred implementation, the dynamic region feature extraction module of the present invention processes images of multiple scales output by the shared encoder, and dynamically specifies the number of regions to be concerned about for each feature map in this module. The feature map input into the feature correction and calibration module undergoes convolution operations of different scales according to the number of regions, and then sequentially passes through a 3×3 convolution, a normalization layer, a RELU activation function, a 1×1 convolution, and a sigmoid activation function to obtain the overall weighted fusion feature; the feature map input into the dynamic region feature extraction module undergoes convolution operations of different scales according to the number of regions, and then calculates the weight coefficients of different convolutions through two linear layers and a RELU activation function, thereby dynamically determining the weight coefficients that each region needs to be concerned about. Then, a Softmax function is used to normalize the coefficients. After obtaining the normalized coefficients, the extracted features corresponding to each region are multiplied by the corresponding feature weights, and all regions are summed to obtain the overall weighted fusion feature..

[0029] Further, during training, the predicted label and the true label of semantic segmentation are respectively spliced with the image fusion result, and the splicing results are respectively dimensionally elevated through multiple cascaded convolutional layers to obtain high-dimensional vectors, and the distance between the two high-dimensional vectors is calculated as the consistency loss.

[0030] Aiming at the problems that it is difficult to extract effective features in multi-modal fusion and it is difficult to optimize the training of joint tasks, the present invention designs an image fusion and semantic segmentation method based on multi-modal fusion and collaborative task optimization. By designing a multi-modal fusion network to aggregate feature information from different modalities, and at the same time improving the internal consistency between the image fusion and semantic segmentation tasks through collaborative optimization, the effect of achieving a win-win situation in the optimization of multiple tasks is realized. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is a schematic diagram of the model of a collaborative optimization method for image fusion and semantic segmentation according to the present invention;

[0032] Figure 2 It is a schematic diagram of the structure of the global feature enhancement module adopted by the present invention;

[0033] Figure 3 It is a schematic diagram of the structure of the local feature enhancement module adopted by the present invention;

[0034] Figure 4 It is a schematic diagram of the structure of the local feature fusion module adopted by the present invention;

[0035] Figure 5Schematic diagram of the global feature fusion module structure adopted by the present invention;

[0036] Figure 6 Schematic diagram of the spatial recalibration alignment module structure adopted by the present invention;

[0037] Figure 7 Schematic diagram of the MAVSSBlock structure adopted by the present invention. Detailed implementation manners

[0038] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0039] The present invention provides a collaborative optimization method for image fusion and semantic segmentation, including constructing an encoder, a decoder for the image fusion task, and a decoder for the semantic segmentation task. All decoders share the data output by the encoder. During the training process, the output of the decoder for the image fusion task is respectively concatenated with the output of the decoder for the semantic segmentation task and the label corresponding to the output, and the consistency of the two concatenated vectors is used as one of the loss functions to optimize the network parameters of the decoder and all encoders.

[0040] In this embodiment, a specific implementation manner of a collaborative optimization method for image fusion and semantic segmentation is given, which specifically includes the following steps:

[0041] Step 1: Prepare a multi-modal image dataset and preprocess the image dataset. The dataset includes a dataset of infrared-visible light images, that is, a visible light sensor and an infrared sensor are used to capture images of the same scene;

[0042] Step 2: Construct a shared encoder for extracting multi-modal features. The shared encoder consists of a basic feature encoder based on ConvNext, a global feature enhancement module based on the Mamba self-attention mechanism, a global feature fusion module based on the Mamba cross-attention mechanism, a local feature enhancement module based on multi-core convolution, a local feature fusion module based on pooling and depthwise separable convolution, and a fusion module based on the coordinated attention mechanism;

[0043] Step 3: Construct specific decoders according to different task requirements, and process the data output by the weight-sharing encoder according to the constructed decoders. The specific decoders consist of a semantic segmentation head based on the Mamba and hybrid attention mechanisms and an image fusion head based on pixel feature extraction and CNN;

[0044] Step 4: Achieve collaborative optimization by exploring the internal consistency of different tasks. The method for achieving collaborative optimization is to splice the corresponding prediction results and labels and map them to a high-dimensional space, and finally calculate the distance loss between different tasks through an appropriate distance loss function.

[0045] For example Figure 1 , in this embodiment, a visible light image (RGB image) and an infrared image are simultaneously input into the basic encoder to obtain the basic features of the visible light image and the basic features of the infrared image respectively. The basic features of the visible light image and the basic features of the infrared image are respectively subjected to local feature enhancement and global feature enhancement, and then the two local feature enhancements are subjected to local feature fusion, and the two global feature enhancements are subjected to global feature fusion. The fused results are subjected to spatial recalibration alignment, and finally input into the semantic segmentation head and the image fusion head for downstream tasks. According to the prediction results of the downstream tasks, the network parameters are optimized using task consistency regularization as one of the loss functions during the backpropagation process.

[0046] As an alternative implementation, this embodiment provides a specific implementation of preprocessing an image dataset, including:

[0047] First, load the RGB image and the infrared image, convert the channel order of the RGB image, and expand the infrared image into a pseudo-RGB image;

[0048] Next, in the training stage, use data augmentation methods, including random horizontal flipping and random scaling, and perform normalization operations on the images;

[0049] Then, the system randomly crops according to the target size, fills the excess part and records the edge information, and finally converts the image format to the CHW format and generates a binary Mask to mark the valid area and the filled area;

[0050] Finally, in the verification stage, the processing is simplified, and only normalization, channel conversion, and Tensor conversion are performed.

[0051] In this embodiment, the basic feature encoder is used to extract the corresponding basic feature maps from the RGB image or the infrared image, specifically including the following steps:

[0052] The input image is first subjected to preliminary size compression through a shared downsampling layer, which includes a convolution operation and aims to perform a preliminary mapping of the number of channels of the input image and reduce the spatial dimension;

[0053] Next, layer normalization is used to ensure the stability of subsequent calculations;

[0054] Then, the RGB image and the infrared image are respectively input into the convolution and normalization operations with the same structure. Each stage contains multiple residual blocks. In the blocks, the image first undergoes depthwise separable convolution to independently extract features for each channel and then is normalized;

[0055] Subsequently, two fully connected layers are used to further capture the relationships between features, and a GELU activation function is introduced to introduce non-linear transformation;

[0056] Finally, the residual block adopts residual connection to add the input and output, and a dropout strategy is introduced to discard some paths, thereby obtaining richer basic features;

[0057] In the above operations, specifically, through the module of the convolutional neural network and the framework design of Transformer, preliminary features can be better extracted while reducing the computational complexity.

[0058] In this embodiment, the global feature enhancement module based on the Mamba self-attention mechanism takes the output of the basic encoder based on ConvNext as input, as Figure 2 , and the specific processing process specifically includes the following steps:

[0059] First, the image features of different modalities are respectively subjected to feature extraction through a linear layer and a depthwise separable convolution module;

[0060] Subsequently, corresponding A, B, C, D, and Δ parameters are obtained in the Selective Scan Space State Sequential Model (abbreviated as the S6 module) (these parameters are generated through the results of the linear layer, embedding layer, etc. of the S6 module. Those skilled in the art can obtain these parameters according to any existing S6 module selected. The selective scan space state sequential model adopted in this embodiment is a mature feature processing module in the prior art, and this embodiment will not elaborate on it). After feature extraction through the selective scan space state sequential model, normalization processing is performed through a normalization module and a linear layer;

[0061] Finally, residual connection is adopted to obtain self-modal enhanced features.

[0062] In the above operations, the self-attention mechanism can effectively achieve a global receptive field. However, due to the high computational complexity requirement of the traditional dot product operation, the present invention adopts a selective scan space to optimize the implementation of the self-attention mechanism.

[0063] In this embodiment, the global feature fusion module based on the Mamba cross-attention mechanism receives the output of the global feature enhancement module based on the Mamba self-attention mechanism as input, as Figure 5, the specific processing process specifically includes the following steps:

[0064] First, after processing the self-modal enhancement features of different modalities through a shared linear layer (Linear) and depthwise separable convolution (DWConv), they are fed into the S6 module (Selective Scan Space State Sequential Model, hereinafter referred to as the S6 module or S6 Blocks) to generate different A, B, C, D, and Δ;

[0065] Next, the C values of the two features are swapped, and further feature extraction is performed through a 2D selective scan module (2D Selective ScanModule, hereinafter referred to as the SS2D module);

[0066] Subsequently, after passing through a layer normalization layer (Layer Normalization, LN) and a linear layer, and combined with the initial self-modal enhancement features for residual feature enhancement, that is, the input feature map and the output of the last linear layer are subjected to element-wise summation;

[0067] Finally, the features of different modalities are concatenated, and an attention map is generated through global average pooling and a sigmoid activation function. This attention map is multiplied element-wise with the original feature map, and added to the concatenated features to finally obtain the global feature fusion output; in this implementation, the cross-attention mechanism has a significant effect on the global context feature extraction of different modalities. Similarly, this embodiment also uses a selective state space sequence for optimization to reduce memory consumption.

[0068] In this embodiment, the local feature enhancement module based on multi-core convolution receives the output of the basic encoder based on ConvNext as input, such as Figure 3 , the specific processing process specifically includes the following steps:

[0069] First, the input feature maps of different modalities are adjusted in channel number through 1×1 convolution (Conv 1×1);

[0070] Subsequently, the feature maps are respectively subjected to multi-scale feature extraction through 3×3 convolution (Conv 3×3), 5×5 convolution (Conv 5×5), and 7×7 convolution (Conv 7×7). The feature maps of different scales are added to obtain preliminary local features;

[0071] Next, a spatial scale feature map is obtained through global average pooling (Global Average Pooling), and its channel number is adjusted through depthwise separable convolution;

[0072] Finally, the Sigmoid activation function is used to generate spatial information weights, and these weights are fused with the initial feature map using matrix multiplication to enhance the spatial information;

[0073] In the above operations, local receptive fields of features are extracted through convolution, and multi-core convolution helps to obtain information from different scales, thus achieving a richer feature effect.

[0074] In this embodiment, the local feature fusion module based on pooling and depthwise separable convolution receives the output of the local feature enhancement module based on multi-core convolution as input, as Figure 4 , and the specific processing process specifically includes the following steps:

[0075] First, the local feature enhancement maps from different modalities are concatenated by tensors (Tensor Concatenation), and the features are further fused through depthwise separable convolution;

[0076] Next, the concatenated feature map is divided into two parts along the channel dimension using the chunk operation. One part generates a weight map through the Sigmoid activation function, and then performs the Hadamard product with the other part of the feature map to finally obtain the fused local feature map;

[0077] In the above operations, depthwise separable convolution can efficiently aggregate the fused features and integrate the features of different channels through the attention map, improving the channel modeling ability of the model.

[0078] In this embodiment, the spatial recalibration alignment module based on the coordinated attention mechanism receives the outputs of the global feature fusion module based on the Mamba cross-attention mechanism and the local feature fusion module based on pooling and depthwise separable convolution as input, as Figure 6 , and the specific processing process specifically includes the following steps:

[0079] First, the feature maps from different fusion modules are added together, and further feature integration and channel number adjustment are performed through a convolutional layer and a normalization layer;

[0080] Next, two different sizes of pooling operations are applied to pool the feature map along the height and width directions respectively to generate two pooled feature maps;

[0081] Subsequently, a corresponding attention map is generated through a 1×1 convolution and the Sigmoid activation function;

[0082] Finally, these attention maps are multiplied element-wise with the initial fused features to finally obtain the fused feature map.

[0083] In this embodiment, an image fusion and semantic segmentation method based on multi-modal fusion and collaborative task optimization constructs a decoder based on different task requirements and processes the data output by the shared encoder according to the constructed decoder, including:

[0084] Construct a semantic segmentation head based on Mamba and hybrid attention mechanism. This segmentation head receives the multi-scale fusion feature map from the above-mentioned spatial recalibration alignment module and generates the final semantic segmentation result. The processing of the data by this segmentation head includes: First, the input feature map will be processed by multiple MAVSSBlocks in ascending order, as Figure 7 , in each block, the input feature map is first processed by a normalization layer, and then divided into two parts through a chunk operation. One part is further feature-extracted through a fully connected layer and a depthwise separable convolution, and a global context is established through a 2D selective scanning module. Finally, the feature enhancement result is obtained through normalization; the other part generates corresponding attention maps through a channel attention and a spatial attention module (during the process of generating the attention maps, the input feature map is respectively subjected to average pooling and max pooling operations. The two features after pooling are added and then processed through an activation function, and then continue to be divided into two branches and respectively subjected to average pooling and max pooling operations again. The two feature maps after the second pooling operation pass through), and the two parts of the feature maps are then multiplied element by element to obtain the enhanced result. After each feature map is processed by the corresponding block, it will be upsampled to the same size as the upper-level feature map, then added and sent to the next-level block for processing. Finally, after a total upsampling, the final semantic segmentation result is obtained through a multi-layer perceptron;

[0085] Since the CNN encoder provides more channels, another branch is added to the original VSSBlock in the present invention. The multi-channel integration ability of the VSSBlock is better improved through the channel attention and spatial attention modules, and the features are Figure 1 upsampled step by step and fused through a design similar to FPN to obtain the final result. Therefore, an image fusion head based on pixel feature extraction and CNN is constructed, which receives the multi-scale fusion feature map from the above-mentioned spatial recalibration alignment module to obtain the final image fusion result. Specifically, it includes: First, the input feature map will be subjected to preliminary feature extraction through a convolutional neural network and upsampled to the initial image size. Subsequently, the original multi-modal image is subjected to feature extraction through a pixel feature extraction module and then spliced with the corresponding feature map. Finally, the spliced result is reconstructed through multiple convolutions and RELU activation functions to obtain the image fusion result;

[0086] Since image fusion highly depends on the features of the original images, the present invention takes the initial input as a reference and inputs it. A pixel feature extraction module is used to obtain the rich visual characteristics of the original images, and the effectiveness of CNN in low-level tasks is utilized to obtain the corresponding fused image results.

[0087] An image fusion and semantic segmentation method based on multi-modal fusion and collaborative task optimization realizes collaborative optimization by exploring the inherent consistency between different tasks, specifically including: concatenating the segmentation results output by the above semantic segmentation head, the true labels of the segmentation, and the results of image fusion respectively in the channel dimension. In this embodiment, since there is no corresponding true label for image fusion, the present embodiment directly concatenates the image fusion results with the true labels of the image segmentation results, and then extracts the features of the concatenated images through multiple convolutional layers to obtain the corresponding high-dimensional latent representations. Subsequently, a cosine distance function is used to calculate the similarity between two high-dimensional latent representation vectors in the channel dimension, and a new distance loss function is constructed in the above manner and added to the original loss function to adjust the optimization direction of training.

[0088] This embodiment considers the potential inherent consistency between different image processing tasks, so the finally obtained results are concatenated and their high-dimensional features are extracted, and the distance loss between the high-dimensional features is taken as one aspect of the optimization of the present invention, so as to achieve implicit alignment between multiple tasks and further realize joint optimization.

[0089] In addition, it should be noted that the encoder and decoder of the present invention adopt the U-shaped network structure, and the corresponding upsampling and downsampling adopt skip connections.

[0090] Although the embodiments of the present invention have been shown and described, for those of ordinary skill in the art, it can be understood that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirits of the present invention, and the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A collaborative optimization method for image fusion and semantic segmentation, constructing an encoder, a decoder for image fusion tasks, and a decoder for semantic segmentation tasks, characterized in that: All decoders share the data output by the encoder. During the training process, the output of the decoder for the image fusion task is concatenated with the output of the decoder for the semantic segmentation task and the label corresponding to the output. The consistency of the two concatenated vectors is used as one of the loss functions to optimize the network parameters of the decoder and all encoders.

2. The collaborative optimization method for image fusion and semantic segmentation according to claim 1, characterized in that: The encoder includes a basic feature encoder, a global feature enhancement module, a global feature fusion module, a local feature enhancement module, a local feature fusion module, and a spatial recalibration alignment module.

3. The collaborative optimization method for image fusion and semantic segmentation according to claim 2, characterized in that: The basic feature encoder is composed of multiple cascaded feature extraction layers, and the image input to the basic feature encoder is residually connected to the output of the last feature extraction layer. The feature extraction layer uses depthwise separable convolution, fully connected layer, fully connected layer, and GELU activation function layer in sequence to process the input image to obtain the basic feature map.

4. The collaborative optimization method for image fusion and semantic segmentation according to claim 2, characterized in that: The global feature enhancement module processes the basic feature map to obtain a global enhanced feature map, specifically including: sequentially using a linear layer, a depth-wise separable convolution module, a selective state space module, a normalization module, and a linear layer to process the input features, and then performing a residual connection between the processed features and the input features to obtain a global enhanced feature map.

5. A collaborative optimization method for image fusion and semantic segmentation according to claim 2, 3 or 4, characterized in that: The RGB image and the infrared image are respectively passed through the basic feature encoder and the global feature enhancement module to obtain the corresponding global enhanced feature map, and the two global enhanced feature maps are fused through the global feature fusion module to obtain the global fused feature map, which specifically includes: The global enhanced feature maps of RGB images and infrared images are processed respectively using linear layers with shared parameters and depthwise separable convolutions. The exchange selective state space module processes the observation matrices of the RGB image and the infrared image, and then performs feature selection corresponding to the RGB image through the observation matrix of the infrared image and performs feature selection corresponding to the observation matrix through the observation matrix of the RGB image; The feature map after feature selection is processed by the normalization module and the linear layer in sequence, and the processed features are residually connected with the corresponding global enhanced feature map; The feature map obtained by residual connection is subjected to global average pooling and sigmoid activation function to generate an attention map; The attention map is multiplied element-by-element with the feature map after residual splicing, and the feature map obtained by multiplication is added to the feature map after residual splicing to obtain a global fusion feature map.

6. The collaborative optimization method for image fusion and semantic segmentation according to claim 2, characterized in that: The local feature enhancement module processes the basic feature map to obtain local enhanced features, specifically including: adjusting the number of channels of the basic feature map through a 1×1 convolution kernel, and then using 3×3, 5×5 and 7×7 convolution kernels to extract multi-scale features, splicing the multi-scale features together for global average pooling and then adjusting the number of channels through convolution, and then using the Sigmoid activation function to generate spatial information weights, and using the spatial information weights to multiply the feature map input to the local feature enhancement module element by element to obtain local enhanced features.

7. A collaborative optimization method for image fusion and semantic segmentation according to claim 2, 3 or 6, characterized in that: The RGB image and the infrared image are processed by the basic feature encoder and the local feature enhancement module respectively to obtain the corresponding local enhancement features, and the local enhancement features of the two images are fused by the local feature fusion module to obtain the local fusion features, which specifically include: The local enhanced features of the RGB image and infrared image are spliced ​​together and processed using depthwise separable convolution; The chunk operation is used to divide the feature map into two parts along the channel dimension. One part is activated by the Sigmoid function to generate a weight map, which is then multiplied element by element with the other part of the feature map to finally obtain the local fusion feature.

8. The collaborative optimization method for image fusion and semantic segmentation according to claim 2, characterized in that: The spatial recalibration alignment module is used to align the global enhanced feature map output by the global feature fusion module with the local enhanced feature map output by the local feature fusion module to obtain the fusion feature. The alignment process includes: The global-local feature is obtained by adding the global enhancement feature map to the local enhancement feature map, and then integrated and channel-adjusted through the convolution layer and the normalization layer; Use two pooling operations of different sizes to pool the feature maps in the height and width directions respectively, reshape one of the pooled feature maps, and then splice the two pooled feature maps together; After the concatenated feature maps are processed by 1×1 convolution and Sigmoid activation function, the obtained feature maps are divided into two feature vectors according to the height and width of the global-local features; The two feature vectors are processed by 1×1 convolution and Sigmoid activation function respectively, and then multiplied element by element to obtain the attention map; The obtained attention map is element-wise multiplied with the global-local feature to obtain the fused feature.

9. The collaborative optimization method for image fusion and semantic segmentation according to claim 2, characterized in that: The decoder includes a semantic segmentation head and an image fusion head, where: The semantic segmentation head includes multiple cascaded MAVSSBlocks. Each MAVSSBlock processes the input feature map in the following ways: normalizing the feature map and dividing the feature map into two parts through chunk operation. The first part is processed by linear layer, depthwise separable convolution, two-dimensional selective scanning module and normalization layer in sequence to obtain an enhanced feature map. The second part is processed by channel attention and spatial attention modules to generate the corresponding attention map. The enhanced feature map is multiplied element by element with the attention map to obtain the output of MAVSSBlock. The image fusion head processes data as follows: RGB images and infrared images are spliced ​​together, and then the original features are extracted through the feature correction and calibration module. The images of multiple scales output by the shared encoder are processed by the dynamic area feature extraction module to obtain multiple feature maps. The multiple feature maps are upsampled to the original feature map size and then added and fused. After that, they are spliced ​​together with the original features, and the splicing results are input into the image reconstruction layer to obtain the fusion result.

Images