A hybrid prompt architecture fusion method for multi-modal semantic segmentation

Abstract

A hybrid cueing architecture fusion method for multimodal semantic segmentation belongs to the field of computer vision. The implementation method of this invention is as follows: a multimodal embedding module is constructed based on an RGB pre-trained model; for the initial cueing output by the multimodal embedding module and the RGB features input by the backbone network, linear mapping is used to project them onto a low-rank subspace to complete feature alignment; a mixing matrix is ​​introduced to fuse the RGB features of the low-rank subspace with the cueing information; the new cueing information is fused with the RGB features encoded by the backbone network; a lightweight multi-subspace alignment and hybrid cueing module is introduced; the multi-resolution self-attention encoder of the backbone network is used to encode the RGB image features; auxiliary image information is used to generate the initial cueing through the multimodal embedding module; the initial cueing information and RGB image features are fused with the hybrid cueing module through multi-subspace alignment to form new cueing information, which is then fused with the RGB image features, thus fusing the semantic information of the RGB image and the auxiliary modality.

Description

Technical Field

[0001] This invention relates to a multimodal fusion method, and more particularly to a hybrid cue architecture fusion method for multimodal semantic segmentation, belonging to the field of computer vision. Background Technology

[0002] Semantic segmentation, a fundamental task in computer vision, aims to assign predefined semantic labels to each pixel in an input image. In recent years, with the release of large-scale RGB benchmark datasets such as Cityscapes and ADE20K, semantic segmentation technology has made significant progress, providing crucial support for the training and evaluation of segmentation models. However, in complex environments such as low light and inclement weather, segmentation methods relying solely on RGB images often perform poorly, primarily because visual information may be severely occluded or distorted, leading to decreased segmentation accuracy. To overcome these limitations, researchers have proposed multimodal semantic segmentation methods, which enhance segmentation performance by incorporating auxiliary sensor data (such as depth images and thermal infrared images). For example, depth images provide precise spatial location information, while thermal infrared images can capture target features imperceptible to visible light under low light or smoke conditions. This fusion of multimodal data not only improves the model's ability to recognize instance boundaries but also enhances its understanding of contextual information, thus significantly improving segmentation results. Therefore, multimodal segmentation methods outperform traditional RGB-only segmentation methods, opening up new directions for the research and application of semantic segmentation.

[0003] However, the application of multimodal semantic segmentation technology still faces two major challenges: 1. Increased model complexity. To handle additional modal data, it is usually necessary to design independent backbone network branches for each modality to extract features and introduce complex fusion modules to integrate multimodal features. This architectural expansion not only significantly increases the number of model parameters but also greatly increases computational complexity, making its deployment in resource-constrained environments (such as mobile devices or embedded systems) difficult. 2. Scarcity of multimodal data. Compared with RGB datasets, multimodal segmentation datasets are typically smaller in size, which limits the training performance and generalization ability of the model. Summary of the Invention

[0004] To address the issues of high complexity and data scarcity in existing multimodal semantic segmentation models, this invention aims to provide a hybrid prompting architecture fusion method for multimodal semantic segmentation. Through the collaborative optimization of pre-trained model reuse and dynamic prompting learning, it achieves efficient fusion of cross-modal features and parameter simplification. It can efficiently integrate auxiliary modal data without changing the underlying architecture of the pre-trained RGB segmentation model, thereby significantly improving segmentation performance while reducing model complexity.

[0005] The main objective of this invention is achieved through the following technical solution:

[0006] This invention discloses a hybrid prompting architecture fusion method for multimodal semantic segmentation. It constructs a multimodal embedding module based on an RGB pre-trained model to initialize the embedding information of auxiliary modalities, introduces a multi-subspace aligned hybrid prompting fusion method to generate multi-level prompting information, realizes multimodal information fusion, improves model segmentation accuracy, and at the same time, the lightweight hybrid prompting architecture reduces model complexity compared to the traditional two-stream fusion architecture, reducing the computational power requirements for model training and inference.

[0007] This invention discloses a hybrid prompting architecture fusion method for multimodal semantic segmentation, comprising the following steps:

[0008] Step 1: Construct a multimodal embedding module based on the RGB pre-trained model to generate high-quality initial prompt information from the input depth image, thermal infrared image and other auxiliary images, effectively encode the semantic information of the auxiliary modalities, avoid the interference of redundant information in the original auxiliary images, and improve the adaptability of different modal auxiliary information to the RGB feature space.

[0009] Given auxiliary image information such as depth images and thermal infrared images Where C f H and W represent the number of channels, height, and width of the image, respectively. An enhanced image is obtained using channel alignment transformation and data augmentation methods. Where C in This represents the number of channels in the RGB image. An initial block of a pre-trained ResNet-50 is used as the multimodal embedding network, and the initial cue e0 is extracted by aligning it with the RGB image features through a mapping layer. The extraction expression for the initial cue e0 is shown in equation (1):

[0010]

[0011] Where, x aux Indicates auxiliary modal input information. As the initial block for pre-training ResNet-50, the shallow architecture of the pre-trained ResNet-50 network is used to extract features of auxiliary modality information, so that the extracted auxiliary modality features match the shallow information distribution of the RGB backbone network. proj As a mapping layer, a single linear layer is used to map the extracted features to a specific feature space, consistent with the dimension of the RGB backbone model, to achieve high-quality initial prompt information generation.

[0012] Step 2: For the initial prompts output by the multimodal embedding module and the RGB features input by the Segformer backbone network, a linear mapping is used to project them onto the low-rank subspace to complete feature alignment. A mixing matrix is ​​introduced to fuse the RGB features of the low-rank subspace with the prompt information to generate new prompt information. The new prompt information is then fused with the RGB features encoded by the Segformer backbone network. Without modifying the Segformer backbone network architecture, a lightweight multi-subspace alignment and mixing prompt module is introduced to provide auxiliary modal feature information for the backbone model, effectively improving semantic segmentation performance.

[0013] Step 2.1: Project the initial cue output by the multimodal embedding module and the RGB features input by the Segformer backbone network to complete multi-subspace alignment. Given an RGB image... Where C in H and W represent the number of RGB channels, height, and width of the image, respectively. An enhanced image is obtained using a data augmentation method involving random horizontal flipping and perturbation. The enhanced data is normalized and then input into the embedding module, where it is divided into 4×4 blocks. These blocks are then used as input to the Segformer backbone network.

[0014] The RGB features input to the Segformer backbone network and the auxiliary modal features in the cue branch exist in different feature spaces, requiring alignment of these mismatched feature spaces. A low-rank subspace alignment method is introduced, which uses linear projection to align the RGB modal features and the cue information of the auxiliary modality into a low-rank subspace. The aligned features are then fused through an addition operation. The multi-subspace alignment expression is shown in Equation (2):

[0015]

[0016] Where i∈{1,2,…,N} represents the layer index, P i Represents the i-th multi-subspace hybrid hint module, e i Indicates the Lth i The layer's prompt message, h i Indicates the Lth i The output hidden state of the layer, It is a downsampling module. This is an upsampling module. Furthermore, let d be the original feature space, r << d, and r is the dimension of the low-rank aligned subspace. Using a smaller value for r allows the cueing module to be parameterized.

[0017] To improve information utilization efficiency, multiple pairs of down-projection and up-projection modules are introduced. Each pair of projections aligns the RGB modalities and auxiliary modalities in unique subspaces, thereby improving the utilization rate of useful information. However, this method significantly increases the number of parameters. Inspired by the low-rank adaptation fine-tuning method for large language models, even with extremely low ranks, the pre-trained neural network model still retains good performance because it operates in low intrinsic dimensions. At the same time, the pre-trained Segformer segmentation model is also in low intrinsic dimensions. Therefore, by reducing the rank of each subspace to maintain the parameter efficiency of the overall model, the RGB modal features and auxiliary modal cue information are projected into multiple subspaces, as shown in equations (3) and (4):

[0018] [W rgb,1 h i-1 W rgb,2 h i-1 ,...,W rgb,n h i-1 ], (3)

[0019] [W f,1 h i-1 W f,2 h i-1 ,...,W f,n h i-1 ], (4)

[0020] in, n represents the number of subspaces used. Dividing the original rank r by n ensures that introducing multiple subspaces does not introduce additional parameters.

[0021] Step 2.2: Introduce a hybrid matrix to fuse the RGB features of the low-rank subspace with the cue information to generate new cue information. The new cue information is then fused with the RGB features encoded by the Segformer backbone network. Without modifying the Segformer backbone network architecture, a lightweight multi-subspace alignment and hybrid cue module is introduced to provide auxiliary modality feature information for the backbone model, effectively improving semantic segmentation performance.

[0022] In the low-rank subspace alignment process, a hybrid cue generation method is introduced to enable information interaction between RGB features and auxiliary modal features in the cue branch, thereby improving the expressive power of the newly generated auxiliary cue information. In equations (3) and (4), the RGB modal features and auxiliary modal cue information are projected onto multiple subspaces, and E rgb,n h i-1 Represented as Introduce a mixing matrix Used for exchanging information between different subspaces. Features after mixing. The expression is shown in equation (5):

[0023]

[0024] Among them, M i,j This represents the value of the corresponding element in the mixing matrix M. Similarly, we get the hint e. i-1 low-rank representation Information from the RGB modality is injected into the low-rank cue through addition, and then linear projection is used. Projecting it back into the original feature space generates a new cue e. i Main layer L i The RGB features are fused with the new cue through addition. The expression is shown in equation (6):

[0025]

[0026] Step 3: Use the Segformer backbone network's multi-resolution self-attention encoder to encode RGB image features. Auxiliary image information such as depth image and infrared image is generated as initial prompts through the multimodal embedding module. The initial prompt information and RGB image features are fused together through the multi-subspace alignment and hybrid prompt module to form new prompt information and RGB image features. This efficiently fuses the semantic information of RGB images and auxiliary modalities, improves the model's inference speed, and enhances semantic segmentation performance.

[0027] Given an RGB image After data preprocessing into 4×4 blocks, the multi-resolution self-attention encoder of the Segformer backbone network takes the divided blocks as input to obtain the original image resolution in {1 / 4, 1 / 8, 1 / 16, 1 / 32}. The feature expression of the extracted image is shown in Equation (7):

[0028]

[0029] in, This refers to RGB image blocks after data preprocessing. This is a multi-resolution self-attention encoder, where θ represents the encoder parameters, f rgb This represents the extracted image features, corresponding to different levels of hidden states h. i From equation (1), the initial prompt information e0 can be obtained. The obtained e0, together with the embedded original RGB input h0, is used by the prompt module to generate a new prompt e1. After passing through the model layer, e1 is used to obtain the output hidden state h1, as shown in equations (8) and (9):

[0030] e1=P(h0,e0), (8)

[0031] h1=L(h0,e1), (9)

[0032] Where P and L represent the multi-subspace alignment and hybrid cueing module and the multi-resolution self-attention encoder layer, respectively. For different resolutions, the input feature is the cueing information e from the previous layer. i-1 and hidden state h i-1 The input features are reduced to a low-rank subspace through linear projection and then divided into multiple subspaces. A mixture matrix is ​​used to exchange and fuse information between the RGB modal subspace features. The fused features are combined through addition and then restored to the original feature dimension through linear projection to generate the final cue e. i Message e i With the previous hidden state h i-1 The sums are input into the encoder layer. The expressions are shown in equations (10), (11), (12), (13), and (14):

[0033]

[0034] h i ←L i (h i-1 +e i ), (14)

[0035] Among them W x and W rgb For the mapping layer, `reshape` is the transformation operation, and `flatten` is the flattening operation. W is an intermediate state of the hidden state. up For the upsampling layer, the mixing matrix M acts as a router, assigning different importances to each subspace through different weight mixtures. Simultaneously, the reparameterization of the mixing cue module enables the model to utilize a multi-branch architecture, further promoting more effective propagation of auxiliary information and helping the model achieve better semantic segmentation performance.

[0036] The obtained hidden state, i.e., the fused features, is input into the multilayer perceptron encoder. The MLP layer unifies the multilayer feature channel dimensions, upsamples, and then concatenates the channels. The MLP layer processes the concatenated features to obtain the reconstructed pixel-level segmentation mask. The expression is shown in Equation (15):

[0037]

[0038] In the formula, Linear represents the linear MLP module, Up represents the upsampling module, and Concat represents the connection operation. This represents the segmentation result corresponding to the fused features of r. Meanwhile, for the training loss function of the semantic segmentation task, the cross-entropy loss function is used, as shown in equation (16):

[0039]

[0040] Where N is the total number of pixels in the image, M is the total number of segmentation categories, and y k,c For the true label of the k-th pixel in category c, p k,c This indicates the probability that the model predicts the pixel belongs to category c. The aforementioned multimodal semantic segmentation hybrid prompting architecture can efficiently fuse multiple auxiliary modalities such as infrared images and depth images with RGB images, improving semantic segmentation performance while further reducing the complexity and inference cost of the multimodal segmentation model.

[0041] Beneficial effects

[0042] 1. The present invention discloses a hybrid prompt architecture fusion method for multimodal semantic segmentation. It uses a multimodal embedding module based on an RGB pre-trained model to generate high-quality initial prompt information from input auxiliary images such as depth images and thermal infrared images. Compared with traditional prompt initialization structures, it only needs to introduce a small number of parameters, which can effectively encode auxiliary modal semantic information, avoid interference from redundant information in the original auxiliary images, and improve the adaptability of different modal auxiliary information to the RGB feature space.

[0043] 2. The present invention discloses a hybrid prompting architecture fusion method for multimodal semantic segmentation. The hybrid prompting framework is used to convert auxiliary modalities into prompting information and fuse the prompting information with RGB image features. Compared with the traditional dual-stream embedding fusion strategy, it can effectively reduce model complexity and inference cost, and can adaptively fuse multiple auxiliary modal information to further improve semantic segmentation accuracy.

[0044] 3. The present invention discloses a hybrid prompting architecture fusion method for multimodal semantic segmentation, which introduces a lightweight multi-subspace alignment and hybrid prompting module, projects modal information onto multiple low-rank subspaces for alignment, and uses a hybrid matrix to further enhance the representational ability of the generated prompting information. Compared with the traditional semantic model prompting architecture, it can effectively reduce the computational power requirement for prompting information generation, and at the same time handle the generation of multiple auxiliary modal prompts, thereby improving the model's generalization ability. Attached Figure Description

[0045] Figure 1 This is a schematic diagram of the multimodal cue segmentation framework based on multi-subspace alignment and hybrid cueing in this embodiment;

[0046] Figure 2 This is a schematic diagram of the multi-subspace alignment and blending prompt module in this embodiment;

[0047] Figure 3 This is a flowchart of the multimodal cue segmentation framework based on multi-subspace alignment and hybrid cueing in this embodiment; Detailed Implementation

[0048] The present invention will now be described in detail with reference to the accompanying drawings and embodiments, while also discussing the technical problems solved by the present invention and its beneficial effects. It should be noted that the described embodiments are intended to facilitate understanding of the present invention and do not constitute any limitation thereof.

[0049] Example 1

[0050] This embodiment discloses a hybrid prompting architecture fusion method for multimodal semantic segmentation, applied to the Segformer model. The specific steps are as follows:

[0051] Step 1: Construct a multimodal embedding module based on the RGB pre-trained model to generate high-quality initial prompt information from the input depth image, effectively encode auxiliary modal semantic information, avoid interference from redundant information in the original auxiliary image, and improve the adaptability of different modal auxiliary information to the RGB feature space.

[0052] like Figure 1 As shown, this example uses the Segformer-B5 model as the backbone network architecture, employing a pre-trained Mix Transformer encoder (MiT) on the ADE20K dataset as the backbone encoder, and a 512-dimensional MLP decoder architecture. The MiT backbone encoder features a multi-scale feature extraction architecture, with features at four different scales from stages 1 to 4. The dataset is NYU Depth V2, containing 1449 RGB-D samples covering 40 categories, with all RGB and depth images having a resolution of 480×640.

[0053] Given depth image information Where H and W represent the number of channels, height, and width of the image, respectively, the enhanced image is obtained using channel alignment transformation and data augmentation methods. Where C in This indicates the number of channels in an RGB image. For example... Figure 2 As shown, the design uses ResNet50 as the feature extractor, loads pre-trained weights for initialization, selects the feature map obtained in the first stage of the ResNet50 model, and completes the alignment of the feature map with the feature dimension of the first stage of the backbone network through a single-layer linear mapping layer. The extraction expression of the initial prompt e0 is shown in Equation (1):

[0054]

[0055] Where, x aux This represents the input information of the auxiliary modal depth image. As the initial block for pre-training ResNet-50, the shallow architecture of the pre-trained ResNet-50 network is used to extract features of auxiliary modality information, so that the extracted auxiliary modality features match the shallow information distribution of the RGB backbone network. proj As a mapping layer, a single linear layer is used to map the extracted features to a specific feature space, consistent with the dimension of the RGB backbone model, to achieve high-quality initial prompt information generation.

[0056] Furthermore, the design of the pre-trained feature extractor was investigated, and the impact of initializing the extractor with pre-trained weights on performance was analyzed. The results are shown in Table 1. When using a randomly initialized ResNet50 cue extractor, the average intersection-over-union (IoU) ratio was 59.5%, while after initialization with pre-trained weights, the performance improved to 60.1%. This demonstrates the effectiveness of using a pre-trained RGB model to extract initial cues. The pre-trained backbone network enables the model to extract more meaningful representations from auxiliary modalities, ultimately achieving better segmentation results.

[0057] Table 1: The effect of initializing the ResNet50 cue extractor with pre-trained weights

[0058] Feature extractor initialization mIoU random 59.5％ Pre-training 60.1％

[0059] Next, the impact of using different feature extractor architectures in the multimodal embedding module was evaluated, and the results are shown in Table 2. Since the backbone network uses a MiT encoder architecture, the performance of various MiT variant encoders as feature extractors was evaluated. The results for MiT-B1 show that deeper extractors lead to lower mean intersection-over-union ratios (MUI), while progressively reducing depth improves performance, with the best results achieved using only the first stage. ResNet50 outperformed MiT-B1, demonstrating the effectiveness of convolutional models in extracting initial cues. Increasing the size of the MiT extractor did not bring further improvement, further reinforcing the advantages of convolutional extractors. These findings suggest that pre-trained convolutional cue extractors focusing on early features are more beneficial for cue segmentation.

[0060] Table 2: The impact of different prompt extractor architectures and the number of layers used.

[0061] Feature Extractor Types Number of feature layers used mIoU MiT-B1 Stages {1,2,3,4} 59.2％ MiT-B1 Stage {1,2,3} 59.5％ MiT-B1 Phase {1,2} 59.6％ MiT-B1 Phase {1} 59.7％ MiT-B2 Phase {1} 59.6％ MiT-B4 Phase {1} 59.2％ MiT-B5 Phase {1} 59.3％ ResNet50 Stages {1,2,3,4} 58.9％ ResNet50 Stage {1,2,3} 59.5％ ResNet50 Phase {1,2} 59.9％ ResNet50 Phase {1} 60.1％

[0062] Step 2: For the initial prompts output by the multimodal embedding module and the RGB features input by the Segformer backbone network, a linear mapping is used to project them onto the low-rank subspace to complete feature alignment. A mixing matrix is ​​introduced to fuse the RGB features of the low-rank subspace with the prompt information to generate new prompt information. The new prompt information is then fused with the RGB features encoded by the Segformer backbone network. Without modifying the Segformer backbone network architecture, a lightweight multi-subspace alignment and mixing prompt module is introduced to provide auxiliary modal feature information for the backbone model, effectively improving semantic segmentation performance.

[0063] Step 2.1: Project the initial cue output by the multimodal embedding module and the RGB features input by the Segformer backbone network to complete multi-subspace alignment. Given an RGB image... Where C in H and W represent the number of RGB channels, height, and width of the image, respectively. An enhanced image is obtained using a data augmentation method involving random horizontal flipping and perturbation. The enhanced data is normalized and then input into the embedding module, where it is divided into 4×4 blocks. These blocks are then used as input to the Segformer backbone network.

[0064] The RGB modal features and auxiliary modal cue information are aligned into a low-rank subspace using a linear projection method. The aligned features are then fused through an addition operation. The multi-subspace alignment expression is shown in equation (2):

[0065]

[0066] Where i∈{1,2,…,N} represents the index of the layer. P i This represents the i-th multi-subspace hybrid hint module. i Indicates the Lth i The layer's prompt message, h i Indicates the Lth i The output hidden state of the layer, It is a downsampling module. This is an upsampling module. Furthermore, let d be the original feature space, r << d, and r is the dimension of the low-rank aligned subspace. Using a smaller value for r allows the cueing module to be parameterized.

[0067] To improve information utilization efficiency, multiple pairs of downward and upward projection modules are introduced. Each pair of projections aligns the RGB modality and auxiliary modality in a unique subspace, thereby improving the utilization rate of useful information. By reducing the rank of each subspace to maintain the parameter efficiency of the overall model, the RGB modality features and auxiliary modality cue information are projected into multiple subspaces, as shown in equations (3) and (4):

[0068] [W rgb,1 h i-1 W rgb,2 h i-1 ,...,W rgb,n h i-1 ], (3)

[0069] [W f,1 h i-1 W f,2 h i-1 ,...,W f,n h i-1 ], (4)

[0070] in, n represents the number of subspaces used. Dividing the original rank r by n ensures that introducing multiple subspaces does not introduce additional parameters.

[0071] Step 2.2: Introduce a hybrid matrix to fuse the RGB features of the low-rank subspace with the cue information to generate new cue information. The new cue information is then fused with the RGB features encoded by the Segformer backbone network. Without modifying the Segformer backbone network architecture, a lightweight multi-subspace alignment and hybrid cue module is introduced to provide auxiliary modality feature information for the backbone model, effectively improving semantic segmentation performance.

[0072] In the low-rank subspace alignment process, a hybrid cue generation method is introduced to enable information interaction between RGB features and deep modal features in the cue branch, thereby improving the expressive power of the newly generated auxiliary cue information. In equations (3) and (4), the RGB modal features and auxiliary modal cue information are projected onto multiple subspaces, and W... rgb,n h i-1 Represented as Introduce a mixing matrix Used for exchanging information between different subspaces. Features after mixing. The expression is shown in equation (5):

[0073]

[0074] Among them, M i,j This represents the value of the corresponding element in the mixing matrix M. Similarly, we get the hint e. i-1 low-rank representation Information from the RGB modality is injected into the low-rank cue through addition, and then linear projection is used. Projecting it back into the original feature space generates a new cue e. i Main layer L i The RGB features are fused with the new cue through addition. The expression is shown in equation (6):

[0075]

[0076] like Figure 2 As shown, in the multi-subspace alignment and blending cue module, the input features are the cue information and hidden states from the previous layer. A linear projection layer reduces the dimensionality of the input features to a low-rank subspace, which is then divided into four subspaces. Subsequently, a blending matrix is ​​used to exchange and fuse the subspace features of the RGB modality. The fused features are combined through an additive operation and then restored to the original feature dimension through linear projection to generate the final cue.

[0077] This study investigates the effects of different subspace cue blending techniques applied to the RGB or depth modality. The results are shown in Table 3. The model achieves an average intersection-union (MUI) of 59.3% without subspace blending. Introducing subspace blending into the RGB modality significantly improves performance to 60.1%, demonstrating the effectiveness of optimizing RGB cues through subspace alignment. Conversely, applying subspace blending only in the auxiliary modality yields only a small improvement of 59.7%. Enabling subspace blending in both modalities does not provide further improvement; instead, it slightly reduces the mIoU to 59.8%, slightly lower than the setting that only optimizes the RGB modality. These results indicate that enhancing the RGB modality through subspace blending plays a more crucial role in improving segmentation performance. In contrast, applying the same strategy in the auxiliary modality yields limited improvement, while applying it in both modalities may introduce redundant or conflicting information. Therefore, prioritizing subspace blending in the RGB cueing is recommended for optimal results.

[0078] Table 3: The impact of different prompt extractor architectures and the number of layers used.

[0079] method mIoU No mixing 59.3％ RGB mode mixing 60.1％ Auxiliary mode mixing 59.7％ RGB + auxiliary mode dual hybrid 59.8％

[0080] Step 3: Use the Segformer backbone network's multi-resolution self-attention encoder to encode RGB image features. Auxiliary image information such as depth image and infrared image is generated as initial prompts through the multimodal embedding module. The initial prompt information and RGB image features are fused together through the multi-subspace alignment and hybrid prompt module to form new prompt information and RGB image features. This efficiently fuses the semantic information of RGB images and auxiliary modalities, improves the model's inference speed, and enhances semantic segmentation performance.

[0081] The process is as follows Figure 3 As shown, given an RGB image After data preprocessing into 4×4 blocks, the multi-resolution self-attention encoder of the Segformer backbone network takes the divided blocks as input to obtain the original image resolution in {1 / 4, 1 / 8, 1 / 16, 1 / 32}. The feature expression of the extracted image is shown in Equation (7):

[0082]

[0083] in, This refers to RGB image blocks after data preprocessing. This is a multi-resolution self-attention encoder, where θ represents the encoder parameters, f rgb This represents the extracted image features, corresponding to different levels of hidden states h. i From equation (1), the initial prompt information e0 can be obtained. The obtained e0, together with the embedded original RGB input h0, is used by the prompt module to generate a new prompt e1. After passing through the model layer, e1 is used to obtain the output hidden state h1, as shown in equations (8) and (9):

[0084] e1=P(h0,e0), (8)

[0085] h1=L(h0,e1), (9)

[0086] Where P and L represent the multi-subspace alignment and hybrid cueing module and the multi-resolution self-attention encoder layer, respectively. For different resolutions, the input feature is the cueing information e from the previous layer. i-1 and hidden state h i-1 The input features are reduced to a low-rank subspace by linear projection and then divided into multiple subspaces.

[0087] The input data was normalized, and data augmentation strategies included random flipping and random scaling within a range of [0.5, 1.75]. Auxiliary modal data was uniformly converted to three-channel feature representations. The backbone model used a pre-trained Segformer-B5 model to extract RGB image information. Auxiliary modalities were projected into the feature space through a multimodal embedding module to initialize cue information. Subsequently, these cue information were fused at multiple scales through four hybrid cue modules, effectively integrating depth information into different layers of the RGB backbone network. The decoder output was matched to the dimension of the actual predicted number of classes through a linear mapping layer. The training iterations were 500, with an initial learning rate of 0.04 and a multinomial decay scheme. SGD was used as the optimizer, with weight decay at 5e-4. Data augmentation techniques such as random flipping and random cropping were used during training. The backbone network parameters were frozen, and only the parameters of the multimodal embedding module, the multi-subspace hybrid module, and the final output linear mapping layer were trained.

[0088] During inference, multi-scale inference is used to enhance the results for comparison with other methods, and the mean intersection over union (mIoU) is used to evaluate the segmentation accuracy, as shown in equations (10) and (11):

[0089]

[0090] Where TP, FP, and FN represent the number of correctly predicted, false positives, and false negatives, respectively, and N is the total number of categories. The multi-scale inference test results are shown in Table 4. The depth image modality effectively enhances the feature representation of the RGB modality. In terms of multi-modal fusion, the method improves IoU by 4.3 mIoU compared to the two-stream semantic segmentation architecture and by 1.9 mIoU compared to the cue-based semantic segmentation method. Furthermore, compared to the traditional dual-path feature extraction architecture, the cue-based hybrid module of the auxiliary modality has negligible parameter count compared to the backbone architecture, requiring only half the number of parameters.

[0091] Table 4: Performance comparison of different modality segmentation methods.

[0092] Model mIoU Single-modal segmentation architecture 52.3％ Two-stream semantic segmentation architecture 56.9％ Hints on learning semantic segmentation methods 59.3％ Multi-subspace hybrid hinting method 61.2％

[0093] Simultaneously, ablation experiments were conducted on key hyperparameters. The rank reduction ratio was analyzed. The ratio controls the dimensionality of intermediate features and effectively determines the rank of the low-rank subspace. A larger ratio corresponds to a smaller rank and fewer trainable parameters. The results, shown in Table 5, indicate that increasing the dimensionality reduction ratio from 1 to 4 improves mIoU from 59.4% to 60.1%, demonstrating that moderate rank reduction can improve feature efficiency. However, when the ratio is further increased to 8, the performance slightly decreases to 59.6%, suggesting that too small a rank may lead to information loss and affect performance. The optimal trade-off is achieved at a ratio of 4, which is selected as the final parameter configuration.

[0094] Table 5: Ablation study on the rank reduction ratio.

[0095]

[0096] This study investigates the impact of the number of subspaces, *n*, used for blending cues. Unlike the rank-based dimensionality reduction ratio, this parameter does not affect the model's parameter efficiency. The results, shown in Table 6, demonstrate that increasing the number of subspaces from 1 to 4 progressively improves performance, reaching a peak mIoU of 60.1%. However, when the number of subspaces is further increased to 8, performance declines, indicating that overly complex cue blending may introduce unnecessary redundancy.

[0097] Table 6: Ablation experiments on the number of mixed subspaces.

[0098]

[0099]

[0100] Therefore, the present invention discloses a hybrid prompting architecture fusion method for multimodal semantic segmentation. It constructs a multimodal embedding module based on an RGB pre-trained model to initialize the embedding information of auxiliary modalities, introduces a multi-subspace aligned hybrid prompting fusion method, generates multi-level prompting information, realizes multimodal information fusion, effectively improves model accuracy, reduces model computational complexity and training cost, and provides theoretical support for the development of multimodal information fusion models.

[0101] The above detailed description further illustrates the purpose, technical solution, and beneficial effects of the invention. It should be understood that the above description is merely a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A hybrid prompting architecture fusion method for multimodal semantic segmentation, characterized in that: Includes the following steps, Step 1: Construct a multimodal embedding module based on the RGB pre-trained model to generate high-quality initial prompt information from the input auxiliary image, effectively encode the semantic information of the auxiliary modality, avoid interference from redundant information in the original auxiliary image, and improve the adaptability of different modal auxiliary information to the RGB feature space. Step 2: For the initial prompts output by the multimodal embedding module and the RGB features input by the Segformer backbone network, a linear mapping is used to project them onto the low-rank subspace to complete feature alignment. A mixing matrix is ​​introduced to fuse the RGB features of the low-rank subspace with the prompt information to generate new prompt information. The new prompt information is then fused with the RGB features encoded by the Segformer backbone network. Without modifying the Segformer backbone network architecture, a lightweight multi-subspace alignment and mixing prompt module is introduced to provide auxiliary modal feature information for the backbone model, effectively improving semantic segmentation performance. In the low-rank subspace alignment process, a hybrid cue generation method is introduced to enable information interaction between RGB features and auxiliary modal features in the cue branch, thereby improving the expressive power of the newly generated auxiliary cue information. In equations (3) and (4), the RGB modal features and auxiliary modal cue information are projected onto multiple subspaces, thereby... Represented as , For downsampling module, Indicates the first The layer outputs the hidden state and introduces a blending matrix. Used to exchange information between different subspaces; the hybrid features The expression is shown in equation (5): （5） in, Represents the mixture matrix The value of the corresponding element; similarly, a prompt is received. low-rank representation Information from the RGB modalities is injected into the low-rank cue through addition, and then linear projection is used. Projecting it back into the original feature space generates new hints. Main layer The RGB features are fused with the new cue through addition; the expression is shown in equation (6): （6） Step 3: Use the Segformer backbone network's multi-resolution self-attention encoder to encode RGB image features. Auxiliary image information is used to generate initial prompts through the multimodal embedding module. The initial prompt information and RGB image features are fused together through the multi-subspace alignment and hybrid prompt module to form new prompt information. This process fuses the semantic information of the RGB image with the auxiliary modality, improving the model's inference speed and enhancing semantic segmentation performance.

2. The hybrid prompting architecture fusion method for multimodal semantic segmentation as described in claim 1, characterized in that: The implementation method for step one is as follows: Given depth image, thermal infrared image, and auxiliary image information ,in , , and represent the number of channels, height, and width of an image, respectively. An enhanced image is obtained using channel alignment transformation and data augmentation methods. ,in These represent the number of channels in an RGB image; The initial blocks of a pre-trained ResNet-50 are used as the multimodal embedding network, and the initial cue is extracted by aligning it with RGB image features through a mapping layer. Initial prompt The extraction expression is shown in equation (1): （1） in, Indicates auxiliary modal input information, As the initial block for pre-training ResNet-50, the shallow architecture of the pre-trained ResNet-50 network is used to extract features of auxiliary modality information, so that the extracted auxiliary modality features match the distribution of shallow information of the RGB backbone network. As a mapping layer, a single linear layer is used to map the extracted features to a specific feature space, consistent with the dimension of the RGB backbone model, to achieve high-quality initial prompt information generation.

3. The hybrid prompting architecture fusion method for multimodal semantic segmentation as described in claim 2, characterized in that: The second step is implemented as follows: Step 2.1: Project the initial cue output by the multimodal embedding module and the RGB features input by the Segformer backbone network to complete multi-subspace alignment; given an RGB image ,in , , and represent the number of RGB channels, height, and width of the image, respectively. An enhanced image is obtained using a data augmentation method involving random horizontal flipping and perturbation. The enhanced, normalized data is then input into the embedding module and divided into segments of size [size missing]. The blocks are used as input to the Segformer backbone network; The RGB features input to the Segformer backbone network and the auxiliary modal features in the cue branch exist in different feature spaces, requiring alignment of these mismatched feature spaces. A low-rank subspace alignment method is introduced, which uses linear projection to align the RGB modal features and the cue information of the auxiliary modality into a low-rank subspace. The aligned features are then fused through an addition operation. To improve information utilization efficiency, multiple pairs of downward and upward projection modules are introduced; each pair of projections aligns the RGB modes and auxiliary modes in a unique subspace, thereby improving the utilization rate of useful information. By reducing the rank of each subspace to maintain the parameter efficiency of the overall model, RGB modal features and auxiliary modal cue information are projected onto multiple subspaces.

4. The hybrid prompting architecture fusion method for multimodal semantic segmentation as described in claim 3, characterized in that: In step 2.1, the multi-subspace alignment expression is shown in equation (2): （2） in, Index of the presentation layer; Indicates the first A multi-subspace hybrid prompt module; Indicates the first The output hidden state of the layer, It is a downsampling module. It is an upsampling module; in addition, for For the original feature space, ,and It is the dimension of the low-rank aligned subspace.

5. The hybrid prompting architecture fusion method for multimodal semantic segmentation as described in claim 4, characterized in that: In step 2.1, the RGB modal features and auxiliary modal cue information are projected into multiple subspaces, as shown in equations (3) and (4): （3） （4） in, , Indicates the number of subspaces used, and the original rank Divide by Ensure that introducing multiple subspaces does not introduce additional parameters.

6. The hybrid prompting architecture fusion method for multimodal semantic segmentation as described in claim 4, characterized in that: The method for implementing step three is as follows: Given an RGB image After data preprocessing, the size is The Segformer backbone network's multi-resolution self-attention encoder takes the divided blocks as input to obtain the... The original image resolution is used to extract the image feature expression as shown in equation (7): （7） in, This is a RGB image block after data processing. For multi-resolution self-attention encoders, Indicates the parameters of the encoder, This represents the extracted image features, corresponding to different levels of hidden states. The initial prompt information can be obtained from equation (1). , obtained As an initial prompt and embedded raw RGB input A new prompt is generated after the prompt module. , After passing through the model layer, the output hidden state is obtained. The expressions are shown in equations (8) and (9): （8） （9） in, and These represent the multi-subspace alignment and hybrid cueing module and the multi-resolution self-attention encoder layer, respectively. For different resolutions, the input features are the cue information from the previous layer. and hidden state The input features are reduced to a low-rank subspace through linear projection and then divided into multiple subspaces. A mixture matrix is ​​used to exchange and fuse the subspace features of the RGB modalities. The fused features are combined through addition and then restored to the original feature dimension through linear projection to generate the final prompt. Message With the previous hidden state The sums are input into the encoder layer; the expressions are shown in equations (10), (11), (12), (13), and (14): （10） （11） (reshape (12) （13） （14） in and This is a mapping layer; `reshape` is a transformation operation. Flattening operation, This is an intermediate state of the hidden state. For upsampling layer, the mixing matrix As a router, different importance is assigned to each subspace through different weight mixtures; at the same time, the reparameterization of the hybrid prompt module enables the model to utilize a multi-branch architecture, further promoting more effective propagation of auxiliary information and helping the model achieve better semantic segmentation performance. The obtained hidden state, i.e. the fused features, is input into the multilayer perceptron encoder. The MLP layer unifies the multilayer feature channel dimensions, and after upsampling, the channels are concatenated. The MLP layer is used to process the concatenated features to obtain the reconstructed pixel-level segmentation mask. The expression is shown in Equation (15): （15） In the formula, For linear MLP modules, For upsampling module, For connection operations, This represents the segmentation result corresponding to the r-fused features; The aforementioned multimodal semantic segmentation hybrid prompting architecture can achieve efficient fusion of infrared images, depth images, and RGB images, improving semantic segmentation performance. At the same time, it further reduces the complexity and inference cost of the multimodal segmentation model.

7. The hybrid prompting architecture fusion method for multimodal semantic segmentation as described in claim 6, characterized in that: In step three, For the training loss function of the semantic segmentation task, the cross-entropy loss function is used, and its expression is shown in equation (16): （16） in, The total number of pixels in the image. To divide the total number of categories, For the first 1 pixel in The true label of the category, This indicates that the model predicts the pixel belongs to The probability of the category.

8. A hybrid prompting architecture fusion method for multimodal semantic segmentation as described in claims 1, 2, 3, 4, 5, 6, or 7, characterized in that: The input auxiliary images include depth images and thermal infrared images.

Images