A RGB-D image semantic segmentation method based on multimodal feature fusion

Abstract

The present invention belongs to the field of computer vision, and specifically relates to a RGB-D image semantic segmentation method based on multimodal feature fusion. Due to the inherent differences between RGB and depth features, how to more effectively fuse the two features remains a problem to be solved. In order to solve this problem, an attention-guided multimodal cross-fusion segmentation network (ACFNet) is proposed. The present invention adopts an encoder-decoder structure to encode the depth map into an HHA image, designs an asymmetric two-stream feature extraction network, and uses ResNet-101 and ResNet-50 as the backbone networks of the RGB and depth encoders, respectively, and adds a global-local feature extraction module (GL) to the RGB encoder. In order to effectively fuse RGB and depth features, an attention-guided multimodal cross-fusion module (ACFM) is proposed to better utilize the fused enhanced feature representation in multiple stages.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision, and in particular relates to an RGB-D image semantic segmentation method based on multimodal feature fusion. Background Art

[0002] Image semantic segmentation is a basic task that is the focus of research in the field of computer vision. Its purpose is to assign a category label to each pixel of the image to achieve pixel-level scene understanding. It is widely used in medical imaging, autonomous driving, face recognition, target detection, etc. In general, due to the complex indoor scene environment, cluttered and trivial objects, and severe occlusion, the performance requirements for segmentation algorithms are higher and more challenging. Depending on whether deep neural networks are applied, image semantic segmentation methods can be divided into traditional methods and deep learning-based methods. Traditional methods can be divided into six categories: threshold-based, edge-based, region-based, clustering-based, graph theory-based, and specific theory-based methods. However, segmentation tasks that require a lot of semantic information are not suitable for traditional image segmentation methods. Therefore, methods that introduce deep learning into the field of image segmentation have emerged. This method can make full use of the semantic information of the image.

[0003] The early image segmentation method was to extract feature information from color images (RGB images) for segmentation. In 2014, Long et al. proposed a fully convolutional neural network (FCN), which can use images of any size as input data. It is the first segmentation network model that achieves end-to-end processing. When the FCN network is downsampled, the image resolution is reduced, some feature information is lost, and the segmentation effect is not ideal. To address this problem, Lin et al., Ronneberger et al., and Zhao et al. proposed methods such as using a decoder to restore image resolution, combining high-level semantic information and low-level detail features, and aggregating contextual information of different scales to improve segmentation performance. The problem with RGB image segmentation is that the image only provides two-dimensional visual information such as color, texture, and shape. For objects with similar colors and textures, there is a lack of corresponding geometric information, which easily causes many pixel classification errors and cannot obtain good semantic segmentation results. Therefore, the study of RGB-D image semantic segmentation methods has been launched.

[0004] Simply concatenating or adding RGB and depth information to an existing single-modality RGB segmentation network improves the segmentation effect, but does not fully utilize the geometric information provided by the depth map, and cannot effectively fuse RGB and depth information. Using two independent network branches to process RGB and depth information respectively and designing a feature fusion module is conducive to the efficient use of RGB and depth features. Although these methods have achieved good segmentation results, how to fully utilize the complementarity and differences between the two modalities, and how to enhance the interaction, transmission, and feature extraction capabilities of the encoder are still issues that need to be further studied. Summary of the invention

[0005] The purpose of this invention is to provide an efficient RGB-D indoor scene semantic segmentation method, introduce deep image enhancement image feature representation, so as to solve the limitation of RGB image segmentation. According to the end-to-end RGB-D segmentation network structure, a RGB-D indoor scene semantic segmentation network based on multimodal feature fusion is designed to efficiently solve the semantic segmentation task in the field of computer vision.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] A RGB-D image semantic segmentation method based on multimodal feature fusion includes the following steps:

[0008] Step 1: data preprocessing, converting the single-channel depth image into a three-channel HHA image;

[0009] In step 2, the RGB and HHA images are taken as input data and input into the attention-guided multimodal cross-fusion segmentation network model (ACFNe), which follows an encoder-decoder structure. The encoder extracts semantic features from the input, and the decoder uses upsampling technology to restore the input resolution and assign a semantic category to each input pixel.

[0010] Furthermore, the HHA image in step 1 contains more spatial information, and the three channels respectively represent the height of the horizontal disparity above the ground, the local surface normal of the pixel, and the angle of the inferred gravity direction.

[0011] Furthermore, in step 2, the encoder uses an asymmetric dual-stream branch for RGB and HHA images, including an RGB encoder and a depth encoder. The RGB encoder and the depth encoder use ResNet-101 network and ResNet-50 network as the backbone network respectively. The ResNet-101 network and the ResNet-50 network both include 7×7 convolution, maximum pooling operation and four stages of Stage 1, Stage 2, Stage 3 and Stage 4. The basic blocks constituting the backbone network are improved, that is, a Maxpool parallel module is added to Bottleneck, denoted as MP_Bottleneck, and the feature extraction performance is improved by increasing the network width, as shown in formula (1):

[0012] F E_out =W3(Cat(W2(W1(F in )),M_P(W1(F in )) (1)

[0013] Among them, F in ∈R H×W×C, H, W and C represent the height, width and number of channels of the feature map respectively, W1, W2, W3 represent 1×1, 3×3, 1×1 convolution operations respectively, Cat represents concatenation operation, M_P represents maximum pooling operation, F E_out Represents the output of each stage of the ResNet network.

[0014] Furthermore, a global feature extraction module is added to the four stages of Stage 1, Stage 2, Stage 3, and Stage 4 of the RGB encoder, and the global feature extraction module is used to process F E_out Extract global features, and combine the final global features and local features by adding them together to form a global-local feature extraction module. The above process is shown in equations (2) to (4):

[0015] F GL =F E_out +GC(F E_out ) (2)

[0016] GC=x+Conv1×1(ReLu(LN(Conv1×1(y)))) (3)

[0017] y=x×SoftMax(Conv1×1(x)) (4)

[0018] Among them, x represents the input of the global feature extraction module, GC represents the global feature, F GL Represents global-local features, Conv 1×1 represents a 1×1 convolution operation, and LN represents a batch normalization operation.

[0019] Furthermore, the encoder further includes a feature fusion module, which fuses RGB features and depth features after the 7×7 convolution and each stage, and transmits the fused features to the RGB encoder and the depth encoder, reweights the RGB features and the depth features at each stage of the encoder, and introduces an attention mechanism in the feature fusion module to perform attention operations on the channel and spatial dimensions, as shown in equations (5) to (8):

[0020] F a_rgb =CBAM(F rgb_in ) (5)

[0021] F a_hha =CBAM(F hha_in ) (6)

[0022] CBAM=ca×σ(Conv 7×7 (Cat(Avgpool(ca),MaxPool(ca))) (7)

[0023] ca=x×σ(MLP(Avgpool(x)+MLP(Maxpool(x))) (8)

[0024] Among them, CBAM means that the attention mechanism adopts the CBAM module, x represents the input of the CBAM module, and F rgb_in 、F hha_in They represent the RGB features and depth features of the input feature fusion module, respectively. a_rgb 、F a_hha represents the feature output after being processed by the CBAM attention mechanism, σ represents the Sigmoid activation function, Conv 7×7 represents a 7×7 convolution operation, Cat represents a concatenation operation, Avgpool and Maxpool represent average pooling and maximum pooling operations respectively, and MLP represents a multi-layer perceptron consisting of two 1×1 convolutional layers.

[0025] Furthermore, the feature fusion module uses a cross-multiplication method to select one modality feature processed by the attention mechanism and multiply it by another unprocessed modality feature, selects distinguishable information from one modality to assist in correcting the other modality, and then performs a cross residual connection, cross-adds the feature processed by the CBAM module and the original modality information corrected by the other modality information, further reuses the detail information lost in the network processing process, enhances the complementarity between the two modalities, and finally fuses the two modality information by adding them, as shown in equations (9) to (11):

[0026] F m_rgb =F rgb_in ×F a_hha (9)

[0027] F m_hha =F hha_in ×F a_rgb (10)

[0028] F f_out =(F a_rgb +F m_hha )+(F a_hha +F m_rgb ) (11)

[0029] Among them, × represents the element-by-element multiplication operation, F m_rgb 、F m_hha They represent the feature outputs corrected by the multiplication operation, F f_out Represents the final output result of the feature fusion module.

[0030] Furthermore, the decoder is composed of 5 identical decoder units, each of which is composed of a convolution layer + a BN layer + a ReLU activation function and an upsampling operation, and a short-circuit connection is added therein to reuse the information lost during the training process. The short-circuit connection uses a 1×1 convolution to change the number of output channels of the encoder stage so that it is the same as the number of channels of the corresponding stage of the decoder and can perform an addition operation.

[0031] Furthermore, the convolution layer includes one 1×1 convolution and two 3×3 convolutions, and the upsampling operation adopts a bilinear interpolation method. After 5 upsampling operations, the image resolution is restored to the input size.

[0032] Furthermore, the number of input channels of the first decoder unit is 2048, and the number of channels gradually decreases as the image resolution increases. The last decoder unit is used to output the network prediction result, remove the batch normalization operation after the last convolution layer, and set the number of output channels to the number of semantic categories.

[0033] Compared with the prior art, the present invention has the following advantages:

[0034] (1) In order to improve the deficiency of ResNet network in extracting local features of images, a global-local feature extraction module is added to the RGB encoder.

[0035] (2) In order to effectively fuse RGB and depth features, the attention mechanism is applied to the feature fusion module, and an attention-guided feature cross-fusion module is proposed, which fully utilizes the complementarity and differences between the two modalities to better utilize the fused enhanced feature representation in multiple stages.

[0036] (3) The ACFNet proposed in this paper outperforms most existing methods and shows competitive segmentation performance on the NYUD V2 dataset. It can better adapt to the complexity and diversity of indoor scenes, which shows the effectiveness of the proposed method. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 The overall architecture of the ACFNet network model of the present invention;

[0038] Figure 2 Detailed design of the encoder structure, where (a) is MP_Bottleneck, (b) is the global feature extraction module (GC), and (c) is the global-local feature extraction module (GL);

[0039] Figure 3 The specific structure of the feature fusion module, where (a) is the CBAM module and (b) is the ACFM fusion module;

[0040] Figure 4is a decoder unit;

[0041] Figure 5 This is a visual comparison of the segmentation results of the present invention on the NYUD V2 dataset. DETAILED DESCRIPTION

[0042] Example 1

[0043] A RGB-D image semantic segmentation method based on multimodal feature fusion includes the following steps:

[0044] Step 1: Data preprocessing: converting the single-channel depth image into a three-channel HHA image, where the three channels represent the height of the horizontal disparity above the ground, the local surface normal of the pixel, and the angle of the inferred gravity direction;

[0045] Step 2: Take the RGB and HHA images as input data and input them into the attention-guided multimodal cross-fusion segmentation network model (such as Figure 1 As shown in the figure, the model follows an encoder-decoder structure, where the encoder extracts semantic features from the input, and the decoder uses upsampling technology to restore the input resolution and assign a semantic category to each input pixel.

[0046] The encoder uses an asymmetric dual-stream branch for RGB and HHA images, including an RGB encoder and a depth encoder. The RGB encoder and the depth encoder use ResNet-101 network and ResNet-50 network as the backbone network respectively. The ResNet-101 network and the ResNet-50 network both include 7×7 convolution, maximum pooling operation and four stages of Stage1, Stage2, Stage3 and Stage4, and improve the basic blocks constituting the backbone network, that is, add the Maxpool parallel module in Bottleneck, denoted as MP_Bottleneck, as shown in Figure 2 As shown in (a), the feature extraction performance is improved by increasing the network width, as shown in formula (1):

[0047] F E_out =W3(Cat(W2(W1(F in )),M_P(W1(F in )) (1)

[0048] Among them, F in ∈R H×W×C , H, W and C represent the height, width and number of channels of the feature map respectively, W1, W2, W3 represent 1×1, 3×3, 1×1 convolution operations respectively, Cat represents concatenation operation, M_P represents maximum pooling operation, F E_out Represents the output of each stage of the ResNet network.

[0049] A global feature extraction module (GC) is added to the four stages of Stage 1, Stage 2, Stage 3, and Stage 4 of the RGB encoder, such as Figure 2 As shown in (b), the global feature extraction module is used to process F E_out Extract global features, and combine the final global features and local features by adding them together to form a global-local feature extraction module (GL), such as Figure 2 As shown in (c), the above process is shown in equations (2) to (4):

[0050] F GL =F E_out +GC(F E_out ) (2)

[0051] GC=x+Conv1×1(ReLu(LN(Conv1×1(y)))) (3)

[0052] y=x×SoftMax(Conv1×1(x)) (4)

[0053] Among them, x represents the input of the global feature extraction module, GC represents the global feature, F GL Represents global-local features, Conv 1×1 represents a 1×1 convolution operation, and LN represents a batch normalization operation.

[0054] The encoder also includes a feature fusion module, which fuses RGB features and depth features after the 7×7 convolution and each stage, and transmits the fused features to the RGB encoder and the depth encoder. The RGB features and the depth features are reweighted at each stage of the encoder, and an attention mechanism is introduced in the feature fusion module to perform attention operations on the channel and spatial dimensions, as shown in equations (5) to (8):

[0055] F a_rgb =CBAM(F rgb_in ) (5)

[0056] F a_hha =CBAM(F hha_in ) (6)

[0057] CBAM=ca×σ(Conv 7×7 (Cat(Avgpool(ca),MaxPool(ca))) (7)

[0058] ca=x×σ(MLP(Avgpool(x)+MLP(Maxpool(x))) (8)

[0059] Among them, CBAM means that the attention mechanism adopts CBAM module (such as Figure 3 As shown in (a), x represents the input of the CBAM module, F rgb_in 、F hha_in They represent the RGB features and depth features of the input feature fusion module, respectively. a_rgb 、F a_hha represents the feature output after being processed by the CBAM attention mechanism, σ represents the Sigmoid activation function, Conv 7×7 represents a 7×7 convolution operation, Cat represents a concatenation operation, Avgpool and Maxpool represent average pooling and maximum pooling operations respectively, and MLP represents a multi-layer perceptron consisting of two 1×1 convolutional layers.

[0060] The feature fusion module is as follows Figure 3 As shown in (b), a cross-multiplication method is used to select one modality feature processed by the attention mechanism and multiply it by another modality feature that has not been processed. Distinguishable information is selected from one modality to assist in correcting the other modality. After that, a cross residual connection is performed. The features processed by the CBAM module are cross-added with the original modality information corrected by the other modality information. The detail information lost in the network processing process is further reused to enhance the complementarity between the two modalities. Finally, the two modal information is fused by adding them together, as shown in equations (9) to (11):

[0061] F m_rgb =F rgb_in ×F a_hha (9)

[0062] F m_hha =F hha_in ×F a_rgb (10)

[0063] F f_out =(F a_rgb +F m_hha )+(F a_hha +F m_rgb ) (11)

[0064] Among them, × represents the element-by-element multiplication operation, F m_rgb 、F m_hha They represent the feature outputs corrected by the multiplication operation, F f_out Represents the final output result of the feature fusion module.

[0065] The decoder consists of 5 identical decoder units, such as Figure 4As shown in the figure, each decoder unit consists of a convolution layer + BN layer + ReLU activation function and an upsampling operation, and a short-circuit connection is added to it to reuse the information lost during the training process. The short-circuit connection uses a 1×1 convolution to change the number of output channels of the encoder stage so that it is the same as the number of channels of the corresponding stage of the decoder and can be added.

[0066] The convolution layer includes one 1×1 convolution and two 3×3 convolutions. The upsampling operation adopts a bilinear interpolation method. After 5 upsampling operations, the image resolution is restored to the input size.

[0067] The number of input channels of the first decoder unit is 2048, and the number of channels gradually decreases as the image resolution increases. The last decoder unit is used to output the network prediction result, remove the batch normalization operation after the last convolutional layer, and set the number of output channels to the number of semantic categories, that is, 40.

[0068] Example 2

[0069] The segmentation network model proposed in this paper is applied to the public RGB-D indoor dataset NYUD V2, and the effectiveness of the network model is demonstrated through experiments.

[0070] The NYUD V2 dataset contains 1449 labeled RGB-D image pairs from 464 different indoor scenes in three cities, of which 796 image pairs are used for training and 654 image pairs are used for testing. The classification method is to classify 894 different objects into 40 categories. The effectiveness of the segmentation network ACFNet proposed in this invention is demonstrated by providing segmentation visualization comparison results. The network structure after removing the feature fusion module ACFM and the global-local feature extraction module (GL) is recorded as the baseline network (Baseline). Figure 5 Figure 2 shows some visualization results of ACFNet on the public dataset NYUD V2, where the first, second and third columns represent RGB images, HHA images and semantic labels respectively, and the fourth and fifth columns represent the semantic segmentation results of the baseline network and ACFNet respectively, and the comparison areas are marked with boxes.

[0071] The semantic segmentation performance metrics used in the present invention are the common pixel accuracy (PixAcc) and average intersection-over-union (mIou), where PixAcc is the ratio of correctly labeled pixels to total pixels, and its calculation formula is shown in formula (12); mIou is the average value of the intersection-over-union (Iou) of all object categories, which is essentially the intersection-over-union ratio of the ground truth and the predicted value, and its calculation formula is shown in formula (13):

[0072]

[0073]

[0074] Among them, k+1 represents the total number of categories, i represents the true value, j represents the predicted value, and p ij It indicates the number of predicted values ​​of true value i as j, p ii represents the number of correct predictions, p ij and p ji They represent the number of false positives (FP) and false negatives (FN), respectively.

[0075] The ACFNet proposed in the present invention is compared with other existing methods on the NYUD V2 dataset. The NYUD V2 dataset is divided into 40 semantic categories. The results are shown in Table 1, which lists the segmentation performance of the proposed method and other methods in terms of pixel accuracy (PixAcc) and mean intersection-over-union (MIOU).

[0076] Table 1 Comparison with other methods on the NYUD V2 dataset

[0077]

[0078] As shown in Table 1, the proposed method uses ResNet-101+ResNet-50 as the backbone network and conducts experiments on the NYUD V2 dataset of 40 semantic categories. The two evaluation indicators, pixel accuracy and average intersection-over-union ratio, are 77.0% and 51.5%, respectively.

[0079] Compared with RefineNet, which only uses RGB images as input, the pixel accuracy and average intersection-over-union ratio are improved by 3.4% and 5.0% respectively. The experimental results show that adding deep features can improve the segmentation performance to a certain extent.

[0080] Compared with ACNet and CANet, which also use the attention mechanism, the average intersection-over-union ratio is improved by 3.2% and 0.3% respectively;

[0081] Compared with SGNet, which uses ResNet-101 as the backbone network, not only is the network model simplified, but the pixel accuracy and average intersection-over-union ratio are also improved by 0.2% and 0.5% respectively;

[0082] Compared with RAFNet, which also uses a bidirectional multi-stage propagation strategy for fusion features, the pixel accuracy and average intersection-over-union ratio are improved by 3.2% and 4.0% respectively;

[0083] Compared with the newer segmentation algorithms TSNet and RGB×D, although the number of network layers is relatively large, the average intersection-over-union ratio is also greatly improved, by 5.4% and 0.4% respectively.

[0084] In summary, the ACFNet proposed in this paper outperforms most of the existing methods, shows competitive segmentation performance on the NYUD V2 dataset, and can better adapt to the complexity and diversity of indoor scenes, which shows the effectiveness of the method proposed in this paper.

Claims

1. A RGB-D image semantic segmentation method based on multimodal feature fusion, characterized in that: The following steps are involved: Step 1: data preprocessing, converting the single-channel depth image into a three-channel HHA image; Step 2: Take the RGB and HHA images as input data and input them into the attention-guided multimodal cross-fusion segmentation network model. The model follows an encoder-decoder structure. The encoder extracts semantic features from the input, and the decoder uses upsampling technology to restore the input resolution and assign a semantic category to each input pixel. In step 2, the encoder uses an asymmetric dual-stream branch for RGB and HHA images, including an RGB encoder and a depth encoder. The RGB encoder and the depth encoder use ResNet-101 network and ResNet-50 network as the backbone network respectively. The ResNet-101 network and the ResNet-50 network both include 7×7 convolution, maximum pooling operation and four stages of Stage 1, Stage 2, Stage 3 and Stage 4. The basic blocks constituting the backbone network are improved, that is, a Maxpool parallel module is added to Bottleneck, denoted as MP_Bottleneck, and the feature extraction performance is improved by increasing the network width, as shown in formula (1): FE_out=W3(Cat(W2(W1(Fin))),M_P(W1(Fin)) (1) Among them, F in ∈R H×W×C , H, W and C represent the height, width and number of channels of the feature map respectively, W1, W2, W3 represent 1×1, 3×3, 1×1 convolution operations respectively, Cat represents concatenation operation, M_P represents maximum pooling operation, F E_out Represents the output of each stage of the ResNet network; The global feature extraction module is added to the four stages of Stage 1, Stage 2, Stage 3, and Stage 4 of the RGB encoder, and the global feature extraction module is used to process F E_out Extract global features, and combine the final global features and local features by adding them together to form a global-local feature extraction module, as shown in equations (2) to (4): F GL =F E_out +GC(F E_out ) (2) GC=x+Conv1×1(ReLu(LN(Conv1×1(y)))) (3) y=x×SoftMax(Conv1×1(x)) (4) Among them, x represents the input of the global feature extraction module, GC represents the global feature, F GL Represents global-local features, Conv 1×1 represents a 1×1 convolution operation, and LN represents a batch normalization operation; The encoder also includes a feature fusion module, which fuses RGB features and depth features after the 7×7 convolution and each stage, and transmits the fused features to the RGB encoder and the depth encoder. The RGB features and the depth features are reweighted at each stage of the encoder, and an attention mechanism is introduced in the feature fusion module to perform attention operations on the channel and spatial dimensions, as shown in equations (5) to (8): Fa_rgb=CBAM(Frgb_in) (5) Fa_hha=CBAM(Fhha_in) (6) CBAM=ca×σ(Conv7×7(Cat(Avgpool(ca),MaxPool(ca))) (7) ca=x×σ(MLP(Avgpool(x)+MLP(Maxpool(x)) (8) Among them, CBAM means that the attention mechanism adopts the CBAM module, x represents the input of the CBAM module, and F rgb_in 、F hha_in They represent the RGB features and depth features of the input feature fusion module, respectively. a_rgb 、F a_hha represents the feature output after being processed by the CBAM attention mechanism, σ represents the Sigmoid activation function, Conv 7×7 represents a 7×7 convolution operation, Cat represents a concatenation operation, Avgpool and Maxpool represent average pooling and maximum pooling operations respectively, and MLP represents a multi-layer perceptron consisting of two 1×1 convolutional layers.

2. The RGB-D image semantic segmentation method based on multimodal feature fusion according to claim 1, characterized in that: The three channels in step 1 respectively represent the height of the horizontal disparity above the ground, the local surface normal of the pixel, and the angle of the inferred gravity direction.

3. The RGB-D image semantic segmentation method based on multimodal feature fusion according to claim 1, characterized in that: The feature fusion module uses a cross-multiplication method to select one modality feature processed by the attention mechanism and multiply it by another modality feature that has not been processed, selects distinguishable information from one modality to assist in correcting the other modality, and then performs a cross residual connection. The feature processed by the CBAM module is cross-added with the original modality information corrected by the other modality information, further reuses the detail information lost in the network processing process, enhances the complementarity between the two modalities, and finally fuses the two modality information by adding them, as shown in equations (9) to (11): Fm_rgb=Frgb_in×Fa_hha (9) Fm_hha=Fhha_in×Fa_rgb (10) Ff_out=(Fa_rgb+Fm_hha)+(Fa_hha+Fm_rgb) (11) Among them, × represents the element-by-element multiplication operation, F m_rgb 、F m_hha They represent the feature outputs corrected by the multiplication operation, F f_out Represents the final output result of the feature fusion module.

4. The RGB-D image semantic segmentation method based on multimodal feature fusion according to claim 1, characterized in that: The decoder consists of 5 identical decoder units, each of which consists of a convolution layer + BN layer + ReLU activation function and an upsampling operation, and a short-circuit connection is added to it to reuse the information lost during the training process. The short-circuit connection uses a 1×1 convolution to change the number of output channels of the encoder stage to make it the same as the number of channels of the corresponding stage of the decoder for addition operation.

5. The RGB-D image semantic segmentation method based on multimodal feature fusion according to claim 4, characterized in that: The convolution layer includes one 1×1 convolution and two 3×3 convolutions. The upsampling operation adopts a bilinear interpolation method. After 5 upsampling operations, the image resolution is restored to the input size.

6. The RGB-D image semantic segmentation method based on multimodal feature fusion according to claim 5, characterized in that: The number of input channels of the first decoder unit is 2048, and the number of channels gradually decreases as the image resolution increases. The last decoder unit is used to output the network prediction result, remove the batch normalization operation after the last convolution layer, and set the number of output channels to the number of semantic categories.

Images