RGB-D semantic segmentation method based on dual-channel transformer cross-fusion

The RGB-D semantic segmentation method using dual-channel transformer cross-fusion utilizes a channel attention cross-fusion module and a depth-enhanced RGB module to fuse features in the encoder, and then aggregates features in the decoder through a multi-layer aggregation decoder module. This solves the problems of insufficient feature fusion and loss of fine-grained details in existing technologies, achieving higher semantic segmentation accuracy and robustness.

CN118038060BActive Publication Date: 2026-11-10ZHEJIANG SCI-TECH UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410353418.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-03-27
Publication Date
2026-11-10
Estimated Expiration
2044-03-27

AI Technical Summary

Technical Problem

Existing RGB-D semantic segmentation methods suffer from insufficient feature fusion during downsampling and loss of fine-grained details during upsampling.

Method used

An RGB-D semantic segmentation method based on dual-channel transformer cross-fusion is adopted. Through a dual-stream backbone network and a multi-level fusion module, feature fusion is performed in the encoder using a channel attention cross-fusion module and a depth enhancement RGB module. Feature aggregation is performed in the decoder through a multi-layer aggregation decoder module, combining information from RGB images and depth images.

Benefits of technology

It improves the accuracy and robustness of semantic segmentation, enhances the expressive power of image features, enables a better understanding of the global contextual information and spatial relationships of images, and improves the performance and expressiveness of the model.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118038060B_ABST
    Figure CN118038060B_ABST
Patent Text Reader

Abstract

This invention relates to the fields of computer vision and deep learning technology, and discloses an RGB-D semantic segmentation method based on dual-channel transformer cross-fusion. The method includes acquiring RGB images and depth images, converting the depth image into a three-channel HHA-encoded image using camera parameters in a computer, thereby obtaining the RGB image and corresponding HHA-encoded image to be semantically segmented. Then, the RGB image and HHA-encoded image are scaled to a fixed size and input into an RGB-D semantic segmentation model based on dual-channel transformer cross-fusion to obtain the segmentation result. This invention avoids the problems of insufficient feature fusion during downsampling and loss of fine-grained details during upsampling when transmitting fused features when using a dual-channel encoder-decoder for semantic segmentation, thus enhancing the expressive power of image features.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention belongs to the fields of computer vision and deep learning technology, specifically relating to an RGB-D semantic segmentation method based on dual-channel transformer cross-fusion. Background Technology

[0002] Image semantic segmentation has become an important research direction in computer vision, aiming to accurately understand image scenes and content. Semantic segmentation classifies each pixel in an image, determining the category of each point (such as belonging to the background, edge, or body), thus understanding the image at the pixel level. Currently, semantic segmentation technology is relatively mature and has wide applications in fields such as autonomous driving, medical image segmentation, and land cover classification. When analyzing indoor scenes, relying solely on RGB images may not yield accurate segmentation results, while depth information can supplement semantic information. Ordinary RGB images provide texture and color, while depth maps provide geometric contour information and spatial structure location information. Existing encoder-decoder methods for semantic segmentation fall into two categories: one uses two encoders to extract features from the RGB image and depth map respectively, fusing them during downsampling; however, the features extracted in this way may not be fully integrated. The second method directly fuses the two features during upsampling after feature extraction in the encoder branches, but it ignores the contribution of each feature to the result during the process. Therefore, existing techniques for pixel-based semantic segmentation using encoders need improvement. Summary of the Invention

[0003] The technical problem to be solved by the present invention is to provide an RGB-D semantic segmentation method based on dual-channel transformer cross-fusion, so as to avoid the problems of insufficient feature fusion during downsampling and loss of fine-grained details when upsampling transmits fused features when using dual-channel codec to implement semantic segmentation.

[0004] To address the aforementioned technical problems, this invention provides an RGB-D semantic segmentation method based on dual-channel transformer cross-fusion, comprising acquiring RGB images and depth images, converting the depth images into three-channel HHA encoded images in a computer using camera parameters, thereby obtaining the RGB images and corresponding HHA encoded images to be semantically segmented, and then scaling and fixing the size of the RGB images and HHA encoded images before inputting them into the RGB-D semantic segmentation model based on dual-channel transformer cross-fusion to obtain the segmentation results.

[0005] As an improvement to the RGB-D semantic segmentation method based on dual-channel transformer cross-fusion of the present invention:

[0006] The RGB-D semantic segmentation model based on dual-channel transformer cross-fusion is based on the transformer model. The encoder includes a dual-stream backbone network and a multi-level fusion module, and the decoder adopts a multi-layer aggregation decoder module. The dual-stream backbone network includes parallel RGB branches and depth branches. Both the RGB branch and the depth branch include four Transformer layers. After downsampling, the RGB features and depth features extracted by each Transformer layer are fed into the channel attention cross-fusion module for fusion. Then, the depth-enhanced RGB module is used as the input of the next transformer layer of the RGB branch. At the same time, the features output by the channel attention cross-fusion module of each layer are fed into the multi-layer aggregation decoder module.

[0007] As a further improvement to the RGB-D semantic segmentation method based on dual-channel transformer cross-fusion of the present invention:

[0008] The operation of the channel attention cross-fusion module is as follows:

[0009]

[0010]

[0011]

[0012] Where i = 1, 2, 3, 4 are the Transformer layer numbers in the encoder. and F represents the output color feature and depth feature of the i-th transformer layer, respectively. i Pay attention to the fusion characteristics output by the cross-fusion module for the channel.

[0013] As a further improvement to the RGB-D semantic segmentation method based on dual-channel transformer cross-fusion of the present invention:

[0014] The operation of the depth-enhanced RGB module is as follows:

[0015]

[0016] in, This is the RGB feature output of the i-th multi-level fusion module.

[0017] As a further improvement to the RGB-D semantic segmentation method based on dual-channel transformer cross-fusion of the present invention:

[0018] The operation of the multi-layer feature aggregation decoder module is as follows:

[0019] The fusion feature F output by the channel attention cross-fusion feature iUpsample and denoted as F' i (F' 1 , F' 2 , F' 3 and F' 4 ),Then:

[0020] F SUM =Conv(F' 1 +F' 2 +F' 3 +F' 4 (5)

[0021]

[0022] Where Conv() represents 1′1 and 3′3 convolution operations, This represents the output of the multi-layer feature aggregation decoder module.

[0023] As a further improvement to the RGB-D semantic segmentation method based on dual-channel transformer cross-fusion of the present invention:

[0024] The RGB-D semantic segmentation model based on dual-channel transformer cross-fusion is trained using the NYU-Dv2 dataset. Images from the NYU-Dv2 dataset are used as RGB images during training, while camera parameters are used to convert each image into a three-channel HHA encoded image for use as a depth image during training.

[0025] The beneficial effects of this invention are mainly reflected in:

[0026] 1. This invention utilizes a channel attention cross-fusion module to achieve channel fusion of RGB images and depth images, learns effective global image context information, and improves the accuracy of segmentation.

[0027] 2. The depth enhancement RGB module of the present invention introduces spatial correlation and geometric information into RGB images in addition to color information, thereby enhancing the expressive power of image features.

[0028] 3. The multi-layer aggregation decoder module designed in this invention can effectively mine contextual information in images, improving the robustness of the model while also enhancing its performance and expressiveness. Attached Figure Description

[0029] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0030] Figure 1 This is a schematic diagram of the RGB-D semantic segmentation model based on dual-channel Transformer cross-fusion according to the present invention;

[0031] Figure 2 for Figure 1 The diagram shows the structure of the cross-fusion module in the channel diagram.

[0032] Figure 3 for Figure 1 A schematic diagram of the structure of the multi-layer feature aggregation decoder module in the diagram;

[0033] Figure 4 This is a visual example of the semantic segmentation results. Detailed Implementation

[0034] The present invention will be further described below with reference to specific embodiments, but the scope of protection of the present invention is not limited thereto:

[0035] Example 1: An RGB-D semantic segmentation method based on dual-channel transformer cross-fusion, such as... Figure 1 As shown, the RGB-D semantic segmentation model based on dual-channel Transformer cross-fusion (hereinafter referred to as the RGB-D semantic segmentation model) of this invention employs two branches (RGB branch and depth branch) of Transformer to cross and fuse, thereby better extracting feature relationships. Simultaneously, a multi-layer aggregation decoder module is introduced, enabling it to better understand and capture long-range dependencies and semantic associations in the data, thus improving the understanding of the input data. This invention, by introducing Transformer, effectively combines the features of the RGB and Depth channels through channel attention cross-fusion during downsampling. During upsampling, the fused multi-layer features are aggregated into the decoder. This allows for full feature extraction while enabling the decoder to better capture the complex features of the input data and produce more accurate output. Specifically, it includes the following steps:

[0036] S1. Design a multi-level fusion module and a multi-layer feature aggregation decoder module.

[0037] The multi-level fusion module consists of two parts: a channel attention cross-fusion module and a depth-enhanced RGB module. The multi-layer feature aggregation decoder module enhances feature interaction and information flow between different feature levels.

[0038] (1) Channel attention cross-fusion module

[0039] RGB images contain more appearance and texture information, while depth images reveal more spatial cues and geometric structure. It is necessary to fuse features from different modes. Considering that information from RGB and depth images is complementary, the Channel Attention Cross-Fusion (CACF) module primarily fuses features from both channels during downsampling. It emphasizes task-relevant feature channels and suppresses irrelevant ones, thereby improving overall segmentation accuracy and model performance. The CACF module is shown in the attached diagram. Figure 2 As shown, the specific structure is as follows:

[0040] Given the output color features of the i-th transformer layer and depth features First, the two modalities are concatenated. To obtain global spatial information and effectively suppress noise interference, a global average pooling (GAP) operation is used instead of a fully connected layer to fuse the learned deep features, performing channel-dimensional fusion. Then, a multilayer perceptron (MLP) network is used to better capture information from different locations. The formula is expressed as:

[0041]

[0042] In the formula, The output representing channel attention is an intermediate weight transfer process. Cat(·) represents concatenation. GAP(·) represents global average pooling. MLP(·) represents a multilayer perceptron network, which includes connection layers, ReLU activation function, and sigmoid function. These can better capture the geometric contours of RGB appearance and depth, enhance the model's expressive power, and improve model performance.

[0043] Will Different from the original and Multiplication followed by cross-addition, i.e., after channel calibration and noise removal, feature fusion is performed, can be defined as:

[0044]

[0045] After performing multiplication and cross-addition operations on the original features, two new features F1 can be obtained respectively. i and Since different branches may focus on different feature levels, such as low-level details and high-level semantics, multiplication can emphasize the feature regions that both branches share interest in, while weakening the features unique to each branch. Cross-addition can supplement the features of both branches and enhance the expressive power of the overall feature. This enables multi-scale feature fusion. Finally, the two new features are added together to obtain the fused feature F. iIt demonstrates a powerful representational ability:

[0046]

[0047] (2) Deeply Enhanced RGB Module:

[0048] To learn richer features from the limited RGB branches, this invention also adds fusion information to the RGB branches. It integrates features from the deep branches into the input of the RGB branches by cross-mapping the feature maps of the channel attention fusion module at each feature level. Specifically:

[0049] The operation of the deep enhancement RGB module is as follows: output the i-th layer of the RGB branch. The fusion feature F with the two branches of the i-th layer i Adding them together helps the algorithm better understand the boundaries and scale relationships of objects in the image.

[0050]

[0051] In the formula, This is the RGB feature output of the i-th Transformer layer. This is the RGB feature output of the i-th multi-level fusion module.

[0052] The output of the depth enhancement RGB module is the output of the multi-layer fusion module. It is also the input of the RGB branch of the next Transformer layer, enabling the RGB branch to obtain features with more details and more accurate depth information.

[0053] The multi-level fusion module is executed three times, enhancing the RGB image. Compared to depth-based features, RGB-based features provide more comprehensive scene information. The fusion features of the channel attention cross-module combine spatial information from depth and RGB representations, eliminating some redundant and repetitive information. Therefore, the features of the multi-level channel attention cross-module are an enhancement of RGB-based features. The depth-enhanced RGB module makes the RGB branches more robust to environmental influences from changes in lighting. It can better understand the distances and spatial relationships between objects in a scene, unaffected by external environmental interference. It can better reflect the three-dimensionality and distance information of the scene, enabling people to better perceive the position and shape of objects.

[0054] (3) Multi-layer Feature Aggregation Decoder Module (MAD)

[0055] When the encoder downsamples, the resulting feature map still loses fine-grained details during the process of being passed to the decoder. Therefore, a multi-layer aggregation decoder module was designed, as shown in the attached figure. Figure 3 As shown, the specific structure is as follows:

[0056] The multi-layer aggregation decoder module aggregates channel attention cross-fusion features from multiple different levels, and outputs fused features F from the four channel attention cross-fusion features in the encoder. i Upsampling and denoting it as F' i (i.e., F') 1 , F' 2 , F' 3 and F' 4 It uses a structure of information aggregation and distribution, known as "squeezing and expanding". First, it uses element-wise additive layers and convolutional layers to compress the decoder block, represented as:

[0057] F SUM =Conv(F' 1 +F' 2 +F' 3 +F' 4 (5)

[0058] In the formula, F SUM This is the result of the compression operation. Then, the features output after 1′1 and 3′3 convolutions are unfolded and added back to each original feature.

[0059]

[0060] In the formula, This is the result of the extended operation. The multi-layer aggregation decoder module will improve the segmentation accuracy and robustness of the model, and can also help solve the problem of blurry or inaccurate segmentation results due to information loss.

[0061] S2. Building an RGB-D semantic segmentation model

[0062] An RGB-D semantic segmentation model based on dual-channel Transformer cross-fusion is shown in the attached figure. Figure 1As shown, the model is built on the Transformer model as the base model. The encoder mainly consists of a dual-stream backbone network and a multi-level fusion module, while the decoder uses a multi-layer aggregation decoder module. The dual-stream backbone network includes parallel RGB and depth branches. Both the RGB and depth branches contain four Transformer layers. Each Transformer layer, like the base Transformer model decoder, includes a multi-head self-attention mechanism and a fully connected layer. After downsampling, the RGB and depth features extracted by each Transformer layer are fed into the channel attention cross-fusion module for fusion. Then, the depth-enhanced RGB module serves as the input to the next Transformer layer of the RGB branch. Simultaneously, the features output from the channel attention cross-fusion module of each layer are fed into the multi-layer aggregation decoder module, allowing the features to be aggregated during upsampling. Specifically:

[0063] First, the RGB image and depth image are input into the first Transformer layer of the RGB branch and the depth branch, respectively, to extract RGB features. and depth features The input is then fed into the channel cross-attention module to obtain the fused feature F. 1 Then, the fusion feature F 1 With RGB features The common inputs are fed into the depth-enhanced RGB module to obtain RGB feature outputs. As the input to the second transformer layer in the RGB branch, deep features This serves as the input to the second transformer layer of the depth branch. Similarly, the second transformer layers of the RGB branch and the depth branch output RGB features respectively. and depth features RGB features and depth features The fusion feature F is obtained after passing through the channel intersection attention module. 2 , fusion feature F 2 and RGB features The common inputs are fed into the depth-enhanced RGB module to obtain RGB feature outputs. As the input to the third transformer layer in the RGB branch, deep features This is the input to the third transformer layer of the depth branch. The third transformer layer of both the RGB branch and the depth branch outputs RGB features respectively. and depth features RGB features and depth features The fusion feature F is obtained after passing through the channel intersection attention module.3 , fusion feature F 3 and RGB features The common inputs are fed into the depth-enhanced RGB module to obtain RGB feature outputs. As input to the fourth transformer layer of the RGB branch, deep features This is the input to the fourth transformer layer of the depth branch. The RGB features are output by the fourth transformer layer of both the RGB branch and the depth branch. and depth features The fusion feature F is obtained after passing through the channel intersection attention module. 4 .

[0064] Then, the outputs of the four-channel cross-attention module are fused to form the feature F. 1 F 2 F 3 and F 4 This serves as the input to the multi-layer feature aggregation decoder. The multi-layer feature aggregation decoder first upsamples the four fused features of the input image, then adds the upsampled features element-wise, and then adds them back to each of the original feature layers after passing through 1′1 and 3′3 convolutions. Finally, it obtains the semantic segmentation prediction result that classifies each pixel of the input image into different semantic categories.

[0065] S3. Model Training

[0066] This invention uses the NYU-Dv2 (New York University Deep Dataset V2) dataset to train and validate an RGB-D semantic segmentation model. The NYU-Dv2 dataset comprises 1449 RGB-D images, each 640×480 pixels in size, containing 464 different indoor scenes and 894 different object classes. Each image in the NYU-Dv2 dataset includes a ground truth label and camera parameters. First, using the camera parameters, the images are converted into three-channel HHA-encoded images for use as depth images. Then, 795 images (as RGB images) along with their corresponding ground truth labels and HHA-encoded images are used as the training set, and the remaining 654 images (as RGB images) along with their corresponding ground truth labels and HHA-encoded images are used as the validation set for both training and testing segmentation.

[0067] This invention employs the publicly available PyTorch framework for training, implemented on an ASUS TUF RTX3090 GPU. For the optimizer, Adam is used with a weight decay of 0.01. The learning rate is initialized to 6e-5, momentum is set to 0.9, and the batch size is set to 2. Cross-entropy loss is used as a loss function, and training is performed for 475 epochs. RGB images, depth images, and corresponding ground truth label images are input into the RGB-D semantic segmentation model. The network and its parameters are initialized, including the weights and biases in each layer. The network parameters are constructed using training samples to obtain the trained model.

[0068] S4. Online usage process

[0069] After model testing, in the actual use process, the weights with the best performance after training are selected and loaded into the RGB-D semantic segmentation model to achieve the segmentation of RGB and depth images and visualize the segmentation results. In the image visualization results, different objects can be represented by different colors to better display and distinguish the segmentation results of each type.

[0070] First, RGB images and corresponding depth images are acquired using a color digital camera and a depth camera. In the computer, camera parameters are used to convert the depth images into three-channel HHA-encoded images, thus obtaining the RGB images and corresponding HHA-encoded images required for semantic segmentation. The image size resolution is then fixed to 640′480 pixels by scaling. These 640′480 pixel RGB and HHA-encoded images are then input into an RGB-D semantic segmentation model with optimal weights. Four Transformer layers of a dual-stream backbone network extract color and depth features, obtaining feature maps of different sizes after each downsampling. The RGB and depth features obtained after each downsampling are fed into a channel attention cross-fusion module, fusing the RGB and depth features along the channel dimension. A depth-enhanced RGB module further enriches the RGB channels with depth geometric information. The fused features are then introduced into a multi-layer aggregation decoder for 2D semantic segmentation prediction, obtaining the final segmentation result, including segmentation annotation and segmentation contour results.

[0071] experiment:

[0072] (1) Evaluation indicators

[0073] This invention uses three metrics—mean Intersection over Union (mIoU), pixel precision (PA), and mean pixel precision (MPA)—to evaluate the model's performance and effectiveness. IoU is calculated as the intersection-to-union ratio of the true label and the predicted result for each class. mIoU is the average IoU across all classes, and it is the most commonly used metric for evaluating segmentation accuracy. The calculation method is as follows:

[0074]

[0075] Where i represents the true value, j represents the predicted value, and p ij This indicates that the prediction i is j, p ii p ij and p ji They represent TP, FP, and FN respectively; while k+1 is the class count, which represents true positives (TP), false negatives (FN), false positives (FP), and true negatives (TN).

[0076] Pixel precision (PA): The proportion of pixels correctly predicted to the total number of pixels, which can be expressed as:

[0077]

[0078] Average Pixel Precision (MPA): Calculates the number of correct pixels in each category and the proportion of all pixels in that category, then averages the results across all categories. It can be expressed as:

[0079]

[0080] (2) Comparative experiment:

[0081] The proposed method was compared with other state-of-the-art methods on the NYU-Dv2 dataset, and the results are shown in Table 1.

[0082] Table 1: Comparative Experiment Results

[0083]

[0084]

[0085] Most methods use depth maps as one of the input sources, and the method of this invention also utilizes RGB images and depth images (RGB-D) as input for semantic segmentation, achieving leading performance. Table 1 shows that, when using encoder-decoder structures, for different fusion methods, RDFNet uses the core idea of ​​residual learning in its multimodal feature fusion module, achieving an mIoU of 50.1%, PA of 76.0%, and MPA of 62.8%. The fusion method in the Zig-zag network is a sawtooth network, which connects the backbone structure and the decoder structure to exchange enhanced feature maps, achieving an mIoU of 47.8%, PA of 73.4%, and MPA of 57.5%. ESANet utilizes non-bottleneck-1D blocks and an attention-based fusion to merge RGB and depth information, achieving an mIoU of 50.3%. SGNet-16s embeds Spatial Information Guided Convolution (S-Conv) into a network called Spatial Information Guided Convolution, achieving an mIoU of 50.2%, PA of 76.4%, and MPA of 62.1%. PCGNet proposes a novel Pyramid Context Guided Fusion (PCGF) module to fully utilize the complementary information of depth and RGB features, achieving an mIoU of 52.1% and PA of 77.6%. The channel attention cross-fusion module and depth-enhanced RGB module of this invention can more fully integrate mutual features. The method of this invention outperforms other methods in most categories, achieving an mIoU of 52.9%, PA of 78.0%, and MPA of 66.0%.

[0086] Appendix Figure 4 The image shows the visual representation of the semantic segmentation result. RGB input is the input RGB image, HHA input is the depth map (converted to HHA before input), Ground Truth is the ground truth image, w / o MAD is the segmentation result without the multi-layer feature aggregation decoder module (dual-channel Transformer backbone network + channel attention cross-fusion module + depth-enhanced RGB module), and Our is the semantic segmentation output of the overall model of this invention. The results show that the addition of depth allows for excellent extraction of geometric information. For example, some geometric contours on the wall are segmented very well.

[0087] (3) Ablation experiment:

[0088] To understand how each component of the method of this invention affects segmentation, a series of ablation studies were conducted on the NYU-Dv2 dataset. The results of the ablation experiments are shown in Table 2:

[0089] Table 2: Ablation Experiment Results

[0090]

[0091] Seg+HHA: The baseline model of this invention, i.e., RGB and depth map (HHA) as input, and Transformer (Seg) as the backbone network of the two-stream network.

[0092] Seg+HHA+CACF: Based on the Seg+HHA network, a Channel Attention Cross-Fusion Module (CACF) is added after each of the four stages.

[0093] Seg+HHA+CACF+Enhanced: Based on the Seg+HHA+CACF network, a deep enhancement RGB module (Enhanced) is added to the attention cross-fusion module of each channel.

[0094] Res+HHA+CACF+Enhanced: Based on Seg+HHA+CACF+Enhanced, the dual-stream backbone network Transformer is modified to ResNet-50(Res), which serves as the backbone network of the dual-stream network.

[0095] Seg+HHA+CACF+Enhanced+MAD: This is the present invention.

[0096] As shown in Table 2:

[0097] (1) Effectiveness of the Transformer backbone: The ResNet backbone was replaced with a Transformer backbone to examine the effectiveness of the dual-channel Transformer backbone network. When using ResNet as the dual-channel backbone network (Res+HHA+...

[0098] In the case of CACF+Enhanced, the mIoU reaches 48.6% and the PA reaches 75.6%. When using Transformer as the dual-channel backbone network (Seg+HHA+CACF+Enhanced), the mIoU reaches 51.8% and the PA reaches 77.3%. Through the above comparison, Transformer improves mIoU by 3.2% and PA by 1.7% compared to ResNet. This is because convolutional structures like ResNet only consider feature extraction from local receptive domains. When processing large-scale images, Transformer can better capture long-distance semantic relationships.

[0099] (2) Effectiveness of the Deep Enhanced RGB Module: Compared to some modules without a Deep Enhanced RGB module, using only a dual-channel Transformer backbone network and a channel attention cross-fusion module, without the Deep Enhanced RGB module (Seg+HHA+

[0100] The mIoU of CACF is 50.5%, and the PA is 76.7%. After adding this module, even using the Transformer backbone network, channel attention cross-fusion module, and depth-enhanced RGB module (Seg+HHA+CACF+Enhanced), the mIoU reaches 51.8%, and the PA reaches 77.3%. The mIoU of this invention is 1.3% higher. In addition to introducing color information, this module can also introduce spatial correlation and geometric information, thereby improving the expressive power of image features. The addition of depth information and shape enables the image representation capability of the RGB branch to be enhanced even in the presence of illumination changes and texture loss.

[0101] (3) Effectiveness of the Channel Attention Cross-Fusion Module: Using only the dual-channel Transformer backbone network without the channel attention cross-fusion module, i.e., the baseline network (Seg+HHA), the mIoU is 49.5% and the PA is 76.2%. After using the dual-channel Transformer backbone network and performing feature fusion using the channel attention cross-fusion module (Seg+HHA+CACF), the mIoU reaches 50.5% and the PA reaches 76.7%. The use of the channel attention cross-fusion module improves the mIoU by 1.0%. First, this module assigns weights to each feature map, improving the overall feature expressiveness. Second, by calculating the similarity and correlation between feature maps, cross-feature fusion is effectively achieved. Finally, by adjusting the weights according to different tasks and scenarios, it has strong robustness and generalization ability.

[0102] (4) Effectiveness of the multi-layer feature aggregation decoder module: When using a dual-channel Transformer backbone network, a channel attention cross-fusion module, and a deep enhancement RGB module (Seg+HHA+CACF+Enhanced), the mIoU was 51.8% and the PA was 77.3%. However, after using a dual-channel Transformer backbone network, a channel attention cross-fusion module, a deep enhancement RGB module, and a multi-layer feature aggregation decoder module (Seg+HHA+CACF+Enhanced+MAD), the mIoU reached 52.9% and the PA reached 78.0%. Adding the multi-layer feature aggregation decoder module increased the mIoU by 1.1%. Through multi-layer feature extraction, low-level details can be better captured into high-level semantic information, providing richer and more distinctive feature representations. This invention utilizes a squeeze-unfold operation to integrate and reconstruct the multi-layer features extracted by the decoder, extracting shared features between different layers and adding them to the features of the original layers. This can improve the accurate extraction of target regions in semantic segmentation tasks and alleviate the problem of fine-grained information loss.

[0103] Finally, it should be noted that the above examples are merely some specific embodiments of the present invention. Obviously, the present invention is not limited to the above embodiments and many variations are possible. All variations that can be directly derived or conceived by those skilled in the art from the disclosure of the present invention should be considered within the scope of protection of the present invention.

[0104] [1]Lin, D., Huang, H.: Zig-zag network for semantic segmentation of rgb-dimages. IEEE Transactions on Pattern Analysis and Machine Intelligence 42(10),2642–2655(2020) https: / / doi.org / 10.1109 / TPAMI.2019.2923513 .

[0105] [2] Zhou, F., Lai, Y.-K., Rosin, PL, Zhang, F., Hu, Y.: Scale-aware network with modality-awareness for rgb-d indoor semantic segmentation. Neurocomputing492, 464–473 (2022).

[0106] [3]Lee,S.,Park,S.-J.,Hong,K.-S.:Rdfnet:Rgb-d multi-level residualfeature fusion for indoor semantic segmentation.In:2017IEEE InternationalConference on Computer Vision(ICCV),pp.4990–4999(2017). https: / / doi.org / 10.1109 / ICCV .2017.533。

[0107] [4]Seichter,D.,K¨ohler,M.,Lewandowski,B.,Wengefeld,T.,Gross,H.-M.:Efficient rgb-d semantic segmentation for indoor scene analysis.In:2021IEEEInternational Conference on Robotics and Automation(ICRA),pp.13525–13531(2021). https: / / doi.org / 10.1109 / ICRA48506.2021.9561675 。

[0108] [5]Chen,L.-Z.,Lin,Z.,Wang,Z.,Yang,Y.-L.,Cheng,M.-M.:Spatialinformation guided convolution for real-time rgbd semantic segmentation.IEEETransactions on Image Processing 30,2313–2324(2021) https: / / doi.org / 10.1109 / TIP. 2021 .3049332。

[0109] [6]Xiaokang,C.,Lin,K.-Y.,Wang,J.,Wu,W.,Qian,C.,Li,H.,Zeng,G.:Bi-directional Cross-Modality Feature Propagation with Separation-andAggregationGate for RGB-D Semantic Segmentation,pp.561–577(2020). https: / / doi.org / 10.1007 / 978-3-030-58621-8 33。

[0110] [7]Liu,H.,Guo,L.,Zhou,Z.,Zhang,H.:Pyramid-context guided featurefusion for rgb-d semantic segmentation.In:2022 IEEE International Conferenceon Multimedia and ExpoWorkshops(ICMEW),pp.1–6(2022). https: / / doi.org / 10.1109 / ICMEW56448.2022.9859353 。

Claims

1. An RGB-D semantic segmentation method based on dual-channel transformer cross-fusion, characterized in that: The process involves acquiring RGB and depth images, converting the depth images into three-channel HHA encoded images using camera parameters in a computer, thereby obtaining the RGB images and corresponding HHA encoded images that need to be semantically segmented, and then scaling and fixing the size of the RGB and HHA encoded images before inputting them into an RGB-D semantic segmentation model based on dual-channel transformer cross-fusion to obtain the segmentation results. The RGB-D semantic segmentation model based on dual-channel transformer cross-fusion is based on the transformer model. The encoder includes a dual-stream backbone network and a multi-level fusion module, and the decoder adopts a multi-layer aggregation decoder module. The dual-stream backbone network includes parallel RGB branches and depth branches. Both the RGB branch and the depth branch include four Transformer layers. After downsampling, the RGB features and depth features extracted by each Transformer layer are fed into the channel attention cross-fusion module for fusion. Then, the depth-enhanced RGB module is used as the input of the next transformer layer of the RGB branch. At the same time, the features output by the channel attention cross-fusion module of each layer are fed into the multi-layer aggregation decoder module. The operation of the channel attention cross-fusion module is as follows: (1) (2) (3) in, This represents the number of Transformer layers in the encoder. and These are the output color features and depth features of the i-th transformer layer, respectively. Pay attention to the fusion characteristics output by the cross-fusion module for the channel; The operation of the depth-enhanced RGB module is as follows: (4) in, This is the RGB feature output of the i-th multi-level fusion module; The operation of the multi-layer aggregation decoder module is as follows: The fusion feature output by the channel attention cross-fusion feature Upsampled separately and denoted as ,Right now, , , and ,Then: (5) (6) in, express and Convolution operation, This indicates the output of the multi-layer aggregation decoder module.

2. The RGB-D semantic segmentation method based on dual-channel transformer cross-fusion according to claim 1, characterized in that: The RGB-D semantic segmentation model based on dual-channel transformer cross-fusion is trained using the NYU-Dv2 dataset. Images in the NYU-Dv2 dataset are used as RGB images during training, while camera parameters are used to convert each image into a three-channel HHA encoded image for use as a depth image during training.