RGB-D oriented fusion model and target detection method thereof

Abstract

The application provides an RGB-D fusion model and a target detection method thereof. First, an edge image of a ground truth image is generated by using a Canny edge detection algorithm, and the edge image is used as an additional input feature of a backbone network to enrich feature information and assist deep learning of the model. Then, a MobileNetV2 is used to extract multi-level features of an RGB image, and an encoder based on a residual network ResNet-152 is used to extract multi-level features of a depth image and an edge image, so that edge feature information is integrated into features of the RGB image extracted by the MobileNetV2 encoder. A multi-scale perception fusion module and a global fusion module are used to combine complementary semantic information of RGB features and depth features, and layered fusion of cross-modal features is completed. Finally, the obtained multiple fusion features are input into an edge strengthening fusion module in a decoding process layer by layer, the fusion of RGB stream and depth stream detection results is supervised by using a designed network loss, and finally, a significant detection result is output, so that the detection effect in the case that the background is complex and the boundary of the significant target is fuzzy is improved.

Description

Technical Field

[0001] This invention belongs to the field of computer image processing technology, specifically, it is a fusion model for RGB-D and its target detection method. Background Technology

[0002] The continuous advancements in computer vision have made salient object detection a major focus of current research. Most current conventional salient object detection algorithms are designed for data acquired from ordinary optical cameras, resulting in low robustness. In situations with complex backgrounds and blurred salient object boundaries, they easily overlook intricate or subtle boundaries, often producing salient maps with blurry edges. Using depth data acquired from depth cameras can effectively alleviate these problems, extracting relevant information from depth images to fill in insufficiently defined information in RGB images, thereby improving detection accuracy. The fundamental goal of salient object detection is to automatically identify and segment the regions or objects of greatest interest to the human eye in a given scene. It has been widely applied to many computer vision tasks, such as image retrieval, visual tracking, action recognition, and autonomous driving.

[0003] Over time, many RGB-D-based salient object detection algorithms have been proposed. Traditional image algorithms utilize various handcrafted features, such as color cues, boundary cues, texture cues, global contrast, and spatial priors, to extract useful feature information from RGB and depth images, and then use this information to predict saliency maps. Examples include saliency prediction networks and boundary preservation networks that simultaneously focus on the regions and boundaries of salient objects. With the rapid development of deep learning, compared to the limitations of handcrafted features in information representation, the feature information extracted by convolutional neural networks possesses rich semantic information and powerful representational capabilities. Most RGB-D methods primarily focus on exploring the fusion of RGB modal features and depth modal features. Fusion strategies are categorized into three types: early fusion, multi-scale fusion, and late fusion. Early fusion directly concatenates RGB features with depth features, adding an input channel to the network; while late fusion... The fusion approach uses two independent network streams to extract high-level features from RGB and depth data respectively. These features are concatenated and then used to generate the final saliency prediction. Multi-scale fusion is proposed to effectively explore the correlation between RGB images and depth maps. For example, a multi-scale, multi-path fusion network integrating RGB images and depth maps with a cross-modal interaction module (MMCI) is used to explore the complementarity between low-level and high-level representations. The BiANet network uses a multi-scale bilateral attention module (MBAM) to capture better global information from multiple layers. The BBS-Net network uses a bifurcated backbone strategy (BBS) and has developed a depth enhancement module (DEM) to explore the information portion of the depth map from spatial and channel views. However, the above salient object detection algorithms do not solve the problem of blurred salient object boundaries, which leads to a decrease in accuracy when complex or detailed boundaries are easily ignored. Summary of the Invention

[0004] The purpose of this invention is to address the challenges in existing object detection algorithms, such as the difficulty of fusing RGB and depth image features, the potential performance degradation due to poorly designed fusion mechanisms, and the poor detection performance caused by blurred target boundaries in salient object detection. Specifically, this invention employs an RGB-D fusion model and its object detection method. By introducing a generated edge map, the algorithm further enriches the edge information representation, effectively mitigating the performance degradation caused by blurred target boundaries. Simultaneously, three fusion modules are designed to efficiently fuse complementary semantic information between RGB and depth features at different feature levels, thereby enhancing the collaborative expression capability of cross-modal information and ensuring full integration of RGB and depth modal features.

[0005] The specific technical solution adopted in this invention is as follows:

[0006] A fusion model for RGB-D and its target detection method include the following steps:

[0007] (1) The edge map of the ground truth map is generated by the Canny edge detection algorithm and used as an additional input feature of the backbone network to enrich the feature information and assist the deep learning of the model.

[0008] (2) Multi-level feature extraction of RGB images is performed using MobileNetV2, and multi-level feature extraction of depth images is performed using an encoder based on ResNet-152 residual network, thereby analyzing and capturing key features of RGB images and depth images at different levels. At the same time, feature extraction of edge maps is performed using the ResNet-152 encoder, and the obtained edge feature information is integrated into the RGB image features extracted by the MobileNetV2 encoder, thereby enhancing the feature representation capability of RGB images.

[0009] (3) Using the multi-scale perceptual fusion module (MSP-Fusion) and the global fusion module (Global-Fusion), the complementary semantic information of RGB features and deep features is combined at the low-level and high-level feature levels to complete the hierarchical fusion of cross-modal features.

[0010] (4) After decoding, the multiple fusion features obtained in step three are input layer by layer into the edge-refinement fusion module in the decoding process. This module effectively enhances the detail information of the edges of salient objects and the overall global semantic information by combining RGB image features with low-level and high-level RGB-D fusion features. Through the meticulous fusion of features at different levels, the edge-refinement module can not only improve the detail perception ability of the edge region, but also optimize the global semantic expression, thereby producing more accurate and semantically rich saliency detection results.

[0011] (5) Design image fusion loss and network overall loss to achieve the fusion of RGB stream and depth stream detection results and supervised learning of the network, and output the final saliency detection result.

[0012] As a further improvement of the present invention, step 1) firstly involves coloring the ground truth map. Since the ground truth map is a structure obtained through pixel classification, it needs to be pixel-colored for further processing. Next, the classic Canny edge detection algorithm is used to extract edges from the colored ground truth map. The Canny algorithm, through multi-level gradient detection and non-maximum suppression, can accurately extract target edges and generate high-quality salient target edge maps. In this process, key edge information of salient targets is effectively extracted, which not only enhances the boundary clarity of feature representation but also significantly enriches the feature information of the model. The introduction of the edge map provides the model with an accurate description of the boundaries of salient targets, serving as additional guiding information in multimodal fusion.

[0013] As a further improvement of the present invention, in step 2), MobileNetV2 performs layer-by-layer downsampling on the RGB image to extract multi-level feature information. The feature extraction of depth image and edge image is completed by ResNet-152 encoder. The residual connection structure of this network alleviates the gradient vanishing problem while extracting deep features. After downsampling at each layer, the edge image features are embedded in the corresponding scale of RGB features. Feature fusion is achieved through addition operation, which improves the diversity of features, supports model learning, and enhances the edge information extraction capability.

[0014] As a further improvement of the present invention, in step 3), the RGB image carries rich scene and detail information, while the depth image provides distance information from each pixel to the camera, containing more spatial structure information. Since the low-level feature map contains rich detail information, we introduce the MSPF module during the fusion process. This module generates preliminary fused features through the summation of RGB and depth features. Next, the number of channels is adjusted using 1×1 convolution, and 1×3 and 3×1 convolutions are used to extract the horizontal and vertical features of the image, respectively, thereby capturing spatial information in different directions. Then, the extracted features are deeply fused using 1×1 and 3×3 convolution operations to enhance the expressiveness of the features. Finally, the fused feature map is combined with the original RGB and depth features to form a richer feature representation.

[0015] Building upon this, the concatenated feature map is processed using Depthily Separable Convolution (DW), Batch Normalization (BN), and the Swish activation function to further extract deeper features. Subsequently, the feature map is subjected to dimensionality reduction and nonlinear mapping using Pointwise Convolution (PW), BN, and the Swish activation function to optimize its representational power. To capture broader spatial feature associations, the feature map is further processed using a Spatial Attention (SA) mechanism to enhance its perception of global information and promote effective fusion between RGB and deep features. To enhance feature representation capabilities, this module designs three independent branches: the main branch extracts features using deep DW-BN, while the shortcut branch processes features using BN and PW-BN. Finally, the feature maps from the three branches are fused element-wise and nonlinearly transformed using the GELU activation function to generate the final comprehensive feature map. The calculation formula is as follows:

[0016]

[0017] This fusion process effectively combines the detailed information of the RGB image with the spatial information of the depth image, thereby generating a richer and more comprehensive feature map.

[0018] When processing high-level feature maps, which contain more semantic information but have a smaller data volume, the GF module is used to efficiently fuse these features and better utilize semantic information. First, 1×1 convolution operations are performed on RGB and depth features respectively, followed by 3×3 convolution and the Sigmoid activation function to extract deeper semantic information. Next, cross-fusion is performed through element-wise multiplication to fuse RGB and depth features, and the results are added and concatenated to form an initial fused feature map. This initial fused feature map is then concatenated with the previously extracted deep semantic features to further enrich the feature representation. Subsequently, feature extraction is performed using DW-BN and Swish activation functions, and further optimized using PW-BN and Swish. Based on this, SA is used to enhance the spatial correlation between RGB and depth features, promoting more effective fusion. Finally, the SA-processed feature map is added to the original RGB features, and a non-linear mapping is performed using the GELU activation function to obtain the final fused feature map, providing rich representation for the ERF module. The calculation process can be represented as follows:

[0019]

[0020] Through this fusion strategy, GF can effectively integrate the semantic information of RGB and depth images. At the same time, it enhances the ability to capture global features by leveraging the attention mechanism, generating a feature representation that fuses semantic information with rich global information.

[0021] As a further improvement of the present invention, in step 4), during the upsampling decoding of the RGB image and the depth image, an ERF module is interspersed layer by layer. This module aims to better combine the RGB image with the low-level and high-level RGB-D fusion features, thereby improving the accuracy of saliency detection. The low-level detail information extracted by MSPF is fused in the first two layers of the ERF module to enhance the detail perception capability; the high-level abstract information extracted by GF is fused in the last two layers of the ERF module to provide more globally perceptive semantic information.

[0022] The fusion information extracted by MSPF or GF is mainly used to fill in the missing edge information in the RGB features to enhance the expressive power of edge details. First, the fusion information is added element-wise to the RGB features. Then, the fused features are processed by the channel attention mechanism (CA) to further extract more edge information. In this process, the RGB features and the extracted edge information are multiplied and added element-wise again to further supplement the details and edge information that may be lost in the RGB features.

[0023] Subsequently, a 1×1 convolution is used to adjust the channel dimension of the feature map to prepare for subsequent concatenation and fusion operations, thereby obtaining a preliminary fused feature representation. On this basis, to further enhance the semantic information of the deep features, a spatial attention mechanism (SA) is used to process the deep feature map and strengthen the global spatial correlation of the deep features. Then, two convolution operations are applied to extract multi-scale information of the deep features to capture richer semantic features.

[0024] Finally, the RGB feature map and the depth feature map are multiplied element-wise and concatenated along the channel dimension to further fuse their feature information. After the DW-BN-Swish operation, the final fused feature map is obtained. To obtain the final segmentation result map, the number of channels in the fused feature map is adjusted to match the number of classes, and combined with high-level abstraction information, the final segmentation prediction is performed. The calculation process can be represented as follows:

[0025]

[0026] As a further improvement of the present invention, in step 5), in order to extract the residual features of the RGB feature map from the depth feature map for semantic segmentation, the depthstream needs to have the ability to independently perform semantic segmentation. Therefore, during the training process, it is necessary to calculate the ground truth map y and the depthstream output. The loss between them. This loss function is calculated using cross-entropy, and the specific formula is as follows:

[0027]

[0028] Final classification results It also needs to be supervised by the truth graph y, and the calculation is still done using cross-entropy, as shown in the following formula:

[0029]

[0030] Meanwhile, in the edge enhancement fusion module, to better combine RGB image features with low- and high-level RGB-D fusion features, the fusion module also needs supervised learning using the ground truth map y. The output of the fusion module... In this process, it is necessary to compare it with the true label map y, and calculate the error using the cross-entropy loss function. The specific formula is as follows:

[0031]

[0032] Therefore, the final loss function is:

[0033] L total =L d +L rgb +L n

[0034] This total loss function combines the loss from the depth flow, the loss from the RGB flow, and the loss from the edge enhancement and fusion module to optimize the model's training process and improve the final semantic segmentation accuracy.

[0035] The beneficial effects of this invention are as follows: By employing the Canny algorithm to generate ground truth-labeled edge maps, the problem of blurred boundaries of salient targets is effectively solved, especially when complex or detailed boundaries are easily ignored, significantly improving the blurring of salient map edges. Simultaneously, considering the significant differences between depth images and RGB images, this invention proposes a cross-modal fusion strategy at different feature levels, fully combining the detailed information of RGB images with the spatial information of depth images, thereby significantly improving the network's comprehensive perception ability of multimodal information. Furthermore, the edge enhancement fusion module introduced in the upsampling stage can fully utilize the fusion information from different feature levels in the early stages to achieve high-precision restoration of the original image details. Finally, this invention designs a jointly optimized loss function, which not only provides strong supervision for the fusion module but also globally constrains the training of the entire network. Attached Figure Description

[0036] Figure 1 This is a diagram of the overall network structure in this invention.

[0037] Figure 2 This is a network structure diagram of the Multi-Scale Perception Fusion Module (MSPF) in this invention.

[0038] Figure 3 This is a network structure diagram of the Global Fusion Module (GF) in this invention.

[0039] Figure 4 This is a network structure diagram of the Edge Enhancement Fusion Module (ERF). Detailed Implementation

[0040] The overall network structure of the model is as follows Figure 1 As shown, an RGB-D fusion model and its target detection method are presented. Figure 1 The specific steps are as follows:

[0041] In step 1, the ground truth map is first colored. Since the ground truth map is a structure obtained through pixel classification, pixel coloring is necessary for further processing. Then, the classic Canny edge detection algorithm is used to extract edges from the colored ground truth map. The Canny algorithm, through multi-level gradient detection and non-maximum suppression, can accurately extract target edges, generating a high-quality salient target edge map. In this process, key edge information of salient targets is effectively extracted, which not only enhances the boundary clarity of feature representation but also significantly enriches the model's feature information. The introduction of the edge map provides the model with an accurate description of the boundaries of salient targets, serving as additional guiding information in multimodal fusion.

[0042] In step 2, MobileNetV2 is used to extract multi-level features from the RGB image, and a ResNet-152-based encoder is used to extract multi-level features from the depth image, thereby analyzing and capturing key features of the RGB and depth images at different levels. Simultaneously, the ResNet-152 encoder extracts features from the edge map, integrating the obtained edge feature information into the RGB image features extracted by the MobileNetV2 encoder, thus enhancing the feature representation capability of the RGB image. Figure 1 As shown.

[0043] In step 3, the MSPF module and GF module are used in the last four convolutional layers of feature extraction, respectively. The MSPF module is used to process low-level feature maps, which contain rich details and are an important source of target edge and texture features. To extract this information more effectively, this module sums the RGB and depth feature information to obtain a preliminary fused feature. Then, this fused feature is concatenated with the original RGB and depth features, followed by depthwise separable convolution (DW), batch normalization (BN), and Swish activation function processing. Pointwise convolution (PW), BN, and Swish activation function are then performed on the processed feature map. The Wish activation function is used to reduce the dimensionality of the feature maps. To capture broader spatial feature associations, the feature maps are processed by a spatial attention (SA) mechanism. To enhance feature representation capabilities, the module is designed with three branches: the main branch extracts features using DW and BN, and the shortcut branch processes features using BN and PW-BN. Finally, the feature maps from the three branches are fused element-wise and then processed using the GELU activation function to obtain the final feature map. This module effectively combines the detailed information of the RGB image with the spatial features of the depth image while preserving low-level feature details, generating richer feature maps, such as... Figure 2 As shown.

[0044] The GF module is primarily used to process high-level feature maps. High-level feature maps typically contain rich semantic information and have low feature dimensionality, making them suitable for cross-modal feature fusion operations. First, RGB and depth features are extracted separately and then cross-fused using a multiplication method, that is, RGB features and depth features are multiplied element-wise. The resulting feature maps are then combined through addition and concatenation operations to obtain a preliminary comprehensive feature representation. Next, feature extraction is performed using DW-BN and Swish activation functions, followed by further feature optimization using PW-BN and Swish, effectively extracting global spatial information from high-level features and improving the model's ability to express high-level semantic features and segmentation accuracy. Based on this, SA is used to enhance the spatial correlation between RGB and depth features, promoting more effective fusion. Finally, the SA-processed feature map is added to the original RGB features and non-linearly mapped using the GELU activation function to obtain the final fused feature map, providing rich representations for the ERF module, such as... Figure 3 As shown.

[0045] In step 4, during the upsampling decoding of the RGB and depth images, ERF modules are interspersed layer by layer. This module aims to better combine the RGB image with the low-level and high-level RGB-D fusion features, thereby improving the accuracy of saliency detection. The low-level detail information extracted by MSPF is fused in the first two layers of the ERF module to enhance detail perception; the high-level abstract information extracted by GF is fused in the last two layers of the ERF module to provide more globally perceptive semantic information.

[0046] After concatenating the extracted fusion features with the RGB feature information, the expressive power of the features is further enhanced through a channel attention (CA) mechanism. The feature representation is further enriched by performing element-wise multiplication and summation of the RGB features and the fusion features. Next, a 1×1 convolution is used to adjust the number of channels to prepare for subsequent concatenation operations, ultimately generating the integrated feature map. Simultaneously, to further enhance the semantic information of the deep features, SA is applied to process the deep features, and a deep feature map is extracted through two convolution operations. Finally, the RGB feature map and the deep feature map are multiplied element-wise and concatenated along the channel dimension, followed by a DW operation to obtain the final fusion feature map. Finally, combining high-level abstraction information, the number of channels is adjusted to match the number of classifications to obtain the final segmentation result map.

[0047] In step 5, in order to extract the residual features of the RGB feature map from the depth feature map for semantic segmentation, the deepflow algorithm needs to have the ability to independently perform semantic segmentation. Therefore, during training, it is necessary to compute the ground truth map y and the deepflow output. The loss between them. This loss function is calculated using cross-entropy, and the specific formula is as follows:

[0048]

[0049] Final classification results It also needs to be supervised by the truth graph y, and the calculation is still done using cross-entropy, as shown in the following formula:

[0050]

[0051] Meanwhile, in the edge enhancement fusion module, to better combine RGB image features with low- and high-level RGB-D fusion features, the fusion module also needs supervised learning using the ground truth map y. The output of the fusion module... In this process, it is necessary to compare it with the true label image y, and calculate the error using the cross-entropy loss function. The specific formula is as follows:

[0052]

[0053] Therefore, the final loss function is:

[0054] L total =L d +L rgb +L n

[0055] This total loss function combines the loss from the depth flow, the loss from the RGB flow, and the loss from the edge enhancement fusion module to optimize the model's training process and improve the final semantic segmentation accuracy.

Claims

1.A RGB-D oriented fusion model and a target detection method thereof, characterized in that, Includes the following steps: Step 1: Use the Canny edge detection algorithm to generate an edge map of the ground truth map, and use it as an additional input feature of the backbone network to enrich the feature information and assist the deep learning of the model. Step 2: Multi-level feature extraction is performed on the RGB image using MobileNetV2, and multi-level feature extraction is performed on the depth image using a ResNet-152-based encoder, thereby analyzing and capturing key features of the RGB and depth images at different levels; at the same time, edge image features are extracted using the ResNet-152 encoder, and the obtained edge feature information is integrated into the RGB image features extracted by the MobileNetV2 encoder, thereby enhancing the feature representation capability of the RGB image; Step 3: Utilize the multi-scale perceptual fusion module MSP-Fusion and the global fusion module Global-Fusion to complete the hierarchical fusion of cross-modal features; wherein, the multi-scale perceptual fusion module is used to perform low-level fusion of detailed information features and depth features of RGB images; the global fusion module is used to perform efficient semantic information feature fusion of RGB features and depth features at a high-level feature level containing more semantic information. Step 4: After decoding, the multiple fusion features obtained in Step 3 are input layer by layer into the edge enhancement fusion module in the decoding process; this module effectively enhances the detail information of significant object edges and the overall global semantic information by combining RGB image features with low-level and high-level RGB-D fusion features. Step 5: Design image fusion loss and overall network loss to achieve the fusion of RGB stream and depth stream detection results and supervised learning of the network, and output the final saliency detection result. 2.The RGB-D oriented fusion model and the target detection method thereof according to claim 1, wherein, In step one, the ground truth map is first colored; then, the Canny algorithm is used to perform edge detection on the colored edge map to generate the edge map of the salient target. This process can effectively extract the edge information of the salient target, enrich the feature information, and promote the model's assisted learning in salient target detection. 3.The RGB-D oriented fusion model and the target detection method thereof according to claim 1, wherein, In step two, MobileNetV2 performs layer-by-layer downsampling on the RGB image to extract multi-level feature information; the feature extraction of depth image and edge image is completed by ResNet-152 encoder. The residual connection structure of this network alleviates the gradient vanishing problem while extracting deep features; the edge image features are embedded into the corresponding scale of RGB features after downsampling at each layer, and feature fusion is achieved through addition operation to improve feature diversity, support model learning and enhance the edge information extraction capability. 4.The RGB-D oriented fusion model and the target detection method thereof according to claim 1, characterized in that, In step three, since the low-level feature map contains rich detailed information, we introduced the MSPF module in the fusion process. This module generates preliminary fusion features by summing the RGB and depth features. Then, the fused feature is concatenated with the original RGB and depth features, followed by depthwise separable convolution (DW), batch normalization (BN), and Swish activation. The processed feature map is then subjected to pointwise convolution (PW), BN, and Swish activation to reduce the feature map's dimensionality. To capture broader spatial feature associations, the feature map is processed by spatial attention (SA). To enhance feature representation capabilities, this module is designed with three independent branches: the main branch extracts features using DW and BN, while the shortcut branch processes features using BN and PW-BN. Finally, the feature maps of the three branches are fused by element-wise summation, and the final feature representation is obtained by the GELU activation function. This process effectively combines the detailed information of the RGB image with the spatial features of the depth image, generating richer feature maps. When processing high-level feature maps, which contain more semantic information but have a lower data volume, the GF module is used to efficiently fuse these features and better utilize semantic information. First, RGB and depth features are extracted separately and then cross-fused by multiplication, that is, RGB features and depth features are multiplied element-wise. The resulting feature maps are then combined by addition and concatenation operations to obtain a preliminary comprehensive feature representation. Next, feature extraction is performed by processing with DW-BN and Swish activation functions, and then the features are further optimized by PW-BN and Swish. Based on this, SA is used to enhance the spatial correlation between RGB and depth features, promoting more effective fusion. Finally, the SA-processed feature map is added to the original RGB features and nonlinearly mapped through the GELU activation function to obtain the final fused feature map, providing rich representation for the ERF module. 5.The RGB-D oriented fusion model and the target detection method thereof according to claim 1, wherein, In step four, during the upsampling decoding of the RGB and depth images, ERF modules are interspersed layer by layer. This module aims to better combine the RGB image with the RGB-D fusion features of low and high layers, thereby improving the accuracy of saliency detection. The low-level detail information extracted by MSPF is fused in the first two layers of the ERF module to enhance the ability to perceive details. The high-level abstract information extracted by GF is fused in the last two layers of the ERF module to provide more globally perceptive semantic information. After concatenating the extracted fusion features with the RGB feature information, the expressive power of the features is further enhanced by the channel attention mechanism CA. The feature representation is further enriched by performing element-wise multiplication and summation of the RGB features and the fusion features. Next, a 1×1 convolution is used to adjust the number of channels in preparation for subsequent concatenation operations, and finally, the integrated feature map is generated. At the same time, in order to further enhance the semantic information of the depth features, SA is applied to process the depth features, and the depth feature map is extracted through two convolution operations. Finally, the RGB feature map and the depth feature map are multiplied element-wise and concatenated along the channel dimension, and the final fusion feature map is obtained by performing a DW operation. Finally, by combining high-level abstract information, the number of channels is adjusted to match the number of categories to obtain the final segmentation result image. 6.The RGB-D oriented fusion model and the target detection method thereof according to claim 1, wherein, In step five, in order to extract the residual features of the RGB feature map from the depth feature map for semantic segmentation, the depthstream needs to have the ability to independently perform semantic segmentation; therefore, during training, it is necessary to calculate the ground truth map y and the depthstream output. The loss between them; this loss function is calculated using cross-entropy, and the specific formula is as follows: Final classification result It is also necessary to supervise by the true value map y, still using cross entropy to calculate, the formula is as follows: Meanwhile, in the edge enhancement fusion module, in order to better combine the RGB image features with the low-level and high-level RGB-D fusion features, the fusion module also needs to be supervised by the ground truth map y. At the output of the fusion module The error is calculated by the cross-entropy loss function, which is compared with the real label map y, and the specific formula is as follows: Therefore, the final loss function is: L total = L d + L rgb + L n This total loss function combines the loss from the depth flow, the loss from the RGB flow, and the loss from the edge enhancement fusion module to optimize the model's training process and improve the final semantic segmentation accuracy.

Images