An RGB-D real-time instance segmentation method based on cross-modal correction and context aggregation

Abstract

The application discloses an RGB-D real-time instance segmentation method based on cross-modal correction and context aggregation. First, the RGB image and the depth image of the scene are obtained and converted into HHA representation, and features are extracted in parallel by inputting a double-flow backbone network; second, a cross-modal feature correction module is introduced at each feature extraction stage, which calibrates features in both directions and suppresses noise through channel-level and spatial-level correction; then, a residual context aggregation module is integrated at the deepest layer of the network, which restores small-scale target details through global double-pooling and residual injection mechanism; finally, after multi-scale features are integrated by a pyramid encoder, the sparse instance activation decoder outputs the instance segmentation mask and classification results. The application effectively improves the geometric perception and small target segmentation accuracy, and realizes high frame rate real-time inference to meet the needs of robot navigation.

Description

Technical Field

[0001] This invention relates to the field of computer vision and image processing technology, and in particular to a real-time RGB-D instance segmentation method based on cross-modal correction and context aggregation for complex indoor scenes, applicable to scenarios such as autonomous navigation, grasping, and human-computer interaction of service robots. Background Technology

[0002] In the rapidly developing field of indoor service robots, robust scene understanding is a prerequisite for achieving downstream tasks such as autonomous navigation, robot grasping, and human-robot interaction. Unlike outdoor environments, indoor scenes are characterized by severe occlusion, complex layouts, and diverse object scales. Therefore, instance segmentation technology, which not only needs to identify object categories but also accurately delineate their boundaries, has become a core foundational technology for solving these challenges.

[0003] Currently, existing indoor scene instance segmentation technologies mainly face the following challenges and technical bottlenecks:

[0004] First, unimodal RGB methods suffer from geometric perception limitations. While existing RGB-based methods have achieved significant success, they often perform poorly in distinguishing objects with similar textures or chaotic geometric arrangements. RGB data is susceptible to variations in lighting and texture. In contrast, depth maps provide explicit geometric constraints by directly encoding physical distance and 3D spatial structure, serving as a powerful prior information for addressing geometric ambiguity. However, efficiently fusing RGB and depth modalities remains a challenge.

[0005] Second, existing RGB-D fusion methods suffer from high computational latency, making them difficult to meet real-time requirements. To incorporate depth information, existing RGB-D instance segmentation methods typically rely on computationally intensive Transformer-based attention mechanisms (such as the publicly disclosed Intra-Inter Attention), which incurs excessive computational costs. Traditional two-stage methods (such as the publicly disclosed Mask R-CNN algorithm) depend on complex candidate region generation steps, which also struggle to meet the demands of real-time applications such as robotics.

[0006] Third, high-speed single-stage models suffer from insufficient geometric perception and loss of details for small objects. To balance speed, pure sparse instance activation frameworks (such as the publicly available SparseInst algorithm) have emerged in recent years. These frameworks highlight information regions by predicting a set of sparse instance activation maps, achieving extremely fast inference speeds. However, when applied to complex indoor scenes, sparse activation frameworks, as single-modal RGB models, reveal key flaws: they struggle to distinguish geometrically similar objects or handle severe occlusions with misleading visual textures. Furthermore, during downsampling as the network increases its stride, the lack of global context guidance inevitably leads to the loss of fine-grained geometric details of small objects (such as lamps and cups), and fragile local features are easily contaminated by the high-frequency noise inherent in deep modalities.

[0007] In summary, existing deep learning methods struggle to balance real-time inference speed with robustness to complex geometry in indoor instance segmentation. There is an urgent need for a lightweight framework that can inherit the high-speed characteristics of sparse activation while effectively integrating the geometric robustness of RGB-D fusion and recovering details of small objects. Summary of the Invention

[0008] Technical problem: The technical problem that this invention aims to solve and the objective that it aims to achieve.

[0009] This invention primarily addresses the technical challenges of geometric perception limitations, high computational latency, and loss of detail in small targets during instance segmentation in complex indoor scenes. Existing single-modal RGB methods are highly susceptible to changes in lighting and texture when processing objects with similar textures or chaotic geometric arrangements, exhibiting severe limitations in geometric perception. Traditional RGB-D fusion methods incorporating depth information typically rely on computationally intensive attention mechanisms, resulting in excessively high computational costs and failing to meet the real-time requirements of downstream tasks such as service robots. Furthermore, existing high-speed sparse activation frameworks, lacking global context guidance during network downsampling, suffer from significant loss of fine-grained geometric details in small-scale targets and are highly susceptible to high-frequency noise inherent in depth modalities. Therefore, the goal of this invention is to improve the geometric perception accuracy and small target boundary localization accuracy of instance segmentation in complex indoor scenes, while significantly reducing computational latency and improving overall inference speed. To achieve the above objectives, this invention constructs a real-time instance segmentation framework based on cross-modal correction and context aggregation. First, RGB features and deep geometric features are extracted in parallel through a dual-stream backbone network. At each stage, a cross-modal feature correction module (CM-FRM) is introduced to actively calibrate semantics and suppress noise. At the same time, a residual context aggregation module (RCAM) is integrated in the deepest layer of the network to recover details of small targets. Finally, the fused features are fed into a sparse instance activation decoder to achieve high-precision, low-latency RGB-D real-time instance segmentation.

[0010] Technical solution: The complete technical means and methods of this invention.

[0011] To solve the above-mentioned technical problems, the specific technical solution adopted by the present invention is as follows:

[0012] S1: Multimodal data input and preprocessing.

[0013] Acquire RGB images and raw depth maps of the target indoor scene, and explicitly encode the raw depth map into a three-channel HHA representation that includes horizontal parallax, ground clearance, and local surface normal angle features;

[0014] S2: Dual-stream feature extraction and cross-modal feature correction.

[0015] A dual-stream backbone network is constructed, and the RGB image and the three-channel HHA representation are input into the dual-stream backbone network to extract RGB features and depth geometric features in parallel. A cross-modal feature correction module is introduced into each feature extraction stage of the dual-stream backbone network to perform bidirectional feature calibration on the RGB features and the depth geometric features through channel-level correction and spatial-level correction, and output multi-scale fusion features.

[0016] S3: Global context aggregation and detail recovery.

[0017] In the deepest layer of the dual-stream backbone network, the corresponding deep multi-scale fusion features are input into the residual context aggregation module; global scene context features are extracted in parallel through global average pooling and global max pooling, and the refined global scene context features are injected back into the backbone feature stream through residual connections to obtain global enhanced multi-scale features.

[0018] S4: Multi-scale feature decoding and sparse instance activation.

[0019] The pyramid encoder integrates the calibrated and aggregated multi-scale features from each stage, and then feeds them into the sparse instance activation decoder. The decoder predicts a set of sparse instance activation maps to aggregate features and outputs the instance segmentation mask and classification results of the target indoor scene.

[0020] Further, the channel-level correction in step S2 specifically includes: aggregating global statistical information of the RGB features and the depth geometric features using global average pooling; concatenating the global statistical information of the RGB features and the depth geometric features obtained by aggregation into a joint statistical descriptor, and then mapping it through a multilayer perceptron and processing it through a Sigmoid activation function to separate and output RGB channel weights and depth channel weights; multiplying the RGB channel weights with the depth geometric features at the channel level, and multiplying the depth channel weights with the RGB features at the channel level to obtain the channel-level corrected features.

[0021] Further, the spatial correction process in step S2 specifically includes: concatenating the RGB features with the depth geometric features, extracting local neighborhood information and aggregating it to generate an intermediate feature map through a 3x3 convolutional layer, a non-linear activation function, and a 1x1 convolutional layer in sequence; processing the intermediate feature map with a Sigmoid activation function to separate it into RGB spatial weights and depth spatial weights; and performing spatial multiplication of the RGB spatial weights with the depth geometric features, and performing spatial multiplication of the depth spatial weights with the RGB features to obtain the spatial correction features.

[0022] Furthermore, the process of the cross-modal feature correction module outputting multi-scale fusion features specifically involves: introducing learnable parameters, and performing weighted residual fusion of the channel-level correction features and the spatial-level correction features with the corresponding RGB features and the depth geometric features, respectively, to obtain the multi-scale fusion features.

[0023] Further, the processing procedure of the residual context aggregation module in step S3 specifically includes: performing global average pooling and global max pooling operations in parallel on the deep multi-scale fusion features input to the module to obtain global average pooling vectors and global max pooling vectors; concatenating the global average pooling vectors and global max pooling vectors, and generating channel-level attention weights by sequentially passing them through a multilayer perceptron and a sigmoid activation function; multiplying the channel-level attention weights with the deep multi-scale fusion features at the channel level to obtain context-aware features; refining the context-aware features through a 1x1 convolutional layer, and directly injecting the refined features back into the deep multi-scale fusion features through residual connections to generate the globally enhanced multi-scale features.

[0024] Furthermore, in step S4: the sparse instance activation decoder does not rely on dense candidate boxes, but directly generates mask kernels, category predictions, and instance confidence scores based on the instance activation graph; during the model training phase, a bipartite graph matching strategy is used to calculate the loss function between the prediction results and the true labels.

[0025] Beneficial effects: The benefits and indicators achieved by this invention.

[0026] 1. Overcoming Geometric Perception Limitations and Enhancing Robustness in Complex Scenes: This invention introduces a cross-modal feature correction module (CM-FRM), breaking through the limitations of traditional pure RGB methods. This module is not merely a simple feature stitching process, but actively utilizes depth geometric information to bidirectionally calibrate RGB semantic features, effectively suppressing modality-specific noise and solving the perceptual blurring problem caused by severe indoor occlusion and changes in lighting.

[0027] 2. Significantly improves the accuracy of small-scale target recognition and boundary segmentation: To address the issue of small target feature loss caused by network downsampling, this invention designs a Residual Context Aggregation (RCAM) module at the network bottleneck. This module utilizes global context to guide local features, successfully filtering out the interference of depth noise and achieving improved average accuracy for small targets (…). It significantly improves upon the previous version and exhibits high geometric fidelity in the fit of the overall object boundary.

[0028] 3. Achieves an excellent balance between accuracy and speed (meeting stringent real-time requirements): This invention significantly improves geometric perception accuracy while inheriting the efficient nature of sparse instance activation. By avoiding computationally intensive Transformer attention mechanisms and complex candidate region generation steps, this method achieves a real-time inference speed of 44.5 frames per second (FPS) while ensuring high-precision instance segmentation, perfectly suited for downstream robot grasping, navigation, and other practical applications with extremely low latency requirements. Attached Figure Description

[0029] Figure 1 A general flowchart of an RGB-D real-time instance segmentation method based on cross-modal correction and context aggregation provided in an embodiment of the present invention;

[0030] Figure 2 A network architecture diagram (DSR-Net) provided for an embodiment of the present invention;

[0031] Figure 3 This is a schematic diagram of the internal structure of the cross-modal feature correction module (CM-FRM) provided in an embodiment of the present invention;

[0032] Figure 4 This is a schematic diagram of the internal structure of the Residual Context Aggregation Module (RCAM) provided in an embodiment of the present invention. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0034] This invention provides a real-time RGB-D instance segmentation method based on cross-modal correction and context aggregation, combined with... Figure 1 The overall flowchart shown illustrates the specific steps included in this method:

[0035] S1: Multimodal data input and preprocessing.

[0036] Acquire RGB images and raw depth maps of complex indoor scenes. Since the raw single-channel depth map lacks discriminative patterns, this embodiment explicitly encodes it into a three-channel HHA representation: horizontal parallax, ground clearance, and local surface normal angle. This transformation not only provides robust geometric cues unaffected by lighting variations but also aligns with the input dimensions of the pre-trained ResNet backbone network.

[0037] S2: Dual-stream feature extraction and cross-modal feature correction.

[0038] like Figure 2 As shown in (a) of the dual-stream feature extraction backbone network, a dual-stream ResNet backbone network is constructed to extract RGB features in parallel. HHA characteristics To fully utilize the complementarity of both and suppress specific modal noise, cross-modal feature correction modules (CM-FRM) are introduced at various stages of the network. Combined with... Figure 3 The internal structure of the cross-modal feature correction module is shown below. The specific steps of the correction process of this module are as follows:

[0039] S21: Channel-level calibration.

[0040] Reference Figure 3 The upper channel-level correction branch first uses global average pooling (GAP) to aggregate global statistics for both modalities, then concatenates them into a joint statistical descriptor. Then through a multilayer perceptron ( The values ​​are mapped to channel-level attention weights, processed by an internal sigmoid activation function (normalized to between 0 and 1), and then subjected to a separation operation. ) Obtain RGB channel weights and depth channel weights The calculation formula is shown below:

[0041]

[0042] In the formula, This represents the Sigmoid activation function. Subsequently, the aforementioned weights are multiplied at the channel level by the features of another modality. (), complete the cross-injection, and obtain the channel-level correction features. and The calculation formula is shown below:

[0043]

[0044] S22: Spatial level correction.

[0045] Reference Figure 3The lower spatial correction branch concatenates the RGB and HHA features using a non-linear activation function (ReLU, denoted as...) )of Convolutional layers extract local neighborhood information to suppress high-frequency depth noise, followed by... Convolutional aggregation generates intermediate feature maps The calculation formula is shown below:

[0046]

[0047] Next, the intermediate feature map Separate into RGB space weights and depth space weights The calculation formula is shown below:

[0048]

[0049] Finally, these spatial weights are spatially multiplied with the input features ( ), to obtain spatial-level correction features and The calculation formula is shown below:

[0050]

[0051] S23: Adaptive feature fusion.

[0052] Reference Figure 3 The feature fusion section on the right introduces learnable parameters. and (Initialized to 0.5), the channel-level correction features, spatial-level correction features, and original features are weighted and fused using residuals. This mechanism allows the network to adaptively determine the optimal fusion balance to obtain the final output fused features. and The calculation formula is shown below:

[0053]

[0054] S3: Global context aggregation and detail recovery.

[0055] In the deepest layer of the feature extraction network (i.e., the output of Layer 4) Fusion features, corresponding At the bottleneck of scale downsampling, fine-grained geometric details are easily lost due to high-frequency noise contamination. Therefore, this embodiment integrates a Residual Context Aggregation (RCAM) module at this point, such as... Figure 4 The internal structure of the residual context aggregation module is shown below, and its specific steps are as follows:

[0056] S31: Generate contextual attention weights.

[0057] like Figure 4 As shown, the fusion features Parallel input is fed to the Global Average Pooling (GAP) and Global Max Pooling (GMP) branches. The pooling vectors are concatenated ( The input is then fed into a multilayer perceptron and processed by a sigmoid activation function to generate channel attention weights. The calculation formula is shown below:

[0058]

[0059] S32: Residual Injection and Feature Refinement.

[0060] Subsequently, this weight is multiplied by the input features through channels, and then... After convolutional thinning, the features are directly injected into the original feature stream through lightweight residual connections to generate global context-enhanced features. The calculation formula is shown below:

[0061]

[0062] This module effectively restores the details of small-scale indoor targets (such as cups and table lamps) and sharpens object boundaries without increasing feature dimensions.

[0063] S4: Multi-scale feature decoding and sparse instance activation.

[0064] like Figure 2 As shown in (b) the pyramid encoder and (c) the sparse instance activation decoder, the calibrated and aggregated multi-scale features are input into the pyramid encoder for multi-scale feature fusion. Subsequently, they are fed into the decoder based on the sparse instance activation architecture. This decoder abandons dense anchor boxes or complex region proposal mechanisms, mainly comprising an instance feature branch and a mask feature branch. First, a set of sparse instance activation maps (IAMs) is predicted through the instance feature branch to highlight the information regions of the target. Then, features are aggregated based on the instance activation maps to directly generate instance mask kernels, class predictions, and instance confidence scores. Simultaneously, the mask feature branch generates high-resolution mask features and operates on them with the generated mask kernels to generate the final mask. During the model training phase, a bipartite matching strategy is used to calculate the loss function between the predicted results and the true labels, achieving end-to-end model optimization. Finally, while achieving inference speeds that meet real-time requirements, the output is an instance segmentation mask and classification results with accurate boundaries for the target indoor scene.

Claims

1. A real-time RGB-D instance segmentation method based on cross-modal correction and context aggregation, characterized in that, Includes the following steps: S1: Multimodal data input and preprocessing. Acquire RGB images and raw depth maps of the target indoor scene, and explicitly encode the raw depth map into a three-channel HHA representation that includes horizontal parallax, ground clearance, and local surface normal angle features; S2: Dual-stream feature extraction and cross-modal feature correction. A dual-stream backbone network is constructed, and the RGB image and the three-channel HHA representation are input into the dual-stream backbone network to extract RGB features and depth geometric features in parallel. A cross-modal feature correction module is introduced into each feature extraction stage of the dual-stream backbone network to perform bidirectional feature calibration on the RGB features and the depth geometric features through channel-level correction and spatial-level correction, and output multi-scale fusion features. S3: Global context aggregation and detail recovery. In the deepest layer of the dual-stream backbone network, the corresponding deep multi-scale fusion features are input into the residual context aggregation module; global scene context features are extracted in parallel through global average pooling and global max pooling, and the refined global scene context features are injected back into the backbone feature stream through residual connections to obtain global enhanced multi-scale features. S4: Multi-scale feature decoding and sparse instance activation. The pyramid encoder integrates the calibrated and aggregated multi-scale features from each stage, and then feeds them into the sparse instance activation decoder. The decoder predicts a set of sparse instance activation maps to aggregate features and outputs the instance segmentation mask and classification results of the target indoor scene.

2. The method according to claim 1, characterized in that, The channel-level correction in step S2 specifically includes: Global average pooling is used to aggregate global statistical information for the RGB features and the depth geometric features respectively; The global statistical information of the RGB features and the depth geometric features obtained by aggregation are concatenated into a joint statistical descriptor, which is then mapped through a multilayer perceptron and processed by the Sigmoid activation function to separate the output into RGB channel weights and depth channel weights. The RGB channel weights are multiplied with the depth geometric features at the channel level, and the depth channel weights are multiplied with the RGB features at the channel level to obtain the channel-level correction features.

3. The method according to claim 1, characterized in that, The spatial-level correction process in step S2 specifically includes: The RGB features are concatenated with the deep geometric features, and local neighborhood information is extracted and aggregated through a 3x3 convolutional layer, a non-linear activation function, and a 1x1 convolutional layer in sequence to generate an intermediate feature map. After the intermediate feature map is processed by the Sigmoid activation function, it is separated into RGB spatial weights and depth spatial weights; the RGB spatial weights are then multiplied spatially with the depth geometric features, and the depth spatial weights are then multiplied spatially with the RGB features to obtain spatially corrected features.

4. The method according to claims 2 and 3, characterized in that, The process by which the cross-modal feature correction module outputs multi-scale fused features is as follows: Learnable parameters are introduced, and the channel-level correction features and the spatial-level correction features are respectively weighted and residually fused with the corresponding RGB features and the depth geometric features to obtain the multi-scale fusion features.

5. The method according to claim 1, characterized in that, The processing procedure of the residual context aggregation module in step S3 specifically includes: The deep multi-scale fusion features input to this module are subjected to parallel global average pooling and global max pooling operations to obtain global average pooling vectors and global max pooling vectors. The global average pooling vector and the global max pooling vector are concatenated, and then channel-level attention weights are generated by sequentially passing them through a multilayer perceptron and a sigmoid activation function. The channel-level attention weights are multiplied with the deep multi-scale fusion features at the channel level to obtain context-aware features; The context-aware features are refined using 1x1 convolutional layers, and the refined features are directly injected back into the deep multi-scale fusion features through residual connections to generate the globally enhanced multi-scale features.

6. The method according to claim 1, characterized in that, In step S4: The sparse instance activation decoder does not rely on dense candidate boxes, but directly generates mask kernels, category predictions, and instance confidence scores based on the instance activation map; During the model training phase, a bipartite graph matching strategy is used to calculate the loss function between the predicted results and the true labels.

Images