An RGBD image instance segmentation method and system based on feature fusion and boundary enhancement

Abstract

This invention relates to the field of image processing technology, and more particularly to an RGBD image instance segmentation system based on adaptive fusion and boundary enhancement. The system includes a data preprocessing module, a dual-stream feature extraction backbone network, an adaptive cross-modal fusion module, a multi-scale feature fusion network, a selective boundary aggregation module, and an instance segmentation prediction head. Specifically, the data preprocessing module encodes the RGB and depth images into a dual-modal input; the dual-stream feature extraction backbone network extracts RGB and depth features in parallel; the adaptive cross-modal fusion module employs a separation and aggregation gating mechanism to suppress noise and adaptively fuse the dual-modal features at multiple levels; the multi-scale feature fusion network aggregates the fused multi-level features; the selective boundary aggregation module extracts detailed information from shallow features and selectively injects high-level semantic features to refine the boundaries; and the instance segmentation prediction head finally generates the instance segmentation result. This invention significantly improves instance segmentation accuracy in complex indoor scenes, especially at small targets and blurred boundaries, through adaptive fusion and boundary refinement.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically, to an RGBD image instance segmentation method and system based on feature fusion and boundary enhancement. Background Technology

[0002] With the rapid development of applications such as smart homes and service robots, the importance of indoor scene understanding technology is becoming increasingly prominent. Instance segmentation, as one of the core tasks of scene understanding, requires algorithms to simultaneously distinguish different object instances and identify their categories at the pixel level. However, indoor environments often suffer from severe object occlusion, weak features of small objects, and confusion between regions with similar textures, leading to shortcomings in existing instance segmentation methods when handling such scenarios, such as blurred boundaries, missed detections of small objects, or false detections.

[0003] Early instance segmentation methods primarily relied on hand-designed features (such as HOG and SIFT) and classifiers (such as SVM and Random Forest). However, these methods are highly sensitive to changes in lighting and texture similarity, exhibit poor generalization ability, and struggle to handle the diversity of indoor scenes. With the development of deep learning, methods based on convolutional neural networks have gradually become mainstream. For example, Mask R-CNN achieves end-to-end instance segmentation by adding mask branches, but its two-stage structure is computationally intensive and difficult to meet real-time requirements. Single-stage methods such as YOLACT achieve real-time segmentation through prototype masks and dynamic convolutional kernels, but their accuracy remains insufficient for small objects and complex boundaries.

[0004] In recent years, the widespread adoption of RGBD sensors has provided a new dimension for indoor scene understanding. Depth information can provide geometric structure and spatial context, helping to distinguish objects with similar textures but different spatial locations. However, most existing RGBD instance segmentation methods use simple feature concatenation or addition for fusion, failing to effectively handle noise in depth data and ignoring the feature complementarity between RGB and depth modalities, resulting in poor fusion performance. Furthermore, the spatial resolution of deep features decreases after multiple downsampling, and boundary details are severely lost, further exacerbating the problems of small object segmentation and boundary blurring.

[0005] Therefore, how to fully leverage the complementary advantages of RGB and depth information while maintaining real-time performance, and enhance the network's ability to perceive boundaries and small objects, is a key issue that current RGBD indoor instance segmentation technology urgently needs to address. Summary of the Invention

[0006] Technical Problem: This invention aims to solve the following technical problems existing in current RGBD instance segmentation methods: Insufficient cross-modal fusion: Simple feature concatenation or addition fails to effectively utilize the complementarity between RGB and depth information, and noise in depth data easily contaminates RGB features; Loss of boundary details: The spatial resolution of deep features decreases after multiple downsampling, resulting in blurred boundaries of the prediction mask; Low segmentation accuracy of small objects: Existing methods are not good at sensing small targets, and are prone to missed detections or false detections.

[0007] To achieve the above objectives, the present invention provides an RGBD image instance segmentation system based on adaptive fusion and boundary enhancement, the system comprising: The data preprocessing module is used to acquire RGB images and depth images of the same scene, encode the depth image to obtain an encoded depth feature map, and use the RGB image and the encoded depth feature map as bimodal inputs. The dual-stream feature extraction backbone network includes a first feature extraction stream and a second feature extraction stream with identical and parallel structures. The first feature extraction stream is used to extract RGB feature maps at multiple levels, and the second feature extraction stream is used to extract depth feature maps at multiple levels. An adaptive cross-modal fusion module is connected to multiple target layers in the dual-stream feature extraction backbone network, and is used to perform cross-modal adaptive fusion of RGB feature maps and depth feature maps of the same layer at each target layer, and output the fused feature map. A multi-scale feature fusion network, connected to the adaptive cross-modal fusion module, is used to receive and fuse the fused feature maps from different levels and output multi-scale fused features. The selective boundary aggregation module, connected to the multi-scale feature fusion network, is used to select shallow features from the multi-scale fused features, extract their detailed information, and selectively inject the detailed information into high-level features to output high-level features with enhanced boundaries. The instance segmentation prediction head, connected to the selective boundary aggregation module, is used to generate a segmentation mask for each instance in the image based on the high-level features enhanced by the boundary.

[0008] This invention also provides an RGBD image instance segmentation method based on adaptive fusion and boundary enhancement, applied to the above-mentioned system, including steps such as data preprocessing, dual-stream feature extraction, cross-modal fusion, multi-scale fusion, boundary enhancement, and instance segmentation.

[0009] Beneficial effects: Compared with the prior art, the present invention has the following beneficial effects: 1. Improve cross-modal fusion quality: Through the separation-aggregation gating mechanism of the adaptive cross-modal fusion module, noise in deep data is effectively suppressed, complementary features are dynamically selected for fusion, and the robustness of feature representation is enhanced; 2. Enhanced boundary segmentation accuracy: By injecting shallow detail features into high-level semantic features through a selective boundary aggregation module, the boundary clarity of instance segmentation is significantly improved, and the adhesion of adjacent instances is reduced; 3. Improved small object segmentation capability: The boundary enhancement mechanism preserves the spatial details of small objects. Experiments on the SUNRGBD dataset show that the segmentation accuracy of small objects is improved by 31.4%. 4. Maintain real-time processing capability: While improving accuracy, the number of model parameters and computational load increases only to a limited extent, still meeting the needs of real-time applications. Attached Figure Description

[0010] Figure 1 A flowchart illustrating the overall process of the indoor instance segmentation method based on RGBD images provided in this embodiment of the invention; Figure 2 This is a schematic diagram of the RTM-FuseBound network structure provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the separation-aggregation gating fusion module structure provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the selective boundary aggregation module structure provided in an embodiment of the present invention; Figure 5 This is an example image of instance segmentation results on the SUNRGBD dataset according to an embodiment of the present invention. Detailed Implementation

[0011] This paper develops an RGBD image instance segmentation method and system based on feature fusion and boundary enhancement. The method is an improvement on the single-stage instance segmentation network—RTMDet-Ins—by introducing a separation-aggregation gated fusion module and a selective boundary aggregation module to construct an RTM-FuseBound network. In the first stage, deep features of the RGB and HHA images are extracted separately through a dual-stream coding backbone network. In the second stage, the separation-aggregation gated fusion module achieves adaptive fusion of cross-modal features. In the third stage, the selective boundary aggregation module enhances the boundary information of deep features. Finally, the detection and segmentation head outputs target detection boxes and instance segmentation masks. The flowchart is shown below. Figure 1 As shown.

[0012] S1: Experimental Data Acquisition and Preprocessing This invention utilizes the publicly available dataset SUNRGBD, which contains 10,335 RGBD images of indoor scenes, covering various indoor environments, including bedrooms, living rooms, dining rooms, and offices. The data was acquired using four different depth sensors to ensure data diversity.

[0013] Data preprocessing includes the following steps: S11: Perform HHA encoding transformation on the depth image. HHA encoding includes three channels: horizontal disparity, height above ground, and surface normal angle, which can effectively represent the geometric structure and spatial information of the depth image. The specific transformation process is as follows: First, calculate the disparity value from the original depth map; then estimate the height of each pixel relative to the ground based on the camera parameters; finally, calculate the angle between the surface normal and the direction of gravity.

[0014] S12: Perform uniform size adjustment on RGB and HHA images, scaling all images to 640×640 pixels while maintaining the original aspect ratio, and padding any insufficient areas.

[0015] S13: Normalize the images. RGB images are normalized using the mean and standard deviation of the ImageNet dataset; HHA images are normalized using a specific statistic calculated from the training set.

[0016] S14: Data augmentation. Random horizontal flipping (probability 0.5) and YOLOX-style HSV color perturbation are used to increase the diversity of training data and improve the model's generalization ability.

[0017] S2: RTM-FuseBound Network Construction The RTM-FuseBound network is an improvement on the single-stage instance segmentation model RTMDet-Ins, with the overall network structure as follows: Figure 2 As shown, it mainly includes the following components: a dual-stream coding backbone network, a separation-aggregation gating fusion module, a feature pyramid network, a selective boundary aggregation module, and a detection and segmentation head.

[0018] The specific improvements are reflected in the following aspects: S21: Construction of a Two-Stream Coding Backbone Network The original single-stream CSPNEXt backbone of RTMDet-Ins was extended into a dual-stream parallel structure to process input images in RGB and HHA modalities respectively. Both streams use the same CSPNEXt architecture but do not share weights to extract texture color features for the RGB modality and geometric structure features for the HHA modality, respectively. Each branch contains five stages, extracting feature maps at different scales through progressive downsampling. The feature maps from the last three stages (C3, C4, and C5) are selected for subsequent cross-modal fusion. These three stages have spatial resolutions of 80×80, 40×40, and 20×20, with 256, 512, and 1024 channels, respectively.

[0019] S22: Construction of Separation-Aggregation Gated Fusion Module like Figure 3 As shown, the separation-aggregation gated fusion module is set at the output of layers C3, C4, and C5 of the dual-stream coding backbone network, and is used to perform adaptive cross-modal fusion of RGB feature maps and HHA feature maps at the same layer. This module consists of two parts: a separation unit and an aggregation unit.

[0020] S221: Separation Unit Design. The separation unit employs a cross-modal channel attention mechanism to achieve noise suppression and feature enhancement in the following ways: First, the input RGB feature map... and HHA feature map Global average pooling is performed separately to obtain global feature descriptors for both modalities. These descriptors are then input into a shared multilayer perceptron to generate channel attention weights. These generated channel attention weights are applied to the feature maps of the other modality to achieve cross-modal noise suppression, resulting in filtered features. and Finally, the filtered features are added to the original features through residual connections to achieve bidirectional feature enhancement, resulting in the enhanced features. and The calculation formula is as follows: S222: Aggregation Unit Design. The aggregation unit employs a spatial attention mechanism to achieve feature fusion in the following way: the two enhanced feature maps are concatenated along the channel dimension, and an adaptive fusion weight map in the spatial dimension is generated through a 3×3 convolutional layer and a softmax function. This weight map represents the contribution ratio of RGB features and HHA features at each spatial location; the two enhanced features are weighted and summed according to the generated weight map to obtain the fused feature. Finally, the fused features and the original RGB features are weighted and averaged to obtain the final output features for this level. The calculation formula is as follows: in To achieve optimal performance, the fusion weighting coefficient was set to 0.5 based on the experimental results of parameter sensitivity analysis.

[0021] S23: Feature Pyramid Network Construction The feature pyramid network adopts a CSP-PAFPN structure and receives fused multi-level features. As input, the network transmits high-level semantic information to lower levels via a top-down path, and then transmits low-level detailed information to higher levels via a bottom-up path, achieving full fusion of multi-scale features. The final output consists of feature maps at three scales: P3, P4, and P5, corresponding to high-resolution, medium-resolution, and low-resolution features, respectively.

[0022] S24: Construction of Selective Boundary Aggregation Module like Figure 4 As shown, the selective boundary aggregation module is placed after the feature pyramid network, receiving shallow feature map P3 and deep feature map P5 as input, and performs boundary enhancement through the following sub-steps: S241: Detail Feature Extraction. The shallow feature map P3 is input into two consecutive 3×3 convolutional layers, followed by a batch normalization layer and a ReLU activation function after each convolutional layer to extract detailed features rich in boundary information. The calculation formula is as follows: S242: Spatial weighting. This involves assigning detailed features... Spatial attention map is generated by passing a 1×1 convolutional layer followed by a sigmoid function. The attention map has a value range of [0,1], representing the probability that each spatial location belongs to the boundary. The spatial attention map is then multiplied element-wise with the detail features to obtain the weighted detail features. To highlight detailed information in boundary areas and suppress noise in flat areas, the calculation formula is as follows: S243: Feature Injection. The weighted detail features are then subjected to a 1×1 convolution for channel projection, so that the number of channels is equal to that of the P5 features. Figure 1 To achieve this, the injected features were obtained by upsampling to the same spatial size as P5 using bilinear interpolation. The calculation formula is as follows: The injected features are added element-wise to P5 to enhance its boundaries; finally, a 3×3 convolutional layer and residual connections are used for smoothing to obtain the deep feature map with enhanced boundaries. The calculation formula is as follows: S25: Detection and Segmentation Head Construction The detection and segmentation head consists of two parts: the detection head branch and the mask head branch.

[0023] S251: Detection Head Branch Design. The detection head branch employs a design with shared weights but independent batch normalization layers. That is, feature maps of different scales share convolutional layer weights but are equipped with independent batch normalization layers to adapt to the statistical characteristics of their respective scales. The detection head outputs the target's class probability and bounding box coordinates.

[0024] S252: Mask Head Branch Design. The mask head branch consists of a mask feature head and a kernel prediction head. The mask feature head contains four convolutional layers, extracting an 8-channel mask feature map from the multi-scale feature map. The kernel prediction head predicts a 169-dimensional vector for each detected instance, which contains all the parameters required by the three dynamic convolutional layers, thus forming three dynamic convolutional kernels. During the inference phase, the dynamic convolutional kernels are convolved with the mask feature map to generate a binary mask for the instance. This dynamic convolution mechanism avoids traditional RoI operations, significantly reducing computational overhead.

[0025] S26: Loss Function Design During model training, a multi-task loss function is used for joint optimization, and the total loss function is: S261: Classification Loss The Quality Focal Loss (QFL) loss function is used to optimize the accuracy of target category prediction.

[0026] S262: Bounding Box Regression Loss We introduce the Generalized Intersection over Union (GIoU) ​​loss. The GIoU loss solves the gradient vanishing problem of traditional IoU when the bounding boxes do not overlap by calculating the minimum closed rectangle between the predicted bounding box and the true bounding box and measuring the difference between the closed rectangle and the union.

[0027] S263: Masking Loss The Dice loss function is used to optimize the overlap between the predicted mask and the real mask. Simultaneously, during the label assignment stage, a soft region prior based on the mask centroid is introduced to further improve mask localization accuracy.

[0028] S3: Model Training and Inference In this stage, the RTM-FuseBound network is trained, and supervised learning enables the model to accurately predict the bounding box, category, and instance segmentation mask of the target.

[0029] S31: Dataset Labeling Using the annotations from the SUNRGBD dataset, 19 of the most common indoor object categories (including beds, sofas, tables, chairs, monitors, etc.) were selected as target categories. The annotation information includes the target category, bounding box coordinates, and instance segmentation mask. The annotation results were converted into a JSON file in COCO format.

[0030] S32: Model Training The RTM-FuseBound network was trained using a labeled dataset. During training, a multi-task loss function was used to jointly optimize the performance of object classification, bounding box regression, and mask segmentation. The AdamW optimizer was employed, with an initial learning rate set to [value missing]. The weight decay coefficient was 0.05, and a cosine annealing learning rate scheduling strategy was adopted. Training lasted for 36 epochs with a batch size of 8. Exponential moving average and gradient clipping were introduced to enhance training stability.

[0031] S33: Model Reasoning The trained RTM-FuseBound network is used to perform inference on new RGBD images. The RGBD image to be processed is input into the model, and the model outputs the detection box, class label, and corresponding instance segmentation mask for each instance.

[0032] S4: Ablation Experiments and Parameter Analysis To verify the effectiveness of each module of the present invention, an ablation experiment was conducted on the SUNRGBD dataset.

[0033] S41: Module Validity Verification Four sets of comparative experiments were set up: the baseline RTMDet-Ins model, the model with only the SA-Gate module added, the model with only the SBA module added, and the complete RTM-FuseBound model. Experimental results show that the complete model achieves optimal performance, with an average mask accuracy of 0.272, a 5.8% improvement over the baseline; and an average detection box accuracy of 0.302, a 4.1% improvement over the baseline. In particular, the small object segmentation accuracy reached 0.046, a 31.4% improvement over the baseline, verifying the effectiveness of the proposed method in handling small objects and complex boundaries.

[0034] S42: Sensitivity Analysis of Fusion Weight Parameters For fusion weight coefficients Perform parameter sensitivity analysis and set The values ​​range from 0.1 to 0.9, with intervals of 0.2. Experimental results show that when The model achieves optimal performance when the accuracy is 0.5, with an average mask accuracy of 0.269 and an average detection box accuracy of 0.306. A value that is too small (0.1) cannot fully utilize depth information. An excessively large value (0.9) may introduce deep noise pollution features. Based on this analysis, the present invention will... Set it to 0.5 as the default parameter.

[0035] S5: Comparison Experiment with Existing Methods The method of this invention is compared with existing mainstream instance segmentation methods SOLOv2 and YOLACT under the same experimental conditions.

[0036] S51: Quantitative Comparison Experimental results show that the average mask accuracy of the method of the present invention is 0.272, which is significantly better than SOLOv2's 0.231 and YOLACT's 0.217, representing improvements of 17.7% and 25.3%, respectively. In terms of model efficiency, the method of the present invention has 111M parameters and 142 GFLOPs of computation, which is slightly higher than the comparative methods, but still within an acceptable range.

[0037] S52: Qualitative Comparison Visualization results demonstrate that the method of this invention outperforms the comparative methods in terms of boundary clarity, small target recognition capability, and instance discrimination. Particularly in complex scenes with object occlusion and similar textures, the method of this invention can better utilize depth geometric information to distinguish different instances, reducing missegmentation and boundary adhesion phenomena.

[0038] S6: Output Results For each instance detected in the image, output its bounding box coordinates, class label, and instance segmentation mask. Visualize the results and save or export them as a structured data file for subsequent analysis and applications.

[0039] The above description is merely a preferred embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. An RGBD image instance segmentation system based on adaptive fusion and boundary enhancement, characterized in that, The system includes a data preprocessing module, a dual-stream feature extraction backbone network, an adaptive cross-modal fusion module, a multi-scale feature fusion network, a selective boundary aggregation module, and an instance segmentation prediction head, wherein: The data preprocessing module is used to acquire RGB images and depth images of the same scene, encode the depth image to obtain an encoded depth feature map, and use the RGB image and the encoded depth feature map as bimodal inputs. The dual-stream feature extraction backbone network includes a first feature extraction stream and a second feature extraction stream with identical and parallel structures. The first feature extraction stream is used to process the RGB image to extract RGB feature maps at multiple levels; the second feature extraction stream is used to process the encoded depth feature map to extract depth feature maps at multiple levels. An adaptive cross-modal fusion module is connected to multiple target layers in the dual-stream feature extraction backbone network, and is used to perform cross-modal adaptive fusion of the input RGB feature map and depth feature map of the same layer at each target layer, and output the fused feature map. A multi-scale feature fusion network, connected to the adaptive cross-modal fusion module, is used to receive and fuse the fused feature maps from different levels and output multi-scale fused features. The selective boundary aggregation module, connected to the multi-scale feature fusion network, is used to select shallow features containing rich detailed information from the multi-scale fused features, extract the detailed information of the shallow features, and selectively inject the detailed information into high-level features containing rich semantic information, and output the high-level features with enhanced boundaries. The instance segmentation prediction head, connected to the selective boundary aggregation module, is used to generate a segmentation mask for each instance in the image based on the high-level features enhanced by the boundary.

2. The RGBD image instance segmentation system based on adaptive fusion and boundary enhancement as described in claim 1, characterized in that, The adaptive cross-modal fusion module further includes a separation unit and an aggregation unit; The separation unit is configured to generate a first channel attention weight based on the RGB feature map, and perform noise suppression on the depth feature map based on the first channel attention weight to obtain a first cleaned feature; simultaneously, it generates a second channel attention weight based on the depth feature map, and performs noise suppression on the RGB feature map based on the second channel attention weight to obtain a second cleaned feature; The aggregation unit is used to generate a spatial selection weight map based on the first enhanced feature obtained by fusing the RGB feature map with the first purification feature and the second enhanced feature obtained by fusing the depth feature map with the second purification feature, and to perform a weighted summation of the first enhanced feature and the second enhanced feature based on the spatial selection weight map to obtain a fused feature map.

3. The RGBD image instance segmentation system based on adaptive fusion and boundary enhancement as described in claim 2, characterized in that, The adaptive cross-modal fusion module is further used to perform a weighted average of the fused feature map and the RGB feature map to obtain the final output feature map.

4. The RGBD image instance segmentation system based on adaptive fusion and boundary enhancement as described in claim 1, characterized in that, The selective boundary aggregation module further includes: The detail extraction unit is used to perform a convolution operation on the shallow features to extract a detail feature map; A spatial selection unit is used to convolve and normalize the detail feature map to generate a spatial selection mask, which is used to identify boundary regions in the image. The feature injection unit is used to weight the detail feature map using the spatial selection mask, and then upsample and channel-project the weighted detail feature map to match the size and number of channels of the high-level feature. Finally, the processed detail feature map is added to the high-level feature to obtain the boundary-enhanced high-level feature.

5. The RGBD image instance segmentation system based on adaptive fusion and boundary enhancement as described in claim 1, characterized in that, The data preprocessing module encodes the depth image, specifically by encoding the depth image into an HHA three-channel feature map containing horizontal parallax, ground height, and surface normal angle.

6. The RGBD image instance segmentation system based on adaptive fusion and boundary enhancement as described in claim 1, characterized in that, The dual-stream feature extraction backbone network is the CSPNEXt network, and the adaptive cross-modal fusion module is connected after the last three target layers of the CSPNEXt network.

7. The RGBD image instance segmentation system based on adaptive fusion and boundary enhancement as described in claim 1, characterized in that, The multi-scale feature fusion network is the Path Aggregation Feature Pyramid Network (PAFPN), which is used to perform bidirectional fusion of the multi-level feature maps output by the adaptive cross-modal fusion module from top to bottom and bottom to top.

8. The RGBD image instance segmentation system based on adaptive fusion and boundary enhancement as described in claim 1, characterized in that, The instance segmentation prediction head further includes a detection head branch and a mask branch; The detection head branch is used to predict the bounding box and category for each instance; The mask branch includes a mask feature generation head and a kernel prediction head. The kernel prediction head is used to predict a dynamic convolution kernel for each instance. The mask feature generation head is used to generate a mask feature map and convolve the mask feature map with the dynamic convolution kernel to generate an instance segmentation mask.

9. A method for RGBD image instance segmentation based on adaptive fusion and boundary enhancement, applied to the system described in any one of claims 1-8, characterized in that, The method includes: Step S1: Acquire RGB images and depth images of the same scene, and encode the depth image to obtain an encoded depth feature map; Step S2: Extract multiple levels of RGB feature maps and depth feature maps from the RGB image and the encoded depth feature map respectively through a parallel dual-stream feature extraction backbone network; Step S3: At multiple target levels, perform cross-modal adaptive fusion of RGB feature maps and depth feature maps at the same level, and output the fused feature map; Step S4: Perform multi-scale fusion on the fused feature maps from different levels using a multi-scale feature fusion network to output multi-scale fused features; Step S5: Select shallow features from the multi-scale fused features, extract their detailed information, and selectively inject the detailed information into high-level features to output the high-level features with enhanced boundaries; Step S6: Generate a segmentation mask for each instance in the image based on the enhanced high-level features of the boundary.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method as described in claim 9.

Images