RGB-D Scene 3D Object Detection Method Guided by Object Region Information

Abstract

This invention relates to a method for 3D object detection in RGB-D scenes guided by object region information. The invention inputs RGB-D scene data, extracts the 2D region of the target object from the color image and performs coarse classification of the object, then elevates the 2D bounding box of the object region to a 3D oblique cone region and converts it into point cloud data. Next, feature extraction is performed on the oblique cone point cloud using object region information, and global and local features of the point cloud are effectively fused using feature transformation and max pooling operations. Using the fused features, foreground and background points are segmented, and background points are removed to form a masking point cloud. Finally, object center point voting is performed in the masking point cloud, and suggestions and 3D object predictions are made using object region information. This invention, guided by object region information and fusing global and local point cloud features, can effectively eliminate background interference and efficiently and accurately detect 3D target objects in complex scenes.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology and relates to a method for 3D target detection in RGB-D scenes guided by object region information. Background Technology

[0002] In 3D computer vision, 3D object detection is crucial for numerous real-world applications, such as autonomous driving and home robots. Its purpose is to detect specific target objects or regions of interest in a 3D scene and determine the object's location and category. Object detection has become fundamental to many complex and high-level vision tasks, such as scene understanding and object tracking. However, due to the diverse appearances, shapes, and poses of various target objects, as well as interference from factors such as lighting and shielding during imaging, object detection remains a challenging problem.

[0003] Mainstream methods for 3D object detection include global feature-based methods and local feature-based methods. Global feature-based viewpoint feature histogram (VFH) object detection methods are robust to scene surface noise and lost depth information, enabling them to quickly detect objects. However, their poor adaptability to occluded environments leads to low detection accuracy. Local feature-based object detection methods first extract key points from the scene or template and calculate feature descriptors for each key point, then perform object detection or pose estimation based on these descriptors. These methods have poor adaptability to complex scene backgrounds and are computationally time-consuming. Summary of the Invention

[0004] The purpose of this invention is to provide a 3D target detection method for RGB-D scenes guided by object region information. Addressing the shortcomings of 3D target detection in RGB-D scenes, such as poor adaptability to complex backgrounds and difficulty in effectively utilizing object region information and scene point cloud features, this invention proposes a 3D target detection method based on object region information guidance. This method integrates global and local point cloud features while eliminating background interference, enabling efficient and accurate detection of 3D target objects in complex scenes.

[0005] Based on the input scene RGB-D data, the 2D region of the target object to be detected is first extracted from the color image and coarsely classified. Then, the 2D bounding box of the object region is raised to a 3D oblique cone region and transformed into point cloud data. Next, feature extraction is performed on the oblique cone point cloud using object region information, and global and local features of the point cloud are effectively fused using feature transformation and max pooling. Then, the fused features are used to predict the probability score of the correlation between each sampling point and the foreground and background. Based on this probability score, the scene foreground points and background points are segmented, and the scene background points are removed to form a masking point cloud. Finally, object center point voting is performed in the masking point cloud, and suggestions and 3D target prediction are made with the help of object region information.

[0006] The present invention specifically includes the following steps:

[0007] 1) Based on scene color Figure 2 A 3D cone point cloud is generated from bounding boxes, and RGB-D data is converted into point cloud data. Using the scene's RGB image and its depth information as input, a 2D object detector extracts 2D object regions from the image and classifies target objects in the scene. During the 3D cone point cloud generation process, the known camera projection matrix is ​​used to elevate the 2D bounding boxes of the target objects to be predicted to the 3D cone region, so that each target object in the scene can obtain its corresponding cone region. This cone defines the 3D search space of the target object. Then, all sampling points within each cone in the scene are collected to form the entire scene's 3D cone point cloud. The cone orientation of the target object is rotated towards the center view of the scene, and the central axis of the target object's cone is orthogonal to the scene's RGB image plane to normalize each cone.

[0008] 2) Combine global and local features of the oblique cone point cloud to remove background point clouds and use them for foreground-background separation in the scene point cloud. Utilize the global features within the oblique cone region and the local features of each sampling point to segment the entire scene point cloud into foreground and background points, and remove the background points. Finally, combine all foreground point clouds to form a masking point cloud.

[0009] 3) Perform 3D object detection in the scene cloud of the oblique cone foreground and refine the suggestions in normalized coordinates to obtain the final object detection result. A voting process is performed on the center of each object to generate voting center points. Then, the sampling points near the center points are clustered. Simultaneously, high-quality object suggestions are generated using object information obtained within the oblique cone region, and 3D bounding boxes are generated through 3D non-maximum suppression, thus completing the 3D object detection.

[0010] The RGB-D scene 3D target detection method guided by object region information provided by this invention can efficiently and accurately detect 3D target objects from complex scenes, while effectively eliminating background interference during the detection process. Attached Figure Description

[0011] Figure 1 This invention presents an RGB-D scene 3D target detection framework guided by object region information.

[0012] Figure 2 This is an example diagram illustrating the 3D target detection effect of the present invention in a meeting room setting;

[0013] Figure 3 This is an example diagram illustrating the 3D target detection effect of the present invention in a hotel room scenario. Detailed Implementation

[0014] The technical method and target detection effect of the present invention will be further described and explained below with reference to the accompanying drawings.

[0015] like Figure 1 As shown, a method for 3D object detection in an RGB-D scene guided by object region information includes the following steps:

[0016] 1) Generation of oblique cone regions: based on scene color Figure 2 The bounding box generates a 3D oblique cone point cloud, and the RGB-D data is converted into point cloud data.

[0017] In the process of generating a 3D oblique cone point cloud, the 2D bounding box of the target object to be predicted is raised to the 3D oblique cone region using the known camera projection matrix, so that each target object in the scene can obtain its corresponding oblique cone region. The oblique cone defines the 3D search space of the target object. Then, all sampling points in each oblique cone in the scene are collected to form the oblique cone point cloud of the entire scene.

[0018] However, since the orientation and size of the oblique cones of each target object are not the same, the coordinate range of the point cloud position of each object will change significantly. Therefore, the orientation of the oblique cones of the target objects is rotated toward the center view of the scene, and the central axis of the oblique cones of the target objects is made orthogonal to the RGB image plane of the scene to normalize each oblique cone. This normalization operation will help improve the rotation invariance of the point cloud extraction of the oblique cones of the target objects in the oblique cone proposal generation.

[0019] 2) Foreground / Background Separation: The background point cloud is removed by combining global and local features of the oblique cone point cloud, and then used for foreground / background separation in the scene point cloud. Using the generated 3D oblique cone point cloud as input, a weighted multilayer perceptron is used to extract local features of the scene sampling points, which are then combined with the global features obtained through max pooling. The point cloud features are then dimensionality reduced, and the probability that each sampling point belongs to the target object of interest is predicted.

[0020] Specifically, the segmentation network takes scene oblique cone point cloud data as input, utilizes input transformation and feature transformation, passes through a weight-sharing multilayer perceptron, and aggregates global features of sampling points using max pooling operation.

[0021] However, relying solely on the global features of the oblique cone point cloud is insufficient for foreground-background segmentation of each sampling point. Therefore, after obtaining the global features of the oblique cone point cloud, it is necessary to further connect the global features with the local features of each sampling point and feed them back to the features of each sampling point. Then, new sampling point features are extracted based on the combined point features, so that each sampling point feature can simultaneously possess both local and global information, which is equivalent to a single sampling point performing a retrieval in the global features. Finally, each sampling point is scored and output, thereby achieving the segmentation of the scene's foreground and background point clouds. After segmentation, the scene's oblique cone point cloud can extract sampling points classified as objects of interest to form the foreground point cloud, while scene background points are removed to form a masking point cloud for 3D object prediction.

[0022] 3) 3D object detection: 3D object detection is performed in the scene cloud with the foreground of the oblique cone, and the suggestions are refined in normalized coordinates to obtain the final object detection results.

[0023] Specifically, for the input scene foreground cloud of a sloping cone, features of each sampling point after fusion by a segmentation network are extracted, and votes are generated based on these features. The purpose of these votes is to predict the object center. Therefore, voting clustering occurs near the object center. Then, based on the target object information of the previous object region, aggregation is performed to generate 3D proposals for detecting object bounding boxes. However, since multiple bounding boxes may be detected for the same target, the method uses 3D non-maximum suppression to select the target detection box with the highest score as the final target bounding box, thus completing 3D object detection. During the detection process, the search space for target objects is effectively reduced, while background noise interference is effectively eliminated, thereby enabling efficient and accurate detection of different target objects in complex scenes.

[0024] Appendix Figure 2 This is an example image showing the 3D object detection effect in a meeting room setting. Figure 2 a and appendix Figure 2 b represents the RGB image and depth image of the input scene, respectively. Figure 2 c and appendix Figure 2 d are example images showing the effect of 3D object detection in the scene from different perspectives; (attached) Figure 3 Here are example images of 3D object detection results for a hotel room setting, with attached. Figure 3 a and appendix Figure 3 b represents the RGB image and depth image of the input scene, respectively. Figure 3 c and appendix Figure 3 d are example images of the 3D object detection effect in the scene from different perspectives.

[0025] From the appendix Figure 2 and attached Figure 3 As can be seen, this patented method achieves robust 3D object detection results in scenarios such as meeting rooms or hotel rooms with defects including cluttered object distribution, object occlusion, and incomplete object display. For example, see attached... Figure 2 In this scenario, the patented method can detect patterned sofas that are clearly truncated and closest to the viewpoint; (attached) Figure 3 In this scenario, the method of this patent can detect the leftmost bed where only a corner is visible.

Claims

1. A method for 3D object detection in RGB-D scenes guided by object region information, characterized in that: Specifically, the steps include the following: 1) Generate a 3D oblique cone point cloud based on the 2D bounding box of the scene color image, and convert the RGB-D data into point cloud data; use the scene RGB image and its depth information as input, and then use a 2D object detector to extract the 2D object region in the image and classify the target objects in the scene; In the process of generating a 3D oblique cone point cloud, the 2D bounding box of the target object to be predicted is raised to the 3D oblique cone region using the known camera projection matrix, so that each target object in the scene can obtain its corresponding oblique cone region. This oblique cone defines the 3D search space of the target object. Then, all sampling points in each oblique cone in the scene are collected to form the 3D oblique cone point cloud of the entire scene. The oblique cone orientation of the target object is rotated toward the center view of the scene, and the central axis of the oblique cone of the target object is orthogonal to the RGB image plane of the scene to normalize each oblique cone. 2) Combine global and local features of the oblique cone point cloud to remove the background point cloud and use it for foreground-background separation in the scene point cloud; use the global features in the oblique cone region and the local features of each sampling point to segment the entire scene point cloud into foreground points and background points and remove the scene background points, and finally combine all foreground point clouds to form a shielding point cloud. 3) Perform 3D object detection in the scene cloud foreground of the oblique cone, and refine the suggestions in normalized coordinates to obtain the final object detection results; Each target object center is voted on to generate a voting center point. Then, the sampling points near the center point are clustered. At the same time, the target object information obtained in the oblique cone region is used to generate high-quality target suggestions. Finally, 3D target bounding boxes are generated through 3D non-maximum suppression, thus completing 3D target detection.

2. The RGB-D scene 3D target detection method guided by object region information as described in claim 1, characterized in that: Step 2) specifically involves: using the scene's 3D oblique cone point cloud data as input, employing input transformation and feature transformation, and then aggregating the global features of the sampling points through a weight-sharing multilayer perceptron and max-pooling operation. Since it is difficult to segment the foreground and background of each sampling point using only the global features of the oblique cone point cloud, after obtaining the global features, it is necessary to further connect the global features with the local features of each sampling point and feed them back to the features of each sampling point. Then, new sampling point features are extracted based on the combined point features, so that each sampling point feature can simultaneously possess both local and global information, which is equivalent to a single sampling point performing a retrieval in the global features. Finally, each sampling point is scored and output, thereby achieving the segmentation of the scene's foreground and background point clouds. After segmentation, the scene's oblique cone point cloud can extract sampling points classified as objects of interest to form the foreground point cloud, and remove scene background points to form a masking point cloud for 3D target prediction.

Images