Semantic boundary guided robot working scene point cloud segmentation method

Abstract

The application discloses a kind of semantic boundary guided robot operation scene point cloud segmentation methods. Firstly, the three-dimensional point cloud of robot operation scene is obtained;Then, the semantic boundary point in the scene three-dimensional point cloud is predicted using point cloud learning network;The scene three-dimensional point cloud is input into point cloud Transformer model for feature extraction, and the semantic boundary point is used as the extended key point of self-attention mechanism in Transformer model, wherein for the point feature of each boundary region, a contrast learning loss function is added to reduce the semantic ambiguity of boundary point feature;Finally, based on the point feature of each point, the semantic class of each point is predicted to obtain the point cloud semantic segmentation result. The application enhances the learned point feature by introducing semantic boundary information in the point cloud feature extraction process, and improves the misclassification problem caused by boundary ambiguous features, effectively improves the point cloud segmentation quality of robot operation scene, and has good engineering practical value.

Description

Technical Field

[0001] This invention relates to a point cloud segmentation method in the fields of computer vision and robotics, and particularly to a point cloud segmentation method for robot operation scenes guided by semantic boundaries. Background Technology

[0002] With the development of various depth sensors (such as 3D scanners, depth cameras, and LiDAR), environmental semantic perception from 3D point cloud data has shown valuable application potential in many practical tasks, such as robot operation, autonomous driving, augmented reality, and digital twins. Especially in robotic systems, accurate semantic segmentation of point clouds in the work environment, as a fundamental 3D visual understanding technology, can help robots autonomously perceive their surrounding work environment, thereby improving the intelligence and flexibility of the robotic system.

[0003] Traditional methods rely on manually defined rules to classify specific point regions, which largely depends on human intervention and prior knowledge. With the rapid development of deep learning technology, data-driven point cloud segmentation methods have gradually achieved better results and have attracted attention from industry. These methods extract learned features from 3D point clouds through various point cloud learning networks and predict the semantic category of each point based on these features.

[0004] However, unlike two-dimensional images, three-dimensional point clouds are disordered and irregular, making it difficult for existing learning networks to capture effective long-range dependencies in point cloud data, leading to decreased segmentation performance. On the other hand, semantic boundaries in three-dimensional point clouds have a significant impact on segmentation quality. Semantic boundaries are located in transitional regions between points of different semantic categories, and the learned features in these transitional regions are often ambiguous, potentially leading to incorrect semantic classification and thus reducing overall segmentation quality. Summary of the Invention

[0005] To address the problems existing in the background technology, this invention proposes a semantic boundary-guided point cloud segmentation method for robot operation scenes. This invention performs semantic segmentation of robot operation scenes from 3D point clouds based on deep learning technology. By introducing semantic boundary information into the point cloud learning network to enhance the learned point features, and by using contrastive learning to reduce misclassification of boundary points, the overall point cloud segmentation quality is improved.

[0006] The specific technical solution created by this invention is as follows:

[0007] S1: Capture the robot's work scene from a single perspective using an RGB-D camera to obtain a scene depth map, and then convert the scene depth map into a 3D point cloud of the scene;

[0008] S2: Use a point cloud learning network to predict and obtain the semantic boundary point set in the 3D point cloud of the scene;

[0009] S3: Combine the semantic boundary point set obtained from the prediction, input the scene 3D point cloud into the point cloud Transformer model for feature extraction, and obtain the point feature set;

[0010] S4: Based on the obtained point feature set, predict the semantic category of each point in the 3D point cloud of the scene, and use the prediction result as the point cloud segmentation result.

[0011] In S2, the semantic boundary point set in the scene's 3D point cloud is a set composed of the intersection points of regions with different semantic categories.

[0012] In S2, during the training phase of the point cloud learning network, the first loss function is used. Supervised training of a point cloud learning network, first loss function The formula is as follows:

[0013]

[0014] Among them, b i This represents the true boundary label of the i-th point in the 3D point cloud of the scene. α1 represents the predicted boundary label of the i-th point in the scene's 3D point cloud, N represents the total number of points in the scene's 3D point cloud, and α2 represents the first and second balancing hyperparameters.

[0015] In step S3, the set of semantic boundary points obtained from the prediction is used as the extended key points of the self-attention mechanism in the point cloud Transformer model.

[0016] Specifically, S3 is:

[0017] The point cloud Transformer model uniformly divides the input 3D point cloud of the scene into several cubic grids. Each cubic grid contains several points. In each cubic grid, a self-attention mechanism is used in parallel to extract the point features corresponding to the points it contains. The key point set in the self-attention mechanism includes the set of semantic boundary points obtained by prediction.

[0018] In each cube grid, the query point set consists of several points contained in the current cube grid, and the key point set consists of several points contained in the current cube grid and the predicted semantic boundary point set. The point feature f corresponding to each query point in the query point set is calculated using the following formula:

[0019]

[0020] Where q represents the query volume corresponding to the query point, and kj and v j Let represent the key and value of the j-th point in the key set, respectively; σ(·) represents the weight calculation function; and M represents the total number of points in the key set.

[0021] In S3, during the training phase of the point cloud Transformer model, the loss function for the point features corresponding to the semantic boundary points in the 3D point cloud of the scene also includes a contrastive learning loss function. The formula is as follows:

[0022]

[0023] Where, N b This represents the number of points in the semantic boundary point set B. This represents a point in the semantic boundary point set B. Point Neighborhood One point in the middle, Point The set of positive examples P positive One point in the middle, point The set of positive examples P positive For point Neighborhood Midpoint A set of points with the same semantic label. Representing points respectively point point The corresponding point features are represented by d(·), which represents the distance function in the feature space, and τ represents the temperature parameter in the contrastive learning.

[0024] The beneficial effects of this invention are as follows:

[0025] (1) This invention realizes semantic segmentation of point cloud for robot operation scenarios. By introducing semantic boundary information in the point cloud feature extraction process, the network can effectively perceive the semantic transition region in the scene point cloud and pay more attention to it to enhance the learned point features, thus solving the problem of poor segmentation accuracy of existing methods.

[0026] (2) By using contrastive learning technology on the semantic boundary regions in the scene point cloud, the present invention effectively reduces the ambiguity of the boundary point features, thereby avoiding misclassification of boundary points and effectively improving the overall segmentation quality of the scene point cloud.

[0027] (3) The point cloud segmentation method provided by the present invention can be directly deployed to various downstream robot application tasks, effectively improving the flexibility and intelligence of the robot system.

[0028] In summary, the segmentation method provided by this invention enhances the learned point features by introducing boundary information and improves the misclassification problem caused by blurred boundary features, effectively improving the segmentation quality of robot operation scenarios and having good engineering practical value. Attached Figure Description

[0029] Figure 1 This is a flowchart of point cloud segmentation for robot operation scenarios guided by semantic boundaries in this invention.

[0030] Figure 2 This is a schematic diagram of the query point set and key point set corresponding to a single grid in an embodiment of the present invention.

[0031] Figure 3 This is a schematic diagram of a robot operation scenario in an embodiment of the present invention.

[0032] Figure 4 It refers to the target object category contained in the self-built point cloud dataset in this embodiment of the invention.

[0033] Figure 5 This is a visualization result of point cloud semantic segmentation in a robot operation scene according to an embodiment of the present invention. Detailed Implementation

[0034] The invention will be further illustrated below using a self-built real point cloud dataset for robot operation scenarios:

[0035] In this embodiment, the semantic boundary-guided point cloud segmentation method for robot operation scenarios provided by the present invention is trained and tested on a self-built real point cloud dataset. The self-built real point cloud dataset includes approximately 10,000 point cloud data samples, which are located in an actual robot operation scenario (see...). Figure 3 The scene was captured by data collection. Multiple target objects were randomly placed on the work platform. These target objects are common industrial parts (see...). Figure 4 The data is categorized into 8 classes. The goal of this example is to perform semantic segmentation on target objects in a robot's operational scene point cloud, providing effective localization information for subsequent robot operations. The basic process is as follows: Figure 1 As shown.

[0036] Based on this dataset, the specific steps of the point cloud segmentation method provided by this invention are as follows:

[0037] S1: Capture the robot's work scene from a single perspective using an RGB-D camera to obtain a scene depth map, and then convert the scene depth map into a 3D point cloud of the scene;

[0038] In this embodiment, an Intel Realsense-D455 camera was used to capture the scene. The depth pixel resolution was 640×480 and the depth scale was 1.0. Each point in the 3D point cloud converted from the depth map contains a spatial coordinate and a corresponding RGB value.

[0039] S2: Use a point cloud learning network to predict and obtain the semantic boundary point set in the scene's 3D point cloud; where the semantic boundary point set in the scene's 3D point cloud is a set of intersection points of different semantic category regions. In specific implementation, the boundary label b is used to represent the boundary of each point. If a point in the scene's 3D point cloud is a semantic boundary point, then b = 1, otherwise b = 0.

[0040] During the training phase of the point cloud learning network, the first loss function is used. Supervised training of a point cloud learning network, first loss function The formula is as follows:

[0041]

[0042] Among them, b i This represents the true boundary label of the i-th point in the 3D point cloud of the scene. Let αi represent the predicted boundary label of the i-th point in the scene's 3D point cloud, N represent the total number of points in the scene's 3D point cloud, and α1 and α2 represent the first and second balancing hyperparameters, respectively. After supervised training, the point cloud learning network generates a boundary label for each point in the scene's point cloud to indicate whether the point is a boundary point.

[0043] In this embodiment, the point cloud learning network uses the PointNet++ network to reduce computational overhead. α1 and α2 are set to 1 and 5, respectively.

[0044] S3: Combining the semantic boundary point set obtained from prediction, specifically, using the explicitly predicted semantic boundary point set as the extension key points of the self-attention mechanism in the point cloud Transformer model, thereby introducing boundary information to enhance the point features extracted by the model. The 3D point cloud of the scene is input into the point cloud Transformer model for feature extraction to obtain the point feature set;

[0045] S3 specifically refers to:

[0046] The point cloud Transformer model uniformly divides the input 3D point cloud of the scene into several cubic meshes. In this embodiment, the size of each mesh is set to 0.1m × 0.1m × 0.1m. Each cubic mesh contains several points, and the self-attention mechanism is used in parallel within each cubic mesh to extract the point features corresponding to the points it contains. The key point set in the self-attention mechanism includes the set of semantic boundary points obtained from the prediction.

[0047] Within each cube grid, the query point set consists of several points contained in the current cube grid, and the key point set consists of several points contained in the current cube grid and the predicted semantic boundary point set, such as... Figure 2 As shown, the query point set is Figure 2 The key point set of the dashed outline points in (a) is... Figure 2 The dashed outline points in (b) are used, where the points outside the thick solid grid are the predicted semantic boundary points. The point feature f corresponding to each point in the cube grid, i.e., each query point in the query point set, is calculated using the following formula:

[0048]

[0049] Where q represents the number of queries corresponding to this query point, and k j and v j Let represent the key and value of the j-th point in the key set, respectively. σ(ε) represents the weight calculation function, which is the soft-max function used in this embodiment. M represents the total number of points in the key set.

[0050] During the training phase of the point cloud Transformer model, for the point features corresponding to semantic boundary points in the 3D point cloud of the scene (i.e., points in the set of semantic boundary points predicted in S2), the loss function also includes a contrastive learning loss function. The contrastive learning loss function and the original loss function of the point cloud Transformer model are weighted and summed to obtain the final loss function for the point features corresponding to the semantic boundary points. Adding the contrastive learning loss function to the point features of the boundary region reduces the semantic ambiguity of the boundary point features. The formula is as follows:

[0051]

[0052] Where, N b This represents the number of points in the semantic boundary point set B. This represents a point in the semantic boundary point set B. Point Neighborhood One point in the middle, Point The set of positive examples P positive One point in the middle, point The set of positive examples P positive For point Neighborhood Midpoint A set of points with the same semantic label. Representing points respectively point point The corresponding point features are represented by d(·), which represents the distance function in the feature space, and τ represents the temperature parameter in the contrastive learning.

[0053] In this embodiment, the neighborhood Indicated by point Let k be the k nearest neighbors of the center, where k is 16.

[0054] S4: Based on the obtained point feature set, predict the semantic category of each point in the 3D point cloud of the scene, and use the prediction result as the point cloud segmentation result.

[0055] In this embodiment, 70% of the data from a self-built real point cloud dataset was used, and the model was trained based on the NVIDIA A100 graphics processor and the PyTorch framework. After the model training was completed, 30% of the data from the self-built real point cloud dataset was used for testing.

[0056] The quantitative test results are shown in Table 1. The mean Intersection over Union (mIoU) and mean class accuracy (mAcc) were used as evaluation metrics for the segmentation results. The method provided by this invention was also compared with two existing methods: Method 1 uses the classic PointNet network for segmentation, and Method 2 uses the KPConv network. The test results show that for target objects of a single semantic category in robot operation scenarios, the class-wise IoU score of the method of this invention reaches over 70%, and the overall mIoU and mAcc scores are significantly better than the two existing methods, verifying the effectiveness and superiority of this invention in point cloud segmentation in practical robot operation scenarios.

[0057] Table 1 shows the semantic segmentation results (%) of the method of the present invention in robot operation scenarios.

[0058]

[0059] In addition, some qualitative segmentation results, such as Figure 5 As shown, Figure 5 (a) is the real point cloud of the scene. Figure 5 (b) is the semantic segmentation point cloud. Different gray levels are used to display target object points of different categories. It can be seen that the method of this invention can successfully segment target objects from the point cloud of the robot's operational scene, further verifying the effectiveness of the method.

[0060] This invention is not limited to the above-described embodiments, but is only used to help understand the method and core idea of ​​this invention. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from the principles of this invention, and these improvements and modifications also fall within the protection scope of the claims of this application. Contents not described in detail in this specification are prior art known to those skilled in the art.

Claims

1. A semantic boundary-guided point cloud segmentation method for robot operation scenes, characterized in that, Includes the following steps: S1: Capture the robot's work scene from a single perspective using an RGB-D camera to obtain a scene depth map, and then convert the scene depth map into a 3D point cloud of the scene; S2: Use a point cloud learning network to predict and obtain the semantic boundary point set in the 3D point cloud of the scene; During the training phase of the point cloud learning network, the first loss function is used. Supervised training of a point cloud learning network, first loss function The formula is as follows: in, This represents the true boundary label of the i-th point in the 3D point cloud of the scene. This represents the predicted boundary label of the i-th point in the scene's 3D point cloud, where N represents the total number of points in the scene's 3D point cloud. and Indicates the first and second equilibrium hyperparameters; S3: Combine the semantic boundary point set obtained from the prediction, input the scene 3D point cloud into the point cloud Transformer model for feature extraction, and obtain the point feature set; During the training phase of the point cloud Transformer model, for the point features corresponding to the semantic boundary points in the 3D point cloud of the scene, the loss function also includes a contrastive learning loss function. The formula is as follows: in, This represents the number of points in the semantic boundary point set B. This represents a point in the semantic boundary point set B. Point Neighborhood One point in the middle, Point positive example set One point in the middle, point positive example set For point Neighborhood Midpoint A set of points with the same semantic label. Representing points respectively ,point ,point Corresponding point features, The distance function representing the feature space. This indicates the temperature parameter in the comparative learning process; S4: Based on the obtained point feature set, predict the semantic category of each point in the 3D point cloud of the scene, and use the prediction result as the point cloud segmentation result.

2. The semantic boundary-guided point cloud segmentation method for robot operation scenes according to claim 1, characterized in that, In S2, the semantic boundary point set in the scene's 3D point cloud is a set composed of the intersection points of regions with different semantic categories.

3. The semantic boundary-guided point cloud segmentation method for robot operation scenes according to claim 1, characterized in that, In step S3, the set of semantic boundary points obtained from the prediction is used as the extended key points of the self-attention mechanism in the point cloud Transformer model.

4. The semantic boundary-guided point cloud segmentation method for robot operation scenes according to claim 1, characterized in that, Specifically, S3 is: The point cloud Transformer model uniformly divides the input 3D point cloud of the scene into several cubic grids. Each cubic grid contains several points. In each cubic grid, a self-attention mechanism is used in parallel to extract the point features corresponding to the points it contains. The key point set in the self-attention mechanism includes the set of semantic boundary points obtained by prediction.

5. The semantic boundary-guided point cloud segmentation method for robot operation scenes according to claim 4, characterized in that, In each cube grid, the query point set consists of several points contained in the current cube grid, and the key point set consists of several points contained in the current cube grid and the predicted semantic boundary point set. The point features corresponding to each query point in the query point set... All are obtained through the following formula: Where q represents the query volume corresponding to this query point. and Let these represent the key and value of the j-th point in the key set, respectively. This represents the weight calculation function, and M represents the total number of points in the key point set.

Images