A component detection method and system for performing cross-category object perception and manipulation

Abstract

The present invention relates to the technical field of component detection, and provides a component detection method and system for performing cross-category object perception and manipulation, including: S1: extracting backbone features from object point clouds point by point; S2: performing component semantic classification, and predicting the vector from each point to the center of the component point cloud to which it belongs; S3: clustering the point cloud of the component center; S4: normalizing the clustered point cloud component instances to a sphere of unit radius, and performing feature extraction; S5: predicting confidence using the global features of the point cloud component instances, and using the segmentation all basic model to correct the point cloud component instances with confidence below a preset threshold; S6: re-inferring the confidence of the point cloud component instances, and using non-maximum suppression to delete the point cloud component instances with large overlap; S7: using the point cloud component instances point by point features to predict the initial NOCS coordinates, and then re-estimating the posture of the re-NOCS coordinates. The quality of cross-object category component detection and the accuracy of posture estimation are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of component detection, and in particular to a component detection method and system for performing cross-category object perception and manipulation. Background Art

[0002] Generalizable perception and manipulation are the core of building intelligent robots. Previous research on generalizable perception and manipulation has mainly focused on the setting of category-level objects, including pose estimation of category-level objects, pose estimation of category-level joint objects, and manipulation of category-level joint objects. Researchers have collected large datasets for these tasks, and improved algorithms have also been explored.

[0003] However, object-level classification imposes severe limitations on the generalization ability of collected data. A rigid object category is limited to smaller intra-category geometric variations, while an articulated object category restricts its objects to have the same type of kinematic chains, including the same number and type of joints. Therefore, subsequent research has promoted the development of cross-category generalizable perception and manipulation. One type of research uses test-time adaptation to update the posterior knowledge of objects through a small amount of online interactions, but the additional requirement of online interactions limits its practical application. Another type of work that is more related to our research, represented by GAPartNet and PartManip, proposes to achieve cross-category generalization by decomposing perception and manipulation from the object level to the part level. These operable parts have simpler shape geometries and can be perceived in unseen object categories, thereby achieving better cross-category generalization.

[0004] Although GAPartNet has demonstrated cross-category generalizable perception and manipulation, we emphasize that perception accuracy is critical to manipulation quality. GAPartNet uses the 3D SparseUNet backbone network for instance segmentation of parts, but its generalization ability for unseen object categories is not reliable. GAPartNet uses the classic NOCS pose estimator. Currently, there are many improved algorithms based on NOCS, which also provide room for improvement for this pose estimation task. The NOCS pose estimation algorithm first defines a standard space for a class of objects. The objects in the standard space are in the initial pose, and the point cloud coordinates of the objects in the standard control are called NOCS coordinates. By predicting the NOCS coordinates point by point for objects in the camera space, a point-by-point correspondence between objects in the camera space and objects in the standard space can be established. Then, the Umeyama algorithm is used to solve the rotation, translation, and scaling implied in the point-by-point correspondence, and the pose of the category-level objects can be solved.

[0005] In summary, the existing technology has the following defects: (1) Because the existing technology is applied to cross-object category (on untrained object categories) component detection, this application places high requirements on the generalization of the perception model. However, the background technology uses 3D component detection based on the SparseUNet backbone network, and does not optimize the generalization at the model level. (2) The accuracy of the existing classic NOCS pose estimation method needs to be further improved, and there is room for optimization. Summary of the invention

[0006] In response to the above problems, the object of the present invention is to provide a component detection method and system for performing cross-category object perception and manipulation, so as to improve the quality of component detection across object categories and the accuracy of posture estimation, thereby improving the success rate of component operation across object categories.

[0007] The above-mentioned object of the present invention is achieved through the following technical solutions:

[0008] A method for performing part detection for cross-category object perception and manipulation, comprising:

[0009] S1: input an object point cloud containing a plurality of component point clouds, and each component corresponding to the component point cloud is provided with a corresponding semantic label for indicating the component category, and extract backbone features from the object point cloud point by point;

[0010] S2: Using the extracted backbone features to perform semantic classification of components point by point, and predicting the vector from each point to the center of the component point cloud to obtain a point cloud of the component center predicted point by point;

[0011] S3: clustering the predicted point cloud of the component center to obtain a predicted point cloud component instance, and evaluating the clustering quality of the point cloud component instance;

[0012] S4: normalizing each clustered point cloud component instance into a sphere of unit radius, and performing feature extraction on the point cloud component instance;

[0013] S5: predicting the confidence of the point cloud component instance using the global feature of the point cloud component instance, and modifying the point cloud component instance whose confidence is below a preset threshold using the segmentation all basic models;

[0014] S6: re-inferring the confidence of the point cloud component instance according to the modified segmentation of the point cloud component instance, and deleting the point cloud component instance with a large overlap by using non-maximum suppression;

[0015] S7: using the point-by-point features of the point cloud component instance to predict the initial NOCS coordinates, performing an initial pose estimation of the point cloud component instance on the initial NOCS coordinates, obtaining a secondary NOCS coordinate based on the result of the initial pose estimation, and then performing a secondary pose estimation on the secondary NOCS coordinates.

[0016] Furthermore, in step S1, an object point cloud containing a plurality of component point clouds is input, and each component corresponding to the component point cloud is provided with a corresponding semantic label for indicating the component category, and backbone features are extracted point by point from the object point cloud, specifically:

[0017] The object point cloud is obtained by back-projecting the pre-collected RGB-D image, wherein the object point cloud includes a plurality of the component point clouds;

[0018] The object point cloud is segmented by manually annotating training data, and the object point cloud is divided into a plurality of component point clouds. For the segmented component point clouds, a semantic label is assigned to each of the component point clouds according to the characteristics of the component point clouds and prior knowledge;

[0019] The SparseUNet network is used as the backbone network to extract the backbone features point by point. The encoder part of the SparseUNet network gradually downsamples and extracts features from the input point cloud to obtain feature maps at different levels. The decoder part of the SparseUNet network gradually upsamples and fuses these feature maps, and finally outputs the vector of the backbone features of each point.

[0020] Further, in step S2, the extracted backbone features are used to perform semantic classification of components point by point, and the vector from each point to the center of the component point cloud is predicted point by point to obtain the point cloud of the component center predicted point by point, specifically:

[0021] Pre-training a semantic segmentation model, and using a focal loss function in the semantic segmentation model to solve the problem of class imbalance, giving higher weights to difficult-to-classify samples, using a dice loss function to measure the overlap between the predicted results and the true labels, and processing the irregularly shaped point cloud of the component;

[0022] The input backbone features are predicted using the semantic segmentation model. When the semantic segmentation model predicts that the backbone features belong to a certain component category, the current backbone features are marked as the corresponding component category. If the semantic segmentation model predicts that the backbone features do not belong to any component category, the current backbone features are marked as the background category.

[0023] Pre-training a component center prediction model, and using a vector direction loss in the component center prediction model to calculate a direction difference between a predicted vector and a true vector, using a vector length loss to ensure that the length of the predicted vector is close to the length of the true vector, and performing a weighted combination of the vector direction loss and the vector length loss to obtain a final loss function;

[0024] The component center prediction model is used to predict the prediction vector of the point to the center of the component point cloud to which the backbone feature is input point by point, and the point cloud of the component center predicted point by point is obtained by moving each point along the prediction vector.

[0025] Furthermore, in step S3, the predicted point cloud of the component center is clustered to obtain a predicted point cloud component instance, and the clustering quality of the point cloud component instance is evaluated, specifically:

[0026] The spherical radius r of the cluster and the maximum number of cluster points are preset, and an unprocessed point is randomly selected from the point cloud at the center of the component as the starting point, and the nearest point with the same semantics within the spherical radius r from the point is connected to the starting point, and then starting from the point newly included in the connected block, the nearest point with the same semantics within the spherical radius r from the new starting point is recursively included in the connected block until there is no point that meets the requirements within the spherical radius r, or until the number of points in the cluster is greater than the pre-specified threshold of the maximum number of cluster points, so as to obtain the clustered point cloud component instance;

[0027] Each clustered point cloud component instance is compared with the real point cloud component annotation to obtain the intersection over union (IoU) ratio between the point cloud component instance and the real point cloud component, which is used to indicate the quality of the current point cloud component instance clustering.

[0028] Furthermore, in step S4, each clustered point cloud component instance is normalized to a sphere of unit radius, and feature extraction is performed on the point cloud component instance, specifically:

[0029] Normalizing each clustered point cloud component instance to a sphere with a unit radius;

[0030] A small SparseUNet backbone network is constructed, and the normalized point cloud component instance is input into the small SparseUNet backbone network for feature extraction, and finally a feature vector of each point is output.

[0031] Furthermore, in step S5, the confidence of the point cloud component instance is predicted using the global feature of the point cloud component instance, and the point cloud component instance with the confidence below the preset threshold is corrected using the segmentation basic model, specifically:

[0032] Using a pooling operation in deep learning to map features in the point cloud component instance to a vector of the global feature, using the predicted intersection over union (IoU) of the global feature, and interpreting the IoU as confidence, while using an L2 loss function to train confidence prediction;

[0033] The point cloud component instances whose confidence level is below the preset threshold are selected, and modified using the segmentation basic model. The modification process is specifically as follows:

[0034] Project the point cloud into an RGB-D image, map the depth channel into the color space, and then superimpose the RGB image and the depth image to obtain an image with depth information;

[0035] Randomly select a number of points in the point cloud component instance as foreground click prompts of the segmentation all basic models, and then project these foreground click prompts into the image space;

[0036] Input the foreground click prompt and the image with depth information into the segmentation basic model, the model infers the 2D component segmentation, and finally maps the 2D component segmentation result back to the 3D point cloud component instance to obtain the corrected segmentation of the point cloud component instance;

[0037] At the same time, in order to improve the effect of the segmentation basic model on the component dataset, a lightweight adapter module is used to fine-tune the segmentation basic model. The input is the picture with depth information and the foreground click prompt, and the output is the segmentation annotation on the real 2D component segmentation picture.

[0038] Further, in step S6, the confidence of the point cloud component instance is re-inferred according to the segmentation of the modified point cloud component instance, and the point cloud component instance with large overlap is deleted by using non-maximum suppression, specifically:

[0039] Re-reasoning the confidence of the point cloud component instance according to the modified segmentation of the point cloud component instance;

[0040] Non-maximum suppression is used to delete the point cloud component instances with large overlap. First, the confidence levels of all the point cloud component instances are sorted from high to low, starting from the clustering of the first point cloud component instance. If the confidence level of a subsequent point cloud component instance and the current point cloud component instance is greater than the threshold of the non-maximum suppression, the subsequent point cloud component instance is deleted.

[0041] Further, in step S7, the point-by-point features of the point cloud component instance are used to predict the initial NOCS coordinates, the initial posture of the point cloud component instance is estimated for the initial NOCS coordinates, and the secondary NOCS coordinates are obtained according to the result of the initial posture estimation, and then the secondary posture estimation is performed on the secondary NOCS coordinates, specifically:

[0042] The initial NOCS coordinates are predicted using the point-by-point features of the point cloud component instance, the initial posture estimation of the component instance is performed on the initial NOCS coordinates using the Umeyama algorithm, the object point cloud in the camera space is transformed into a coordinate system of a neighboring standard space according to the result of the initial posture estimation, and the relative transformation between the object point cloud in the camera space and the standard space depends on the error of the initial posture estimation, which is called a pseudo-standard space, point-by-point features are extracted for the point cloud component instance in the pseudo-standard space, the features are used to iteratively perform the second NOCS coordinate prediction, and the posture estimation of the NOCS coordinates is performed again using the Umeyama algorithm;

[0043] The L2 loss is used to train the prediction of the initial NOCS coordinates and the secondary NOCS coordinates at the same time.

[0044] The component detection method for performing cross-category object perception and manipulation of the present invention also includes: performing a simulated operation experiment in a Sapien simulator to verify the effect of the improved component detection for cross-category object perception and manipulation.

[0045] A component detection system for performing cross-category object perception and manipulation for performing the component detection method for performing cross-category object perception and manipulation as described above, comprising:

[0046] An object point cloud input module is used to input an object point cloud containing a plurality of component point clouds, and each component corresponding to the component point cloud is provided with a corresponding semantic label for indicating the component category, and to extract backbone features from the object point cloud point by point;

[0047] A center point cloud prediction module is used to use the extracted backbone features to perform semantic classification of components point by point, and predict the vector from each point to the center of the component point cloud to obtain a point cloud of the component center predicted point by point;

[0048] A point cloud component clustering module, used for clustering the predicted point cloud of the component center to obtain a predicted point cloud component instance, and at the same time evaluating the clustering quality of the point cloud component instance;

[0049] A point cloud component normalization module, used to normalize each clustered point cloud component instance into a sphere of unit radius, and perform feature extraction on the point cloud component instance;

[0050] A point cloud component correction module, used to predict the confidence of the point cloud component instance using the global features of the point cloud component instance, and to correct the point cloud component instance whose confidence is below a preset threshold using the segmentation basic model;

[0051] A point cloud component deletion module, used for re-inferring the confidence of the point cloud component instance according to the segmentation of the modified point cloud component instance, and deleting the point cloud component instance with large overlap by using non-maximum suppression;

[0052] An iterative attitude estimation module is used to use the point-by-point features of the point cloud component instance to predict the initial NOCS coordinates, perform the initial attitude estimation of the point cloud component instance on the initial NOCS coordinates, obtain the secondary NOCS coordinates according to the result of the initial attitude estimation, and then perform the secondary attitude estimation on the secondary NOCS coordinates.

[0053] Compared with the prior art, the present invention has at least one of the following beneficial effects:

[0054] (1) We propose a modified part instance segmentation method using the Split Everything base model to achieve more generalized cross-category part detection. We achieve better performance by fine-tuning Split Everything on our task data.

[0055] (2) A simple yet effective iterative pose optimization method is introduced to achieve more accurate part pose estimation.

[0056] (3) We conducted simulated operation experiments in the Sapien simulator to verify that the improved perception model of the present invention can bring better operation results. In addition, we also demonstrated our robot operation framework on a real-world robot. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 The overall flow chart of the component detection method for performing cross-category object perception and manipulation of the present invention;

[0058] Figure 2 The overall structural diagram of the component detection system for performing cross-category object perception and manipulation of the present invention. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0060] Those skilled in the art will appreciate that, unless otherwise stated, the singular forms "a", "an", "said" and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present invention refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0061] When building intelligent robots, it is ideal to be able to perform cross-category object perception and manipulation. A promising approach is to define the concept of generalizable and actionable parts (GAParts), such as buttons, handles, and drawers, where a class of parts appears on multiple classes of objects, and the part perception model can generalize to untrained object categories. However, accurate cross-category part perception remains challenging due to the huge variation in object shapes across categories.

[0062] To address this problem, we propose SAMIR (Segment Anything Model for Improved Robotic perception), a new framework that uses a Segment Anything base model to correct part detection and an iterative method for part pose optimization, which can be used for part detection and manipulation. First, we use the Segment Anything base model as a geometric regularizer to correct uncertain instance segmentation, and the zero-shot generalization ability of the Segment Anything base model enables cross-category generalization. Then, we introduce a lightweight adaptation module to fine-tune Segment Anything on our task data, achieving better segmentation performance while avoiding catastrophic forgetting. Next, we propose a simple but effective iterative method to obtain more accurate part pose estimates.

[0063] Our experiments on the GAPartNet dataset (Generalizable and Actionable Parts Network dataset) show that SAMIR consistently outperforms baseline methods on instance segmentation and pose estimation tasks. Our robot manipulation experiments in the Sapien simulator show that SAMIR improves the manipulation success rate. We also deploy the method on a real robot for real-world execution.

[0064] The following is described by specific examples:

[0065] First embodiment

[0066] like Figure 1 As shown, this embodiment provides a component detection method for performing cross-category object perception and manipulation, including:

[0067] S1: Input an object point cloud containing a plurality of component point clouds, and each component corresponding to the component point cloud is provided with a corresponding semantic label for indicating the component category, and extract backbone features from the object point cloud point by point.

[0068] The object point cloud is obtained by back-projecting the pre-collected RGB-D image, and several component point clouds are included in the object point cloud. The object point cloud is segmented by manually annotating training data, and the object point cloud is divided into several component point clouds. For the segmented component point clouds, a semantic label is assigned to each component point cloud according to the characteristics of the component point cloud and prior knowledge, specifically including the following schemes:

[0069] (1) Obtaining RGB-D images

[0070] Capture RGB-D images using a suitable depth camera or sensor device that can capture both color images (RGB) and depth information.

[0071] The collected RGB-D images are preprocessed, including noise removal, color and depth information correction, etc., to improve the image quality.

[0072] (2) Back-projection to generate object point cloud

[0073] According to the intrinsic parameter matrix and extrinsic parameter matrix of the depth camera, the depth value of each pixel in the RGB-D image is converted into point coordinates in three-dimensional space.

[0074] Combined with the color information of the color image, each generated 3D point is assigned a corresponding color value, thereby obtaining an object point cloud containing color information.

[0075] (3) Component point cloud segmentation and semantic labeling

[0076] Using manually annotated training data, the object point cloud is segmented and divided into several component point clouds. For each segmented component point cloud, a semantic label is assigned to it based on its characteristics and prior knowledge, such as "drawer", "button", "lid", etc. Semantic labeling can be performed by manual labeling, automatic labeling using machine learning algorithms, or a combination of the two.

[0077] During the entire technical implementation process, it is necessary to ensure the accuracy and efficiency of each step to obtain high-quality object point clouds and accurate component semantic labels to provide reliable input data for subsequent tasks such as component detection and robotic operation.

[0078] For the point cloud taken during actual use, the present invention can be used to identify the component point cloud contained therein and predict the semantic label of the component.

[0079] The SparseUNet network is used as the backbone network to extract the backbone features point by point. The encoder part of the SparseUNet network gradually downsamples and extracts features from the input point cloud to obtain feature maps at different levels. The decoder part of the SparseUNet network gradually upsamples and fuses these feature maps, and finally outputs the vector of the backbone features of each point, which specifically includes the following processes:

[0080] (1) Constructing the SparseUNet network structure

[0081] SparseUNet usually consists of two parts: encoder and decoder.

[0082] Encoder part: gradually downsample the point cloud and extract features at different levels. This can be achieved using a series of sparse convolutional layers and pooling layers. Sparse convolutional layers can effectively process sparse point cloud data and only perform calculations at locations with data, improving computational efficiency.

[0083] Decoder part: gradually upsample the feature maps and fuse them with the features of the corresponding level in the encoder to restore the detail information of the point cloud. Upsampling can be achieved using deconvolution layers or interpolation operations.

[0084] At different levels of the network, some additional modules, such as attention mechanism, residual connection, etc., can be added to enhance the performance of the network and its ability to extract features.

[0085] (2) Feature extraction process

[0086] Input the preprocessed object point cloud into the constructed SparseUNet network.

[0087] The encoder part of the network gradually downsamples and extracts features from the input point cloud to obtain feature maps at different levels.

[0088] The decoder part gradually upsamples and fuses these feature maps, and finally outputs the feature vector of each point. These feature vectors are point-by-point features, which can reflect the local and global feature information of each point in the object point cloud.

[0089] S2: Use the extracted backbone features to perform semantic classification of components point by point, and predict the vector from each point to the center of the component point cloud to obtain a point cloud of the component center predicted point by point.

[0090] Pre-train the semantic segmentation model, and use the focal loss function in the semantic segmentation model to solve the category imbalance problem, give higher weights to difficult-to-classify samples, use the dice loss function to measure the overlap between the prediction result and the true label, and process the irregularly shaped component point cloud; use the semantic segmentation model to predict the input backbone features, and when the semantic segmentation model predicts that the backbone features belong to a certain component category, the current backbone features are marked as the corresponding component category. If the semantic segmentation model predicts that the backbone features do not belong to any of the component categories, the current backbone features are marked as the background category. Training the semantic segmentation model specifically includes the following processes:

[0091] (1) Data preparation

[0092] Organize the backbone features of the object point cloud extracted by SparseUNet to ensure the accuracy and completeness of the features.

[0093] Identify the component point cloud data with semantic labels and divide them into training set, validation set and test set. Ensure that the data distribution of each set is reasonable, covering different types of components and various possible scenarios.

[0094] (2) Feature processing

[0095] The backbone features are further analyzed and processed, such as feature scaling and normalization, to make different features comparable in numerical range, thereby improving the training efficiency and stability of the model.

[0096] (3) Model selection and construction

[0097] The model structure is constructed using a fully connected neural network deep learning model, which usually includes an input layer, a hidden layer, and an output layer. The input layer receives the processed backbone features, the hidden layer is responsible for extracting higher-level feature representations, and the output layer gives the semantic classification results of each point.

[0098] (4) Training process

[0099] The point cloud data in the training set and its corresponding backbone features and semantic labels are input into the semantic prediction model.

[0100] Define the loss function. Here, we use a combination of focal loss and dice loss. Focal loss solves the problem of class imbalance, giving higher weights to samples that are difficult to classify, prompting the model to pay more attention to these samples. Dice loss measures the overlap between the predicted results and the true labels, and is suitable for processing irregularly shaped part point clouds.

[0101] Select a suitable optimization algorithm, such as Adam optimizer, etc. The optimization algorithm minimizes the loss function by continuously adjusting the parameters of the model.

[0102] During the training process, set an appropriate learning rate adjustment strategy. For example, you can use the learning rate decay method to gradually reduce the learning rate as the training progresses to better converge to the optimal solution.

[0103] Regularly use the validation set to evaluate the performance of the model. By calculating indicators such as accuracy and recall on the validation set, determine whether the model is overfitting or underfitting, and adjust the model's hyperparameters and structure based on the evaluation results.

[0104] (5) Prediction and classification

[0105] After the model training is completed, the test set is used for the final evaluation of the model. The point cloud data in the test set is input into the model, and the model outputs the semantic classification prediction result of each point.

[0106] For each point, if the model predicts that it belongs to a certain part category, it is labeled as the corresponding part category; if the model predicts that the point does not belong to any part, it is classified as the "background" class.

[0107] Pre-train a component center prediction model, and use vector direction loss in the component center prediction model to calculate the direction difference between the predicted vector and the true vector, use vector length loss to ensure that the length of the predicted vector is close to the length of the true vector, and perform a weighted combination of the vector direction loss and the vector length loss to obtain a final loss function; use the component center prediction model to predict the predicted vector of the point to the center of the component point cloud to which the backbone feature is input point by point, and obtain a point cloud of the component center predicted point by point by moving each point along the prediction vector. Training the component center prediction model specifically includes:

[0108] (1) Data preparation and preprocessing

[0109] Ensure that the backbone features of the object point cloud have been extracted, and clearly identify the component point cloud to which each point belongs (determined by previous semantic classification results or other methods).

[0110] The backbone features are subjected to necessary normalization to ensure that different features are comparable in numerical range, which is helpful for subsequent calculations and training.

[0111] (2) Building a prediction model

[0112] A suitable deep learning model or machine learning algorithm (fully connected neural network) can be selected to predict the point-to-part center vector.

[0113] The input of the model is the backbone feature of each point, and the output is the vector from the point to the center of the point cloud of the component to which it belongs. This vector can be represented by three-dimensional coordinates (x, y, z).

[0114] (3) Define the loss function

[0115] Part center prediction is trained using vector direction and length loss.

[0116] Vector direction loss: The direction difference between the predicted vector and the true vector can be calculated (using the Cos function).

[0117] Vector length loss: Make sure the length of the predicted vector is close to the length of the true vector. The mean squared error (MSE) loss can be used to account for the length difference.

[0118] Comprehensive loss: The direction loss and length loss are weighted together to obtain the final loss function.

[0119] (5) Model training

[0120] The labeled point cloud data (including the backbone features of each point and its true vector to the center of the point cloud of the corresponding part) is input into the constructed prediction model.

[0121] Use optimization algorithms (such as stochastic gradient descent, Adam, etc.) to minimize the loss function and continuously adjust the model parameters so that the model can gradually and accurately predict the vector from the point to the center of the component.

[0122] During the training process, the validation set can be used regularly to evaluate the performance of the model, and the model's hyperparameters and training strategies can be adjusted based on the loss value and other indicators on the validation set (such as the average distance between the predicted vector and the true vector).

[0123] (6) Obtain the point cloud of the component center predicted point by point

[0124] When the model training is completed, the new object point cloud data is input into the model, and the model outputs the prediction vector from each point to the center of the point cloud of the component to which it belongs.

[0125] By moving each point along the prediction vector, we can get a point cloud of the predicted component center point by point. Specifically, for each point, its predicted component center is C=P+prediction vector.

[0126] S3: Clustering the predicted point cloud of the component center to obtain a predicted point cloud component instance, and evaluating the clustering quality of the point cloud component instance.

[0127] The spherical radius r of the cluster and the maximum number of cluster points are preset, and an unprocessed point is randomly selected from the point cloud at the center of the component as the starting point, and the nearest point with the same semantics within the spherical radius r from the point is connected to the starting point, and then starting from the point newly included in the connected block, the nearest point with the same semantics within the spherical radius r from the new starting point is recursively included in the connected block until there are no points that meet the requirements within the spherical radius r, or until the number of points in the cluster is greater than the pre-specified threshold of the maximum number of cluster points, so as to obtain the clustered point cloud component instance. The specific implementation process is:

[0128] (1) Data preparation

[0129] Make sure that you have point cloud data processed in the previous steps, including the location information of each point, semantic labels, and the predicted vector from the point to the center of the component. Determine appropriate initial parameters, including the maximum number of cluster points and an initial spherical radius r.

[0130] (2) Determine clustering parameters

[0131] According to the actual situation and experience, try different values ​​of the spherical radius r, combine the prior knowledge of the data and the experimental results, and choose a radius r that is small enough but can effectively cluster points with the same semantics. This process can be determined by experimenting and adjusting on the validation set.

[0132] Set the maximum number of cluster points. This value should be large enough to ensure that most of the real part point clouds are included, but not too large to cause over-clustering. A reasonable maximum number of cluster points can be determined by analyzing the size and point cloud density of different parts in the dataset.

[0133] (3) Clustering algorithm implementation

[0134] An unprocessed point is randomly selected as the starting point from the component center point cloud moved along the point-by-point predicted intermediate center vector.

[0135] Find the nearest points with the same semantics within a radius r from the starting point. Connect these points with the starting point to form a connected block.

[0136] Starting from the point newly included in the connected block, repeat the above steps and recursively include the nearest points with the same semantics within a radius r from the new starting point into the connected block.

[0137] In each iteration, check whether one of the following two conditions is met:

[0138] A: There are no points that meet the requirements within the radius r, that is, there are no more points with the same semantics and within a distance of r that can be added to the connected block.

[0139] B: The number of points in the cluster is greater than the pre-specified maximum cluster point threshold.

[0140] When one of the above conditions is met, the expansion of the current connected block is stopped and it is marked as a cluster, that is, a predicted component instance.

[0141] Repeat the above steps until all points in the point cloud are processed, that is, they are all assigned to a cluster or determined to be isolated points (not belonging to any cluster).

[0142] Each clustered point cloud component instance is compared with the real point cloud component annotation to obtain the intersection-over-union ratio (IoU) between the point cloud component instance and the real point cloud component, which is used to indicate the quality of the current point cloud component instance clustering. The specific technical solution is as follows:

[0143] (1) Data preparation

[0144] Make sure that the point cloud clustering operation has been completed and the point cloud components of each cluster have been obtained. At the same time, there must be annotated data of the real point cloud components, which can be obtained through manual annotation or other reliable methods.

[0145] (2) Calculate IoU

[0146] For each clustered point cloud component (and for each real point cloud component, since we don’t know the real component point cloud B corresponding to component point cloud A, we calculate IoU for all (A, B) pairs and select the real component point cloud with the largest IoU and the same semantic label as B corresponding to A), perform the following steps respectively:

[0147] It is determined that the point set of the clustered point cloud component is A, and the point set annotated by the real point cloud component is B.

[0148] Compute the intersection of two point sets , that is, the set of points that belong to both the clustered point cloud component and the real point cloud component annotation.

[0149] Calculate the number of intersection points .

[0150] Calculate the number of points of clustered point cloud components separately and the number of points marked by the real point cloud components .

[0151] According to the definition of IoU, the IoU value between the clustered point cloud component and the real point cloud component annotation is calculated as .

[0152] 3. Evaluating the quality of component clustering

[0153] After obtaining the IoU value of each clustered point cloud component, the quality of component clustering can be evaluated based on these values.

[0154] If the IoU value is close to 1, it means that the clustered point cloud parts are very close to the real point cloud parts annotation, and the clustering quality is high.

[0155] If the IoU value is low, it means that the clustering result is quite different from the actual situation and the clustering quality is low.

[0156] S4: normalizing each clustered point cloud component instance into a sphere of unit radius, and performing feature extraction on the point cloud component instance.

[0157] Each clustered point cloud component instance is normalized to a sphere of unit radius; a small SparseUNet backbone network is constructed, and the normalized point cloud component instance is input into the small SparseUNet backbone network for feature extraction, and finally a feature vector of each point is output, which specifically includes the following technical solutions:

[0158] (1) Normalization of point cloud components

[0159] For each predicted point cloud component, its center point is first determined. The coordinates of the center point can be obtained by calculating the average coordinates of all points in the point cloud component.

[0160] Calculate the distance from each point in the point cloud component to the center point. , the center point is , then the distance from this point to the center point is .

[0161] Normalize all points in the point cloud component. For each point , and update its coordinates to , where d is the maximum distance from all points to the center point. After this processing, the point cloud component is normalized to a sphere with a unit radius.

[0162] (2) Use small SparseUNet for feature extraction

[0163] Build a small SparseUNet backbone network. This network structure can be similar to the larger SparseUNet used previously, but it is appropriately simplified in terms of the number of layers and parameters to adapt to the task of feature extraction of component instances.

[0164] The normalized point cloud components are input into a small SparseUNet.

[0165] The network starts to extract features. Similar to the traditional SparseUNet, it consists of an encoder and a decoder.

[0166] The encoder part gradually downsamples the point cloud to extract features at different levels. Using operations such as sparse convolutional layers and pooling layers, more and more abstract feature representations are extracted from the input point cloud components.

[0167] The decoder part gradually upsamples the feature map and fuses it with the features of the corresponding level in the encoder to restore the detail information of the point cloud. Through deconvolution layers or interpolation operations, the low-resolution feature map is restored to a higher resolution, and the features from the encoder are fused to obtain a richer feature representation.

[0168] Finally, the small SparseUNet outputs the feature vector of each point, which is the feature extraction result of the component instance. It can be used for subsequent tasks, such as predicting the confidence of the component instance and performing posture estimation.

[0169] S5: Predicting confidence levels using the global features of the point cloud component instances, and using the segmentation base model to modify the point cloud component instances whose confidence levels are below a preset threshold.

[0170] The pooling operation in deep learning is used to map the features in the point cloud component instance to a vector of the global feature, and the predicted intersection over union (IoU) of the global feature is used, and the IoU is interpreted as confidence, and the confidence prediction is trained using the L2 loss function. The specific technical solution is as follows:

[0171] (1) Extracting global features of component instances

[0172] For a given part instance, its global features need to be extracted first. Using pooling operations in deep learning, such as global maximum pooling or global average pooling, the features of the point cloud are mapped to a global feature vector.

[0173] (2) Predict IoU and interpret it as confidence

[0174] A prediction model is used to take the extracted global features of the component instance as input to predict the Intersection over Union (IoU) value between the component instance and the true annotation.

[0175] This prediction model can be a simple fully connected neural network or a more complex deep learning model, depending on the complexity of the task and the characteristics of the data.

[0176] After training, the model can output a predicted intersection-over-union (IoU) value based on the input global features.

[0177] Since IoU reflects the quality of a component instance, it is interpreted as confidence. The higher the IoU value, the closer the component instance is to the true annotation, and the higher the confidence; conversely, the lower the IoU value, the lower the confidence.

[0178] (3) Use L2 loss to train confidence prediction

[0179] Define the loss function. In this case, the L2 loss is calculated as , where n is the number of samples, is the true IoU value (i.e., target value), is the IoU value predicted by the model.

[0180] You can also use the following loss:

[0181] ,

[0182] Where y is the IoU, y is in the interval [0,1], and p is the logit predicted by the fully connected neural network, and sigmoid(logit) is y_hat, which is the neural network's prediction of confidence. This ensures that the neural network's predictions fall in the interval [0,1].

[0183] Prepare training data. Collect a batch of part instances with real annotations. For each part instance, extract its global features and calculate the IoU value between it and the real annotation as the target value.

[0184] Perform model training. Take the global features of the component instance as input, the actual IoU value as the target value, and use an optimization algorithm (such as stochastic gradient descent, Adam, etc.) to minimize the loss function. During the training process, the parameters of the model will be continuously adjusted to make the predicted IoU value closer and closer to the actual value.

[0185] Evaluation and adjustment. During the training process, the validation set can be used regularly to evaluate the performance of the model, and the model's hyperparameters, network structure, etc. can be adjusted based on the evaluation results to improve the accuracy of the confidence prediction.

[0186] The point cloud component instances whose confidence level is below the preset threshold are selected, and modified using the segmentation basic model. The modification process is specifically as follows:

[0187] Project the point cloud into an RGB-D image, map the depth channel into the color space, and then superimpose the RGB image and the depth image to obtain an image with depth information;

[0188] Randomly select a number of points in the point cloud component instance as foreground click prompts of the segmentation all basic models, and then project these foreground click prompts into the image space;

[0189] Input the foreground click prompt and the image with depth information into the segmentation basic model, the model infers the 2D component segmentation, and finally maps the 2D component segmentation result back to the 3D point cloud component instance to obtain the corrected segmentation of the point cloud component instance;

[0190] At the same time, in order to improve the effect of the segmentation basic model on the component dataset, a lightweight adapter module is used to fine-tune the segmentation basic model. The input is the picture with depth information and the foreground click prompt, and the output is the segmentation annotation on the real 2D component segmentation picture.

[0191] The foreground click prompt and the image with depth information are input into the segmentation basic model, and the model infers the 2D component segmentation. Finally, the 2D component segmentation result is mapped back to the 3D point cloud component instance to obtain the corrected segmentation of the point cloud component instance, specifically:

[0192] (1) Model architecture

[0193] The architecture of Segment Anything Model (SAM) consists of three main components designed to achieve suggestible segmentation:

[0194] Image Encoder: SAM uses a powerful image encoder based on Visual Transformer (ViT), which is pre-trained with Masked Autoencoder (MAE). The encoder processes high-resolution images and generates C×H×W image embeddings that are only calculated once per image and can be used for different cues, thus improving efficiency.

[0195] Hint Encoder: The hint encoder is flexible and can handle various types of hints, including sparse hints (points, boxes, text) and dense hints (masks). Sparse hints are represented by combining positional encoding with learned embeddings, while dense hints are embedded using convolutional layers.

[0196] Mask Decoder: A lightweight mask decoder combines the information from the image encoder and hint encoder to predict the segmentation mask.

[0197] (2) Training process

[0198] 1. Data preparation:

[0199] Use our parts dataset for data preparation. Since the point clouds in the parts dataset are back-projected from RGB-D images, these RGB-D images can be naturally used to fine-tune Segment Anything.

[0200] The data is preprocessed, including image normalization and data augmentation (such as random rotation, flipping, scaling, etc.) to increase data diversity and improve the generalization ability of the model.

[0201] 2. Loss function:

[0202] The cross entropy loss function or other loss functions suitable for image segmentation tasks are usually used. The loss function measures the difference between the segmentation results predicted by the model and the true annotations, and the model adjusts the parameters by minimizing the loss function.

[0203] 3. Optimization algorithm:

[0204] Select a suitable optimization algorithm, such as stochastic gradient descent (SGD), Adam, etc., to update the model parameters. The optimization algorithm adjusts the model parameters according to the gradient information of the loss function to gradually improve the performance of the model.

[0205] 4. Training strategy:

[0206] Using batch training, a batch of images and corresponding annotations are input into the model for training.

[0207] You can use learning rate adjustment strategies, such as learning rate decay, to gradually reduce the learning rate as training progresses to better converge to the optimal solution.

[0208] The training is performed for multiple iterations until the performance of the model on the validation set reaches a stable state or a specific stopping condition is met.

[0209] (3) Reasoning process

[0210] 1. Input preparation: Take the foreground click prompt and the image with depth information as the input of the model. The foreground click prompt can provide prior information about the location of the component, helping the model to segment more accurately. (The foreground click prompt here is obtained by sampling on the predicted component point cloud, randomly sampling 4 to 8 points)

[0211] 2. Forward propagation: The input data is forward propagated through the model's encoder, intermediate layer, and decoder to obtain the model's category prediction for each pixel (each forward propagation only performs binary classification on the foreground and background of one component segmentation).

[0212] 3. Post-processing: Post-process the segmentation results output by the model, such as removing noise, smoothing boundaries, etc., to improve the quality of segmentation.

[0213] 4. Output results: The final output is the 2D component segmentation result, that is, the component category to which each pixel belongs.

[0214] S6: re-inferring the confidence of the point cloud component instance according to the modified segmentation of the point cloud component instance, and using non-maximum suppression to delete the point cloud component instance with a large overlap.

[0215] The confidence of the point cloud component instance is re-inferred according to the modified segmentation condition of the point cloud component instance.

[0216] Non-Maximum Suppression is used to delete the point cloud component instances with large overlaps. First, the confidence of all the point cloud component instances is sorted from high to low, starting from the clustering of the first point cloud component instance. If the confidence of the subsequent point cloud component instance and the current point cloud component instance is greater than the threshold of the non-maximum suppression, the subsequent point cloud component instance is deleted. The specific technical steps are:

[0217] (1) Preparation

[0218] Make sure you have calculated the confidence predictions for all part instances, and also calculated the Intersection over Union (IoU) between each part instance and every other part instance.

[0219] (2) Sorting component instance

[0220] All part instances are sorted from high to low according to their confidence prediction values.

[0221] (3) Non-maximum suppression process

[0222] 1. Start with the first instance of the sorted component and mark it as the currently processed component.

[0223] 2. Traverse the subsequent component instances: For each subsequent component instance, calculate its IoU value with the currently processed component. If this IoU value is greater than the preset non-maximum suppression threshold, it is considered that this subsequent component instance has a large overlap with the currently processed component and deletes this subsequent component instance. If the IoU value is less than or equal to the threshold, retain this subsequent component instance and continue to check the next subsequent component instance.

[0224] 3. After processing all subsequent component instances, mark the next unprocessed component instance as the currently processed component, and repeat step 2 until all component instances have been processed.

[0225] Through the above steps, confidence prediction can be used to perform non-maximum suppression and remove component instances with large overlaps, thereby obtaining more accurate and concise component detection results.

[0226] S7: using the point-by-point features of the point cloud component instance to predict the initial NOCS coordinates, performing an initial pose estimation of the point cloud component instance on the initial NOCS coordinates, obtaining a secondary NOCS coordinate based on the result of the initial pose estimation, and then performing a secondary pose estimation on the secondary NOCS coordinates.

[0227] The initial NOCS coordinates are predicted using the point-by-point features of the point cloud component instance, and the initial posture estimation of the component instance is performed on the initial NOCS coordinates using the Umeyama algorithm. According to the result of the initial posture estimation, the object point cloud in the camera space is transformed into a coordinate system of a neighboring standard space, and its relative transformation with the standard space depends on the error of the initial posture estimation, which is called a pseudo-standard space. Point-by-point features are extracted in the pseudo-standard space for the point cloud component instance, and the features are used to iteratively predict the secondary NOCS coordinates, and the Umeyama algorithm is used to perform a secondary posture estimation on the NOCS coordinates; at the same time, the L2 loss is used to train the prediction of the initial NOCS coordinates and the secondary NOCS coordinates. The specific technical solution is:

[0228] (1) Predict the initial NOCS coordinates and perform initial attitude estimation

[0229] 1. Predict the initial NOCS coordinates using point-by-point features:

[0230] For a given component instance, its point-by-point features are first extracted. These features can be obtained by the methods used in the previous steps, such as using a specific neural network to extract features from the component point cloud.

[0231] These point-by-point features are input into a prediction model, which is trained to predict the initial Normalized Object Coordinate Space (NOCS) coordinates based on the input features. This prediction process can be based on a deep learning algorithm, which can establish a mapping relationship from point-by-point features to NOCS coordinates by learning a large amount of annotated component instance data.

[0232] 2. Use Umeyama algorithm for initial attitude estimation:

[0233] Once the initial NOCS coordinates are obtained, the Umeyama algorithm can be used to perform an initial pose estimate on the component instance. The Umeyama algorithm is a method for calculating the rigid transformation (including rotation and translation) between two point clouds. In this step, the point cloud of the component instance in camera space is compared with the standard reference point cloud determined by the initial NOCS coordinates, and the rotation and translation parameters that can best align the two point clouds are calculated by the Umeyama algorithm, thereby obtaining the initial pose estimate of the component instance in camera space.

[0234] (2) Constructing a pseudo-standard space

[0235] 1. Perform point cloud transformation based on the initial attitude estimation results:

[0236] The object point cloud in the camera space is transformed using the rotation and translation parameters obtained from the initial pose estimation. The purpose of this transformation is to transform the object from the camera space to a coordinate system in a nearby standard space. The specific transformation process is to transform the coordinates of each point cloud point in the camera space to a new coordinate system through rotation and translation operations.

[0237] 2. Determine the pseudo-standard space:

[0238] Since this new coordinate system is based on the result of the initial attitude estimation, and the initial attitude estimation may have certain errors, there is a certain deviation between this new coordinate system and the true standard space. Therefore, we call this new coordinate system a pseudo-standard space. The relative transformation of this pseudo-standard space depends on the error of the initial attitude estimation. If the initial attitude estimation is more accurate, then the pseudo-standard space will be closer to the true standard space; conversely, if the initial attitude estimation error is large, then the gap between the pseudo-standard space and the standard space will be larger.

[0239] (3) Re-NOCS coordinate prediction and attitude estimation in pseudo-standard space

[0240] 1. Extract point-by-point features:

[0241] In the pseudo-standard space, point-by-point features are extracted again for the part instance. Since the object has already gone through the process of initial pose estimation and transformation to the pseudo-standard space, these new point-by-point features may contain more geometric information about the part instance in a near-standard space. These features can be obtained by a similar method to the initial extraction of point-by-point features, but may need to be adjusted according to the characteristics of the pseudo-standard space.

[0242] 2. Iteratively perform NOCS coordinate prediction again:

[0243] Use the point-by-point features extracted in the pseudo-standard space and input them into another prediction model to perform a second NOCS coordinate prediction. This prediction model can be similar to the initial NOCS coordinate prediction model, but can also be optimized based on the characteristics in the pseudo-standard space. Through this step, it is hoped that more accurate NOCS coordinate prediction results can be obtained.

[0244] 3. Use Umeyama algorithm to estimate the posture again:

[0245] Using the re-predicted NOCS coordinates, the Umeyama algorithm is used again to estimate the pose of the component instance. This process is similar to the initial pose estimation, but is performed in the pseudo-standard space. By comparing the point cloud of the component instance in the pseudo-standard space with the standard reference point cloud determined by the re-NOCS coordinates, the rotation and translation parameters that can best align the two point clouds are calculated, thereby obtaining the re-pose estimation of the component instance in the pseudo-standard space.

[0246] 4. Experiments show that the attitude estimation result is better:

[0247] Through experimental verification, it is found that the result of the second pose estimation is usually better than the initial pose estimation. This is because the pose estimation is decomposed into multiple iterative subtasks, which reduces the amount of information required to learn for each subtask, reduces the learning burden of the model, and improves the model accuracy. The model continuously adjusts and improves the pose estimation, making the pose of the component instance more accurately close to the standard pose.

[0248] In addition, this embodiment also includes a simulated operation experiment in the Sapien simulator to verify the effect of the improved cross-category object perception and manipulation component detection. The operation experiments we conducted include "pushing and pulling the door", "pushing and pulling the drawer", "pressing the button", and "grabbing the round handle". The specific operation strategy of the robot is a heuristic strategy defined according to the component posture. "Pushing and pulling the door": after grabbing the door handle, the rotation axis of the door is determined according to the width and posture of the door, and the movement direction of the mechanical claw is determined according to the position of the rotation axis; "Pushing and pulling the drawer": after grabbing the drawer handle, the mechanical claw moves in the horizontal axis direction of the drawer; "Pressing the button": close the mechanical claw, move the gripper to the top of the z-axis of the button, and move in the negative direction of the z-axis with a posture parallel to the z-axis until the button is pressed; "Grabbing the round handle": close the mechanical claw, move the gripper to the top of the z-axis of the round handle, and move in the negative direction of the z-axis with a posture parallel to the z-axis until the gripper reaches the center of the round handle, then close the gripper, and then move the gripper in the positive direction of the z-axis of the round handle.

[0249] Second embodiment

[0250] like Figure 2 As shown, this embodiment provides a component detection system for performing cross-category object perception and manipulation for performing the component detection method for performing cross-category object perception and manipulation in the first embodiment, including:

[0251] The object point cloud input module 1 is used to input an object point cloud containing a plurality of component point clouds, and each component corresponding to the component point cloud is provided with a corresponding semantic label for indicating the component category, and to extract backbone features from the object point cloud point by point;

[0252] The center point cloud prediction module 2 is used to use the extracted backbone features to perform semantic classification of components point by point, and predict the vector from each point to the center of the component point cloud to obtain the point cloud of the component center predicted point by point;

[0253] A point cloud component clustering module 3 is used to cluster the predicted point cloud of the component center to obtain a predicted point cloud component instance, and to evaluate the clustering quality of the point cloud component instance;

[0254] The point cloud component normalization module 4 is used to normalize each clustered point cloud component instance into a sphere of unit radius and perform feature extraction on the point cloud component instance;

[0255] A point cloud component correction module 5, configured to predict the confidence of the point cloud component instance using the global features of the point cloud component instance, and to correct the point cloud component instance whose confidence is below a preset threshold using the segmentation basic model;

[0256] A point cloud component deletion module 6 is used to re-infer the confidence of the point cloud component instance according to the segmentation of the modified point cloud component instance, and delete the point cloud component instance with large overlap by using non-maximum suppression;

[0257] The iterative attitude estimation module 7 is used to use the point-by-point features of the point cloud component instance to predict the initial NOCS coordinates, perform the initial attitude estimation of the point cloud component instance on the initial NOCS coordinates, obtain the secondary NOCS coordinates according to the result of the initial attitude estimation, and then perform the secondary attitude estimation on the secondary NOCS coordinates.

[0258] A computer-readable storage medium stores computer code. When the computer code is executed, the above method is executed. A person skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program. The program can be stored in a computer-readable storage medium. The storage medium can include: a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.

[0259] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

[0260] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0261] It should be noted that the above embodiments can be freely combined as needed. The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered as the protection scope of the present invention.

Claims

1. A component detection method for performing cross-category object perception and manipulation, characterized in that: include: S1: input an object point cloud containing a plurality of component point clouds, and each component corresponding to the component point cloud is provided with a corresponding semantic label for indicating the component category, and extract backbone features from the object point cloud point by point; S2: Using the extracted backbone features to perform semantic classification of components point by point, and predicting the vector from each point to the center of the component point cloud to obtain a point cloud of the component center predicted point by point; S3: clustering the predicted point cloud of the component center to obtain a predicted point cloud component instance, and evaluating the clustering quality of the point cloud component instance; S4: normalizing each clustered point cloud component instance into a sphere of unit radius, and performing feature extraction on the point cloud component instance; S5: predicting the confidence of the point cloud component instance using the global feature of the point cloud component instance, and modifying the point cloud component instance whose confidence is below a preset threshold using the segmentation basic model; S6: re-inferring the confidence of the point cloud component instance according to the modified segmentation of the point cloud component instance, and deleting the point cloud component instance with a large overlap by using non-maximum suppression; S7: using the point-by-point features of the point cloud component instance to predict the initial NOCS coordinates, performing an initial pose estimation of the point cloud component instance on the initial NOCS coordinates, obtaining a secondary NOCS coordinate based on the result of the initial pose estimation, and then performing a secondary pose estimation on the secondary NOCS coordinates.

2. The component detection method for performing cross-category object perception and manipulation according to claim 1, characterized in that: In step S1, an object point cloud containing several component point clouds is input, and each component corresponding to the component point cloud is provided with a corresponding semantic label for indicating the component category, and backbone features are extracted point by point from the object point cloud, specifically: The object point cloud is obtained by back-projecting the pre-collected RGB-D image, wherein the object point cloud includes a plurality of the component point clouds; The object point cloud is segmented by manually annotating training data, and the object point cloud is divided into a plurality of component point clouds. For the segmented component point clouds, a semantic label is assigned to each of the component point clouds according to the characteristics of the component point clouds and prior knowledge; The SparseUNet network is used as the backbone network to extract the backbone features point by point. The encoder part of the SparseUNet network gradually downsamples and extracts features from the input point cloud to obtain feature maps at different levels. The decoder part of the SparseUNet network gradually upsamples and fuses these feature maps, and finally outputs the vector of the backbone features of each point.

3. The component detection method for performing cross-category object perception and manipulation according to claim 1, characterized in that: In step S2, the extracted backbone features are used to perform semantic classification of components point by point, and the vector from each point to the center of the component point cloud is predicted point by point to obtain the point cloud of the component center predicted point by point, specifically: Pre-training a semantic segmentation model, and using a focal loss function in the semantic segmentation model to solve the problem of class imbalance, giving higher weights to difficult-to-classify samples, using a dice loss function to measure the overlap between the predicted results and the true labels, and processing the irregularly shaped point cloud of the component; The input backbone features are predicted using the semantic segmentation model. When the semantic segmentation model predicts that the backbone features belong to a certain component category, the current backbone features are marked as the corresponding component category. If the semantic segmentation model predicts that the backbone features do not belong to any component category, the current backbone features are marked as the background category. Pre-training a component center prediction model, and using a vector direction loss in the component center prediction model to calculate a direction difference between a predicted vector and a true vector, using a vector length loss to ensure that the length of the predicted vector is close to the length of the true vector, and performing a weighted combination of the vector direction loss and the vector length loss to obtain a final loss function; The component center prediction model is used to predict the prediction vector of the point to the center of the component point cloud to which the backbone feature is input point by point, and the point cloud of the component center predicted point by point is obtained by moving each point along the prediction vector.

4. The component detection method for performing cross-category object perception and manipulation according to claim 3, characterized in that: In step S3, the predicted point cloud of the component center is clustered to obtain a predicted point cloud component instance, and the clustering quality of the point cloud component instance is evaluated, specifically: The spherical radius r of the cluster and the maximum number of cluster points are preset, and an unprocessed point is randomly selected from the point cloud at the center of the component as the starting point, and the nearest point with the same semantics within the spherical radius r from the point is connected to the starting point, and then starting from the point newly included in the connected block, the nearest point with the same semantics within the spherical radius r from the new starting point is recursively included in the connected block until there is no point that meets the requirements within the spherical radius r, or until the number of points in the cluster is greater than the pre-specified threshold of the maximum number of cluster points, so as to obtain the clustered point cloud component instance; Each clustered point cloud component instance is compared with the real point cloud component annotation to obtain the intersection over union (IoU) ratio between the point cloud component instance and the real point cloud component, which is used to indicate the quality of the current point cloud component instance clustering.

5. The component detection method for performing cross-category object perception and manipulation according to claim 1, characterized in that: In step S4, each clustered point cloud component instance is normalized to a sphere of unit radius, and feature extraction is performed on the point cloud component instance, specifically: Normalizing each clustered point cloud component instance to a sphere with a unit radius; A small SparseUNet backbone network is constructed, and the normalized point cloud component instance is input into the small SparseUNet backbone network for feature extraction, and finally a feature vector of each point is output.

6. The component detection method for performing cross-category object perception and manipulation according to claim 1, characterized in that: In step S5, the confidence of the point cloud component instance is predicted using the global features, and the point cloud component instance with the confidence below the preset threshold is corrected using the segmentation basic model, specifically: Using a pooling operation in deep learning to map features in the point cloud component instance to a vector of the global feature, using the predicted intersection over union (IoU) of the global feature, and interpreting the IoU as confidence, while using an L2 loss function to train confidence prediction; The point cloud component instances whose confidence level is below the preset threshold are selected, and modified using the segmentation basic model. The modification process is specifically as follows: Project the point cloud into an RGB-D image, map the depth channel into the color space, and then superimpose the RGB image and the depth image to obtain an image with depth information; Randomly select a number of points in the point cloud component instance as foreground click prompts of the segmentation all basic models, and then project these foreground click prompts into the image space; Input the foreground click prompt and the image with depth information into the segmentation basic model, the model infers the 2D component segmentation, and finally maps the 2D component segmentation result back to the 3D point cloud component instance to obtain the corrected segmentation of the point cloud component instance; At the same time, in order to improve the effect of the segmentation basic model on the component dataset, a lightweight adapter module is used to fine-tune the segmentation basic model. The input is the picture with depth information and the foreground click prompt, and the output is the segmentation annotation on the real 2D component segmentation picture.

7. The component detection method for performing cross-category object perception and manipulation according to claim 1, characterized in that: In step S6, according to the modified segmentation of the point cloud component instance, the confidence of the point cloud component instance is re-inferred, and the point cloud component instance with large overlap is deleted by using non-maximum suppression, specifically: Re-reasoning the confidence of the point cloud component instance according to the modified segmentation of the point cloud component instance; Non-maximum suppression is used to delete the point cloud component instances with large overlap. First, the confidence levels of all the point cloud component instances are sorted from high to low, starting from the clustering of the first point cloud component instance. If the confidence level of a subsequent point cloud component instance and the current point cloud component instance is greater than the threshold of the non-maximum suppression, the subsequent point cloud component instance is deleted.

8. The component detection method for performing cross-category object perception and manipulation according to claim 1, characterized in that: In step S7, the point-by-point features of the point cloud component instance are used to predict the initial NOCS coordinates, the initial posture of the point cloud component instance is estimated for the initial NOCS coordinates, and the secondary NOCS coordinates are obtained according to the result of the initial posture estimation, and then the secondary posture estimation is performed on the secondary NOCS coordinates, specifically: The initial NOCS coordinates are predicted using the point-by-point features of the point cloud component instance, the initial posture estimation of the component instance is performed on the initial NOCS coordinates using the Umeyama algorithm, the object point cloud in the camera space is transformed into a coordinate system of a neighboring standard space according to the result of the initial posture estimation, and the relative transformation between the object point cloud in the camera space and the standard space depends on the error of the initial posture estimation, which is called a pseudo-standard space, point-by-point features are extracted for the point cloud component instance in the pseudo-standard space, the features are used to iteratively perform the second NOCS coordinate prediction, and the posture estimation of the NOCS coordinates is performed again using the Umeyama algorithm; The L2 loss is used to train the prediction of the initial NOCS coordinates and the secondary NOCS coordinates at the same time.

9. The component detection method for performing cross-category object perception and manipulation according to claim 1, characterized in that: Also includes: Simulated operation experiments are conducted in the Sapien simulator to verify the effectiveness of improved cross-category object perception and manipulation component detection.

10. A component detection system for performing cross-category object perception and manipulation for performing the component detection method for performing cross-category object perception and manipulation as claimed in any one of claims 1 to 9, characterized in that: include: An object point cloud input module is used to input an object point cloud containing a plurality of component point clouds, and each component corresponding to the component point cloud is provided with a corresponding semantic label for indicating the component category, and to extract backbone features from the object point cloud point by point; A center point cloud prediction module is used to use the extracted backbone features to perform semantic classification of components point by point, and predict the vector from each point to the center of the component point cloud to obtain a point cloud of the component center predicted point by point; A point cloud component clustering module, used for clustering the predicted point cloud of the component center to obtain a predicted point cloud component instance, and at the same time evaluating the clustering quality of the point cloud component instance; A point cloud component normalization module, used for normalizing each clustered point cloud component instance into a sphere of unit radius, and performing feature extraction on the point cloud component instance; A point cloud component correction module, used to predict the confidence of the point cloud component instance using the global features of the point cloud component instance, and to correct the point cloud component instance whose confidence is below a preset threshold using the segmentation basic model; A point cloud component deletion module, used for re-inferring the confidence of the point cloud component instance according to the segmentation of the modified point cloud component instance, and deleting the point cloud component instance with large overlap by using non-maximum suppression; An iterative attitude estimation module is used to use the point-by-point features of the point cloud component instance to predict the initial NOCS coordinates, perform the initial attitude estimation of the point cloud component instance on the initial NOCS coordinates, obtain the secondary NOCS coordinates according to the result of the initial attitude estimation, and then perform the secondary attitude estimation on the secondary NOCS coordinates.

Images