A method and system for robotic manipulation of articulated objects

Abstract

This invention discloses a method and system for robot manipulation of articulated objects, comprising the following steps: standardizing the input 3D point cloud data to process it at a uniform scale; segmenting and encoding the point cloud data using a multi-scale feature extraction module to extract global and local features, and then generating point-level features using a point-level feature extraction module; segmenting the object into parts using a segmentation network, determining the mobility of the parts, and estimating the joint type, orientation, position, and state of the movable parts in a joint parameter estimation network; further generating operation suggestions and scores for the end effector through an operation feasibility assessment network, determining the optimal operation direction and interaction point, and calculating the operation trajectory of the end effector through a trajectory planning module to complete operation commands such as pushing and pulling on the object. The system of this invention introduces self-supervised learning for pre-training during the training phase to improve its generalization ability and ensure high performance even on objects of unseen categories.

Description

Technical Field

[0001] This invention relates to the field of robot operation and intelligent control technology, and in particular to a method and system for a robot to manipulate articulated objects. Background Technology

[0002] In the field of service robots, robot operating systems face increasing demands for the recognition and manipulation of multi-jointed objects. Manipulating multi-jointed objects involves complex geometric structures and motion relationships, such as opening and closing cabinet doors, extending and retracting drawers, and manipulating multi-jointed hinged objects, placing higher demands on the robot system's perception, analysis, and path planning. However, existing methods have several shortcomings in handling multi-jointed object manipulation. First, existing object segmentation methods rely on simple 2D or 3D segmentation algorithms, which are affected by object occlusion, lighting changes, and pose differences, making it difficult to accurately identify joints and manipulable parts in complex scenes. Furthermore, these methods have poor generalization ability under scene changes or new object categories, limiting their practical application. Second, joint parameter estimation relies on geometric assumptions or template matching algorithms, which cannot provide reliable parameter estimates when dealing with objects with complex structures or diverse joint degrees of freedom. They are highly dependent on the object's shape and joint type, and are sensitive to noisy data, easily generating errors when perception data is interfered with, making it difficult to meet the needs of practical applications.

[0003] Furthermore, current operational methods typically rely on static predefined paths or heuristic algorithms, such as simple motion planning algorithms like A*, which struggle to generate dynamic operation paths based on actual joint structures and operational requirements. This results in insufficient versatility and flexibility, making it difficult to adapt to real-time changes in different operation points and joint states. Simultaneously, existing technologies are heavily dependent on large amounts of labeled data, but accurately labeling multi-joint object datasets is time-consuming and costly, limiting model generalization and robustness. Although self-supervised learning techniques have made progress in mitigating label dependency in recent years, their application in 3D point cloud data remains immature. Therefore, to improve the applicability and efficiency of robot systems in multi-joint object recognition and manipulation, there is an urgent need to develop a general-purpose robot operating system capable of accurate segmentation, dynamic path planning, and efficient joint parameter estimation.

[0004] In existing technologies, research on manipulation of articulated objects mainly focuses on single aspects such as part segmentation, joint parameter estimation, and availability analysis. For example, RPM-Net has made some progress in joint parameter estimation (RPM-Net_Recurrent Prediction of Motion and Parts from Point Cloud), but it often relies on a large amount of labeled data and has insufficient generalization ability for new object categories. The Cart method attempts to predict parameters through command-driven methods (Command-Driven Articulated Object Understanding and Manipulation CVPR2023 paper), but it lacks accurate modeling of the overall structure of multi-jointed objects in complex interaction scenarios. MARS performs well in multimodal data fusion (Multimodal Active Robotic Sensing for Articulated Characterization), but data acquisition is complex and has high requirements for scene matching. In addition, the Where2Act method has also achieved some results in operational feasibility analysis and action suggestion (Where2Act From Pixels to Actions for Articulated 3D Objects ICCV 2021 paper), but they usually rely on fixed action primitives and are difficult to cope with the ever-changing practical operational requirements. Summary of the Invention

[0005] The purpose of this invention is to develop a general-purpose robot operating system and method based on multi-scale feature extraction and joint parameter estimation, enabling service robots to efficiently identify and manipulate multi-jointed objects in complex and diverse environments. This method can accurately segment the operable parts of multi-jointed objects and generate dynamic operation paths based on the object's joint type, orientation, and position, thereby solving the problems of insufficient segmentation accuracy, inaccurate joint parameter estimation, and lack of path planning flexibility encountered by current robot operating systems in complex scenarios. By introducing self-supervised learning technology, this invention reduces the dependence on large amounts of labeled data, improves the system's generalization ability, and makes the robot more adaptable to practical application scenarios such as home and medical settings.

[0006] The present invention is achieved by at least one of the following technical solutions.

[0007] A method for a robot to manipulate an articulated object includes the following steps:

[0008] Step S1: Obtain the 3D point cloud data of the target object and standardize the 3D point cloud data so that it can be processed at a uniform scale;

[0009] Step S2: Divide and encode the point cloud data using the Multi-Scale Feature Extraction (MSFE) module to extract global and local features;

[0010] Step S3: Generate point-level features through the point-level feature extraction module (PFE) for object segmentation and operation planning;

[0011] Step S4: The component segmentation module uses the segmentation network (Seg-Net) to segment the object into components, determines the mobility of the components, and estimates the joint type, orientation, position and state of the movable components in the joint parameter estimation module (Para-Net).

[0012] Step S5: Further generate operational suggestions and scores for the end effector through the Afford-Net operational feasibility assessment network to determine the optimal operational direction and interaction point;

[0013] Step S6: Plan the operation trajectory of the end effector through the trajectory planning module to complete the push and pull operation command of the object.

[0014] Furthermore, the standardization of 3D point cloud data is achieved by modifying the input 3D point cloud data matrix. After normalization, a unified coordinate system is established. The normalization formula is as follows:

[0015]

[0016] Where X represents the input 3D point cloud data matrix, containing N points, each point having three coordinate components: x, y, and z. i X is the coordinate of the i-th point, and X′ represents the normalized point cloud data. This represents a matrix containing N points, each with three coordinate components (x, y, z).

[0017] Furthermore, the multi-scale feature extraction module employs multi-level farthest point sampling (FPS) and k-nearest neighbor (KNN) algorithms to divide the point cloud data into multi-scale segments, representing each layer of the point cloud as... The specific formula is as follows:

[0018]

[0019] Among them, C i Let m represent the set of center points obtained by sampling at the i-th scale. i Let k be the number of center points in the i-th layer. i The number of neighboring points for each center point, P i This represents the block data of the i-th layer;

[0020] Multi-scale features are encoded using a Vision Transformer (ViT), with the input for each scale being:

[0021]

[0022] Where, Mini-PointNet(P1) represents the preliminary feature extraction of the input point cloud block data P1 through the Mini-PointNet network, θ enc (.) represents the positional encoding of the center points. C1 is the set of center points at the first scale, and T1 is the point cloud feature matrix processed by the visual transformer, with dimensions m1×d1, where m1 is the number of center points in the point cloud at this scale, and d1 is the dimension after feature encoding. This represents the feature matrix with dimension m1×d1.

[0023] Furthermore, the point-level feature extraction module utilizes multi-scale feature propagation to extract point-level features, aggregating the encoded features from each scale to obtain point-level features, as shown in the formula:

[0024]

[0025] Where f is a point-level feature. T represents the feature connection operation. i For the feature at the i-th scale, For feature propagation operation, n is the number of scales.

[0026] Furthermore, the segmentation network (Seg-Net) is used to identify movable and immovable parts within an object. For each point, it performs binary classification to predict whether it belongs to a movable part, outputting a mobility label τ and a part category label γ.

[0027] τ=θ mov (F), γ=θ seg (F), τ∈{0,1};

[0028] Where, θ mov (.) and θ seg (.) represents the prediction function, F represents the extracted point cloud features, and the loss function of the segmentation network includes binary cross-entropy loss (BCE) and multi-class cross-entropy loss, with the specific formula as follows:

[0029]

[0030] Among them, L seg L represents the loss of the entire network. bce (.) represents the binary cross-entropy loss, L ce(.) represents multi-class cross-entropy loss. and These are the actual labels for mobility and segmentation, respectively.

[0031] Furthermore, the joint parameter estimation module (Para-Net) is used to estimate the joint type, orientation, position, and current state, specifically including the following steps:

[0032] Step S3-1, Joint Type Classification: Determine the joint type ρ using binary classification, where ρ = 0 represents a rotational joint and ρ = 1 represents a prismatic joint. The classification loss function is:

[0033]

[0034] Among them, L type Represents classification loss, It predicts joint type;

[0035] Step S3-2, Joint Position Estimation: For a rotational joint, estimate the joint position u and compare it with the actual position. The projected distance between them is used for path planning, and its loss function is defined as:

[0036]

[0037] in, Used to measure the estimated rotational joint position u compared to the actual joint position. In the direction of the actual joint axis Projection difference on is the unit vector in the joint direction;

[0038] Step S3-3, Joint Orientation Estimation: Calculate the estimated joint orientation value o and the true value. The angle between Defined as:

[0039]

[0040] Step S3-4, Joint State Estimation: Apply L1 loss to the current state s of the joint and compare it with the true state. contrast:

[0041]

[0042] Here L state The regression loss represents the joint state and is used to measure the difference between the model's predicted current joint state s and the actual joint state. The differences between them.

[0043] Furthermore, the operational feasibility analysis module (Afford-Net) is used to generate operational feasibility scores and the rotation angle of the end effector, specifically including:

[0044] Step S4-1, Operation posture generation: Generate operation posture R for each operation point. z Let be the three-degree-of-freedom rotation matrix in SO(3) space, and the formula for calculating the operation attitude is:

[0045]

[0046] Where, θ AP (.) represents the operation pose generation function, F represents the point cloud features, z represents random Gaussian noise, and a t This refers to the operation type, with a value of 0 indicating a push or 1 indicating a pull. Represents a 3×3 matrix;

[0047] Step S4-2, Operational Feasibility Score Calculation: The feasibility score as of the operation point is calculated using binary classification and is defined as follows:

[0048] L AS =r·log(θ) AS (F p|S ,R,a t ))+(1-r)·log(1-θ AS (F p|S ,R,a t ));

[0049] Among them, L AS θ represents the loss of the operational feasibility scoring module. AS The rating generation function, where r is the true label with a value of 0 or 1, and F... p|S R represents the characteristics of the current operation point, and R denotes the 3×3 rotation matrix in SO(3) space.

[0050] Furthermore, the planning of the end effector's operating trajectory in step S6 is based on joint parameters and the target state, and the trajectory is generated through interpolation calculation, including:

[0051] For prism joints, the motion trajectory is planned according to the translation path, and is represented as follows:

[0052] p(t)=p+t·(l·o),t∈[0,1];

[0053] Where p(t) represents the spatial position of the end effector or the operated point at the interpolation parameter time t, p is the starting point, l is the moving distance, and o is the moving direction;

[0054] For a rotary joint, the rotation path is calculated using the Rodriguez formula, defined as:

[0055] p(t)=R(o,l·t)(pu)+u,t∈[0,1];

[0056] Where R(o,l@t) represents the rotation matrix about axis o, p is the starting point, and u is the position of the rotation joint.

[0057] A system for implementing the method of robot manipulating an articulated object includes:

[0058] The multi-scale feature extraction module divides point cloud data into multiple scales and extracts object features at different scales, providing comprehensive feature information for subsequent modules to adapt to diverse object structures and scales. On the normalized point cloud, the farthest point sampling and k-nearest neighbor algorithm are used to divide the point cloud data into multi-scale point blocks.

[0059] The point-level feature extraction module propagates multi-scale features to each point and integrates information from each scale to generate point-level features for object segmentation and operation planning.

[0060] The component segmentation module identifies operable components in an object, distinguishes between movable and immovable parts, and generates segmentation information that provides the basis for the joint parameter estimation module to estimate operable components.

[0061] The joint parameter estimation module provides geometric and kinematic information of the joints by predicting their type, orientation, position, and current state.

[0062] The operational feasibility analysis module generates the posture and score of the operation point to determine the interaction parameters of the robot's end effector.

[0063] The dynamic path planning module generates operation paths based on joint parameter estimation and operation feasibility analysis results, meeting the operation requirements of multi-joint objects and generating different paths for different types of joints.

[0064] A computer device according to the present invention includes: a memory and a processor, and a computer program stored in the memory, wherein when the computer program is executed on the processor, it implements a method for a robot to manipulate an articulated object.

[0065] Compared with existing technologies, the beneficial effects of the present invention are as follows:

[0066] This invention discloses a method for robot manipulation of articulated objects, comprising multi-scale point cloud feature extraction and aggregation, modular segmentation and joint parameter estimation, self-supervised learning pre-training, and dynamic trajectory planning and operation evaluation. Through multi-scale feature extraction and point-level feature aggregation, accurate perception of objects of different sizes and shapes is achieved; the segmentation and joint parameter estimation modules provide detailed object information; self-supervised learning enhances the system's generalization ability, enabling it to adapt to various real-world scenarios; dynamic trajectory planning ensures the robot can autonomously generate paths that meet task requirements, improving operational performance in complex environments. Innovations in point cloud data processing, modular design, and adaptive operation path planning significantly enhance the application potential of service robots in manipulating multi-jointed objects, applicable to various jointed structures, including but not limited to cabinet doors, drawers, revolving doors, and foldable furniture. Attached Figure Description

[0067] Figure 1 This is a flowchart illustrating a method for a robot to manipulate an articulated object according to an embodiment;

[0068] Figure 2 This is a diagram illustrating the process of dividing a point cloud into multi-scale patches using multi-level FPS and KNN in an embodiment.

[0069] Figure 3 This is a visualization of the segmentation, joint parameter estimation, and availability suggestions for eight instances across four categories in the embodiments.

[0070] Figure 4 This is a diagram showing the results of a real-world robot experiment in the example. Detailed Implementation

[0071] To better understand the above-mentioned objects, features, and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Furthermore, it should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. Therefore, the scope of protection of the present invention is not limited to the specific embodiments disclosed below.

[0072] To achieve the aforementioned objectives, this invention constructs a complete robot operating system, from perception to manipulation, through the collaborative work of multiple modules. The overall system flow includes six core modules: multi-scale feature extraction, point-level feature extraction, component segmentation, joint parameter estimation, operational feasibility analysis, and dynamic path planning. This embodiment provides a robot manipulation system for articulated objects, comprising:

[0073] The multi-scale feature extraction module divides point cloud data into multiple scales and extracts object features at different scales, providing comprehensive feature information for subsequent modules to adapt to diverse object structures and scales. On the normalized point cloud, the farthest point sampling and k-nearest neighbor algorithm are used to divide the point cloud data into multi-scale point blocks.

[0074] The point-level feature extraction module propagates multi-scale features to each point and integrates information from each scale to generate point-level features for object segmentation and operation planning.

[0075] The component segmentation module identifies operable components in an object, distinguishes between movable and immovable parts, and generates segmentation information that provides the basis for the joint parameter estimation module to estimate operable components.

[0076] The joint parameter estimation module provides geometric and kinematic information of the joints by predicting their type, orientation, position, and current state.

[0077] The operational feasibility analysis module generates the posture and score of the operation point to determine the interaction parameters of the robot's end effector.

[0078] The dynamic path planning module generates operation paths based on joint parameter estimation and operation feasibility analysis results, meeting the operation requirements of multi-joint objects and generating different paths for different types of joints.

[0079] like Figure 1 As shown in the figure, a method for a robot to manipulate an articulated object according to this embodiment includes the following steps:

[0080] Step S1: The multi-scale feature extraction module acquires the 3D point cloud data of the target object and standardizes the 3D point cloud data to enable processing at a uniform scale.

[0081] Collect point cloud data of objects Normalize the data to represent it in the same coordinate system, which facilitates subsequent processing. The normalization formula is as follows:

[0082]

[0083] Where X represents the input 3D point cloud data matrix, containing N points, each point having three coordinate components: x, y, and z. i X is the coordinate of the i-th point, and X′ represents the normalized point cloud data. This represents a matrix containing N points, each with three coordinate components (x, y, z). After normalization, the point cloud data X′ reduces the impact of data offset and scale differences, which helps improve the stability of subsequent feature extraction.

[0084] Step S2: Divide and encode the point cloud data using the Multi-Scale Feature Extraction (MSFE) module to extract global and local features;

[0085] On the normalized point cloud, the point cloud data is divided into multi-scale point blocks using multi-layer farthest point sampling (FPS) and k-nearest neighbor (KNN) algorithms. For each scale layer i, the set of center points C... i Use KNN to obtain the block data P i :

[0086]

[0087] Where, m i Let k be the number of center points in the i-th layer. i The number of neighboring points for each center point. This multi-scale partitioning strategy can adapt to objects of different scales and extract structural information at different levels.

[0088] Feature encoding: For each layer of block data P i Feature encoding is performed using a visual transformer. The input encoding calculation formula is:

[0089]

[0090] Where, Mini-PointNet(P1) represents the preliminary feature extraction of the input point cloud data P1 through the Mini-PointNet network, θ enc (.) represents the location encoding of the center points, C1 is the set of center points at the first scale, and T1 is the processed point cloud feature matrix with dimensions m1×d1, where m1 is the number of center points in the point cloud at this scale, and d1 is the dimension after feature encoding. This represents a feature matrix with dimensions m1×d1. Through multi-scale feature encoding, the system can perform detailed analysis of complex object structures, providing support for subsequent component segmentation and parameter estimation.

[0091] Step S3: Generate point-level features through the point-level feature extraction module (PFE) for object segmentation and operation planning;

[0092] The point-level feature extraction module utilizes multi-scale feature propagation to extract point-level features, aggregating the encoded features at each scale to obtain point-level features. Multi-scale feature aggregation: Propagates multi-scale features to each point and integrates information from each scale to generate point-level features f.

[0093]

[0094] in, T represents the feature connection operation. i For the feature at the i-th scale, For feature propagation operation, n is the number of scales.

[0095] Feature pooling and dimensionality reduction: Max pooling and average pooling operations are performed on the integrated features, and one-dimensional convolution is used for dimensionality reduction to reduce computational costs and provide point-level features for subsequent modules.

[0096] Step S4: The component segmentation module uses the segmentation network (Seg-Net) to segment the object into components, determines the mobility of the components, and estimates the joint type, orientation, position and state of the movable components in the joint parameter estimation network (Para-Net).

[0097] The component segmentation module aims to identify operable parts within an object, distinguishing between movable and immovable sections. The steps include:

[0098] 3.1 Component Classification: Based on multi-scale features, binary classification is performed on each point in the point cloud to predict whether it belongs to a movable component, generating a mobility label τ and a component category label γ to determine whether each point belongs to an operable component.

[0099] τ=θ mov (F), γ=θ seg (F), τ∈{0,1};

[0100] Where, θ mov (.) and θ seg (.) is the prediction function, F represents the extracted point cloud features, movable parts refer to those parts that can be manipulated by the robot or have their position or orientation changed, such as hinged doors or drawers, while static parts refer to those parts that are fixed in the object and cannot have their position or orientation changed.

[0101] 3.2 Loss Function Optimization: The segmentation results are optimized using Binary Cross-Entropy (BCE) to ensure the model can accurately segment joint structure components in complex environments. The loss function is:

[0102]

[0103] Among them, L seg L represents the loss of the entire network. bce (.) represents the binary cross-entropy loss, L ce (.) represents multi-class cross-entropy loss. and These are the true labels for mobility and segmentation, respectively. The segmentation information generated by this module provides an accurate basis for subsequent joint parameter estimation of the maneuvering components.

[0104] The joint parameter estimation module provides geometric and kinematic information about the joints by predicting their type, orientation, position, and current state. The steps are as follows:

[0105] (1) Feature aggregation: The features F of the segmented movable parts are aggregated. κ The aggregation is performed using the following formula:

[0106]

[0107] Where F represents the point-level features extracted from the point cloud, and κ represents the selected component number. This indicates an aggregation operation.

[0108] (2) Joint type classification: The joint type ρ is determined by binary classification, where ρ = 0 represents a rotational joint and ρ = 1 represents a prismatic joint. The classification loss function is:

[0109]

[0110] Among them, L type Represents classification loss, It predicts the joint type. The accuracy of this classification affects the applicability of subsequent path planning.

[0111] (3) Joint position estimation: For rotational joints, the position projection distance is used as the loss index to estimate the error of the joint position u, and the estimated joint position u is compared with the true position. The projected distance between them is used for path planning, and its loss function is defined as:

[0112]

[0113] in, Used to measure the estimated joint position u compared to the actual joint position. In the direction of the actual joint axis Projection difference on It is a unit vector quantity in the joint direction.

[0114] (4) Joint orientation estimation: The estimated joint orientation o is calculated and compared with the true orientation. The angle between the two sides is used to define the direction loss function:

[0115]

[0116] Accurate orientation estimation enables the robot to perform the correct rotational movements.

[0117] (5) Joint state estimation: Apply L1 loss to the current state s of the joint and compare it with the true state. contrast:

[0118]

[0119] Where L state The regression loss represents the joint state and is used to measure the difference between the model's predicted current joint state s and the actual joint state. The differences between them. Joint state information determines the degree of robot execution under the current operation.

[0120] Step S5: Further generate operational suggestions and scores for the end effector through the Afford-Net operational feasibility assessment network to determine the optimal operational direction and interaction point. Specific steps include:

[0121] 5.1 Operation posture generation: Generate operation posture R for each operation point. z Let represent the three-degree-of-freedom rotation matrix in SO(3) space. The formula for calculating the operation attitude is:

[0122]

[0123] Where, θ AP Let F be the pose generation function, z be the point cloud feature, and a be the random Gaussian noise. t The operation type is 0 (push) or 1 (pull). This indicates the existence of a 3×3 matrix, meaning that a three-dimensional rotation matrix has been generated for each of the N' points (or candidate interaction positions) in the data;

[0124] 5.2 Calculation of Operational Feasibility Score: The feasibility score (as) of an operation point is calculated using a binary classification method and is defined as follows:

[0125] L AS =r·log(θ) AS (F p|S ,R,a t ))+(1-r)·log(1-θ AS (F p|S ,R,a t ));

[0126] Among them, L AS θ represents the loss of the operational feasibility scoring module. AS The rating generation function, where r is the true label with a value of 0 or 1, and F... p|S R represents the characteristics of the current operation point, and R denotes the 3×3 rotation matrix in SO(3) space.

[0127] Step S6: The trajectory planning module plans the operation trajectory of the end effector to complete the push-pull operation command on the object. This module generates the operation path based on joint parameter estimation and operation feasibility analysis results to meet the operation requirements of multi-joint objects. Different paths are generated for different types of joints.

[0128] For prism joints, the motion trajectory is planned according to the translation path, and is represented as follows:

[0129] p(t)=p+t·(l·o),t∈[0,1];

[0130] Where p(t) represents the spatial position of the end effector (or the operated point) at the interpolation parameter time t, p is the starting point, l is the distance traveled, and o is the direction of travel;

[0131] For a rotary joint, the rotation path is calculated using the Rodriguez formula, defined as:

[0132] p(t)=R(o,l·t)(pu)+u,t∈[0,1];

[0133] Where R(o,l·t) represents the rotation matrix about axis o, p is the starting point, and u is the position of the rotation joint;

[0134] To illustrate the effectiveness of the method proposed in this invention, as a specific embodiment, the following detailed description of the above technical solution of this invention is provided through a specific example, comprising data preparation, pre-training, fine-tuning training, model inference, path planning, and operation execution modules. The specific implementation steps are as follows:

[0135] Step 1: Dataset collection and preparation;

[0136] Step 1-1: Dataset selection;

[0137] This example uses the PartNet-Mobility and Shape2Motion datasets, which include various types of multi-jointed objects such as cabinets, refrigerators, tables, and safes. Each object contains multiple instances, and the joint states are randomly adjusted multiple times to enrich the dataset.

[0138] Steps 1-2: Data partitioning;

[0139] The dataset was divided into training, validation, and test sets in a 9:1:1 ratio.

[0140] Steps 1-3: Data preprocessing;

[0141] Capture multi-view point cloud data of objects using an RGB-D camera or other depth sensor. Generate multi-view point cloud frames for each object, and perform noise reduction and normalization on the point clouds to ensure the data is in a unified coordinate system. Output standardized point cloud data files, each containing multi-view point cloud frames of the object at different joint states.

[0142] Step 2, Pre-training process;

[0143] Step 2-1: Preparing the pre-training dataset;

[0144] The ShapeNet dataset was chosen for pre-training of the multi-scale feature extraction module. The ShapeNet dataset contains more than 50,000 3D models, covering a wide range of shapes and categories.

[0145] Step 2-2: Multi-scale point block division;

[0146] The point cloud parameters are defined as M = {512, 256, 64} and the number of points in each block is K = {32, 8, 8}, capturing object features at different scales. The point cloud data is segmented using farthest point sampling (FPS) and k-nearest neighbor (KNN) methods to generate multi-scale segmented point clouds as input data. Figure 2 This demonstrates the process of dividing a point cloud into blocks. First, a 3D point cloud is obtained. Where N represents the number of points, and each point contains three-dimensional coordinates (x, y, z). For example, corresponding to Figure 2 The leftmost 3D model. At the first scale, m1 = 512 center points were selected from the point cloud X using farthest point sampling, denoted as... Here, 512 comes from the first value in parameter M, indicating that more center points are retained at the highest resolution to cover the surface details of the object. For each center point, k1 = 32 nearest points are found in its local neighborhood using the k-nearest neighbor algorithm, and these neighboring points are combined with the center point to form a patch. Let... This represents the set of blocks corresponding to all center points, i.e., the block data of the first layer. At this point, (m1 = 512, k1 = 32) correspond to the number of center points and the number of neighbor points, respectively. Next, at the second scale, FPS is performed again on the center point set C1 obtained from the previous layer, resulting in m2 = 256 new center points, denoted as... Then, for each center point, KNN is used to select k2 = 8 neighbor points to form the second layer of blocks. The values ​​(m2 = 256, k2 = 8) come from the second set of values ​​for parameters M = {512, 256, 64} and K = {32, 8, 8}. Similarly, at the third scale, the farthest point is sampled again from the set of center points C2 of the second layer, selecting m3 = 64 center points. Then, k3 = 8 neighbors are selected for each center point to obtain the third layer of blocks. At this point, center points and neighborhood point patches at three different scales have been obtained. By repeating the FPS+KNN process at different resolutions, multi-scale point cloud blocks can be obtained. These blocks can then be fed into the corresponding feature encoding network, and the multi-scale feature propagation module can be used to fuse the features of each layer into the point-level features, thereby obtaining a representation that combines local details and global structure.

[0147] Steps 2-3: Use self-supervised learning techniques to pre-train the multi-scale feature extraction module to enhance the model's generalization ability;

[0148] The AdamW optimizer was used to train for 300 epochs with a learning rate of 0.001 and a batch size of 32 on the ShapeNet dataset to improve the model's ability to extract multi-scale features. Cross-entropy loss and L2 regularization were used to ensure that the model could efficiently capture shape and geometric features in point cloud data. A pre-trained Multi-Scale Feature Extraction (MSFE) module was used to extract rich shape and scale information.

[0149] Step 3: Fine-tune the training;

[0150] Step 3-1: Model Input;

[0151] The preprocessed multi-scale point cloud data is fed into the following task modules via the MSFE module:

[0152] Part segmentation: The Seg-Net module uses multi-scale features to perform binary classification on each point, generating a mobility label τ and a part category label γ for each point.

[0153] Joint parameter estimation: The Para-Net module predicts the object's joint type (rotational or prismatic joint), joint orientation, joint position, and current state.

[0154] Operational feasibility analysis: The Afford-Net module generates the pose matrix R of the operation points. z And operation score as, to select the best operation point.

[0155] Step 3-2: Loss Function and Optimization;

[0156] Component segmentation loss: Binary cross-entropy (BCE) and cross-entropy loss are used, as shown in the following formula:

[0157]

[0158] Joint parameter loss: This includes estimation losses for joint type, position, and orientation. The joint type loss is as follows:

[0159]

[0160] As one example, the AdamW optimizer was used, trained with a learning rate of 0.001 and a batch size of 32 for 100 epochs.

[0161] Step 4: Model Reasoning Process

[0162] Step 4-1, Input;

[0163] A system that uses point cloud data acquired by a depth camera or other 3D sensor to input into a novel method for manipulating articulated objects in robots.

[0164] Step 4-2, Reasoning Process;

[0165] Figure 3 Figures a and b respectively illustrate the visualization results of the inference performed by the method of the present invention on two different instances of four categories (door, computer, cabinet, microwave oven). To clearly illustrate the processing, the results of the four stages of input, segmentation, availability, and relation are given for each instance of each category from left to right, as explained in detail below:

[0166] Input: refers to the raw point cloud or 3D model data input into the system. It can be seen that the geometric appearance and posture of different instances vary considerably; for example, the door exhibits a distinct rectangular plate-like structure, while the computer has a flip-top cover and a base.

[0167] Segmentation: This column shows the results of the system performing component segmentation on the point cloud, with different colors corresponding to different components or partitions. For example, door handles and door panels are segmented into movable and fixed components; computers are segmented into screen covers and keyboard bases; cabinets include drawers and outer shells; and microwave ovens are also distinguished between the door panel and the body.

[0168] Availability: This column displays a heatmap of each point on every movable part, indicating where it is more feasible for the robot to perform an operation. Brighter (or warmer) colors generally indicate that the point is more suitable for actions such as pushing or pulling. By observation, the system will show high-scoring areas at locations such as door edges, drawer handles, or microwave doors, thus indicating feasible areas for operation.

[0169] Joints: This column displays the system's visualization of joint parameter estimates, including joint type (rotational / prismatic), joint position, joint orientation, and current opening / closing or extending / retracting state. The orientation of the joints is indicated by arrows or axial line segments in the diagram to visually understand the movement of the components. For example, doors and microwave oven doors are rotary joints, the flip-up part of a computer lid can also be considered a rotary joint, while cabinet drawers are often prismatic joints.

[0170] Component segmentation: The Seg-Net module identifies movable components based on the input point cloud data and generates corresponding segmentation labels, outputting the mobility label τ and component category label γ for each point.

[0171] Joint parameter estimation: The Para-Net module estimates the joint type (rotational or prismatic), orientation, position, and current state based on the segmentation results. The output includes the joint type label ρ, the joint orientation vector o, the joint position u, and the current state s.

[0172] Operational feasibility analysis: The Afford-Net module generates the attitude rotation matrix R of the operation point. z The operation score (as) is used to output the operation posture parameters to determine the optimal operation path.

[0173] Step 5: Route Planning and Operation Execution

[0174] Step 5-1, Path Planning;

[0175] Based on the operational feasibility analysis results, a dynamic operation path is generated for the robot's end effector.

[0176] For prism joints, path generation follows the translation path planning formula:

[0177] p(t) = p + t·(l·o), t∈[0,1]

[0178] Where p is the starting point, l is the distance moved, and o is the direction of movement.

[0179] For a rotary joint, the rotation path is calculated using the Rodriguez formula, defined as:

[0180] p(t)=R(o,l·t)(pu)+u,t∈[0,1]

[0181] Where R(o,l·t) represents the rotation matrix about axis o, p is the trajectory point, and u is the position of the rotation joint.

[0182] Step 5-2: Operation execution;

[0183] During execution, compliance control is used to dynamically adjust the attitude of the end effector, reducing the impact of path deviation and parameter errors.

[0184] Step 6: Experimental verification;

[0185] Step 6-1: Simulation experiment;

[0186] A novel method for manipulating articulated objects in a robot was validated on the Sapien simulator. Tests included pushing and pulling operations on various jointed objects to verify the system's success rate in the simulation environment.

[0187] Step 6-2: Real-world testing;

[0188] This study validates the performance of a novel robotic method for manipulating articulated objects in real-world scenarios, including the manipulation of cabinets, drawers, and doors. Through multiple experiments, the success rate of push-pull operations is recorded to evaluate the system's generalization and robustness. Figure 4 'a' refers to the equipment used in real-world experiments. Figure 4 Figures b, c, and d show the experimental results on a door, a microwave oven, and a cabinet, respectively.

[0189] Experiments show that a novel method for manipulating articulated objects in robots achieves success rates of 36.94% and 21.72% in push and pull operations in a simulated environment, respectively. In real-world scenarios, the system successfully manipulates various object types, demonstrating good operational accuracy and adaptability to multi-jointed objects.

[0190] The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, enabling those skilled in the art to better understand and utilize the invention.

Claims

1. A method for a robot to manipulate an articulated object, characterized in that, Includes the following steps: Step S1: Obtain the 3D point cloud data of the target object and standardize the 3D point cloud data so that it can be processed at a uniform scale; Step S2: Divide and encode the point cloud data using the multi-scale feature extraction module to extract global and local features; The multi-scale feature extraction module employs multi-layer far-point sampling and the k-nearest neighbor algorithm to divide the point cloud data into multi-scale segments, representing each layer of the point cloud as a block. The specific formula is as follows: ； in, Indicates the first The center point set obtained by downsampling at each scale For the first Number of center points of the layer The number of neighboring points for each center point. Indicates the first Layered data blocks; Multi-scale features are encoded using a visual transformer, with the input for each scale being: ； in, This indicates that the input point cloud data is segmented using the Mini-PointNet network. Perform preliminary feature extraction. Encode the position of the center point. It is the set of center points at the first scale. It is a point cloud feature matrix processed by a visual transformer, with dimensions of ,in It is the number of center points of the point cloud at this scale. It is the dimension after feature encoding. The dimension is The feature matrix; Step S3: Generate point-level features through the point-level feature extraction module for use in object segmentation and operation planning; Step S4: The component segmentation module uses a segmentation network to segment the object into components, determines the mobility of the components, and estimates the joint type, orientation, position, and state of the movable components in the joint parameter estimation module. Step S5: Further generate operation suggestions and scores for the end effector through the operation feasibility assessment network to determine the optimal operation direction and interaction point; Step S6: Plan the operation trajectory of the end effector through the trajectory planning module to complete the push and pull operation command of the object; The planned operating trajectory of the end effector is generated through interpolation calculation based on joint parameters and target state, including: For prism joints, the motion trajectory is planned according to the translation path, and is represented as follows: ； in, This indicates the time of the end effector or the point of operation at the interpolation parameter. Spatial location, Starting point For the distance traveled, Indicates the direction of movement; For a rotary joint, the rotation path is calculated using the Rodriguez formula, defined as: ； in, Indicates about the axis Rotation matrix of rotation, Starting point This indicates the position of the rotational joint.

2. The method for a robot to manipulate an articulated object according to claim 1, characterized in that, Standardizing 3D point cloud data involves processing the input 3D point cloud data matrix. After normalization, a unified coordinate system is established. The normalization formula is as follows: ； in, This represents the input 3D point cloud data matrix, containing There are points, each point has Three coordinate components, It is the first The coordinates of the points This represents the normalized point cloud data. Indicates a containing N There are 10 points, each with three coordinate components. x,y,z A matrix of ).

3. The method for a robot to manipulate an articulated object according to claim 1, characterized in that, The point-level feature extraction module utilizes multi-scale feature propagation to extract point-level features, aggregating the encoded features from each scale to obtain point-level features, as shown in the formula: ； in, Point-level features Indicates feature connection operation, For the first Features at each scale For feature propagation operation, It is the number of scales.

4. A method for a robot to manipulate an articulated object according to claim 1, characterized in that, Segmentation networks are used to identify movable and immovable parts within an object. For each point, a binary classification is performed to predict whether it belongs to a movable part, and a mobility label is output. and component category labels : ； in, and For the prediction function, The extracted point cloud features are represented by the loss function of the segmentation network, which includes binary cross-entropy loss and multi-class cross-entropy loss. The specific formula is as follows: ； in, This represents the loss of the entire network. It is a binary cross-entropy loss. Represents multi-class cross-entropy loss. and These are the actual labels for mobility and segmentation, respectively.

5. A method for a robot to manipulate an articulated object according to claim 1, characterized in that, The joint parameter estimation module is used to estimate the joint type, orientation, position, and current state, and specifically includes the following steps: Step S3-1, Joint Type Classification: Determine joint type through binary classification. ,in =0 indicates a rotary joint. =1 represents a prism joint, and the classification loss function is: ； in, Represents classification loss, It predicts joint type; Step S3-2, Joint Position Estimation: For rotational joints, estimate the joint position. With real location The projected distance between them is used for path planning, and its loss function is defined as: ； in, Used to measure and estimate the position of rotational joints Compared with the actual joint position In the direction of the actual joint axis Projection difference on is the unit vector in the joint direction; Step S3-3, Joint Orientation Estimation: Calculate the estimated joint orientation value. Compared with the true value The angle between Defined as: ； Step S3-4, Joint State Estimation: Estimating the current state of the joint. use Loss, and the actual state contrast: ； here The regression loss represents the joint state and is used to measure the model's prediction of the current joint state. Compared to the actual joint state The differences between them.

6. A method for a robot to manipulate an articulated object according to claim 1, characterized in that, The operational feasibility analysis module is used to generate an operational feasibility score and the rotation angle of the end effector, specifically including: Step S4-1, Operation posture generation: Generate the operation posture for each operation point. Let be the three-degree-of-freedom rotation matrix in SO(3) space, and the formula for calculating the operation attitude is: ； in, This is a function for generating the operational attitude. Point cloud features It is random Gaussian noise. This refers to the operation type, with a value of 0 indicating a push or 1 indicating a pull. Represents a 3×3 matrix; Step S4-2, Calculation of Operational Feasibility Score: Feasibility score of the operation point Based on binary classification, it is defined as follows: ； in, This indicates the loss in the operational feasibility assessment module. Rating generation function, This is the actual label, with a value of 0 or 1. The characteristics of the current operation point, Let represent a 3×3 rotation matrix in SO(3) space.

7. A system for implementing the method of robot manipulating an articulated object as described in claim 1, characterized in that, include: The multi-scale feature extraction module divides point cloud data into multiple scales and extracts object features at different scales, providing comprehensive feature information for subsequent modules to adapt to diverse object structures and scales. On the normalized point cloud, the farthest point sampling and k-nearest neighbor algorithm are used to divide the point cloud data into multi-scale point blocks. The point-level feature extraction module propagates multi-scale features to each point and integrates information from each scale to generate point-level features for object segmentation and operation planning. The component segmentation module identifies operable components in an object, distinguishes between movable and immovable parts, and generates segmentation information that provides the basis for the joint parameter estimation module to estimate operable components. The joint parameter estimation module provides geometric and kinematic information of the joints by predicting their type, orientation, position, and current state. The operational feasibility analysis module generates the posture and score of the operation point to determine the interaction parameters of the robot's end effector. The dynamic path planning module generates operation paths based on joint parameter estimation and operation feasibility analysis results, meeting the operation requirements of multi-joint objects and generating different paths for different types of joints.

8. A computer device, characterized in that, It includes: a memory and a processor, and a computer program stored in the memory, which, when executed on the processor, implements a method for a robot to manipulate an articulated object as described in any one of claims 1 to 6.

Images