End pose generation method and system based on target consistency constraint

Abstract

This invention relates to the field of robot control technology and proposes a method and system for generating end-effector pose based on target consistency constraints. The method includes: aligning RGB and depth images in a segmented region of a target object to extract the point cloud of the target object; constructing a candidate set based on the target object point cloud, and pre-constraining the range of the grasping center value within the candidate set; generating multiple grasping candidate poses corresponding to each grasping center point based on the point cloud of the target object, and constructing a grasping contact area corresponding to the grasping candidate poses; identifying the proportion of the target object point cloud in the grasping contact area, filtering the grasping candidate poses, and finally selecting the optimal pose. This invention improves the accuracy of robot end-effector pose generation and increases the success rate of robot grasping by using pre-constrained grasping center point selection and post-filtering based on target consistency.

Description

Technical Field

[0001] This invention relates to the field of robot control technology, specifically to a method and system for generating end-effector pose based on target consistency constraints. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] With the increasing prevalence of robots in industrial sorting, warehousing and handling, and home services, robots need to stably grasp designated target objects according to user instructions in complex environments such as occlusion, stacking, and sparse point clouds. Six-degree-of-freedom grasping can improve posture flexibility, but in real-world scenarios, the grasping system must not only be able to grasp, but also grasp accurately and correctly to the target; otherwise, it is easy to miss or grasp the wrong object, affecting the success rate of the task and the reliability of human-computer interaction.

[0004] Existing methods for robotic object grasping typically involve preprocessing RGB (Red-Green-Blue) images using object detection or segmentation, followed by prediction and filtering of grasping candidates on the target region's point cloud. This approach has several drawbacks: First, object detection or segmentation is usually only applied to cropped and filtered point clouds, limiting the input to the grasping network. However, the candidate generation process still relies on the grasping network predicting within a continuous spatial context. When the point cloud is sparse, the target object is occluded, or segmentation errors occur, the grasping center may appear at the edge of the target object or in a location without a point cloud, potentially leading to the system missing or grasping the wrong object. Second, after generating grasping candidates, existing methods typically only filter them based on grasping scores, lacking further assessment of the relationship between the robotic arm's gripper and the target object during the grasping process. In scenarios with stacked objects and severe occlusion, the highest-scoring pose generated by the grasping network may result in the gripper's closed area landing on a non-target object, potentially leading to a grasped object that differs from the user-specified object. Summary of the Invention

[0005] To address the aforementioned problems, this invention proposes an end-effector pose generation method and system based on target consistency constraints. By applying pre-constraints to the gripping center before generating gripping candidates and post-constraints based on the relationship between the gripper contact area and the target object during the screening stage after gripping candidate generation, the target object of the final gripping action is kept consistent with the target object specified by the user, thereby improving the accuracy of robot end-effector pose generation and increasing the robot end-effector gripping success rate.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] The first aspect of this invention provides a method for generating end pose based on target consistency constraints, comprising the following steps:

[0008] Acquire the RGB and depth images of the target scene, and process the RGB images to obtain the segmented regions of the target objects;

[0009] In the segmented region of the target object, the RGB image and the depth image are aligned, and the region of the target object is selected from the aligned target scene point cloud to extract the point cloud of the target object.

[0010] A candidate set is constructed based on the point cloud of the target object, and the range of values ​​of the grab center is limited to the candidate set as a pre-constraint for grabbing candidates.

[0011] Based on each grasping center point selected by the pre-constraint, and combined with the point cloud of the target object, multiple grasping candidate poses corresponding to the grasping center points are generated, and a grasping contact area corresponding to the grasping candidate poses is constructed.

[0012] Identify the proportion of target object point cloud in the grasping contact area, construct post-constraints for target consistency in grasping posture, filter candidate grasping postures, and score the filtered candidate grasping postures to select the optimal posture.

[0013] A second aspect of the present invention provides an end-effector pose generation system based on target consistency constraints, comprising:

[0014] The perception module is configured to acquire RGB and depth images of the target scene, and process the RGB images to obtain segmented regions of the target object.

[0015] The target point cloud extraction module is configured to align the RGB image and depth image in the segmented region of the target object, filter out the region of the target object from the aligned target scene point cloud, and extract the point cloud of the target object.

[0016] The pre-constraint module is configured to construct a candidate set based on the point cloud of the target object, and limit the value range of the grasping center to within the candidate set, serving as a pre-constraint for the grasping candidates;

[0017] The grasping candidate generation module is configured to generate multiple grasping candidate poses corresponding to each grasping center point selected based on the pre-constraints, combined with the point cloud of the target object, and construct the grasping contact area corresponding to the grasping candidate poses.

[0018] The posture selection module is configured to identify the proportion of the target object point cloud in the grasping contact area, construct a post-constraint for the consistency of the grasping posture, filter the grasping candidate postures, and score the filtered grasping candidate postures to select the optimal posture.

[0019] A third aspect of the present invention provides an end-effector pose generation system based on target consistency constraints, comprising:

[0020] Image acquisition device, grasping execution terminal, and processor;

[0021] The processor is configured to execute the steps of the above-described end-effector pose generation method based on target consistency constraints, generating a grasping pose to control the grasping execution end to perform a grasping action.

[0022] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0023] This invention introduces a pre-constraint on the grasping center during the grasping candidate generation stage, strictly limiting the candidate set of grasping centers to the effective range of the target object's point cloud. This makes it less likely for the grasping center to fall on the edge of the target object, the transition region between the target and non-target objects, or a spatial location without point cloud. This enhances the spatial correlation between grasping candidates and the target object from the source, reducing the risk of grasping the wrong object or missing the target due to grasping center offset when the point cloud is sparse, the target is occluded, or there are errors in target segmentation. This improves the consistency and reliability of grasping the specified target object.

[0024] This invention introduces a post-consistent constraint on the target consistency of grasping postures during the candidate grasping stage. By analyzing the spatial relationship between the grasping contact area and the target point cloud, and calculating the proportion of the target point cloud within the grasping contact area or the distribution relationship between target and non-target point clouds, it can effectively eliminate candidate grasping postures that may contact, interfere with, or clamp non-target objects within the gripper's closed area before executing the grasping action. This overcomes the problem of insufficient judgment on the "gripper-target object" interaction relationship when relying solely on grasping scores for screening. Especially in scenarios with multiple adjacent, occluded, or stacked objects, it can significantly reduce the probability of grasping high-scoring postures but actually clamping non-target objects, improving the matching accuracy between the grasping action and the user-specified target object, and increasing the overall grasping success rate.

[0025] The advantages of the present invention, as well as its additional advantages, will be described in detail in the following specific embodiments. Attached Figure Description

[0026] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute a limitation thereof.

[0027] Figure 1 This is a flowchart of the end pose generation method based on target consistency constraints according to Embodiment 1 of the present invention;

[0028] Figure 2This is a diagram illustrating the overall architecture of the end-effector pose generation method in Embodiment 1 of the present invention.

[0029] Figure 3 This is a schematic diagram of the construction process of the pre-constraint for the grasping center in Embodiment 1 of the present invention;

[0030] Figure 4 This is a schematic diagram illustrating the consistency between the grasping contact area and the target in Embodiment 1 of the present invention;

[0031] Figure 5 This is a schematic diagram of multi-scale cylindrical local feature modeling in Embodiment 1 of the present invention;

[0032] Figure 6 This is a schematic diagram of explicit geometric feature enhancement in Embodiment 1 of the present invention;

[0033] Figure 7 This is a structural block diagram of the end pose generation system based on target consistency constraints according to Embodiment 2 of the present invention. Detailed Implementation

[0034] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0035] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0036] It should be noted that the terminology used herein is for describing particular embodiments only and is not intended to limit the exemplary embodiments of the present invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof. It should be noted that, without conflict, the various embodiments and features within those embodiments can be combined with each other. The embodiments will now be described in detail with reference to the accompanying drawings.

[0037] Example 1

[0038] In one or more of the technical solutions disclosed in the embodiments, such as Figures 1 to 6 As shown, the end-effector pose generation method based on target consistency constraints includes the following steps:

[0039] Step 1: Obtain the RGB image and depth image of the target scene, and process the RGB image to obtain the segmented region of the target object;

[0040] Step 2: In the segmented region of the target object, align the RGB image and the depth image, filter out the region of the target object from the aligned target scene point cloud, and extract the point cloud of the target object.

[0041] Step 3: Construct a candidate set based on the point cloud of the target object, and limit the value range of the grab center to within the candidate set as a prerequisite constraint for grabbing candidates;

[0042] Step 4: Based on each grasping center point selected by the pre-constraints, and combined with the point cloud of the target object, generate multiple grasping candidate poses corresponding to the grasping center points, and construct the grasping contact area corresponding to the grasping candidate poses.

[0043] Step 5: Identify the proportion of the target object point cloud in the grasping contact area, construct a post-constraint for the target consistency of the grasping posture, filter the grasping candidate postures, and score the filtered grasping candidate postures to select the optimal posture.

[0044] In the above implementation, by introducing a pre-constraint on the grasping center during the grasping candidate generation stage, the candidate set of grasping centers is strictly limited to the effective range of the target object's point cloud. This makes it less likely for the grasping center to fall on the edge of the target object, the transition region between the target and non-target objects, or a spatial location without point cloud. This enhances the spatial correlation between the grasping candidates and the target object from the source, reducing the risk of grasping the wrong object or missing the target due to grasping center offset when the point cloud is sparse, the target is occluded, or there are errors in target segmentation. This improves the consistency and reliability of grasping the specified target object.

[0045] In the candidate grasping stage, a post-constraint on target consistency of grasping posture is introduced. By analyzing the spatial relationship between the grasping contact area and the target point cloud, the proportion of the target point cloud or the distribution relationship of the target / non-target point cloud within the grasping contact area is calculated. Before executing the grasping action, candidate grasping postures that may contact, interfere with, or clamp non-target objects within the gripper's closed area can be effectively eliminated. This can compensate for the insufficient judgment of the interaction relationship between the gripper and the target object when relying solely on grasping scores for screening. Especially in scenarios with multiple adjacent, occluded, or stacked objects, it can significantly reduce the probability of grasping high-scoring postures but actually clamping non-target objects, improve the matching accuracy between the grasping action and the user-specified target object, and increase the overall grasping success rate.

[0046] In step 1, the object to which the end-effector pose generation in this embodiment is applicable can be a robotic arm or an intelligent agent such as a robot;

[0047] In step 1, image data can be acquired by using depth vision sensor devices such as depth cameras to obtain RGB images and depth images of the scene;

[0048] In step 1, the acquired RGB image and depth image are processed, including:

[0049] Step 11: Spatially align the RGB image and the depth image in the same coordinate system to ensure that the two-dimensional information and the three-dimensional information correspond.

[0050] Step 12: Obtain the user input instructions, perform target detection and segmentation on the target object in the capture scene based on the RGB image, and determine the target region of the target object in the RGB image;

[0051] Specifically, the target region of a target object can be obtained by performing target detection on the target object to generate a detection box, and then performing target segmentation on the detection box to obtain a pixel-level segmented region.

[0052] In step 2, after determining the target object region, based on the position of the target object region in the RGB image, the 3D data belonging to the target region is filtered out from the depth image or scene point cloud aligned with the RGB image, thus obtaining the target point cloud corresponding to the target object.

[0053] Combination Figure 3 A candidate set is constructed based on the point cloud of the target object. Specifically, after obtaining the point cloud of the target object, the point cloud of the target object is marked as 1, and the remaining point cloud of the current scene is marked as 0. A target object point cloud set is created, that is, the point cloud set marked as 1. The created set is used as the candidate set of the grab center, and the value range of the grab center is limited.

[0054] Furthermore, the capture center is obtained by selecting or sampling three-dimensional points from the target point cloud, and the value range of the capture center is limited to the target point cloud or its spatial neighborhood;

[0055] The neighborhood typically refers to a very small spatial region surrounding the target point cloud. The neighborhood range is a spatially extended region constructed based on the target object's point cloud. For potential gripping locations close to the target object's surface, the candidate geometry can include these locations. The neighborhood range can be obtained by extending the target object's point cloud by distance, for example, constructing a neighborhood region within a preset distance threshold range centered on each point in the target point cloud; or it can be determined through methods such as K-nearest neighbor search. The distance threshold can be set according to the gripper size or gripping scale.

[0056] In the above embodiments, the spatial location of the grasping center is strictly limited to the target point cloud itself or its close neighborhood, so that the grasping center is limited to the three-dimensional space of the target object as much as possible, which avoids the center falling on the edge of the target or in a region without point cloud, and also prevents the center from deviating from the target due to sparse point cloud or segmentation error.

[0057] In step 3, before generating the candidate grasping poses, several 3D points are selected from the candidate point cloud as grasping center points, and these center points are used as the geometric references for generating the grasping candidates. Specifically:

[0058] Step 31: For each point in the candidate point cloud, predict whether it is a target object (objectness) and its grabbability score (graspness), and construct a candidate set of grab centers by combining the target mask (target_mask). The formula represents:

[0059] ;

[0060] in: This represents the set of candidate centers for capture; Represents the first point in the scene point cloud Points; objectness ) represents a point Is it a prediction result for the target object point? (graspness) ) represents a point Scrabability score; This is the crawlability threshold; Point The point cloud region corresponding to the target object.

[0061] Through the above screening, only points that belong to the target object and have a high feasibility of being grasped are retained as the set of candidate grasping centers.

[0062] Step 32: In the crawl center candidate set Within this process, Farthest Point Sampling (FPS) is performed, selecting a set number of points as the set of capture center points. It is represented as:

[0063] ;

[0064] Optional, 1024 is a good choice;

[0065] in: This represents the set of crawl center points, which is the candidate set of crawl centers. The set of points selected by sampling from the farthest point; Indicates the first One capture center point, and ; This is the preset number of sampling points.

[0066] In this step, sampling at the farthest point ensures that the distribution of the grab center point in space is relatively uniform, thereby improving the coverage of the grab candidate pose generation.

[0067] Step 33: For each gripping center point, determine the direction in which the gripper approaches the gripping center point, predict the score corresponding to different approach directions, and select the viewpoint with the highest score as the optimal approach direction for that gripping center point; construct a local geometric reference coordinate system using the optimal approach direction as the main direction of the local coordinate system.

[0068] Specifically, the optimal approach direction is taken as the principal direction of the local coordinate system, and two directions orthogonal to it are constructed in combination with the principal direction. Finally, a local geometric reference coordinate system at the center point is constructed from the three orthogonal direction vectors, and this coordinate system is represented as a rotation matrix:

[0069] ;

[0070] in: To capture the center point The rotation matrix, consisting of three orthogonal unit vectors arranged in columns, is used to represent the directional relationship between the local geometric reference coordinate system and the target scene point cloud coordinate system; This indicates the direction vector to be captured; , This represents the unit vector of the coordinate axis direction that is orthogonal to the grabbing direction.

[0071] In the above process, the target scene point cloud is defined in a unified three-dimensional coordinate system, and the subsequent grasping center point, rotation matrix and gripper geometry model are all represented and calculated in this coordinate system.

[0072] Step 34: Grab the center point With rotation matrix As a geometric reference for generating grabbing candidates, the local coordinate system is aligned with the grabbing direction, and local neighborhood construction, feature extraction, and grabbing posture parameter prediction are performed in this coordinate system.

[0073] This embodiment, through the above implementation method, achieves the effect of grasping the center point. From only the candidate set The point cloud of the target object is sampled, thus limiting the grab center to the target object's point cloud range. This avoids the grab center appearing at the edge of the target object, in the transition area between the target and non-target objects, or in empty areas without point clouds, thereby improving the stability and accuracy of grab pose generation.

[0074] In step 4 of this embodiment, under the pre-constraint of the grasping center, for each grasping center point, combined with the point cloud information of the target object, multiple grasping candidate poses corresponding to the grasping center point, such as six-degree-of-freedom grasping candidate poses, are generated through the grasping network. Multi-scale cylindrical local feature modeling and explicit geometric feature enhancement are introduced into the grasping network to improve the ability of the grasping candidate generation stage to express the local geometric structure of the target object, thereby improving the stability and robustness of the grasping candidate generation.

[0075] As a further technical solution, the grabbing candidate poses are generated, and the grabbing network is improved as follows:

[0076] The multi-scale feature modeling approach can construct multiple local spatial neighborhoods of different scales in the target point cloud of the grasping center when generating grasping candidate poses, and perform feature extraction and feature fusion on the point cloud in each scale neighborhood to enhance the network's ability to grasp objects of different scales.

[0077] The explicit geometric feature enhancement method calculates the explicit geometric features of the target point cloud, including normals, curvature, and density, during the modeling process in the local spatial neighborhood at each scale. The explicit geometric features are then fused with the original features to enhance the network's ability to grasp objects with different geometric shapes.

[0078] Step 4, which generates multiple candidate grasping poses corresponding to each grasping center point selected based on pre-constraints and combined with the point cloud of the target object, includes the following steps:

[0079] Step 41: Multi-scale cylinder local feature modeling: For each grasping center point, construct nested cylinder neighborhoods at multiple scales and sample point cloud data. After extracting features, fuse them to obtain the multi-scale fused features corresponding to each grasping center point. ;in, Indicates the first The feature number corresponding to each capture center point Indicates the first A multi-scale fusion feature vector that captures the center point.

[0080] Furthermore, in step 41, multi-scale fusion features The generation process includes:

[0081] Step 411: Construct an alignment coordinate system: For each grab center point Using the rotation matrix corresponding to the candidate pose Establish a local alignment coordinate system, using the grabbing approach direction as the axis of the coordinate system, so that subsequent neighborhood modeling is consistent with the grabbing direction.

[0082] Step 412: Construct a multi-scale nested cylindrical neighborhood: using the center point as the starting point. Centered on a coordinate system, construct multiple cylindrical neighborhoods with different radii;

[0083] Step 413, Neighborhood Sampling at Each Scale: Sampling of the point cloud within the neighborhood of each cylinder;

[0084] Step 414, Shared Feature Extraction and Aggregation: Input the coordinates of the sampling points into a shared multilayer perceptron (MLP), and after obtaining the point features, pool and aggregate them to obtain local features of a uniform dimension (e.g., 256 dimensions) at each scale;

[0085] Step 415, Lightweight Scale Attention Fusion: Weights are calculated for local features at different scales based on single-head attention, and then weighted and fused to obtain the corresponding current capture center point. The multi-scale fusion features; the output of this step still maintains the same dimension (such as 256 dimensions), avoiding dimensional expansion caused by direct splicing, thereby maintaining the network economy.

[0086] Traditional grasping models typically use a fixed-scale cylinder with a fixed radius, such as 5cm, to model the point cloud at the grasping center. However, when dealing with small objects, an excessively large cylinder radius introduces a large number of background points and other object point clouds, making it difficult to capture the point cloud features of the target object. Furthermore, a single-scale cylinder cannot simultaneously capture both local details and global features when facing a target object. In this step, by constructing a multi-scale nested cylinder neighborhood for the same grasping center and adaptively fusing it with scale attention, the model can simultaneously acquire small-scale local details and large-scale structural context. This reduces the risk of background noise and feature overload introduced by the fixed large radius, thereby improving the accuracy of representing the target's local geometry during the grasping candidate generation stage, enhancing the stability of candidate pose prediction, and improving robustness to scale changes, occlusion, and point cloud sparsity, while maintaining the output feature dimension unchanged to control computational and parameter costs.

[0087] In one specific implementation, this embodiment introduces multi-scale cylindrical local feature modeling into the grasping model, and the structure can be as follows: Figure 5 As shown, step 41 is illustrated by constructing four nested cylinders of different sizes.

[0088] The multi-scale cylinder local feature module constructs four nested cylinders of different scales with radii of 1.25cm, 2.5cm, 3.75cm, and 5cm, corresponding to 0.25 times, 0.5 times, 0.75 times, and 1 times the original radius, respectively. Excessively large scales may introduce too much background noise, and high-resolution small-scale features are needed for fine-grained tasks like grasping.

[0089] For each grab center point An alignment coordinate system is constructed based on the grasping direction predicted by the grasping network. Within this coordinate system, four cylinders with different radii are defined. A fixed number of samples is obtained within each cylinder using cylinder queries: 1.25cm corresponds to 16 points, 2.5cm to 24 points, 3.75cm to 24 points, and 5cm to 32 points. The point cloud coordinates are then input into a shared MLP (Multi-Level Processing), which extracts the features of each point. These features are then pooled into four 256-dimensional local features.

[0090] The local features extracted at each scale can be represented as:

[0091] ;

[0092] in: Let be the radius of the neighborhood of the cylinder. To correspond to the number of sampling points in the neighborhood of the cylinder, 256 is the unified feature dimension of nested cylinder neighborhoods of different scales.

[0093] For the The feature vector of a sampling point can be represented as:

[0094] ;

[0095] For the feature vector at each scale, a shared two-layer perceptron is used to extract features from the feature vector of each point, thus obtaining the point features. The process is represented as:

[0096] ;

[0097] in: This represents the first-layer weight matrix, used for linear transformation and feature compression of the input features, mapping the 256-dimensional features to a smaller space. It is the first layer of MLP, and its function is to extract information useful for judging the importance of the scale from the features at the current scale.

[0098] For ReLU activation function: The role of activation functions is to introduce non-linearity, enabling the network to express the importance of certain feature combinations.

[0099] This represents the second-layer weight matrix, used to map feature vectors to a scalar score. This score is used to represent the importance of features at the current scale and is used for weight calculation in subsequent multi-scale feature fusion.

[0100] To reduce the number of model parameters and computational complexity, direct concatenation is avoided here to prevent the dimensionality from quadrupling. In this embodiment, a lightweight single-head scale attention algorithm is used to calculate the weights, as follows:

[0101] For the same sampling point Point features are normalized using softmax across all scales:

[0102] ;

[0103] in: The number of scales is 4 in this embodiment.

[0104] The purpose of softmax normalization is to allow the network to dynamically assign scale weights to each crawling center, rather than to determine which scales are more important. For the first The sampling point at the th sampling point The weights on each scale represent the importance of that scale to the current crawling center.

[0105] Finally, the 256-dimensional features across the four scales are weighted according to... Weighting is performed to obtain each sampling point The 256-dimensional multi-scale fusion feature achieves effective integration of multi-scale features while keeping the crawling network interface and parameter scale unchanged.

[0106] For the The multi-scale fusion feature of each sampling point is represented as follows:

[0107] ;

[0108] in: Indicates the first Features obtained by multi-scale fusion of sampling points Indicates the number of scales; Indicates the first The sampling point at the th sampling point Weights under each scale; Indicates the first The sampling point at the th sampling point Eigenvectors at various scales.

[0109] Based on this, the fusion features of all sampling points are aggregated to obtain the current capture center point. The corresponding final multi-scale fusion features :

[0110] ;

[0111] in: This indicates a pooling operation, used to aggregate features from multiple sampling points; Indicates the first Multi-scale fusion features of each sampling point; This indicates the number of sampling points at the corresponding scale.

[0112] Step 42, Explicit Geometric Feature Enhancement: Geometric features based on the normal, curvature, and density of the central neighborhood points are obtained through cloud computing and fused with multi-scale features extracted from the network. The enhanced features are obtained by splicing them together; the process of generating the enhanced features includes the following steps 421 to 423.

[0113] While multi-scale cylindrical local feature modeling can perceive the spatial structure of objects at different scales, like other point cloud grasping methods, the model primarily relies on geometric information learned from the point cloud. This makes it difficult for the model to accurately perceive key grasping areas such as object edges and surface variations. Point clouds often contain noise and are frequently occluded, potentially resulting in missing parts of the point cloud. The geometric relationship between the 3D point cloud and the real object surface cannot be fully recovered using pure coordinates, leading to unstable geometric feature learning by the model. Therefore, explicit geometric feature enhancement is introduced into the grasping model, with structures such as... Figure 6 As shown.

[0114] The explicit geometric feature enhancement module introduces explicit geometric features such as normals, curvature, and density to construct a lightweight geometric enhancement module. This enhances the expressive power of local geometric structures in point clouds and improves the accuracy of model pose prediction. Specific methods include the following:

[0115] Step 421: Construct a local neighborhood point set: In the nested cylindrical neighborhood at each scale, capture the center point. and The points obtained by K-Nearest Neighbors (KNN) are used to construct a neighborhood point set. And calculate the mean of the neighborhood points. :

[0116] ;

[0117] in, Represents the neighborhood point set The point in, and ; Indicates the number of neighboring points. This represents the mean of the neighborhood point set;

[0118] Step 422: Calculate explicit geometric features: Based on the mean of the neighborhood points and the points in the neighborhood, construct a local covariance matrix, decompose the covariance matrix features of the neighborhood point set, and obtain the normal features, curvature features and density features.

[0119] Constructing the local covariance matrix ,as follows:

[0120] ;

[0121] For covariance matrix Eigenvalue decomposition is performed. Since the covariance matrix is ​​calculated from the 3D point cloud coordinates, the decomposition yields three eigenvalues. These three eigenvalues ​​are then sorted in ascending order. Among them, the smallest eigenvalue The direction of the corresponding unit eigenvector is the direction of minimum diffusion in the neighborhood point cloud, which serves as the normal vector of the target object. ;

[0122] For a planar region, the point cloud has a large distribution in both directions, so For edge regions, the point cloud is distributed very large in only one direction, so .

[0123] Curvature features are used to characterize the degree of edge or abrupt change. Curvature features are calculated using eigenvalues. :

[0124] ;

[0125] For planar regions, Extremely small, therefore curvature ≈0, for mutation or corner regions, , , Both are relatively large, so the curvature is... Move closer to 1.

[0126] Density characteristics The following formula is used to characterize the sparsity and reliability of point clouds:

[0127] ;

[0128] in, , Represents the radius of the neighborhood of the cylinder at the corresponding scale;

[0129] For regions with sparse point clouds or regions with abrupt edge changes + + The value will increase, leading to It will also increase, so the corresponding density feature It would also be too large to be suitable as a focal point.

[0130] The extracted geometric features are concatenated to construct a geometric feature vector. It is represented as:

[0131] ;

[0132] in, Indicates the corresponding center point of the capture. Geometric eigenvectors, Represents the normal eigenvector. Indicates curvature characteristics. This indicates density characteristics.

[0133] Step 423, Feature Fusion and Dimension Alignment: Combine geometric features with the multi-scale fused features output from Step 41. The concatenation is performed and mapped back to the original dimension (e.g., 256 dimensions) through a linear layer to obtain the enhanced features.

[0134] Specifically, such as Figure 6 As shown, in this embodiment, the 3D normal feature, 1D curvature feature, and 1D density feature corresponding to the current grasping center are concatenated to form a 5D feature vector, namely the geometric feature vector. Then, it is concatenated with the 256-dimensional features of the grasping network to form a 261-dimensional feature vector. Finally, a linear mapping is used to map it back to 256 dimensions, aligning it with the previous network. This introduces geometric feature enhancement without changing the feature dimensions, which aligns with the economical nature of networks.

[0135] Furthermore, during training, the feature fusion and alignment dimensions are initialized using an approximate identity, as shown in the formula:

[0136]

[0137] in: The weight matrix represents the linear mapping. Indicates the current center point of the capture. The corresponding enhanced features, Indicates the corresponding center point of the capture. Geometric eigenvectors, This represents the concatenation of multi-scale fused features and geometric features; Indicates the bias term;

[0138] During the initialization phase, the weight matrix... and bias terms Set to:

[0139] ;

[0140] express The identity matrix, This represents the zero matrix, where the weights of the corresponding geometric features are initialized to 0.

[0141] In the initial state, geometric features It has no effect on the output:

[0142] ;

[0143] At the start of training, the explicit geometry augmentation module does not alter the behavior of the original model or disrupt the learned multi-scale features. As training iterates, the weights of the geometric features are dynamically adjusted to gradually enhance their effectiveness, ultimately achieving a stable perception of the target's geometric structure.

[0144] In step 42 of this embodiment, explicit geometric feature enhancement is introduced. Based on the geometric quantities such as normal, curvature and density of the point cloud in the grasping center neighborhood, it is fused with the network learning features. This enables the grasping model to stably perceive the orientation changes, edge abrupt changes and sparse and unreliable regions of the target surface even under point cloud noise, missing and occluded conditions. This improves the ability to identify key graspable regions, reduces the probability of grasping points falling on edge holes or low-confidence regions, and thus improves the accuracy and consistency of six-degree-of-freedom grasping candidate pose prediction. At the same time, by mapping the fused features back to the original dimension and using approximate identity initialization, the new module can enhance the geometric perception capability without significantly increasing interface complexity and parameter overhead and without destroying the initial behavior of the original model.

[0145] Step 43: Generate candidate grasping poses: For each grasping center point, generate multiple candidate grasping poses based on the enhanced features;

[0146] The six-DOF grasping candidate poses include grasping position parameters and grasping pose parameters, which can represent the actual grasping pose of the gripper in three-dimensional space. Multiple grasping candidate poses are generated around the grasping center point, providing a basis for subsequent selection. Specifically, the grasping candidate poses can include: grasping approach direction, rotation angle around the approach direction, grasping depth along the approach direction, gripper opening width, and grasping quality score.

[0147] In this embodiment, step 4 proposes multi-scale cylindrical local feature modeling. By fusing features from cylinders of multiple scales, the model's ability to perceive objects of different scales is enhanced. Then, a geometric feature enhancement module is implemented, fusing normal, curvature, and density geometric features based on covariance decomposition to enhance the grasping network's ability to represent complex geometric structures of objects. After completing multi-scale feature fusion and geometric feature enhancement, the enhanced local features are fed into the grasping head network to regress grasping posture parameters, evaluate and rank the grasping quality, and finally output an executable grasping pose.

[0148] Step 4 involves constructing a gripping contact area corresponding to each candidate gripping posture based on the gripper's structural parameters, dimensional information, and generated candidate gripping posture parameters. This process includes the following steps:

[0149] Step 401: Determine the position and orientation of the gripper in three-dimensional space based on the gripping candidate posture parameters; the gripping candidate posture parameters include the gripping center point. Position and rotation matrix And the gripper width parameter.

[0150] Capture the center point The position is used to determine the gripper's grasping position in space; rotation matrix The gripper width parameter is used to determine the spatial orientation of the gripper and to determine the opening range of the gripper during the closing process.

[0151] Step 402: Establish a geometric model of the gripper based on its structural parameters, including the length, width, and thickness of the gripper.

[0152] In one embodiment, the gripper geometry is modeled as a three-dimensional structure, such as a cuboid, cube, prism, or pyramid; the spatial dimensions of the three-dimensional structure are determined based on the structural parameters of the gripper using this three-dimensional structure.

[0153] Step 403: Based on the grasping candidate posture parameters, perform spatial transformation on the gripper geometry model, including transformation based on the grasping center point. Translation transformation and rotation matrix based The rotational transformation aligns the gripper geometry with the candidate gripping pose in three-dimensional space.

[0154] Step 404: Based on the position and orientation of the aligned gripper geometry model, within the gripper's closed area, determine the spatial region where the gripper may contact the object, as the gripping contact area. The gripping contact area represents the spatial range within which the gripper may contact the object during the gripping process, and is used for subsequent target consistency determination.

[0155] In the above embodiments, the grasping contact area can represent the spatial range in which the gripper may come into contact with the object during the grasping process, providing a basis for subsequent target consistency determination.

[0156] Step 5: Identify the proportion of the target object point cloud in the grasping contact area, construct a post-constraint for target consistency in the grasping posture, filter the candidate grasping postures, and score the filtered candidate grasping postures to select the optimal posture; combined with Figure 4 After constructing the grasping contact area, the target consistency of the grasping candidate poses is determined based on the spatial relationship between the grasping contact area and the target point cloud.

[0157] The post-constraint for the consistency of the grasping posture is as follows: a ratio threshold is set, and the proportion of the point cloud of the target object in the grasping contact area exceeds the set ratio threshold as a post-constraint for filtering grasping candidate postures.

[0158] Specifically, the proportion of the target object point cloud within the grasping contact area is statistically analyzed. Step 3 previously marked the point clouds, and the set of point clouds marked as 1 is used as the candidate set for grasping centers. For each grasping candidate posture, its corresponding gripper contact area is determined, and the ratio of the number of point clouds belonging to the target object within that contact area to the total number of point clouds within that area is calculated. When the proportion of the target object point cloud within the grasping contact area meets a certain threshold—that is, when the statistically calculated proportion of point clouds marked as 1 meets the set ratio threshold—the corresponding grasping candidate posture is determined to satisfy the target consistency constraint; grasping candidate postures that do not meet the threshold are discarded.

[0159] In this embodiment, the target consistency determination step can effectively exclude grasping candidate postures that are inconsistent with the specified target object and grasping candidate postures that may come into contact with non-target objects before grasping is executed.

[0160] A further technical solution involves ranking the candidate grasping postures that meet the target consistency constraint, taking into account grasping quality evaluation and collision detection, and selecting the optimal grasping posture for the target object grasping task. Finally, the robotic arm and end effector are controlled to perform the grasping operation according to the selected target grasping posture, thereby completing the six-degree-of-freedom grasping of the specified target object.

[0161] To illustrate the effectiveness of the pose generation method in this embodiment, an experimental comparison was conducted, and the details are as follows;

[0162] Experiments were conducted using the large-scale 6-DoF grasping detection dataset GraspNet-1 Billion. This dataset contains 97,280 images and over 1.1 billion grasping poses from 190 cluttered real-world scenes captured by two popular RGB-D (Red-Green-Blue-Depth) cameras, RealSense and Kinect. The experiments followed the official training and testing split, with the first 100 scenes used for training and the remaining 90 scenes for testing. The test set was divided into three categories based on object features: seen objects, similar but unseen objects, and novel objects, with 30 scenes in each category.

[0163] Tables 1 and 2 show the comparison results of the method in this embodiment (hereinafter referred to as MSGeoGrasp) with existing representative models on the GraspNet-1 Billion dataset under RealSense and Kinect camera settings, respectively, with the best results marked in bold. MSGeoGrasp achieves a high level in both camera settings, with particularly outstanding performance under the RealSense camera, outperforming other methods in the Seen, Similar, and Novel scenarios;

[0164] Table 1 shows the performance comparison under RealSense camera settings;

[0165]

[0166] Table 2 shows the performance comparison under Kinect camera settings;

[0167]

[0168] In Tables 1 and 2, CD (Collision Detection) represents collision detection, used to determine whether the generated grab pose collides with an object or the environment. The best results without using collision detection (CD) are highlighted in bold; the explanations of existing methods or models are as follows:

[0169] GPD: A grasping pose detection method based on point cloud geometric features, which generates grasping candidate poses and uses a convolutional neural network to evaluate grasping quality.

[0170] PointNetGPD: A grasping posture detection method based on the PointNet point cloud feature extraction network, which evaluates grasping quality by learning local geometric features of the point cloud.

[0171] GraspNet: An end-to-end six-DOF grasping pose detection network model based on deep learning.

[0172] TransGrasp: A grasping posture detection method based on the Transformer structure, which improves grasping detection performance by modeling long-distance feature relationships between point clouds.

[0173] GraNet: A grasping pose generation method based on graph neural networks, which learns point cloud spatial features by constructing a multi-layer graph structure.

[0174] TSB: A grasping detection method based on multi-scale feature learning, which enhances local geometric representation through multi-scale cylinder feature extraction.

[0175] GSNet: A crawl detection network based on crawlability assessment to filter high-quality crawl regions.

[0176] EconomicGrasp: A six-degree-of-freedom grasping detection method based on economic supervision strategies to reduce model training resource consumption.

[0177] The results in Tables 1 and 2 show that, under the RealSense camera settings, MSGeoGrasp improves the average accuracy (AP) by 2.67% compared to EconomicGrasp, with the most significant improvement of 4.72% in the Seen scene; and improvements of 1.67% and 1.61% in the Similar and Novel scenes, respectively. This indicates that the proposed multi-scale cylinder local feature modeling and geometric feature enhancement module can effectively improve grasping detection performance under high-quality point cloud conditions.

[0178] With the Kinect camera setup, due to higher point cloud noise and sparsity, the overall performance of all methods decreased compared to the RealSense camera setup. MSGeoGrasp improved the average AP by 0.16% compared to EconomicGrasp, achieving a 2.10% improvement in the Seen scene; in the Similar and Novel scenes, performance decreased slightly due to the impact of point cloud quality.

[0179] The results show that under low-quality point cloud conditions, the performance of multi-scale cylinder local feature modeling and geometric feature enhancement superposition is affected to some extent, but MSGeoGrasp can maintain competitive performance compared with existing models in most scenarios.

[0180] Example 2

[0181] Based on Example 1, this embodiment provides an end-effector pose generation system based on target consistency constraints, such as... Figure 7 As shown, it includes:

[0182] The target point cloud extraction module is configured to align the RGB image and depth image in the segmented region of the target object, filter out the region of the target object from the aligned target scene point cloud, and extract the point cloud of the target object.

[0183] The pre-constraint module is configured to construct a candidate set based on the point cloud of the target object, and limit the value range of the grasping center to within the candidate set, serving as a pre-constraint for the grasping candidates;

[0184] The grasping candidate generation module is configured to generate multiple grasping candidate poses corresponding to each grasping center point selected based on the pre-constraints, combined with the point cloud of the target object, and construct the grasping contact area corresponding to the grasping candidate poses.

[0185] The posture selection module is configured to identify the proportion of the target object point cloud in the grasping contact area, construct a post-constraint for the consistency of the grasping posture, filter the grasping candidate postures, and score the filtered grasping candidate postures to select the optimal posture.

[0186] Furthermore, the candidate generation module includes:

[0187] The multi-scale feature modeling module is configured to construct nested cylindrical neighborhoods of multiple scales for each grasping center point, sample point cloud data, extract features, and then fuse them to obtain the multi-scale fused features corresponding to each grasping center point. Specifically, it is configured to perform step 41 in embodiment 1;

[0188] An explicit geometric feature enhancement module is configured to compute geometric features based on the normal, curvature, and density of the center neighborhood points, and fuse them with multi-scale features extracted from the network. The enhanced features are then spliced ​​together; specifically, step 42 in Example 1 is executed.

[0189] The generation module is configured to generate multiple grasping candidate poses based on the enhanced features for each grasping center point.

[0190] It should be noted that each module in this embodiment corresponds one-to-one with each step in embodiment 1, and their specific implementation process is the same, so it will not be repeated here.

[0191] Example 3

[0192] Based on Example 1, this embodiment provides an end-effector pose generation system based on target consistency constraints, including:

[0193] Image acquisition device, used to acquire RGB and depth images of a target scene;

[0194] The end of the capture process is used to perform the capture action;

[0195] The processor is used to execute the steps of the end pose generation method based on target consistency constraints described in Example 1, generate a grasping pose and control the grasping execution end to complete the grasping of the target object.

[0196] In the above embodiments, the image acquisition device can be an RGB-D camera, such as a structured light RGB-D camera, a ToF RGB-D camera, or an active binocular depth camera.

[0197] The RGB-D camera can be mounted in any of the following locations:

[0198] The camera is fixed near the end of the robotic arm or on the side of the gripper for tracking and observing the target.

[0199] The camera is fixed on a bracket, gantry, or robot workstation frame above the workbench for viewing overhead / oblique scenes.

[0200] In the above embodiments, the grasping end effector can be part of an industrial robot system, which includes a robotic arm body, a robot control cabinet, and an end effector.

[0201] Capturing the execution end can specifically be as follows:

[0202] Two-finger parallel grippers, such as electric parallel grippers or pneumatic parallel grippers, are used to grasp regular or semi-regular objects;

[0203] Adaptive grippers, such as three-finger adaptive grippers or flexible grippers, are used for objects with irregular shapes or fragile surfaces;

[0204] The suction end, such as a vacuum suction cup or a vacuum generator connected to a suction cup array, is used to flatten surfaces or packages;

[0205] The gripper combined with the suction cup at the end is used for mixed gripping of target objects of different shapes.

[0206] In the above embodiments, the processor can be an industrial computer, an embedded GPU computing unit, or a computing module in the robot control cabinet. The processor can be connected to the image acquisition device via USB 3.0, Ethernet, or CSI interface to receive image streams. The processor can communicate with the robot control cabinet via Ethernet, fieldbus, or industrial protocols to send end pose, gripper opening width / absorption parameters, and gripping action trigger signals.

[0207] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0208] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A method for generating end-effector pose based on target consistency constraints, characterized in that, Includes the following steps: Acquire the RGB and depth images of the target scene, and process the RGB images to obtain the segmented regions of the target objects; In the segmented region of the target object, the RGB image and the depth image are aligned, and the region of the target object is selected from the aligned target scene point cloud to extract the point cloud of the target object. A candidate set is constructed based on the point cloud of the target object, and the range of values ​​of the grab center is limited to the candidate set as a pre-constraint for grabbing candidates. Based on each grasping center point selected by the pre-constraint, and combined with the point cloud of the target object, multiple grasping candidate poses corresponding to the grasping center points are generated, and a grasping contact area corresponding to the grasping candidate poses is constructed. Identify the proportion of target object point cloud in the grasping contact area, construct post-constraints for target consistency in grasping posture, filter grasping candidate postures, score the filtered grasping candidate postures, and select the optimal posture. The process of generating multiple grasping candidate poses corresponding to each grasping center point based on the selected grasping center point and the point cloud of the target object includes the following steps: For each grasping center point, nested cylindrical neighborhoods of multiple scales are constructed and point cloud data is sampled. After feature extraction, the data is fused to obtain the multi-scale fused features corresponding to each grasping center point. ; Geometric features of the normal, curvature, and density of the central neighborhood points are obtained through cloud computing and fused with multi-scale features extracted from the network. The enhanced features are obtained by splicing the data together. For each grasping center point, multiple grasping candidate poses are generated based on the enhanced features; Multi-scale fusion features The generation process includes: For each grab center point, a local alignment coordinate system is established with the grab network prediction or the current candidate grab approach direction as the axis; Using the grab center point as the center, construct multiple cylindrical neighborhoods with different radii in the aligned coordinate system; Sampling of the point cloud occurs within the neighborhood of each cylinder; The coordinates of the sampling points are input into a shared multilayer perceptron. After obtaining the point features, pooling and aggregation are performed to obtain local features of a uniform dimension at each scale. Weights are calculated for local features at different scales using single-head attention and then weighted and fused to obtain the corresponding current grab center point. Multi-scale fusion features.

2. The end-effector pose generation method based on target consistency constraints as described in claim 1, characterized in that, The acquired RGB and depth images are processed, including: The RGB image and depth image are spatially aligned in the same coordinate system to ensure that the two-dimensional information corresponds to the three-dimensional information. The system acquires user input commands, performs target detection and segmentation on target objects in the capture scene based on RGB images, and determines the target region of the target object in the RGB image.

3. The end-effector pose generation method based on target consistency constraints as described in claim 1, characterized in that, The capture center is obtained by selecting or sampling 3D points from the target object point cloud, and the value range of the capture center is limited to the target object point cloud or its spatial neighborhood.

4. The end-effector pose generation method based on target consistency constraints as described in claim 1, characterized in that, Geometric features of the normal, curvature, and density of the central neighborhood points are obtained through cloud computing and fused with multi-scale features extracted from the network. The enhanced features are obtained by splicing them together, including the following steps: Within the neighborhood of nested cylinders at each scale, the center point will be captured. and Nearest neighbors are constructed as a set of neighborhood points. And calculate the mean of the neighborhood points. ; Based on the mean of the neighborhood points and the points within the neighborhood, a local covariance matrix is ​​constructed. The features of the covariance matrix of the neighborhood point set are decomposed to obtain the normal features, curvature features, and density features. Integrating geometric features with multi-scale features The features are concatenated and mapped back to the original dimension via a linear layer to obtain the enhanced features.

5. The end-effector pose generation method based on target consistency constraints as described in claim 4, characterized in that, The post-constraint for the consistency of the grasping posture is as follows: a ratio threshold is set, and the proportion of the point cloud of the target object in the grasping contact area exceeds the set ratio threshold as a post-constraint for filtering candidate grasping postures.

6. An end-effector pose generation system based on the end-effector pose generation method based on target consistency constraints according to any one of claims 1-5, characterized in that, include: The perception module is configured to acquire RGB and depth images of the target scene, and process the RGB images to obtain the segmented regions of the target objects; The target point cloud extraction module is configured to align the RGB image and depth image in the segmented region of the target object, filter out the region of the target object from the aligned target scene point cloud, and extract the point cloud of the target object. The pre-constraint module is configured to construct a candidate set based on the point cloud of the target object, and limit the value range of the grasping center to within the candidate set, serving as a pre-constraint for the grasping candidates; The grasping candidate generation module is configured to generate multiple grasping candidate poses corresponding to each grasping center point selected based on the pre-constraints, combined with the point cloud of the target object, and construct the grasping contact area corresponding to the grasping candidate poses. The pose selection module is configured to identify the proportion of the target object point cloud in the grasping contact area, construct a post-constraint for the consistency of the grasping pose, filter the grasping candidate poses, score the filtered grasping candidate poses, and select the optimal pose.

7. The end-effector pose generation system based on target consistency constraints as described in claim 6, characterized in that, The candidate generation module includes: The multi-scale feature modeling module is configured to construct nested cylindrical neighborhoods of multiple scales for each grasping center point, sample point cloud data, extract features, and then fuse them to obtain the multi-scale fused features corresponding to each grasping center point. ; An explicit geometric feature enhancement module is configured to compute geometric features based on the normal, curvature, and density of the center neighborhood points, and fuse them with multi-scale features extracted from the network. The enhanced features are obtained by splicing the data together. The generation module is configured to generate multiple grasping candidate poses based on the enhanced features for each grasping center point.

8. An end-effector pose generation system based on target consistency constraints, characterized in that, include: Image acquisition device, grasping execution terminal, and processor; The processor is configured to perform the steps of the end-effector pose generation method based on target consistency constraints as described in any one of claims 1-5, generating a grasping pose to control the grasping execution end to perform a grasping action.

Images