Target consistency constraint-based tail end pose generation method and system

Abstract

The invention relates to the technical field of robot control, and provides a tail end pose generation method and system based on target consistency constraint, and the method comprises the steps: carrying out the alignment of an RGB image and a depth image in a segmentation region of a target object, and extracting the point cloud of the target object; constructing a candidate set based on the target object point cloud, and limiting the value range of the grabbing center in the candidate set for preposition constraint; a plurality of grabbing candidate postures corresponding to the grabbing center points are generated by combining the point cloud of the target object on the basis of each grabbing center point selected by the preposed constraint, and grabbing contact areas corresponding to the grabbing candidate postures are constructed; and the point cloud proportion of the target object in the grabbing contact area is recognized, the grabbing candidate postures are screened, and finally the optimal posture is selected. According to the method, through grabbing center point selection of front constraint and rear screening based on target consistency, the accuracy of robot tail end pose generation is improved, and the grabbing success rate of the robot is increased.

Description

A Method and System for End-Factory Pose Generation Based on Target Consistency Constraints Technical Field

[0001] This invention relates to the field of robot control technology, specifically to a method and system for generating end-effector pose based on target consistency constraints. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] With the increasing prevalence of robots in industrial sorting, warehousing and handling, and home services, robots need to stably grasp designated target objects according to user instructions in complex environments such as occlusion, stacking, and sparse point clouds. Six-degree-of-freedom grasping can improve posture flexibility, but in real-world scenarios, the grasping system must not only be able to grasp, but also grasp accurately and correctly to the target; otherwise, it is easy to miss or grasp the wrong object, affecting the success rate of the task and the reliability of human-computer interaction.

[0004] Existing methods for robotic object grasping typically involve preprocessing RGB (Red-Green-Blue) images using object detection or segmentation, followed by prediction and filtering of grasping candidates on the target region's point cloud. This approach has several drawbacks: First, object detection or segmentation is usually only applied to cropped and filtered point clouds, limiting the input to the grasping network. However, the candidate generation process still relies on the grasping network predicting within a continuous spatial context. When the point cloud is sparse, the target object is occluded, or segmentation errors occur, the grasping center may appear at the edge of the target object or in a location without a point cloud, potentially leading to the system missing or grasping the wrong object. Second, after generating grasping candidates, existing methods typically only filter them based on grasping scores, lacking further assessment of the relationship between the robotic arm's gripper and the target object during the grasping process. In scenarios with stacked objects and severe occlusion, the highest-scoring pose generated by the grasping network may result in the gripper's closed area landing on a non-target object, potentially leading to a grasped object that differs from the user-specified object. Summary of the Invention

[0005] To address the aforementioned problems, this invention proposes an end-effector pose generation method and system based on target consistency constraints. By applying pre-constraints to the gripping center before generating gripping candidates and post-constraints based on the relationship between the gripper contact area and the target object during the screening stage after gripping candidate generation, the target object of the final gripping action is kept consistent with the target object specified by the user, thereby improving the accuracy of robot end-effector pose generation and increasing the robot end-effector gripping success rate.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: The first aspect of the present invention provides an end-effector pose generation method based on target consistency constraints, comprising the following steps: acquiring an RGB image and a depth image of a target scene; processing the RGB image to obtain a segmented region of the target object; aligning the RGB image and the depth image within the segmented region of the target object; selecting the region of the target object from the aligned point cloud of the target scene; extracting the point cloud of the target object; constructing a candidate set based on the point cloud of the target object; limiting the value range of the grasping center to within the candidate set as a pre-constraint for the grasping candidate; generating multiple grasping candidate poses corresponding to each grasping center point based on the pre-constraint and combining them with the point cloud of the target object; constructing a grasping contact region corresponding to the grasping candidate poses; identifying the proportion of the target object point cloud in the grasping contact region; constructing a post-constraint for target consistency of the grasping pose; filtering the grasping candidate poses; and scoring the filtered grasping candidate poses to select the optimal pose.

[0007] A second aspect of the present invention provides an end-effector pose generation system based on target consistency constraints, comprising: a perception module configured to acquire RGB and depth images of a target scene, and process the RGB images to obtain segmented regions of the target object; a target point cloud extraction module configured to align the RGB and depth images within the segmented regions of the target object, filter out regions of the target object from the aligned target scene point cloud, and extract the point cloud of the target object; a pre-constraint module configured to construct a candidate set based on the point cloud of the target object, and limit the value range of the grasping center within the candidate set as a pre-constraint for the grasping candidates; a grasping candidate generation module configured to generate multiple grasping candidate poses corresponding to each grasping center point selected by the pre-constraints, and construct a grasping contact region corresponding to the grasping candidate poses; and a pose selection module configured to identify the proportion of the target object point cloud in the grasping contact region, construct a post-constraint for target consistency of the grasping pose, select the grasping candidate poses, and score the selected grasping candidate poses to select the optimal pose.

[0008] A third aspect of the present invention provides an end-effector pose generation system based on target consistency constraints, comprising: an image acquisition device, a grasping execution end, and a processor; the processor is configured to execute the steps of the above-described end-effector pose generation method based on target consistency constraints to generate a grasping pose to control the grasping execution end to perform a grasping action.

[0009] Compared with existing technologies, the beneficial effects of this invention are as follows: By introducing a pre-constraint on the grasping center during the grasping candidate generation stage, this invention strictly limits the grasping center candidate set to the effective range of the target object's point cloud, making it less likely for the grasping center to fall on the edge of the target object, the transition region between the target and non-target objects, or a spatial location without point cloud. This enhances the spatial correlation between the grasping candidates and the target object from the source, reducing the risk of grasping the wrong object or missing the target due to grasping center offset when the point cloud is sparse, the target is occluded, or there are errors in target segmentation, thereby improving the consistency and reliability of grasping the specified target object.

[0010] This invention introduces a post-consistent constraint on the target consistency of grasping postures during the candidate grasping stage. By analyzing the spatial relationship between the grasping contact area and the target point cloud, and calculating the proportion of the target point cloud within the grasping contact area or the distribution relationship between target and non-target point clouds, it can effectively eliminate candidate grasping postures that may contact, interfere with, or clamp non-target objects within the gripper's closed area before executing the grasping action. This overcomes the problem of insufficient judgment on the "gripper-target object" interaction relationship when relying solely on grasping scores for screening. Especially in scenarios with multiple adjacent, occluded, or stacked objects, it can significantly reduce the probability of grasping high-scoring postures but actually clamping non-target objects, improving the matching accuracy between the grasping action and the user-specified target object, and increasing the overall grasping success rate.

[0011] The advantages of the present invention, as well as its additional advantages, will be described in detail in the following specific embodiments. Attached Figure Description

[0012] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute a limitation thereof.

[0013] Figure 1 is a flowchart of the end-effector pose generation method based on target consistency constraints according to Embodiment 1 of the present invention; Figure 2 is an overall architecture diagram of the end-effector pose generation method according to Embodiment 1 of the present invention; Figure 3 is a schematic diagram of the construction process of the grasping center pre-constraint according to Embodiment 1 of the present invention; Figure 4 is a schematic diagram of the grasping contact area and target consistency according to Embodiment 1 of the present invention; Figure 5 is a schematic diagram of multi-scale cylinder local feature modeling according to Embodiment 1 of the present invention; Figure 6 is a schematic diagram of explicit geometric feature enhancement according to Embodiment 1 of the present invention; Figure 7 is a structural block diagram of the end-effector pose generation system based on target consistency constraints according to Embodiment 2 of the present invention. Detailed Implementation

[0014] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0015] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0016] It should be noted that the terminology used herein is for describing particular embodiments only and is not intended to limit the exemplary embodiments of the present invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof. It should be noted that, without conflict, the various embodiments and features within those embodiments can be combined with each other. The embodiments will now be described in detail with reference to the accompanying drawings.

[0017] Example 1, in one or more of the technical solutions disclosed in the embodiments, as shown in Figures 1 to 6, describes an end-effector pose generation method based on target consistency constraints, comprising the following steps: Step 1: Acquire RGB and depth images of the target scene, and process the RGB images to obtain segmented regions of the target object; Step 2: Align the RGB and depth images within the segmented regions of the target object, filter out the target object region from the aligned target scene point cloud, and extract the target object point cloud; Step 3: Construct a candidate set based on the target object point cloud, and limit the value range of the grasping center to within the candidate set as a pre-constraint for the grasping candidates; Step 4: Based on each grasping center point selected by the pre-constraint, combine it with the target object point cloud to generate multiple grasping candidate poses corresponding to the grasping center points, and construct a grasping contact region corresponding to the grasping candidate poses; Step 5: Identify the target object point cloud proportion in the grasping contact region, construct a post-constraint for target consistency of the grasping pose, filter the grasping candidate poses, and score the filtered grasping candidate poses to select the optimal pose.

[0018] In the above implementation, by introducing a pre-constraint on the grasping center during the grasping candidate generation stage, the candidate set of grasping centers is strictly limited to the effective range of the target object's point cloud. This makes it less likely for the grasping center to fall on the edge of the target object, the transition region between the target and non-target objects, or a spatial location without point cloud. This enhances the spatial correlation between the grasping candidates and the target object from the source, reducing the risk of grasping the wrong object or missing the target due to grasping center offset when the point cloud is sparse, the target is occluded, or there are errors in target segmentation. This improves the consistency and reliability of grasping the specified target object.

[0019] In the candidate grasping stage, a post-constraint on target consistency of grasping posture is introduced. By analyzing the spatial relationship between the grasping contact area and the target point cloud, the proportion of the target point cloud or the distribution relationship of the target / non-target point cloud within the grasping contact area is calculated. Before executing the grasping action, candidate grasping postures that may contact, interfere with, or clamp non-target objects within the gripper's closed area can be effectively eliminated. This can compensate for the insufficient judgment of the interaction relationship between the gripper and the target object when relying solely on grasping scores for screening. Especially in scenarios with multiple adjacent, occluded, or stacked objects, it can significantly reduce the probability of grasping high-scoring postures but actually clamping non-target objects, improve the matching accuracy between the grasping action and the user-specified target object, and increase the overall grasping success rate.

[0020] In step 1, the object applicable to the end-effector pose generation in this embodiment can be a robotic arm or a robot or other intelligent agent; in step 1, the image data is acquired by acquiring RGB images and depth images of the scene through depth vision sensor devices such as depth cameras; in step 1, the acquired RGB images and depth images are processed, including: step 11, aligning the RGB images and depth images in the same coordinate system to ensure the correspondence between two-dimensional and three-dimensional information; step 12, acquiring the user input command, and performing target detection and segmentation on the target object in the grasping scene based on the RGB image to determine the target region of the target object in the RGB image; specifically, the target region of the target object can be obtained by generating a detection box by performing target detection on the target object, and then segmenting the detection box to obtain pixel-level segmented regions.

[0021] In step 2, after determining the target object region, based on the position of the target object region in the RGB image, the 3D data belonging to the target region is filtered out from the depth image or scene point cloud aligned with the RGB image, thus obtaining the target point cloud corresponding to the target object.

[0022] Referring to Figure 3, a candidate set is constructed based on the target object point cloud. Specifically, after obtaining the target object point cloud, the target object point cloud is marked as 1, while the remaining point clouds in the current scene are marked as 0, creating a target object point cloud set, i.e., the point cloud set marked as 1. This created set serves as the candidate set for the grasping center, limiting the value range of the grasping center. Further, the grasping center is obtained by selecting or sampling 3D points from the target point cloud, and the value range of the grasping center is limited to the target point cloud or its spatial neighborhood. The neighborhood typically refers to a very small spatial area around the target point cloud. The neighborhood range is a spatially extended area constructed based on the target object point cloud. For potential grasping positions close to the target object surface, the candidate geometry can include these positions. The neighborhood range can be obtained by extending the target object point cloud by distance, for example, constructing a neighborhood region within a preset distance threshold range centered on each point in the target point cloud; or determined by methods such as K-nearest neighbor search. The distance threshold can be set according to the gripper size or grasping scale.

[0023] In the above embodiments, the spatial location of the grasping center is strictly limited to the target point cloud itself or its close neighborhood, so that the grasping center is limited to the three-dimensional space of the target object as much as possible, which avoids the center falling on the edge of the target or in a region without point cloud, and also prevents the center from deviating from the target due to sparse point cloud or segmentation error.

[0024] In step 3, before generating the grabbing candidate poses, several 3D points are selected from the candidate set point cloud as grabbing center points, and these grabbing center points are used as geometric references for generating grabbing candidates. Specifically: Step 31, for each point in the candidate set point cloud, predict whether it is a target object point (objectness) and its grabbability score (graspness), and construct a grabbing center candidate set by combining it with the target mask (target_mask). The formula represents: ;in: This represents the set of candidate centers for capture; Represents the first point in the scene point cloud Points; objectness ) represents a point Is it a prediction result for the target object point? (graspness) ) represents a point Scrabability score; This is the crawlability threshold; Point The point cloud region corresponding to the target object.

[0025] Through the above screening, only points that belong to the target object and have a high feasibility of being grasped are retained as the set of candidate grasping centers.

[0026] Step 32: In the crawl center candidate set Within this process, Farthest Point Sampling (FPS) is performed, selecting a set number of points as the set of capture center points. It is represented as: ; optional One can take 1024; where: This represents the set of crawl center points, which is the candidate set of crawl centers. The point set selected by sampling from the farthest point; Indicates the first One capture center point, and ; This is the preset number of sampling points.

[0027] In this step, sampling at the farthest point ensures that the distribution of the grab center point in space is relatively uniform, thereby improving the coverage of the grab candidate pose generation.

[0028] Step 33: For each gripping center point, determine the direction in which the gripper approaches the gripping center point, predict the score corresponding to different approach directions, and select the viewpoint with the highest score as the optimal approach direction for that gripping center point; construct a local geometric reference coordinate system using the optimal approach direction as the principal direction of the local coordinate system; specifically, using the optimal approach direction as the principal direction of the local coordinate system, and combining it with the principal direction to construct two directions orthogonal to it, finally constructing the local geometric reference coordinate system at the gripping center point using three orthogonal direction vectors, and representing this coordinate system as a rotation matrix: ;in: To capture the center point The rotation matrix, consisting of three orthogonal unit vectors arranged in columns, is used to represent the directional relationship between the local geometric reference coordinate system and the target scene point cloud coordinate system; This indicates the direction vector to be captured; , This represents the unit vector of the coordinate axis direction that is orthogonal to the grabbing direction.

[0029] In the above process, the target scene point cloud is defined in a unified three-dimensional coordinate system, and the subsequent grasping center point, rotation matrix and gripper geometry model are all represented and calculated in this coordinate system.

[0030] Step 34: Grab the center point With rotation matrix As a geometric reference for generating grabbing candidates, the local coordinate system is aligned with the grabbing direction, and local neighborhood construction, feature extraction, and grabbing posture parameter prediction are performed in this coordinate system.

[0031] This embodiment, through the above implementation method, achieves the effect of grasping the center point. From only the candidate set The point cloud of the target object is sampled, thus limiting the grab center to the target object's point cloud range. This avoids the grab center appearing at the edge of the target object, in the transition area between the target and non-target objects, or in empty areas without point clouds, thereby improving the stability and accuracy of grab pose generation.

[0032] In step 4 of this embodiment, under the pre-constraint of the grasping center, for each grasping center point, combined with the target object point cloud information, the grasping network generates multiple grasping candidate postures corresponding to the grasping center point, such as six-degree-of-freedom grasping candidate postures. Multi-scale cylindrical local feature modeling and explicit geometric feature enhancement are introduced into the grasping network to improve its ability to express the local geometric structure of the target object during the grasping candidate generation stage, thereby improving the stability and robustness of the grasping candidate generation. As a further technical solution, the grasping network is improved as follows to generate grasping candidate postures: Multi-scale feature modeling: When generating grasping candidate postures, multiple local spatial neighborhoods of different scales can be constructed in the target point cloud of the grasping center, and feature extraction and feature fusion are performed on the point clouds in each scale neighborhood to enhance the network's grasping ability for objects of different scales; Explicit geometric feature enhancement: During the modeling process of each scale local spatial neighborhood, explicit geometric features of the target point cloud, including normal, curvature, and density, can be calculated, and the explicit geometric features are fused with the original features to enhance the network's grasping ability for objects with different geometric shapes.

[0033] In step 4, the process of generating multiple grasping candidate poses corresponding to each grasping center point selected based on the pre-constraints and combining them with the point cloud of the target object includes the following steps: Step 41, Multi-scale cylinder local feature modeling: For each grasping center point, construct a nested cylinder neighborhood at multiple scales and sample point cloud data, extract features, and then fuse them to obtain the multi-scale fused features corresponding to each grasping center point. ;in, Indicates the first The feature number corresponding to each capture center point Indicates the first A multi-scale fusion feature vector that captures the center point.

[0034] Furthermore, in step 41, multi-scale fusion features The generation process includes: Step 411, constructing an alignment coordinate system: for each grab center point Using the rotation matrix corresponding to the candidate pose Establish a local alignment coordinate system, using the grabbing approach direction as the axis of the coordinate system, so that subsequent neighborhood modeling is consistent with the grabbing direction.

[0035] Step 412: Construct a multi-scale nested cylindrical neighborhood: using the center point as the starting point. Centered on a central point, construct multiple cylindrical neighborhoods with different radii in an aligned coordinate system; Step 413, neighborhood sampling at each scale: sample the point cloud within each cylindrical neighborhood; Step 414, shared feature extraction and aggregation: input the coordinates of the sampled points into a shared multilayer perceptron (MLP), obtain the point features, and then pool and aggregate them to obtain local features of a uniform dimension (e.g., 256 dimensions) at each scale; Step 415, lightweight scale attention fusion: calculate weights for local features at different scales based on single-head attention and perform weighted fusion to obtain the corresponding current grasping center point. The multi-scale fusion features; the output of this step still maintains the same dimension (such as 256 dimensions), avoiding dimensional expansion caused by direct splicing, thereby maintaining the network economy.

[0036] Traditional grasping models typically use a fixed-scale cylinder with a fixed radius, such as 5cm, to model the point cloud at the grasping center. However, when dealing with small objects, an excessively large cylinder radius introduces a large number of background points and other object point clouds, making it difficult to capture the point cloud features of the target object. Furthermore, a single-scale cylinder cannot simultaneously capture both local details and global features when facing a target object. In this step, by constructing a multi-scale nested cylinder neighborhood for the same grasping center and adaptively fusing it with scale attention, the model can simultaneously acquire small-scale local details and large-scale structural context. This reduces the risk of background noise and feature overload introduced by a fixed large radius, thereby improving the accuracy of representing the target's local geometry during the grasping candidate generation stage, enhancing the stability of candidate pose prediction, and improving robustness to scale changes, occlusion, and point cloud sparsity, while maintaining the output feature dimension unchanged to control computational and parameter costs.

[0037] In one specific implementation, this embodiment introduces multi-scale cylindrical local feature modeling into the grasping model. The structure can be as shown in Figure 5. Specifically, the process of step 41 is illustrated by constructing four nested cylinders of different scales.

[0038] The multi-scale cylinder local feature module constructs four nested cylinders of different scales with radii of 1.25cm, 2.5cm, 3.75cm, and 5cm, corresponding to 0.25 times, 0.5 times, 0.75 times, and 1 times the original radius, respectively. Excessively large scales may introduce too much background noise, and high-resolution small-scale features are needed for fine-grained tasks like grasping.

[0039] For each grab center point An alignment coordinate system is constructed based on the grasping direction predicted by the grasping network. Within this coordinate system, four cylinders with different radii are defined. A fixed number of samples is obtained within each cylinder using cylinder queries: 1.25cm corresponds to 16 points, 2.5cm to 24 points, 3.75cm to 24 points, and 5cm to 32 points. The point cloud coordinates are then input into a shared MLP (Multi-Level Processing), which extracts the features of each point. These features are then pooled into four 256-dimensional local features.

[0040] The local features extracted at each scale can be represented as: ;in: Let be the radius of the neighborhood of the cylinder. To correspond to the number of sampling points in the neighborhood of the cylinder, 256 is the unified feature dimension of nested cylinder neighborhoods of different scales.

[0041] For the The feature vector of a sampling point can be represented as: For the feature vector at each scale, a shared two-layer perceptron is used to extract features from the feature vector of each point, thus obtaining the point features. The process is represented as: ;in: This represents the first-layer weight matrix, used for linear transformation and feature compression of the input features, mapping the 256-dimensional features to a smaller space. It is the first layer of MLP, and its function is to extract information useful for judging the importance of the scale from the features at the current scale.

[0042] For ReLU activation function: The role of activation functions is to introduce non-linearity, enabling the network to express the importance of certain feature combinations.

[0043] This represents the second-layer weight matrix, used to map feature vectors to a scalar score. This score is used to represent the importance of features at the current scale and is used for weight calculation in subsequent multi-scale feature fusion.

[0044] To reduce the number of model parameters and computational complexity, direct concatenation is avoided to prevent the dimensionality from quadrupling. This embodiment uses a lightweight single-head scale attention algorithm to calculate weights, as follows: for the same sampling point... Point features are normalized using softmax across all scales: ;in: The number of scales is 4 in this embodiment.

[0045] The purpose of softmax normalization is to allow the network to dynamically assign scale weights to each crawling center, rather than to determine which scales are more important. For the first The sampling point at the th sampling point The weights on each scale represent the importance of that scale to the current crawling center.

[0046] Finally, the 256-dimensional features across the four scales are weighted according to... Weighting is performed to obtain each sampling point The 256-dimensional multi-scale fusion feature achieves effective integration of multi-scale features while keeping the crawling network interface and parameter scale unchanged.

[0047] For the The multi-scale fusion feature of each sampling point is represented as follows: ;in: Indicates the first Features obtained by multi-scale fusion of sampling points Indicates the number of scales; Indicates the first The sampling point at the th sampling point Weights under each scale; Indicates the first The sampling point at the th sampling point Eigenvectors at various scales.

[0048] Based on this, the fusion features of all sampling points are aggregated to obtain the current capture center point. The corresponding final multi-scale fusion features : ;in: This indicates a pooling operation, used to aggregate features from multiple sampling points; Indicates the first Multi-scale fusion features of each sampling point; This indicates the number of sampling points at the corresponding scale.

[0049] Step 42, Explicit Geometric Feature Enhancement: Geometric features based on the normal, curvature, and density of the central neighborhood points are obtained through cloud computing and fused with multi-scale features extracted from the network. The enhanced features are obtained by splicing the data. The enhanced feature generation process includes steps 421 to 423. Although multi-scale cylindrical local feature modeling can perceive the spatial structure of objects at different scales, like other point cloud grasping methods, the model mainly relies on geometric information learned from the point cloud. This makes it difficult for the model to accurately perceive key grasping areas such as the edges and surface changes of the object. Point clouds often contain noise and are frequently occluded, which may result in the loss of some point clouds. The geometric relationship between the 3D point cloud and the surface of the real object cannot be fully recovered by pure coordinates, which leads to unstable geometric feature learning by the model. Therefore, explicit geometric feature enhancement is introduced into the grasping model, as shown in Figure 6.

[0050] The explicit geometric feature enhancement module introduces explicit geometric features such as normals, curvature, and density to construct a lightweight geometric enhancement module, enhancing the expressive power of the local geometric structure of the point cloud and improving the accuracy of the model's grasping pose prediction. The specific method includes the following: Step 421, Constructing a local neighborhood point set: In the nested cylindrical neighborhood at each scale, the grasping center point... and The points obtained by K-Nearest Neighbors (KNN) are used to construct a neighborhood point set. And calculate the mean of the neighborhood points. : ;in, Represents the neighborhood point set The point in, and ; Indicates the number of neighboring points. The mean of the neighborhood point set is represented; Step 422: Calculate explicit geometric features: Based on the mean of the neighborhood points and the points within the neighborhood, construct a local covariance matrix, decompose the covariance matrix features of the neighborhood point set to obtain normal features, curvature features, and density features; construct the local covariance matrix. ,as follows: For the covariance matrix Eigenvalue decomposition is performed. Since the covariance matrix is ​​calculated from the 3D point cloud coordinates, the decomposition yields three eigenvalues. These three eigenvalues ​​are then sorted in ascending order. Among them, the smallest eigenvalue The direction of the corresponding unit eigenvector is the direction of minimum diffusion in the neighborhood point cloud, which serves as the normal vector of the target object. For planar regions, the point cloud has a large distribution in both directions, therefore For edge regions, the point cloud is distributed very large in only one direction, so .

[0051] Curvature features are used to characterize the degree of edge or abrupt change. Curvature features are calculated using eigenvalues. : For planar regions, Extremely small, therefore curvature ≈0, for mutation or corner regions, , , Both are relatively large, so the curvature is... Move closer to 1.

[0052] Density characteristics The following formula is used to characterize the sparsity and reliability of point clouds: ;in, , This represents the radius of the neighborhood of the cylinder at the corresponding scale; for sparse regions or regions with abrupt changes in the edge of the point cloud, + + The value will increase, leading to It will also increase, so the corresponding density feature It would also be too large to be suitable as a focal point.

[0053] The extracted geometric features are concatenated to construct a geometric feature vector. It is represented as: ;in, Indicates the corresponding center point of the capture. Geometric eigenvectors, Represents the normal eigenvector. Indicates curvature characteristics. This indicates density characteristics.

[0054] Step 423, Feature Fusion and Dimension Alignment: Combine geometric features with the multi-scale fused features output from Step 41. The concatenation is performed and mapped back to the original dimension (e.g., 256 dimensions) through a linear layer to obtain the enhanced features.

[0055] Specifically, as shown in Figure 6, in this embodiment, the 3D normal feature, 1D curvature feature, and 1D density feature corresponding to the current grasping center are concatenated to form a 5D feature vector, namely the geometric feature vector. Then, it is concatenated with the 256-dimensional features of the grasping network to form a 261-dimensional feature vector. Finally, a linear mapping is used to map it back to 256 dimensions, aligning it with the previous network. This introduces geometric feature enhancement without changing the feature dimensions, which aligns with the economical nature of networks.

[0056] Furthermore, during training, the feature fusion and alignment dimensions are initialized using an approximate identity, as shown in the formula:

[0057] in: The weight matrix represents the linear mapping. Indicates the current center point of the capture. The corresponding enhanced features, Indicates the corresponding center point of the capture. Geometric eigenvectors, This represents the concatenation of multi-scale fused features and geometric features; This represents the bias term; during the initialization phase, the weight matrix... and bias terms Set to: ; express The identity matrix, This represents the zero matrix, where the weights of the corresponding geometric features are initialized to 0; in the initial state, the geometric features... It has no effect on the output: At the start of training, the explicit geometric enhancement module does not change the behavior of the original model or destroy the learned multi-scale features. As training iterates, the weights of the geometric features are dynamically adjusted to gradually exert their enhancement effect, ultimately achieving stable perception of the target's geometric structure. In step 42 of this embodiment, explicit geometric feature enhancement is introduced. Based on the geometric quantities such as normal, curvature, and density of the grasping center neighborhood points, it is fused with the network-learned features. This enables the grasping model to stably perceive the orientation changes, edge abrupt changes, and sparse and unreliable regions of the target surface even under point cloud noise, missing points, and occlusion. This improves the ability to identify key graspable regions, reduces the probability of grasping points falling on edge holes or low-confidence regions, and thus improves the accuracy and consistency of six-degree-of-freedom grasping candidate pose prediction. At the same time, by mapping the fused features back to the original dimensions and using approximate identity initialization, the new module achieves enhanced geometric perception capabilities without significantly increasing interface complexity and parameter overhead or destroying the initial behavior of the original model.

[0058] Step 43: Generate candidate grasping poses: For each grasping center point, multiple candidate grasping poses are generated based on the enhanced features. The six-DOF candidate grasping poses include grasping position parameters and grasping pose parameters, which can represent the actual grasping pose of the gripper in three-dimensional space. Generating multiple candidate grasping poses around the grasping center point provides a basis for subsequent selection. Specifically, the candidate grasping poses may include: grasping approach direction, rotation angle around the approach direction, grasping depth along the approach direction, gripper opening width, and grasping quality score. In this embodiment, step 4 proposes multi-scale cylindrical local feature modeling. By fusing features from cylinders of multiple scales, the model's ability to perceive objects of different scales is enhanced. Then, a geometric feature enhancement module is performed, fusing normal, curvature, and density geometric features based on covariance decomposition to enhance the grasping network's ability to express complex geometric structures of objects. After completing multi-scale feature fusion and geometric feature enhancement, the enhanced local features are fed into the grasping head network to regress the grasping pose parameters, evaluate and rank the grasping quality, and finally output an executable grasping pose.

[0059] In step 4, for each candidate gripping posture, a method for constructing the gripping contact area corresponding to the candidate gripping posture is provided based on the gripper's structural parameters, dimensional information, and generated gripping candidate posture parameters. This method includes the following steps: Step 401: Determine the position and orientation of the gripper in three-dimensional space based on the gripping candidate posture parameters; the gripping candidate posture parameters include the gripping center point. Position and rotation matrix And the gripper width parameter.

[0060] Capture the center point The position is used to determine the gripper's grasping position in space; rotation matrix The gripper width parameter is used to determine the spatial orientation of the gripper and to determine the opening range of the gripper during the closing process.

[0061] Step 402: Establish a geometric model of the gripper based on its structural parameters, including the length, width, and thickness of the gripper.

[0062] In one embodiment, the gripper geometry is modeled as a three-dimensional structure, such as a cuboid, cube, prism, or pyramid. Based on the gripper's structural parameters, the spatial dimensions of the three-dimensional structure are determined. Step 403 involves spatially transforming the gripper geometry based on the candidate gripping posture parameters, including transformations based on the gripping center point. Translation transformation and rotation matrix based The rotation transformation aligns the gripper geometry model with the candidate gripping posture in three-dimensional space; Step 404: Based on the position and orientation of the aligned gripper geometry model, within the gripper's closed range, determine the spatial area where the gripper may contact the object, as the gripping contact area. The gripping contact area represents the spatial range where the gripper may contact the object during the gripping process, and is used for subsequent target consistency determination.

[0063] In the above embodiments, the grasping contact area can represent the spatial range in which the gripper may come into contact with the object during the grasping process, providing a basis for subsequent target consistency determination.

[0064] Step 5: Identify the proportion of the target object point cloud in the grasping contact area, construct a post-constraint for the target consistency of the grasping posture, filter the grasping candidate postures, score the filtered grasping candidate postures and select the optimal posture; referring to Figure 4, after constructing the grasping contact area, the target consistency of the grasping candidate postures is judged based on the spatial relationship between the grasping contact area and the target point cloud.

[0065] The post-constraint for target consistency in grasping postures is as follows: A proportional threshold is set. If the proportion of the target object point cloud in the grasping contact area exceeds this threshold, it serves as a post-constraint for filtering candidate grasping postures. Specifically, the proportion of the target object point cloud in the grasping contact area is statistically analyzed. The point clouds were marked in step 3, and the set of point clouds marked as 1 is used as the candidate set for grasping centers. For each candidate grasping posture, its corresponding gripper contact area is determined, and the ratio of the number of point clouds belonging to the target object within that contact area to the total number of point clouds in that area is calculated. When the proportion of the target object point cloud in the grasping contact area meets a certain threshold (i.e., the proportion of point clouds marked as 1 meets the set proportional threshold), the corresponding candidate grasping posture is determined to satisfy the target consistency constraint. Candidate grasping postures that do not meet the threshold are discarded.

[0066] In this embodiment, the target consistency determination step can effectively exclude grasping candidate postures that are inconsistent with the specified target object and grasping candidate postures that may come into contact with non-target objects before grasping is executed.

[0067] A further technical solution involves ranking the candidate grasping postures that meet the target consistency constraint, taking into account grasping quality evaluation and collision detection, and selecting the optimal grasping posture for the target object grasping task. Finally, the robotic arm and end effector are controlled to perform the grasping operation according to the selected target grasping posture, thereby completing the six-degree-of-freedom grasping of the specified target object.

[0068] To illustrate the effectiveness of the pose generation method in this embodiment, experimental comparisons were conducted, detailed below. The experiments used the large-scale 6-DoF grasping detection dataset GraspNet-1Billion. This dataset contains 97,280 images and over 1.1 billion grasping poses from 190 cluttered real-world scenes captured by two popular RGB-D (Red-Green-Blue-Depth, i.e., color and depth images) cameras, RealSense and Kinect. The experiments followed the official training and testing partitioning method, with the first 100 scenes used for training and the remaining 90 scenes used for testing. The test set was divided into three categories based on the characteristics of the scene objects: seen objects, similar but unseen objects, and novel objects, with 30 scenes in each category.

[0069] Tables 1 and 2 show the performance comparison results of the method in this embodiment (hereinafter referred to as MSGeoGrasp) with existing representative models on the GraspNet-1 Billion dataset under RealSense and Kinect camera settings, respectively, with the best results marked in bold. MSGeoGrasp achieves a high level in both camera settings, with particularly outstanding performance under the RealSense camera, outperforming other methods in the Seen, Similar, and Novel scenarios; Table 1 shows the performance comparison under the RealSense camera setting.

[0070] Table 2 shows the performance comparison under Kinect camera settings;

[0071] In Tables 1 and 2, CD (Collision Detection) represents collision detection, used to determine whether the generated grasping pose collides with an object or the environment. The best results without using collision detection (CD) are marked in bold; the existing methods or models are explained as follows: GPD: A grasping pose detection method based on point cloud geometric features, which generates grasping candidate poses and uses a convolutional neural network to evaluate the grasping quality.

[0072] PointNetGPD: A grasping posture detection method based on the PointNet point cloud feature extraction network, which evaluates grasping quality by learning local geometric features of the point cloud.

[0073] GraspNet: An end-to-end six-DOF grasping pose detection network model based on deep learning.

[0074] TransGrasp: A grasping posture detection method based on the Transformer structure, which improves grasping detection performance by modeling long-distance feature relationships between point clouds.

[0075] GraNet: A grasping pose generation method based on graph neural networks, which learns point cloud spatial features by constructing a multi-layer graph structure.

[0076] TSB: A grasping detection method based on multi-scale feature learning, which enhances local geometric representation through multi-scale cylinder feature extraction.

[0077] GSNet: A crawl detection network based on crawlability assessment to filter high-quality crawl regions.

[0078] EconomicGrasp: A six-degree-of-freedom grasping detection method based on economic supervision strategies to reduce model training resource consumption.

[0079] The results in Tables 1 and 2 show that, under the RealSense camera settings, MSGeoGrasp improves the average accuracy (AP) by 2.67% compared to EconomicGrasp, with the most significant improvement of 4.72% in the Seen scene; and improvements of 1.67% and 1.61% in the Similar and Novel scenes, respectively. This indicates that the proposed multi-scale cylinder local feature modeling and geometric feature enhancement module can effectively improve grasping detection performance under high-quality point cloud conditions.

[0080] With the Kinect camera setup, due to higher point cloud noise and sparsity, the overall performance of all methods decreased compared to the RealSense camera setup. MSGeoGrasp improved the average AP by 0.16% compared to EconomicGrasp, achieving a 2.10% improvement in the Seen scene; in the Similar and Novel scenes, performance decreased slightly due to the impact of point cloud quality.

[0081] The results show that under low-quality point cloud conditions, the performance of multi-scale cylinder local feature modeling and geometric feature enhancement superposition is affected to some extent, but MSGeoGrasp can maintain competitive performance compared with existing models in most scenarios.

[0082] Example 2 is based on Example 1. This example provides an end-effector pose generation system based on target consistency constraints, as shown in Figure 7. It includes: a target point cloud extraction module, configured to align RGB images and depth images in the segmented region of the target object, filter out the region of the target object from the aligned target scene point cloud, and extract the point cloud of the target object; a pre-constraint module, configured to construct a candidate set based on the point cloud of the target object, and limit the value range of the grasping center to within the candidate set as a pre-constraint for the grasping candidate; a grasping candidate generation module, configured to generate multiple grasping candidate poses corresponding to each grasping center point selected by the pre-constraint, combined with the point cloud of the target object, and construct a grasping contact region corresponding to the grasping candidate poses; and a pose selection module, configured to identify the proportion of the target object point cloud in the grasping contact region, construct a post-constraint for the target consistency of the grasping pose, select the grasping candidate poses, and score the selected grasping candidate poses to select the optimal pose.

[0083] Furthermore, the candidate generation module includes a multi-scale feature modeling module, which is configured to construct nested cylindrical neighborhoods of multiple scales for each grasping center point, sample point cloud data, extract features, and then fuse them to obtain multi-scale fused features corresponding to each grasping center point. Specifically, it is configured to perform step 41 in embodiment 1; the explicit geometric feature enhancement module is configured to compute geometric features based on the normal, curvature, and density of the central neighborhood points, and fuse them with multi-scale features extracted from the network. The features are concatenated to obtain enhanced features; specifically, step 42 in embodiment 1 is executed; the generation module is configured to generate multiple grasping candidate poses based on the enhanced features for each grasping center point.

[0084] It should be noted that each module in this embodiment corresponds one-to-one with each step in embodiment 1, and their specific implementation process is the same, so it will not be repeated here.

[0085] Example 3 is based on Example 1. This example provides an end-effector pose generation system based on target consistency constraints, including: an image acquisition device for acquiring RGB images and depth images of a target scene; a grasping execution end for performing grasping actions; and a processor for executing the steps of the end-effector pose generation method based on target consistency constraints described in Example 1, generating a grasping pose and controlling the grasping execution end to complete the grasping of the target object.

[0086] In the above embodiments, the image acquisition device can be an RGB-D camera, such as a structured light RGB-D camera, a ToF RGB-D camera, or an active binocular depth camera.

[0087] The RGB-D camera can be mounted in any of the following locations: the camera is fixed near the end of the robotic arm or the side of the gripper for tracking and observing the target; the camera is fixed on a bracket above the worktable, gantry, or robot workstation frame for viewing overhead / oblique scenes.

[0088] In the above embodiments, the grasping end effector can be part of an industrial robot system, which includes a robotic arm body, a robot control cabinet, and an end effector.

[0089] The gripping execution end can be specifically categorized as follows: two-finger parallel grippers, such as electrically driven or pneumatically driven parallel grippers, for gripping regular or semi-regular objects; adaptive grippers, such as three-finger adaptive grippers or flexible grippers, for irregularly shaped or easily damaged objects; suction end, such as vacuum suction cups or vacuum generator connected to suction cup arrays, for flattening surfaces or packaging; and gripper combined with suction cup combination end: for mixed gripping of target objects of different shapes.

[0090] In the above embodiments, the processor can be an industrial computer, an embedded GPU computing unit, or a computing module in the robot control cabinet. The processor can be connected to the image acquisition device via USB 3.0, Ethernet, or CSI interface to receive image streams. The processor can communicate with the robot control cabinet via Ethernet, fieldbus, or industrial protocols to send end pose, gripper opening width / absorption parameters, and gripping action trigger signals.

[0091] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0092] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A method for generating end-effector pose based on target consistency constraints, characterized in that, The process includes the following steps: acquiring the RGB image and depth image of the target scene; processing the RGB image to obtain the segmented region of the target object; aligning the RGB image and depth image within the segmented region of the target object; filtering out the region of the target object from the aligned point cloud of the target scene; and extracting the point cloud of the target object. A candidate set is constructed based on the point cloud of the target object, and the range of values ​​of the grab center is limited to the candidate set as a pre-constraint for grabbing candidates. For each grasping center point selected based on the pre-constraints, multiple grasping candidate poses corresponding to the grasping center points are generated by combining the point cloud of the target object, and a grasping contact area corresponding to the grasping candidate poses is constructed. The proportion of the target object point cloud in the grasping contact area is identified, a post-constraint for the target consistency of the grasping pose is constructed, the grasping candidate poses are screened, the screened grasping candidate poses are scored, and the optimal pose is selected.

2. The end-effector pose generation method based on target consistency constraints as described in claim 1, characterized in that, The acquired RGB and depth images are processed, including: spatially aligning the RGB and depth images in the same coordinate system to ensure correspondence between two-dimensional and three-dimensional information; acquiring user input commands; performing target detection and segmentation on target objects in the capture scene based on the RGB images; and determining the target region of the target object in the RGB images.

3. The end-effector pose generation method based on target consistency constraints as described in claim 1, characterized in that, The capture center is obtained by selecting or sampling 3D points from the target object point cloud, and the value range of the capture center is limited to the target object point cloud or its spatial neighborhood.

4. The end-effector pose generation method based on target consistency constraints as described in claim 1, characterized in that, The process of generating multiple grasping candidate poses corresponding to each grasping center point based on the selected grasping center point and the point cloud of the target object includes the following steps: For each grasping center point, constructing nested cylindrical neighborhoods of multiple scales and sampling point cloud data, extracting features and fusing them to obtain multi-scale fused features corresponding to each grasping center point. Geometric features of normal, curvature, and density are obtained by capturing the central neighborhood points and cloud computing, and then fused with multi-scale features extracted from the network. The enhanced features are obtained by concatenating the features; for each grasping center point, multiple grasping candidate poses are generated based on the enhanced features.

5. The end-effector pose generation method based on target consistency constraints as described in claim 4, characterized in that, Multi-scale fusion features The generation process includes: for each grasping center point, establishing a local alignment coordinate system with the grasping network prediction or the current candidate grasping approach direction as the axis; constructing multiple cylindrical neighborhoods with different radii in the alignment coordinate system with the grasping center point as the center; sampling the point cloud within each cylindrical neighborhood; inputting the sampled point coordinates into a shared multilayer perceptron, obtaining point features, and then pooling and aggregating them to obtain local features of a unified dimension at each scale; calculating weights for local features at different scales based on single-head attention and weighted fusion to obtain the corresponding current grasping center point. Multi-scale fusion features.

6. The end-effector pose generation method based on target consistency constraints as described in claim 4, characterized in that, Geometric features of the normal, curvature, and density of the central neighborhood points are obtained through cloud computing and fused with multi-scale features extracted from the network. The enhanced features are obtained by stitching together the data, including the following steps: Within the nested cylindrical neighborhood at each scale, the center point is extracted. and Nearest neighbors are constructed as a set of neighborhood points. And calculate the mean of the neighborhood points. Based on the mean of neighborhood points and the points within the neighborhood, a local covariance matrix is ​​constructed. The covariance matrix features of the neighborhood point set are decomposed to obtain normal features, curvature features, and density features. Geometric features are then fused with multi-scale features. The features are concatenated and mapped back to the original dimension via a linear layer to obtain the enhanced features.

7. The end-effector pose generation method based on target consistency constraints as described in claim 6, characterized in that, The post-constraint for the consistency of the grasping posture is as follows: a ratio threshold is set, and the proportion of the target object point cloud in the grasping contact area exceeds the set ratio threshold as a post-constraint for filtering grasping candidate postures.

8. An end-effector pose generation system based on target consistency constraints, characterized in that, include: The perception module is configured to acquire RGB and depth images of the target scene, and process the RGB images to obtain the segmented regions of the target objects; The target point cloud extraction module is configured to align the RGB image and depth image in the segmented region of the target object, filter out the region of the target object from the aligned target scene point cloud, and extract the point cloud of the target object. The pre-constraint module is configured to construct a candidate set based on the point cloud of the target object, and limit the value range of the grasping center to within the candidate set, serving as a pre-constraint for the grasping candidates; The grasping candidate generation module is configured to generate multiple grasping candidate poses corresponding to each grasping center point selected based on pre-constraints, combined with the point cloud of the target object, and construct a grasping contact area corresponding to the grasping candidate poses; the pose selection module is configured to identify the proportion of the target object point cloud in the grasping contact area, construct a post-constraint for the target consistency of the grasping pose, select the grasping candidate poses, score the selected grasping candidate poses, and select the optimal pose.

9. The end-effector pose generation system based on target consistency constraints as described in claim 8, characterized in that, The candidate generation module includes a multi-scale feature modeling module, which is configured to construct nested cylindrical neighborhoods of multiple scales for each grasping center point, sample point cloud data, extract features, and then fuse them to obtain multi-scale fused features corresponding to each grasping center point. The explicit geometric feature enhancement module is configured to compute geometric features based on the normal, curvature, and density of the center neighborhood points, and fuse them with multi-scale features extracted from the network. The features are concatenated to obtain enhanced features; the generation module is configured to generate multiple grasping candidate poses based on the enhanced features for each grasping center point.

10. An end-effector pose generation system based on target consistency constraints, characterized in that, include: An image acquisition device, a grasping execution end, and a processor; the processor is configured to perform the steps of the end pose generation method based on target consistency constraints according to any one of claims 1-7, generating a grasping pose to control the grasping execution end to perform a grasping action.

Images