Three-dimensional vision-guided robotic arm adaptive grasping method and system

Abstract

The application provides a three-dimensional vision-guided mechanical arm adaptive grasping method and system, which predicts a grasping pose by improving an attention mechanism to fuse local geometry and global constraints, after extracting local geometric features of sampling points, using surface normal and principal curvature information to construct key and value vectors containing rich local surface morphology, using global position and neighborhood information of the sampling points to generate a query vector, the core lies in using a grasping compatibility matrix based on the kinematics and geometric constraints of the mechanical arm end, the matrix is used as prior knowledge to modulate the initial attention score calculated by Q and K, generate an attention weight fused with feasibility, weight V based on the weight, generate a context-enhanced feature for each point, through a pose regression network, parallelly solve the grasping pose parameters and confidence corresponding to each point, and output the optimal grasping scheme through sorting and screening.

Description

Technical Field

[0001] This application belongs to the field of grasping, and in particular relates to a three-dimensional vision-guided adaptive grasping method and system for robotic arms. Background Technology

[0002] Traditional robotic arm grasping methods largely rely on 3D models of objects and predefined grasping poses. While this model-based approach offers high accuracy in specific scenarios, it imposes strict requirements on the geometric and physical properties of the object, making it unsuitable for handling diverse, unknown objects in unstructured environments and exhibiting poor generalization ability. Data-driven grasping methods, on the other hand, typically learn grasping strategies from point cloud or image data acquired by sensors. By constructing deep neural networks, they predict feasible grasping poses end-to-end from the raw data. For example, some methods utilize point cloud feature extraction networks to generate feature representations for each point in the scene, and then use a grasping evaluation network to regress the grasping quality and corresponding grasping parameters for each point.

[0003] At the feature representation level, many methods lack fine-grained perception of local object geometry and extract relatively general features, failing to fully integrate microscopic geometric information such as surface normals and curvature, which are crucial for grasping stability. This results in inaccurate assessment of grasping pose when dealing with objects with complex surfaces or edge features. Most deep learning models implicitly learn geometric relationships from data during the learning process, without explicitly modeling the characteristics of the robotic arm's end effector. For example, a visually plausible grasping point pair may be impractical due to gripper size, opening / closing range, or kinematic limitations. Therefore, integrating the robot's physical constraints into the feature extraction and relational reasoning processes of deep learning to achieve deep coupling between geometric perception and feasibility is key to improving the performance of 3D vision-guided grasping methods. Summary of the Invention

[0004] To address the challenges of existing technologies in integrating the physical constraints of robots into the feature extraction and relational reasoning processes of deep learning, and in achieving deep coupling between geometric perception and feasibility.

[0005] In a first aspect, the present invention provides a three-dimensional vision-guided adaptive grasping method for a robotic arm, comprising the following steps: The three-dimensional point cloud data of the target object is acquired and sampled. Initial local geometric features are extracted for each sampled point. A query vector Q is constructed for each sampled point. The query vector Q integrates the global position information of the sampled point in the object coordinate system with the local geometric information in the neighborhood. For each sampling point, the surface normal and principal curvature are calculated, and the initial local geometric features are modulated using the normal and curvature to generate a key vector K and a value vector V that encode the local surface morphology; based on the kinematic model and geometric constraints of the robotic arm end effector, a grasping compatibility matrix is ​​generated for the sampling point pair. An initial attention score is calculated using the query vector Q and the key vector K, and the initial attention score is biased using the crawling compatibility matrix to generate an attention weight matrix; the value vector V is weighted and aggregated based on the attention weight matrix, and a context enhancement feature is output for each sampling point. The context-enhanced features are input into a pose regression network, which calculates the corresponding grasp confidence and grasp pose parameters for each sampling point. All calculated grasp poses are sorted and filtered according to the grasp confidence, and the optimal grasp pose is output.

[0006] Optionally, the step of acquiring the three-dimensional point cloud data of the target object and sampling the point cloud to extract initial local geometric features for each sampling point includes: The farthest point sampling method is used to sample N points from the original 3D point cloud data; For each sampling point, the k-nearest neighbor algorithm or ball query algorithm is used to search for neighboring points to form a local point set; The three-dimensional coordinates of all points in the local point set are centered relative to the sampling point; The coordinate information of the centered local point set is input into a feature extraction network with shared weights, and a high-dimensional vector is output as the initial local geometric features of the sampling points.

[0007] Optionally, the step of constructing a query vector Q for each sampling point, wherein the query vector Q integrates the global position information of the sampling point in the object coordinate system with the local geometric information in the neighborhood, including: The centroid of the 3D point cloud of the target object is normalized, and the geometric center of all points is calculated as the origin of the coordinate system. For each sampling point, the normalized coordinates of the sampling point are used as global position information; For each sampling point, calculate the relative position vector between the sampling point and all points in the neighborhood, and use the aggregation result of all relative position vectors as local geometric information; The global location information and local geometric information are concatenated and then nonlinearly mapped through a multilayer perceptron network to generate the query vector Q.

[0008] Optionally, the step of calculating the surface normal and principal curvature for each sampling point, and using the normal and curvature to modulate the initial local geometric features to generate a key vector K and a value vector V encoding the local surface morphology, includes: Principal component analysis is performed on the neighborhood point set of each sampling point to obtain the eigenvalues ​​and eigenvectors of the covariance matrix. The eigenvector corresponding to the smallest eigenvalue is used as the surface normal of the sampling point, and the principal curvature is calculated using the eigenvalues. The initial local geometric features, surface normal vector, and principal curvature values ​​are concatenated into a composite feature vector. The composite feature vector is input into two independent multilayer perceptron networks, one of which outputs a key vector K and the other outputs a value vector V.

[0009] Optionally, the kinematic model and geometric constraints based on the robotic arm end effector generate a grasping compatibility matrix for the sampling point pairs, including: For any pair of sampling points Determine whether the sampling point pair simultaneously meets the following preset conditions: The Euclidean distance between two points lies between the minimum and maximum opening width of the end effector; The angle between the surface normal vectors of two points is greater than a preset angle threshold. Based on the 3D model of the end effector, when the grasping posture is determined by the sampling point pair, the end effector model does not interfere with the point cloud of the target object; When a sampling point pair satisfies all of the above conditions, the corresponding element M(i,j) in the capture compatibility matrix M is assigned a value of 0; otherwise, it is assigned a sufficiently small negative number.

[0010] Optionally, the step of calculating an initial attention score using the query vector Q and key vector K, and then biasing the initial attention score using the crawling compatibility matrix to generate an attention weight matrix, includes: The initial attention score matrix S is obtained by calculating the dot product of the query vector Q and the transpose of the key vector K using matrix multiplication. The initial attention score matrix S and the crawling compatibility matrix M are added element-wise to obtain the biased attention score matrix. The attention weight matrix is ​​generated by normalizing each row of the biased attention score matrix using the Softmax function.

[0011] Optionally, the step of inputting the context enhancement features into a pose regression network, wherein the network calculates the corresponding grasp confidence and grasp pose parameters for each sampling point, including: The pose regression network consists of a shared backbone network and three independent branch networks; The shared backbone network performs deep encoding on the input context-enhanced features; The first branch head network outputs a scalar in the range [0,1], which serves as the capture confidence score; The second branch network outputs a seven-dimensional vector, where three dimensions represent the offset of the grasp center point relative to the sampling point, and four dimensions represent the rotation quaternion of the grasp posture. The third branch head network outputs a scalar value, which is constrained between the minimum and maximum opening width of the end effector, representing the opening width of the end effector.

[0012] Optionally, the step of sorting and filtering all calculated grasping poses according to grasping confidence and outputting the optimal grasping pose includes: Set a grasp confidence threshold and remove all grasp poses with a confidence level lower than the threshold. The physical feasibility of the remaining grasping poses is verified, and grasping poses in which the end effector collides with the point cloud of the target object are eliminated. The remaining grasping poses are sorted in descending order of confidence level; The non-maximum suppression algorithm is applied to remove redundancy from the sorted grasping poses: starting from the grasping pose with the highest confidence, other poses that are too close to the current pose in terms of spatial position and rotation are iteratively removed. The grasping pose with the highest confidence after being filtered by the suppression algorithm is output as the optimal grasping pose.

[0013] In another aspect, the present invention also proposes a three-dimensional vision-guided adaptive grasping system for a robotic arm, comprising the following modules: The construction module is used to acquire the three-dimensional point cloud data of the target object, sample the point cloud, extract initial local geometric features for each sampled point, and construct a query vector Q for each sampled point. The query vector Q integrates the global position information of the sampled point in the object coordinate system and the local geometric information in the neighborhood. The generation module is used to calculate the surface normal and principal curvature for each sampling point, and use the normal and curvature to modulate the initial local geometric features to generate a key vector K and a value vector V that encode the local surface morphology; based on the kinematic model and geometric constraints of the robotic arm end effector, a grasping compatibility matrix is ​​generated for the sampling point pair. The calculation module is used to calculate an initial attention score using the query vector Q and the key vector K, and to bias the initial attention score using the crawling compatibility matrix to generate an attention weight matrix; based on the attention weight matrix, the value vector V is weighted and aggregated to output a context enhancement feature for each sampling point; The output module is used to input the context enhancement features into a pose regression network, which calculates the corresponding grasp confidence and grasp pose parameters for each sampling point; all calculated grasp poses are sorted and filtered according to the grasp confidence, and the optimal grasp pose is output.

[0014] Further, the acquisition of the three-dimensional point cloud data of the target object, and the sampling of the point cloud to extract initial local geometric features for each sampling point, includes: The farthest point sampling method is used to sample N points from the original 3D point cloud data; For each sampling point, the k-nearest neighbor algorithm or ball query algorithm is used to search for neighboring points to form a local point set; The three-dimensional coordinates of all points in the local point set are centered relative to the sampling point; The coordinate information of the centered local point set is input into a feature extraction network with shared weights, and a high-dimensional vector is output as the initial local geometric features of the sampling points.

[0015] Furthermore, a query vector Q is constructed for each sampling point. The query vector Q integrates the global position information of the sampling point in the object coordinate system with the local geometric information within its neighborhood, including: The centroid of the 3D point cloud of the target object is normalized, and the geometric center of all points is calculated as the origin of the coordinate system. For each sampling point, the normalized coordinates of the sampling point are used as global position information; For each sampling point, calculate the relative position vector between the sampling point and all points in the neighborhood, and use the aggregation result of all relative position vectors as local geometric information; The global location information and local geometric information are concatenated and then nonlinearly mapped through a multilayer perceptron network to generate the query vector Q.

[0016] Further, the step of calculating the surface normal and principal curvature for each sampling point, and using the normal and curvature to modulate the initial local geometric features to generate a key vector K and a value vector V encoding the local surface morphology, includes: Principal component analysis is performed on the neighborhood point set of each sampling point to obtain the eigenvalues ​​and eigenvectors of the covariance matrix. The eigenvector corresponding to the smallest eigenvalue is used as the surface normal of the sampling point, and the principal curvature is calculated using the eigenvalues. The initial local geometric features, surface normal vector, and principal curvature values ​​are concatenated into a composite feature vector. The composite feature vector is input into two independent multilayer perceptron networks, one of which outputs a key vector K and the other outputs a value vector V.

[0017] Furthermore, the kinematic model and geometric constraints based on the robotic arm end effector generate a grasping compatibility matrix for the sampling point pairs, including: For any pair of sampling points Determine whether the sampling point pair simultaneously meets the following preset conditions: The Euclidean distance between two points lies between the minimum and maximum opening width of the end effector; The angle between the surface normal vectors of two points is greater than a preset angle threshold. Based on the 3D model of the end effector, when the grasping posture is determined by the sampling point pair, the end effector model does not interfere with the point cloud of the target object; When a sampling point pair satisfies all of the above conditions, the corresponding element M(i,j) in the capture compatibility matrix M is assigned a value of 0; otherwise, it is assigned a sufficiently small negative number.

[0018] Further, the step of calculating an initial attention score using the query vector Q and key vector K, and then biasing the initial attention score using the crawling compatibility matrix to generate an attention weight matrix, includes: The initial attention score matrix S is obtained by calculating the dot product of the query vector Q and the transpose of the key vector K using matrix multiplication. The initial attention score matrix S and the crawling compatibility matrix M are added element-wise to obtain the biased attention score matrix. The attention weight matrix is ​​generated by normalizing each row of the biased attention score matrix using the Softmax function.

[0019] Further, the context enhancement features are input into a pose regression network, which calculates the corresponding grasp confidence and grasp pose parameters for each sampling point, including: The pose regression network consists of a shared backbone network and three independent branch networks; The shared backbone network performs deep encoding on the input context-enhanced features; The first branch head network outputs a scalar in the range [0,1], which serves as the capture confidence score; The second branch network outputs a seven-dimensional vector, where three dimensions represent the offset of the grasp center point relative to the sampling point, and four dimensions represent the rotation quaternion of the grasp posture. The third branch head network outputs a scalar value, which is constrained between the minimum and maximum opening width of the end effector, representing the opening width of the end effector.

[0020] Furthermore, the process of sorting and filtering all calculated grasping poses based on grasping confidence and outputting the optimal grasping pose includes: Set a grasp confidence threshold and remove all grasp poses with a confidence level lower than the threshold. The physical feasibility of the remaining grasping poses is verified, and grasping poses in which the end effector collides with the point cloud of the target object are eliminated. The remaining grasping poses are sorted in descending order of confidence level; The non-maximum suppression algorithm is applied to remove redundancy from the sorted grasping poses: starting from the grasping pose with the highest confidence, other poses that are too close to the current pose in terms of spatial position and rotation are iteratively removed. The grasping pose with the highest confidence after being filtered by the suppression algorithm is output as the optimal grasping pose.

[0021] This invention achieves deep, multi-dimensional perception of object surface features by fusing global position and local geometric information of sampling points into the query vector of the attention mechanism, and modulating the key and value vectors using surface normals and principal curvatures. It utilizes a grasping compatibility matrix based on robotic arm kinematics and geometric constraints, and uses this matrix as prior knowledge to bias the attention score, enabling the attention mechanism to focus on physically feasible grasping point pairs and avoid paying attention to unreasonable areas. The generated context-enhanced features include not only geometric context but also feasibility constraints, thereby improving the accuracy of the pose regression network in calculating the grasping pose and ensuring the success rate of the output grasping scheme. Attached Figure Description

[0022] Figure 1 A flowchart of a three-dimensional vision-guided adaptive grasping method for a robotic arm provided in an embodiment of the present invention; Figure 2 A schematic diagram of a query vector Q generation architecture based on global location and local geometry provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the crawling parameter prediction structure based on a shared backbone network provided in an embodiment of the present invention. Detailed Implementation

[0023] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0024] In this application, a three-dimensional vision-guided adaptive grasping method for a robotic arm is described, such as... Figure 1 As shown, it includes the following steps: S1, acquire the 3D point cloud data of the target object, sample the point cloud, and extract the initial local geometric features for each sampled point; The target object is scanned using an RGB-D camera such as an Intel RealSense series or Microsoft Kinect to obtain the original 3D point cloud. The farthest point sampling algorithm (FPS) is used to downsample the original point cloud, resulting in N sampling points. For each sampling point, the k-nearest neighbor algorithm (k-NN) is used to search for k nearest neighbors within the sampling point's neighborhood. The relative coordinates of these k nearest neighbors are used as input, and an initial 256-dimensional local geometric feature vector is extracted through a multilayer perceptron (MLP) network with shared weights, namely PointNet.

[0025] In an optional embodiment, acquiring the 3D point cloud data of the target object and sampling the point cloud to extract initial local geometric features for each sampling point includes: The farthest point sampling method is used to sample N points from the original 3D point cloud data; For each sampling point, the k-nearest neighbor algorithm or ball query algorithm is used to search for neighboring points to form a local point set; The three-dimensional coordinates of all points in the local point set are centered relative to the sampling point; The coordinate information of the centered local point set is input into a feature extraction network with shared weights, and a high-dimensional vector is output as the initial local geometric features of the sampling points.

[0026] The input raw 3D point cloud is normalized, for example, by scaling it to a unit sphere to enhance the algorithm's adaptability to objects of different scales. The furthest-point sampling method is used to select N sampling points with good coverage from tens of thousands of raw points, where the preferred range for N is 512 to 2048, and in a typical embodiment, N is set to 1024. For each of the 1024 sampling points, the k-nearest neighbor algorithm is used to search for the k nearest neighbors of that sampling point, forming a local neighborhood. The parameter k ranges from 16 to 64, and is typically set to 32 to strike a balance between detecting sufficient local detail and controlling computational complexity.

[0027] After obtaining the local neighborhood point set for each sampling point, a centering process is performed to achieve translation invariance. The centered coordinates are fed into a shared-weight feature extraction network, which is a neural network based on a multilayer perceptron. The structure consists of a series of shared-weight multilayer perceptron layers that independently transform the features of each neighborhood point, for example, with layer dimensions of 64, 128, or 256. Then, the features of all neighborhood points are aggregated through a max-pooling symmetric function. The input of the network is a tensor of centered neighborhood coordinates for N sampling points, with a dimension of N×k×3, where N is the number of sampling points and k is the number of neighborhood points. The output of the network is the initial local geometric features of the N sampling points. The features are a high-dimensional feature tensor with a dimension of N×D, where D is the feature dimension, for example, 256. The features encode the microscopic geometric structure around the point.

[0028] S2, construct a query vector Q for each sampling point, the query vector Q is a fusion of the global position information of the sampling point in the object coordinate system and the local geometric information in the neighborhood; The 3D coordinates (x, y, z) of all sampling points are normalized to obtain global position information. The `torch.cat` function is used to concatenate the 3D global position information of each sampling point with the 256-dimensional initial local geometric feature vector extracted in the previous step, resulting in a 259-dimensional fused feature vector. This fused feature vector is then mapped through a linear transformation layer to generate the query vector Q.

[0029] In some embodiments, the step of constructing a query vector Q for each sampling point, wherein the query vector Q integrates the global position information of the sampling point in the object coordinate system with the local geometric information in the neighborhood, including: The centroid of the 3D point cloud of the target object is normalized, and the geometric center of all points is calculated as the origin of the coordinate system. For each sampling point, the normalized coordinates of the sampling point are used as global position information; For each sampling point, calculate the relative position vector between the sampling point and all points in the neighborhood, and use the aggregation result of all relative position vectors as local geometric information; The global location information and local geometric information are concatenated and then nonlinearly mapped through a multilayer perceptron network to generate the query vector Q.

[0030] Before constructing the query vector Q, the centroid C of the entire target object point cloud is calculated, and the coordinates of all points are subtracted from the centroid coordinates to achieve centroid normalization of the point cloud. For any point among N sampling points... Normalized three-dimensional coordinates This information will be used as the global position information for that point. This information provides an absolute position reference for each point in the object's own coordinate system, which helps the attention mechanism understand the global distribution of points. Simultaneously, the information from the previous step will be used as the global position information for that point. The extracted 256-dimensional initial local geometric features serve as local geometric information, which already contain rich representations of the neighborhood structure.

[0031] The 3D global location information is concatenated with the 256-dimensional local geometric information to obtain a 259-dimensional composite vector. This composite vector is then input into a multilayer perceptron network to generate the query vector Q, such as... Figure 2 As shown, the network structure is a multilayer perceptron with two layers, for example, with a layer dimension of 256 or 128. The input of the network is a composite feature vector for each sampling point, and the data format is a tensor with dimension N×(3+D), where N is the number of sampling points and D is the dimension of local geometric features. The output of the network is a query vector Q for each sampling point, and the data format is a tensor with dimension N×(3+D). The tensor, in which The query vector dimension is, for example, 128.

[0032] S3, calculate the surface normal and principal curvature for each sampling point, and use the normal and curvature to modulate the initial local geometric features to generate a key vector K and a value vector V that encode the local surface morphology; For each sampling point and its k nearest neighbors, Principal Component Analysis (PCA) is applied to calculate the covariance matrix. The eigenvector corresponding to the smallest eigenvalue of the covariance matrix is ​​the surface normal vector n at that point. Curvature is calculated based on the eigenvalues. The 3D normal vector and 1D curvature information of each sampling point are concatenated with the 256-dimensional initial local geometric feature vector to form a 260-dimensional modulation feature vector. This modulation feature vector is then passed through two independent linear transformation layers to generate the key vector K and the value vector V.

[0033] In one embodiment, the step of calculating the surface normal and principal curvature for each sampling point, and using the normal and curvature to modulate the initial local geometric features to generate a key vector K and a value vector V encoding the local surface morphology, includes: Principal component analysis is performed on the neighborhood point set of each sampling point to obtain the eigenvalues ​​and eigenvectors of the covariance matrix. The eigenvector corresponding to the smallest eigenvalue is used as the surface normal of the sampling point, and the principal curvature is calculated using the eigenvalues. The initial local geometric features, surface normal vector, and principal curvature values ​​are concatenated into a composite feature vector. The composite feature vector is input into two independent multilayer perceptron networks, one of which outputs a key vector K and the other outputs a value vector V.

[0034] For each sampling point and its k neighboring points, the local surface geometric properties are estimated using principal component analysis. The 3×3 covariance matrix of this point set is calculated, and eigenvalue decomposition is performed on the matrix to obtain three eigenvalues. and the corresponding eigenvectors. Among them, the smallest eigenvalue will be... The corresponding eigenvector is considered as the surface normal vector n at that sampling point, and its direction is kept consistent with the viewpoint direction to eliminate directional ambiguity. Simultaneously, the eigenvalues ​​can be used to calculate the surface variability or principal curvature; a commonly used curvature measure is... , It is a scalar quantity that represents the flatness of the surface.

[0035] The previously extracted 256-dimensional initial local geometric features, the calculated 3-dimensional surface normal vector n, and the 1-dimensional curvature value are combined. The vectors are concatenated to form a 260-dimensional composite feature vector. This vector is then input into two independent multilayer perceptron networks to generate a key vector K and a value vector V. The first network is a key vector generation network, structured as a multilayer perceptron. Its input is a 260-dimensional composite feature vector for each sampling point, with data in the form of a tensor of dimension N×(D+3+1). The network's output is the key vector K, with data in the form of a tensor of dimension N×(D+3+1). The tensor, in which The first network is the key vector dimension, for example, 128. The second network is a value vector generation network, which has the structure of another independent multilayer perceptron. The input of this network is the same as the key vector generation network, and the output is a value vector V, with a data format of one dimension. The tensor, in which The dimension of the value vector, for example, 256.

[0036] S4, based on the kinematic model and geometric constraints of the robotic arm end effector, generates a grasping compatibility matrix for the sampling point pairs; Create an N×N zero matrix as the capture compatibility matrix. Iterate through all sample point pairs. and Perform the following constraint checks: Calculate the Euclidean distance between the two points and determine whether the distance is between the minimum and maximum opening width of the robotic arm's end effector. Calculate the surface normals at the two points. and The dot product is calculated, and it is determined whether the dot product is less than a preset negative threshold to ensure that the normal directions are approximately opposite. A system is constructed based on... and The algorithm creates a 3D model of the gripper at the contact point and performs collision detection using the Flexible Collision Library (FCL) to determine whether the gripper model collides with the remaining point cloud of the target object. If any check fails, the element at the corresponding position in the matrix is ​​set to a sufficiently small negative number, such as -1e9; if all checks pass, the element remains zero.

[0037] In some embodiments of this application, the kinematic model and geometric constraints based on the robotic arm end effector generate a grasping compatibility matrix for the sampling point pairs, including: For any pair of sampling points Determine whether the sampling point pair simultaneously meets the following preset conditions: The Euclidean distance between two points lies between the minimum and maximum opening width of the end effector; The angle between the surface normal vectors of two points is greater than a preset angle threshold to ensure that the directions of the two normals are approximately opposite. Based on the 3D model of the end effector, when the grasping posture is determined by the sampling point pair, the end effector model does not interfere with the point cloud of the target object; When a sampling point pair satisfies all of the above conditions, the corresponding element M(i,j) in the capture compatibility matrix M is assigned a value of 0; otherwise, it is assigned a sufficiently small negative number.

[0038] For each element M(i,j) in the matrix, it corresponds to a pair of sampling points. Feasibility of potential grabbing contact points. Perform distance constraint check: calculate the Euclidean distance between the two points. And verify whether the distance falls within the working range of the gripper at the end of the robotic arm, i.e. For example, for a typical parallel gripper, the width range might be [0.01m, 0.08m]. Perform a normal opposition constraint check: calculate the surface normals at two points. and The angle between To ensure stable opposing grasping, this included angle needs to be greater than a threshold. The preferred range of the threshold is 120° to 170°, with a typical value of 150°.

[0039] Perform collision-free constraint checks: based on point pairs and normal Construct a complete 6-DOF candidate grasping pose. The grasping center is defined as the midpoint between two points. The three orthogonal axes of the grasping coordinate system can be defined as: the principal axes along... point to The direction; the approach direction is through the average vector of the normal. The third axis is obtained by performing Gram-Schmidt orthogonalization, ensuring it is perpendicular to the principal axes; the third axis is obtained by the cross product of the x and z axes. The 3D geometric model of the end effector is placed on this candidate pose, and the model is checked for volumetric intersections or interferences with the complete point cloud of the target object. Only when a point pair... A point pair is considered physically compatible only when it meets all three constraints: distance, normal, and no collision. In this case, the corresponding elements M(i,j) and M(j,i) in matrix M are assigned a value of 0. For any pair of points that fails to meet at least one condition, the corresponding matrix element is assigned a sufficiently small negative value, such as -1e9, to mask the pair in subsequent attention calculations.

[0040] S5, calculate the initial attention score using the query vector Q and the key vector K, and use the crawling compatibility matrix to bias the initial attention score to generate an attention weight matrix; The `torch.matmul` function is used to calculate the dot product of the query vector matrix Q and the transpose of the key vector matrix K, yielding the initial attention score matrix. This score matrix is ​​then scaled by dividing it by the square root of the key vector dimension. The crawling compatibility matrix generated in the previous step is added to the scaled initial attention score matrix to achieve biasing. Finally, the softmax function is applied to each row of the biased score matrix to calculate the attention weight matrix.

[0041] In an optional embodiment, the step of calculating an initial attention score using the query vector Q and key vector K, and then biasing the initial attention score using the crawling compatibility matrix to generate an attention weight matrix, includes: The initial attention score matrix S is obtained by calculating the dot product of the query vector Q and the transpose of the key vector K using matrix multiplication. The initial attention score matrix S and the crawling compatibility matrix M are added element-wise to obtain the biased attention score matrix. The attention weight matrix is ​​generated by normalizing each row of the biased attention score matrix using the Softmax function.

[0042] The 128-dimensional query vectors from all N sampling points are combined into an N×128 matrix Q, and the N key vectors are combined into an N×128 matrix K. The transposes of Q and K are then calculated. The dot product of these values ​​yields an N×N initial attention score matrix. To stabilize the training process, the score is typically scaled, i.e. ,in This is the dimension of the key vector, therefore the scaling factor is approximately 11.3. Matrix Each element in the expression represents the original attention intensity of sampling point i to sampling point j, which is determined purely by the learned feature similarity.

[0043] To incorporate physical constraints into the attention mechanism, the scaled score matrix obtained in the previous step... The attention score matrix is ​​obtained by element-wise addition to the pre-calculated crawling compatibility matrix M, resulting in a biased attention score matrix. For physically incompatible grab point pairs (i,j), since M(i,j) is a sufficiently small negative number, This will also become a sufficiently small negative number; and for compatible point pairs, M(i,j) is 0, and the original attention score... Remain unchanged. (Regarding) Each row of the matrix is ​​independently normalized using the Softmax function. This causes the attention weights corresponding to incompatible point pairs to approach 0 after exponential operation, while the weights of compatible point pairs are redistributed based on feature similarity. The resulting N×N matrix is ​​the attention weight matrix, which guides the model to interact only between physically feasible sampled point pairs.

[0044] S6, Based on the attention weight matrix, the value vector V is weighted and aggregated to output a context enhancement feature for each sampling point; The `torch.matmul` function is used to multiply the attention weight matrix generated in the previous step with the value vector matrix V. This matrix multiplication operation performs a weighted summation of the value vector V for each sampling point, with the weights being the attention weights. This generates a context-enhanced feature vector for each sampling point that aggregates global contextual information and feasibility information.

[0045] S7, The context enhancement features are input into a pose regression network, and the network calculates the corresponding grasp confidence and grasp pose parameters for each sampling point; A multilayer perceptron consisting of multiple torch.nn.Linear fully connected layers and ReLU activation functions is constructed as the pose regression network. This network takes the contextual augmentation features at each sampling point as input and sets up multiple output heads. The first output head contains a single neuron that outputs a grasp confidence score between 0 and 1 through a sigmoid activation function. The second output head outputs seven-dimensional grasp pose parameters: a three-dimensional offset plus a four-dimensional rotation quaternion. The third output head outputs the grasp width.

[0046] In some embodiments of this application, the step of inputting the context enhancement features into a pose regression network, wherein the network calculates the corresponding grasp confidence and grasp pose parameters for each sampling point, includes: The pose regression network consists of a shared backbone network and three independent branch networks; The shared backbone network performs deep encoding on the input context-enhanced features; The first branch head network outputs a scalar in the range [0,1], which serves as the capture confidence score; The second branch network outputs a seven-dimensional vector, where three dimensions represent the offset of the grasp center point relative to the sampling point, and four dimensions represent the rotation quaternion of the grasp posture. The third branch head network outputs a scalar value, which is constrained between the minimum and maximum opening width of the end effector, representing the opening width of the end effector.

[0047] The pose regression network is a neural network consisting of a shared backbone network and three independent branch networks. The shared backbone network comprises several multilayer perceptron layers, for example, with layer dimensions of 256, 512, and 256. This backbone network is then connected to three branch networks: a confidence prediction head (a multilayer perceptron network with layer dimensions of, for example, 256, 64, and 1, followed by a sigmoid activation function); a pose parameter prediction head (a multilayer perceptron network with layer dimensions of, for example, 256, 128, and 7); and a grasp width prediction head (a multilayer perceptron network with layer dimensions of, for example, 256, 64, and 1, followed by a custom activation function to constrain the output range, such as...). .

[0048] The pose regression network is input to N sampling points of context enhancement features, and the data format is a one-dimensional feature. The tensor, in which To enhance the contextual feature dimension, for example, to 256; the network outputs the grasping solution for each sampling point, including: a grasping confidence scalar in the range [0,1] output by the confidence prediction head; and a seven-dimensional vector output by the pose parameter prediction head, where the three dimensions represent the offset of the grasping center point relative to the sampling point. The four-dimensional rotational quaternion q represents the grasping posture; a scalar output by the grasping width prediction head represents the opening width w of the end effector, the width value being constrained by... Between, such as Figure 3 As shown.

[0049] The aforementioned pose regression network and attention mechanism-related feature mapping network can be trained end-to-end based on public datasets or custom datasets for robot grasping. The training process uses the AdamW optimizer, and the loss function is a weighted combination of the binary cross-entropy loss of grasping confidence, the L2 loss of grasping pose offset and rotation quaternion, and the L2 loss of grasping width. The network parameters are updated through backpropagation until convergence.

[0050] S8 sorts and filters all calculated grasping poses according to grasping confidence and outputs the optimal grasping pose.

[0051] Collect the grasping poses generated from all N sampling points and their corresponding grasping confidence scores. Sort all grasping poses from highest to lowest grasping confidence score. Apply Non-Maximum Suppression (NMS) algorithm for filtering: retain the grasping pose with the highest confidence score and remove all other grasping poses that are too close to it in spatial position and rotation direction. Repeat this process until the list is empty. Output the grasping pose with the highest confidence score after NMS filtering as the optimal grasping pose.

[0052] In some embodiments, the step of sorting and filtering all calculated grasping poses according to grasping confidence and outputting the optimal grasping pose includes: Set a grasp confidence threshold and remove all grasp poses with a confidence level lower than the threshold. The physical feasibility of the remaining grasping poses is verified, and grasping poses in which the end effector collides with the point cloud of the target object are eliminated. The remaining grasping poses are sorted in descending order of confidence level; The non-maximum suppression algorithm is applied to remove redundancy from the sorted grasping poses: starting from the grasping pose with the highest confidence, other poses that are too close to the current pose in terms of spatial position and rotation are iteratively removed. The grasping pose with the highest confidence after being filtered by the suppression algorithm is output as the optimal grasping pose.

[0053] Perform initial filtering on the N candidate crawls output by the network, and set a confidence threshold. For example, a confidence level of 0.8 is used to discard all grasping schemes with a confidence level below this threshold. A fine-grained physical feasibility verification is then performed on the grasping schemes that pass the initial filtering. Although the previous compatibility matrix has been pre-screened, the poses regressed by the network may have slight deviations. This step serves as a safety check to ensure that the generated poses are physically feasible, and any grasping schemes that interfere with the object will be eliminated.

[0054] After the above filtering, the remaining grasping poses are sorted in descending order of confidence score. To eliminate redundant grasping poses that are spatially similar, a non-maximum suppression algorithm is used. This algorithm starts with the grasping pose with the highest confidence score. Begin by using the aforementioned crawl as a baseline and then iterating through the remaining crawls in the list. Through calculation and The comparison is made based on differences in translation and rotation. The translation difference is represented by calculating the Euclidean distance between the two gripping centers; if it is less than a distance threshold... If the positions are close, they are considered similar. Rotational difference is represented by the absolute value of the dot product of the quaternions of the two poses. If this value is greater than the rotational similarity threshold, it is equivalent to the included angle being less than a threshold. For example, if the angle corresponds to 15 degrees, the posture is considered similar. When a grasping... and When the proximity condition is satisfied simultaneously in translation and rotation, Redundant positions are considered redundant and removed from the list. This process is repeated iteratively until the list is fully traversed. After NMS processing, the position with the highest confidence in the list is determined as the optimal grasping pose and is output to the robotic arm for execution.

[0055] Based on any of the above embodiments, a three-dimensional vision-guided adaptive grasping method for a robotic arm is provided. This application also provides a three-dimensional vision-guided adaptive grasping system for a robotic arm, comprising the following modules: The construction module is used to acquire the three-dimensional point cloud data of the target object, sample the point cloud, extract initial local geometric features for each sampled point, and construct a query vector Q for each sampled point. The query vector Q integrates the global position information of the sampled point in the object coordinate system and the local geometric information in the neighborhood. The generation module is used to calculate the surface normal and principal curvature for each sampling point, and use the normal and curvature to modulate the initial local geometric features to generate a key vector K and a value vector V that encode the local surface morphology; based on the kinematic model and geometric constraints of the robotic arm end effector, a grasping compatibility matrix is ​​generated for the sampling point pair. The calculation module is used to calculate an initial attention score using the query vector Q and the key vector K, and to bias the initial attention score using the crawling compatibility matrix to generate an attention weight matrix; based on the attention weight matrix, the value vector V is weighted and aggregated to output a context enhancement feature for each sampling point; The output module is used to input the context enhancement features into a pose regression network, which calculates the corresponding grasp confidence and grasp pose parameters for each sampling point; all calculated grasp poses are sorted and filtered according to the grasp confidence, and the optimal grasp pose is output.

[0056] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0057] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A three-dimensional vision-guided adaptive grasping method for a robotic arm, characterized in that, Includes the following steps: Acquire the 3D point cloud data of the target object, sample the point cloud, and extract initial local geometric features for each sampled point; A query vector Q is constructed for each sampling point. The query vector Q integrates the global position information of the sampling point in the object coordinate system with the local geometric information in the neighborhood. For each sampling point, the surface normal and principal curvature are calculated, and the initial local geometric features are modulated using the normal and curvature to generate a key vector K and a value vector V that encode the local surface morphology; based on the kinematic model and geometric constraints of the robotic arm end effector, a grasping compatibility matrix is ​​generated for the sampling point pair. An initial attention score is calculated using the query vector Q and the key vector K, and the initial attention score is biased using the crawling compatibility matrix to generate an attention weight matrix; the value vector V is weighted and aggregated based on the attention weight matrix, and a context enhancement feature is output for each sampling point. The context-enhanced features are input into a pose regression network, which calculates the corresponding grasp confidence and grasp pose parameters for each sampling point. All calculated grasp poses are sorted and filtered according to the grasp confidence, and the optimal grasp pose is output.

2. The method according to claim 1, characterized in that, The process of acquiring 3D point cloud data of the target object and sampling the point cloud to extract initial local geometric features for each sampling point includes: The farthest point sampling method is used to sample N points from the original 3D point cloud data; For each sampling point, the k-nearest neighbor algorithm or ball query algorithm is used to search for neighboring points to form a local point set; The three-dimensional coordinates of all points in the local point set are centered relative to the sampling point; The coordinate information of the centered local point set is input into a feature extraction network with shared weights, and a high-dimensional vector is output as the initial local geometric features of the sampling points.

3. The method according to claim 1, characterized in that, The step involves constructing a query vector Q for each sampling point. The query vector Q integrates the global position information of the sampling point in the object coordinate system with the local geometric information within its neighborhood, including: The centroid of the 3D point cloud of the target object is normalized, and the geometric center of all points is calculated as the origin of the coordinate system. For each sampling point, the normalized coordinates of the sampling point are used as global position information; For each sampling point, calculate the relative position vector between the sampling point and all points in the neighborhood, and use the aggregation result of all relative position vectors as local geometric information; The global location information and local geometric information are concatenated and then nonlinearly mapped through a multilayer perceptron network to generate the query vector Q.

4. The method according to claim 1, characterized in that, The step of calculating the surface normal and principal curvature for each sampling point, and using the normal and curvature to modulate the initial local geometric features to generate a key vector K and a value vector V encoding the local surface morphology, includes: Principal component analysis is performed on the neighborhood point set of each sampling point to obtain the eigenvalues ​​and eigenvectors of the covariance matrix. The eigenvector corresponding to the smallest eigenvalue is used as the surface normal of the sampling point, and the principal curvature is calculated using the eigenvalues. The initial local geometric features, surface normal vector, and principal curvature values ​​are concatenated into a composite feature vector. The composite feature vector is input into two independent multilayer perceptron networks, one of which outputs a key vector K and the other outputs a value vector V.

5. The method according to claim 1, characterized in that, The kinematic model and geometric constraints based on the robotic arm end effector generate a grasping compatibility matrix for the sampling point pairs, including: For any pair of sampling points Determine whether the sampling point pair simultaneously meets the following preset conditions: The Euclidean distance between two points lies between the minimum and maximum opening width of the end effector; The angle between the surface normal vectors of two points is greater than a preset angle threshold. Based on the 3D model of the end effector, when the grasping posture is determined by the sampling point pair, the end effector model does not interfere with the point cloud of the target object; When a sampling point pair satisfies all of the above conditions, the corresponding element M(i,j) in the capture compatibility matrix M is assigned a value of 0; otherwise, it is assigned a sufficiently small negative number.

6. The method according to claim 3, characterized in that, The process involves calculating an initial attention score using the query vector Q and the key vector K, and then biasing the initial attention score using the crawling compatibility matrix to generate an attention weight matrix, including: The initial attention score matrix S is obtained by calculating the dot product of the query vector Q and the transpose of the key vector K using matrix multiplication. The initial attention score matrix S and the crawling compatibility matrix M are added element-wise to obtain the biased attention score matrix. The attention weight matrix is ​​generated by normalizing each row of the biased attention score matrix using the Softmax function.

7. The method according to claim 1, characterized in that, The context enhancement features are input into a pose regression network, which calculates the corresponding grasp confidence and grasp pose parameters for each sampling point, including: The pose regression network consists of a shared backbone network and three independent branch networks; The shared backbone network performs deep encoding on the input context-enhanced features; The first branch head network outputs a scalar in the range [0,1], which serves as the capture confidence score; The second branch network outputs a seven-dimensional vector, where three dimensions represent the offset of the grasp center point relative to the sampling point, and four dimensions represent the rotation quaternion of the grasp posture. The third branch head network outputs a scalar value, which is constrained between the minimum and maximum opening width of the end effector, representing the opening width of the end effector.

8. The method according to any one of claims 1-7, characterized in that, The process of sorting and filtering all calculated grasping poses based on grasping confidence and outputting the optimal grasping pose includes: Set a grasp confidence threshold and remove all grasp poses with a confidence level lower than the threshold. The physical feasibility of the remaining grasping poses is verified, and grasping poses in which the end effector collides with the point cloud of the target object are eliminated. The remaining grasping poses are sorted in descending order of confidence level; The non-maximum suppression algorithm is applied to remove redundancy from the sorted grasping poses: starting from the grasping pose with the highest confidence, other poses that are too close to the current pose in terms of spatial position and rotation are iteratively removed. The grasping pose with the highest confidence after being filtered by the suppression algorithm is output as the optimal grasping pose.

9. A three-dimensional vision-guided adaptive grasping system for a robotic arm, characterized in that, Includes the following modules: The module is used to acquire the 3D point cloud data of the target object, sample the point cloud, and extract initial local geometric features for each sampled point. A query vector Q is constructed for each sampling point. The query vector Q integrates the global position information of the sampling point in the object coordinate system with the local geometric information in the neighborhood. The generation module is used to calculate the surface normal and principal curvature for each sampling point, and use the normal and curvature to modulate the initial local geometric features to generate a key vector K and a value vector V that encode the local surface morphology; based on the kinematic model and geometric constraints of the robotic arm end effector, a grasping compatibility matrix is ​​generated for the sampling point pair. The calculation module is used to calculate an initial attention score using the query vector Q and the key vector K, and to bias the initial attention score using the crawling compatibility matrix to generate an attention weight matrix; based on the attention weight matrix, the value vector V is weighted and aggregated to output a context enhancement feature for each sampling point; The output module is used to input the context enhancement features into a pose regression network, which calculates the corresponding grasp confidence and grasp pose parameters for each sampling point; all calculated grasp poses are sorted and filtered according to the grasp confidence, and the optimal grasp pose is output.

10. The system according to claim 9, characterized in that, The process of acquiring 3D point cloud data of the target object and sampling the point cloud to extract initial local geometric features for each sampling point includes: The farthest point sampling method is used to sample N points from the original 3D point cloud data; For each sampling point, the k-nearest neighbor algorithm or ball query algorithm is used to search for neighboring points to form a local point set; The three-dimensional coordinates of all points in the local point set are centered relative to the sampling point; The coordinate information of the centered local point set is input into a feature extraction network with shared weights, and a high-dimensional vector is output as the initial local geometric features of the sampling points.

Images