A robot grasping acquisition method

Abstract

The embodiment of the present application provides a robot grasping acquisition method, comprising: simulating human grasping operation to construct a grasping representation method combining grasping posture and contact force information; establishing a grasping contact model and obtaining stable collision-free grasping constraint conditions, taking the grasping contact model and the stable collision-free grasping constraint conditions as evaluation indexes for constructing robot stable grasping; training a robot grasping force range estimation module according to the mapping relationship between the grasping target and the contact force; then, according to the grasping representation algorithm, the stable grasping evaluation index, the grasping force range estimation module and the grasping posture and grasping force mapping relationship, a grasping generation network integrating visual and tactile is constructed, and the grasping generation network is used for detecting the robot grasping posture and grasping force corresponding to the target. According to the technical scheme provided by the embodiment of the present application, a more optimal robot grasping pose and grasping force in terms of stability, accuracy and the like can be obtained, so as to better drive the robot to realize dexterous grasping.

Description

Technical Field

[0001] This invention relates to the field of robot operation technology, and in particular to a robot grasping and acquiring method. Background Technology

[0002] Grasping and manipulating unknown objects in unstructured real-world environments has been a long-standing challenge in robotics research. Ideally, we hope that robots can propose multiple reliable grasping strategies (grasping poses and grasping forces) based on observed 3D object information, from which collision-free, slip-free, and kinematically feasible movements can be performed. However, many challenges remain to be addressed throughout the entire grasping process, from perception to planning and control.

[0003] Robots typically acquire unstructured environmental information based on visual information provided by their vision systems. Robotic grasping methods using 2.5D and 3D data can better identify the external features of the objects being manipulated. These methods often employ panoramic binocular cameras to acquire point cloud data, combined with multi-stage techniques involving cascaded training or networks with individual losses, to generate the grasping posture of unknown objects. The geometric information carried by depth data enhances the robot's grasping accuracy in unstructured environments. However, while existing 3D-based methods can accurately acquire the visible features of the objects being manipulated, they cannot reliably grasp various unknown targets. The unseen intrinsic properties of objects, such as softness, weight, and surface roughness, cannot be correctly identified using only visual information.

[0004] Studies focusing solely on human grasping techniques have shown that individuals with fingertip paralysis cannot maintain a stable grip, and children with tactile deficits struggle to perform manipulative tasks. Tactile sensors provide robots with information about physical contact, enabling autonomous robotic hands to manipulate unknown objects in unstructured environments. Existing technologies, based on tactile data, propose methods for robot grasping stability detection, object recognition and exploration, and sliding detection. While these methods provide tactile feedback, they cannot be applied to the grasping pose generation stage, as they all require physical contact with the target and do not consider the entire grasping process. From an implementation perspective, there is a lack of a scheme to actively generate tactile grasping force to guide the robot in performing grasping operations of varying dexterity and fine granularity. Therefore, integrating visual-tactile sensing into robotic grasping is a pressing technical problem that needs to be solved. Summary of the Invention

[0005] In view of this, the present invention provides a robot grasping method, which constructs a grasping generation network that integrates visual and tactile perception to obtain the robot grasping posture and grasping force of the corresponding target.

[0006] This invention provides a robot grasping and acquisition method, comprising:

[0007] A grasping representation algorithm is constructed by simulating human grasping operations to generate grasping posture and contact force information.

[0008] Establish a grasping contact model and obtain stable collision-free grasping constraints. Use the grasping contact model and stable collision-free grasping constraints as evaluation indicators for constructing stable robot grasping.

[0009] Based on the mapping relationship between the grasping target and the contact force, train the robot's grasping force range estimation module;

[0010] Based on the grasping representation algorithm, stable grasping evaluation index, grasping force range estimation module, and the mapping relationship between grasping posture and grasping force, a grasping generation network integrating vision and touch is constructed. The grasping generation network is used to detect the robot's grasping posture and grasping force for the corresponding target.

[0011] In the above method, the grasping representation algorithm that constructs grasping posture and contact force information by simulating human grasping operations includes:

[0012] Randomly extract sampling points p1 from the surface of the target object's point cloud. Starting from p1, randomly find another contact point p2 on the object's surface, obtaining a pair of feasible grasping contact points (p1, p2). The center of the two contact points is the end effector position p. The grasping posture in 3D space is represented as g = (p, φ), where the grasping position... grip angle

[0013] The gripping force formula is expressed as f = μMg, where M and μ represent the mass of the gripped target and the surface friction coefficient, respectively.

[0014] The robot's grasping algorithm is as follows:

[0015] g = (x, y, z, φ, f)

[0016] Where f∈[0,10] represents the expected grasping force.

[0017] In the above method, establishing the grasping contact model includes:

[0018] The condition for force closure is that the lines connecting the contact points between the object and the left and right sides of the gripper are both inside the two-point friction cone, which will decompose the contact surface between the gripper and the object into multiple contact points.

[0019] Establish a grasping contact model:

[0020]

[0021] in, Let the friction cone be represented algebraically. It refers to the generalized force exerted by the fingers on an object.

[0022] In the above method, establishing the grasping contact model and obtaining stable, collision-free grasping constraints includes:

[0023] Within this grasping contact model, the object and the rigid finger only contact at one point, and friction and normal forces exist at that point. For the two objects to not slide against each other, the static sliding friction must be greater than and equal to the tangential component of the force acting on the object. This serves as a stable, collision-free grasping constraint, namely:

[0024] f t ≤μf3

[0025] Critical state:

[0026] f t =μf3

[0027]

[0028] Where f is the contact force, f t f3 is the tangential component of the force along the surface of the object, f3 is the normal component of the force along the surface of the object, and μ is the coefficient of static friction.

[0029] Under the condition of force closure, forces are applied to k contact points on the object surface to balance all the screws, establishing a gripping screw space as a stable, collision-free gripping constraint, i.e.:

[0030]

[0031] Where w is the sum of the surface spins of k objects, w i For the surface spin of an object, a collision is determined by calculating the radius of the largest inscribed sphere whose center is located at the far point, which is contained within the convex hull of the original contact force spiral geometry in the spin space.

[0032] In the above method, the step of training the robot's grasping force range estimation module based on the mapping relationship between the grasping target and the contact force includes:

[0033] Based on two-dimensional images Furthermore, the mapping relationship between the target and the contact force is obtained, and the contact force is input into a 3×3 convolution module to generate features.

[0034] feature Enhanced features are obtained by using inverse residual-linear bottleneck layer modules with 3 x 3 and 8 x 5 convolutional kernels. Represented as:

[0035] V = SE(V1 + V2)

[0036] Where SE represents the convolution module operation;

[0037] The training process uses the h-wish activation function, with ReLU6 being a non-linear activation function, resulting in:

[0038]

[0039] Enhanced features Adaptive average pooling is performed, and target features are extracted through a 1×1 convolutional layer. The target features are then input into a 1×1 fully connected layer, which predicts the grasping range f∈[0,10].

[0040] The prediction results are compared with the preset sample range to obtain the loss value. When the loss value is less than or equal to the preset loss threshold or the number of training times reaches the preset number threshold, the training is considered complete, and the trained robot grasping force range estimation module is obtained.

[0041] In the above method, the step of constructing a grasping generation network that integrates visual and haptic feedback based on the grasping representation algorithm, stable grasping evaluation index, grasping force range estimation module, and the mapping relationship between grasping posture and grasping force includes:

[0042] The grasping force, obtained from the mapping relationship between grasping posture and grasping force, and the disordered point cloud of the grasping target, are used as the input {p1, p2, ..., p} of the grasping generation network to be trained. n}, The number of point clouds is set to 750. The grasping generation network predicts the grasping of each target to obtain the grasping pose and grasping force, represented as:

[0043]

[0044] Where γ is the feature extraction function, and h is the mapping function that uses a multilayer perceptron network to map the point cloud from low dimension to high dimension.

[0045] In the above method, the multilayer perceptron network is formed by connecting a sampling layer, a clustering layer, and a PointNet layer in series;

[0046] The grasping generative network predicts the grasping of each target, obtaining the grasping pose and grasping force, including:

[0047] Randomly select an initial point, and use the farthest point sampling method to sample the target points to obtain each sampling point;

[0048] A sphere of radius R is defined centered at each sampling point, and the point cloud contained within the sphere is considered as a cluster {p 1i ,p 2i ,…,p ni};

[0049] Input a point set of size N×(d+C) and center coordinates of size N'×d into the aggregation layer, and output a point set group of size N'×K×(d+C), where N is the number of points, N' is the number of center points, K is the number of points in the neighborhood of the center points, d is the dimension of the points, and C is the dimension of the feature of each point.

[0050] The PointNet layer extracts local structural features N'×(d+C') from the point cloud clustered near the center point of each point set;

[0051] By integrating local structural features from different dimensions, a global feature is obtained, which serves as the output of the multilayer perceptron network. The output global feature is then input into the fully connected network in the grasping generation network, enabling the fully connected network to acquire and output the grasping pose and grasping force based on the input global feature, thereby improving the safety and stability of the robot's operation in grasping tasks. Attached Figure Description

[0052] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0053] Figure 1 This is an operation flowchart provided in the embodiments of the present invention;

[0054] Figure 2 This is a schematic diagram of a two-finger gripper grasping a point cloud target in an embodiment of the present invention;

[0055] Figure 3 This is a diagram of the grasping and generation network framework that integrates visual and haptic feedback in an embodiment of the present invention.

[0056] Figure 4 This is a comparison chart of the crawling accuracy obtained using the embodiments of the present invention and the baseline network. Detailed Implementation

[0057] To better understand the technical solution of the present invention, the embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0058] It should be understood that the described embodiments are merely some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0059] This invention provides a robot grasping and acquisition method; please refer to it. Figure 1 The method includes the following steps:

[0060] Step 101: Construct a grasping representation algorithm by simulating human grasping operations to obtain grasping posture and contact force information.

[0061] This study focuses on a two-finger gripper, including the target point cloud and the gripper's grasping posture. Figure 2 As shown, ① is a two-finger gripper, ② is the target point cloud, and the geometric parameters of the gripper include depth, maximum opening width, and gripper thickness. The end effector coordinate system is located at the midpoint of the gripper's closing direction axis.

[0062] A sampling point p1 is randomly extracted from the surface of the target object's point cloud. Starting from p1, another contact point p2 is randomly found on the object's surface, resulting in a pair of feasible grasping contact points (p1, p2). Random perturbations are added during the sampling process to increase the robustness of the grasping pose. Considering the gripper configuration, the width of the two sampling points is less than the maximum width of the gripper's open end. The center of the two contact points is the end effector position p, and the grasping angle corresponding to the linear axis around these two points is... Finally, the grasping pose in 3D space is represented as g = (p, φ), where the grasping position... grip angle All sampled candidate poses are converted from grid to point cloud coordinates.

[0063] The grasping force formula is expressed as f = μMg, where M and μ represent the mass of the grasped target and the surface friction coefficient, respectively. It can be seen that for rigid objects, changes in mass can easily cause slippage, leading to grasping failure. Therefore, the grasping force is introduced into the robot grasping representation.

[0064] g = (x, y, z, φ, f)

[0065] Where f∈[0,10] represents the expected grasping force.

[0066] In one feasible implementation, based on the hand parameters g, the point cloud p obtained by the visual sensor, and the grasping force f obtained by the tactile sensor, a corresponding grasping quality metric Q(g,p,f) model is trained, ultimately achieving grasping prediction from the point cloud.

[0067] Step 102: Establish a grasping contact model and obtain stable collision-free grasping constraints. Use the grasping contact model and stable collision-free grasping constraints as evaluation indicators for constructing stable robot grasping.

[0068] Given a grasping sample and an object state s, consider a two-finger gripper grasping an object; the grasping action is as follows: The force representing the generalized force exerted by the finger on the object satisfies the condition of force closure: the lines connecting the contact points between the object and the left and right sides of the gripper are both inside the two-point friction cone. The contact surface between the gripper and the object is decomposed into multiple contact points to establish a gripping contact model:

[0069]

[0070] In the formula, Let be the algebraic representation of the friction cone. It refers to the generalized force exerted by the fingers on an object.

[0071] Within this model, the object and the rigid body finger only contact at one point, and friction and normal force exist at that point. For the two objects to not slide against each other, the static sliding friction must be greater than the tangential component of the force acting on the object, i.e.:

[0072] f t ≤μf3

[0073] Critical state:

[0074] f t =μf3

[0075]

[0076] Where f is the contact force, f t Let f' be the tangential component of the force along the object's surface, f3 be the normal component along the object's surface, and μ be the coefficient of static friction. Clearly, the force enclosure of the gripping action depends not only on the positions of each gripping point but also on the type of friction at the contact points. During the gripping process, the prerequisite for no relative sliding between the object and the gripper is that the resultant gripping force is located inside the friction cone.

[0077] Under the condition of force closure, apply forces at k contact points on the object surface to balance all the spinors, thus establishing a grasping spinor space:

[0078]

[0079] Where w is the sum of the surface spins of k objects, w i For the surface spin of an object, a collision is determined by calculating the radius of the largest inscribed sphere whose center is located at the far point, which is contained within the convex hull of the original contact force spiral geometry in the spin space.

[0080] In a stable, collision-free state, the corresponding force constraints are set with different grasping force thresholds based on the different physical characteristics of the grasping target. For each grasping target, an average grasping force and grasping quality score are set and input into the network.

[0081] Input = {αQ} fc (s,g)+βQ gws (s,g)+γQ f (s,g,f)}

[0082] Where α, β, and γ are the weights of the force closure, grasping collision, and grasping force evaluation indicators, respectively, and Q fcLet Q be the force-closed evaluation function. gws To capture the collision evaluation function, Q f (s,g,f) represents the gripping force in this state.

[0083] Step 103: Based on the mapping relationship between the grasping target and the contact force, train the robot's grasping force range estimation module.

[0084] The RGB images captured by the ZED stereo camera are fed into a convolutional neural network architecture to establish a mapping relationship between the grasping target and the grasping force, predicting the grasping force used to grasp the object, and providing a foundation for further exploration of the intrinsic relationship between grasping pose and grasping force. Figure 3 In the grasping force range estimation part, the grasping force prediction module uses depthwise separable convolution, and through a linear bottleneck inverse residual structure, it incorporates an attention mechanism to extract two-dimensional image features and dynamically updates the parameters to obtain the grasping force estimation range.

[0085] like Figure 3 As shown, given a two-dimensional image Input features generated by 3×3 convolutional modules The feature layer dimension is compressed to C / d using 1×1 convolutions. Then, an inverse residual-linear bottleneck module, consisting of three 3×3 kernels and eight 5×5 kernels of depthwise separable convolutions, is applied. This module contains an inverse residual structure with a linear bottleneck, which first compresses the input features to a low dimension and then expands them to a high dimension. Lightweight convolutional filtering projects the features onto a low-dimensional compressed representation. In other words, the feature matrix is ​​calculated using depthwise separable convolutions, and the feature space dimension is restored using 1×1 convolutional layers. Each individual pixel channel d in a convolutional layer contains a variety of different data information. n convolutional layers together form a convolutional neural network. Successive convolutional and activation layers constitute data content that aligns with the target task's bias. This data can be transformed and embedded into the next low-dimensional subspace. Extending a branch adds the input and output feature matrices and introduces an attention mechanism to adjust the weights of each channel, resulting in enhanced features. Represented as

[0086] V = SE(V1 + V2)

[0087] Where SE represents the convolution module operation;

[0088] The training process uses the h-wish activation function and ReLU6, a non-linear activation function, to improve network accuracy.

[0089]

[0090] Enhanced features Adaptive average pooling is performed, and target features are extracted through a 1×1 convolutional layer. The target features are then input into a 1×1 fully connected layer, which predicts the grasping range f∈[0,10].

[0091] Step 104: Based on the grasping representation algorithm, stable grasping evaluation index, grasping force range estimation module, and the mapping relationship between grasping posture and grasping force, a grasping generation network integrating visual and tactile perception is constructed. The grasping generation network is used to detect the robot's grasping posture and grasping force for the corresponding target.

[0092] like Figure 3 As shown, the grasping force obtained based on the mapping relationship between the grasping target and the grasping force, and the disordered point cloud of the grasping target, are used as the input {p1,p2,...,p} of the grasping generation network to be trained. n}, The point cloud is set to 750 points, which are then integrated into multiple candidate grasping forces and candidate grasping poses. These are fed into the grasping generation network, which predicts the grasping of each target to obtain the grasping pose and grasping force. A schematic diagram is shown below. Figure 3 As shown in the grasping pose and grasping force generation section, the network includes a feature extraction module, a grasping quality evaluation module, and a fully connected layer.

[0093]

[0094] Where γ is the feature extraction function, and h is the mapping function that uses a multilayer perceptron network to map the point cloud from low dimension to high dimension. The capture quality evaluation module consists of a series of sampling layers, aggregation layers, and PointNet layers connected in series.

[0095] Randomly select an initial point and use the farthest point sampling method to sample the target number of points. Draw a sphere with radius R centered on the sampling point, and group the point cloud contained within it into a cluster {p}. 1i ,p 2i ,...,p ni};

[0096] Input a point set of size N×(d+C) and centroid coordinates of size N'×d into the aggregation layer, and output a point set group of size N'×K×(d+C), where N is the number of points, N' is the number of center points, K is the number of points in the neighborhood of the center points, d is the dimension of the points, and C is the dimension of the feature of each point.

[0097] The PointNet layer extracts local structural features N'×(d+C') from the point cloud clustered near each center point; then, it integrates local structural features of different dimensions to obtain global features, and outputs grasping pose and grasping force through a fully connected network;

[0098] Inputting 3D point cloud and tactile data generates 140 grasping postures and corresponding grasping forces. By learning the grasping quality score, the network selects the grasping estimate based on the feature point with the most information, thereby generating the grasp with the highest grasping quality score.

[0099] We constructed a view-touch fusion grasping dataset by selecting grasping targets with varying degrees of hardness and softness from the YCB object set. This dataset includes 350K real point clouds, two-finger grasping, grasping pose, grasping force, and binary labels generated from grasping evaluation metrics. The final dataset contains 16800×20 grasping samples, with at least 16800 grasping records for each object, and a total of 120 viewpoints. We divided the dataset into training and testing sets in a 9:1 ratio.

[0100] For example, Figure 4 This comparison shows the accuracy of the robot's grasping method versus the PointnetGPD method in grasping nine everyday objects, including bananas, apples, strawberries, tennis balls, and screwdrivers. These nine objects, selected from a visual-touch fusion grasping dataset, exhibit different degrees of hardness, size, and mass to validate the grasping algorithm's performance. Figure 4 As shown, the light-colored bars represent the PointnetGPD method, and the dark-colored bars represent the robot grasping method. It can be seen that the accuracy of grasping everyday objects obtained using this embodiment is much higher than that of the PointnetGPD network, which has significant advantages in terms of robot grasping stability and accuracy, thus verifying the effectiveness of the method provided in this embodiment.

[0101] The technical solutions of the embodiments of the present invention have the following beneficial effects:

[0102] A grasping representation method combining grasping posture and contact force information was constructed by simulating human grasping operations. A grasping contact model was established, and stable, collision-free grasping constraints were obtained. The grasping contact model and stable, collision-free grasping constraints were used as evaluation metrics for constructing stable robot grasping. Based on the mapping relationship between the grasping target and the contact force, a robot grasping force range estimation module was trained. Then, based on the grasping representation algorithm, stable grasping evaluation metrics, grasping force range estimation module, and the mapping relationship between grasping posture and grasping force, a grasping generation network integrating visual and tactile senses was constructed. This grasping generation network is used to detect the robot's grasping posture and grasping force for the corresponding target. Furthermore, the performance of the obtained grasping posture and grasping force in terms of grasping stability and accuracy is superior to that of the baseline network.

[0103] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0104] The contents not described in detail in this specification are common knowledge to those skilled in the art.

Claims

1. A robot grasping and acquisition method, characterized in that, The method includes: A grasping representation algorithm is constructed by simulating human grasping operations to generate grasping posture and contact force information, including: Randomly extract sampling points from the surface of the target object's point cloud. ,by Find another contact point on the surface of an object randomly, starting from the first point. A pair of feasible gripping contact points are obtained. The center of the two contact points is the position of the end effector. Representing the grasping posture in 3D space as The location to be captured Handle angle ; The formula for gripping force is expressed as follows: ,in and These represent the mass of the target being grasped and the coefficient of surface friction, respectively. The robot's grasping algorithm is as follows: in, Indicates the desired grasping force; A grasping contact model is established, and stable, collision-free grasping constraints are obtained. The grasping contact model and stable, collision-free grasping constraints are used as evaluation metrics for constructing a stable grasping robot, including: The condition for force closure is that the lines connecting the contact points between the object and the left and right sides of the gripper are both inside the two-point friction cone, which will decompose the contact surface between the gripper and the object into multiple contact points. Establish a grasping contact model: in, Let the friction cone be represented algebraically. This refers to the generalized force exerted by the fingers on an object. Within this grasping contact model, the object and the rigid finger only contact at a single point, where friction and normal forces exist. For the two objects to not slide against each other, the static friction force must be greater than and equal to the tangential component of the force acting on the object. This serves as a stable, collision-free grasping constraint, namely: Critical state: in For the force at the contact point, For the tangential component of the force along the surface of the object, For the normal component of the force along the surface of the object, The coefficient of static friction; Under closed force conditions, on the surface of the object Forces are applied to each contact point to balance all the helices, establishing a gripping helice space as a stable, collision-free gripping constraint, i.e.: in for The surface rotation of an object and For the surface spin of an object, the collision is determined by calculating the radius of the largest inscribed sphere whose center is located at the far point, which is contained in the convex hull of the original contact force spiral geometry in the spin space. Based on the mapping relationship between the grasping target and the contact force, train the robot's grasping force range estimation module; Based on the grasping representation algorithm, stable grasping evaluation index, grasping force range estimation module, and the mapping relationship between grasping posture and grasping force, a grasping generation network integrating vision and touch is constructed. The grasping generation network is used to detect the robot's grasping posture and grasping force for the corresponding target.

2. The method according to claim 1, characterized in that, The module for estimating the robot's grasping force range, based on the mapping relationship between the grasping target and the contact force, includes: Based on two-dimensional images Furthermore, the mapping relationship between the grasping target and the contact force is obtained, and the contact force is input. Convolutional modules are used to generate features. ; feature After 3 convolution kernels and 8 convolution kernels The inverse residual-linear bottleneck layer module yields enhanced features. , is represented as: in Indicates the operation of the convolution module; The training process uses the h-wish activation function. As a non-linear activation function, we get: Enhanced features Perform adaptive average pooling, through The convolutional layer extracts the target features, and the target features are then input into... The fully connected layer, composed of the Fully connected layer gripping range The prediction; The prediction results are compared with the preset sample range to obtain the loss value. When the loss value is less than or equal to the preset loss threshold or the number of training times reaches the preset number threshold, the training is considered complete, and the trained robot grasping force range estimation module is obtained.

3. The method according to claim 1, characterized in that, The method involves constructing a grasping generation network that integrates visual and haptic feedback, based on a grasping representation algorithm, a stable grasping evaluation index, a grasping force range estimation module, and a mapping relationship between grasping posture and grasping force. The grasping force, obtained from the mapping relationship between grasping posture and grasping force, and the disordered point cloud of the grasping target, are used as inputs to the grasping generation network to be trained. , The number of point clouds is set to 750. The grasping generation network predicts the grasping of each target to obtain the grasping pose and grasping force, represented as: in For feature extraction function, This is a mapping function that uses a multilayer perceptron network to map point clouds from low dimensions to high dimensions.

4. The method according to claim 3, characterized in that, The multilayer perceptron network is formed by connecting a sampling layer, a aggregation layer, and a PointNet layer in series. The grasping generative network predicts the grasping of each target, obtaining the grasping pose and grasping force, including: Randomly select an initial point, and use the farthest point sampling method to sample the target points to obtain each sampling point; A sphere of radius R is defined centered at each sampling point, and the point cloud contained within the sphere is considered as a cluster. ; The size is The point set and The center coordinates of the input aggregation layer, the output size is The point set group, where, For points, The number of the center point, The number of points in the neighborhood of the center point. For the dimension of a point, The dimension of the feature for each point; The PointNet layer extracts local structural features from the point cloud clustered near the center point of each point set. ; The local structural features of different dimensions are integrated to obtain global features, which are used as the output of the multilayer perceptron network. The output global features are input into the fully connected network in the grasping generation network, so that the fully connected network can obtain and output grasping pose and grasping force based on the input global features.

Images