Method for optimizing 6D grabbing pose of mechanical arm based on deep reinforcement learning

Abstract

The invention discloses a method for optimizing the 6D grabbing pose of a mechanical arm, and the method comprises the steps: adding a model of the mechanical arm and a depth camera in a Pybullet simulation environment, obtaining the point cloud information of a to-be-grabbed object from the depth camera, and dividing the point cloud into two types through a K-means clustering method; surrounding cuboids are constructed along the classified point clouds, and proper simulated grabbing poses are constructed according to the cuboids; and an optimized expert grabbing pose is selected from the object grabbing data set according to the screened simulated grabbing pose, a grabbing heuristic supervised learning target is constructed, and learning and training are carried out in combination with simulated learning and deep reinforcement learning methods. According to the method, a simulated grabbing point more suitable for grabbing is constructed by utilizing object surface structure features, an expert grabbing pose closest to the simulated grabbing point is selected from an object grabbing data set for grabbing training, and optimization of the grabbing pose by a mechanical arm is finally achieved by optimizing a heuristic target.

Description

Technical Field

[0001] The present invention relates to the field of robot technology application technology, and in particular to a method for optimizing a 6D grasping posture of a robot arm in a continuous motion space. Background Art

[0002] Learning human-level object manipulation skills is very demanding for robots, and learning these skills allows robots to automatically repeat tasks in the real world. Current research on grasping of intelligent robots mainly focuses on how to improve the robot's ability to identify unknown objects, select grasping postures, and execute grasping. With the development of deep reinforcement learning (DRL) technology, robots have made great progress in basic tasks involving simple manipulation skills and have shown strong generalization. However, it is still a huge challenge for deep reinforcement learning to enable robots to learn general strategies that can handle more complex tasks, especially 6D grasping tasks of robotic arms, such as grasping objects of arbitrary shapes, because for objects with complex shapes, we hope that the robotic arm can grasp the object with a better posture, and improving the grasping posture of the robotic arm is a relatively difficult problem that is still under research compared to completing the grasping of the target object.

[0003] We cited the GA-DDPG method and made improvements to it. The GA-DDPG method is to construct a set of expert grasping poses of the target object, and then select a grasping pose closest to the robot arm from the set to construct a heuristic goal for supervised learning, so that the robot arm always moves closer to this target during movement. Through experiments, we found that since the initial position of the robot arm is generally higher than the target object, the grasping pose closest to the robot arm is often the upper half of the object, causing the trained model to often choose the upper half to grasp the object when grasping the object. This may cause the best grasping pose of the object to be missed, and ignore the influence of the center of mass of the object in the real environment, resulting in a significantly lower grasping success rate in the real environment compared to the simulation environment. Therefore, we propose an improvement: obtain the point cloud of the target object from the state space, perform k-means clustering on the feature point cloud of the object, and use the principal component analysis method to obtain the main direction of each category of point cloud. We construct a cuboid that surrounds the point cloud based on the main direction of the point cloud so that it can well surround the target object, and filter the cuboids generated at different times to obtain the cuboid that can best surround the clustered point cloud. Then, for the middle and long axis of the cuboid, that is, the central axis passing through the center point of the cuboid and parallel to the long side of the cuboid, construct a simulated grasping posture that meets certain characteristic conditions, use point matching loss to find the expert grasping posture near the grasping posture, and construct a new supervised learning target based on the expert grasping posture. That is, optimize the original heuristic supervised learning target through the filtered simulated grasping points to avoid the robot arm only approaching the grasping posture closest to it, thereby increasing the possibility of exploration until learning to grasp the object in the optimal posture. Summary of the invention

[0004] The present invention proposes a method for improving the 6D grasping posture of a robotic arm based on the 3D point cloud data of the target object combined with supervised learning and the Deep Deterministic Policy Gradient (DDPG) algorithm. The robotic arm can autonomously and dynamically plan the grasping path and adjust the grasping posture according to environmental feedback, thereby achieving accurate grasping of any target object.

[0005] This paper proposes a method to optimize the 6D grasping posture of a robot arm, which includes the following main steps:

[0006] Step S1: Building an experimental platform for 6D pose grasping of a robotic arm in a simulation environment, and constructing a 6D pose grasping network in the simulation environment;

[0007] Step S2: Sample the point cloud data observed by the depth camera to obtain the target object point cloud P, and then output the point cloud set S with the target object features through the point cloud feature network. At the same time, set a series of initial goals, use the expert planner OMG to generate expert grasping samples for these goals, and store the grasping samples in the experience replay pool. When the number of samples reaches a certain threshold, start training the network;

[0008] Step S3: Divide the target object point cloud P into P by clustering j (j=1,2) two types, along the main direction of the point cloud, a cuboid is constructed to enclose the two types of point clouds Make the long axis of the cuboid (i.e. the central axis passing through the center point of the cuboid and parallel to the longest side of the cuboid) consistent with the main direction of the point cloud, and according to the cuboid Construct the corresponding simulated grasping posture located in the middle of the cuboid and perpendicular to the long axis of the cuboid

[0009] Step S4: As the robot moves toward the target object, new enclosing cuboids are continuously generated. Select a pair of cuboids with good segmentation and enclosing point cloud effects from the accumulated enclosing cuboids. At the same time, the simulated grasping posture corresponding to the cuboid group is retained Select the simulated grasping pose that is closest to the robot gripper Then according to the simulated grasping posture Find expert grasp poses from the object grasping dataset G to construct the supervised learning objective

[0010] Step S5: Grasp the pose according to the expert Set the heuristic goal t t , combined with the predicted targets output by the policy network and the value network for supervised learning, using supervised learning to reduce the difference between the network predicted target and the heuristic target, supervise the gripper to approach the target, and combine imitation learning and deep reinforcement learning losses to ultimately achieve optimization of the grasping posture;

[0011] Furthermore, the construction of the simulation platform and network described in step S1 specifically includes the following steps:

[0012] Step S1-1: Add a robotic arm model and a two-finger gripper model in the Pybullet simulation environment, load an RGB-D camera on the two-finger gripper model, and import a table model for placing the object to be grasped;

[0013] Step S1-2: constructing a PointNet++ point cloud network to extract feature information of the target object;

[0014] Step S1-3: Use two fully convolutional neural networks to build the policy network π θ and value network Q φ , the strategy network is responsible for deciding the 6D posture grasping action of the robot arm, and the value network is responsible for judging the quality of the actions decided by the strategy network during training;

[0015] Furthermore, step S2 specifically includes the following steps:

[0016] Step S2-1: randomly place an object to be grasped on the table, obtain the point cloud data of the target object through the RGB-D depth camera, and evenly sample 1024 points from it to form a point cloud set P of the target object;

[0017] Step S2-2: Use the PointNet point cloud feature network to extract features from the point cloud set P, and use the extracted feature point cloud as the input state S of the deep reinforcement learning network;

[0018] Step S2-3: Set a series of target objects, obtain the state S of the target object according to step S2-2, and use the state as the input of the expert planner. The expert planner outputs the expert action of the robot gripper under the current state, and records the states and corresponding actions experienced by the robot gripper from the initial moment to the successful grasping of the target object, and forms a grasping trajectory. When the number of trajectories reaches a certain threshold, proceed to the next step;

[0019] Furthermore, in step S3, the point cloud bounding cuboid is constructed according to the point cloud subset. And its corresponding simulated grasping pose The specific steps include:

[0020] Step S3-1: Use the K-means clustering algorithm to divide the target object point cloud set P obtained in step S2-1 into two different point cloud subsets P1 and P2;

[0021] Step S3-2: Obtain point cloud data P j The point cloud center and the main direction of the point cloud are projected to the horizontal direction (i.e., the x and y directions), and the rotation angle μ of the point cloud in the horizontal direction is calculated. j ;

[0022] Step S3-3: Centralize the point cloud data so that the point cloud is centered on the origin, and then calculate the point cloud data according to μ j Construct a rotation matrix to rotate the point cloud to be parallel to the x-axis, and construct a bounding cuboid around the rotated point cloud And construct the corresponding simulated grasping posture g located in the middle of the cuboid and perpendicular to the long axis of the cuboid j ;

[0023] Step S3-4: Enclosing the cuboid and simulated grasping pose g j Perform rotation and coordinate offset to complete the enclosing cuboid And its corresponding simulated grasping posture The construction of

[0024] Furthermore, the horizontal rotation angle μ of the calculated point cloud in step S3-2 is j , specifically including the following steps:

[0025] Step S3-2-1: Point cloud data P j By N j Points p ji =[x ji ,y ji , z ji ] T (i=1,2,...,N j ) to calculate the point cloud center Then subtract the center coordinates from each point Get centralized data The processed point cloud data is centered on the origin O, which is convenient for subsequent covariance matrix calculation;

[0026] Step S3-2-2: Construct the covariance matrix To represent the distribution of point cloud data on different coordinate axes;

[0027] Step S3-2-3: Construct characteristic equation C using covariance matrix j v j =λ j v j , where v j is a feature vector, representing the point cloud P j (j=1,2) a main direction, λ j is the eigenvalue, corresponding to the variance of the point cloud in the main direction of the eigenvector;

[0028] Step S3-2-4: Solve the characteristic equation det(C j -λ j I)=0 to obtain the eigenvalue λ j，1 ,λ j,2 ,λ j，3 , sorted by eigenvalue λ j，1 <λ j，2 <λ j，3 , thus obtaining the corresponding eigenvector λ j，1 ,λ j，2 ,λ j，3 , select the eigenvector λ with the largest eigenvalue j，3 As the first main direction of the point cloud;

[0029] Step S3-2-5: Set λ j，3 The horizontal projection is divided into The inverse tangent function is used to calculate the angle relative to the x-axis.

[0030] Step S3-2-6: If λ j，3 Parallel to the z-axis, that is, the first principal direction is perpendicular to the horizontal plane, then λ j，2 As the second main direction, j，2 The horizontal projection is divided into The inverse tangent function is used to calculate the angle relative to the x-axis.

[0031] Furthermore, the horizontally enclosing cuboid described in step S3-3 and its corresponding simulated grasping posture g j The construction of includes the following steps:

[0032] Step S3-3-1: Set the angle μ j Convert to radians and take the negative value to get the rotation angle

[0033] Step S3-3-2: According to the rotation angle α j Construct the rotation matrix R j :

[0034]

[0035] Step S3-3-3: The centralized point cloud q obtained in step S3-2-1 j，i Applying a rotation matrix Make the main direction of the point cloud parallel to the x-axis, where R j ·q jl is the matrix multiplication operation;

[0036] Step S3-3-4: Known point cloud With the origin O as the center, construct an enclosing cuboid with the coordinate origin O as the center So that in the base coordinate system, the three sides of the cuboid have coordinate axes parallel to them, and the cuboid The three sides of are:

[0037] a j =x j，max -x j，min

[0038] b j =y j，max -y j，main

[0039] c j =z j，max -z j，min

[0040] Where (x j，max ,x j，min ,y j，max ,y j，min ,z max , z j，min ) are vectors Extreme values ​​on various components;

[0041] Step S3-3-5: Along the horizontal cuboid Construct the long axis j , l j Set the cuboid as the central axis passing through the origin and parallel to the longest side of the cuboid. The three faces facing the gripper of the robot arm are A j , B j , C j , mid-long axis l j Perpendicular to A j and A j Faces intersect at center B j The area of ​​the surface is greater than C j noodle;

[0042] Step S3-3-6: Set the gripping posture structure according to the robot gripper model, and use six symmetrical points to represent the gripper, which correspond to the palm base, finger base, fingertip and other six key parts of the gripper from top to bottom. The spatial positions of the two points at the palm base coincide, and the six points are all on the same plane. The spatial coordinates of the six gripper reference points are constructed as (x n ,y n , z n ) n＝1，2,...,6 ;

[0043] Step S3-3-7: Select C j Center of face As the center point of the gripper, set the vector Set the vector for the z-axis (the direction of the gripper) of the gripper The x-axis (vertical to the plane of the jaws) of the jaws is in the direction of the long axis l j Parallel, by vector cross product Get the direction of the gripper's y-axis and construct a horizontal cuboid The simulated grasping pose g j ;

[0044] Furthermore, the enclosing cuboid described in step S3-4 And its corresponding simulated grasping posture The construction of includes the following steps:

[0045] Step S3-4-1: Construct the rotation angle μ j The corresponding rotation matrix R′ j ,in

[0046]

[0047] According to the rotation matrix R′ j The cuboid Rotate into a cuboid

[0048] Step S3-4-2: Translate the rotated cuboid back to the original point cloud space position to obtain the final cuboid surrounding the point cloud in Point cloud data P j The center coordinates of

[0049] Step S3-4-3: Simulate the grasping posture g j In the base coordinate system, it can be expressed as g j =[x j ,y j , z j , α j , β j , γ j ] T ,in Indicates the center point of the gripper Position, [α j , β j , γ j ] T Represents the Euler angle of the gripper, which can be converted into the form of a rotation matrix:

[0050]

[0051] Therefore, the simulated grasping posture g can be constructed j The homogeneous transformation matrix

[0052]

[0053] Step S3-4-4: According to the rotation matrix R′ j Rotation Matrix and jaw center position Rotate to form the rotated gripper position The corresponding homogeneous transformation matrix;

[0054]

[0055] Step S3-4-5: Perform similar operations as step S3-4-2 to translate the rotated gripper to the final enclosing cuboid. The corresponding position of to obtain the simulated grasping posture The homogeneous transformation matrix

[0056]

[0057] Step S3-4-6: According to step S3-4-3, the gripper position can be set With the homogeneous transformation matrix The conversion relationship between them is as follows, where h g (·) and h T (·) are all transformation operators, that is, posture The corresponding rotation matrix is Rotation Matrix The corresponding gripper position is

[0058]

[0059]

[0060] Therefore, the final enclosing cuboid can be obtained Corresponding simulated grasping pose

[0061] Furthermore, the step S4 described in accordance with the simulated grasping posture Selecting expert grasp poses for constructing supervised learning objectives The specific steps include:

[0062] Step S4-1: As the gripper approaches the target, the point cloud information of the object gradually becomes richer, and the cuboid constructed in step S3 The effect of enclosing the point cloud gradually becomes better, and the volume of the enclosing cuboid gradually decreases. Set the threshold Select a smaller cuboid from many surrounding cuboids;

[0063] Step S4-2: Select a pair of cuboids with better segmentation and enclosing point cloud effects based on the center position and volume of the selected cuboids. At the same time, the simulated grasping posture of the corresponding enclosing cuboid in the cuboid group is retained

[0064] Step S4-3: According to the simulated grasping posture Filter out the simulated grasping posture closest to the current moment's robot gripper Then according to the simulated grasping posture Find expert grasp poses from the object grasping dataset G to construct the supervised learning objective

[0065] Furthermore, the volume threshold setting described in step S4-1 The specific steps include:

[0066] Step S4-1-1: Repeat steps S3-2, S3-3 and S3-4 to construct the point cloud enclosing cuboid of the entire target object point cloud P And calculate the volume of the cuboid through step S3-3-4

[0067] Step S4-1-2: Setting the maximum volume threshold If the current time is the initial time, that is, t = 1, then otherwise constant;

[0068] Step S4-1-3: Calculate the cuboid at time t Volume if but And save the cuboid The center position and the corresponding simulated grasping point And record the number T=t at this time; if but No change, do not save the cuboid In order to avoid the error causing the volume of the retained cuboid to be too small, a minimum volume threshold is set. Ensure that the volume of all cuboids is greater than

[0069] Step S4-1-4: When the target object point cloud is updated, Update to the new target object point cloud bounding box Volume

[0070] Furthermore, the final cuboid group selected in step S4-2 And retain the simulated grasping posture of the corresponding enclosing cuboid in the cuboid group The specific steps include:

[0071] Step S4-2-1: Pair the cuboids recorded in step S4-1-3 in pairs to construct m pairs of cuboids, and calculate the Euclidean distance L between the center coordinates of each pair of cuboids. m , and calculate the average of all Euclidean distances

[0072] Step S4-2-2: Analyze the cuboid numbers saved in step S4-1-3. The smaller the volume of the cuboid, the larger the corresponding number T. Filter out the distance between the center points of two cuboids. Then select the pair of cuboids with the largest sum of their numbers (T1+T2) from these cuboid pairs. If there are cuboid pairs with the same sum of numbers, select the pair with the largest sum of their numbers (T1+T2). m The largest cuboid pair, as the two final cuboid groups selected at the current moment

[0073] Step S4-2-3: Enclose a cuboid based on each point cloud in step S3-4 The corresponding simulated grasping postures are constructed Therefore, the final cuboid group can be obtained The corresponding two simulated grasping poses

[0074] Furthermore, the expert grasping posture obtained in step S4-3 is used to construct the supervised learning target The specific steps include:

[0075] Step S4-3-1: The position of the gripper in the base coordinate system at time t is denoted as g t =[x t ,y t ,z t , α t , β t , γ t ] T , where [x t ,y t ,z t ] T Indicates the center position of the gripper, [α t , β t , γ t ] T represents the Euler angle of the gripper, and uses f(·) transformation to represent the gripper position g at time t t To the 6 gripper reference points (x n, y n , z n )(n=1,2,...,6), the specific representation method is as follows:

[0076] f(g t )=[x1(t), γ1(t), z1(t),..., x6(t), y6(t), z6(t)] T

[0077] Step S4-3-2: Quantify the gripper position g using the point matching loss formula t+1 With g t Distance between:

[0078]

[0079] In the formula ||f(g t+1 )-f(g t )||1 means to find the 1-norm, and then from Select the distance g from the robot gripper at time t t The most recent grasping pose

[0080]

[0081] In the formula, arg min represents the variable value operation to find the minimum function value;

[0082] Step S4-3-3: Each object surface has 100 pre-prepared grasping postures, which constitute the grasping set G of the object to be grasped. The distance simulation grasping posture is calculated using the point matching loss formula. The nearest expert grasp pose

[0083]

[0084] Furthermore, the step S5 of constructing a heuristic target auxiliary network for training specifically includes the following steps:

[0085] Step S5-1: Improved target grasping posture obtained in step S4-3 and the current position of the gripper g t , construct the heuristic goal t t and the corresponding loss function;

[0086] Step S5-2: Apply the loss function calculated in step S5-1 to the policy network π θ and value network Q φ During the training process, the robot arm can be promoted to approach the target object better;

[0087] Furthermore, the step S5-1 of constructing the heuristic target specifically includes the following steps:

[0088] Step S5-1-1: Combined with step S3-4-6, define the homogeneous transformation matrix h g (t t ) is the current gripper position g t To the target gripper position The transfer matrix is ​​calculated as follows:

[0089]

[0090] Step S5-1-2: According to the calculation relationship of step S5-1-1 and the conversion operator set in step S3-4-6, the improved heuristic target can be obtained, where (h g {g t )) -1 is the gripper position g t The corresponding inverse transformation of the homogeneous transformation matrix:

[0091]

[0092] Step S5-1-3: Use the policy network π constructed in step S2 θ and value network Q φ Generate the corresponding network prediction target and Combined with the formula proposed in step S4-3-2, the loss function of the network prediction target and the heuristic target is calculated as follows:

[0093]

[0094]

[0095] Where θ is the parameter of the policy network, φ′ is the parameter of the target value network in the DDPG network, which is used to update the parameters of the main value network;

[0096] Furthermore, the network training process described in step S5-2 specifically includes the following steps:

[0097] Step S5-2-1: Assume that the environment in which the robot is located is completely observable. At each time step t, the agent changes from the current state s t Take action a t After reaching state s t+1 , where state s t Represents the point cloud features obtained from step S2-2, and the reward return r of the current action t The robot generates a large number of trajectory samples (s t , a t , r t ,s t+1 ), store these samples in the experience replay pool to train the reinforcement learning network;

[0098] Step S5-2-2: The policy network output action is recorded as a t , that is, the gripper posture g at time step t+1 t+1 The position offset of the gripper position gt at time step t is calculated as follows:

[0099] h g (a t )=(h g (g t )) -1 h g (g t+1 )

[0100] a t =h T ((h g (g t )) -1 h g (g t+1 ))=[Δx,Δy,Δz,Δα,Δβ,Δγ]

[0101] Where Δx, Δy, Δz are position offsets, and Δα, Δβ, Δγ are angle offsets. In the autonomous decision-making process of the robot arm, the trajectory of the gripper from the 0th time step to the Tth time step is T r ={g0, g0, ..., g T}, where g T Indicates the gripper position when the maximum number of grasps is reached;

[0102] Step S5-2-3: Use the expert planner OMG to generate state s t Expert Actions Combined with the policy network output action σ in step S5-2-2 t And the point matching loss formula proposed in step S4-3-2, construct the loss function of behavior cloning:

[0103]

[0104] Step S5-2-4: Value Network Q φ The overall loss function expression is as follows:

[0105]

[0106] y represents the target Q value to be approached, and its expression is as follows:

[0107] y=r t +γQ φ′ (s t+1 , π θ ′(s t+1 )+ε)

[0108] Where Q φ′ and π θ′ are the target value network and the target policy network, θ′ is the parameter of the target policy network, γ is the reward discount factor, set γ = 0.95, ε is the pre-set shear noise, r t is the reward value of the robot grasping. When the grasping is successful and the object is stably lifted 20 cm, a positive reward is provided, otherwise a negative reward is given. The grasping reward function is defined as follows:

[0109]

[0110] Step S5-2-5: Policy network π θ The overall loss function expression is as follows:

[0111]

[0112] BC loss helps prevent the learning strategy from deviating too much from the expert strategy, and λ is a weighted hyperparameter. Setting λ = 0.2 is mainly used to coordinate the balance between the expert loss and the learning value function loss. DDPG (s t , a t ) is the deterministic strategy loss, whose goal is to maximize the Q value, which can be expressed as follows:

[0113] L DDPG (s t , a t )=-Q φ (s t , a t )

[0114] Step S5-2-6: Soft update target value network Q φ and the target policy network π θ ',

[0115] φ′←τφ+(1-τ)φ′

[0116] θ′←-τθ+(1-τ)θ′

[0117] Where φ′ is the target value network Q φ′ , φ is the parameter of the value network Q φ , θ′ is the parameter of the target policy network π θ′ , θ is the parameter of the policy network π θ Parameters, τ is the soft update coefficient, set τ = 0.999;

[0118] Step S5-2-7: Repeat the above steps S3 to S5 multiple times to achieve the grasping and grasping optimization of the target object by the robot arm.

[0119] The beneficial effects of the present invention are as follows:

[0120] This method proposes a method for optimizing the 6D grasping posture of a robotic arm, which provides a systematic framework that combines the error correction performance of supervised learning with the perception and decision-making capabilities of deep reinforcement learning. It autonomously screens the grasping posture through the network, and uses the continuous interaction between the robotic arm and the environment to allow the robotic arm to learn how to grasp in a more human-like grasping manner, thereby improving the posture of the robotic arm grasping objects, thereby improving the efficiency and robustness of the robotic arm in grasping objects of arbitrary shapes in unstructured environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0121] Figure 1 This is the overall flow chart of this method;

[0122] Figure 2 Robot arm simulation environment;

[0123] Figure 3 Some training scene diagrams of ShapeNet;

[0124] Figure 4 Schematic diagram of three types of enclosing cuboids;

[0125] Figure 5 The top view obtained by centering the cuboid obtained by clustering;

[0126] Figure 6 Schematic diagram of the angle between the main direction of the corresponding point cloud and the x-axis;

[0127] Figure 7 Schematic diagram of the cuboid obtained by rotating according to the main direction of the point cloud;

[0128] Figure 8 Construct a schematic diagram of the simulated gripper based on the enclosing cuboid;

[0129] Fig. 9 Schematic diagram of rotating and offsetting the simulated gripper;

[0130] Fig.10 Schematic diagram of the final enclosing cuboid obtained through screening and its corresponding simulated grasping posture;

[0131] Fig.11 Overall algorithm structure diagram;

[0132] Fig.12 The effect of grasping an object using the trained model. DETAILED DESCRIPTION

[0133] The present invention proposes an improved 6D grasping posture method of a robotic arm based on deep reinforcement learning, which will be described in detail and completely in conjunction with the technical methods in the accompanying drawings and embodiments of the present invention. The specific description is as follows:

[0134] Figure 1It is a flow chart of the method of the present invention, which includes all the steps of the method. The specific steps are as follows:

[0135] Step S1: Build a simulation experimental platform for 6D pose grasping of the robotic arm in the Ubuntu environment, and construct a 6D pose grasping network;

[0136] Step S1-1: The simulation experiment platform includes a Pybullet scene and the object to be grasped in the scene. An XARM robotic arm equipped with an xArm parallel robotic claw is built, and an RGB-D depth camera is added to obtain the visual information of the target object. At the same time, a table model for placing the object to be grasped is imported. The position of the robotic arm base is fixed, and the position of the target object is randomly generated by the system. Before each grasping experiment begins, the position of the robotic arm gripper is randomly initialized so that it always faces the target object; the maximum range of the agent's decision is T = 20, and once the object is successfully grasped, a round will end; the experimental platform environment is as follows Figure 2 As shown;

[0137] Step S1-2: 1500 objects are selected from the ShapeNet dataset for training. These objects cover 169 different categories. Each object is equipped with 100 preset expert grasping poses. Ten different categories of objects in the YCB (Yale-CMU-Berkeley) dataset are used for testing. Some training scenarios are as follows: Figure 3 As shown;

[0138] Step S1-3: constructing a PointNet++ point cloud network to extract feature information of the target object;

[0139] Step S1-4: Use two fully convolutional neural networks to build the policy network π θ and value network Q φ , the strategy network is responsible for deciding the 6D posture grasping action of the robot arm, and the value network is responsible for judging the quality of the actions decided by the strategy network during training;

[0140] Step S1-4-1: The value network and the policy network each consist of a three-layer fully connected multilayer perceptron with a ReLU activation function as the core. During the training phase, this study uses the Adam optimizer and adjusts the learning rate I by adopting a segmented approach. r , to optimize network performance;

[0141] Step S1-4-2: Randomly initialize the parameters of the policy network and the value network as well as the parameters of the target policy network and the target value network;

[0142] Step S2: Sample the point cloud data observed by the depth camera to obtain the target object point cloud P, and then output the point cloud set S with the target object features through the point cloud feature network. At the same time, set a series of initial goals, use the expert planner OMG to generate expert grasping samples for these goals, and store the grasping samples in the experience replay pool. When the number of samples reaches a certain threshold, start training the network;

[0143] Step S2-1: Place an object to be grasped randomly on the table, obtain the point cloud data of the target object through the RGB-D depth camera, uniformly sample 1024 points from it to form a point cloud set P of the target object, use the PointNet point cloud feature network to extract features from the point cloud set P, and use the extracted 512-dimensional feature point cloud as the input state S of the deep reinforcement learning network;

[0144] Step S2-2: Set up an experience replay pool to store the experience data generated by the interaction between the robot and the environment. Before training, a certain amount of successful grasping experience data must be collected through the expert planner. By giving the starting point and the target grasping point of the target object, the expert planner OMG will autonomously generate a continuous and successful grasping motion trajectory. The trajectory reflects the state reached by the robot during the exploration process and the action of the robot's decision in this state. Only when the trajectory of successful grasping reaches a certain level will it enter the next step;

[0145] Step S3: Collect heuristic target grab samples and construct corresponding loss functions;

[0146] Step S3-1: Divide the target object point cloud P into two categories P1 (j=1, 2) by clustering, and construct cuboids enclosing the two categories of point clouds in sequence along the main direction of the point cloud Make the long axis of the cuboid (i.e. the central axis passing through the center point of the cuboid and parallel to the longest side of the cuboid) consistent with the main direction of the point cloud, and Construct the corresponding simulated grasping posture located in the middle of the cuboid and perpendicular to the long axis of the cuboid

[0147] Step S3-1-1: Use the K-means clustering algorithm to divide the target object point cloud set P obtained in step S2-1 into two different point cloud subsets P1 and P2;

[0148] Step S3-1-2: Obtain point cloud data P j The point cloud center and the main direction of the point cloud are projected to the horizontal direction (i.e., the x and y directions), and the rotation angle μ of the point cloud in the horizontal direction is calculated. j , the corresponding rectangular projection diagram is as follows Figure 4 and Figure 5 As shown;

[0149] Step S3-1-2-1: Point cloud data P j By N j Points Composition, calculation of point cloud center Then subtract the center coordinates from each point Get centralized data The processed point cloud data is centered on the origin O, which is convenient for subsequent covariance matrix calculation;

[0150] Step S3-1-2-2: Construct the covariance matrix To represent the distribution of point cloud data on different coordinate axes;

[0151] Step S3-1-2-3: Construct characteristic equation C using covariance matrix j v j =λ j v j , where v j is a feature vector, representing the point cloud P j (j=1,2) a main direction, λ j is the eigenvalue, corresponding to the variance of the point cloud in the main direction of the eigenvector;

[0152] Step S3-1-2-4: Solve the characteristic equation det(C j -λ j I)=0 to obtain the eigenvalue λ j，1 ,λ j,2 ,λ j，3 , sorted by eigenvalue λ j，1 <λ j，2 <λ j，3 , thus obtaining the corresponding eigenvector λ j，1 ,λ j，2 ,λ j，3 , select the eigenvector λ with the largest eigenvalue j，3 As the first main direction of the point cloud;

[0153] Step S3-1-2-5: Set λ j，3 The horizontal projection is divided into The inverse tangent function is used to calculate the angle relative to the x-axis. Angle calculation diagram Figure 6 As shown;

[0154] Step S3-1-2-6: If λ j，3 Parallel to the z-axis, that is, the first principal direction is perpendicular to the horizontal plane, then λ j，2 As the second main direction, j，2 The horizontal projection is divided into The inverse tangent function is used to calculate the angle relative to the x-axis.

[0155] Step S3-1-3: Centralize the point cloud data so that the point cloud is centered on the origin, and then construct a rotation matrix based on μ to rotate the point cloud to be parallel to the x-axis, and construct a bounding cuboid around the rotated point cloud And construct the corresponding simulated grasping posture g located in the middle of the cuboid and perpendicular to the long axis of the cuboid j ;

[0156] Step S3-1-3-1: Set the angle μ j Convert to radians and take the negative value to get the rotation angle

[0157] Step S3-1-3-2: According to the rotation angle α j Construct the rotation matrix R j :

[0158]

[0159] Step S3-1-3-3: The centralized point cloud q obtained in step S3-1-2-1 ji Applying a rotation matrix Make the main direction of the point cloud parallel to the x-axis, where R j ·q ji is the matrix multiplication operation;

[0160] Step S3-1-3-4: Known point cloud With the origin O as the center, construct an enclosing cuboid with the coordinate origin O as the center So that in the base coordinate system, the three sides of the cuboid have coordinate axes parallel to them, and the cuboid The three sides of are:

[0161] a j =x j，max -x j，min

[0162] b j =y j，max -y j，min

[0163] c j =z j,max -z j，min

[0164] Where (x j，max , x j，min ,y j，max ,y j，min , z max , z j，min) are vectors The extreme values ​​of each component, enclosing the cuboid The schematic diagram is as follows Figure 7 As shown;

[0165] Step S3-1-3-5: Along the horizontal cuboid Construct the long axis j , l j Set the cuboid as the central axis passing through the origin and parallel to the longest side of the cuboid. The three faces facing the gripper of the robot arm are A j , B j , C j , mid-long axis l j Perpendicular to A j and A j Faces intersect at center B j The area of ​​the surface is greater than C j noodle;

[0166] Step S3-1-3-6: Set the gripping posture structure according to the robot gripper model, and use six symmetrical points to represent the gripper, which correspond to the palm base, finger base, fingertip and other six key parts of the gripper from top to bottom. The spatial positions of the two points at the palm base coincide, and the six points are all on the same plane. The spatial coordinates of the six gripper reference points are constructed as (x n ,y n , z n )(n=1,2,...,6);

[0167] Step S3-1-3-7: Select C j Center of face As the center point of the gripper, set the vector Set the vector for the z-axis (the direction of the gripper) of the gripper The x-axis (vertical to the plane of the jaws) of the jaws is in the direction of the long axis l j Parallel, by vector cross product Get the direction of the gripper's y-axis and construct a horizontal cuboid The simulated grasping pose g j , simulated gripper g j The construction of Figure 8 As shown;

[0168] Step S3-1-4: Enclosing the cuboid and simulated grasping pose g j Perform rotation and coordinate offset to complete the enclosing cuboid And its corresponding simulated grasping posture The construction of

[0169] Step S3-1-4-1: Construct rotation angle μ j The corresponding rotation matrix R′ j ,in

[0170]

[0171] According to the rotation matrix The cuboid Rotate into a cuboid

[0172] Step S3-1-4-2: Translate the rotated cuboid back to the original point cloud space position to obtain the final cuboid surrounding the point cloud in Point cloud data P j The center coordinates of

[0173] Step S3-1-4-3: Simulate the grasping posture g j In the base coordinate system, it can be expressed as g j =[x j ,y j , z j , α j , β j , γ j ] T ,in Indicates the center point of the gripper Position, [α j , β j , γ j ] T Represents the Euler angle of the gripper, which can be converted into the form of a rotation matrix:

[0174]

[0175] Therefore, the simulated grasping posture g can be constructed j The homogeneous transformation matrix

[0176]

[0177] Step S3-1-4-4: According to the rotation matrix R′ j Rotation Matrix and jaw center position Rotate to form the rotated gripper position The corresponding homogeneous transformation matrix:

[0178]

[0179] Step S3-1-4-5: Perform similar operations as step S3-1-4-2 to translate the rotated gripper to the final enclosing cuboid. The corresponding position of to obtain the simulated grasping posture The homogeneous transformation matrix

[0180]

[0181] Step S3-1-4-6: According to step S3-1-4-3, the gripper position can be set With the homogeneous transformation matrix The conversion relationship between them is as follows, where h g (·) and h T (·) are all transformation operators, that is, posture The corresponding rotation matrix is Rotation Matrix The corresponding gripper position is

[0182]

[0183]

[0184] Therefore, the final enclosing cuboid can be obtained Corresponding simulated grasping pose The rotation and displacement process of the gripper jaws is as follows: Fig. 9 As shown;

[0185] Step S3-2: As the robot moves toward the target object, new enclosing cuboids are continuously generated. Select a pair of cuboids with good segmentation and enclosing point cloud effects from the accumulated enclosing cuboids. At the same time, the simulated grasping posture corresponding to the cuboid group is retained Select the simulated grasping pose that is closest to the robot gripper Then according to the simulated grasping posture Find expert grasp poses from the object grasping dataset G to construct the supervised learning objective

[0186] Step S3-2-1: As the gripper approaches the target, the point cloud information of the object gradually becomes richer. The cuboid constructed in step S3-1 The effect of enclosing the point cloud gradually becomes better, and the volume of the enclosing cuboid gradually decreases. Set the threshold Select a smaller cuboid from many surrounding cuboids;

[0187] Step S3-2-1-1: Repeat steps S3-1-2, S3-1-3 and S3-1-4 to construct the point cloud enclosing cuboid of the entire target object point cloud P And calculate the volume of the cuboid through step S3-1-3-4

[0188] Step S3-2-1-2: Setting the maximum volume threshold If the current time is the initial time, that is, t = 1, then otherwise constant;

[0189] Step S3-2-1-3: Calculate the cuboid at time t Volume if but And save the cuboid The center position and the corresponding simulated grasping point And record the number T=t at this time; if but No change, do not save the cuboid In order to avoid the error causing the volume of the retained cuboid to be too small, a minimum volume threshold is set. Ensure that the volume of all cuboids is greater than

[0190] Step S3-2-1-4: When the target object point cloud is updated, Update to the new target object point cloud bounding box Volume

[0191] Step S3-2-2: Select a pair of cuboids with better segmentation and enclosing point cloud effects based on the center position and volume of the selected cuboids. At the same time, the simulated grasping posture of the corresponding enclosing cuboid in the cuboid group is retained

[0192] Step S3-2-2-1: Pair the cuboids recorded in step S3-2-1-3, construct m pairs of cuboids, and calculate the Euclidean distance L between the center coordinates of each pair of cuboids. m , and calculate the average of all Euclidean distances

[0193] Step S3-2-2-2: Analyze the cuboid numbers saved in step S3-2-1-3. The smaller the volume of the cuboid, the larger the corresponding number T. Filter out the distance between the center points of two cuboids. Then select the pair of cuboids with the largest sum of their numbers (T1+T2) from these cuboid pairs. If there are cuboid pairs with the same sum of numbers, select the pair with the largest sum of their numbers (T1+T2). m The largest cuboid pair, as the two final cuboid groups selected at the current moment

[0194] Step S3-2-2-3: Enclose a cuboid based on each point cloud in step S3-1-4 The corresponding simulated grasping postures are constructed Therefore, the final cuboid group can be obtained The corresponding two simulated grasping poses The final rectangular group obtained according to the above screening is shown in the figure below Fig.10 As shown;

[0195] Step S3-2-3: According to the simulated grasping posture Filter out the simulated grasping posture closest to the current moment's robot gripper Then according to the simulated grasping posture Find expert grasp poses from the object grasping dataset G to construct the supervised learning objective

[0196] Step S3-2-3-1: The position of the gripper in the base coordinate system at time t is denoted as g t =[x t， y t , z t ,α t， β t , γ t ] T , where [x t ,y t ,z t ] T Indicates the center position of the gripper, [α t , β t , γ t ] T represents the Euler angle of the gripper, and uses f(·) transformation to represent the gripper position g at time t t To the 6 gripper reference points (x n ,y n , z n )(n=1,2,...,6), the specific representation method is as follows:

[0197] f(g t )=[x1(t), y1(t), z1(t),..., x6(t), y6(t), z6(t)] T

[0198] Step S3-2-3-2: Quantify the gripper position g using the point matching loss formula t+1 With g t Distance between:

[0199]

[0200] In the formula ||f(g t+1 )-f(g t )||1 means to find the 1-norm, and then from Select the distance g from the robot gripper at time t t The most recent grasping pose

[0201]

[0202] In the formula, arg min represents the variable value operation to find the minimum function value;

[0203] Step S3-2-3-3: Each object surface has 100 pre-prepared grasping postures, which constitute the grasping set G of the object to be grasped. The distance simulation grasping posture is calculated using the point matching loss formula. The nearest expert grasp pose

[0204]

[0205] Step S3-3: Grasp the pose according to the expert Set the heuristic goal t t , combine it with the predicted targets output by the policy network and the value network for supervised learning, use supervised learning to reduce the difference between the network prediction target and the heuristic target, and supervise the gripper to move closer to the target;

[0206] Step S3-3-1: Combined with step S3-1-4-6, define the homogeneous transformation matrix h g (t t ) is the current gripper position g t To the target gripper position The transfer matrix is ​​calculated as follows:

[0207]

[0208] Step S3-3-2: According to the calculation relationship of step S3-3-1 and the conversion operator set in step S3-1-4-6, the improved heuristic target can be obtained, where (h g (g t )) -1 is the gripper position g t The corresponding inverse transformation of the homogeneous transformation matrix:

[0209]

[0210] Step S3-3-3: Use the policy network π constructed in step S2 θ and value network Q φ Generate the corresponding network prediction target and Combined with the formula proposed in step S3-2-3-2, the loss function of the network prediction target and the heuristic target is calculated as follows:

[0211]

[0212]

[0213] Where θ is the parameter of the policy network, φ′ is the parameter of the target value network in the DDPG network, which is used to update the parameters of the main value network;

[0214] Step S4: Combine imitation learning and deep reinforcement learning losses to finally optimize the grasping posture;

[0215] Step S4-1: Assume that the environment in which the robot is located is completely observable. At each time step t, the agent changes from the current state s t Take action a t After reaching state s t+1 , where state s t Represents the component of the point cloud feature S obtained from step S2-1 at time step t, and the reward r of the current action is obtained at the same time t The robot generates a large number of trajectory samples (s t ,a t , r t ,s t+1 ), store these samples in the experience replay pool to train the reinforcement learning network;

[0216] Step S4-2: The policy network output action is recorded as a t , that is, the position offset between the gripper position gt+1 at the t+1 time step and the gripper position gt at the t time step, and its calculation formula is as follows:

[0217] h g (a t )=(h g (g t )) -1 h g (g t+1 )

[0218] a t =h T ((h g (gt )) -1 h g (g t+1 ))=[Δx,Δy,Δz,Δα,Δβ,Δγ]

[0219] Where Δx, Δy, Δz are position offsets, and Δα, Δβ, Δy are angle offsets. In the autonomous decision-making process of the robot arm, the trajectory of the gripper from the 0th time step to the Tth time step is T r ={g0,g0,...,g T}, where g T Indicates the gripper position when the maximum number of grasps is reached;

[0220] Step S4-3: Use the expert planner OMG to generate state s t Expert Actions Combined with the policy network output action a in step S4-2 t And the point matching loss formula proposed in step S3-2-3-2, construct the loss function of behavior cloning:

[0221]

[0222] This loss helps prevent the strategy learned by the policy network from deviating too much from the expert strategy, while ensuring that the robot arm can approach the target object in the early stages of training, reducing the time the robot arm spends exploring the space;

[0223] Step S4-4: Value Network Q φ The overall loss function expression is as follows:

[0224]

[0225] y represents the target Q value to be approximated, and its expression is as follows;

[0226] y=r t +γQ φ ′(s t+1 , π θ ′(s t+1 )+ε)

[0227] Where Q φ′ and π θ ′ are the target value network and the target policy network, θ′ is the parameter of the target policy network, γ is the reward discount factor, set γ = 0.95, ε is the pre-set shear noise, r t is the reward value of the robot grasping. When the grasping is successful and the object is stably lifted 20 cm, a positive reward is provided, otherwise a negative reward is given. The grasping reward function is defined as follows:

[0228]

[0229] Step S4-5: Policy Network π θ The overall loss function expression is as follows:

[0230]

[0231] λ is a weighted hyperparameter, which is mainly used to coordinate the balance between expert loss and learning value function loss. Set λ = 0.2, L DDPG (s t , a t ) is the deterministic strategy loss, whose goal is to maximize the Q value, which can be expressed as follows:

[0232] L DDpG (s t , a t )=-Q φ (s t ,a t )

[0233] Step S4-6: Soft update target value network Q φ′ and the target policy network πτ θ neck:

[0234] φ′←τφ+(1-τ)φ′

[0235] θ′←τθ+(1-τ)θ′

[0236] Where φ′ is the target value network Q φ′ , φ is the parameter of the value network Q φ , θ′ is the parameter of the target policy network π θ′ , θ is the parameter of the policy network π θ Parameters, τ is the soft update coefficient, set τ = 0.999, the network training structure is as follows Fig.11 As shown;

[0237] Step S4-7: Repeat the above steps S3 and S4 multiple times to achieve the grasping and grasping optimization of the target object by the robot arm.

[0238] After experimental training, the obtained robotic arm model has a certain improvement in the object grasping effect. The specific results are as follows: Fig.12 As shown (increased from left to right).

[0239] In summary, the present invention can divide the target objects into two categories according to the physical properties of the objects, construct the corresponding enclosing cuboid according to the point cloud distribution of the objects, and generate stable simulated grasping points. With the help of the screening performance of the simulated grasping points, the grasping data can be autonomously and efficiently selected from the grasping data set to perform supervised learning, and combined with the imitation learning and reinforcement learning methods, the grasping posture of the robot arm for the object can be dynamically improved, thereby improving the grasping performance of the robot arm.

[0240] The present invention can also classify the target objects into multiple categories (including but not limited to two categories) according to the classification method of the object point cloud, and can optimize the simulated grasping points as the classification effect improves, so as to better supervise the 6D grasping performance of the robot arm, and has strong applicability and generalization.

[0241] All the above contents are detailed descriptions of the implementation methods of the present invention, and it cannot be determined that the example process of the present invention is limited to the above contents. For the technical field to which the present invention belongs, changes made by relevant technicians within the original scope of the present invention, such as replacement, deletion, etc., also belong to the protection scope of the present invention.

Claims

1. A method for optimizing the 6D grasping posture of a robot arm, characterized in that: The steps include: Step S1: Build an experimental platform for 6D pose grasping of a robotic arm in a simulation environment, and construct a strategy network and a value network; Step S2: sampling the point cloud data observed by the depth camera to obtain the point cloud of the target object, and then outputting a point cloud set with the features of the target object through the point cloud feature network; Step S3: Divide the object point cloud into two categories, and construct cuboids enclosing the two categories of point clouds in sequence along the main direction of the point cloud, so that the long axis of the cuboid (i.e., the central axis passing through the center point of the cuboid and parallel to the longest side of the cuboid) is consistent with the main direction of the point cloud, and then construct a simulated grasping posture for each enclosing cuboid; Step S4: As the gripper approaches the target, the point cloud information of the object gradually becomes richer, the effect of the rectangular block enclosing the point cloud constructed in step S3 gradually becomes better, and the volume of the enclosing rectangular block gradually decreases; Step S5: Filter out two simulated grasping postures corresponding to two cuboids with better effects of enclosing and segmenting the point cloud; Step S6: Select a simulated grasping posture closest to the gripper of the robot arm from the simulated grasping postures obtained in step S5, and find an expert grasping posture used to construct a supervised learning target from the object grasping data set according to the simulated grasping posture; Step S7: setting a heuristic grasping target according to the expert grasping posture, and combining it with the target predicted by the network for supervised learning, using supervised learning to reduce the difference between the network predicted target and the heuristic target, guiding the gripper to approach the object to be grasped, thereby optimizing the gripping posture of the gripper; Step S8: Combine imitation learning and reinforcement learning to optimize the grasping posture.

2. A method for optimizing the 6D grasping posture of a robot arm according to claim 1, characterized in that: The step S2 of obtaining point cloud data specifically includes the following steps: Step S2-1: randomly place an object to be grasped in the scene, obtain the point cloud data of the target object through the RGB-D depth camera, and evenly sample 1024 points from it to form a point cloud set P of the target object; Step S2-2: Use the PointNet point cloud feature network to extract features from the point cloud set P, and use the extracted feature point cloud as the input state S of the deep reinforcement learning network.

3. The method for optimizing the 6D grasping posture of a robot arm according to claim 1, characterized in that: Step S3: Constructing a point cloud bounding cuboid The specific steps include: Step S3-1: Use the K-means clustering algorithm to divide the target object point cloud set P obtained in step S2-1 into two different point cloud subsets P1 and P2; Step S3-2: Obtain point cloud data P j The point cloud center and the main direction of the point cloud are projected to the horizontal direction (i.e., the x and y directions), and the rotation angle μ of the point cloud in the horizontal direction is calculated. j ; Step S3-3: Centralize the point cloud data so that the point cloud is centered on the origin, and then calculate the point cloud data according to μ j Construct a rotation matrix to rotate the point cloud to be parallel to the x-axis, and construct a bounding cuboid around the rotated point cloud And construct the corresponding simulated grasping posture g located in the middle of the cuboid and perpendicular to the long axis of the cuboid j ; Step S3-4: Enclosing the cuboid and simulated grasping pose g j Perform rotation and coordinate offset to complete the enclosing cuboid And its corresponding simulated grasping posture 's construction.

4. The method for constructing a point cloud enclosing cuboid according to claim 3 It is characterized in that The horizontal rotation angle μ of the calculated point cloud described in step S3-2 j , specifically including the following steps: Step S3-2-1: Point cloud data P j (j=1,2) by N j Points p ji =[x ji ,y ji , z ji ] T (i=1,2,...,N j ) to calculate the point cloud center Then subtract the center coordinates from each point Get centralized data The processed point cloud data is centered on the origin O, which is convenient for subsequent covariance matrix calculation; Step S3-2-2: Construct the covariance matrix To represent the distribution of point cloud data on different coordinate axes; Step S3-2-3: Construct characteristic equation C using covariance matrix j v j =λ j v j , where v j is a feature vector, representing the point cloud P j (j=1,2) a main direction, λ j is the eigenvalue, corresponding to the variance of the point cloud in the main direction of the eigenvector; Step S3-2-4: Solve the characteristic equation det(C j -λ j I)=0 to obtain the eigenvalue λ j，1 ,λ j，2 ,λ j，3 , sorted by eigenvalue λ j，1 <λ j，2 <λ j，3 , thus obtaining the corresponding eigenvector v j，1 ,v j，2 , v j，3 , select the eigenvector v with the largest eigenvalue j,3 As the first main direction of the point cloud; Step S3-2-5: Set v j，3 The horizontal projection is divided into The inverse tangent function is used to calculate the angle relative to the x-axis. Step S3-2-6: If v j，3 Parallel to the z-axis, that is, the first principal direction is perpendicular to the horizontal plane, then v is used j，2 As the second main direction, v j，2 The horizontal projection is divided into The inverse tangent function is used to calculate the angle relative to the x-axis.

5. The method for constructing a point cloud enclosing cuboid according to claim 3 It is characterized in that The horizontal enclosing cuboid described in step S3-3 and its corresponding simulated grasping posture g j The construction of includes the following steps: Step S3-3-1: Set the angle μ j Convert to radians and take the negative value to get the rotation angle Step S3-3-2: According to the rotation angle α j Construct the rotation matrix R j : Step S3-3-3: The centralized point cloud q obtained in step S3-2-1 ji Applying a rotation matrix Make the main direction of the point cloud parallel to the x-axis, where R j ·q ji is the matrix multiplication operation; Step S3-3-4: Known point cloud With the origin O as the center, construct an enclosing cuboid with the coordinate origin O as the center So that in the base coordinate system, the three sides of the cuboid have coordinate axes parallel to them, and the cuboid The three sides of are: a j =x j，max -x j，min b j =and j，max -and j，min c j =from j，max -With j，min Where (x j，max ,x j，min ,y j，max ,y j，min ,z max ,z j，min ) are vectors Extreme values ​​on various components; Step S3-3-5: Along the horizontal cuboid Construct the long axis j , l j Set the cuboid as the central axis passing through the origin and parallel to the longest side of the cuboid. The three faces facing the gripper of the robot arm are A j , B j , C j , mid-long axis l j Perpendicular to A j and A j Faces intersect at center B j The area of ​​the surface is greater than C j noodle; Step S3-3-6: Set the gripping posture structure according to the robot gripper model, and use six symmetrical points to represent the gripper, which correspond to the palm base, finger base, fingertip and other six key parts of the gripper from top to bottom. The spatial positions of the two points at the palm base coincide, and the six points are all on the same plane. The spatial coordinates of the six gripper reference points are constructed as (x n ,y n , z n )(n=1,2,...,6); Step S3-3-7: Select C j Center of face As the center point of the gripper, set the vector Set the vector for the z-axis (the direction of the gripper) of the gripper The x-axis (vertical to the plane of the jaws) of the jaws is in the direction of the long axis l j Parallel, by vector cross product Get the direction of the gripper's y-axis and construct a horizontal cuboid The simulated grasping pose g j。 6. The method for constructing a point cloud-enclosing cuboid according to claim 3 It is characterized in that The enclosing cuboid described in step S3-4 And its corresponding simulated grasping posture The construction of includes the following steps: Step S3-4-1: Construct the rotation angle μ j The corresponding rotation matrix R′ j ,in According to the rotation matrix R′ j The cuboid Rotate into a cuboid Step S3-4-2: Translate the rotated cuboid back to the original point cloud space position to obtain the final cuboid surrounding the point cloud in is the center coordinate of the point cloud data; Step S3-4-3: Simulate the grasping posture g j In the base coordinate system, it can be expressed as g j =[x j ,y j , z j , α j , β j , γ j ] T ,in Indicates the center point of the gripper Position, [α j , β j , γ j ] T Represents the Euler angle of the gripper, which can be converted into the form of a rotation matrix: Therefore, the simulated grasping posture g can be constructed j The homogeneous transformation matrix Step S3-4-4: According to the rotation matrix R′ j Rotation Matrix and jaw center position Rotate to form the rotated gripper position The corresponding homogeneous transformation matrix: Step S3-4-5: Perform similar operations as step S3-4-2 to translate the rotated gripper to the final enclosing cuboid. The corresponding position of to obtain the simulated grasping posture The homogeneous transformation matrix Step S3-4-6: According to step S3-4-3, the gripper position can be set With the homogeneous transformation matrix The conversion relationship between them is as follows, where h g (·) and h T (·) are all transformation operators, that is, posture The corresponding rotation matrix is Rotation Matrix The corresponding gripper position is Therefore, the final enclosing cuboid can be obtained Corresponding simulated grasping pose 7. The method for optimizing the 6D grasping posture of a robot arm according to claim 1, characterized in that: The final cuboid group is obtained as described in step S5 The corresponding two simulated grasping poses The specific steps include: Step S5-1: Repeat steps S3-2, S3-3 and S3-4 to construct a point cloud enclosing cuboid of the entire target object point cloud P And calculate the volume of the cuboid through step S3-3-4 Step S5-2: Setting the maximum volume threshold If the current time is the initial time, that is, t = 1, then otherwise constant; Step S5-3: Calculate the cuboid at time t Volume if but And save the cuboid The center position and the corresponding simulated grasping point And record the number T=t at this time; if but No change, do not save the cuboid In order to avoid the error causing the volume of the retained cuboid to be too small, a minimum volume threshold is set. Ensure that the volume of all cuboids is greater than Step S5-4: When the target object point cloud is updated, Update to the new target object point cloud bounding box Volume Step S5-5: Pair the cuboids recorded in step S5-3 in pairs to construct m pairs of cuboids, and calculate the Euclidean distance L between the center coordinates of each pair of cuboids. m , and calculate the average of all Euclidean distances Step S5-6: Analyze the cuboid numbers saved in step S5-3. The smaller the volume of the cuboid, the larger the corresponding number T. Filter out the distance between the center points of two cuboids. Then select the pair of cuboids with the largest sum of their numbers (T1+T2) from these cuboid pairs. If there are cuboid pairs with the same sum of numbers, select the pair with the largest sum of their numbers (T1+T2). m The largest cuboid pair, as the two final cuboid groups selected at the current moment Step S5-7: Enclose a cuboid based on each point cloud in step S3-4 The corresponding simulated grasping postures are constructed Therefore, the final cuboid group can be obtained The corresponding two simulated grasping poses 8. The method for optimizing the 6D grasping posture of a robot arm according to claim 1, characterized in that: Step S6 described in the simulation grasping posture Constructing object grasp poses for supervised learning The specific steps include: Step S6-1: The position of the gripper in the base coordinate system at time t is denoted as g t =[x t ,y t , z t , t ,β t , γ t ] T , where [x t ,y t , z t ] T Indicates the center position of the gripper, [α t , β t , γ t ] T represents the Euler angle of the gripper, and uses f(·) transformation to represent the gripper position g at time t t To the 6 gripper reference points (x n ,y n , z n )(n=1,2,...,6), the specific representation method is as follows: f(g t )=[x1(t),y1(t),z1(t),...,x6(t),y6(t),z6(t)] T Step S6-2: Quantify the gripper position g using the point matching loss formula t+1 With g t Distance between: In the formula ||f(g t+1 )-f(g t )||1 means to find the 1-norm, and then from Select the distance g from the robot gripper at time t t The most recent grasping pose In the formula, arg min represents the variable value operation to find the minimum function value; Step S6-3: Each object surface has 100 pre-prepared grasping postures, which constitute the grasping set G of the object to be grasped. The distance simulation grasping posture is calculated using the point matching loss formula. The nearest expert grasp pose 9. The method for optimizing the 6D grasping posture of a robot arm according to claim 1, characterized in that: Step S7: the improved target grasping posture based on the one obtained in step S6 and the current position of the gripper g t , construct the heuristic goal t t , specifically including the following steps: Step S7-1: Combined with step S3-4-6, define the homogeneous transformation matrix h g (t t ) is the current gripper position g t To the target gripper position The transfer matrix is ​​calculated as follows: Step S7-2: According to the calculation relationship of step S7-1 and the conversion operator set in step S3-4-6, the improved heuristic target can be obtained, where (h g (g t )) -1 is the gripper position g t The corresponding inverse transformation of the homogeneous transformation matrix: Step S7-3: Using the policy network π constructed in step S2 θ and value network Q φ Generate the corresponding network prediction target and Combined with the formula proposed in step S6-2, the loss function of the network prediction target and the heuristic target is calculated as follows: Where θ is the parameter of the policy network, and φ′ is the parameter of the target value network in the DDPG network, which is used to update the parameters of the main value network.

10. The method for optimizing the 6D grasping posture of a robot arm according to claim 1, characterized in that: The combination of imitation learning and deep reinforcement learning loss described in step S8 ultimately optimizes the grasping posture, and specifically includes the following steps: Step S8-1: Assume that the environment in which the robot is located is completely observable. At each time step t, the agent changes from the current state s t Take action a t After reaching state s t+1 , where state s t Represents the component of the point cloud feature S obtained from step S2-2 at time step t, and the reward r of the current action is obtained at the same time t The robot generates a large number of trajectory samples (s t , a t , r t ,s t+1 ), store these samples in the experience replay pool to train the reinforcement learning network; Step S8-2: The policy network output action is recorded as a t , that is, the gripper posture g at time step t+1 t+1 The gripper position g at time step t t The pose offset is calculated as follows: h g (a t )=(h g (g t )) -1 h g (g t+1 ) a t =h T ((h g (g t )) -1 h g (g t+1 ))=[Δx,Δy,Δz,Δα,Δβ,Δγ] Where Δx, Δy, Δz are position offsets, and Δα, Δβ, Δγ are angle offsets. In the autonomous decision-making process of the robot arm, the trajectory of the gripper from the 0th time step to the Tth time step is T r ={g0, g0, ..., g T }, where g T Indicates the gripper position when the maximum number of grasps is reached; Step S8-3: Use the expert planner OMG to generate state s t Expert Actions Combined with the policy network output action a in step S8-2 t And the point matching loss formula proposed in step S6-2, construct the loss function of behavior cloning: This loss helps prevent the strategy learned by the policy network from deviating too much from the expert strategy, while ensuring that the robot arm can approach the target object in the early stages of training, reducing the time the robot arm spends exploring the space; Step S8-4: Value Network Q φ The overall loss function expression is as follows: y represents the target Q value to be approached, and its expression is as follows: y=r t +γQ φ′ (s t+1 ,π θ′ (s t+1 )+ε) Where Q φ′ and π θ′ are the target value network and the target policy network, θ′ is the parameter of the target policy network, γ is the reward discount factor, set γ = 0.95, ε is the pre-set shear noise, r t is the reward value of the robot grasping. When the grasping is successful and the object is stably lifted 20 cm, a positive reward is provided, otherwise a negative reward is given. The grasping reward function is defined as follows: Step S8-5: Policy network π θ The overall loss function expression is as follows: λ is a weighted hyperparameter, which is mainly used to coordinate the balance between expert loss and learning value function loss. Set λ = 0.2, L DDPG (s t , a t ) is the deterministic strategy loss, whose goal is to maximize the Q value, which can be expressed as follows: L DDPG (s t ,a t )=-Q φ (s t ,a t ) Step S8-6: Soft update target value network Q φ′ and the target policy network π θ′ : φ′←τφ+(1-τ)φ′ θ′←τθ+(1-τ)θ′ Where φ′ is the target value network Q φ′ , φ is the parameter of the value network Q φ , θ′ is the parameter of the target policy network π θ′ , θ is the parameter of the policy network π θ Parameters, τ is the soft update coefficient, set τ = 0.999; Step S8-7: Repeat the above steps S3 to S8 multiple times to achieve the grasping and grasping optimization of the target object by the robot arm.

Images