A method for improving a mechanical arm grasping posture by fusing reinforcement learning

Abstract

The application discloses a mechanical arm grasping posture improvement method fusing reinforcement learning and belongs to the field of robot intelligent control.The application combines the intelligent decision-making capability of reinforcement learning and the accurate prediction and optimization capability of model predictive control, the agent of reinforcement learning interacts with the environment, on the basis of the current environment state, cooperates with the feedback of the reward function, constantly explores the optimized control strategy, the weight parameter of the model predictive controller is no longer artificially modified, but is learned by reinforcement learning according to the current system state, and the problem that the model predictive control is strongly dependent on parameters is optimized to a certain extent, and the adaptability of the system to the environment is improved.

Description

Technical Field

[0001] This invention belongs to the field of robot intelligent control, specifically involving a method for improving the grasping posture of a robotic arm in a kitchen setting using reinforcement learning. Background Technology

[0002] With the advancement of technology and the development of industrialization, robotic arms are playing an increasingly important role in both daily life and industry. As robotic arms are increasingly used in logistics sorting, household services, and precision assembly, their autonomous grasping capabilities have become the core for achieving complex tasks. However, traditional grasping planning methods often rely on pre-defined geometric models and rules (such as calculating the object's center of mass or the force closure point of parallel grippers). While this works reasonably well in structured industrial scenarios, the adaptability of these methods drops sharply when faced with irregularly shaped, stacked, or diverse unknown objects, making it difficult to generate stable and reliable grasping points.

[0003] To enhance adaptability to unstructured environments, reinforcement learning methods have been introduced into robotic arm control. Existing research largely focuses on directly outputting joint torques or grasping actions end-to-end from visual input, which does indeed enhance adaptability to the environment to some extent. However, this end-to-end reinforcement learning framework has significant shortcomings at the grasping point decision level: on the one hand, due to the lack of physical constraints during the exploration process, grasping points with strange postures or poor alignment between the gripper and the object's edge often occur, leading to low grasping success rates and safety hazards; on the other hand, to cover the high-dimensional state space, the training process often requires millions of trial and error samples, resulting in high training costs and a significant "reality gap" problem when transferring from simulation to reality, leading to joint control jitter and unstable grasping execution. Summary of the Invention

[0004] To address the above shortcomings, this invention combines reinforcement learning with traditional grasping methods. By interacting with the environment, it improves the grasping posture, increases the success rate of grasping, and also improves the grasping effect on irregular objects.

[0005] The technical solution of this invention is: a method for improving the grasping posture of a robotic arm by incorporating reinforcement learning, the method comprising the following steps:

[0006] S1. Obtain the initial gripping point of the object;

[0007] S2. Develop a reinforcement learning module to improve the initial grasp point;

[0008] S3. Add domain randomization processing and course learning to increase the generalization of the target object shape and accelerate the convergence speed.

[0009] S4. Reachability detection and scene collision detection for the final grasping posture.

[0010] Furthermore, in step S1, the initial grasping point of the object is obtained:

[0011] GraspNet is used as the base model to obtain the initial grasping pose of the target object. First, the scene's RGB and depth images are obtained from images acquired by a single-view camera. After being converted into point cloud format, the scene point cloud is obtained. The point cloud of the target object is then segmented and input into the GraspNet model to obtain the initial grasping points of the target object. GraspNet mainly uses three loss functions during training:

[0012] ;

[0013] in, Indicates near-network loss. Point The grabbability binary result, Point From the perspective The confidence level is below. Represents the actual value. Indicates the angle difference. This represents the binary classification softmax loss. Indicates smoothness loss, Indicates the total number of samples in the classification. Point Real tags for crawlability This represents the weighting balance coefficient. This represents the total number of regression samples. Indicates the direction of approach predicted by the network. Indicates the actual direction of approach;

[0014] ;

[0015] in, Indicates operational network loss. These represent the grasp confidence level, gripper width, and proximity distance, respectively. This represents the classification probability of the in-plane rotation angle predicted by the network. This represents the true confidence level of the crawl. The actual label representing the rotation angle in the plane. This indicates the total number of boxes that are close to each other. This represents the weighting balance coefficient. Indicates the first The classification loss function under distance binning, Indicates the first Regression loss function under distance bins This indicates the actual gripper width;

[0016] ;

[0017] in, This indicates the model's tolerance loss. This indicates the maximum disturbance distance that the grasping posture can withstand. This indicates the maximum disturbance distance that the actual grasping posture can withstand;

[0018] The final loss function is obtained by weighting the above three losses. :

[0019] ;

[0020] in, These are the weight parameters of the loss function.

[0021] Furthermore, the specific method for writing the reinforcement learning module in step S2 to improve the initial grasping points in S1 is as follows:

[0022] The PPO algorithm is used for training. Based on the current state: the point cloud information of the target object, the initial grasping point of the target object in step S1, the object's position in the gripper coordinate system, the gripper joint angles, and the information from the gripper collision sensor, the modification amount for the initial grasping point is obtained. The PPO loss function... for:

[0023] ;in, This indicates taking the expected value. This indicates an editing mechanism that limits the range of variation in the importance sampling ratio and prevents excessive strategy updates; Indicates the current strategy. Indicates the importance sampling ratio, Represents the dominance function. Indicates the editing factor. Weight parameters representing value loss, Indicates the actual value of the current action. Indicates the target value. Represents the entropy coefficient. Represents the entropy of the strategy.

[0024] Furthermore, in step S3, domain randomization and course learning are added to increase the generalization of the target object shape and accelerate the convergence speed.

[0025] S3.1 Added domain randomization processing, which randomizes the size, shape, and position of the target object, the interference force received by the gripper, and the mass of the object, making it easier to deploy the model in sim2real.

[0026] S3.2 Add course learning. The entire task is divided into two stages. The first stage encourages the gripper to approach the graspable position of the target object. The second stage encourages the gripper to firmly grasp the object.

[0027] Furthermore, in step S4, the final reachability detection and scene collision detection of the grasping posture are performed:

[0028] S4.1 Obtain the processed grasping posture and check whether the end effector of the robotic arm can achieve the final grasping posture in the coordinate system of the robotic arm base.

[0029] S4.2. The model of the gripper needs to be obtained. The gripper model is processed into a point cloud. The scene point cloud and the target object point cloud are separated in the scene point cloud. The distance between the gripper point cloud and the scene point cloud is calculated in turn. The gripping posture of the collision scene is excluded.

[0030] Furthermore, in step S2, during reinforcement learning, the observation state space for reinforcement learning is first determined. and output motion space Reinforcement learning requires the current observation state as input. The observation state space is composed of multiple state vectors.

[0031] in, This represents the point cloud information of the target object at time k. This represents the initial capture point information at time k. This represents the spatial pose of the object relative to the gripper at time k. This represents the joint angle of the gripper at time k. This represents the collision information between the gripper and the object at time k.

[0032] Before each training round begins, the system state needs to be initialized. Based on the input observation state Reinforcement learning decision networks will output the optimal action for the current state. ;

[0033] in, Indicates the position of the gripper in space;

[0034] Reinforcement learning undergoes a Markov decision process during training; reinforcement learning will adjust its actions based on the current output. Make a state transition, from state Transition to state In reinforcement learning, the value network will adjust according to the state. Give action The score determines the reward value for this action, which is composed of multiple parts, as follows:

[0035] ;

[0036] in, The weights representing the reward values Indicates the penalty for an object falling. The reward indicates the collision between the gripper's fingers and an object. This indicates the penalty for the remaining parts of the gripper colliding with the object. This refers to the reward for the gripper holding an object and maintaining it. This indicates a reward for the gripper successfully securing the object. This indicates a penalty for an object exceeding the gripper's height.

[0037] This invention combines the intelligent decision-making capabilities of reinforcement learning with the accurate prediction and optimization capabilities of model predictive control. Reinforcement learning interacts with the environment through an agent, and based on the current environmental state and the feedback from the reward function, continuously explores and optimizes the control strategy. The weight parameters of the model predictive controller are no longer manually modified, but are learned by reinforcement learning based on the current system state. This optimizes the problem of strong parameter dependence in model predictive control to a certain extent and improves the system's adaptability to the environment. Attached Figure Description

[0038] Figure 1 This is a flowchart of the method for adjusting the grasping posture of a robotic arm using reinforcement learning, as proposed in this invention. Detailed Implementation

[0039] The following is in conjunction with the appendix Figure 1 The flowcharts provide a detailed description of embodiments of the present invention.

[0040] First, it is necessary to obtain the initial grasping pose of the target object. This invention uses GraspNet as the basic model. The RGB image and depth map of the scene are obtained through the camera, converted into point cloud format, and the point cloud of the target object is segmented. Then, it is input into the GraspNet network model to obtain the initial grasping pose of the target object.

[0041] To adjust the gripper's grasping posture using reinforcement learning, the observation state space for reinforcement learning must first be determined. There is also room for output. Reinforcement learning requires the current observation state as input. The observation state space is composed of multiple state vectors.

[0042] in, This represents the point cloud information of the target object at time k. This represents the initial capture point information at time k. This represents the spatial pose of the object relative to the gripper at time k. This represents the joint angle of the gripper at time k. This represents the collision information between the gripper and the object at time k.

[0043] Before each training round begins, the system state needs to be initialized. Based on the input observation state Reinforcement learning decision networks will output the optimal action for the current state. ;

[0044] in, Indicates the position of the gripper in space;

[0045] Reinforcement learning involves a Markov decision process during training. Reinforcement learning will adjust its actions based on the current output. Make a state transition, from state Transition to state In reinforcement learning, the value network will adjust according to the state. Give action The score determines the reward value for this action, which is composed of multiple parts, as follows:

[0046] ;

[0047] in, The weights representing the reward values Indicates the penalty for an object falling. The reward indicates the collision between the gripper's fingers and an object. This indicates the penalty for the remaining parts of the gripper colliding with the object. This refers to the reward for the gripper holding an object and maintaining it. This indicates a reward for the gripper successfully securing the object. This indicates a penalty for an object exceeding the gripper's height.

[0048] After obtaining the reward value, the PPO network can calculate the loss function.

[0049] ;in, This indicates taking the expected value. This indicates an editing mechanism that limits the range of variation in the importance sampling ratio to prevent excessive strategy updates. Indicates the current strategy. Indicates the importance sampling ratio, Represents the dominance function. Indicates the editing factor. Weight parameters representing value loss, Indicates the actual value of the current action. Indicates the target value. Represents the entropy coefficient. Represents the entropy of the strategy.

[0050] Before updating, you must set the hyperparameters, including the learning rate. There are also discount factors. Based on the loss function, the parameters of the reinforcement learning decision network are updated in reverse. Update value network parameters using the TD algorithm The formula is as follows:

[0051] ;

[0052] ;

[0053] ;

[0054] ;

[0055] in, Indicated in the new strategy state Take action below The probability of. Indicating the old strategy state Take action below The probability of. This represents the dominance function. This indicates the editing parameters. This represents the learning rate of the policy network. Indicates the old state The reward value below. Indicates the next state The reward value. Indicates the state Below, the immediate reward obtained after taking an action. This represents the discount factor that reduces the reward. This represents the learning rate of the value network.

[0056] After obtaining the action parameters output by the reinforcement learning decision network, the action parameters are inversely normalized to restore the actual gripper pose adjustment, which is then added to the initial gripping posture to adjust it. It is worth noting that during training, due to the severe action and reward delay issues in this task, the conventional PPO algorithm cannot be used; modifications and innovations are needed at the network update level. During algorithm updates, the actor and critic networks in the PPO algorithm are innovatively updated as two separate parts, each using different historical trajectories. This is because in the task of adjusting the gripping posture, the actions generated by the actor network are used discretely, but the score of the current observation state needs to be estimated continuously using the critic network. This is the fundamental contradiction; the two networks operate on different trajectories, so it is not possible to use the same entire historical trajectory to update both networks simultaneously as in the ordinary PPO algorithm. The processing method of this invention is to perform calibration. When generating action parameters, the action parameters are labeled according to the characteristics of the observation environment. Only the action parameters generated by the actor network are labeled, and their corresponding historical trajectories are also labeled. When updating the actor network, only the labeled historical trajectories are used to update the actor network, and all historical trajectories are used to update the critic network. This is the key to this invention.

[0057] In addition, domain randomization and learning were added. Domain randomization is essential for the successful implementation of sim2real, randomizing the size, shape, weight, and position of the target object, as well as the disturbance forces received by the gripper, to better simulate real-life situations. Learning was added to accelerate the convergence speed and effectiveness of reinforcement learning and prevent the undesirable result of getting trapped in local optima. Learning divides the entire training process into two parts: the first part encourages the gripper to continuously approach the target object, and the second part encourages the gripper to firmly grasp the object. The second part is the focus of the entire task, and the training iterations for the two parts of the task are reasonably allocated.

[0058] Finally, there is pose reachability detection and scene collision handling for the grasping posture. The previously processed grasping posture is only relative to the target object and does not actually take into account the reachability of the robotic arm end effector and scene collision. After obtaining the adjusted grasping posture, it is necessary to perform end effector reachability detection and scene collision detection to eliminate grasping postures that the robotic arm end effector cannot reach. Then, the model of the gripper is extracted, converted into a point cloud, and the distance between the gripper point cloud and the scene point cloud is calculated to eliminate unachievable grasping postures.

[0059] Model predictive control (MMCC) is derived based on a robotic arm model. Under the conditions of satisfying the robotic arm's safety constraints and joint limitations, it predicts the future state of the robotic arm, receives weight parameters from reinforcement learning output, optimizes the cost function, and obtains the optimal control input for the current moment. Compared to other control methods, this approach better balances the system's adaptability to the environment and its safety.

[0060] The final grasping results, compared with the original GraspNet network grasping results without post-processing, show that when there are no occlusions or interfering objects and it is a single-view camera, the shape of the object is diverse. The success rate of GraspNet network is 70%, while the grasping success rate of this invention is 75%. When there are occlusions and interfering objects (collision in grasping pose) and it is a single-view camera, the success rate of GraspNet is 30%, while the success rate of this invention is close to 45%.

Claims

1. A method for improving the grasping posture of a robotic arm by incorporating reinforcement learning, the method comprising the following steps: S1. Obtain the initial gripping point of the object; S2. Develop a reinforcement learning module to improve the initial grasp point; S3. Add domain randomization processing and course learning to increase the generalization of the target object shape and accelerate the convergence speed. S4. Reachability detection and scene collision detection for the final grasping posture.

2. The method for improving the grasping posture of a robotic arm by incorporating reinforcement learning as described in claim 1, characterized in that, In step S1, the initial grasping point of the object is obtained: GraspNet is used as the base model to obtain the initial grasping pose of the target object. First, the scene's RGB and depth images are obtained from images acquired by a single-view camera. After being converted into point cloud format, the scene point cloud is obtained. The point cloud of the target object is then segmented and input into the GraspNet model to obtain the initial grasping points of the target object. GraspNet mainly uses three loss functions during training: ； in, Indicates near-network loss. Point The grabbability binary result, Point From the perspective The confidence level is below. Represents the actual value. Indicates the angle difference. This represents the binary classification softmax loss. Indicates smoothness loss, Indicates the total number of samples in the classification. Point Real tags for crawlability This represents the weighting balance coefficient. This represents the total number of regression samples. Indicates the direction of approach predicted by the network. Indicates the actual direction of approach; ； in, Indicates operational network loss. These represent the grasp confidence level, gripper width, and proximity distance, respectively. This represents the classification probability of the in-plane rotation angle predicted by the network. This represents the true confidence level of the crawl. The actual label representing the rotation angle in the plane. This indicates the total number of boxes that are close to each other. This represents the weighting balance coefficient. Indicates the first The classification loss function under distance binning, Indicates the first Regression loss function under distance bins This indicates the actual gripper width; ； in, This indicates the model's tolerance loss. This indicates the maximum disturbance distance that the grasping posture can withstand. This indicates the maximum disturbance distance that the actual grasping posture can withstand; The final loss function is obtained by weighting the above three losses. : ； in, These are the weight parameters of the loss function.

3. The method for improving the grasping posture of a robotic arm by incorporating reinforcement learning as described in claim 1, characterized in that, The specific method for writing the reinforcement learning module in step S2 to improve the initial grasping points in S1 is as follows: The PPO algorithm is used for training. Based on the current state: the point cloud information of the target object, the initial grasping point of the target object in step S1, the object's position in the gripper coordinate system, the gripper joint angles, and the information from the gripper collision sensor, the modification amount for the initial grasping point is obtained. The PPO loss function... for: ;in, This indicates taking the expected value. This indicates an editing mechanism that limits the range of variation in the importance sampling ratio and prevents excessive strategy updates; Indicates the current strategy. Indicates the importance sampling ratio, Represents the dominance function. Indicates the editing factor. Weight parameters representing value loss, Indicates the actual value of the current action. Indicates the target value. Represents the entropy coefficient. Represents the entropy of the strategy.

4. The method for improving the grasping posture of a robotic arm by incorporating reinforcement learning as described in claim 1, characterized in that, In step S3, domain randomization and course learning are added to increase the generalization of the target object shape and accelerate the convergence speed. S3.1 Added domain randomization processing, which randomizes the size, shape, and position of the target object, the interference force received by the gripper, and the mass of the object, making it easier to deploy the model in sim2real. S3.2 Add course learning. The entire task is divided into two stages. The first stage encourages the gripper to approach the graspable position of the target object. The second stage encourages the gripper to firmly grasp the object.

5. The method for improving the grasping posture of a robotic arm by incorporating reinforcement learning as described in claim 1, characterized in that, Step S4 ultimately involves reachingability detection and scene collision detection for the grasping posture: S4.1 Obtain the processed grasping posture and check whether the end effector of the robotic arm can achieve the final grasping posture in the coordinate system of the robotic arm base. S4.

2. The model of the gripper needs to be obtained. The gripper model is processed into a point cloud. The scene point cloud and the target object point cloud are separated in the scene point cloud. The distance between the gripper point cloud and the scene point cloud is calculated in turn. The gripping posture of the collision scene is excluded.

6. The method for improving the grasping posture of a robotic arm by incorporating reinforcement learning as described in claim 1, characterized in that, In step S2, during reinforcement learning, the observation state space for reinforcement learning is first determined. and output motion space ; Reinforcement learning requires the current observation state as input. The observation state space is composed of multiple state vectors. in, This represents the point cloud information of the target object at time k. This represents the initial capture point information at time k. This represents the spatial pose of the object relative to the gripper at time k. This represents the joint angle of the gripper at time k. This represents the collision information between the gripper and the object at time k. Before each training round begins, the system state needs to be initialized. Based on the input observation state Reinforcement learning decision networks will output the optimal action for the current state. ; in, Indicates the position of the gripper in space; Reinforcement learning undergoes a Markov decision process during training; reinforcement learning will adjust its actions based on the current output. Make a state transition, from state Transition to state In reinforcement learning, the value network will adjust according to the state. Give action The score determines the reward value for this action, which is composed of multiple parts, as follows: ； in, The weights representing the reward values Indicates the penalty for an object falling. The reward indicates the collision between the gripper's fingers and an object. This indicates the penalty for the remaining parts of the gripper colliding with the object. This refers to the reward for the gripper holding an object and maintaining it. This indicates a reward for the gripper successfully securing the object. This indicates a penalty for an object exceeding the gripper's height.

Images