Five-finger flexible manipulator grasping action optimization method based on reinforcement learning

Abstract

The present application belongs to the field of mechanical hand grasping action algorithm, and particularly relates to a five-finger flexible mechanical hand grasping action optimization method based on reinforcement learning, comprising: S1: defining the state space of the system, S2: implementing a reinforcement learning strategy, S3: training a generative adversarial network while generating a style reward R1, S4: obtaining a task reward R2, combining the style reward R1 to generate a reward sum R obtained by the mechanical hand at the tth step t , calculating its long-term return returns, S5: repeating the steps of S3-S5 to complete the training of all stages and obtain a final reinforcement learning model, S6: inputting information of an object that has not been grasped by a trained mechanical hand or a data glove into the trained model to verify the result of the mechanical hand when grasping.

Description

Technical Field

[0001] This invention belongs to the field of robotic arm grasping motion algorithms, specifically involving an optimization method for grasping motions of a five-finger flexible robotic arm based on reinforcement learning. Background Technology

[0002] Five-finger dexterous robotic hand grasping control is an important direction in robot development, involving multiple aspects such as perception, planning, and control. It plays a crucial role in industrial fields such as manufacturing and logistics, as well as civilian fields such as prosthetics and exoskeletons for assisted grasping. In practical applications, the design of controllers for five-finger dexterous robotic hands remains challenging, with shortcomings in grasping accuracy, precision, and flexibility, which seriously affect the grasping efficiency and widespread application of five-finger dexterous robotic hands.

[0003] Currently, in order to enable five-fingered dexterous robotic hands to adapt well to the physical properties, initial state, motion state and task objectives of objects when facing unstructured environments, and to ensure the flexibility, stability and adaptability of grasping control, self-learning five-fingered dexterous robotic hands have long been designed. Through multiple training tasks, they can adaptively adjust grasping strategies to improve the accuracy and precision of grasping.

[0004] For example, the Chinese invention patent with announcement number "CN111260072A" discloses "a reinforcement learning exploration method based on generative adversarial networks". It proposes a GANE method to train a generator and generate virtual state samples of the robot. This method depends on the network structure and the adjustment of hyperparameters, which has a high time cost. Moreover, the quality of the virtual samples generated by the generator network is not high, and the excessive amount of computation causes problems such as training instability and mode collapse, resulting in the generated motion behavior not being realistic enough. Summary of the Invention

[0005] Therefore, the technical problem to be solved by this invention is to overcome the fact that the current self-learning grasping method of five-fingered dexterous robotic hands mainly relies on network generators. The adjustment of hyperparameters is time-consuming, and the quality of virtual samples generated by the generator network is not high. The excessive amount of computation causes problems such as training instability and mode collapse, resulting in the generated motion behavior not being realistic enough.

[0006] To address the aforementioned technical problems, this invention provides a method for optimizing the grasping action of a five-finger flexible robotic hand based on reinforcement learning, comprising the following steps: S1: Define the system's state space, i.e., the state information when the operator, wearing data gloves, performs step t. This includes the coordinates of the five-fingered robotic hand in space, the coordinates of the centroid of the grasped target, the angles, joint velocities, and angular velocities of each finger joint, as well as the coordinates of the fingertips contacting the target object. The trajectory of the operator wearing data gloves is then converted and calculated into expert information. S2: Implement reinforcement learning strategy: Construct multi-stage training tasks for the robotic arm from easy to difficult. Before constructing each new stage of training task, expert information needs to be collected once, and the same object is required to be grasped when performing each new stage of training task. Phase 1: The robotic arm makes numerous attempts to grasp evenly distributed objects, receiving feedback each time. Using this feedback, the robotic arm gradually adjusts and optimizes its grasping motions to achieve accurate grasping of evenly distributed objects. During training, the reinforcement learning model continuously interacts with the environment, adjusting its strategy based on environmental feedback to maximize cumulative rewards. The second stage involves grasping a large number of objects with larger volume and mass to learn the extreme posture and torque during grasping. The parameters in the reinforcement model of the first stage are initialized into the reinforcement learning model of the second stage to accelerate the convergence speed of the model and improve training efficiency. The third stage involves grasping softer toys and learning the posture and torque for grasping irregular objects. The parameters from the reinforcement learning model in the second stage are used to initialize the third stage reinforcement learning model to further optimize the gripping ability of the robotic arm. The final reinforcement learning model is obtained after all training stages are completed. S3: Perform the following operations for each training task in S2: First, feed the collected expert information and the robot's real-state information into the discriminator D to train the generative adversarial network, and simultaneously generate style rewards. ; Discriminator D, which has an internal network, is continuously trained to distinguish whether the input information is expert information or real state information. The network parameters of discriminator D are updated with the minimum loss function, and the output is a scalar d, which is in the range of [0,1]. A value close to 1 indicates that discriminator D considers the input data to be expert information, and a value close to 0 indicates that discriminator D considers the input data to be real state information from the five-fingered robotic hand. During each training phase, discriminator D receives input data from expert information and the real-world state information of the five-fingered robotic hand, and learns different features and patterns to distinguish between the two. By continuously adjusting the network parameters, discriminator D gradually improves its ability to distinguish between expert information and real-world state information, thus enabling it to more accurately determine the source of the input data. Throughout the training process of the five-fingered robotic hand system, the learning ability and accuracy of discriminator D will continuously improve; style reward. Describe the differences between real-world information and expert information regarding the five-fingered robotic hand; ; As the robot's real-world information closely approximates the expert information, style reward increases. The larger; S4: Following the reinforcement learning strategy in S2, based on the current input state information... Output the action that the five-finger robotic hand should perform. The five-fingered robotic arm performs the selected action. Task rewards obtained after interacting with the environment And update the state of the robotic arm based on the result of the action execution. Combined with the style rewards Generate the total reward obtained by the robotic arm in step t. Calculate its long-term return value and use it as a feedback signal to update the reinforcement learning algorithm; S5: The training process iterates continuously until the termination condition is met, at which point the best reinforcement learning model for that stage is obtained. The parameters of the previous stage of reinforcement learning are initialized for the next stage. Steps S3 to S5 are repeated to complete the training of all stages and obtain the final reinforcement learning model. S6: Validate the reinforcement learning model by inputting the unknown target into the trained model and validating the training results.

[0007] Preferably, the This includes the coordinates of the data glove in space, the coordinates of the centroid of the grasped target, the angles of each finger joint, joint velocities and angular velocities, and the coordinates of the fingertips contacting the target object.

[0008] Preferred method for converting operator's trajectory while wearing data gloves into expert information: The operator wears data gloves to grasp an object and obtain a series of action sequences, calculates the angles of each joint through inverse kinematics, calculates the hand position using forward kinematics, and calculates the joint velocity and angular velocity using finite difference kinematics.

[0009] Preferably, the task reward Depend on , and It consists of 3 parts. To provide a positive reward, ensure that the finger joints remain within the threshold range for each joint angle. To achieve a negative reward, the robotic arm's coordinates and the target's coordinates should be as close as possible; a successful grab yields a reward. Positive reward;

[0010] ;

[0011] ;

[0012] ;

[0013] Where C is a constant, set according to the actual situation; This represents the angle of the i-th joint. The threshold for the angle of the i-th joint, ( , , ) represents the centroid coordinates of the target being grasped, ( , , () represents the coordinates of the robotic arm.

[0014] Preferably, the total reward obtained by the robotic arm when performing step t. for:

[0015]

[0016] in , These are the weights for style rewards and task rewards, respectively. In a crawling task, the success or failure of the crawl is related to... The relationship is more significant, so when setting weights, It can be larger; the specific value can be designed according to the actual situation.

[0017] Preferably, Returns is defined as the long-term return, and its calculation method is as follows: Returns= , 0 < <1，

[0018] Here, γ is a discount factor, meaning that the current reward is more important than the reward of future feedback.

[0019] The technical solution of this invention has the following beneficial effects: This invention incorporates expert information to help a five-fingered dexterous robotic hand better understand and perform grasping tasks. By introducing data and knowledge from various scenarios during the progressive training phase, the robotic hand can learn effective grasping strategies more quickly, thereby improving the accuracy and efficiency of grasping.

[0020] The present invention is based on reinforcement learning. The robot's motion behavior learned directly through interaction with the environment is used as a real state sample. No additional generator is needed to generate virtual samples. The sample is directly input into the discriminator D along with expert information. The discriminator D continuously optimizes by comparing the differences between the two. The training results directly affect the robot's motion behavior. This method simplifies the model structure and training process, making the training results more accurate. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of the grasping action optimization method and system provided in the embodiments of the present invention;

[0022] Figure 2This is a flowchart of the grasping action optimization method provided in an embodiment of the present invention;

[0023] Figure 3 This is a flowchart of a multi-stage training process for reinforcement learning provided in an embodiment of the present invention;

[0024] Figure 4 This is a schematic diagram of the discriminator D network provided in an embodiment of the present invention.

[0025] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort. Detailed Implementation

[0026] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0027] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or component referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0028] This invention provides a method for optimizing the grasping action of a five-finger dexterous robotic hand based on reinforcement learning, comprising the following steps:

[0029] S1: Define the system's state space, i.e., the state information of each object when the operator, wearing data gloves, grasps different objects and executes step t. Next, the trajectory of the operator wearing data gloves is converted and calculated into expert information; the purpose of defining expert information is to provide a reference standard for the robot arm when learning and imitating, and also to make the robot arm learn faster.

[0030] S2: Implement reinforcement learning strategy: Construct multi-stage training tasks for the robotic arm from easy to difficult. Before constructing each new stage of training task, expert information needs to be collected once, and the same object is required to be grasped when performing each new stage of training task.

[0031] Phase 1: The robotic arm makes numerous attempts to grasp evenly distributed objects, receiving feedback each time. Using this feedback, the robotic arm gradually adjusts and optimizes its grasping actions to achieve accurate grasping of evenly distributed objects. During training, the reinforcement learning model continuously interacts with the environment, adjusting its strategy based on environmental feedback to maximize cumulative reward. Through continuous trial and error and learning, the five-fingered robotic arm gradually establishes an effective strategy for grasping tennis balls, ultimately achieving the training objective of Phase 1. Maximizing Cumulative Reward: Each time the robotic arm interacts with the environment, the environment awards a reward value based on the robotic arm's actions. After many interactions, a complete grasping process is completed, and the cumulative reward value is the sum of all rewards for completing this process. The goal of reinforcement learning is to maximize this cumulative reward.

[0032] After training, the robotic arm will receive the first-stage reinforcement learning model, which contains the best strategies learned during the process of grasping the tennis ball. This model will provide reference and guidance for the robotic arm in the next stage, improving the success rate and efficiency of grasping.

[0033] The second stage involves grasping a large number of objects with greater volume and mass to learn the extreme postures and torques during grasping. The parameters from the first stage's reinforcement model are then used to initialize the second-stage reinforcement learning model (the parameters learned in the first stage are assigned to the network in the second stage, and the model trained in the first stage is used to learn the tasks in the second stage). Since the robotic arm already possesses the ability to grasp uniformly distributed objects early in the second stage of learning, the training task in the second stage will be completed more quickly based on this foundation. This accelerates the model's convergence speed and improves training efficiency. This transfer learning method allows the robotic arm to learn and adapt to new grasping tasks more quickly, while retaining previously learned useful knowledge and experience.

[0034] The third stage involves grasping softer toys and learning the posture and torque for grasping irregular objects. The goal of this stage is to enable the robotic arm to adapt to irregular shapes and softer objects, improving its grasping adaptability and accuracy. The parameters from the reinforcement learning model in the second stage are initialized into the third stage reinforcement learning model (this process is essentially the same as the second stage), further optimizing the robotic arm's grasping ability. The final reinforcement learning model is obtained after all training stages are completed.

[0035] The model obtained by this progressive training method is more flexible, efficient and reliable when facing grasping tasks of varying difficulty and complexity, thus improving the overall performance and application scope of the model.

[0036] S3: Perform the following operations for each training task in S2: First, feed the collected expert information and the robot's real-state information into the discriminator D to train the generative adversarial network, and simultaneously generate style rewards. ;

[0037] One of the internal networks of the discriminator D is continuously trained to distinguish whether the input information is expert information or real state information. The discriminator D network parameters are updated with the minimum loss function, and the output is a scalar d in the range of [0,1]. A value close to 1 indicates that the discriminator D considers the input data to be expert information, while a value close to 0 indicates that the discriminator D considers the input data to be real state information from the five-fingered robotic hand.

[0038] During each training phase, discriminator D receives input data from expert information and the real-world state information of the five-fingered robotic hand, and learns different features and patterns to distinguish between the two. By continuously adjusting the network parameters, discriminator D gradually improves its ability to distinguish between expert information and real-world state information, thus enabling it to more accurately determine the source of the input data. Throughout the training process of the five-fingered robotic hand system, the learning ability and accuracy of discriminator D will continuously improve; style reward. The difference between the real-state information and expert information of the five-fingered robotic hand is described, and its calculation method is as follows:

[0039]

[0040] As the robot's real-world information closely approximates the expert information, style reward increases. The larger the style reward The larger the value, the better the learning effect of the robotic arm.

[0041] S4: Following the reinforcement learning strategy in S2, based on the current input... Output the action that the five-finger robotic hand should perform. The five-fingered robotic arm performs the selected action. Task rewards obtained after interacting with the environment And update the state of the robotic arm based on the result of the action execution. Combined with the style rewards Generate the total reward obtained by the robotic arm in step t. Calculate its long-term return value and use it as a feedback signal to update the reinforcement learning algorithm;

[0042] S5: The training process iterates continuously until the termination condition is met, at which point the best reinforcement learning model for that stage is obtained. The parameters of the previous stage of reinforcement learning are initialized for the next stage. Steps S3 to S5 are repeated to complete the training of all stages and obtain the final reinforcement learning model.

[0043] S6: Validate the reinforcement learning model by inputting the unknown target into the trained model and validating the training results.

[0044] Preferred, This includes the coordinates of the data glove in space, the coordinates of the centroid of the grasped target, the angles of each finger joint, joint velocities and angular velocities, and the coordinates of the fingertips contacting the target object.

[0045] Preferred method for converting operator's trajectory while wearing data gloves into expert information: The operator wears data gloves to grasp an object and obtain a series of action sequences, calculates the angles of each joint through inverse kinematics, calculates the hand position using forward kinematics, and calculates the joint velocity and angular velocity using finite difference kinematics.

[0046] Preferred, task rewards Depend on , and It consists of 3 parts. To provide a positive reward, ensure that the finger joints remain within the threshold range for each joint angle. To achieve a negative reward, the robotic arm's coordinates and the target's coordinates should be as close as possible; a successful grab yields a reward. Positive reward;

[0047]

[0048]

[0049]

[0050] Where C is a constant, set according to the actual situation; This represents the angle of the i-th joint. The threshold for the angle of the i-th joint, ( , , ) represents the centroid coordinates of the target being grasped, ( , , () represents the coordinates of the robotic arm.

[0051]

[0052] in , These are the weights for style rewards and task rewards, respectively. In a crawling task, the success or failure of the crawl is related to... The relationship is more significant, so when setting weights, It can be larger; the specific value can be designed according to the actual situation.

[0053]

[0054] Here, γ is the discount factor, a constant between 0 and 1, usually represented by γ. The discount factor controls the importance of future returns; that is, observed returns are given more weight than future returns. The larger the discount factor, the more important future returns are. In decision-making, immediate returns are usually prioritized over potential future returns, meaning that current rewards are more important than future rewards.

[0055] The key to this invention lies in combining generative adversarial networks (deep learning) and reinforcement learning techniques, enabling the robotic arm to continuously adjust and optimize its grasping actions during actual operation to adapt to objects of different shapes, sizes, and weights. Simultaneously, by collecting expert information in advance as a reference, the number of trial-and-error attempts during the actual grasping process can be effectively reduced, thereby improving the grasping success rate.

[0056] Furthermore, this deep reinforcement learning-based approach can help robotic arms adapt and learn more quickly when faced with new grasping tasks, thereby improving their intelligence and flexibility. By continuously accumulating and updating the expert database, the five-fingered dexterous robotic arm can continuously improve its grasping performance in practical applications, achieving more precise and efficient grasping control.

[0057] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A method for optimizing the grasping action of a five-finger dexterous robotic hand based on reinforcement learning, characterized in that, Includes the following steps: S1: Define the system's state space, i.e., the state information when the operator, wearing data gloves, performs step t. Next, the trajectory of the operator wearing data gloves is converted into expert information; S2: Implement reinforcement learning strategy: Construct multi-stage training tasks for the robotic arm from easy to difficult. Before constructing each new stage of training task, expert information needs to be collected once, and the same object is required to be grasped when performing each new stage of training task. Phase 1: The robotic arm makes numerous attempts to grasp evenly distributed objects, receiving feedback each time. Using this feedback, the robotic arm gradually adjusts and optimizes its grasping actions to achieve accurate grasping of evenly distributed objects. During training, the reinforcement learning model continuously interacts with the environment, adjusting its strategy based on environmental feedback to maximize cumulative rewards: each time the robotic arm interacts with the environment, the environment assigns a reward value based on the robotic arm's action. After many interactions, a complete grasping process is completed, and the cumulative reward value is the sum of all rewards for completing this process. The second stage involves grasping a large number of objects with larger volume and mass to learn the extreme posture and torque during grasping. The parameters in the reinforcement model of the first stage are initialized into the reinforcement learning model of the second stage to accelerate the convergence speed of the model and improve training efficiency. The third stage involves grasping softer toys and learning the posture and torque for grasping irregular objects. The parameters from the reinforcement learning model in the second stage are used to initialize the third stage reinforcement learning model to further optimize the gripping ability of the robotic arm. The final reinforcement learning model is obtained after all training stages are completed. S3: Perform the following operations for each training task in S2: First, feed the collected expert information and the robot's real-state information into the discriminator D to train the generative adversarial network, and simultaneously generate style rewards. ; One of the internal networks of the discriminator D is continuously trained to distinguish whether the input information is expert information or real state information. The network parameters of the discriminator D are updated with the minimum loss function, and the output is a scalar d, which is in the range of [0,1]. A value close to 1 indicates that the discriminator D considers the input data to be expert information, and a value close to 0 indicates that the discriminator D considers the input data to be real state information from the five-fingered robotic hand. During each stage of training, the discriminator D receives input data from expert information and the real state information of the five-fingered robotic hand, and learns different features and patterns to distinguish between the two. By continuously adjusting network parameters; style rewards Describe the differences between real-world information and expert information regarding the five-fingered robotic hand; ； As the robot's real-world information closely approximates the expert's information, style reward increases. The larger; S4: Following the reinforcement learning strategy in S2, based on the current input state information... Output the action that the five-finger robotic hand should perform. The five-fingered robotic arm performs the selected action. Task rewards obtained after interacting with the environment And update the state of the robotic arm based on the result of the action execution. Combined with the aforementioned style rewards Generate the total reward obtained by the robotic arm in step t. Calculate its long-term return value (Returns) and use it as a feedback signal to update the reinforcement learning algorithm; S5: The training process iterates continuously until the termination condition is met, at which point the best reinforcement learning model for that stage is obtained. The parameters of the previous stage of reinforcement learning are initialized for the next stage. Steps S3 to S5 are repeated to complete the training of all stages and obtain the final reinforcement learning model. S6: Validate the reinforcement learning model by inputting information about objects that the robotic arm has not been trained on or that the data glove has not grasped into the trained model to validate the results of the robotic arm when grasping. The task reward Depend on , and It consists of 3 parts. To provide a positive reward, the finger joints must remain within the threshold range for each joint angle. The reward is negative; if the capture is successful, the reward is... Positive rewards; ； ； ； Where C is a constant, This represents the angle of the i-th joint. Let be the threshold value for the i-th joint angle. This indicates the centroid coordinates of the target being captured. This represents the coordinates of the robotic arm.

2. The method for optimizing the grasping action of a five-finger dexterous robotic hand based on reinforcement learning according to claim 1, characterized in that, The status information This includes the coordinates of the data glove in space, the coordinates of the centroid of the grasped target, the angles of each finger joint, joint velocities and angular velocities, and the coordinates of the fingertips contacting the target object.

3. The method for optimizing the grasping action of a five-finger dexterous robotic hand based on reinforcement learning according to claim 1, characterized in that, The method for converting the trajectory of an operator wearing data gloves into expert information is as follows: The operator wears data gloves to grasp an object and obtains a series of action sequences. The angles of each joint are calculated through inverse kinematics, the position of the hand is calculated using forward kinematics, and the joint velocities and angular velocities are calculated using finite difference kinematics.

4. The method for optimizing the grasping action of a five-finger dexterous robotic hand based on reinforcement learning according to claim 1, characterized in that, The total reward obtained by the robotic arm when performing step t for: ； in These are the weights for style rewards and task rewards, respectively.

5. The method for optimizing the grasping action of a five-finger dexterous robotic hand based on reinforcement learning according to claim 4, characterized in that, Returns are defined as the long-term return, and are calculated as follows: ； Where γ is the discount factor.

Images