NAO robot grasping training method based on direct preference optimization
By mimicking the actions of human experts and constructing a preference dataset, and combining a policy network and a value network, the problems of high computational resource consumption and difficulty in designing reward functions in deep reinforcement learning are solved, enabling the NAO robot to achieve stable and efficient grasping in complex scenarios.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- LANZHOU UNIV
- Filing Date
- 2023-08-17
- Publication Date
- 2026-04-28
AI Technical Summary
Existing deep reinforcement learning methods consume a lot of computational resources and are difficult to design reward functions in robot object grasping, resulting in poor training performance, especially in complex scenarios.
By mimicking the actions of human experts, data is collected and a preference dataset is constructed. Combining policy networks and value networks, a proximal policy optimization algorithm and an implicit reward function are used to train the NAO robot's grasping strategy, avoiding the design of explicit reward functions. The strategy is then evaluated using an object detection neural network.
It achieves stable and efficient object grasping in different scenarios, reduces computing costs, improves the universality and stability of robot grasping, conforms to human operating habits, avoids extreme operations, and ensures safety.
Smart Images
Figure CN117103255B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of intelligent robot technology and relates to a robot object grasping training method based on direct preference optimization. Background Technology
[0002] With changing business needs and the development of robotics technology, the operational and grasping capabilities of robots are receiving increasing attention. Currently, the large-scale application of robot operational and grasping skills is mainly in simple industrial scenarios. Robots can only repeatedly pick up and manipulate objects in familiar environments, such as assembly and palletizing. The entire process is generally carried out in response to limited expected changes. Therefore, when robots face some new scenarios (different distances between them and objects, different shapes of objects to be grasped), the grasping effect is usually difficult to be satisfactory.
[0003] Compared to traditional control theory methods, deep reinforcement learning enables robots to learn more flexible and precise action strategies. Robots trained using deep reinforcement learning exhibit better versatility in grasping, meaning they can grasp any object in unfamiliar or complex scenes; and greater stability, meaning they are less sensitive to background, lighting, and table angles.
[0004] However, the process of deep reinforcement learning requires the agent to constantly interact with the environment and calculate gradients, which consumes a lot of hardware computing resources. In addition, the learning process involves modeling the reward function, which directly affects the policy and behavior learned by the agent and is a very important part of reinforcement learning. However, in the training of robot object grasping, only a few actions can obtain rewards, which makes the problem of sparse rewards more likely to occur, making the design of the reward function particularly difficult. Summary of the Invention
[0005] Purpose of the invention: To address the problems of excessive hardware computing resources and difficulty in designing reward functions in conventional deep reinforcement learning, this invention proposes a NAO robot object grasping training method based on direct preference optimization. This method collects robot motion trajectory data by mimicking human experts and using simple reinforcement learning, and constructs a human preference dataset. The model is then trained based on this dataset, achieving a relative balance between exploration and utilization. This avoids the difficulty of designing reward functions and fully incorporates relatively scientific human movement habits, ensuring safety and efficiency to a certain extent. Furthermore, it can train NAO robot grasping capabilities with good generalization and stability at a relatively low cost.
[0006] Technical solution:
[0007] 1. A training method for object grasping in NAO robots based on direct preference optimization, including the following:
[0008] (1) Collect demonstration data from human experts, and use Python to call the API interface of the NAO robot simulator to learn by imitation from human experts, thereby obtaining the state variables of each degree of freedom of the NAO robot. and the amount of motion applied to each degree of freedom The trajectory data is stored in a buffer pool. middle;
[0009] (2) Design parameters are Policy network And the parameters are value network ,in As a pre-trained policy network, the reward function is defined as follows: The policy network was trained using a pre-trained policy on the NAO robot simulator, and the proximal policy optimization (PPO) algorithm was used to optimize the policy network. parameters Value Network parameters Update the parameters and obtain the state variables from the robot's training process. and amount of movement The trajectory data is stored in a buffer pool. middle;
[0010] (3) The buffer pool is labeled by human annotators. The trajectory data is scored, and the expected scores from multiple annotators are taken to obtain the ranking of the good and bad actions corresponding to a state, as well as the ranking of the good and bad actions between every two actions, thus constructing an action preference dataset. ,in For better movements, Poor performance;
[0011] (4) Based on the preference dataset Design Reference Strategy Network And design a parameterized policy network. Combined with reference policy network for Construct the maximum likelihood objective The object grasping strategy model is trained on hardware computing resources and minimized through gradient descent. To obtain the optimal ;
[0012] (5) Define a YOLOv4 object detection neural network, train a model that can recognize target objects using real-world datasets, and apply the optimal strategy. The target object recognition model is deployed on a real NAO robot to evaluate the robot's grasping actions in the real world.
[0013] (6) Iterate through the training and evaluation processes until the NAO robot can successfully grasp the target object;
[0014] 2. The aforementioned collection of human expert demonstration data refers to importing the NAOqi module, using Python to call the API interface of the NAO robot simulator, and imitating and learning from human experts to obtain state variables. and amount of movement The trajectory data refers to the data collected on the degrees of freedom of each joint on the robot's arm when it grasps a target object; the state variables mentioned above... This mainly includes the coordinates of the target object. Coordinates of the robot's hand The bending angles of each joint of the robot arm The robot's hand position (open or closed). These four sets of data together form the state space. These state variables can be obtained by calling the API interface; the corresponding action variables Including the angular velocities of each joint of the robot arm Angular motion direction of each joint of the robot arm Robot's hand opening action Robot's hand closing action These four sets of data together constitute the motion volume space. These motion parameters can be obtained by calling the API interface. Each robot arm has 6 degrees of freedom: 3 degrees of freedom for the shoulder joint, 1 degree of freedom for the elbow joint, 1 degree of freedom for the wrist joint, and 1 degree of freedom for the hand.
[0015] 3. The reward function is as follows: Reward function Rewards include those derived from changes in the distance between the robot's hand and the target object. The reward obtained by the robot's hand touching the target object Rewards obtained by the robot's hand grasping ;;
[0016] 4. The aforementioned use of the PPO algorithm for policy networks parameters Value Network parameters The update process mainly includes the following steps:
[0017] Step 1: Interact with the policy network in the NAO robot simulator to obtain the state variables. Motion volume Reward Value Next state quantity ,Will Input the value network to obtain the network's estimated value. ,Will Inputting the policy network yields the network's estimated value. , quadruple Store the experience;
[0018] Step 2: Once the storage volume reaches a batch size, calculate the discount reward within the buffer. and the dominant function The formula for calculating discount rewards is as follows: ,in The discount rate is given; the advantage function is calculated as follows: ;
[0019] Step 3: Update the policy network parameters Its loss function is Gradient descent is applied to this loss function, and the value is updated simultaneously. ;
[0020] Step 4: Update the value network parameters Its loss function is Gradient descent is applied to this loss function, and the value is updated simultaneously. ;
[0021] Step 5: Repeat the above steps until the policy network can be trained to produce a policy that enables the robot to successfully grasp the target object.
[0022] 5. The aforementioned preference dataset Design Reference Strategy Network This refers to designing a policy network. Its satisfaction That is, to maximize the optimal state action pair The likelihood;
[0023] 6. The aforementioned design parameterization strategy network This refers to the fact that, according to the Bradley-Terry model, the probability of human preference can be expressed as... ,in It is a logical function, a linear transformation. It is a parameter used to prevent parameterized policy networks. With reference policy network If the deviation is too large, the preference probability formula can be used to... Construct the maximum likelihood objective ;
[0024] 7. The aforementioned [context missing] Perform gradient descent to minimize To obtain the optimal , refers to the parameter The relevant gradient can be written as ,in It is a real policy network and reference policy network Implicitly defined reward, minimized using gradient descent This will allow us to obtain the optimal solution. ;
[0025] 8. The YOLOv4 object detection neural network is defined as follows: a model trained using real-world datasets is generated that can identify target objects, and the optimal strategy is applied. Deploying the target object recognition model to a real NAO robot refers to building a YOLOV4 deep neural network, collecting images of target objects in the real world to construct a dataset, and improving generalization by placing the target objects in different backgrounds and clutter. The constructed neural network is trained with the constructed dataset, and the best model is saved. The target object recognition model and the trained object grasping strategy are then deployed to a real NAO robot for evaluation.
[0026] Beneficial effects: Compared with the prior art, the present invention has the following advantages:
[0027] (1) The present invention uses a NAO robot object grasping training method based on direct preference optimization, which has better universality and higher stability. It is not specific and is applicable to different scenarios and target objects.
[0028] (2) The present invention uses a NAO robot object grasping training method based on direct preference optimization, which does not require high computational cost and does not require the design of an explicit reward function. The strategy can be improved through an implicit reward function.
[0029] (3) The present invention uses a NAO robot object grasping training method based on direct preference optimization, which is more in line with human operating habits and will not cause extreme operating situations. It can ensure safety while being highly efficient. Attached Figure Description
[0030] Figure 1 This is a flowchart of the NAO robot object grasping training method based on direct preference optimization according to the present invention.
[0031] Figure 2 This is a schematic diagram of the robot arm's degrees of freedom in the NAO robot object grasping training method based on direct preference optimization according to the present invention.
[0032] Figure 3This is a model training diagram for the NAO robot object grasping training method based on direct preference optimization according to the present invention. Detailed Implementation
[0033] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. After reading the present invention, any modifications of the present invention in various equivalent forms by those skilled in the art will fall within the scope defined by the appended claims.
[0034] This invention relates to a training method for object grasping in NAO robots based on direct preference optimization. The research object is the NAO robot, and the method used is direct preference optimization. Figure 1 As shown, the entire process of NAO robot object grasping training based on direct preference optimization described in this invention consists of 5 steps:
[0035] In step S101, human expert demonstration data is collected, and behavioral cloning is used in the NAO robot simulator to imitate and learn the expert's grasping actions. Data on the degrees of freedom of each joint on the robot's arm when grasping the target object is collected. The state variables are... This mainly includes the coordinates of the target object. Coordinates of the robot's hand The bending angles of each joint of the robot arm The robot's hand position (open or closed). These four sets of data together form the state space. These state variables can be obtained by calling the API interface; the corresponding action variables Including the angular velocities of each joint of the robot arm Angular motion direction of each joint of the robot arm Robot's hand opening action Robot's hand closing action These four sets of data together constitute the motion volume space. These action quantities can be obtained by calling the API interface, and the obtained state quantities can be... and amount of movement Trajectory data is stored in a buffer pool. middle.
[0036] In step S102, the construction parameters are: Policy network And the parameters are value network Define the reward function This includes rewards derived from changes in the distance between the robot's hand and the target object. The reward obtained by the robot's hand touching the target object Rewards obtained by the robot's hand grasping ,in This indicates the distance between the robot's hand and the target object. It is a binary value (1 when the robot's hand sensor touches the target object, and 0 otherwise). Then it depends entirely on (The robot receives a reward only when it touches the target object and then grasps it.) The network is updated using the Proximity Policy Optimization (PPO) algorithm, resulting in a value network. parameters The update method is to adjust the loss function. Perform gradient descent, policy network parameters The update method is to adjust the loss function. Perform gradient descent and simultaneously obtain the state variables. and amount of movement Trajectory data is stored in a buffer pool. middle.
[0037] In step S103, human annotators annotate the buffer pool. The trajectory data is scored, and the expected scores from multiple annotators are taken to avoid subjectivity. This yields a ranking of the good and bad actions corresponding to a given state, as well as a ranking of the good and bad actions between any two actions, thus constructing a human preference dataset. ,in For better movements, This is a poor performance.
[0038] In step S104, based on the human preference dataset Design Reference Strategy Network This network can maximize the optimal state-action pair Likelihood, constructing parameterized policy network (Strategies that require optimization for practical applications) According to the Bradley-Terry model, the probability of human preference can be expressed as... ,in It is a logical function, a linear transformation. It is a parameter used to prevent parameterized policy networks. With reference policy network If the deviation is too large, use the preference probability formula and the reference policy network. It can be Construct the maximum likelihood objective , obtain parameters Relevant gradients ,in It is a real policy network and reference policy network Implicitly defined rewards, for Gradient descent is used to obtain the optimal strategy .
[0039] In step S105, a YOLOv4 object detection neural network is defined. Images of target objects are collected in the real world to construct a dataset. The generalization ability can be improved by placing the target objects in different backgrounds and clutter. The constructed neural network is trained with the constructed dataset, and the best-performing model is saved. The target object recognition model and the optimal strategy are then combined. Deployed on a real NAO robot for evaluation.
[0040] Figure 2 The diagram shows the degrees of freedom of a robot arm based on the NAO robot object grasping training method using direct preference optimization. The robot arm has a total of 6 degrees of freedom: 3 degrees of freedom for the shoulder joint, 1 degree of freedom for the elbow joint, 1 degree of freedom for the wrist joint, and 1 degree of freedom for the hand.
[0041] Figure 3 The image shows the model training diagram of the NAO robot object grasping training method based on direct preference optimization.
[0042] The pre-trained model in step S304 includes a policy network. and value network The network is updated using the Proximal Policy Optimization (PPO) algorithm.
[0043] The reference policy network in step S308 is Parameterized policy network By pre-trained policy network It was copied.
[0044] The objective function in step S309 is ,right Gradient descent is used to obtain the optimal strategy ,Will Applied to real NAO robots.
Claims
1. A training method for NAO robot object grasping based on direct preference optimization, characterized in that: Includes the following: (1) Collect demonstration data from human experts, and use Python to call the API interface of the NAO robot simulator to learn by imitation from human experts, thereby obtaining the state variables of each degree of freedom of the NAO robot. and the amount of motion applied to each degree of freedom The trajectory data is stored in a buffer pool. middle; (2) Design parameters are Policy network And the parameters are value network ,in As a pre-trained policy network, the reward function is defined as follows: The policy network was trained using a pre-trained policy network on the NAO robot simulator, and the Proximal Policy Optimization (PPO) algorithm was used to optimize the policy network. parameters Value Network parameters Update the parameters and obtain the state variables from the robot's training process. and amount of movement The second trajectory data is stored in a buffer pool. middle; The reward function Rewards include those derived from changes in the distance between the robot's hand and the target object. The reward obtained by the robot's hand touching the target object Rewards obtained by the robot's hand grasping ;in, This indicates the distance between the robot's hand and the target object; It is a binary value, which is 1 when the robot's hand touches the target object, and 0 otherwise; (3) The buffer pool was annotated by human annotators. Scoring is performed on all trajectory data, and the expected scores from multiple annotators are taken to obtain the ranking of the good and bad actions corresponding to a state variable, as well as the ranking of the good and bad actions between every two actions, thus constructing an action preference dataset. in, For better movements, Poor performance; (4) Based on the preference dataset Design Reference Strategy Network And construct a parameterized policy network. Combined with reference policy network for Construct the maximum likelihood objective The object grasping strategy model is trained on hardware computing resources and minimized through gradient descent. To obtain the optimal policy network ; (5) Define a YOLOv4 object detection neural network, train an object recognition model using a real-world dataset, and apply the optimal policy network. The target object recognition model is deployed on a real NAO robot to evaluate the robot's grasping actions in the real world. (6) Iterate the training and evaluation processes until the NAO robot can successfully grasp the target object.
2. The NAO robot object grasping training method based on direct preference optimization according to claim 1, characterized in that, The collection of human expert demonstration data refers to importing the NAOqi module, using Python to call the API interface of the NAO robot simulator, and imitating and learning from human experts to obtain state variables. and amount of movement The trajectory data refers to the data collected on the degrees of freedom of each joint on the robot's arm when it grasps a target object; the state variables mentioned above... Including the coordinates of the target object Coordinates of the robot's hand Bending angles of each joint of the robot arm The robot's hand status Among them, the robot's hand state is either open or closed. These four sets of data together constitute the state variable space. State variables The action quantity is obtained by calling the API interface; Including the angular velocities of each joint of the robot arm Angular motion direction of each joint of the robot arm Robot's hand opening action Robot's hand closing action These four sets of data together constitute the motion volume space. Motion volume This was obtained by calling the API interface; the robot's single arm has a total of 6 degrees of freedom, namely 3 degrees of freedom for the shoulder joint, 1 degree of freedom for the elbow joint, 1 degree of freedom for the wrist joint, and 1 degree of freedom for the hand.
3. The NAO robot object grasping training method based on direct preference optimization according to claim 1, characterized in that, The aforementioned PPO algorithm for optimizing policy networks using proximal policies. parameters Value Network parameters To perform an update, the following steps are required: Step 1: In the NAO robot simulator, interact with the policy network. Get state variables through interaction Motion volume Reward Value Next state quantity ,Will Input Value Network Obtain the network's estimated value ,Will Input Policy Network Obtain the network's estimated value , quadruple Store the experience; Step 2: Once the storage volume reaches a certain batch size, calculate the discount reward in the buffer. and the dominant function The formula for calculating discount rewards is as follows: ,in The discount rate is given; the advantage function is calculated as follows: ; Step 3: Update the policy network parameters Its loss function is Gradient descent is applied to this loss function, and the value is updated simultaneously. ; Step 4: Update the value network parameters Its loss function is Gradient descent is applied to this loss function, and the value is updated simultaneously. ; Step 5: Repeat the above steps until the policy network is established. It is possible to train a strategy that enables the robot to successfully grasp the target object.
4. The NAO robot object grasping training method based on direct preference optimization according to claim 1, characterized in that, The aforementioned preference dataset Design Reference Strategy Network This refers to designing a policy network. Its satisfaction That is, to maximize the optimal state action pair The likelihood.
5. The NAO robot object grasping training method based on direct preference optimization according to claim 1, characterized in that, The aforementioned Perform gradient descent to minimize To obtain the optimal policy network , refers to the parameter The relevant gradient is written as in It is composed of the optimal policy network and reference policy network Implicitly defined reward, minimized using gradient descent This yields the optimal policy network. .
6. The NAO robot object grasping training method based on direct preference optimization according to claim 1, characterized in that, The aforementioned definition of a YOLOv4 object detection neural network uses real-world datasets to train an object recognition model, and then applies the optimal policy network. Deploying a target object recognition model onto a real NAO robot to evaluate the robot's grasping actions in the real world involves building a YOLOv4 object detection neural network, collecting images of target objects in the real world to construct a dataset, placing the target objects in different backgrounds to improve generalization, training the YOLOv4 object detection neural network with the constructed dataset, saving the best-performing model, and implementing the optimal policy network. The target object recognition model was deployed on a real NAO robot for evaluation.
Citation Information
Patent Citations
Stainless steel weld joint ultrasonic defect detection method based on deep and shallow feature fusion
CN112232400A
Light-emitting device, light-emitting apparatus, display device, electronic apparatus, and lighting device
CN113811589A