Robot grabbing system based on PPO algorithm and meta learning and control method thereof
By combining the PPO algorithm and Meta-Learning (MAML), a robot grasping system was constructed, which solved the problems of flexibility and adaptability of traditional robot grasping systems in complex environments. It achieved efficient and stable grasping performance and generalization ability, and reduced training costs and time.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-31
- Publication Date
- 2026-04-07
AI Technical Summary
Traditional robotic grasping systems suffer from poor flexibility and adaptability, low learning efficiency, and insufficient generalization ability when facing complex and ever-changing environments. Furthermore, the existing DDPG algorithm is unstable during training, sensitive to hyperparameters, and prone to getting stuck in local optima, making it difficult to achieve efficient grasping.
By combining the PPO algorithm and meta-learning (MAML), a robot grasping system is constructed. The meta-learning module learns task-independent initial parameters, and combined with the stable policy update mechanism of PPO, it can quickly adapt to new tasks, improve generalization ability and learning efficiency.
It achieves efficient grasping of robots in complex environments, with strong generalization ability, high learning efficiency, and stable grasping performance. It reduces training costs and time, and avoids model oscillation and local optima problems in new tasks.
Smart Images
Figure CN121798602A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The application belongs to the technical field of robot control, and particularly relates to a robot grasping system based on a PPO algorithm and meta-learning and a control method thereof. BACKGROUND
[0002] Through the robot grasping system, the robot can perceive environmental information in real time and perform complex grasping tasks, realizing automatic operation and improving production efficiency and safety. In traditional robot grasping systems, robots usually need to rely on precise modeling and pre-set control strategies to complete tasks, which requires a large number of sensors and complex programming, resulting in poor flexibility and adaptability of the system. In addition, the traditional method often fails to achieve ideal grasping effect when facing complex and variable environments and objects of different shapes and materials, which seriously limits the application of robots in diversified scenarios.
[0003] To solve this problem, in order to improve the grasping efficiency and adaptability of the robot and enable it to complete diversified tasks in complex environments, the robot grasping system needs to introduce a more intelligent control strategy without additional hardware upgrades. This intelligent control method can further improve the flexibility of the robot and maximize the demand for industrial automation and intelligent manufacturing. However, the existing robot grasping system has the problems of low learning efficiency and insufficient generalization ability when introducing an intelligent control strategy.
[0004] The advantage of deep reinforcement learning (DRL) lies in its excellent decision-making ability. It is similar to the biological learning mechanism, and the agent learns through repeated interaction with the environment, and learns the best decision-making strategy by maximizing the cumulative reward. However, the traditional DRL algorithm (such as PPO) usually needs to start learning from scratch when facing new tasks, which limits its adaptability and learning efficiency in dynamic environments.
[0005] DDPG algorithm combines deterministic policy gradient (DPG) and deep learning technology, and is suitable for reinforcement learning problems in continuous action space. It outputs deterministic actions through the Actor network and evaluates the value of actions through the Critic network. Although the DDPG algorithm performs well in solving reinforcement learning problems in continuous action space, it still has some significant shortcomings in practical applications, which may limit its performance and efficiency in robot grasping tasks:
[0006] DDPG is prone to instability during training, which is mainly due to its deterministic policy and the updating mechanism of the target network. The deterministic policy makes it difficult for DDPG to effectively handle randomness when facing complex environments, leading to oscillation during training. The updating frequency and method of the target network also affect the stability of training. If the updating is too fast, it may cause the target Q value to change too quickly, making it difficult for the training process to converge. In addition, DDPG is prone to gradient explosion or disappearance when dealing with high-variance tasks, further exacerbating the instability of training. In contrast, PPO significantly improves the stability of training by limiting the amplitude of policy updates and using truncated probability ratios, reducing the difficulty of debugging.
[0007] DDPG algorithm is extremely sensitive to the selection of hyperparameters, which is particularly evident in robot grasping tasks. For example, a slight change in learning rate can cause dramatic fluctuations in the training process, or even divergence. The choice of discount factor is also crucial. A too high discount factor may cause the algorithm to focus too much on short-term rewards and ignore long-term goals, which is extremely disadvantageous for grasping tasks that require multi-step planning. In addition, the strength of exploration noise directly affects the exploration ability of the algorithm. If not properly set, it may lead to insufficient or excessive exploration, thereby affecting the optimization of the policy. This high sensitivity to hyperparameters makes DDPG require a large number of experiments to adjust parameters in practical applications, increasing the cost and time of debugging.
[0008] DDPG algorithm is prone to local optimum, DDPG algorithm uses deterministic policy, that is, in a given state, the policy network directly outputs a deterministic action, rather than a probability distribution. This deterministic policy may lead the algorithm to fall into local optimum in some cases. The algorithm converges to a suboptimal solution, resulting in lower than expected success rate of grasping. For example, in actual grasping tasks, the robot may fail to successfully grasp the target object or have low grasping efficiency. SUMMARY
[0009] To solve the above problems, the present application discloses a robot grasping system combining PPO (Proximal Policy Optimization) algorithm and MAML (Model-Agnostic Meta-Learning) and its control method, aiming to improve the grasping efficiency and adaptability of robots in complex environments. A robot grasping control method combining PPO algorithm and meta-learning, the specific steps are as follows:
[0010] Step 1: Environment modeling:
[0011] According to the information obtained by the sensor module, an environment model of the grasping task is constructed, including the position, shape, material and other characteristics of the object and the state information of the robot.
[0012] Define the state space and action space , where the state Including the characteristics and actions of robots and objects. This includes the robot's grasping motion.
[0013] Step 2. Policy Network Initialization:
[0014] Initialize policy network Used to determine the current environmental state Output robot grasping action The probability distribution, where These are the parameters of the policy network.
[0015] Step 3. Value Network Initialization:
[0016] Initialize the value network Used to estimate, under the current policy, the state from the current state The initial expected return, of which These are parameters of the value network.
[0017] Step 4. Initialize the meta-learning module:
[0018] Initialize the meta-learning module, including the meta-policy network. Heyuan Value Network , used to learn task-independent policies and value functions, where and These are parameters of the meta-learning module.
[0019] The goal of the meta-learning module is to learn an initial policy and value function, so that when faced with a new task, it can quickly adapt to the new task with a small number of gradient updates.
[0020] Step 5. Data Acquisition and Interaction:
[0021] Control the robot to interact with the environment and collect its state during the grasping process. ,action and rewards The data is stored in the experience replay cache.
[0022] The collected data includes state sequences. Action sequence and reward sequence .
[0023] Step 6. Strategy Optimization:
[0024] Data is sampled from the experience replay buffer, and the parameters of the policy network are updated using the optimization objective function of the PPO algorithm and the gradient descent method.
[0025] The objective function of the PPO algorithm is defined as:
[0026]
[0027] in, It is an advantage estimate. This is the truncation parameter, which is usually set to 0.2.
[0028] Step 7. Value Function Optimization:
[0029] At the same time, the parameters of the value network are updated using the sampled data, thereby improving the accuracy of the value function's estimation of expected returns.
[0030] The loss function of the value function is defined as:
[0031]
[0032] in, It is the target value, which is usually calculated using the GAE (Generalized Advantage Estimation) algorithm.
[0033] Step 8. After each task update, update the parameters of the policy network and value network. and As input to the meta-learning module, the meta-policy network is updated. Heyuan Value Network parameters and ;
[0034] The update formula for the meta-learning module is:
[0035]
[0036]
[0037] in, It's the learning rate. It's the number of tasks. It is the first Data for each task It is the loss function of the task.
[0038] Step 9. Strategy Update and Iteration:
[0039] Repeat the above process of data collection and interaction, strategy optimization, and value function optimization, continuously updating the policy network and value network until the policy converges and the optimal grasping strategy is obtained.
[0040] When faced with new crawling tasks, use a meta-policy network. Heyuan Value Network As the initial strategy and value function, it can quickly adapt to new tasks through a small number of gradient updates.
[0041] The beneficial effects of this invention are:
[0042] This invention integrates the PPO algorithm with Meta-Learning (MAML) to construct a robot grasping system that combines efficient learning capabilities with strong environmental adaptability. Its core advantages revolve around three dimensions: generalization ability, learning efficiency, and grasping performance, as detailed below:
[0043] 1. Significantly improved generalization ability, adapting to complex and ever-changing scenarios.
[0044] The core limitation of traditional robot grasping solutions is "task dependence," meaning that models trained for specific objects and environments experience a sharp decline in performance when faced with new objects or scenarios, requiring retraining to adapt. This invention fundamentally solves this problem through the collaborative design of PPO and MAML:
[0045] The Meta-Learning (MAML) module captures common patterns across different grasping tasks by learning "task-independent initial parameters," forming a generalized initial model. This initial model is not optimized for a specific task, but rather has the potential to "quickly adapt to new tasks"—when faced with new objects or environments, it only requires a small number of gradient updates, without needing to train from scratch, to adjust the network parameters to adapt to the characteristics of the new task.
[0046] The PPO algorithm ensures the stability of policy updates through a "truncated objective function," avoiding performance fluctuations caused by excessive parameter adjustments when adapting the model to new tasks. This ensures that the general initial model from meta-learning can smoothly transition to the optimal policy for a specific task. The combination of these two approaches enables the robot to not only handle familiar, trained scenarios but also efficiently adapt to unseen object types and environmental changes, with generalization capabilities covering diverse grasping needs.
[0047] 2. Learning efficiency is significantly optimized, and training costs are reduced.
[0048] Traditional deep reinforcement learning requires extensive interactive training from scratch when faced with new tasks, consuming significant time and computational resources, and is prone to getting trapped in local optima. The fusion solution of this invention achieves a dual improvement in learning efficiency through "meta-learning + PPO":
[0049] The meta-learning module learns generalized initial parameters in advance through multi-task training. When faced with a new task, the robot does not need to explore from scratch, but starts with the optimal initial parameters obtained from meta-learning. It only needs a small amount of interaction data to complete the policy fine-tuning, which greatly reduces the number of training steps and data requirements for new tasks, and improves training efficiency.
[0050] The advantage of the PPO algorithm lies in its stable optimization mechanism: by limiting the magnitude of policy updates, it avoids gradient explosion or vanishing during parameter updates, while utilizing the GAE algorithm to improve the estimation accuracy of the value function, making each training iteration more efficient. Meta-learning provides high-quality initial parameters, further reducing the optimization difficulty of PPO, enabling the model to converge to the optimal policy faster and shortening the overall training cycle.
[0051] 3. Superior grasping performance, balancing stability and accuracy.
[0052] This invention achieves a comprehensive improvement in crawling performance through algorithm fusion and reward mechanism design:
[0053] The PPO algorithm, through its probabilistic policy output, can effectively cope with the randomness in the environment. Compared with the deterministic policy of DDPG, it is easier to explore the globally optimal grasping action and reduce the risk of getting trapped in local optima. At the same time, its truncated optimization objective ensures the smoothness of policy updates, making the grasping action more coherent and stable, and reducing the risk of objects slipping due to sudden changes in action. Attached Figure Description
[0054] Figure 1 Strengthen the overall learning process;
[0055] Figure 2 .PPO algorithm flowchart;
[0056] Figure 3 Flowchart of this invention patent;
[0057] Figure 4 .MAML-PPO mean reward;
[0058] Figure 5 .ORIGINAL-PPO (Original PPO) Mean Reward. Detailed Implementation
[0059] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. It should be noted that the terms "front," "rear," "left," "right," "up," and "down" used in the following description refer to directions in the accompanying drawings, and the terms "inner" and "outer" refer to directions toward or away from the geometric center of a specific component, respectively.
[0060] like Figure 1 As shown, reinforcement learning involves two entities: the agent and the environment. At time t, the state of the environment is... The reward for reaching this state is The intelligent agent observed and Take appropriate action After the agent takes the corresponding action, the environmental state changes. The corresponding reward becomes .
[0061] like Figure 3 As shown in this embodiment, a robot grasping control method based on the PPO algorithm and meta-learning is presented.
[0062] The steps of this method are as follows:
[0063] Step 100: Construct a robot grasping system;
[0064] Step 101: Initialize the robot body, including the multi-degree-of-freedom robotic arm and the end effector.
[0065] Step 102: Initialize the sensor module, including vision sensors (such as cameras) and force sensors.
[0066] Step 103: Initialize the control unit, which includes a processor and memory, for running the PPO algorithm and MAML module.
[0067] Step 200: Initialize the PPO algorithm and MAML module;
[0068] Step 201: Initialize the policy network and value network ,in and These are network parameters.
[0069] Step 202: Initialize the MAML module, including the meta-policy network. Heyuan Value Network ,in and These are parameters of the MAML module.
[0070] Step 203: Set training-related hyperparameters such as learning rate, batch size, dropout rate, and number of iterations.
[0071] Step 300: Set the environmental status information for the robot's grasping task
[0072] Step 301: Obtain object information in the grasping scene through the sensor module, including the object's position, shape, material and other features.
[0073] Step 302: Obtain interaction information between the robot and the object through the sensor module, including the contact force detected by the force sensor.
[0074] Step 303: Construct the environment model and define the state space. and action space .
[0075] Step 400: Set the reward function
[0076] Step 401: Define a reward function to assign a corresponding reward value based on the success or failure of the grasping task. For example, a positive reward is given for successfully grasping an object, and a negative reward is given for failing to grasp it.
[0077] Step 402: Adjust the reward value based on the stability and accuracy during the crawling process to optimize the crawling strategy.
[0078] Step 500: Begin the interactive training process;
[0079] like Figure 2 As shown,
[0080] Step 501: Control the robot to interact with the environment and collect the status during the grasping process. ,action and rewards The data is stored in the experience replay cache.
[0081] Step 502: Sample data from the experience replay buffer, use the optimization objective function of the PPO algorithm, and update the parameters of the policy network using gradient descent.
[0082] The objective function of the PPO algorithm is defined as:
[0083]
[0084] in, It is an advantage estimate. This is the truncation parameter, which is usually set to 0.2.
[0085] Step 503: Simultaneously update the parameters of the value network using the sampled data to improve the accuracy of the value function's estimation of expected returns.
[0086] The loss function of the value function is defined as:
[0087]
[0088] in, It is the target value, which is usually calculated using the GAE (Generalized Advantage Estimation) algorithm.
[0089] Step 504: After each task update, update the parameters of the policy network and value network. and As input to the MAML module, update the meta-policy network. Heyuan Value Network parameters and .
[0090] The update formula for the MAML module is:
[0091]
[0092]
[0093] in, It's the learning rate. It's the number of tasks. It is the first Data for each task It is the loss function of the task.
[0094] Step 600: After training stabilizes, save the model parameters.
[0095] Step 601: Determine whether the training has converged based on the maximized cumulative reward curve. The method for calculating the maximized cumulative reward is as follows:
[0096]
[0097] in, It is a discount factor. In time step The reward It is the total number of time steps.
[0098] Step 602: If the curve remains stable, the M-DRL algorithm converges, and the trained network parameters are saved; otherwise, training continues until convergence.
[0099] Step 700: In an actual robot grasping environment, load the pre-trained model parameters, apply the model, and output the optimal grasping strategy.
[0100] Step 701: In the actual robot grasping environment, load the pre-trained PPO and MAML model parameters.
[0101] Step 702: Run the model and output the best grabbing action based on the current environment.
[0102] Step 703: Perform the grabbing action to complete the grabbing task.
[0103] Through the above steps, the present invention can effectively combine the PPO algorithm and MAML technology to improve the robot's grasping efficiency and adaptability in complex environments.
[0104] from Figure 4 , Figure 5 The comparison reveals a significant difference in reward curves between the MAML-PPO and original PPO reward functions in the first 1000 episodes. While the original PPO algorithm initially learns slightly faster than the MAML-PPO algorithm, showing a rapid upward trend, its reward values fluctuate considerably in the mid-to-late stages, causing the model to converge only after approximately 700 episodes. In contrast, the MAML-PPO algorithm achieves greater reward rewards with fewer epochs, maintains model stability after rapid convergence, demonstrates strong generalization ability, and exhibits improved convergence speed, thus accelerating the algorithm and resulting in superior overall performance compared to the original.
[0105] The technical means disclosed in this invention are not limited to those disclosed in the above embodiments, but also include technical solutions composed of any combination of the above technical features.
Claims
1. A robot grasping control method based on PPO algorithm and meta-learning, characterized in that: Specifically, the steps include the following: Step 100: Construct a robotic grasping system; Step 200: Initialize the PPO algorithm and MAML module; Step 300: Set the environmental status information for the robot's grasping task; Step 400: Set the reward function; Step 500: Begin the interactive training process; Step 600: After training stabilizes, save the model parameters; Step 700: In an actual robot grasping environment, load the pre-trained model parameters, apply the model, and output the optimal grasping strategy.
2. The robot grasping control method based on PPO algorithm and meta-learning according to claim 1, characterized in that: Step 1 specifically includes: Step 101: Initialize the robot body, including the multi-degree-of-freedom robotic arm and the end effector; Step 102: Initialize the sensor module, including the vision sensor and the force sensor; Step 103: Initialize the control unit, which includes a processor and memory, for running the PPO algorithm and MAML module.
3. The robot grasping control method based on PPO algorithm and meta-learning according to claim 1, characterized in that: Step 2 specifically includes: Step 201: Initialize the policy network and value network Initialize the policy network Used to determine the current environmental status Output robot grasping action The probability distribution of ; where These are the parameters of the policy network. These are parameters of the value network; Step 202: Initialize the MAML module, including the meta-policy network. Heyuan Value Network , used to learn task-independent policies and value functions, where and These are parameters of the MAML module; Step 203: Set training-related hyperparameters such as learning rate, batch size, dropout rate, and number of iterations.
4. The robot grasping control method based on PPO algorithm and meta-learning according to claim 1, characterized in that: Step 3 specifically includes: Step 301: Obtain object information in the grasping scene through the sensor module, including the object's position, shape, and material characteristics; Step 302: Obtain interaction information between the robot and the object through the sensor module, including the contact force detected by the force sensor; Step 303: Construct the environment model and define the state space. and action space ; where the state Including the characteristics and actions of robots and objects. This includes the robot's grasping motion.
5. The robot grasping control method based on PPO algorithm and meta-learning according to claim 1, characterized in that: Step 4 specifically includes: Step 401: Define a reward function to give corresponding reward values based on the success or failure of the capture task; Step 402: Adjust the reward value based on the stability and accuracy during the crawling process to optimize the crawling strategy.
6. The robot grasping control method based on PPO algorithm and meta-learning according to claim 1, characterized in that: Step 5 specifically includes: Step 501: Control the robot to interact with the environment and collect the status during the grasping process. ,action and rewards The data is stored in the experience replay cache; Step 502: Sample data from the experience replay buffer, and update the parameters of the policy network using gradient descent with the optimization objective function of the PPO algorithm; the objective function of the PPO algorithm is defined as: ;in, It is an advantage estimate. This is the truncation parameter, which is usually set to 0.2; Step 503: Simultaneously update the parameters of the value network using the sampled data to improve the accuracy of the value function's estimation of expected returns; the loss function of the value function is defined as: ;in, It is the target value, usually calculated using the GAE (Generalized Advantage Estimation) algorithm; Step 504: After each task update, update the parameters of the policy network and value network. and As input to the MAML module, update the meta-policy network. Heyuan Value Network parameters and ; The update formula for the MAML module is: ； ； in, It's the learning rate. It's the number of tasks. It is the first Data for each task It is the loss function of the task.
7. The robot grasping control method based on PPO algorithm and meta-learning according to claim 1, characterized in that: Step 6 specifically includes: Step 601: Determine whether the training has converged based on the maximum cumulative reward curve; the maximum cumulative reward is calculated as follows: ； in, It is a discount factor. In time step The reward It is the total number of time steps; Step 602: If the curve remains stable, the M-DRL algorithm converges, and the trained network parameters are saved; otherwise, training continues until convergence.
8. The robot grasping control method based on PPO algorithm and meta-learning according to claim 1, characterized in that: Step 700 specifically includes: Step 701: In the actual robot grasping environment, load the pre-trained PPO and MAML model parameters; Step 702: Run the model and output the best grabbing action based on the current environment. Step 703: Perform the grabbing action to complete the grabbing task.
9. A robot grasping control system based on the PPO algorithm and meta-learning, characterized in that: The system includes an environment modeling module, a policy network and value network initialization module, an initialization meta-learning module, a data acquisition and interaction strategy optimization module, a value function optimization module, and a policy update and iteration module. The environment modeling module constructs an environment model for the grasping task based on information acquired by the sensor module, including the position, shape, and material characteristics of objects, as well as the robot's state information. The policy network and value network initialization module initializes the policy network to output the probability distribution of the robot's grasping actions based on the current environment state. The value network initialization module initializes the value network to estimate the expected reward starting from the current state under the current policy. The goal of initializing the meta-learning module is to learn an initial policy and value function, so that when faced with a new task, it can quickly adapt to the new task with a small number of gradient updates. Data acquisition and interaction module: controls the robot body to interact with the environment, collects status, action and reward data during the grasping process, and stores it in the experience playback cache; Policy optimization module: Samples data from the experience replay cache, uses the optimization objective function of the PPO algorithm, and updates the parameters of the policy network using gradient descent; Value function optimization module: Simultaneously updates the parameters of the value network using the sampled data, improving the accuracy of the value function's estimation of expected returns; The strategy update and iteration module repeats the above process of data collection and interaction, strategy optimization and value function optimization, continuously updating the policy network and value network until the strategy converges and the optimal grasping strategy is obtained.