A robotic arm grasping strategy optimization method based on continuous reinforcement learning of sequential tasks
By using sequential task-based continuous reinforcement learning and model distillation techniques, the robotic arm's grasping strategy is optimized, solving the problem of poor grasping performance in multi-task environments and achieving more efficient and accurate grasping operations.
Patent Information
- Application Number
- CN202410831223.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-26
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2044-06-26
AI Technical Summary
Existing reinforcement learning methods mainly focus on single-task optimization, making it difficult to effectively handle a series of tasks in robotic arm grasping, especially in complex and ever-changing real-world environments, where traditional grasping strategies fail to achieve the expected results.
A temporal task-based continuous reinforcement learning approach is adopted. Through task decomposition and model distillation mechanisms, an Actor network, a Critic network, and an experience replay pool are constructed to train a deep reinforcement learning neural network model. The policies of different networks are integrated through policy distillation technology to optimize the grasping strategy of the robotic arm.
It significantly improves the performance and flexibility of the robotic arm's grasping action, shortens training time, and increases grasping accuracy and efficiency, enabling it to better handle complex grasping tasks.
Smart Images

Figure CN118578396B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of robots, and in particular relates to a method for optimizing a robot arm grasping strategy based on continuous reinforcement learning of sequential tasks. Background Art
[0002] As a highly flexible, precise and programmable automation equipment, the robot arm has been widely used in manufacturing, logistics, medical treatment, aerospace and other fields. The grasping operation of the robot arm is a key technology in the field of robotics, which involves many complex factors, such as the type, position, posture of the target object and the dynamic characteristics of the robot arm itself. Traditional grasping strategies are usually based on predefined rules or control strategies that have undergone a lot of manual parameter adjustments, which makes grasping operations face many challenges. For example, the setting of rules and the adjustment of parameters usually involve highly professional knowledge, and may not achieve the expected results when facing complex and changing actual environments.
[0003] With the continuous development of deep learning and reinforcement learning, using reinforcement learning to optimize operation strategies has proven to be a very effective method. Reinforcement learning is a machine learning method whose main goal is to enable intelligent agents to obtain the optimal action strategy through continuous trial and error and learning through interaction with the environment. In the robotic arm grasping task, the reinforcement learning algorithm can make full use of the observed information, such as the visual input of the camera, the state information of the robotic arm, and the previous grasping results, and gradually adjust the control strategy of the robotic arm, so that it can find the optimal grasping strategy through continuous exploration and trial and error.
[0004] However, most existing reinforcement learning methods focus on the optimization of a single task and cannot handle sequential tasks well. In practical applications, robotic arm grasping usually includes a series of tasks, such as first locating the target object, then performing the grasping operation and moving the object to another location. Summary of the invention
[0005] The technical problem to be solved by the present invention is to address the deficiencies of the above-mentioned prior art and to provide a method for optimizing the grasping strategy of a robotic arm based on continuous reinforcement learning of sequential tasks. By introducing task decomposition and model distillation mechanisms, the strategies of different networks are integrated and adjusted to obtain a more comprehensive and optimized grasping strategy. This method can effectively optimize the various subtasks in the grasping operation in a unified manner and perform effective strategy learning, thereby greatly improving the grasping performance and flexibility of the robotic arm, shortening the grasping training time of the sequential task reinforcement learning robotic arm, and improving the grasping accuracy.
[0006] In order to achieve the above technical objectives, the technical solution adopted by the present invention is:
[0007] A method for optimizing a robotic arm grasping strategy based on continuous reinforcement learning of sequential tasks, comprising:
[0008] Step 1: Collect the state vector and target position of the robot arm when the robot arm performs the reaching task of the sequence task;
[0009] Step 2: Use the actor network, critic network and experience replay pool connected in sequence to build a neural network model for deep reinforcement learning;
[0010] Step 3: training a neural network model of deep reinforcement learning according to the state vector and target position of the robot arm when the robot arm performs the reaching task of the sequence task, and obtaining a neural network model I of deep reinforcement learning;
[0011] Step 4: The robot arm executes the neural network model I. When it reaches the target position, it performs the grasping task training. After the grasping task training is completed, the neural network model II of deep reinforcement learning is obtained.
[0012] Step 5: Use neural network model I to distill neural network model II to obtain neural network model III, and obtain the optimized robotic arm grasping strategy;
[0013] Step 6: Evaluate the performance of the neural network model III and check its performance steps on different task scenarios to confirm whether the model III converges.
[0014] To optimize the above technical solutions, the specific measures taken also include:
[0015] The Actor network and Critic network described in step 2 above are both composed of four fully connected layers with sizes of 64, 256, 256, and 64 respectively.
[0016] The above step 3 includes the following sub-steps:
[0017] Step 31: During the robot grasping process, there is a dynamic target. The reset environment algorithm of the simulation environment is called to obtain the instant target g in motion. t , and obtain the corresponding robot arm status information a t , get the corresponding robot state vector s t ;
[0018] Step 32: Transform the robot state vector s t Input into the Actor network and obtain the action probability π(a t |s t ), the action is selected as the robot arm joint angle a through the action probability t =[θ1, θ2, θ3, θ4, θ5, θ6, θ7];
[0019] Step 33: The robotic arm performs joint actions a t =[θ1, θ2, θ3, θ4, θ5, θ6, θ7], get the robot state information s at the next moment t+1 , and obtain immediate reward r through environmental feedback t and the immediate goal at the next moment g t+1 , will (s t , a t , r t ,s t+1 , g t ) is stored as experience in the experience replay pool I;
[0020] Step 34: Extract 128 experiences from the experience replay pool I to update the neural network model of deep reinforcement learning;
[0021] Step 35, repeating steps 32 to 34 until the robot reaches the target position or the number of actions performed reaches the maximum number of times limited by the environment;
[0022] Step 36: Extract 128 experiences from the experience replay pool I to update the neural network model of deep reinforcement learning;
[0023] Step 37: Repeat steps 31 to 36 using the updated deep reinforcement learning neural network until the maximum number of training rounds is reached, thereby completing the training of the deep reinforcement learning neural network model I.
[0024] The above step 33 includes the following sub-steps:
[0025] Step 331: Execute the robot arm joint angle action a in step 32 t =[θ1, θ2, θ3, θ4, θ5, θ6, θ7], obtain the instant position of the gripper at the end of the robot arm and the target's instant location The distance d(t) between them is used to determine whether the target instant position has been reached. Calculate the reward r1 = -5*d(t):
[0026]
[0027] Step 332: Execute the robot arm joint angle action a of step 32 t =[θ1, θ2, θ3, θ4, θ5, θ6, θ7], obtain the instant position of the gripper at the end of the robot arm Determine whether it collides with the ground and the conveyor belt. If a collision occurs, stop calculating the reward r2 in the current round and calculate the total reward r t =r1+r2, then (s t , a t , r t,s t+1 , g t ) is stored as experience in the experience replay pool I, and the neural network model I of deep reinforcement learning is updated with sampled data, where
[0028]
[0029] The above step 34 includes the following sub-steps:
[0030] Step 341: Randomly sample 128 data (s) from the experience replay pool I t , a t , rt, s t+1 , g t ), where s t Indicates the current state of the robot arm, a t Represents the current action of the robot arm, and the immediate reward r for interaction with the environment t and immediate target t ;
[0031] Step 342: Calculate the TD target y(n):
[0032]
[0033] Where γ0 is the initial discount factor, t is the current time step, and T is the total number of time steps; r t+k is the reward at the kth time step, s t+n is the state at time step t+n, π target is the target strategy, Q target is the output of the target Critic network, γ t+k is the dynamic discount factor at time step t+k;
[0034] Step 343: Update the Actor network parameters, the loss L of the Actor network actor for:
[0035]
[0036] in, is the experience replay buffer, Q(s,π(s)) is the state-action value estimated by the Critic network, Indicates the experience pool The expectation of the state s sampled in;
[0037] Step 344, update the critic network parameters, the loss L of the critic network critic for:
[0038]
[0039] Where N is the number of sampled data, (s i , a i , r i , s′ i ) is each sampled data point, y i is the TD target calculated in step 342;
[0040] Step 345: soft update the target network, using the soft update strategy to update the parameters of the target Critic network:
[0041] θ target ←τθ current +(1-τ)θ target
[0042] θ target is the parameter of the target Critic network, θ current is the parameter of the current Critic network, and τ is the soft update coefficient.
[0043] The above step 4 includes the following sub-steps:
[0044] Step 41: Execute the deep reinforcement learning neural network model I trained in step 3, and start recording the current instant target g in motion when the robot reaches the target point t , and obtain the corresponding robot arm joint angle θ t , and the information of the gripper force sensor at the end of the robot arm f t , get the corresponding robot state vector s t ;
[0045] Step 42: transform the robot state vector s t Input the Actor network in the deep reinforcement learning network model II to obtain the action probability π(a t |s t ), the action is selected as the robot arm joint angle a through the action probability t =[θ1, θ2, θ3, θ4, θ5, θ6, θ7];
[0046] Step 43: The robotic arm performs joint action a t =[θ1, θ2, θ3, θ4, θ5, θ6, θ7], get the robot state information s at the next moment t+1 , and obtain immediate reward r through environmental feedback t And whether the immediate target g is captured t+1 , will (s t , a t , r t ,s t+1 , gt , f t ) is stored as experience in the experience replay pool II;
[0047] Step 44, extract 128 experiences from the experience replay pool II to update the neural network of deep reinforcement learning;
[0048] Step 45, repeating steps 42 to 44 until the robot arm successfully grabs the object or the number of actions performed reaches the maximum number limited by the environment;
[0049] Step 46, extract 128 experiences from the experience replay pool II to update the neural network of deep reinforcement learning;
[0050] Step 47: Repeat steps 41 to 46 using the updated deep reinforcement learning neural network until the maximum number of training rounds is reached, thereby completing the training of the deep reinforcement learning neural network model II.
[0051] The above step 5 includes the following sub-steps:
[0052] Step 51: Use the neural network model I to perform reasoning: Under the same robot state vector s t Under this condition, run model I and obtain its output results to obtain the probability distribution of joint angles as soft targets for reference and adjustment by model III during the learning process to generate more accurate joint angles as reaching actions;
[0053] Step 52: Use the neural network model II for reasoning: With the same input robot state vector s t Next, run model II and obtain its output results, which are used to select the joint angle with the maximum probability as the grasping action. The results are used as hard targets for model III to learn.
[0054] Step 53: Train the distillation model: Train Model III by matching the soft target of Model I and / or the hard target of Model II. During the training process, measure the gap between the output of Model III and the soft target / hard target, and then use the gradient descent algorithm to minimize the loss and update the parameters of Model III.
[0055] The process of training the distillation model in step 53 above is as follows:
[0056] Step 531: Train model III to match the soft target of model I. Use KL divergence to measure the difference between the output probability distribution P of model III and the soft target probability distribution Q of model I, so as to use the prediction ability of model I to guide the training of model III. KL divergence is:
[0057]
[0058] Where Q(i) is the probability of category i in the soft target, and P(i) is the probability output by model III;
[0059] Step 532: Model III is trained by matching the hard targets of Model II. When matching the hard targets, a cross entropy loss function is used to measure the difference between the output of Model III and the true label, so that Model III can learn more accurate prediction results. The cross entropy loss function is:
[0060]
[0061] Where y is the one-hot encoded vector of the true label, which represents the corresponding joint angle when the gripping task is successfully completed; is the category probability distribution output by model III. The probability distribution output by model III represents the predicted probability of different joint angles.
[0062] The present invention has the following beneficial effects:
[0063] The present invention decomposes the sequence task into two subtasks, namely the reaching task and the grasping task, and trains and combines the deep reinforcement learning network model I and model II respectively. In the robotic arm grasping task, model I is responsible for guiding the robotic arm to reach the target point and generating the optimal joint angle to control the movement of the robotic arm; model II adjusts the action to achieve the grasping of the object after the robotic arm reaches the target point.
[0064] The present invention optimizes network performance through phased training and experience playback, so that the robot arm can perform grasping tasks more intelligently. In particular, the present invention also introduces strategy distillation technology to combine the advantages of model I and model II to obtain a comprehensive combined strategy network III. During the strategy distillation process, model I and model II provide "soft targets" and "hard targets" respectively to guide the training of network III, so that it can integrate the strategies of the two and generate more accurate and efficient control actions. Through this process, the combined strategy network III not only inherits the navigation ability of model I and the gripping ability of model II, but also optimizes the overall performance, significantly improving the grasping success rate and work efficiency of the robot arm.
[0065] In order to further improve the training efficiency and model performance, the present invention introduces a multi-step TD target. By considering the rewards of multiple time steps, the target is made smoother and more stable, which can better reflect the long-term return and improve the training efficiency. The dynamic discount factor is used to dynamically adjust the discount factor according to the importance of the task, so that the model can handle the importance of different steps more flexibly and improve the overall performance. Compared with the traditional method, the present invention effectively optimizes the various subtasks in the grasping operation, so that the robot arm can perform complex grasping tasks more intelligently and efficiently. BRIEF DESCRIPTION OF THE DRAWINGS
[0066] Figure 1This is a flow chart of a method for optimizing a robot grasping strategy based on continuous reinforcement learning of sequential tasks according to the present invention;
[0067] Figure 2 A schematic diagram of a simulation environment for a reaching task according to the present invention;
[0068] Figure 3 Schematic diagram of the grasping task simulation environment of the present invention. DETAILED DESCRIPTION
[0069] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.
[0070] Although the steps in the present invention are arranged with numbers, they are not used to limit the order of the steps. Unless the order of the steps is clearly stated or the execution of a certain step requires other steps as a basis, the relative order of the steps can be adjusted. It is understood that the term "and / or" used in this article involves and covers any and all possible combinations of one or more of the associated listed items.
[0071] like Figure 1 As shown, the present invention is a method for optimizing the grasping strategy of a robotic arm based on continuous reinforcement learning of sequential tasks, obtains the state of the environment, and uses the collected joint angles and target position information of the robotic arm as input state information; constructs a deep reinforcement learning network model, inputs the joint angles and target positions of the robotic arm into the deep reinforcement learning network, and outputs the current optimal parameterized action to control the movement of the robotic arm to the target position and complete various subtasks. By regularly sampling data from experience replay and combining the principles of maximizing rewards and maximizing entropy to update the network until the maximum number of training rounds is reached, the training of the deep reinforcement learning network is completed, and the optimal parameterized actions for targets facing different motion states are obtained. By executing the optimal parameterized actions of the robotic arm, the robotic arm completes the arrival task and the dynamic target grasping task; then the strategy distillation method is used to distill the two subtask deep reinforcement learning network models into one network model, and the network model strategies of different tasks are integrated and adjusted to obtain a more comprehensive and optimized grasping strategy, which specifically includes:
[0072] Step 1: Collect the state vector and target position of the robot arm when the robot arm performs the reaching task of the sequence task;
[0073] Step 2: Use the actor network, critic network and experience replay pool connected in sequence to build a neural network model for deep reinforcement learning;
[0074] Step 3: training a neural network model of deep reinforcement learning according to the state vector and target position of the robot arm when the robot arm performs the reaching task of the sequence task, and obtaining a neural network model I of deep reinforcement learning;
[0075] Step 4: The robot arm executes the neural network model I. When it reaches the target position, it performs the grasping task training. After the grasping task training is completed, the neural network model II of deep reinforcement learning is obtained.
[0076] Step 5: Use neural network model I to distill neural network model II to obtain neural network model III, and obtain the optimized robotic arm grasping strategy;
[0077] Step 6: Evaluate the performance of the neural network model III and check its performance steps on different task scenarios to confirm whether the model III converges.
[0078] To optimize the above technical solutions, the specific measures taken also include:
[0079] Furthermore, the above step 1 collects the state vector and target position information of the robot arm when the robot arm reaches the task in the first stage of the sequence task.
[0080] The state vector and target position information of the robot arm when the robot arm performs the arrival task of the sequence task are collected, including the state vector of the robot arm That is, the robot arm status information a t and target g t The robot status information Among them, θ i is the current angle θ of the i-th joint at time t, and the target g t is the position coordinate of the instant target that the robot needs to reach at time t
[0081] Furthermore, the above step 2 constructs a neural network model of deep reinforcement learning, wherein the deep reinforcement learning network is connected in sequence by an Actor network, a Critic network, and an experience storage and playback module;
[0082] The Actor network and Critic network are both composed of four fully connected layers with sizes of 64, 256, 256, and 64 respectively.
[0083] Furthermore, the above step 3 inputs the state vector into the neural network model of deep reinforcement learning, obtains the value function array returned by the environment, that is, outputs the joint angle of the robotic arm, and performs corresponding actions according to the joint angle of the robotic arm and inputs them into the environment. The robotic arm interacts with the environment to obtain the next stage state, and updates the neural network of deep reinforcement learning until the maximum number of training rounds is reached. The training of the deep reinforcement learning network I is completed, and the robotic arm completes the task of reaching the target, including the following sub-steps:
[0084] Step 31: During the robot grasping process, there is a dynamic target. Call the reset environment method of the simulation environment to obtain the instant target g in motion t , and obtain the corresponding robot arm status information a t , get the corresponding robot state vector s t ;
[0085] Step 32: Transform the robot state vector s t Input into the Actor network and obtain the action probability π(a t |s t ), the action is selected as the robot arm joint angle a through the action probability t =[θ1, θ2, θ3, θ4, θ5, θ6, θ7].
[0086] Step 33: The robotic arm performs joint actions a t =[θ1, θ2, θ3, θ4, θ5, θ6, θ7], get the robot state information s at the next moment t+1 , and obtain immediate reward r through environmental feedback t and the immediate goal at the next moment g t+1 , will (s t , a t , r t ,s t+1 , g t ) is stored as experience in the experience storage pool I;
[0087] Step 34: Extract 128 experiences from the experience replay pool to update the deep reinforcement learning network model;
[0088] Step 35, repeating steps 32 to 34 until the robot reaches the target object or the number of actions performed reaches the maximum number of times limited by the environment;
[0089] Step 36: Extract 128 experiences from the experience replay pool to update the deep reinforcement learning network;
[0090] Step 37: Repeat steps 31 to 36 using the updated deep reinforcement learning network until the maximum number of training rounds is reached, thereby completing the training of the deep reinforcement learning network model I.
[0091] Furthermore, the above step 33 includes the following sub-steps:
[0092] Step 331: Execute the robot arm joint angle action a in step 32 t =[θ1, θ2, θ3, θ4, θ5, θ6, θ7], obtain the instant position of the gripper at the end of the robot arm and the target's instant location The distance d(t) between them determines whether the target instant position has been reached. Calculate the reward r1 = -5*d(t):
[0093]
[0094] Step 332: Execute the robot arm joint angle action a of step 32 t =[θ1, θ2, θ3, θ4, θ5, θ6, θ7], obtain the instant position of the gripper at the end of the robot arm Determine whether it collides with the ground and the conveyor belt. If a collision occurs, stop calculating the reward r2 in the current round and calculate the total reward r t =r1+r2, then (s t , a t , r t ,s t+1 , g t ) is stored as experience in the experience storage pool, sampling data, and updating the deep reinforcement learning network model I, where
[0095]
[0096] Furthermore, the above step 34 includes the following sub-steps:
[0097] Step 341: Randomly sample 128 data (s) from the experience playback buffer. t , a t , r t ,s t+1 , g t ), where s t Indicates the current state of the robot arm, a t Represents the current action of the robot arm, and the immediate reward r for interaction with the environment t and immediate target t .
[0098] Step 342: Calculate the TD target y(n):
[0099]
[0100] Where γ0 is the initial discount factor, t is the current time step, and T is the total number of time steps. t+k is the reward at the kth time step, s t+n is the state at time step t+n, π target is the target strategy, Q target is the output of the target Critic network, γ t+k is the dynamic discount factor at time step t+k;
[0101] Step 343: Update the Actor network parameters, the loss L of the Actor network actor It can be expressed as:
[0102]
[0103] in, is the experience replay buffer, Q(s,π(s)) is the state-action value estimated by the Critic network, Indicates the experience pool The expectation of the state s sampled in .
[0104] Step 344, update the critic network parameters, the loss L of the critic network critic It can be expressed as:
[0105]
[0106] Where N is the number of sampled data, (s i , a i , r i , s′ i ) is each sampled data point, y i is the TD target calculated in step 342 .
[0107] Step 345: soft update the target network, using the soft update strategy to update the parameters of the target Critic network, θ target is the parameter of the target Critic network, θ current is the parameter of the current Critic network, and τ is the soft update coefficient. It can be specifically expressed as:
[0108] θ target ←τθ current +(1-τ)θ target
[0109] Furthermore, in the above step 4, when the robot arm reaches the task training, a neural network model I of deep reinforcement learning is obtained. The robot arm executes the neural network model I. When it reaches the target position, the training of the second stage grasping task is started. The relevant training method of step 3 is repeated for training. After the grasping task training is completed, the neural network model II of deep reinforcement learning is obtained, which includes the following sub-steps:
[0110] Step 41: Execute the deep reinforcement learning network model I trained in step 3, and start recording the current instant target g in motion when the robot reaches the target point t , and obtain the corresponding robot arm joint angle θ t , and the information of the gripper force sensor at the end of the robot arm f t , get the corresponding robot state vector s t ;
[0111] Step 42: transform the robot state vector s t Input the Actor network in the deep reinforcement learning network model II to obtain the action probability π(a t |s t ), the action is selected as the robot arm joint angle a through the action probability t =[θ1, θ2, θ3, θ4, θ5, θ6, θ7].
[0112] Step 43: The robotic arm performs joint action a t =[θ1, θ2, θ3, θ4, θ5, θ6, θ7], get the robot state information s at the next moment t+1 , and obtain immediate reward r through environmental feedback t And whether the immediate target g is captured t+1 , will (s t , a t , r t ,s t+1 , g t , f t ) is stored as experience in the experience storage pool II.
[0113] Step 44, extract 128 experiences from the experience replay pool to update the deep reinforcement learning network model II;
[0114] Step 45, repeating steps 42 to 44 until the robot arm successfully grabs the object or the number of actions performed reaches the maximum number limited by the environment;
[0115] Step 46: Extract 128 experiences from the experience replay pool to update the deep reinforcement learning network;
[0116] Step 47: Repeat steps 41 to 46 using the updated deep reinforcement learning network until the maximum number of training rounds is reached, thereby completing the training of the deep reinforcement learning network model II.
[0117] Furthermore, in the above step 5, for the neural network model I obtained by the reaching task training and the neural network model II obtained by the grasping task training, the neural network model I is used to distill the neural network model II to obtain the neural network model III;
[0118] Step 5 includes the following sub-steps:
[0119] Step 51: Reasoning using model I: Under the same robot state vector s t Under this condition, run through model I and obtain its output results, which are the probability distribution of joint angles. These results will be used as "soft targets" for reference and adjustment by model III during the learning process to generate more accurate joint angles as reaching actions;
[0120] Step 52: Use Model II for inference: Also on the same input robot state vector s t Run Model II on the , and get its output results. These results are used to select the joint angle with the maximum probability as the grasping action, and the results will be used as "hard targets" for Model III to learn;
[0121] Step 53: Train the distillation model: Train model III by matching the soft target of model I and / or the hard target of model II. In this process, a loss function is needed to measure the gap between the output of model III and the soft target / hard target, and then a gradient descent algorithm is used to minimize the loss and update the parameters of model III.
[0122] When matching soft targets, the Kullback-Leibler divergence (KL divergence) is used to measure the gap between the output of Model III and the output of Model I. When matching hard targets, the cross entropy loss function is used to measure the difference between the output of Model III and the true label.
[0123] Furthermore, step 53 trains the distillation model as follows:
[0124] Step 531: Train model III to match the soft target of model I, using KL divergence to measure the difference between the probability distribution P of the output of model III and the probability distribution Q of the soft target of model I, where Q(i) is the probability of category i in the soft target, and P(i) is the probability of the output of model III. This is to use the predictive power of model I to guide the training of model III.
[0125]
[0126] Step 532: Train model III by matching the hard targets of model II. When matching the hard targets, a cross entropy loss function is used to measure the difference between the output of model III and the true label, so that model III can learn more accurate prediction results.
[0127]
[0128] Where y is the one-hot encoded vector of the true label, is the category probability distribution output by model III. The true label represents the corresponding joint angle when the gripping task is successfully completed, and the probability distribution output by model III represents the predicted probability of different joint angles. By comparing the predicted probability of each category with the true label, the difference between the probability distribution output by the model and the true label is calculated, and the differences of all categories are summed and negative to obtain the final loss value. This process guides model III to gradually adjust its output so that it generates more accurate joint angles, thereby improving the gripping success rate of the robot arm.
[0129] Step 6 evaluates the performance of the deep reinforcement learning neural network model III and checks its performance steps in different task scenarios to confirm whether the model III converges.
[0130] Example
[0131] This embodiment provides a simulation environment for a robotic arm grasping task that considers continuous reinforcement learning of sequential tasks, such as Figure 2 and Figure 3 As shown in the figure. The Gazebo environment simulates a working area with a spatial coordinate range of [-0.8, -1.0, 0] to [0.8, 1.0, 0.8], including the position coordinates of three objects: the robot arm, the target block, and the target area. In this environment, the task of the robot arm is to complete each subtask by selecting reasonable actions. Figure 2 The task is to reach the target object on the conveyor belt. Figure 3 The task is a gripping and placing task, and the ultimate goal is to grip the target object into the target area, that is, the box.
[0132] First, the robot task is decomposed into the reaching task and the gripping task. When training each subtask, the robot can choose the joint angle of each joint, and the continuous parameter value range is [-0.2, 0.2]. The model parameters of the deep reinforcement learning network are set as follows: the Actor network and the Critic network are composed of fully connected layers of size 64, 256, 256, and 64, and the learning rate of the Actor network is 10 -4 , the learning rate of the Critic network (i.e., the Q-value network) is 10 -4, the exploration factor ε of the Q-value network is 0.02, and the discount factor γ is 0.99. The experience replay pool size is set to 100,000, the batch size is 64, the maximum execution step of the robot is 5000, the maximum number of training rounds is 3000, and the optimizer uses the Adam optimizer.
[0133] After the training of the two subtasks, deep neural network models I and II were obtained. Subsequently, the outputs of models I and II were combined using distillation technology to form a comprehensive model III. In this process, the KL divergence was used to measure the difference between the output of model III and the output of model I, and the average KL divergence was 0.1. The output results of model I were used as a reference to further optimize the output strategy. For hard targets, cross entropy loss was used to measure the difference between the output of model III and the true label, and the average loss value was 0.3. Through multiple iterative training, the distillation model III achieved an 85% grasping success rate in the inference stage, and the average grasping accuracy was 0.85, proving its effectiveness in optimizing the robotic arm grasping strategy.
[0134] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.
[0135] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.
Claims
1. A robotic arm grasping strategy optimization method based on sequential task reinforcement learning, characterized in that: include Step 1: Collect the state vector and target position of the robot arm when the robot arm performs the reaching task of the sequence task; Step 2: Use the actor network, critic network and experience replay pool connected in sequence to build a neural network model for deep reinforcement learning; Step 3: Train the neural network model of deep reinforcement learning according to the state vector and target position of the robot arm when the robot arm performs the reaching task of the sequence task, and obtain the neural network model I of deep reinforcement learning; Step 4: The robot arm executes neural network model I. When it reaches the target position, it performs grasping task training. After the grasping task training is completed, the deep reinforcement learning neural network model II is obtained. Step 5: Use neural network model I to distill neural network model II to obtain neural network model III, and obtain the optimized robotic arm grasping strategy; Step 6: Evaluate the performance of the neural network model III and check its performance steps on different task scenarios to confirm whether the model III converges.
2. According to the method for optimizing the robotic arm grasping strategy based on sequential task reinforcement learning in claim 1, it is characterized in that: The Actor network and Critic network described in step 2 are both composed of four fully connected layers with sizes of 64, 256, 256, and 64 respectively.
3. According to the method for optimizing the robotic arm grasping strategy based on sequential task reinforcement learning in claim 1, it is characterized in that: The step 3 includes the following sub-steps: Step 31: During the robot grasping process, there is a dynamic target. Call the reset environment algorithm of the simulation environment to obtain the real-time target in motion , and obtain the corresponding robot arm status information , and get the corresponding robot arm state vector ; Step 32: Transform the robot state vector Input the Actor network to get the robot arm's current neural network model strategy The action probability of all actions , the action is selected as the robot arm joint angle through the action probability ; Step 33: Robotic Arm Performs Joint Actions , get the robot arm status information at the next moment , and obtain immediate rewards through environmental feedback and immediate goals ,Will Stored as experience in Experience Replay Pool I; Step 34, repeating steps 32 and 33 until the robot reaches the target position or the number of actions performed reaches the maximum number of times limited by the environment; Step 35, extract 128 experiences from the experience replay pool I to update the neural network model of deep reinforcement learning; Step 36: Repeat steps 31 to 35 using the updated deep reinforcement learning neural network model until the maximum number of training rounds is reached, thereby completing the training of the deep reinforcement learning neural network model I.
4. According to claim 3, a method for optimizing a robotic arm grasping strategy based on sequential task reinforcement learning is characterized in that: Step 33 includes the following sub-steps: Step 331: Execute the robot arm joint angle action of step 32 , get the instant position of the gripper at the end of the robot arm and the target's immediate location Distance , determine whether the target instant location has been reached , calculate the reward : ; Step 332: Execute the robot arm joint angle action of step 32 , get the instant position of the gripper at the end of the robot arm , determine whether it collides with the ground and the conveyor belt. If a collision occurs, stop calculating the reward in the current round , calculate the total reward , and then The experience is stored in the experience replay pool I, and the deep reinforcement learning neural network model I is updated with sampled data. 。 5. The method for optimizing the grasping strategy of a robotic arm based on sequential task reinforcement learning according to claim 3 is characterized in that: Step 35 includes the following sub-steps: Step 351: Randomly sample 128 data from the experience replay pool I ,in Indicates the current state of the robot arm. Indicates the current action of the robot arm, and provides instant rewards for interacting with the environment and immediate goals ; Step 352: Calculate TD target : ; ; in, is the initial discount factor, is the current time step, is the total number of time steps, For the The reward for each time step is is the time step The state of It is a target strategy. is the output of the target Critic network, is the time step Dynamic discount factor when Step 353: Update Actor network parameters, Actor network loss for: ; in, is the experience replay buffer, is the state-action value estimated by the Critic network, Indicates the experience pool The state of sampling expectations; Step 354: Update the Critic network parameters and the loss of the Critic network for: ; in, is the number of sampled data, is each sampled data point, is the TD target calculated in step 352; Step 355: soft update the target network, using the soft update strategy to update the parameters of the target Critic network: ; are the parameters of the target Critic network, is the parameter of the current Critic network, is the soft update coefficient.
6. The method for optimizing the grasping strategy of a robotic arm based on sequential task reinforcement learning according to claim 1, characterized in that: The step 4 includes the following sub-steps: Step 41: Execute the deep reinforcement learning neural network model I trained in step 3. When the robot reaches the target point, it starts to record the current instant target in motion. , and obtain the corresponding robot arm joint angle , and information about the gripper force sensor at the end of the robot arm , and get the corresponding robot arm state vector ; Step 42: Change the robot state vector Input the Actor network in the deep reinforcement learning network model II to obtain the neural network model strategy of the robot arm in the current deep reinforcement learning The action probability of all actions , the action is selected as the robot arm joint angle through the action probability ; Step 43: The robotic arm performs joint movements , get the robot arm status information at the next moment , and obtain immediate rewards through environmental feedback And whether the immediate target is captured ,Will Stored as experience in Experience Replay Pool II; Step 44, repeating steps 42 and 43 until the robot arm successfully grabs the object or the number of actions performed reaches the maximum number limited by the environment; Step 45, extract 128 experiences from the experience replay pool II to update the neural network of deep reinforcement learning; Step 46: Repeat steps 41 to 45 with the updated deep reinforcement learning neural network until the maximum number of training rounds is reached, thereby completing the training of the deep reinforcement learning neural network model II.
7. The method for optimizing the grasping strategy of a robotic arm based on sequential task reinforcement learning according to claim 1, characterized in that: The step 5 includes the following sub-steps: Step 51: Use neural network model I to perform reasoning: Under this condition, run model I and obtain its output results to obtain the probability distribution of joint angles as soft targets for reference and adjustment by model III during the learning process to generate more accurate joint angles as reaching actions; Step 52: Use the neural network model II for reasoning: Given the same input robot state vector Next, run model II and obtain its output results, which are used to select the joint angle with the maximum probability as the grasping action. The results are used as hard targets for model III to learn. Step 53: Train the distillation model: Train Model III by matching the soft target of Model I and / or the hard target of Model II. During the training process, measure the gap between the output of Model III and the soft target / hard target, and then use the gradient descent algorithm to minimize the loss and update the parameters of Model III.
8. The method for optimizing the grasping strategy of a robotic arm based on sequential task reinforcement learning according to claim 7, characterized in that: The process of training the distillation model in step 53 is as follows: Step 531: Train model III to match the soft target of model I. Use KL divergence to measure the difference between the output probability distribution P of model III and the soft target probability distribution Q of model I, so as to use the prediction ability of model I to guide the training of model III. KL divergence is: ; in It is a category in soft target The probability of is the probability of the output of model III; Step 532: Model III is trained by matching the hard targets of Model II. When matching the hard targets, a cross entropy loss function is used to measure the difference between the output of Model III and the true label, so that Model III can learn more accurate prediction results. The cross entropy loss function is: ; in is the one-hot encoded vector of the true label, which represents the corresponding joint angle when the gripping task is successfully completed; is the category probability distribution output by model III. The probability distribution output by model III represents the predicted probability of different joint angles.
