A robotic arm obstacle avoidance and grasping method based on continuous reinforcement learning

Abstract

This invention relates to a robotic arm obstacle avoidance and grasping method based on continuous reinforcement learning, comprising: acquiring and executing a first-stage task; executing a second-stage task when the reward obtained within a training cycle reaches a threshold and the difference in rewards obtained within each training cycle is within the threshold; acquiring and executing the second-stage task when the reward obtained within a training cycle reaches a threshold and the difference in rewards obtained within each training cycle is within the threshold; acquiring and executing the third-stage task when the reward obtained within a training cycle reaches a threshold and the difference in rewards obtained within each training cycle is within the threshold, and training is completed; wherein the third-stage obstacle is randomly generated. This invention, targeting actual industrial environments, combines grasping and obstacle avoidance settings with a more effective state representation and reward design, thereby improving the robot's learning performance on this task.

Description

Technical Field

[0001] This invention relates to the field of intelligent learning technology, specifically to a robotic arm obstacle avoidance and grasping method based on continuous reinforcement learning. Background Technology

[0002] Currently, robotic arms can be applied to various tasks, such as assembly, picking and placing, and food cutting.

[0003] A robotic arm should grasp a target object and place it at a target location to complete a pick-and-place task. The entire process includes grasping the object and motion planning to reach the target point. Current research focuses on how to grasp objects of different shapes or pick objects from clutter. However, despite improvements in gripping ability, completing pick-and-place tasks in real-world industrial environments requires consideration of obstacles, such as boxes placed in the work area or near the operator. Therefore, another part of the work focuses on robotic arms with obstacle avoidance problems. Some research uses Regression-Enhancing Methods (RL) to solve the obstacle avoidance problem for robotic arms: combining the Artificial Potential Field Method (APFM) with RL methods solves the obstacle avoidance problem for dynamic obstacles. However, despite the significant achievements of RL in grasping and obstacle avoidance, how to combine these two tasks to complete a pick-and-place task in an industrial environment remains an unsolved problem.

[0004] Specifically, for long-cycle combinatorial tasks, RL algorithms suffer from low training efficiency due to long observation periods or insufficient feature extraction capabilities of the network architecture. Researchers address these issues by making the algorithms use training samples more effectively or by constructing networks capable of efficiently extracting features. For some long-cycle tasks, most work relies on well-tuned shaping rewards, which not only facilitates goal delivery to the agent but also alleviates the exploration problem. However, tuning appropriate rewards without causing solution degradation is very difficult. One non-strategy RL algorithm uses demonstration trajectories to quickly guide challenging long-cycle motion tasks, such as various similar insertion tasks. Researchers have investigated replacing these difficult-to-tune shape reward functions with artificial demonstration tasks. Another example is a geometry-aware manipulative space controller for rich contact operations, combining the PPO algorithm to complete trajectory following, block pushing, and long-cycle door opening tasks. Similar research has designed novel experience replay mechanisms or effective reward shaping methods to address the training difficulties of long-cycle tasks. While some research has considered long-cycle tasks, work on solving long-cycle tasks in real-world industrial scenarios is still lacking. Furthermore, RL methods are still inefficient at training long-cycle tasks, and how to effectively train long-cycle tasks remains a challenge.

[0005] The long-cycle grasping and obstacle avoidance problem is defined as a Markov decision process (MDP): given any state at time t...

[0006] s t Under state space s, the agent (i.e., the robot) is based on the state space s. t ∈ A The action environment S and policy π are executed at ∈ A. (st) The agent calculates the reward based on the set reward function R. (St, at) Receive the corresponding reward r t The goal of mastering the problem is to find an optimal strategy π. ∗ To maximize the expected sum of discounted future rewards, that is, the γ-discounted sum of all future returns from t to ∞.

[0007] In this problem, RL is used to find the optimal policy π. ∗ To optimize the grasping process, reduce the probability of collisions, and maximize the success rate of task completion. Furthermore, training efficiency needs to be optimized so that the robotic arm can effectively learn how to complete designed long-cycle tasks.

[0008] In summary, there is an urgent need for a robotic arm obstacle avoidance and grasping method based on continuous reinforcement learning that is tailored to real industrial environments, combines grasping and obstacle avoidance settings with tasks, has more effective state representation and reward design, and improves the robot's learning performance. Summary of the Invention

[0009] The purpose of this invention is to provide a robotic arm obstacle avoidance and grasping method based on continuous reinforcement learning, which combines grasping and obstacle avoidance settings with the task in a real industrial environment, has more effective state representation and reward design, and improves the learning effect of the robot.

[0010] To achieve the above objectives, the technical solution adopted by this invention is: a robotic arm obstacle avoidance and grasping method based on continuous reinforcement learning, comprising the following steps:

[0011] (1) The training tasks are executed sequentially, and the training tasks include at least a first-stage training task, a second-stage training task and a third-stage training task with increasing difficulty.

[0012] (2) Obtaining an obstacle avoidance and grasping model, wherein the obstacle avoidance and grasping model is obtained by performing deep learning on the training task;

[0013] (3) Based on the acquired obstacle avoidance and grasping model, input the task to be executed to realize obstacle avoidance and grasping of the robotic arm.

[0014] Furthermore, the obstacle avoidance and grasping model includes:

[0015] Obtain the first-stage task training model, which is obtained by deep learning on the execution of the first-stage training task. When the first-stage training task is executed, the first-stage training task is completed when the reward obtained in the training cycle reaches the threshold and the difference between the rewards obtained in each training cycle is within the threshold.

[0016] Obtain the second-stage task training model, which is obtained by deep learning of the second-stage training task. The second-stage training task is executed by the first-stage task training model. When the second-stage training task is executed, the second-stage training task is completed when the reward obtained in the training cycle reaches the threshold and the difference in the reward obtained in each training cycle is within the threshold.

[0017] A third-stage task training model is obtained by deep learning of the third-stage training task. The third-stage training task is executed by the second-stage task training model. When the third-stage training task is executed, the third-stage training task is completed when the reward obtained in the training cycle reaches a threshold and the difference in rewards obtained in each training cycle is within the threshold.

[0018] Furthermore, the first stage training task is a target object picking task with first stage obstacles; and the position of the first stage obstacles is fixed.

[0019] The second stage training task is a target object picking task with second-stage obstacles. The second-stage obstacles are fixed in position and more numerous than the first-stage obstacles.

[0020] The third-stage training task is a target object picking task with third-stage obstacles. The positions of the third-stage obstacles are randomly generated, and the number of obstacles is the same as the number of obstacles in the second stage.

[0021] Furthermore, the training tasks include:

[0022] State acquisition, which is used to identify the relative position of the robotic arm with respect to the target object and obstacles, including acquiring the minimum distance between the robotic arm and the obstacles;

[0023] Motion control, the motion control being used to control the movement of a robotic arm, including converting the robotic arm movement into position control in a Cartesian coordinate system;

[0024] Reward acquisition, wherein the reward acquisition is the record of successfully completing a target action; the target action includes arriving, grasping, lifting, and hovering.

[0025] Further, the state acquisition includes: treating several connecting arms, end effectors, and obstacles on the robotic arm as line segments, dividing each line segment into several points, and representing the distance between the points on these connecting arms and the points on the obstacles as the distance between the robotic arm and the obstacles, and defining the minimum distance d between the robotic arm and the obstacles. min As the first part of the state, the position of the two fingers of the robotic arm gripper relative to the y-axis in the end effector coordinate system is y. f As the second part of the state, the position p of the target object in three-dimensional space o As the third part of the state, the position p of the end effector in three-dimensional space. e As the fourth part of the state:

[0026]

[0027] The range of each state is as follows:

[0028]

[0029] Where, d min1 and d min2 These represent the minimum distances between the robotic arm and the two obstacles, respectively; y left and y right These represent the positions of the left and right fingers of the robotic arm gripper relative to the y-axis in the coordinate system of the end effector; x o y o , z o These represent the positions of the target object on the x, y, and z axes in the end effector coordinate system; x e y e , z e These represent the positions of the end effector on the x, y, and z axes in the world coordinate system, respectively.

[0030] Furthermore, the motion control includes: using operation space control (OSC) to convert the robot arm's motion into position control in a Cartesian coordinate system, where the robot arm's motion space 'a' is:

[0031]

[0032] Where Δx, Δy, and Δz represent the offsets of the end effector of the robotic arm on the x, y, and z axes in the Cartesian coordinate system, and Δg represents the opening and closing state of the gripper, which is the clamp on the end effector of the robotic arm. If Δg < 0, the clamp is closed; if Δg = 0, the clamp remains unchanged; if Δg > 0, the clamp is open.

[0033] Furthermore, the reward acquisition includes rewards in four stages: arrival, grabbing, lifting, and hovering.

[0034]

[0035] Where r reach , r grasp , r lift ,r hover These represent the rewards for arrival, grasping, lifting, and hovering, respectively; obstacle This indicates the penalty for a collision between the robotic arm and an obstacle; r time The time penalty incurred before the task is completed; d1 represents the Cartesian distance from the fixture to the target object; z set z object d1 represents the height of the set point and the height of the object to be reached, respectively; d2 represents the Cartesian distance from the target object to the target point, and r represents the height of the target object. hover 1 ,r hover2 α, β, γ, µ, λ, η represent the hovering rewards in the two cases respectively; α, β, γ, µ, λ, η represent the coefficients set in the reward formulas for arrival, gripping, lifting, and hovering respectively; κ represents the coefficient set for collision penalty; τ represents the coefficient set for time penalty.

[0036] Furthermore, there is one obstacle in the first stage; there are two obstacles in the second stage; and there are two obstacles in the third stage.

[0037] Furthermore, the deep learning algorithm is executed using the Soft Actor Critic algorithm.

[0038] The advantages of this invention are:

[0039] 1) The obstacle avoidance and grasping method of the robotic arm based on continuous reinforcement learning described in this invention can be applied to actual industrial environments, such as waste battery recycling, etc. By combining grasping and obstacle avoidance settings with environment and task, it has a more effective state representation and reward design, thereby improving the learning effect of the robot on the task.

[0040] 2) The robotic arm obstacle avoidance and grasping method based on continuous reinforcement learning described in this invention trains the training task in stages according to increasing difficulty and obtains the corresponding training model. Then, the primary training model executes a higher-level training task to obtain a corresponding higher-level training model, which makes the convergence speed of the picking task faster and can achieve higher training efficiency. Attached Figure Description

[0041] Appendix Figure 1 This is a flowchart of the obstacle avoidance and grasping method for robotic arms based on continuous reinforcement learning as described in this invention.

[0042] Appendix Figure 2This is a schematic diagram of the training environment for the first stage training task of the robotic arm obstacle avoidance and grasping method based on continuous reinforcement learning described in this invention.

[0043] Appendix Figure 3 This is a schematic diagram of the training environment for the second stage training task of the robotic arm obstacle avoidance and grasping method based on continuous reinforcement learning described in this invention.

[0044] Appendix Figure 4 This is a schematic diagram of the training environment for the third stage training task of the robotic arm obstacle avoidance and grasping method based on continuous reinforcement learning described in this invention.

[0045] Appendix Figure 5 This is a schematic diagram of the continuous learning mode of the robotic arm obstacle avoidance and grasping method based on continuous reinforcement learning described in this invention.

[0046] In the diagram, 01. Target object; 02. Obstacle; 03. Target bounding box. Detailed Implementation

[0047] The present invention will be further described below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. Furthermore, it should be understood that after reading the description of this invention, those skilled in the art can make various alterations or modifications to the invention, and these equivalent forms also fall within the scope defined by the appended claims.

[0048] Example 1

[0049] This invention provides a robotic arm obstacle avoidance and grasping method based on continuous reinforcement learning, as shown in the attached figure. Figure 1 As shown, attached Figure 1 This is a flowchart of the obstacle avoidance and grasping method for robotic arms based on continuous reinforcement learning according to the present invention, which includes at least the following steps: Steps S10-S20:

[0050] Step S10: Execute training tasks sequentially, wherein the training tasks include at least a first-stage training task, a second-stage training task, and a third-stage training task with progressively increasing difficulty.

[0051] Step S20: Obtain the obstacle avoidance and grasping model, which is obtained by performing deep learning on the training task.

[0052] It should be noted that the robotic arm obstacle avoidance and grasping method based on continuous reinforcement learning described in this invention trains the training task in stages according to the increasing difficulty and obtains the corresponding training model. Then, the primary training model executes a higher-level training task to obtain a corresponding higher-level training model, which makes the convergence speed of the picking task faster and can achieve higher training efficiency.

[0053] Specifically, obtaining the obstacle avoidance and grasping model includes steps S22-S26:

[0054] Step S22: Obtain the first-stage task training model. The first-stage task training model is obtained by deep learning on the execution of the first-stage training task. When the reward obtained in the training cycle reaches the threshold and the difference in the reward obtained in each training cycle is within the threshold, the first-stage training task is completed.

[0055] Please see the appendix Figure 2 , attached Figure 2 This is a schematic diagram of the training environment (robotic arm not shown) for the first stage training task of the robotic arm obstacle avoidance and grasping method based on continuous reinforcement learning described in this invention. The first stage training task is a target object picking task with first-stage obstacles; and the position of the first-stage obstacles is fixed.

[0056] Step S24: Obtain the second-stage task training model. The second-stage task training model is obtained by deep learning on the second-stage training task. The second-stage training task is executed by the first-stage task training model. When the second-stage training task is executed, the second-stage training task is completed when the reward obtained in the training cycle reaches the threshold and the difference in the reward obtained in each training cycle is within the threshold.

[0057] Please see the appendix Figure 3 , attached Figure 3 This is a schematic diagram of the training environment (robotic arm not shown) for the second-stage training task of the robotic arm obstacle avoidance and grasping method based on continuous reinforcement learning described in this invention. The second-stage training task is a target object picking task with second-stage obstacles. The second-stage obstacles have fixed positions and are more numerous than the first-stage obstacles; and the positions of the second-stage obstacles are fixed.

[0058] Step S26: Obtain the third-stage task training model. The third-stage task training model is obtained by deep learning on the third-stage training task. The third-stage training task is executed by the second-stage task training model. When the third-stage training task is executed, the third-stage training task is completed when the reward obtained in the training cycle reaches the threshold and the difference in the reward obtained in each training cycle is within the threshold.

[0059] Step S30: Based on the acquired model, implement obstacle avoidance and grasping by the robotic arm.

[0060] Please see the appendix Figure 4 , attached Figure 4This is a schematic diagram of the training environment (robotic arm not shown) for the third-stage training task of the robotic arm obstacle avoidance and grasping method based on continuous reinforcement learning described in this invention. The third-stage training task is a target object picking task with third-stage obstacles, the positions of which are randomly generated.

[0061] The obstacle avoidance and grasping method for robotic arms based on continuous reinforcement learning described in this invention sets an increasing training difficulty at each training task stage. After the initial training model is established, a more complex training difficulty is used to train the previously trained model to obtain the final complete obstacle avoidance and grasping model. This decomposes the training process and greatly improves training efficiency.

[0062] It should be noted that the deep learning strategy and execution method used in this embodiment are as follows:

[0063] A. SAC

[0064] Soft Actor Critic (SAC) is an algorithm that optimizes stochastic policies in a non-policy manner, combining the advantages of stochastic policy optimization with methods such as DDPG. SAC modifies the DRL problem by requiring not only learning a policy to maximize the cumulative reward, but also the maximum entropy of the policy's output for each action.

[0065] SAC learns a policy π θ and two Q functions Q φ1 Q φ2 SAC uses a neural network to approximate the policy and Q-function. The policy network parameters and Q-network parameters are updated using gradient descent. First, the algorithm samples data from the replay buffer R to compute the objective of the Q-function. The objective is computed as follows:

[0066]

[0067] Where a θ (s) This indicates that the action is derived from strategy π. θ (·|s) SAC uses a pruned double Q-squared function similar to TD3 to learn the value function, taking the minimum Q between two close values. The parameters of the Q-function φ1 and φ2 are updated as follows:

[0068] i=1,2

[0069] The policy parameter θ is updated as follows:

[0070]

[0071] B. Status, Action Representation, and Reward

[0072] 1) State representation

[0073] State `st` is defined as several parts. The robot arm part only considers links 6, 7, 8, and the end effector (links 6, 7, and 8 are components on the robot arm, all within the workspace). Furthermore, links 6, 7, 8, the end effector, and obstacles are considered as line segments, each segment divided into 10 points. The distance between these points on the robot arm components and the points on the obstacles represents the distance between the robot arm and the obstacles. The minimum distance `d` between the robot arm and the obstacles is defined as... min As the first part of the state, the position of the two fingers of the robotic arm gripper relative to the y-axis in the end effector coordinate system is y. f As the second part of the state, the position p of the target object in three-dimensional space o As the third part of the state, the position p of the end effector in three-dimensional space. e As the fourth part of the state:

[0074]

[0075] The range of each state is as follows:

[0076]

[0077] Where, d min1 and d min2 These represent the minimum distances between the robotic arm and the two obstacles, respectively; y left and y right These represent the positions of the left and right fingers of the robotic arm gripper relative to the y-axis in the coordinate system of the end effector; x o y o , z o These represent the positions of the target object on the x, y, and z axes in the end effector coordinate system; x e y e , z e These represent the positions of the end effector on the x, y, and z axes in the world coordinate system, respectively.

[0078] It should be noted that: preferably, the position of the two fingers on the robotic arm gripper relative to the y-axis in the end effector coordinate system is taken as the second part of the state; the position of the target object is taken as the third part of the state; and the position of the end effector is taken as the fourth part of the state.

[0079] 2) Action representation

[0080] The controller determines the high-level control type for a given robotic arm. Operational space control (OSC) is used to translate the robotic arm's motion into position control in a Cartesian coordinate system. The robotic arm's motion space is as follows:

[0081]

[0082] Δx, Δy, and Δz represent the offsets of the end effector along the x, y, and z axes, respectively. Δg represents the state of the gripper's opening and closing motion; a positive value indicates positive motion, and a negative value indicates negative motion. Here, Δx, Δy, and Δz are within the range of the input values ​​and have a proportional relationship of 0.05 with the actual output value. That is, when the x-input is 1, the output moves 0.05 meters along the positive x-axis, where positive and negative values ​​represent the positive and negative directions of motion. The gripper opening and closing motion Δg itself is discrete, including only the opening and closing actions. To apply the RL algorithm to a continuous motion space, the opening and closing actions are processed continuously. If Δg < 0, the gripper is closed; if Δg = 0, the gripper remains unchanged. If Δg > 0, the gripper is open.

[0083] 3) Reward System

[0084] In reinforcement learning, the design of rewards directly affects training effectiveness. In this embodiment, the grasping task is divided into four stages: arrival, grasping, lifting, and hovering. The rewards are set as follows:

[0085]

[0086] Where r reach , r grasp , r lift ,r hover 1 ,r hover2 These represent the rewards for arrival, grasping, lifting, and hovering, respectively; obstacle This indicates the penalty for a collision between the robotic arm and an obstacle; r time The time penalty incurred before the task is completed; d1 represents the Cartesian distance from the fixture to the target object; z set z object d1 represents the height of the set point and the height of the object to be reached, respectively; d2 represents the Cartesian distance from the target object to the target point; α, β, γ, µ, λ, and η represent the coefficients set in the reward formulas for reaching, grasping, lifting, and hovering, respectively; κ represents the coefficient set for the collision penalty; and τ represents the coefficient set for the time penalty. In this embodiment, α = 0.1, β = 0.35, γ = 0.5, µ = 0.75, κ = -5, λ = 15, η = 10, and τ = -0.01 are set.

[0087] At the start of the task, the robotic arm receives a reward for approaching the target object. When both grippers of the robotic arm are in contact with the target object, the robotic arm is considered to have grasped the object and receives a larger reward. grasp When grabbing reward r grasp When > 0, the arm receives a boost bonus r lift This encourages the arm to lift the object. The reward for lifting an object close to the target box is...

[0088] r hover It can be divided into the following two cases: When the lifted object is within the x and y range of the target box, it will receive a reward r at the x and y coordinates. hover1 In other cases, it will receive a relatively small reward r. hover2 Regarding obstacle avoidance, a penalty and reward system was implemented. In this training task, since only the three links of the robotic arm and the end effector were within the workspace, collisions with obstacles were only considered for these four parts of the robotic arm. When these parts collide with obstacles, the robot will be penalized for the collision. Additionally, when the robotic arm has not yet completed its task, i.e., the target object has not been placed in the target box, the robot will always be subject to a time penalty r. time The punishment.

[0089] 4) Continuous learning

[0090] Please see the appendix Figure 5 , attached Figure 5 This is a schematic diagram of the continuous learning mode of the robotic arm obstacle avoidance and grasping method based on continuous reinforcement learning described in this invention.

[0091] Continuous learning can provide auxiliary tasks, allowing the agent to start with simple tasks and gradually progress to more difficult ones. In this embodiment, three task environments of varying difficulty are set up: one with fixed-position obstacles, one with two fixed-position obstacles, and one with two randomly generated obstacles. For the current task, when the robot's reward in one epoch reaches the expected value and the training results stabilize, the training of the current task is considered complete, and the model is then moved to the next environment for training. Training continues until the robot completes the last task.

[0092] Example 2

[0093] Building specific task environments based on the Mujoco physics engine and Robosuite:

[0094] 1) Environment: A table measuring 0.25m long, 0.18m wide, and 0.8m high is placed in the environment. A loaf of bread, representing the target object 01, is randomly placed on the left half of the table, and a cylinder, representing the obstacle 02, is placed on the right side. A four-pane frame is placed on the right side of the table, one of which is the target frame 03. A virtual object is placed on the target frame 03, and the position of the virtual object is the position of the target point. In this environment, the robot's training task is to grasp and place the target object 01 into the target frame 03 without colliding with the obstacle 02. Each epoch and training cycle of the task contains 500 steps. In the first stage of the task environment, a cylindrical obstacle is placed in a fixed position on the right side of the table. In the second stage of the task environment, two cylindrical obstacles are placed in fixed positions on the right side of the table, and the rest is the same as the first stage task environment. In the third stage task environment, two obstacles are randomly generated on the right side of the table, and the rest is the same as the first and second stage task environments.

[0095] 2) Network structure: In the DRL algorithm used, the actor and critic networks use two convolutional layers and a flattening layer of size 256.

[0096] 3) Hyperparameters: The hyperparameter settings of the reinforcement learning algorithm are shown in Table 1. The best results can be achieved when the number of critics N = 5 and the number of atoms M = 25.

[0097] Table 1

[0098] Hyperparameter Value LEARNING RATE <![CDATA[1×10 -3 ]]> BATCH SIZE 256 DISOUNT (γ) 0.98 POLYAK-AVERAGING COEFFICIENT(τ) 0.95 REPLAY BUFFER SIZE <![CDATA[2×10 6 ]]> Numer of Critics 5 Number of Atoms M 25

[0099] 4) Training details: 4000 epochs were trained on a system running Ubuntu 16.04, an Intel Xeon Gold 5118 CPU, an NVIDIA Tesla NVLINK V100 SXM2 graphics card, and CUDA 10.

[0100] Training results:

[0101] 1) State validity:

[0102] Some studies consider the minimum distance between the robot's surface and the obstacle surface. However, this representation leads to a huge computational load and affects the time required to complete the entire training. To reduce computational load, some works use the distance between the joint and the obstacle to represent the relationship between the robot and the obstacle. This state representation is as follows:

[0103]

[0104] d joint6 d joint7 djoint8 j represents the robotic arm oint6 j oint7 j oint8 Distance to obstacles. Two different states, S1 and S2, were compared to demonstrate the effectiveness of training the task in the first-stage task environment.

[0105] The results show that the first state achieved better training results. The first state was used as the state representation in subsequent experiments.

[0106] 2) Data efficiency

[0107] The training results of the model using the CL method for transfer learning are compared with those of the model that started directly from scratch in the third-stage task environment. Model 1 refers to the model that underwent two transfer learning processes, from the first environment to the second environment and then from the second environment to the third environment. Model 2 is the model trained directly from scratch.

[0108] The results show a significant difference in efficiency between the model trained in the third environment after two transfers and the model trained directly from scratch. Model 1 achieves a training time of 1×10⁻⁶ steps. 6 Previously, it converged to very good results; however, model2 is still in a state of lacking rewards.

[0109] 3) Final performance

[0110] The training performance of the two methods was compared under the same training time. Here, Model1 is defined as starting training in the first-stage task environment, and after reaching a high reward (e.g., a cumulative reward exceeding 600 in one epoch), the model is transferred to the next environment for training, and so on. Model2 starts training directly in the third environment. The training duration is the number of steps Model1 takes from the first-stage task environment to the third-stage task environment.

[0111] The results show that in the early stages, the robot spends a certain amount of time training by starting with simple tasks and then moving on to difficult tasks; the convergence speed of the picking task is faster, and the robot can be trained better.

[0112] In addition, the training performance of the model using the CL method for transfer learning was compared with that of the model starting from 0 in a shorter period of time, i.e., the total step size per epoch was reduced from 500 to 300.

[0113] The results show that the transplanted model not only achieves higher training efficiency, but also allows the robotic arm to complete tasks faster and obtain higher rewards in a shorter period. It should be noted that because the number of steps per round is reduced from 500 to 300, the robot's reward will be lower than in the first three environments.

[0114] It should be noted that the obstacle avoidance and grasping method for robotic arms based on continuous reinforcement learning described in this invention is used to optimize the training efficiency of reinforcement learning for long-cycle tasks, significantly reducing the training time required by the robot itself. This enables the robot model to quickly adapt to the next more difficult task after multiple transfers, which also reduces the deployment cost of the task in actual industrial applications.

[0115] Simultaneously, a more efficient state representation is proposed for pick-based training tasks, providing a good starting point for training such long-cycle tasks. To address the difficulty of training reinforcement learning for such long-term tasks, a continuous learning method is adopted to optimize training efficiency, and the ported model exhibits better training performance and can adapt to new working environments more quickly.

[0116] The above description is only a preferred embodiment of the present invention. It should be noted that those skilled in the art can make several improvements and additions without departing from the principle of the present invention, and these improvements and additions should also be considered within the scope of protection of the present invention.

Claims

1. A robotic arm obstacle avoidance and grasping method based on continuous reinforcement learning, characterized in that, Includes the following steps: (1) The training tasks are executed sequentially, and the training tasks include at least a first-stage training task, a second-stage training task and a third-stage training task with increasing difficulty. (2) Obtaining an obstacle avoidance and grasping model, wherein the obstacle avoidance and grasping model is obtained by performing deep learning on the training task; Obtaining the obstacle avoidance and grasping model includes: Obtain the first-stage task training model, which is obtained by deep learning on the execution of the first-stage training task. When the first-stage training task is executed, the first-stage training task is completed when the reward obtained in the training cycle reaches the threshold and the difference between the rewards obtained in each training cycle is within the threshold. Obtain the second-stage task training model, which is obtained by deep learning of the second-stage training task. The second-stage training task is executed by the first-stage task training model. When the second-stage training task is executed, the second-stage training task is completed when the reward obtained in the training cycle reaches the threshold and the difference in the reward obtained in each training cycle is within the threshold. A third-stage task training model is obtained by deep learning of the third-stage training task. The third-stage training task is executed by the second-stage task training model. When the third-stage training task is executed, the third-stage training task is completed when the reward obtained in the training cycle reaches a threshold and the difference in the reward obtained in each training cycle is within the threshold. (3) Based on the acquired obstacle avoidance and grasping model, input the task to be executed to realize obstacle avoidance and grasping of the robotic arm.

2. The obstacle avoidance and grasping method for robotic arms based on continuous reinforcement learning according to claim 1, characterized in that, The first stage training task is a target object pickup task with first stage obstacles; and the positions of the first stage obstacles are fixed. The second stage training task is a target object picking task with second-stage obstacles. The second-stage obstacles are fixed in position and more numerous than the first-stage obstacles. The third-stage training task is a target object picking task with third-stage obstacles. The positions of the third-stage obstacles are randomly generated, and the number of obstacles is the same as the number of obstacles in the second stage.

3. The obstacle avoidance and grasping method for robotic arms based on continuous reinforcement learning according to claim 1, characterized in that, The training tasks include: State acquisition, which is used to identify the relative position of the robotic arm with respect to the target object and obstacles, including acquiring the minimum distance between the robotic arm and the obstacles; Motion control, the motion control being used to control the movement of a robotic arm, including converting the robotic arm movement into position control in a Cartesian coordinate system; Reward acquisition, wherein the reward acquisition is the record of successfully completing a target action; the target action includes arriving, grasping, lifting, and hovering.

4. The obstacle avoidance and grasping method for robotic arms based on continuous reinforcement learning according to claim 3, characterized in that, The state acquisition includes: treating several connecting arms, end effectors, and obstacles on the robotic arm as line segments; dividing each line segment into several points; representing the distance between points on these connecting arms and points on the obstacles as the distance between the robotic arm and the obstacles; and defining the minimum distance d between the robotic arm and the obstacles. min As the first part of the state, the position of the two fingers of the robotic arm gripper relative to the y-axis in the end effector coordinate system is y. f As the second part of the state, the position p of the target object in three-dimensional space o As the third part of the state, the position p of the end effector in three-dimensional space. e As the fourth part of the state: The range of each state is as follows: Where, d min1 and d min2 These represent the minimum distances between the robotic arm and the two obstacles, respectively; y left and y right These represent the positions of the left and right fingers of the robotic arm gripper relative to the y-axis in the coordinate system of the end effector; x o y o , z o These represent the positions of the target object on the x, y, and z axes in the end effector coordinate system; x e y e , z e These represent the positions of the end effector on the x, y, and z axes in the world coordinate system, respectively.

5. The obstacle avoidance and grasping method for robotic arms based on continuous reinforcement learning according to claim 3, characterized in that, The motion control includes: using operation space control to convert the robotic arm's motion into position control in a Cartesian coordinate system, where the robotic arm's operation space 'a' is: Where Δx, Δy, and Δz represent the offsets of the end effector of the robotic arm on the x, y, and z axes in the Cartesian coordinate system, and Δg represents the opening and closing state of the gripper, which is the clamp on the end effector of the robotic arm. If Δg < 0, the clamp is closed; if Δg = 0, the clamp remains unchanged; if Δg > 0, the clamp is open.

6. The obstacle avoidance and grasping method for robotic arms based on continuous reinforcement learning according to claim 3, characterized in that, The reward acquisition includes rewards in four stages: arrival, grabbing, lifting, and hovering. Where r reach , r grasp , r lift ,r hover These represent the rewards for arrival, grasping, lifting, and hovering, respectively; obstacle This indicates the penalty for a collision between the robotic arm and an obstacle; r time The time penalty incurred before the task is completed; d1 represents the Cartesian distance from the fixture to the target object; z set z object d1 represents the height of the set point and the height of the object to be reached, respectively; d2 represents the Cartesian distance from the target object to the target point, and r represents the height of the target object. hover 1 ,r hover2 α, β, γ, µ, λ, η represent the hovering rewards in the two cases respectively; α, β, γ, µ, λ, η represent the coefficients set in the reward formulas for arrival, gripping, lifting, and hovering respectively; κ represents the coefficient set for collision penalty; τ represents the coefficient set for time penalty.

7. The obstacle avoidance and grasping method for robotic arms based on continuous reinforcement learning according to claim 2, characterized in that, There is one obstacle in the first stage; there are two obstacles in the second stage; and there are two obstacles in the third stage.

8. The obstacle avoidance and grasping method for robotic arms based on continuous reinforcement learning according to claim 1, characterized in that, The deep learning algorithm is executed using the Soft Actor Critic algorithm.

Images