A method based on reinforcement deep learning to train the motion trajectory of a seven-DOF redundant robotic arm

Abstract

The present invention provides a method for training the motion trajectory of a seven-degree-of-freedom redundant manipulator based on enhanced deep learning, and relates to the field of manipulator control technology. The method for training the motion trajectory of a seven-degree-of-freedom redundant manipulator based on enhanced deep learning includes the following steps: reinforcement learning basis; spatial manipulator model: collision detection: setting the parameters of the manipulator obstacle avoidance motion trajectory: DQN manipulator path planning: using a three-layer DNN network as the network body of DQN, and automatically training the optimal motion trajectory through the manipulator obstacle avoidance path planning algorithm, avoiding space obstacles, and completing the enhanced deep learning training of the manipulator. Through the three-layer DNN network, the input and output are the state information and motion joint angle of the manipulator, respectively, and combined with offline training, the manipulator can train a motion trajectory close to the optimal by itself, successfully avoid obstacles and reach the target point, and has a strong obstacle avoidance ability.

Description

Technical Field

[0001] The present invention relates to the field of robotic arm control technology, and specifically to a method for training motion trajectories of a seven-degree-of-freedom redundant robotic arm based on enhanced deep learning. Background Art

[0002] For robotic arms, the concept of redundancy is relative and is defined for specific tasks. For a planar task, the commonly used 6-axis (six-degree-of-freedom) robotic arm is also redundant, but in most cases, it is usually adopted for all tasks, and the three-dimensional space can be described by six degrees of freedom. Therefore, the 7-axis (seven-degree-of-freedom) robotic arm is called a redundant robotic arm. The redundant degrees of freedom of the redundant robotic arm are used to achieve additional tasks such as body obstacle avoidance, singularity avoidance, joint limit avoidance, joint torque optimization, and increased operability. At the same time, based on the fact that a human arm also has seven degrees of freedom, from the perspective of bionics, redundant robotic arms are more in line with reality and scenario applications.

[0003] The seven-degree-of-freedom redundant robotic arm is subject to multiple constraints such as joint limits, surrounding environment, self-collision and dynamic balance in the process of performing tasks. Especially in container welding operations, in order to achieve efficient operation requirements, welding robotic arms are widely used in container production lines. However, due to the presence of many obstacles in the operating space, collision factors need to be considered in the process of automated welding. In order to smoothly perform welding operations, it is very necessary to study the obstacle avoidance path planning of the robotic arm.

[0004] To solve the problem of obstacle avoidance path planning for a robot arm, the existing technologies include the following: converting Cartesian space obstacles into configuration space obstacles, and then using the A* algorithm for collision-free path planning; randomly sampling the robot arm from the J space and mapping it to the C space through forward kinematics, and then using the improved RRT algorithm for path planning; using the artificial potential field method to plan the path of the robot arm, artificially simulating a repulsive field at the obstacle, and simulating a gravitational field at the target point, and the robot arm moves under the action of these two fields; using a genetic algorithm to plan the path of the robot arm; mapping the robot arm workspace to a two-dimensional space of a plane, and then using the A* algorithm for path planning in the plane space. According to the analysis of the existing technologies of Shanshu, this method can only be used in specified scenarios and lacks flexibility. At the same time, the above technologies require a more precise environment construction model and cannot be well applied in complex industrial environments.

[0005] Therefore, in order to solve the above problems, a method for training the motion trajectory of a seven-degree-of-freedom redundant robotic arm based on reinforced deep learning is proposed. Summary of the invention

[0006] In view of the shortcomings of the prior art, the present invention provides a method for training the motion trajectory of a seven-degree-of-freedom redundant robotic arm based on enhanced deep learning, which solves the problems of the existing seven-degree-of-freedom redundant robotic arm welding system and the difficulty of adjusting the movement and the lack of flexibility.

[0007] To achieve the above objectives, the present invention is implemented through the following technical solutions: A method for training motion trajectories of a seven-degree-of-freedom redundant robotic arm based on enhanced deep learning, comprising the following steps:

[0008] Step 1: Reinforcement learning foundation: Use the Ql earning algorithm and deep Q network as the underlying logic foundation of the robot arm reinforcement learning;

[0009] Step 2: Spatial robotic arm model: First, the redundant robotic arm is modeled. The redundant robotic arm uses the DH method to establish a mathematical model. At the same time, the relationship between adjacent links of the robotic arm is described by a 4×4 homogeneous transformation matrix, and the robotic arm is adjusted to positive motion, that is, a three-degree-of-freedom serial robotic arm model is obtained to simulate the motion trajectory of the seven-degree-of-freedom robotic arm and deep learning training;

[0010] Step 3, collision detection: the three-degree-of-freedom serial robot model obtained in step 2 is placed in a three-dimensional obstacle space, in which a spherical obstacle is set, the collision parameters between the robot and the spherical obstacle are detected, and the collision result is calculated and analyzed;

[0011] Step 4: Set the robot's obstacle avoidance trajectory parameters: set reward r, state s, and behavior a;

[0012] Reward r: represents the trajectory of the robot's action, the environment generates rewards, and forms the reward function parameters;

[0013] State s: represents the state variable parameters generated by the interaction between the robot and the environment;

[0014] Behavior a: represents the joint rotation angle vector of the robot arm;

[0015] Step 5: DQN robot path planning: A three-layer DNN network is used as the network body of DQN. The input is the state information of the robot, and the output is the motion joint angle of the robot. The robot obstacle avoidance path planning algorithm is used to automatically train the optimal motion trajectory, avoid spatial obstacles, and complete the enhanced deep learning training of the robot.

[0016] Preferably, in step 2, the general form of the homogeneous transformation matrix of the rotation joint is:

[0017]

[0018] Preferably, in step three, the step of determining whether the current position of the three-degree-of-freedom serial robot arm collides with the spherical obstacle is:

[0019] Step 1: Set the center of the spherical obstacle to P(X o ,Y o ,Z o ), with a radius of R o , the radius of the three-degree-of-freedom serial robot is R1, the two end points of the robot axis are A(X1, Y1, Z1) and B(X2, Y2, Z2), and the projection point of the obstacle to the straight line is P c (X c ,Y c ,Z c );

[0020] Step 2: Calculate the direction vector of the straight line: (X1-X2, Y1-Y2, Z1-Z2) = (M, N, P);

[0021] Step 3: Calculate the equation based on the spatial straight line of AB: c =MT+X1,Y c =NT+Y1,Z c =PT+Z1;

[0022] Step 4: The equation calculated based on the intersection relationship between the obstacle and the projection point and the straight line is: M(X0-X C )+N(Y0-Y C )+P(Z0-Z c )=0;

[0023] Step 5: Calculate the projection point coordinates P according to the above two formulas c (X c ,Y c ,Z c ), and then determine the distance from the obstacle to the robotic arm: D1 = √(X1-X C ) 2 +(Y1-Y C ) 2 +(Z1-Z c ) 2 ; Distance from obstacle to endpoint: D2 = √(X1-X0) 2 +(Y1-Y0) 2 +(Z1-Z0) 2 ;

[0024] Step 6: Analysis of collision detection algorithm results:

[0025] Case 1: The projection of the obstacle is within the line segment area, that is, X c Belongs to (X1-X2), Y c Belongs to (Y1, Y2), Z c Belongs to (Z1, Z2), then calculate the center R oThe distance D1 to line segment AB is greater than R o +R1, there is no collision, otherwise there is a collision;

[0026] Case 2: The projection of the obstacle is not within the line segment area, that is, X c Does not belong to (X1-X2), Y c Does not belong to (Y1, Y2), Z c If it does not belong to (Z1, Z2), calculate the center R o The distance D2 to line segment AB is greater than R o +R1, there is no collision, otherwise there is a collision;

[0027] In case three, if none of the three axes collides with obstacles, it is safe, otherwise it is a collision.

[0028] Preferably, in step 4, the reward r is for the path planning of the robot arm, which is mainly manifested in that the robot arm can smoothly avoid obstacles and reach the target point, while generating as few steps as possible. Therefore, the reward function is set as follows: first, reaching the target point is r=r+100, collision is r=r-100, and other r=0.6×P1 or 0.6×P2.

[0029] Preferably, in step 4, the vector form of state s is: S = (X4, Y4, Z4, X3, Y3, Z3, DS, MS, D4, D3).

[0030] Preferably, in step 4, the vector form of behavior a is: a=(θ1, θ2, θ3).

[0031] Preferably, in step 5, the robot arm obstacle avoidance path planning algorithm steps:

[0032] Step 1: Randomly initialize the values ​​Q corresponding to all actions and states, randomly initialize all parameters W of the current Q network, initialize the parameters W'=W of the target network Q', and clear the experience replay set D;

[0033] Step 2: Get the initial state S of the robot from the environment, and then select action a according to the ε-greedy strategy. The robot starts to perform the action and then obtains the reward R from the environment. t， Then obtain the state S' at the next moment;

[0034] Step 3: Store the interaction process between the robot and the environment: store (S, a, R, S') in D;

[0035] Step 4: Randomly sample n minimum batches of samples (S j ,a j , R j , S'j ), then j=1, and so on until the required n times are reached;

[0036] Step 5: The robot reaches the terminal state, making y j＝ R j， but

[0037] Step 6: Loss function Execute gradient descent and update the parameter W. At the same time, update the weight parameter W'=W of the target network every c steps, thus completing the calculation process of the robot's obstacle avoidance path planning.

[0038] The present invention provides a method for training the motion trajectory of a seven-degree-of-freedom redundant robotic arm based on enhanced deep learning. It has the following beneficial effects:

[0039] The present invention solves the path planning problem of the robotic arm by adopting a deep reinforcement learning algorithm. A three-layer DNN network is used, the input is the state information of the robotic arm, and the output is the motion joint angle of the robotic arm. Combined with offline training, the seven-degree-of-freedom redundant robotic arm can train itself to a motion trajectory close to the optimal one, and can successfully avoid obstacles to reach the target point. At the same time, simulation is used to simulate a three-degree-of-freedom spot welding robot on a simulation platform to obtain the motion trajectory of the seven-degree-of-freedom redundant robotic arm. It also shows that the robotic arm using deep reinforcement learning technology can plan a collision-free path for the welding robotic arm and has a strong obstacle avoidance capability. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 A schematic diagram of a three-degree-of-freedom spot welding robot according to the present invention;

[0041] Figure 2 It is a schematic diagram of the three-dimensional obstacle space of the present invention. DETAILED DESCRIPTION

[0042] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0043] Example:

[0044] like Figure 1-2 As shown, an embodiment of the present invention provides a method for training motion trajectories of a seven-degree-of-freedom redundant robotic arm based on enhanced deep learning, comprising the following steps:

[0045] Step 1: Reinforcement learning foundation: Use the Ql earning algorithm and deep Q network as the underlying logic foundation of the robot arm reinforcement learning;

[0046] Step 2: Spatial robotic arm model: First, the redundant robotic arm is modeled. The redundant robotic arm uses the DH method to establish a mathematical model. At the same time, the relationship between adjacent links of the robotic arm is described by a 4×4 homogeneous transformation matrix, and the robotic arm is adjusted to positive motion, that is, a three-degree-of-freedom serial robotic arm model is obtained to simulate the motion trajectory of the seven-degree-of-freedom robotic arm and deep learning training;

[0047] Step 3, collision detection: the three-degree-of-freedom serial robot model obtained in step 2 is placed in a three-dimensional obstacle space, in which a spherical obstacle is set, the collision parameters between the robot and the spherical obstacle are detected, and the collision result is calculated and analyzed;

[0048] Step 4: Set the robot's obstacle avoidance trajectory parameters: set reward r, state s, and behavior a;

[0049] Reward r: represents the trajectory of the robot's action, the environment generates rewards, and forms the reward function parameters;

[0050] State s: represents the state variable parameters generated by the interaction between the robot and the environment;

[0051] Behavior a: represents the joint rotation angle vector of the robot arm;

[0052] Step 5: DQN robot path planning: A three-layer DNN network is used as the network body of DQN. The input is the state information of the robot, and the output is the motion joint angle of the robot. The robot obstacle avoidance path planning algorithm is used to automatically train the optimal motion trajectory, avoid spatial obstacles, and complete the enhanced deep learning training of the robot.

[0053] The present invention, through step one and step five, through reinforcement learning basis, space manipulator model, collision detection, setting manipulator obstacle avoidance motion trajectory parameters and DQN manipulator path planning, enables the seven-degree-of-freedom redundant manipulator to self-train a motion trajectory close to the optimal, and can successfully avoid obstacles to reach the target point, and then a three-degree-of-freedom spot welding robot is simulated on a simulation platform, so that the manipulator passes through the trained neural network, outputs a maximum Q action in each step, and then uses positive kinematics to control the walking of the manipulator. The manipulator is trained by the reinforcement learning model, successfully avoids two obstacles, reaches the target point, and the number of movement steps of the manipulator is small, which is close to the optimal, and then the motion trajectory of the seven-degree-of-freedom redundant manipulator is obtained. At the same time, it also shows that the manipulator using deep reinforcement learning technology can plan a collision-free path for the welding manipulator, and has a strong obstacle avoidance capability.

[0054] In step 2, the general form of the homogeneous transformation matrix of the rotation joint is:

[0055]

[0056] In step 3, the steps for determining whether the current position of the three-degree-of-freedom serial robot arm collides with the spherical obstacle are as follows:

[0057] Step 1: Set the center of the spherical obstacle to P(X o ,Y o ,Z o ), with a radius of R o , the radius of the three-degree-of-freedom serial robot is R1, the two end points of the robot axis are A(X1, Y1, Z1) and B(X2, Y2, Z2), and the projection point of the obstacle to the straight line is P c (X c ,Y c ,Z c );

[0058] Step 2: Calculate the direction vector of the straight line: (X1-X2, Y1-Y2, Z1-Z2) = (M, N, P);

[0059] Step 3: Calculate the equation based on the spatial straight line of AB: c =MT+X1,Y c =NT+Y1,Z c =PT+Z1;

[0060] Step 4: The equation calculated based on the intersection relationship between the obstacle and the projection point and the straight line is: M(X0-X C )+N(Y0-Y C )+P(Z0-Z c )=0;

[0061] Step 5: Calculate the projection point coordinates P according to the above two formulas c (X c ,Y c ,Z c ), and then determine the distance from the obstacle to the robotic arm: D1 = √(X1-X C ) 2 +(Y1-Y C ) 2 +(Z1-Z c ) 2 ; Distance from obstacle to endpoint: D2 = √(X1-X0) 2 +(Y1-Y0) 2 +(Z1-Z0) 2 ;

[0062] Step 6: Analysis of collision detection algorithm results:

[0063] Case 1: The projection of the obstacle is within the line segment area, that is, X c Belongs to (X1-X2), Y c Belongs to (Y1, Y2), Z c Belongs to (Z1, Z2), then calculate the center R o The distance D1 to line segment AB is greater than R o +R1, there is no collision, otherwise there is a collision;

[0064] Case 2: The projection of the obstacle is not within the line segment area, that is, X c Does not belong to (X1-X2), Y c Does not belong to (Y1, Y2), Z c If it does not belong to (Z1, Z2), calculate the center R o The distance D2 to line segment AB is greater than R o +R1, there is no collision, otherwise there is a collision;

[0065] In case three, if none of the three axes collides with obstacles, it is safe, otherwise it is a collision.

[0066] In step 4, the reward r is for the path planning of the robot arm, which mainly means that the robot arm can successfully avoid obstacles and reach the target point, while generating as few steps as possible. Therefore, the reward function is set as follows: r = r + 100 for reaching the target point, r = r - 100 for collision, and r = 0.6 × P1 or 0.6 × P2 for others.

[0067] In step 4, the vector form of state s is: S = (X4, Y4, Z4, X3, Y3, Z3, DS, MS, D4, D3).

[0068] In step 4, the vector form of behavior a is: a=(θ1, θ2, θ3).

[0069] In step 5, the robot arm obstacle avoidance path planning algorithm steps:

[0070] Step 1: Randomly initialize the values ​​Q corresponding to all actions and states, randomly initialize all parameters W of the current Q network, initialize the parameters W'=W of the target network Q', and clear the experience replay set D;

[0071] Step 2: Get the initial state S of the robot from the environment, and then select action a according to the ε-greedy strategy. The robot starts to perform the action and then obtains the reward R from the environment. t， Then obtain the state S' at the next moment;

[0072] Step 3: Store the interaction process between the robot and the environment: store (S, a, R, S') in D;

[0073] Step 4: Randomly sample n minimum batches of samples (S j ,a j , R j , S' j ), then j=1, and so on until the required n times are reached;

[0074] Step 5: The robot reaches the terminal state, making y j＝ R j， but

[0075] Step 6: Loss function Execute gradient descent and update the parameter W. At the same time, update the weight parameter W'=W of the target network every c steps, thus completing the calculation process of the robot's obstacle avoidance path planning.

[0076] The present invention takes a seven-degree-of-freedom redundant robotic arm for spot welding as the research object, and transforms the automatic spot welding of the robotic arm in three-dimensional space into obstacle avoidance path planning of the robotic arm. Through certain training steps, the robotic arm can smoothly plan a collision-free welding path. Compared with traditional planning methods, the planning method based on reinforcement learning is simple and has higher versatility.

[0077] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for training motion trajectories of a seven-degree-of-freedom redundant robotic arm based on enhanced deep learning, characterized in that: The following steps are involved: Step 1: Reinforcement learning foundation: Use q-learning algorithm and deep Q network as the underlying logic foundation of the robot arm reinforcement learning; Step 2: Spatial robotic arm model: First, the redundant robotic arm is modeled. The redundant robotic arm uses the DH method to establish a mathematical model. At the same time, the relationship between adjacent links of the robotic arm is described by a 4×4 homogeneous transformation matrix, and the robotic arm is adjusted to positive motion, that is, a three-degree-of-freedom serial robotic arm model is obtained to simulate the motion trajectory of the seven-degree-of-freedom robotic arm and deep learning training; Step 3, collision detection: the three-degree-of-freedom serial robot model obtained in step 2 is placed in a three-dimensional obstacle space, in which a spherical obstacle is set, the collision parameters between the robot and the spherical obstacle are detected, and the collision result is calculated and analyzed; Step 4: Set the robot's obstacle avoidance trajectory parameters: set reward r, state s, and behavior a; Reward r: represents the trajectory of the robot's action, the environment generates rewards, and forms the reward function parameters; State s: represents the state variable parameters generated by the interaction between the robot and the environment; Behavior a: represents the joint rotation angle vector of the robot arm; Step 5: DQN robot arm path planning: A three-layer DNN network is used as the network body of DQN. The input is the state information of the robot arm, and the output is the motion joint angle of the robot arm. The robot arm obstacle avoidance path planning algorithm is used to automatically train the optimal motion trajectory to avoid spatial obstacles and complete the enhanced deep learning training of the robot arm. In step 3, the step of determining whether the current position of the three-degree-of-freedom serial robot arm collides with the spherical obstacle is as follows: Step 1: Set the center of the spherical obstacle to P(X o ,Y o ,Z o ), with a radius of R o , the radius of the three-degree-of-freedom serial robot is R1, the two end points of the robot axis are A(X1, Y1, Z1) and B(X2, Y2, Z2), and the projection point of the obstacle to the straight line is P c (X c ,Y c ,Z c ); Step 2: Calculate the direction vector of the straight line: (X1-X2, Y1-Y2, Z1-Z2) = (M, N, P); Step 3: Calculate the equation based on the spatial straight line of AB: c =MT+X1,Y c =NT+Y1,Z c =PT+Z1; Step 4: The equation calculated based on the intersection relationship between the obstacle and the projection point and the straight line is: M(X0-X C )+N(Y0-Y C )+P(Z0-Z c )=0; Step 5: Calculate the projection point coordinates P according to the above two formulas c (X c ,Y c ,Z c ), and then determine the distance from the obstacle to the robotic arm: D1 = √(X1-X C ) 2 +(Y1-Y C ) 2 +(Z1-Z c ) 2 ; Distance from obstacle to endpoint: D2 = √(X1-X0) 2 +(Y1-Y0) 2 +(Z1-Z0) 2 ; Step 6: Analysis of collision detection algorithm results: Case 1: The projection of the obstacle is within the line segment area, that is, X c Belongs to (X1-X2), Y c Belongs to (Y1, Y2), Z c Belongs to (Z1, Z2), then calculate the center R o The distance D1 to line segment AB is greater than R o +R1, there is no collision, otherwise there is a collision; Case 2: The projection of the obstacle is not within the line segment area, that is, X c Does not belong to (X1-X2), Y c Does not belong to (Y1, Y2), Z c If it does not belong to (Z1, Z2), calculate the center R o The distance D2 to line segment AB is greater than R o +R1, there is no collision, otherwise there is a collision; Case 3: If none of the three axes collides with obstacles, it is safe, otherwise it is a collision phenomenon; In step 5, the robot arm obstacle avoidance path planning algorithm steps: Step 1: Randomly initialize the values ​​Q corresponding to all actions and states, randomly initialize all parameters W of the current Q network, initialize the parameters W'=W of the target network Q', and clear the experience replay set D; Step 2: Get the initial state S of the robot from the environment, and then select action a according to the ε-greedy strategy. The robot starts to perform the action and then obtains the reward R from the environment. t， Then obtain the state S' at the next moment; Step 3: Store the interaction process between the robot and the environment: store (S, a, R, S') in D; Step 4: Randomly sample n minimum batches of samples (S j ,a j , R j , S' j ), then j=1, and so on until the required n times are reached; Step 5: The robot reaches the terminal state, making y j＝ R j， but Step 6: Loss function Execute gradient descent and update the parameter W. At the same time, update the weight parameter W'=W of the target network every c steps, thus completing the calculation process of the robot's obstacle avoidance path planning.

2. The method for training motion trajectory of a seven-degree-of-freedom redundant manipulator based on enhanced deep learning according to claim 1, characterized in that: In step 2, the general form of the homogeneous transformation matrix of the rotation joint is:

3. The method for training motion trajectory of a seven-degree-of-freedom redundant robotic arm based on enhanced deep learning according to claim 1, characterized in that: In step 4, the reward r is for the path planning of the robot arm, which is mainly manifested in that the robot arm can successfully avoid obstacles and reach the target point, while generating as few steps as possible. Therefore, the reward function is set as follows: first, reaching the target point is r=r+100, collision is r=r-100, and other r=0.6×P1 or 0.6×P2.

4. The method for training motion trajectory of a seven-degree-of-freedom redundant manipulator based on enhanced deep learning according to claim 1, characterized in that: In the step 4, the vector form of the state s is: S = (X4, Y4, Z4, X3, Y3, Z3, DS, MS, D4, D3).

5. The method for training motion trajectory of a seven-degree-of-freedom redundant manipulator based on enhanced deep learning according to claim 1, characterized in that: In the step 4, the vector form of behavior a is: a=(θ1, θ2, θ3).

Images