Seven-degree-of-freedom mechanical arm trajectory planning method based on improved deep reinforcement learning PPO

Abstract

The invention discloses a seven-degree-of-freedom mechanical arm trajectory planning method based on improved deep reinforcement learning PPO, and belongs to the technical field of industrial robot intelligent control. In order to solve the problems of poor learning ability and slow convergence speed of deep reinforcement learning in seven-degree-of-freedom redundant mechanical arm trajectory planning, a staged reward function is designed, and the seven-degree-of-freedom mechanical arm is gradually guided to complete a task through two stages. A layered experience playback mechanism is designed, and experience in the training process is stored in a priority experience storage area and a secondary experience storage area according to reward values; the sampling proportion of the two types of storage areas is balanced through an unbiased sampling strategy, the utilization rate of experience samples is improved, and compared with a traditional priority experience playback mechanism, the method has the advantages that the calculation amount is reduced, and meanwhile, the overfitting problem in the training process is effectively avoided. Through a staged reward function and a layered experience playback mechanism, the learning time of the seven-degree-of-freedom mechanical arm in a complex environment is effectively shortened, and the convergence speed is remarkably increased.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent control of industrial robots, and specifically relates to a seven-degree-of-freedom robotic arm trajectory planning method based on improved deep reinforcement learning PPO. Background Art

[0002] With the continuous advancement of industrial robotics technology, its application areas have expanded to include industrial manufacturing, military engineering, social services, and other fields, making significant contributions to the development of human society. As the working environments and tasks of robotic arms become increasingly complex, the demand for more degrees of freedom for robotic arms is also increasing. Seven-degree-of-freedom robotic arms, with their redundant degree of freedom, offer superior flexibility and are therefore increasingly popular in various application scenarios. However, as the complexity of the robotic arm structure or tasks increases, the control methods for seven-degree-of-freedom redundant robotic arms also become more complex, and trajectory planning has become one of the key challenges in controlling this type of robotic arm.

[0003] Compared to traditional methods, deep reinforcement learning algorithms offer a high degree of automation, intelligence, precision, stability, and flexibility, meeting the demands of intelligent control for redundant seven-degree-of-freedom robotic arms. Despite the in-depth research of artificial intelligence technology in the control field, robotic arms still have limited learning capabilities when faced with unfamiliar environments, with slow convergence rates and often suboptimal planned trajectories. In scenarios where precision and safety are critical, deep reinforcement learning algorithms with improved convergence rates can learn better strategies for trajectory planning, ensuring the smoothness and autonomy of the redundant seven-degree-of-freedom robotic arm's movements and avoiding potential hazards to the surrounding environment and personnel. In summary, deep reinforcement learning trajectory planning methods that consider convergence rate optimization can improve the overall performance, economic efficiency, and reliability of robotic systems, meeting the needs of modern industry and service industries. Summary of the Invention

[0004] Aiming at the problems of poor learning ability and slow convergence speed faced by deep reinforcement learning in trajectory planning of seven-DOF redundant manipulator, the present invention provides a seven-DOF manipulator trajectory planning method based on improved deep reinforcement learning PPO.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] A seven-degree-of-freedom robot arm trajectory planning method based on improved deep reinforcement learning PPO, the method comprising the following steps:

[0007] Step 1: Set the target pose of the end of the seven-degree-of-freedom redundant manipulator;

[0008] In the step 1, the Cartesian space target pose of the end of the seven-degree-of-freedom redundant manipulator is set as the end point pose of the end of the seven-degree-of-freedom redundant manipulator.

[0009] Step 2: Obtain the position and posture of the end of the seven-degree-of-freedom redundant manipulator, and use the arm angle parameter method to perform inverse kinematics solution on the seven-degree-of-freedom redundant manipulator to obtain the current position of each joint of the manipulator;

[0010] In step 2, the position of the end of the seven-degree-of-freedom redundant manipulator is obtained, and the arm angle parameter method is used to perform inverse kinematics on the seven-degree-of-freedom redundant manipulator to obtain the current position of each joint of the manipulator , the specific operations are:

[0011] The robotic arm adopts a "3-1-3" structure, consisting of three shoulder joints, one elbow joint, and three wrist joints. The model labels the shoulder joint hinge point as S, the elbow joint hinge point as E, and the wrist joint hinge point as W.

[0012] When the arm plane WES rotates around the line SW connecting the shoulder joint S and the wrist joint W, the position of the end effector remains unchanged. This kind of motion is called self-motion. Point E' represents the elbow joint hinge point moving from point E to point E' when the seven-degree-of-freedom manipulator performs self-motion. Determines the end posture of the robotic arm. Is the angle between the arm plane WES and the reference plane. The inverse kinematics solution for the seven-degree-of-freedom redundant manipulator can be divided into three steps: First, use the triangle cosine theorem to calculate the angle of the fourth joint ; Then, according to the position of the end effector, the arm angle is calculated Finally, an arm angle value is selected within this range, and each joint angle can be obtained through the end position and arm angle value.

[0013] Assume that the position of the end effector of the seven-degree-of-freedom robot arm is:

[0014]

[0015] in, arrive Represents the rotation relationship between the seven-degree-of-freedom redundant manipulator end coordinate system and the base coordinate system. arrive Each column of corresponds to the unit vector of the terminal coordinate system's X-axis, Y-axis, and Z-axis in the base coordinate system. The first three rows of the last column of the above formula represent the position of the terminal relative to the base coordinate system.

[0016] First, find the origin of each link coordinate system: the end position of the robotic arm is expressed as The position of the wrist joint W point in the seven-degree-of-freedom manipulator end coordinate system is expressed as , use the following formula to calculate the information of wrist joint W relative to the base coordinate system: ;

[0017] The position of the wrist joint S relative to the base coordinate system is , in △SEW, using the triangle cosine theorem we can know: ;

[0018] The position of joint C is calculated using the formula: ;

[0019] Calculate the unit vector of the joint C coordinate system relative to the base coordinate system using the following formula: ;

[0020] Joint E rotates around the Z axis of the coordinate system of joint C, and the radius of rotation is defined as , and assuming the arm angle is , then the position of point E is obtained using the following formula: ;

[0021] Using the above method, the positions of S, E, W, T, and C relative to the base coordinate system can be obtained, and then the angle of each joint of the robotic arm can be solved.

[0022] (1) Solution: yes On the surface The projection and Angle of the axes: ;

[0023] (2) Solution: yes and The angle between: ;

[0024] (3) Solution: Calculate the coordinate system first The transformation matrix between the base coordinate system , the formula is as follows: ;

[0025] In the coordinate system The medium energy is expressed as: ,and yes On the surface The projection and The angle between the axes is given by the following formula: ;

[0026] (4) Solution: yes and The angle between them is given by the following formula: ;

[0027] (5) Solution: First find the coordinate system The transformation matrix between the base coordinate system . In the coordinate system The medium energy is expressed as: . yes On the surface The projection and The angle between the axes is given by the following formula: ;

[0028] (6) Solution: yes and The angle between: ;

[0029] (7) Solution: First get the rotation matrix of the sixth joint coordinate system , then the rotation matrix of the seventh joint is , calculate the angle :

[0030]

[0031]

[0032] in, Representative Matrix The number in the second row and first column; Representative Matrix The number in the second row and second column.

[0033] According to the characteristics of inverse trigonometric functions, , , There are two solutions for each. Therefore, eight sets of joint space solutions are calculated. However, in practice, not all solutions meet the requirements. Solutions that do not satisfy the manipulator's kinematic constraints need to be eliminated.

[0034] Step 3: Perform trajectory planning using the deep reinforcement learning PPO algorithm with a staged reward function and a hierarchical experience replay mechanism, and output the increments of each joint;

[0035] The specific operations of step 3 are:

[0036] Set the input information, output information, reward function and experience replay mechanism of the deep reinforcement learning PPO algorithm;

[0037] The input information is the state space, and the output information is the action space; the reward function is the feedback obtained after the agent performs the action; the experience replay mechanism adopts a hierarchical experience storage structure and an unbiased sampling strategy;

[0038] The state space contains the environment information and the state information of the robot arm, which provides the basis for the decision-making and action execution of the intelligent agent; the position of the end effector , distance from the target location and whether there is a collision with an obstacle Let be the observed value, so the state space is defined as: ,in, It is a three-dimensional array representing the position of the end of the seven-degree-of-freedom manipulator relative to the base coordinate system. 、 and Position in the axial direction; Is a floating point number representing the distance of the robot end effector relative to the target position; Is a two-dimensional array that indicates whether the robot arm collides with an obstacle and which link collides with the obstacle;

[0039] Controlling the movement of the robotic arm is actually achieved by controlling the rotation of the motors of each joint of the robotic arm. Defined as the increment of each joint's rotation angle, that is, directly defined as the change in joint angle, because there is no need for inverse kinematics calculations, but instead the rotation of each joint is directly controlled, the motion control of the robot arm becomes more efficient. This definition not only reduces the amount of algorithm calculation, but also improves the response speed and accuracy of the robot arm's motion control. The formula is as follows: , where arrive Represents the increment of the rotation angle of each joint. In order to avoid the robot arm from moving too violently and unstable due to excessive action output in a single time step, arrive The value range is limited to -0.5rad to 0.5rad.

[0040] During the training process, the reward function defines the feedback obtained after the agent performs an action, which is used to help the agent adjust its strategy and maximize the cumulative reward; the sparse reward function The formula is as follows:

[0041]

[0042]

[0043] in, Indicates the distance between the current robot arm end effector and the target position; Indicates the distance from the target position at the previous moment; Indicates the target location to be set; represents the position of the end of the robotic arm at the t-th movement;

[0044] Dense reward function The formula is as follows: ;

[0045] An ideal reward function should effectively reduce the agent's ineffective exploration while improving training efficiency. In complex environments, to better balance short-term and long-term goals, it's important to combine the strengths of both dense and sparse reward functions. When the distance to the target is far, the algorithm uses a sparse reward function; when the distance is close, the algorithm uses a dense reward function. This design facilitates the design of more refined reward mechanisms, accelerating the agent's learning process and enabling it to better cope with challenges in complex environments.

[0046] In order to more efficiently balance long-term and short-term goals, a phased reward function is designed to gradually guide the agent to achieve its goals through multiple stages. This function can not only guide the agent in the global space, but also optimize the agent's behavior in the local space, thereby significantly improving the stability and efficiency of training and accelerating the agent's learning process in complex environments.

[0047] Since the dense reward function imposes reasonable constraints on the relative distance between the manipulator and the target position, the manipulator will often collide with obstacles in complex environments. Therefore, at the beginning of the task, we do not want the deep reinforcement learning algorithm to use a dense reward function, and propose a staged reward function. During the motion of the seven-degree-of-freedom redundant manipulator, an adjustable coefficient is used. This approach uses different reward functions at different stages. The phased reward function divides the trajectory planning task into two stages: near-region trajectory planning and far-region trajectory planning. In the far-region, a sparse reward function is used as the incentive mechanism; in the near-region, a dense reward function encourages the robot arm to quickly reach the target position.

[0048] use As the boundary, it is divided into near area and far area. This setting can better meet the different tasks in complex environments. , the dense reward function guided by distance will take longer to work, the robot arm will easily collide with obstacles and will have difficulty adapting to complex environments. , the reward mechanism will use sparse reward functions for a longer time, resulting in slow convergence of the algorithm. Therefore, in order to avoid the above situation, The value is 0.5m. When the distance between the end of the manipulator and the target point is greater than 0.5m, it belongs to the far area; when the distance between the end of the manipulator and the target point is less than 0.5m, it belongs to the near area. Adjustable coefficient The calculation formula is as follows: ;

[0049] Phased reward function The calculation formula is as follows: ,in, Indicates whether the end of the robotic arm is in the far area; Indicates whether the end of the robot arm is in the near area; represents transpose;

[0050] The core idea of the experience replay mechanism is to break the temporal correlation of data by repeatedly using past experience, thereby improving sample efficiency and accelerating the learning process. Although the prioritized experience replay mechanism improves training efficiency by prioritizing the extraction of higher-value experiences, it faces some challenges in practical applications. First, prioritized experience replay requires continuous prioritization of samples in each training step, which significantly increases the amount of computation. Second, prioritized experience replay only focuses on high-value experience samples, which may lead to overfitting problems. Because as training progresses, the strategy may rely too much on those experiences with high reward values and ignore other experience samples that may be potentially helpful for strategy learning. In order to solve these problems, a hierarchical experience replay mechanism is designed, which adopts a hierarchical experience storage structure and an unbiased sampling strategy;

[0051] In this hierarchical experience storage structure, the agent distinguishes the value of experiences based on their reward value, classifying them into high-value and low-value experiences. High-value experiences (i.e., experiences with higher reward values) are stored in the priority experience storage area, while low-value experiences are stored in the secondary experience storage area. Both storage areas follow the first-in-first-out (FIFO) principle to ensure timely updates. By categorizing experiences based on reward value, this hierarchical storage structure effectively distinguishes and stores experiences of different values without requiring complex sorting of all samples. Compared to the prioritized experience replay mechanism, this approach not only simplifies the storage and update process but also significantly reduces computational effort.

[0052] However, while using only high-reward experiences for training can accelerate learning, it can also lead to overfitting. During training, if the agent relies solely on experiences with high rewards, it may lose sensitivity to other experiences, leading to overfitting of the policy to certain specific scenarios. Therefore, to address this issue, an unbiased sampling strategy is proposed.

[0053] The unbiased sampling strategy aims to balance the sampling ratio between the priority experience store and the secondary experience store; specifically, it allows experience samples to be drawn from these two stores at different ratios for training. In this way, the agent can fully utilize high-reward experience while avoiding over-reliance on such experience, thereby reducing the risk of overfitting and maintaining the policy's generalization ability.

[0054] In summary, the improved experience replay mechanism effectively addresses the computational overhead of prioritizing experience replay by introducing a hierarchical experience storage structure and an unbiased sampling strategy. It also reduces the risk of overfitting through a reasonable sampling mechanism. This mechanism better balances the efficiency of experience utilization with the generalization ability of the strategy, thereby accelerating strategy convergence and improving training stability. The unbiased sampling strategy formula is as follows: , ,in, Indicates the number of current training; Indicates the maximum number of training times set; Indicates the total number of experience samples extracted from the experience storage area; Indicates the number of experience samples extracted from the priority experience storage area; Indicates the number of experience samples extracted from the secondary experience storage area;

[0055] The deep reinforcement learning PPO algorithm using a staged reward function and a hierarchical experience replay mechanism performs the following steps at each iteration of trajectory planning:

[0056] Step 3.1: Initialize the policy network and value network ,set up , , , , , ;

[0057] Step 3.2: For each iteration of do, when the agent interacts with the environment, , ;

[0058] Step 3.3: When > 0, , when not > 0, then ;

[0059] Step 3.4: For experience sampling, when : ,when , ,otherwise, ;

[0060] Step 3.5: From and Extracted from the experience storage area and Experience samples;

[0061] Step 3.6: For each experience extracted ,calculate , calculate the replacement loss ;

[0062] Step 3.7: Accumulate losses ,in, is the entropy regularization term;

[0063] Step 3.8: Update using the optimizer and To minimize ;

[0064] The algorithm is described as follows:

[0065]

[0066] in, Indicates the current number of training times; Indicates the total number of defined training times; Indicates the size of the priority experience storage area; Indicates the size of the secondary experience storage area; Indicates the number of experiences extracted from the priority experience storage area; Indicates the total number of experiences extracted from the two experience storage areas; A staged reward function is used; lines 4 to 11 indicate that during the training process, experiences are stored separately according to good and bad for subsequent use; lines 12 to 20 indicate that samples are extracted from the stored experiences through an unbiased sampling strategy; lines 21 to 26 indicate that the parameters of the deep reinforcement learning PPO algorithm are optimized and the strategy is updated based on the extracted experiences.

[0067] Step 4: Add the joint increments obtained above to the current positions of the joints of the robotic arm to obtain the joint positions at the next moment for controlling the movement of the robotic arm.

[0068] In step 4, the joint increments output by the improved deep reinforcement learning PPO algorithm are added to the current positions of the joints of the robotic arm to obtain the joint positions at the next moment for controlling the movement of the robotic arm, as follows: , , , , , , ,in, Indicates the position of each joint at the next moment when controlling the movement of the robotic arm.

[0069] Step 5: Repeat steps 2, 3, and 4 until the target pose is reached.

[0070] Compared with the prior art, the present invention has the following advantages:

[0071] The present invention proposes a seven-degree-of-freedom manipulator trajectory planning method based on improved deep reinforcement learning PPO. The method optimizes the learning efficiency during the training process based on planning the trajectory of a seven-degree-of-freedom redundant manipulator. By proposing a phased reward function, the intelligent agent can be guided in the global space and the behavior of the intelligent agent can be optimized in the local space, thereby accelerating the learning ability of the intelligent agent in a complex environment. A hierarchical experience replay mechanism is also proposed, which distinguishes high-value and low-value experiences by reward values, and uses an unbiased sampling strategy to extract experience samples, making more effective use of experience and improving the learning speed of the strategy. The present invention can learn a better strategy for trajectory planning through training, and can better control the movement of a seven-degree-of-freedom redundant manipulator. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] Figure 1 This is a flow chart of a seven-DOF robotic arm trajectory planning method that improves deep reinforcement learning PPO;

[0073] Figure 2 This is a simplified schematic diagram of a seven-degree-of-freedom robotic arm model;

[0074] Figure 3 Schematic diagram of the average reward value and success rate during the training process of various reward functions;

[0075] Figure 4 Schematic diagram of the average reward value and success rate during the training process of various algorithms;

[0076] Figure 5 Schematic diagram of the trajectory of the robot arm controlled by PPO2, PPO3 and PPO4 algorithms;

[0077] Figure 6 Schematic diagram of the trajectory of the robotic arm controlled by PPO, PER-PPO and HM-PPO algorithms. DETAILED DESCRIPTION

[0078] To gain a deeper understanding of the present invention, we will provide a comprehensive and detailed description thereof. However, the present invention has various implementations and is not limited to the specific examples listed herein. These examples are presented to enhance a comprehensive understanding of the present disclosure.

[0079] An improved deep reinforcement learning PPO seven-degree-of-freedom robot arm trajectory planning method, the method flow chart is as follows Figure 1 As shown, the method includes the following steps:

[0080] Step 1: Set the target pose of the end of the seven-degree-of-freedom manipulator;

[0081] In the step 1, the Cartesian space target pose of the end of the seven-degree-of-freedom redundant manipulator is set as the end point pose of the end of the seven-degree-of-freedom redundant manipulator.

[0082] Step 2: Obtain the position and posture of the end of the seven-degree-of-freedom redundant manipulator, and use the arm angle parameter method to perform inverse kinematics solution on the seven-degree-of-freedom redundant manipulator to obtain the current position of each joint of the manipulator;

[0083] In step 2, the position of the end of the seven-degree-of-freedom redundant manipulator is obtained, and the arm angle parameter method is used to perform inverse kinematics on the seven-degree-of-freedom redundant manipulator to obtain the current position of each joint of the manipulator , the specific operations are:

[0084] The robotic arm adopts a "3-1-3" structure, consisting of three shoulder joints, one elbow joint, and three wrist joints. The model labels the shoulder joint hinge point as S, the elbow joint hinge point as E, and the wrist joint hinge point as W.

[0085] Depend on Figure 2 It can be seen that when the arm plane WES of the robot arm rotates around the line SW connecting the shoulder joint S and the wrist joint W, the position of the end effector remains unchanged. This kind of motion is called self-motion. Point E' in the figure represents the elbow joint hinge point moving from point E to point E' when the seven-degree-of-freedom robot arm performs self-motion. During the self-motion process, the arm angle Determines the end posture of the robotic arm. Is the angle between the arm plane WES and the reference plane. The inverse kinematics solution for the seven-degree-of-freedom redundant manipulator can be divided into three steps: First, use the triangle cosine theorem to calculate the angle of the fourth joint ; Then, according to the position of the end effector, the arm angle is calculated Finally, an arm angle value is selected within this range, and each joint angle can be obtained through the end position and arm angle value.

[0086] Assume that the position of the end effector of the seven-degree-of-freedom robot arm is: ,in, arrive Represents the rotation relationship between the seven-degree-of-freedom redundant manipulator end coordinate system and the base coordinate system. arrive Each column of corresponds to the unit vector of the terminal coordinate system's X-axis, Y-axis, and Z-axis in the base coordinate system. The first three rows of the last column of the above formula represent the position of the terminal relative to the base coordinate system.

[0087] First, find the origin of each link coordinate system: the end position of the robotic arm is expressed as The position of the wrist joint W point in the seven-degree-of-freedom manipulator end coordinate system is expressed as , use the following formula to calculate the information of wrist joint W relative to the base coordinate system: ;

[0088] The position of the wrist joint S relative to the base coordinate system is , in △SEW, using the triangle cosine theorem we can know: ;

[0089] The position of joint C is calculated using the formula: ;

[0090] Calculate the unit vector of the joint C coordinate system relative to the base coordinate system using the following formula: ;

[0091] Joint E rotates around the Z axis of the coordinate system of joint C, and the radius of rotation is defined as , and assuming the arm angle is , then the position of point E is obtained using the following formula: ;

[0092] Using the above method, the positions of S, E, W, T, and C relative to the base coordinate system can be obtained, and then the angle of each joint of the robotic arm can be solved.

[0093] (1) Solution: yes On the surface The projection and Angle of the axes: ;

[0094] (2) Solution: yes and The angle between: ;

[0095] (3) Solution: Calculate the coordinate system first The transformation matrix between the base coordinate system , the formula is as follows: ;

[0096] In the coordinate system The medium energy is expressed as: ,and yes On the surface The projection and The angle between the axes is given by the following formula: ;

[0097] (4) Solution: yes and The angle between them is given by the following formula: ;

[0098] (5) Solution: First find the coordinate system The transformation matrix between the base coordinate system . In the coordinate system The medium energy is expressed as: . yes On the surface The projection and The angle between the axes is given by the following formula: ;

[0099] (6) Solution: yes and The angle between: ;

[0100] (7) Solution: First get the rotation matrix of the sixth joint coordinate system , then the rotation matrix of the seventh joint is , calculate the angle :

[0101]

[0102]

[0103] in, Representative Matrix The number in the second row and first column; Representative Matrix The number in the second row and second column.

[0104] According to the characteristics of inverse trigonometric functions, , , There are two solutions for each. Therefore, eight sets of joint space solutions are calculated. However, in practice, not all solutions meet the requirements. Solutions that do not satisfy the manipulator's kinematic constraints need to be eliminated.

[0105] Step 3: Perform trajectory planning using the deep reinforcement learning PPO algorithm with a staged reward function and a hierarchical experience replay mechanism, and output the increments of each joint;

[0106] The specific operations of step 3 are:

[0107] Set the input information, output information, reward function and experience replay mechanism of the deep reinforcement learning PPO algorithm;

[0108] The input information is the state space, and the output information is the action space; the reward function is the feedback obtained after the agent performs the action; the experience replay mechanism adopts a hierarchical experience storage structure and an unbiased sampling strategy;

[0109] The state space contains the environment information and the state information of the robot arm, which provides the basis for the decision-making and action execution of the intelligent agent; the position of the end effector , distance from the target location and whether there is a collision with an obstacle Let be the observed value, so the state space is defined as: ,in, It is a three-dimensional array representing the position of the end of the seven-degree-of-freedom manipulator relative to the base coordinate system. 、 and Position in the axial direction; Is a floating point number representing the distance of the robot end effector relative to the target position; Is a two-dimensional array that indicates whether the robot arm collides with an obstacle and which link collides with the obstacle;

[0110] Controlling the movement of the robotic arm is actually achieved by controlling the rotation of the motors of each joint of the robotic arm. Defined as the increment of each joint's rotation angle, that is, directly defined as the change in joint angle, because there is no need for inverse kinematics calculations, but instead the rotation of each joint is directly controlled, the motion control of the robot arm becomes more efficient. This definition not only reduces the amount of algorithm calculation, but also improves the response speed and accuracy of the robot arm's motion control. The formula is as follows: , where arrive Represents the increment of the rotation angle of each joint. In order to avoid the robot arm from moving too violently and unstable due to excessive action output in a single time step, arrive The value range is limited to -0.5rad to 0.5rad.

[0111] During the training process, the reward function defines the feedback obtained after the agent performs an action, which is used to help the agent adjust its strategy and maximize the cumulative reward; the sparse reward function The formula is as follows: , ,in, Indicates the distance between the current robot arm end effector and the target position; Indicates the distance from the target position at the previous moment; Indicates the target location to be set; represents the position of the end of the robotic arm at the t-th movement;

[0112] Dense reward function The formula is as follows: ;

[0113] A phased reward function is designed to gradually guide the agent to achieve the goal through multiple stages. The phased reward function divides the trajectory planning task into two stages, including near-area trajectory planning and far-area trajectory planning. In the far-area, a sparse reward function is used as an incentive mechanism; in the near-area, a dense reward function encourages the robot arm to quickly reach the target position. The value is 0.5m. When the end of the robotic arm is more than 0.5m away from the target point, it belongs to the far area; when the end of the robotic arm is less than 0.5m away from the target point, it belongs to the near area. An adjustable coefficient is used. Implement different reward functions at different stages with adjustable coefficients The calculation formula is as follows:

[0114]

[0115] Phased reward function The calculation formula is as follows: ,in, Indicates whether the end of the robotic arm is in the far area; Indicates whether the end of the robot arm is in the near area; represents transpose;

[0116] Design a hierarchical experience replay mechanism, using a hierarchical experience storage structure and an unbiased sampling strategy;

[0117] In the hierarchical experience storage structure, the agent distinguishes the value of experience by the reward value of the experience and divides it into high-value experience and low-value experience; high-value experience is stored in the priority experience storage area, and low-value experience is stored in the secondary experience storage area;

[0118] The unbiased sampling strategy aims to balance the sampling ratio between the priority experience storage area and the secondary experience storage area. The unbiased sampling strategy formula is as follows: , ,in, Indicates the number of current training; Indicates the maximum number of training times set; Indicates the total number of experience samples extracted from the experience storage area; Indicates the number of experience samples extracted from the priority experience storage area; Indicates the number of experience samples extracted from the secondary experience storage area;

[0119] The deep reinforcement learning PPO algorithm using a staged reward function and a hierarchical experience replay mechanism performs the following steps at each iteration of trajectory planning:

[0120] Step 3.1: Initialize the policy network and value network ,set up , , , , , ;

[0121] Step 3.2: For each iteration of do, when the agent interacts with the environment, , ;

[0122] Step 3.3: When > 0, , when not > 0, then ;

[0123] Step 3.4: For experience sampling, when : ,when , ,otherwise, ;

[0124] Step 3.5: From and Extracted from the experience storage area and Experience samples;

[0125] Step 3.6: For each experience extracted ,calculate , calculate the replacement loss ;

[0126] Step 3.7: Accumulate losses ,in, is the entropy regularization term;

[0127] Step 3.8: Update using the optimizer and To minimize ;

[0128] The algorithm is described as follows:

[0129]

[0130] in, Indicates the current number of training times; Indicates the total number of defined training times; Indicates the size of the priority experience storage area; Indicates the size of the secondary experience storage area; Indicates the number of experiences extracted from the priority experience storage area; Indicates the total number of experiences extracted from the two experience storage areas; A staged reward function is used; lines 4 to 11 indicate that during the training process, experiences are stored separately according to good and bad for subsequent use; lines 12 to 20 indicate that samples are extracted from the stored experiences through an unbiased sampling strategy; lines 21 to 26 indicate that the parameters of the deep reinforcement learning PPO algorithm are optimized and the strategy is updated based on the extracted experiences.

[0131] Step 4: Add the joint increments obtained above to the current positions of the joints of the robotic arm to obtain the joint positions at the next moment for controlling the movement of the robotic arm.

[0132] In step 4, the joint increments output by the improved deep reinforcement learning PPO algorithm are added to the current positions of the joints of the robotic arm to obtain the joint positions at the next moment for controlling the movement of the robotic arm, as follows: , , , , , ,

[0133] in, Indicates the position of each joint at the next moment when controlling the movement of the robotic arm.

[0134] Step 5: Repeat steps 2, 3, and 4 until the target pose is reached.

[0135] We conducted 5,000 training runs using a binary reward function, a sparse reward function, a dense reward function, and a phased reward function. The effectiveness of the proposed phased reward function was verified by comparing the training data with the experimental results. Figure 3The average reward value and success rate during training under different reward functions are shown in Table 1 and Table 2, which respectively show the data during training and experiment. Among them, PPO1, PPO2, PPO3 and PPO4 correspond to the PPO algorithm under binary reward function, sparse reward function, dense reward function and staged reward function, respectively.

[0136] from Figure 3 (a) As can be seen, when using a binary reward function, the reward value remains close to 0 throughout the training process, and no tasks are successfully completed. The sparse reward function's reward value fluctuates significantly in the early stages of training (between 0 and 1000 training runs), indicating an unstable training process. The dense reward function's average reward value, while fluctuating slightly, is generally low. In contrast, the reward value of the staged reward function changes smoothly and shows a steady upward trend. After approximately 100 training runs, the reward value of the staged reward function exceeds that of the other reward functions, indicating faster training convergence. Figure 3 (b) As can be seen, the PPO algorithm using a phased reward function consistently maintains the highest success rate. The algorithm using a binary reward function consistently achieves a success rate of 0, indicating that no tasks are completed. The algorithm using a sparse reward function achieves a success rate greater than 0 after approximately 500 training epochs. The algorithm using a dense reward function achieves a success rate greater than 0 after approximately 1000 training epochs. The algorithm using a phased reward function achieves a success rate greater than 0 after approximately 100 training epochs. This comparison clearly demonstrates that the phased reward function accelerates the training process of deep reinforcement learning, leading to faster convergence and the rapid development of efficient strategies for trajectory planning.

[0137] As shown in Table 1, the success rates of different reward functions at each training stage varied significantly. Notably, the algorithm using the phased reward function demonstrated an overall performance advantage, not only maintaining the highest success rate across all training stages but also achieving a 27.4% success rate within the first 1,000 training runs. A comparative analysis revealed that the dense reward function, which frequently caused the seven-DOF manipulator to collide with obstacles in the environment, resulted in a generally lower success rate at each stage than both the sparse and phased reward functions. In particular, the phased reward function exhibited a higher success rate compared to the sparse reward function, further validating its effectiveness.

[0138] Table 1 Success rate of each stage in the training process

[0139]

[0140] Finally, we ran 200 experiments on the policies obtained by training the PPO algorithm 5000 times with various reward functions. The experimental results are shown in Table 2. The PPO1 row in the table shows poor training performance, with none of the 200 experiments reaching the target position, so the data is marked with a "-". The average number of steps for the phased reward function algorithm lies between the dense and sparse reward function algorithms, but it has the highest success rate, reaching 95.5%, and the shortest average path length.

[0141] Table 2 Experimental results

[0142]

[0143] We conducted 5,000 training runs using different experience replay mechanisms. By comparing the training data with the experimental results, we verified the superiority of the hierarchical experience replay mechanism. In the experiment, we compared multiple algorithms. PER-PPO stands for the PPO algorithm with prioritized experience replay, and HM-PPO stands for the PPO algorithm with hierarchical experience replay. To further investigate the impact of the sampling ratio of good and bad experience on algorithm performance, we designed the HM1-PPO, HM2-PPO, HM3-PPO, HM4-PPO, and HM5-PPO algorithms based on the HM-PPO algorithm. These algorithms differ in the proportion of samples drawn from the prioritized experience storage area: 20%, 50%, 80%, 90%, and 95%, respectively.

[0144] In order to evaluate the effectiveness of different experience replay mechanisms, we conducted a comparative analysis of the average reward value and success rate during the algorithm training process. In the experiment, PPO, PER-PPO, HM1-PPO, HM2-PPO, HM3-PPO, HM4-PPO, HM5-PPO and HM-PPO algorithms were used for 5000 trainings. All algorithms adopted the staged reward function proposed in the previous section. By comparing the data during the training process and the strategies obtained by training, the effectiveness of the designed hierarchical experience replay mechanism was verified. Table 3 shows the success rate of each algorithm at each stage of training. Figure 4 It shows the changes in average reward value and success rate of different algorithms during training.

[0145] Table 3 Success rate of each stage during training

[0146]

[0147] In Table 3, the success rates of HM1-PPO, HM2-PPO, HM3-PPO, and HM4-PPO algorithms gradually increase in the same training stage. Figure 4The results show that the reward values of the HM1-PPO algorithm are consistently negative, and the success rate is close to 0, indicating poor training results. However, the reward values and success rates of the HM2-PPO, HM3-PPO, and HM4-PPO algorithms gradually increase, indicating that the training results are gradually improving. This suggests that extracting more good experiences during experience replay helps improve the performance of the training strategy. However, in Table 3, the success rate of the HM5-PPO algorithm is lower than that of HM3-PPO and HM4-PPO at each stage. Figure 4 It also shows that the reward value and success rate of HM5-PPO increase more slowly. This shows that extracting too many good experience samples does not necessarily lead to better training results. Over-reliance on good experience samples may lead to overfitting, which in turn affects the generalization ability of the strategy.

[0148] Table 3 shows that during training cycles 3001 to 4000, the HM2-PPO algorithm achieved a success rate of 67.3%, lower than the success rate in the previous phase. This change suggests that the policy trained by the HM2-PPO algorithm is unstable, potentially performing poorly in certain phases and failing to achieve sustained optimization. In contrast, the HM-PPO algorithm consistently outperformed the HM1-PPO through HM5-PPO algorithms in each training phase, demonstrating its more stable training performance and stronger policy optimization capabilities.

[0149] Further, from Figure 4 The HM-PPO algorithm clearly outperforms HM1-PPO through HM5-PPO in training. Of all the algorithms, HM-PPO's reward value increases the fastest and, after a period of training, gradually stabilizes. As training progresses, HM-PPO's strategy is gradually optimized, and the number of robot arm control attempts decreases. Due to the decreasing number of control attempts during training, the reward value obtained by HM-PPO per training session decreases, resulting in the downward trend in average reward value shown in the graph.

[0150] exist Figure 4 In (b), the dark red curve represents the success rate of the HM-PPO algorithm, the dark blue curve represents the success rate of the PER-PPO algorithm, and the blue curve represents the success rate of the PPO algorithm. The figure shows that the improved HM-PPO algorithm converges faster. During the first 1000 training cycles, the success rates of the PPO and PER-PPO algorithms first increase, then decrease, and then increase again, while the success rate of the HM-PPO algorithm maintains a steady increase. Around the 400th training cycle, the success rate of the PER-PPO algorithm briefly exceeds that of the HM-PPO algorithm, but thereafter its success rate remains consistently lower than that of the HM-PPO algorithm. This demonstrates that the HM-PPO algorithm achieves superior training results and is more effective in controlling the movement of the seven-degree-of-freedom redundant manipulator to the target position.

[0151] Table 4 shows the results of 200 experiments after training for various algorithms. The HM1-PPO row in the table shows data marked with a "-" due to poor training performance, with none of the 200 experiments reaching the target location. Compared with the HM2-PPO, HM3-PPO, HM4-PPO, and HM5-PPO algorithms, the HM-PPO algorithm achieved a lower average reward, primarily due to the smaller number of steps in each experiment. However, the HM-PPO algorithm achieved the highest success rate, the shortest planned path, and the fewest steps, validating the effectiveness of the unbiased sampling strategy proposed in this paper. Comparing the PPO, PER-PPO, and HM-PPO algorithms, the HM-PPO algorithm achieved the highest success rate, the shortest path length, and the fewest steps, demonstrating the effectiveness of the designed hierarchical experience replay mechanism.

[0152] Table 4 Experimental results

[0153]

[0154] like Figure 5 Figure 1 shows the trajectory of a seven-DOF redundant manipulator controlled by the PPO2, PPO3, and PPO4 algorithms. The red solid line represents the trajectory of the seven-DOF redundant manipulator controlled by the PPO2 algorithm; the green solid line represents the trajectory of the seven-DOF redundant manipulator controlled by the PPO3 algorithm; and the blue solid line represents the trajectory of the seven-DOF redundant manipulator controlled by the PPO4 algorithm. The blue solid line is smoother and shorter, indicating that the PPO4 algorithm is better at controlling the seven-DOF redundant manipulator to its target point.

[0155] like Figure 6 Figure 1 shows the trajectories of the seven-DOF redundant manipulator controlled by the PPO, PER-PPO, and HM-PPO algorithms. The red solid line represents the trajectory of the seven-DOF redundant manipulator controlled by the PPO algorithm; the green solid line represents the trajectory of the seven-DOF redundant manipulator controlled by the PER-PPO algorithm; and the blue solid line represents the trajectory of the seven-DOF redundant manipulator controlled by the HM-PPO algorithm. The blue solid line is smoother, indicating that the HM-PPO algorithm is better at controlling the seven-DOF redundant manipulator to the target point.

[0156] Any matters not described in detail in this specification are prior art known to those skilled in the art. Although the above description of the present invention is based on specific embodiments to facilitate understanding of the present invention by those skilled in the art, it should be understood that the present invention is not limited to the scope of the specific embodiments. As long as various modifications are within the spirit and scope of the present invention as defined and determined by the appended claims, such modifications will be obvious to those skilled in the art, and all inventions and creations utilizing the concepts of the present invention are protected.

Claims

1. A seven-degree-of-freedom robot arm trajectory planning method based on improved deep reinforcement learning PPO, characterized in that: The method comprises the following steps: Step 1: Set the target pose of the end of the seven-degree-of-freedom redundant manipulator; Step 2: Obtain the position and posture of the end of the seven-degree-of-freedom redundant manipulator, and use the arm angle parameter method to perform inverse kinematics solution on the seven-degree-of-freedom redundant manipulator to obtain the current position of each joint of the manipulator; Step 3: Perform trajectory planning using the deep reinforcement learning PPO algorithm with a staged reward function and a hierarchical experience replay mechanism, and output the increments of each joint; Step 4: Add the joint increments obtained above to the current positions of the joints of the robotic arm to obtain the joint positions at the next moment for controlling the movement of the robotic arm. Step 5: Repeat steps 2, 3, and 4 until the target pose is reached.

2. The seven-degree-of-freedom robot arm trajectory planning method based on improved deep reinforcement learning PPO according to claim 1 is characterized in that: In the step 1, the Cartesian space target pose of the end of the seven-degree-of-freedom redundant manipulator is set as the end point pose of the end of the seven-degree-of-freedom redundant manipulator.

3. The seven-degree-of-freedom robot arm trajectory planning method based on improved deep reinforcement learning PPO according to claim 2 is characterized in that: In step 2, the position of the end of the seven-degree-of-freedom redundant manipulator is obtained, and the arm angle parameter method is used to perform inverse kinematics on the seven-degree-of-freedom redundant manipulator to obtain the current position of each joint of the manipulator .

4. The seven-degree-of-freedom robot arm trajectory planning method based on improved deep reinforcement learning PPO according to claim 3 is characterized in that: The specific operations of step 3 are: Set the input information, output information, reward function and experience replay mechanism of the deep reinforcement learning PPO algorithm; The input information is the state space, and the output information is the action space; the reward function is the feedback obtained after the agent performs the action; the experience replay mechanism adopts a hierarchical experience storage structure and an unbiased sampling strategy; The state space contains the environment information and the state information of the robot arm, which provides the basis for the decision-making and action execution of the intelligent agent; the position of the end effector , distance from the target location and whether there is a collision with an obstacle Let be the observed value, so the state space is defined as: ,in, It is a three-dimensional array representing the position of the end of the seven-degree-of-freedom manipulator relative to the base coordinate system. 、 and Position in the axial direction; Is a floating point number representing the distance of the robot end effector relative to the target position; Is a two-dimensional array that indicates whether the robot arm collides with an obstacle and which link collides with the obstacle; The action space It is defined as the increment of the rotation angle of each joint, that is, it is directly defined as the change in joint angle. The formula is as follows: , where arrive Respectively represent the increment of the rotation angle required for each joint; During the training process, the reward function defines the feedback obtained after the agent performs an action, which is used to help the agent adjust its strategy and maximize the cumulative reward; the sparse reward function The formula is as follows: , ,in, Indicates the distance between the current robot arm end effector and the target position; Indicates the distance from the target position at the previous moment; Indicates the target location to be set; represents the position of the end of the robotic arm at the t-th movement; Dense reward function The formula is as follows: ; A phased reward function is designed to gradually guide the agent to complete the goal through multiple stages. The phased reward function divides the trajectory planning task into two stages, including near-area trajectory planning and far-area trajectory planning. In the far-area, a sparse reward function is used as an incentive mechanism; in the near-area, a dense reward function encourages the robot arm to quickly reach the target position. The value is 0.5m. When the end of the robotic arm is more than 0.5m away from the target point, it belongs to the far area; when the end of the robotic arm is less than 0.5m away from the target point, it belongs to the near area. An adjustable coefficient is used. Implement different reward functions at different stages with adjustable coefficients The calculation formula is as follows: , stage-by-stage reward function The calculation formula is as follows: ,in, Indicates whether the end of the robotic arm is in the far area; Indicates whether the end of the robot arm is in the near area; represents transpose; Design a hierarchical experience replay mechanism, using a hierarchical experience storage structure and an unbiased sampling strategy; In the hierarchical experience storage structure, the agent distinguishes the value of experience by the reward value of the experience and divides it into high-value experience and low-value experience; high-value experience is stored in the priority experience storage area, and low-value experience is stored in the secondary experience storage area; The unbiased sampling strategy aims to balance the sampling ratio between the priority experience storage area and the secondary experience storage area. The unbiased sampling strategy formula is as follows: , ,in, Indicates the number of current training; Indicates the maximum number of training times set; Indicates the total number of experience samples extracted from the experience storage area; Indicates the number of experience samples extracted from the priority experience storage area; Indicates the number of experience samples extracted from the secondary experience storage area; The deep reinforcement learning PPO algorithm using a staged reward function and a hierarchical experience replay mechanism performs the following steps at each iteration of trajectory planning: Step 3.1: Initialize the policy network and value network ,set up , , , , , ; Step 3.2: For each iteration of do, when the agent interacts with the environment, , ; Step 3.3: When > 0, , when not > 0, then ; Step 3.4: For experience sampling, when : ,when , ,otherwise, ; Step 3.5: From and Extracted from the experience storage area and Experience samples; Step 3.6: For each experience extracted ,calculate , calculate the replacement loss ; Step 3.7: Accumulate losses ,in, is the entropy regularization term; Step 3.8: Update using the optimizer and To minimize ; in, Indicates the current number of training times; Indicates the total number of defined training times; Indicates the size of the priority experience storage area; Indicates the size of the secondary experience storage area; Indicates the number of experiences extracted from the priority experience storage area; Indicates the total number of experiences extracted from the two experience storage areas; A staged reward function is used.

5. The seven-degree-of-freedom robot arm trajectory planning method based on improved deep reinforcement learning PPO according to claim 4 is characterized in that: In step 4, the joint increments output by the improved deep reinforcement learning PPO algorithm are added to the current positions of the joints of the robotic arm to obtain the joint positions at the next moment for controlling the movement of the robotic arm, as follows: , , , , , , ,in, Indicates the position of each joint at the next moment when controlling the movement of the robotic arm.

Images