Precise path planning method and system for robotic arm based on back-to-back DDPG

Abstract

The present invention proposes a method and system for accurate path planning of a robotic arm based on back-to-back DDPG, which belongs to the technical field of charging piles, and includes the following steps: S1: the robotic arm includes several segments, each joint can control a segment of the robotic arm, and each joint constructs an independent local action network and a local value network; S2: construct a global value network, which is used to evaluate the impact of the actions of all joints on the overall state; S3: based on the first segment, the robot arm is composed of a plurality of segments, each joint can control a segment of the robot arm, and each joint constructs an independent local action network and a local value network; S4: construct a global value network, and the global value network is used to evaluate the impact of the actions of all joints on the overall state; S5: based on the first segment, the robot arm is composed of a plurality of segments, each joint can control a segment of the robot arm, and each joint constructs an independent local action network and a local value network; S6: based on the first segment, the robot arm is composed of a plurality of segments n After the joints are optimized, the control strategy and path planning are trained. n ‑1 joint, train each joint section by section, and finally obtain the overall control strategy and path planning of the robotic arm. When optimizing each joint strategy, the present invention takes into account its impact on the entire system, achieving the unity of local optimization and global optimization, which can not only improve the optimization effect of a single joint, but also effectively improve the path planning accuracy of the entire robotic arm system and the overall convergence speed of the model.

Description

Technical Field

[0001] The present invention belongs to the technical field of charging piles, and in particular relates to a method and system for precise path planning of a robotic arm based on a back-facing DDPG. Background Art

[0002] The automatic charging of the charging robot arm has become a research focus as an important means to improve the charging efficiency of electric vehicles, adapt to the automatic driving system, avoid the charging time too long, the overtime fee for not leaving the charging station in time after fully charged, and the safety hazards of manual charging. Among them, the accuracy of the robot arm trajectory planning is crucial to achieve the precise execution of the task. Trajectory planning takes into account environmental factors and constraints, avoids potential collision risks through reasonable path selection to ensure the safety of operation, and reduces the movement time and energy consumption of the robot arm, improving resource utilization and energy efficiency.

[0003] The patent with application number 202210501902.8 provides a robotic arm motion planning method based on deep reinforcement learning. It includes training a deep reinforcement learning network in different task scenarios in a simulation environment, and storing the experience data in success and failure experience replay pools. When the data of the two replay pools reaches the set value, the loss value and reward value are input into the neural network, and the weight is output; then data is extracted from the replay pool according to the weight to continue training the deep reinforcement learning network, and iterate until the network converges. Finally, the position and posture of the target object in the real environment are mapped to the simulation environment, and the trained network is input for motion planning to obtain the optimal path.

[0004] The above method may overfit the training data in complex tasks, limit the generalization ability of the model, and reduce its reliability in real-world applications. It also requires a large amount of training data and time to achieve good performance. For complex tasks and environments, the training time may be very long.

[0005] The patent with application number 202310975975.5 provides a multi-objective trajectory planning method and system, including objective functions and constraints; using B-spline interpolation to determine the objective function; converting the constraints into a finite number of control points; finding the Pareto optimal solution through enhanced NSGA-II to form the optimal trajectory.

[0006] The above method has high computational complexity and long running time when dealing with large-scale problems. It also ignores the autonomous learning and generalization capabilities of the robot arm and is prone to falling into local optimal solutions. Summary of the invention

[0007] In order to solve the above problems existing in the prior art, a method and system for precise path planning of a robotic arm based on back-to-back DDPG are provided.

[0008] The technical solution adopted by the present invention to solve its technical problem is:

[0009] This technical solution proposes a method for accurate path planning of a robotic arm based on back-to-back DDPG, including the following steps:

[0010] S1: The robot arm consists of several segments. Each joint can control a corresponding segment of the robot arm. Each joint constructs an independent local action network and local value network.

[0011] S2: Construct a global value network, which is used to evaluate the impact of all joint actions on the overall state;

[0012] S3: Based on n After the joints are optimized, the control strategy and path planning are trained. n -1 joint, train each joint section by section, and finally obtain the overall control strategy and path planning of the robotic arm.

[0013] Preferably, define n The local action network of each joint is , the local value network is , build a global value network ;

[0014] Build for each joint separately MDP Model, define n The state space of each joint is expressed as:

[0015] (1);

[0016] In the formula, For the n Joint to N The angle of the joints, For the n The length of the robot arm corresponding to the joints, is the target position, is the location of multiple obstacles;

[0017] No. n The motion space of the joint is the three-dimensional angle change of the joint, expressed as:

[0018] (2);

[0019] In the formula, For the robot arm x The pitch angle of the axis when it rotates up and down, with positive values ​​for upward rotation and negative values ​​for downward rotation; For the robot arm yThe yaw angle of the axis rotation, with negative values ​​for left rotation and positive values ​​for right rotation.

[0020] Preferably, n The reward of each joint is defined as the Pareto optimal form of its own control penalty, expressed as:

[0021] (3);

[0022] In the formula, For joints n The self-control penalty is expressed as the overall path accuracy of the robot and the individual control accuracy of the robot. is the gripper position of the robot arm, For the N The distance between the robot arm and the obstacle, is a preset weight parameter used to unify the order of magnitude;

[0023] The reward for the collaboration between the robots. When the current robot arm is not moving, i The self-control penalty of each joint is maximized by Obtain the Pareto equilibrium solution.

[0024] Preferably, the action is generated by a local action network , and get rewards , design the local action network loss function of the local action network , expressed as:

[0025] (4);

[0026] In the formula, t is the training iteration variable of the robot arm, is the policy gradient loss of the local action network, used to maximize Q value;

[0027] is the global state change penalty term, To balance the weight;

[0028] It is the comparison value between the control penalty of the current action and the average control penalty of all robotic arms that have made actions, and represents the contribution of the current action to the global optimum.

[0029] Preferably, a global value network is used for system supervision, and the global value network loss function is expressed as:

[0030] (5);

[0031] In the formula, is the global state, Valuing the global value network, The parameters are Global value network valuation, is the action taken by the local action network after state transition, is the discount factor.

[0032] Preferably, based on the obtained global value network loss function, the local value network loss function is obtained by the collaboration of the local value network and the global value network, which is expressed as:

[0033] (6);

[0034] In the formula, For the n Parameters of joints The value network valuation of After the state transfer n The local action network of each joint is based on parameters The generated action, To use n The value network valuation of each joint, To balance the weight.

[0035] Preferably, by monitoring the changes of the local action network loss function and the local value network loss function, when the changes in multiple training rounds are less than the set threshold, the local network is considered to have converged, and the local value network convergence judgment variable is expressed as:

[0036] (7);

[0037] In the formula, is the event function, Indicates that the local value network loss change is less than the threshold ,otherwise ;

[0038] Indicates that the global value network loss change is less than the threshold ,otherwise , To determine the time window size for value network convergence;

[0039] When both conditions are met within the time window , the local value network converges.

[0040] Preferably, the local action network convergence judgment variable expression is:

[0041] (8);

[0042] In the formula, is the preset local action network loss change threshold, is the event function, Indicates that the local action network loss change is less than , To determine the time window size for local action network convergence;

[0043] When the time window And when the local value network converges, the local action network converges.

[0044] When the local section n When the action network and value network of each joint converge, When n The joints converge.

[0045] Preferably, the training sequence is: starting from the end of the robotic arm, training the neural network of each joint section by section.

[0046] A robot arm precise path planning system based on back-facing DDPG, used to execute the robot arm precise path planning method based on back-facing DDPG, comprising:

[0047] The motion acquisition module is responsible for collecting the posture of each joint of the robotic arm, the contact force and position information of the charging socket;

[0048] Local evaluation module, used to evaluate the action and value of each robot joint;

[0049] The global evaluation module is used to evaluate the impact of all robot joint actions on the overall state;

[0050] The loss function calculation module calculates the control penalty of the current action and the average control penalty of all robotic arms that have made actions according to the posture of each joint, indicating the contribution of the current action to the global optimum.

[0051] The convergence judgment calculation module calculates the convergence degree of the robot arm joints by monitoring the changes in the local action network loss function and the local value network loss function;

[0052] The action execution module is responsible for executing the overall control strategy and path planning of the robotic arm.

[0053] Compared with the prior art, the present invention has the following advantages:

[0054] This application adopts a back-to-back training strategy, starting from the end of the robotic arm, to train the neural network of each joint section by section. After each section of the network converges, the neural network of the previous section of the joint is trained in turn, so that the path planning of the previous section of the joint is optimized based on the optimal solution of the next section of the joint. In this way, n The problem of simultaneous optimization of the paths of the multi-section robot arm is split into the optimization problem of a single-section robot arm, which significantly reduces the complexity of the optimization space and improves the convergence speed and computational efficiency of the model.

[0055] In the process of updating local network parameters, this application combines the errors of the local value network and the global value network. Through this collaborative mechanism, the unification of local optimization and global optimization is achieved, which not only improves the optimization effect of a single joint, but also improves the path planning accuracy of the entire robotic arm system, while effectively improving the overall convergence speed of the model.

[0056] This application collaboratively optimizes local networks and global networks to ensure that each joint network does not destroy the coordination performance of the entire system during independent optimization. This not only improves the independent performance of each joint, but also ensures the efficiency and coordination of the overall path planning and control of the robotic arm.

[0057] This application designs an efficient convergence judgment method by comprehensively considering the changes in the loss functions of the local network and the global network. This method can not only consider the impact of each joint strategy on the entire system when optimizing it, but also ensure the balance and coordination between local optimization and global optimization during the training process. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0059] Figure 1 It is the overall flow chart of the present invention;

[0060] Figure 2 It is a schematic diagram of the dynamic planning of the robot path in the present invention. DETAILED DESCRIPTION

[0061] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0062] like Figure 1-Figure 2 As shown, this embodiment proposes a method for accurate path planning of a robotic arm based on back-to-back DDPG, comprising the following steps:

[0063] S1: The robot arm consists of several segments. Each joint can control a corresponding segment of the robot arm. Each joint constructs an independent local action network and local value network.

[0064] The local action network and local value network are used to process the action and value evaluation of each robot segment, thereby achieving fine control of each joint of the robot;

[0065] S2: Build a global value network, which is used to evaluate the impact of all joint actions on the overall state, thereby optimizing global control;

[0066] S3: Based on n After the joints are optimized, the control strategy and path planning are trained. n -1 joint, train each joint section by section, and finally obtain the overall control strategy and path planning of the robotic arm.

[0067] The training sequence is: starting from the end of the robotic arm, the neural network of each joint is trained section by section to ensure that the action and value assessment of each joint can be effectively coordinated.

[0068] It also includes a back-to-back training convergence judgment method based on local-global collaboration, which is used to judge the convergence during the training process to ensure the effectiveness and stability of the training. By training the joint networks of the robotic arm section by section, fine control and global optimization of the robotic arm are achieved.

[0069] A robotic arm consists of n The first joint corresponds to the bottom of the robot arm. n The joints correspond to the head of the robot arm, and each joint can control a section of the robot arm. A local action network and a local value network are constructed for each joint. n The local action network of each joint is , the local value network is In order to evaluate the impact of all robot joint actions on the overall state, a global value network is constructed ;

[0070] Build for each joint separately MDP Model, define n The state space of each joint is expressed as:

[0071] (1);

[0072] In the formula, For the n Joint to N The angle of the joints, For the nThe length of the robot arm corresponding to the joints, is the target position, is the location of multiple obstacles;

[0073] No. n The motion space of the joint is the three-dimensional angle change of the joint, expressed as:

[0074] (2);

[0075] In the formula, For the robot arm x The pitch angle of the axis when it rotates up and down, with positive values ​​for upward rotation and negative values ​​for downward rotation; For the robot arm y The yaw angle of the axis rotation, with negative values ​​for left rotation and positive values ​​for right rotation.

[0076] No. n The reward of each joint is defined as the Pareto optimal form of its own control penalty, so that each robotic arm can maximize its own control effect under given conditions, avoiding the situation where the automatic control penalty of a single robotic arm is reduced, resulting in a reduction in the optimization effect of other robotic arms, expressed as:

[0077] (3);

[0078] In the formula, For joints n The self-control penalty is expressed as the overall path accuracy of the robot and the individual control accuracy of the robot. is the gripper position of the robot arm, For the N The distance between the robot arm and the obstacle, is a preset weight parameter used to unify the order of magnitude;

[0079] The reward for the collaboration between the robots. When the current robot arm is not moving, i The self-control penalty of each joint is maximized by Obtain the Pareto equilibrium solution to help the robotic arm n Decisions are made by comprehensively considering the global state and other robot actions to enhance coordination and avoid falling into individual optimality.

[0080] Traditional robot path planning methods need to consider the control and path planning issues of the entire robot system at the same time, which makes the training process very complicated, especially for multi-joint robot systems. In traditional methods, the entire system is trained at the same time, and the optimization of the previous joints will affect the training results of the subsequent joints, making it difficult for the system to converge. The parameter adjustment of each joint will affect the state of the entire system, and the training process is prone to fall into local optimal solutions or gradient vanishing / exploding problems.

[0081] Therefore, by adopting the back-to-back training strategy, starting from the end of the robot arm, the neural network of each joint is trained section by section. When each network converges, the neural network of the previous joint is trained in turn, so that the path planning of the previous joint is optimized based on the optimal solution of the next joint, thereby gradually ensuring the improvement of the overall performance. In this way, n The problem of simultaneous optimization of the paths of two joints of the robotic arm is split into the optimization problem of a single joint of the robotic arm. Each joint of the robotic arm only needs to solve the optimization problem based on the previous joint, which significantly reduces the complexity of the optimization space and improves the convergence speed and computational efficiency of the model.

[0082] First n Take the training of joints as an example, in the state Next, the action is generated by the local action network , and get rewards , in order to optimize the local action network, so that its output action can not only maximize Q The local action network loss function of the local action network is designed to take into account the global optimality. , expressed as:

[0083] (4);

[0084] In the formula, t is the training iteration variable of the robot arm, is the policy gradient loss of the local action network, used to maximize Q value;

[0085] is the global state change penalty term, To balance the weight;

[0086] It is the comparison value between the control penalty of the current action and the average control penalty of all robotic arms that have made actions, and represents the contribution of the current action to the global optimum.

[0087] To evaluate the action As for the impact on the overall performance of the robotic arm, the global value network is used for system supervision, and the loss function of the global value network is expressed as:

[0088] (5);

[0089] In the formula, is the global state, Valuing the global value network, The parameters are Global value network valuation, is the action taken by the local action network after state transition, is the discount factor.

[0090] Based on the obtained global value network loss function, the local value network loss function is obtained through the collaboration between the local value network and the global value network to update the local value network, which is expressed as:

[0091] (6);

[0092] In the formula, For the n Parameters of joints The value network valuation of After the state transfer n The local action network of each joint is based on parameters The generated action, To use n The value network valuation of each joint, To balance the weight.

[0093] According to the above loss function, independent training of the network corresponding to each joint can be achieved. In the process of updating the local network parameters, the errors of the local value network and the global value network are combined. Through this collaborative mechanism, the unification of local optimization and global optimization is achieved, which not only improves the optimization effect of a single joint, but also improves the path planning accuracy of the entire robotic arm system, while effectively improving the overall convergence speed of the model.

[0094] In the training process of robot path planning and control optimization, convergence judgment is a key link. The back-to-back training method can ensure the independent optimization of each joint network and the coordination performance of the overall system. By monitoring the changes in the local action network loss function and the local value network loss function, when the change is less than the set threshold within multiple training rounds, the local network is considered to have converged. The local value network convergence judgment variable is expressed as:

[0095] (7);

[0096] In the formula, is the event function, Indicates that the local value network loss change is less than the threshold ,otherwise ;

[0097] Indicates that the global value network loss change is less than the threshold ,otherwise , To determine the time window size for value network convergence;

[0098] When both conditions are met within the time window , the local value network converges.

[0099] The local action network convergence judgment variable expression is:

[0100] (8);

[0101] In the formula, is the preset local action network loss change threshold, is the event function, Indicates that the local action network loss change is less than , To determine the time window size for local action network convergence;

[0102] When the time window And when the local value network converges, the local action network converges.

[0103] When the local section n When the action network and value network of each joint converge, When n The joints converge.

[0104] By collaboratively optimizing the local network and the global network, we ensure that the coordination performance of the entire system is not destroyed during the independent optimization of each joint network. This not only improves the independent performance of each joint, but also ensures the efficiency and coordination of the overall path planning and control of the robotic arm.

[0105] By comprehensively considering the changes in the loss functions of the local network and the global network, an efficient convergence judgment method is designed. This method can not only consider the impact of each joint strategy on the entire system when optimizing it, but also ensure the balance and coordination between local optimization and global optimization during the training process. The optimal robotic arm path planning strategy can be obtained after the training of all joints is completed.

[0106] A robot arm precise path planning system based on back-facing DDPG, used to execute the robot arm precise path planning method based on back-facing DDPG, comprising:

[0107] The motion acquisition module is responsible for collecting the posture of each joint of the robotic arm, the contact force and position information of the charging socket;

[0108] Local evaluation module, used to evaluate the action and value of each robot joint;

[0109] The global evaluation module is used to evaluate the impact of all robot joint actions on the overall state;

[0110] The loss function calculation module calculates the control penalty of the current action and the average control penalty of all robotic arms that have made actions according to the posture of each joint, indicating the contribution of the current action to the global optimum.

[0111] The convergence judgment calculation module calculates the convergence degree of the robot arm joints by monitoring the changes in the local action network loss function and the local value network loss function;

[0112] The action execution module is responsible for executing the overall control strategy and path planning of the robotic arm.

[0113] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

Claims

1. A precise path planning method for a robotic arm based on back-to-back DDPG, characterized in that: The following steps are involved: S1: The robot arm consists of several segments. Each joint can control a corresponding segment of the robot arm. Each joint constructs an independent local action network and local value network. S2: Construct a global value network, which is used to evaluate the impact of all joint actions on the overall state; S3: Based on n After the joints are optimized, the control strategy and path planning are trained. n -1 joint, train each joint section by section, and finally get the overall control strategy and path planning of the robot arm; Definition n The local action network of each joint is , the local value network is , build a global value network ; Build for each joint separately MDP Model, define n The state space of each joint is expressed as: （1）； In the formula, For the n Joint to N The angle of the joints, For the n The length of the robot arm corresponding to the joints, is the target position, is the location of multiple obstacles; No. n The motion space of the joint is the three-dimensional angle change of the joint, expressed as: （2）； In the formula, For the robot arm x The pitch angle of the axis when it rotates up and down, with positive values ​​for upward rotation and negative values ​​for downward rotation; For the robot arm y The yaw angle of the axis rotation, with negative values ​​for left rotation and positive values ​​for right rotation.

2. The method for accurate path planning of a robotic arm based on back-to-back DDPG according to claim 1, characterized in that: No. n The reward of each joint is defined as the Pareto optimal form of its own control penalty, expressed as: （3）； In the formula, For joints n The self-control penalty is expressed as the overall path accuracy of the robot and the individual control accuracy of the robot. is the gripper position of the robot arm, For the N The distance between the robot arm and the obstacle, is a preset weight parameter used to unify the order of magnitude; The reward for the collaboration between the robots. When the current robot arm is not moving, i The self-control penalty of each joint is maximized by Obtain the Pareto equilibrium solution.

3. The method for accurate path planning of a robotic arm based on back-to-back DDPG according to claim 2, characterized in that: Generating actions from local action networks , and get rewards , design the local action network loss function of the local action network , expressed as: （4）； In the formula, t is the training iteration variable of the robot arm, is the policy gradient loss of the local action network, used to maximize Q value; is the global state change penalty term, To balance the weight; It is the comparison value between the control penalty of the current action and the average control penalty of all robotic arms that have made actions, and represents the contribution of the current action to the global optimum.

4. The method for accurate path planning of a robotic arm based on back-to-back DDPG according to claim 3, characterized in that: The global value network is used for system supervision, and the global value network loss function is expressed as: （5）； In the formula, is the global state, Valuing the global value network, The parameters are Global value network valuation, is the action taken by the local action network after state transition, is the discount factor.

5. The method for accurate path planning of a robotic arm based on back-to-back DDPG according to claim 4, characterized in that: Based on the obtained global value network loss function, the local value network loss function is obtained through the collaboration between the local value network and the global value network, which is expressed as: （6）； In the formula, For the n Parameters of joints The value network valuation of After the state transfer n The local action network of each joint is based on parameters The generated action, To use n The value network valuation of each joint, To balance the weight.

6. The method for accurate path planning of a robotic arm based on back-to-back DDPG according to claim 5, characterized in that: By monitoring the changes in the local action network loss function and the local value network loss function, when the changes in multiple training rounds are less than the set threshold, the local network is considered to have converged. The local value network convergence judgment variable is expressed as: （7）； In the formula, is the event function, Indicates that the local value network loss change is less than the threshold ,otherwise ; Indicates that the global value network loss change is less than the threshold ,otherwise , To determine the time window size for value network convergence; When both conditions are met within the time window , the local value network converges.

7. The method for accurate path planning of a robotic arm based on back-to-back DDPG according to claim 6, characterized in that: The local action network convergence judgment variable expression is: （8）； In the formula, is the preset local action network loss change threshold, is the event function, Indicates that the local action network loss change is less than , To determine the time window size for local action network convergence; When the time window And when the local value network converges, the local action network converges; When the local number n When the action network and value network of each joint converge, When n The joints converge.

8. A precise path planning system for a robotic arm based on back-to-back DDPG, characterized in that: The method for accurately planning a robot arm path based on back-to-back DDPG according to claim 7 comprises: The motion acquisition module is responsible for collecting the posture of each joint of the robotic arm, the contact force and position information of the charging socket; Local evaluation module, used to evaluate the action and value of each robot joint; The global evaluation module is used to evaluate the impact of all robot joint actions on the overall state; The loss function calculation module calculates the control penalty of the current action and the average control penalty of all robotic arms that have made actions according to the posture of each joint, and represents the contribution of the current action to the global optimum. The convergence judgment calculation module calculates the convergence degree of the robot arm joints by monitoring the changes in the local action network loss function and the local value network loss function; The action execution module is responsible for executing the overall control strategy and path planning of the robotic arm.

Images