Deep deterministic policy gradient based robot path planning method and system

Abstract

The application discloses a mechanical arm path planning method and system based on a deep deterministic policy gradient, and the method comprises the following steps: constructing a mechanical arm multi-task motion model, a deep deterministic policy gradient algorithm network model, a layered reward function of end tracking of the mechanical arm, and an experience sample pool; training the deep deterministic policy gradient algorithm network model based on the layered reward function and the weight of the experience sample, and obtaining the trained deep deterministic policy gradient algorithm network model; and deploying the trained deep deterministic policy gradient algorithm network model to the mechanical arm multi-task motion model, and performing path planning on the mechanical arm. By using the application, the utilization rate and the training speed of the mechanical arm training sample can be improved. The application can be widely applied to the technical field of robot mechanical arm path planning as a mechanical arm path planning method and system based on a deep deterministic policy gradient.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robot arm path planning, and particularly relates to a robot arm path planning method and system based on deep deterministic policy gradient. BACKGROUND

[0002] With the continuous development of artificial intelligence technology and engineering machinery demand, traditional artificial neural networks have achieved great success in pattern recognition, automatic control, signal processing, decision support and many other fields, and the New Generation of Artificial Intelligence Development Plan further emphasizes that with the advancement of industrialization and the arrival of the information age, intelligent robot industry as an important symbol to measure a country's scientific and technological innovation and high-end manufacturing level, its development has attracted more and more attention from countries around the world. The mechanical arm is the most common type of industrial robot, which appears in many fields such as logistics, medical treatment, construction, etc. The traditional mechanical arm operation often needs manual assistance, relying on the task instructions and operation methods provided by human beings. Such mechanical arms have many limitations, such as being unable to automatically complete tasks and requiring high technical requirements for operators. The development of deep learning and reinforcement learning provides a new idea for the intelligentization of mechanical arms. For example, some researchers establish artificial neural networks to enable the mechanical arm to learn and imitate features, which can help the mechanical arm to achieve complex movements. Some researchers apply reinforcement learning algorithms to the mechanical arm and train the mechanical arm as an intelligent agent. The trained mechanical arm can complete path planning or object grasping tasks. These researches provide a new direction for the intelligentization of mechanical arms. In recent years, the rise of intelligent robots, autonomous driving and artificial intelligence has brought great convenience to human life. In the research of these fields, path planning has always been one of the popular research contents. A suitable path planning algorithm can improve the efficiency of movement and reduce the time complexity. For the field of robots, the path planning algorithm provides a feasible, efficient and safe movement route for the robot. Although the existing mechanical arm grasping path planning using machine vision and the improvement of the traditional mechanical arm path planning scheme have greatly improved production efficiency, the algorithm effectiveness is not high, it is time-consuming and the algorithm complexity is high. Therefore, how to use effective visual clues to realize the autonomous obstacle avoidance and path planning of the mechanical arm is a problem to be solved. SUMMARY

[0003] In order to solve the above technical problems, the purpose of the present application is to provide a robot arm path planning method and system based on deep deterministic policy gradient, which can use a multi-layer reward mechanism to enhance the pointing action of the reward function during the training process of the robot arm, use the time difference error as the experience sample weight, and use the data structure characteristics of the binary heap for the experience pool replacement method, further improving the utilization rate and training speed of the robot arm training samples.

[0004] The first technical solution adopted by the present application is a mechanical arm path planning method based on a deep deterministic policy gradient, comprising the following steps:

[0005] Considering the three-dimensional space motion characteristics of the mechanical arm, a mechanical arm multi-task motion model is constructed, the mechanical arm multi-task motion including end tracking of the mechanical arm, pushing of the mechanical arm and grabbing of the mechanical arm;

[0006] Based on an Actor-Critic network structure, a deep deterministic policy gradient algorithm network model is established;

[0007] A preset reward rule is introduced to construct a hierarchical reward function for end tracking of the mechanical arm;

[0008] A priority experience replay mechanism is introduced to accumulate an experience sample pool and obtain the weight of the experience sample;

[0009] The deep deterministic policy gradient algorithm network model is trained based on the hierarchical reward function and the weight of the experience sample to obtain a trained deep deterministic policy gradient algorithm network model;

[0010] The trained deep deterministic policy gradient algorithm network model is deployed to the mechanical arm multi-task motion model to plan a path for the mechanical arm.

[0011] Further, the step of considering the three-dimensional space motion characteristics of the mechanical arm and constructing the mechanical arm multi-task motion model specifically comprises:

[0012] A three-dimensional motion model of the mechanical arm is drawn;

[0013] A virtual environment is initialized, and the three-dimensional motion model of the mechanical arm is set based on a coordinate system O-XYZ in the virtual environment;

[0014] The action space of the mechanical arm, the observation state information and the action information are specified, and the update step of the mechanical arm is set.

[0015] Further, the step of establishing the deep deterministic policy gradient algorithm network model based on the Actor-Critic network structure specifically comprises:

[0016] The deep deterministic policy gradient algorithm network model updates the deterministic policy parameters through the maximum cumulative reward value and outputs the probability of the action space of the mechanical arm, the deep deterministic policy gradient algorithm network model comprising a main network and a target network, the main network comprising an Actor network and a Critic network, and the target network comprising a Target Actor network and a Target Critic network;

[0017] The Actor network adopts a deterministic policy, integrates the input state to output a determined action information given the input state, and uses a deterministic policy gradient function as an updating method for the parameters of the Actor network;

[0018] The Critic network is an evaluation network, used to calculate Q value to evaluate the policy adopted by the Actor network, and uses a state-action value function as an updating method for the parameters of the Critic network;

[0019] The target network is used to calculate target Q value and assist in updating the parameters of the main network.

[0020] Further, the expression of the deep deterministic policy gradient algorithm network model is specifically as follows:

[0021]

[0022] In the above formula, μ ω represents a deterministic policy, J(·) represents a training target, the network is trained by maximizing J(·), r represents an expected value obtained by the network by executing the policy μ ω , s represents a state feedback by the environment, μ ω represents a distribution function of the state, represents integration of .

[0023] Further, the preset reward rule specifically includes:

[0024] The pre-action training task is sent to the multi-task motion model of the robot arm;

[0025] Considering the task completion of the robot arm, for the robot arm that completes the task, a preset target setting value is sent as a reward, and for the robot arm that does not complete the task, no reward value is sent;

[0026] Considering the staying step number of the robot arm completing the task, for the robot arm that stays in the target area for more than a preset step number, a one-time reward is sent, and the reward value is reduced by the consumed step number multiplied by a preset coefficient after completing the target, and for the robot arm that stays in the target area for less than a preset step number, no reward value is sent;

[0027] Considering the step number of the robot arm completing the task, the step number consumed by the robot arm in completing the pre-action training task is calculated, and a preset proportion value multiplied by the consumed step number is sent as a negative reward value;

[0028] Considering the distance between the end of the robot arm and the target, if the distance between the end of the robot arm and the target is less than a set value, a preset reward value is given to the robot arm.

[0029] Further, the step of introducing the priority experience replay mechanism and accumulating the experience sample pool specifically includes:

[0030] observing the state of the current robot arm and inputting it into the Actor network to obtain the corresponding robot arm action output result, and storing the robot arm action output result into the experience sample pool;

[0031] the robot arm multi-task motion model executes the robot arm action output result, updates the state of the robot arm to obtain the observation state of the robot arm at the next time, and calculates the corresponding reward value;

[0032] inputting the state of the current robot arm and the robot arm action output result into the Critic network to obtain the Q estimate value;

[0033] inputting the observation state of the robot arm at the next time into the Target Actor network to obtain the action output result of the robot arm at the next time;

[0034] inputting the observation state of the robot arm at the next time and the action output result of the robot arm at the next time into the Target Critic network to obtain the Q target value;

[0035] performing difference calculation processing on the Q estimate value and the Q target value to obtain the TD-error value, wherein the greater the TD-error value, the greater the potential of experience learning, and the higher the priority;

[0036] repeating the execution steps of the main network and the target network, accumulating the experience sample pool, and setting the TD-error value as the weight of the corresponding experience sample, wherein the data form stored in the experience sample pool is five tuples (s t ,a t ,r t ,s t+1 ,done), wherein s t represents the observation state of the current robot arm, a t represents the current robot arm action output result, r t represents the current corresponding reward value, s t+1 represents the observation state of the robot arm at the next time, and done represents the task completion condition.

[0037] Further, the calculation expression of the TD-error is:

[0038]

[0039] In the above formula, r t+1 represents the cumulative reward value at t+1, represents the TD-target time difference target, and δ tdenotes the time difference error at time t, g denotes a discount factor that balances the importance of current and future rewards, Q(s t ) denotes the value function estimate for the current state.

[0040] Further comprising storing and processing data in the experience sample pool based on the SumTree binary tree and combining the weights of the experience samples, specifically:

[0041] Introducing a minimum heap binary tree data structure, the minimum heap binary tree data structure includes leaf nodes and root nodes, the leaf nodes are used to store five-tuple data and corresponding TD-error values of experience samples, and the root nodes reserve the sum of TD-error values of their leaf nodes;

[0042] Uniformly sampling the nodes in the minimum heap binary tree data structure, and presetting a sampling value;

[0043] Comparing the sampling value with the left child node of the left root node in the minimum heap binary tree data structure, if the sampling value is greater than the left child node of the root node, then the sampling value is subtracted from the value of the left child node of the root node, and the difference value is the sampling value at the next moment, and enters the right root node;

[0044] Comparing the sampling value at the next moment with the left child node of the right root node until the comparison result is greater than the leaf node corresponding to the root node, and storing the five-tuple data of the experience sample into the leaf node corresponding to the root node.

[0045] Further comprising introducing a minimum heap binary tree data structure, and replacing data in the experience sample pool based on the weights of the experience samples, specifically:

[0046] When the experience pool reaches the maximum number of samples, the new experience sample will replace the experience sample with the smallest weight;

[0047] Delete the root node by the minimum heap, cover the experience sample of the root node with the experience sample of the last node in the binary tree, and perform top-down recursive adjustment, the adjustment target is to make the entire binary tree satisfy the data structure of the minimum heap again;

[0048] Insert the new experience sample behind the last node of the binary tree, and move the experience sample with the smallest weight to the root node by upward recursion.

[0049] The second technical scheme adopted by the present application is: a mechanical arm path planning system based on deep deterministic policy gradient, comprising:

[0050] The construction module is used for constructing a multi-task motion model of the mechanical arm, a deep deterministic policy gradient algorithm network model, a layered reward function of end tracking of the mechanical arm and an experience sample pool;

[0051] The training module is used for training the deep deterministic policy gradient algorithm network model based on the layered reward function and the weight of the experience sample, and obtaining a trained deep deterministic policy gradient algorithm network model.

[0052] The planning module is used for deploying the trained deep deterministic policy gradient algorithm network model to the multi-task motion model of the mechanical arm, and planning a path for the mechanical arm.

[0053] The method and system have the advantages that in the action output training process of the mechanical arm, the value of the action is evaluated by using the feedback of the environmental reward value, the cost of the action is learned, the policy network and the value function in the deep deterministic policy gradient algorithm are optimized, the next motion mode is adjusted, the reward value is maximized, the multi-level reward function is designed in view of the complex motion mode of the mechanical arm and the reward sparsity problem of the deep deterministic policy gradient algorithm, the reward value is given from multiple dimensions of the motion of the mechanical arm, the feedback mechanism of the environment is enhanced, the exploration ability of the mechanical arm in the early stage is improved, the training speed of the mechanical arm is accelerated, the experience replay mechanism is introduced in view of the problems that the mechanical arm has few opportunities to learn effective motion and the training efficiency is low in the simulation environment with sparse reward value and high exploration difficulty, the utilization rate of the experience sample is improved by repeatedly sampling data in the experience pool, the weight of the sample is determined by using a suitable measure, the weight of each experience sample in the experience sample pool is considered, a suitable storage mode is designed, the sampling probability of the high-weight experience sample is improved, and finally the data structure characteristics of the binary heap are used in the experience pool replacement method to retain the high-weight experience sample and remove the low-weight experience sample, and the utilization rate of the high-weight experience sample is improved. BRIEF DESCRIPTION OF DRAWINGS

[0054] Figure 1 is a step flowchart of the mechanical arm path planning method based on the deep deterministic policy gradient of the application;

[0055] Figure 2 is a structural block diagram of the mechanical arm path planning system based on the deep deterministic policy gradient of the application;

[0056] Figure 3 is a step flowchart of the existing intelligent mechanical arm path planning intelligent algorithm;

[0057] Figure 4 is an algorithm flowchart of the mechanical arm path planning based on the deep deterministic policy gradient of the application;

[0058] Figure 5is a structural schematic diagram of a SumTree binary tree of the present application;

[0059] Figure 6 is a schematic diagram of a binary tree data structure of a min-heap of the present application;

[0060] Figure 7 is a flowchart of a first round of empirical sample replacement of a specific embodiment of the present application;

[0061] Figure 8 is a flowchart of a second round of empirical sample replacement of a specific embodiment of the present application;

[0062] Figure 9 is a schematic diagram of the completion rate index results of a comparison between the PER algorithm based on the binary heap of the present application and existing algorithms;

[0063] Figure 10 is a schematic diagram of the average return index results of a comparison between the PER algorithm based on the binary heap of the present application and existing algorithms;

[0064] Figure 11 is a schematic diagram of the average step number index results of a comparison between the PER algorithm based on the binary heap of the present application and existing algorithms. DETAILED DESCRIPTION

[0065] The present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. For the step numbers in the following embodiments, they are only set for the convenience of description and explanation, and the order between the steps is not limited in any way, and the execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0066] In recent years, the rise of intelligent robots, autonomous driving and artificial intelligence and other fields has brought great convenience to human life. In the research of these fields, path planning problems have always been one of the popular research contents, and a suitable path planning algorithm can improve the efficiency of movement and reduce the time complexity. For the field of robots, the path planning algorithm provides a feasible, efficient and safe movement route for the robot. The algorithm of the path planning problem is mainly divided into two categories, one is the traditional path planning algorithm, mainly based on the characteristics of the graph and the sampling method, and the other is the intelligent algorithm, such as Figure 3As shown, the robot arm will often face complex multi-obstacle environment in actual work, so the general research on robot arm control mainly studies the acquisition of obstacle information and obstacle avoidance path planning from visual clues; Reliable visual clues can effectively guide the movement of the robot arm, so that it can work efficiently, safely and stably along the effective path; In 2016, <Safe Augmented Motion Planning for Flexible Surgical Manipulators Using Neurodynamics> Chen disclosed a scheme that can reduce unnecessary sweeping motion of flexible surgical manipulators. This method is a new three-dimensional neurodynamics model that can obtain safe augmented trajectories of the robot arm workspace under consideration of minimum scanning area; In 2017, <Motion Planning for Manipulators Using Robust Constraint Control> Zanchettin disclosed a new motion assignment and reactive execution algorithm, which combines traditional trajectory generation techniques and optimal strategies into a unified synchronous motion planning and control framework, effectively solving the motion trajectory planning problem of the robot arm; In 2018, <Dynamic Obstacle Avoidance for Manipulators Using Distance Computation and Discrete Detection> Han proposed Gilbert-Johnson-Keerthi algorithm, which improves the nearest distance calculation to solve the optimal path planning problem of the robot arm, thereby avoiding dynamic obstacles when the robot arm performs manufacturing tasks, and realizing the optimal path planning of the robot arm; In 2019, <Research on Tracking Recognition and Control Method Based on ROS> Wang proposed tracking recognition based on ROS, which uses vision to realize the function of autonomous grasping task of the robot, and the robot arm with machine vision recognizes and grasps, and completes the classification of objects; In 2020, <Three-Dimensional Pose Estimation for Object Detection in Remote Sensing Images> Liu disclosed that based on neural network, multiple feature points of an object are predicted to obtain the isomorphic transformation relationship between object coordinates and image coordinates, and the three-dimensional pose obtained thereby can accurately describe the three-dimensional position and attitude of the object; <Path Planning Research Based on Improved InformedRRT Method> Zhang disclosed that in order to solve the problems of poor purpose, slow convergence speed and low path optimization efficiency of Informed-RRT* algorithm in robot path planning, a method based on greedy algorithm and changing search object was proposed to optimize Informed-RRT* algorithm; Xu disclosed that a mobile robot arm is used to effectively and robustly collect objects stored in different trays, and a complete method for collision-free inverse kinematics solution based on reachability database is proposed to determine the complete set of feasible base positions, which is similar to a set of typical IK solutions, and is particularly useful when IK and collision checking are processed respectively;<Research on industrial robot positioning system based on machine vision> Ma published a set of industrial robot lithium battery current-carrying piece positioning system based on machine vision, using convolutional neural network to segment multi-angle depth image, then match the segmented depth image with the pre-set target object image and apply to the path planning of the robot arm; In 2021, <Indoor environment visual positioning using RGB-D image and improved local aggregation descriptor vector> Zhang proposed a visual positioning method for intelligent mobile devices based on RGB-D image, which calculates the pose of each image in the training set through feature extraction and description, image registration and pose map optimization, and then generates a local aggregation descriptor vector by clustering the training set and the query set in the image retrieval stage; <Research on mechanical arm grabbing based on GPRKRR combined model> Chen et al. In order to avoid the problems of time-consuming kinematics inverse and high computational complexity of visual system calibration in the method of autonomous grabbing of mechanical arm, a combined model of Gaussian process regression and kernel ridge regression is proposed for mechanical arm grabbing. Although the above-mentioned planning of mechanical arm grabbing path using machine vision and the improvement of traditional mechanical arm path planning scheme have greatly improved the production efficiency, the algorithm effectiveness is not high, the time-consuming is high, and the algorithm complexity is high.

[0067] In recent years, with the development of deep learning and reinforcement learning, more and more researchers apply deep reinforcement learning to path planning. Deep reinforcement learning algorithms are mainly divided into two categories: one is value function-based algorithm, including DQN (DeepQ-Network), Dueling DQN (Dueling Deep Q-Network), Double DQN (Double Deep Q-Network), and the other is policy-based method, including DPG (Deterministic Policy Gradient), DDPG (Deep Deterministic Policy Gradient), A3C (Asynchronous Advantage Actor Critic), PPO (Proximal Policy Optimization); Deep reinforcement learning can solve the problem of poor dynamic environment adaptation in conventional path planning algorithms. Deep reinforcement learning algorithm combines deep learning and reinforcement learning. Deep learning provides perception ability, and reinforcement learning provides decision-making ability, so that the agent can learn path selection independently through interaction with the environment and the corresponding feedback mechanism to achieve obstacle avoidance and path optimization; <Robot Exploration Strategy Based on Q-Learning Network> Tai discloses the path planning ability of a robot based on deep Q network algorithm in a maze environment through experiments. The experiment takes images as input and the motion direction of the robot as output. The robot can complete automatic obstacle avoidance and tracking functions by maximizing the reward value function; <Reinforcement Learning with Unsupervised Auxiliary Tasks> Jaderberg discloses that the A3C algorithm is enhanced by adding auxiliary task rewards, and then the agent based on the algorithm is tested through the trajectory in the maze environment to judge its performance. Experiments show that the agent under the algorithm can well perform path finding operation in the maze environment; <One Day to Learn Autonomous Driving> Kendall discloses that the DDPG algorithm is first used for autonomous driving. Through interaction with the surrounding environment, it automatically realizes path planning. Although the above path planning algorithms using deep reinforcement learning can improve the autonomous learning ability of the robot arm, reinforcement learning algorithms generally face the problem of sparse rewards. To solve this problem, the present application designs a reward function and an experience replay strategy for reinforcement learning to solve the problem of difficulty in obtaining reward values.

[0068] In summary, the present project takes the intelligent algorithm of the robot arm in the process of intelligent capture of biological colonies as the research object, designs a robot arm path planning algorithm based on deep deterministic policy gradient algorithm, analyzes the motion mode of the robot arm and the basic principles of deep deterministic policy gradient algorithm, and researches the reward function and experience replay strategy of deep deterministic policy gradient algorithm, specifically:

[0069] (1) A hierarchical reward function based on deep deterministic policy gradient algorithm is proposed. During the training process, the robot uses the feedback of the environmental reward value to evaluate the value of the action, learns the cost of the action, optimizes the policy network and value function in the deep deterministic policy gradient algorithm, adjusts the next motion mode, and realizes the maximization of the reward value. In view of the complex motion mode of the robot and the reward sparsity problem existing in the deep deterministic policy gradient algorithm, the project analyzes the joint angle and activity mode of the robot motion, combines the characteristics of the real three-dimensional space mechanical motion, constructs the end tracking of the robot, the pushing of the robot and the grasping of the robot and other multi-layer task motion models, designs a multi-level reward function, and gives the reward value from multiple dimensions of the robot motion, enhances the feedback mechanism of the environment, improves the exploration ability of the robot in the early stage, and speeds up the training speed of the robot;

[0070] (2) An experience replay strategy based on deep deterministic policy gradient algorithm is proposed. Firstly, in view of the problem that the robot has few opportunities to learn effective motion and the training efficiency is low in the simulation environment with sparse reward value and difficult exploration, the project introduces the experience replay mechanism, opens an experience sample pool, saves the training experience samples of single motion, and improves the utilization rate of experience samples by repeatedly sampling data in the experience pool. Secondly, in view of the problem that the random sampling strategy in the experience replay mechanism has low utilization rate of high-weight experience samples, the project uses a suitable measure to determine the weight of the samples, considers the weight size of each experience sample in the experience sample pool, designs a suitable storage method, and improves the sampling probability of high-weight experience samples. Finally, in view of the problem that the experience sample pool has an upper limit of capacity and the experience sample replacement algorithm has low efficiency, the project uses the data structure characteristics of binary heap in the experience pool replacement method, retains high-weight experience samples, removes low-weight experience samples, and improves the utilization rate of high-weight experience samples.

[0071] With reference to Figure 1 and Figure 4 , the application provides a robot path planning method based on deep deterministic policy gradient, which comprises the following steps:

[0072] S1, constructing a deep deterministic policy gradient algorithm;

[0073] Specifically, in the conventional actor-critic model, the strategy adopted is a random strategy. The random strategy outputs a probability distribution of the action space, that is, the probability of each action, and the probability interval is from 0 to 1, so that the method based on the random strategy samples the entire action space at each output. For a high-dimensional space, the random strategy mass sampling will affect the performance of the algorithm. To solve the above problems, the deterministic policy gradient algorithm is proposed. Unlike the random strategy, the deterministic policy algorithm outputs a probability of the action space, so there is no need to integrate the action space, and the efficiency is improved. The deterministic policy parameter is updated by the maximum cumulative reward value, and the mathematical expression of the objective function of the deterministic policy is as follows:

[0074]

[0075] In the above formula, μ ω represents the deterministic policy, J(·) represents the training target, the network is trained by maximizing J(·), r represents the expected value obtained by the network by executing the policy μ ω , s represents the state feedback of the environment, μ ω represents the distribution function of the state, represents the integral of ;

[0076] Further, the deterministic policy parameter is updated by the maximum cumulative reward value, and the probability of the robot arm action space is output, and the expression is as follows:

[0077]

[0078] In the above formula, represents the probability function distribution of the policy, γ k represents a discount factor for weighing the importance of current and future rewards, p represents the probability of transitioning from state s0 to state s after k time steps, and k represents the number of time points.

[0079] Therefore, the mathematical expression of the deterministic policy gradient is as follows:

[0080]

[0081] In the above formula, represents the gradient with respect to the policy parameter ω, represents the gradient with respect to the action a, μ ω (s) represents the action a taken by the policy ω in the state s, Q μ (s,a) represents the state-action value function, and a represents the action.

[0082] In the Actor-Critic structure, the state-action value function is used as a method for updating the neural network parameters of the Critic, and the deterministic policy gradient function is used as a method for updating the neural network parameters of the Actor. Such a structure is similar to Q-learning, which uses the TD-error of the value function to update the network parameters, but differs from Q-learning in that the output of the action uses a greedy algorithm, while the output of the action here uses a deterministic policy gradient. The mathematical expression of the TD-error of the value function is as follows:

[0083]

[0084] In the above formula, δ t represents the time difference error at time t, r(s, a) represents the reward function for state s and action a, Q(s', μ ω (s')) represents the value function at time t+1, and Q(s, μ ω (s)) represents the value function at time t.

[0085] The Actor network is designed based on a deterministic policy, so the output of the action is deterministic. Compared with the policy gradient algorithm, the deterministic policy gradient algorithm does not need to integrate the action, but only needs to integrate the state, so the sampling of the action is reduced, and the efficiency is improved. The mathematical expression of the deterministic policy gradient used by the Actor network is as follows:

[0086]

[0087] In the above formula, J β (μ ω ) represents the performance index of the policy μ ω , which is used to measure the goodness of the policy and is used as an optimization target to adjust the policy parameters.

[0088] The above solving formula makes the output of the action space of the agent not need the update of the action-value function, and can guarantee that the algorithm reaches local convergence.

[0089] S2, hierarchical reward function based on deep deterministic policy gradient algorithm;

[0090] Specifically, in the reinforcement learning algorithm, the agent always judges the pros and cons of the action by using the feedback of the environment reward value, so as to continuously optimize the policy network or value function in the reinforcement learning. A suitable and excellent reward function can greatly improve the training speed of the agent and the effect degree of the reinforcement learning algorithm. At the beginning of the agent training, due to the lack of prior knowledge, the reward value can only be obtained by continuous exploration. The action output by the reinforcement learning algorithm acts on the environment, and the environment feedbacks the value evaluation of this action, so that the agent learns the value of the action and outputs the next action, so as to realize the movement of the agent towards the direction of maximizing the reward value. Therefore, in the reinforcement learning algorithm, a suitable reward function is particularly important.

[0091] For training the motion control of the robot arm using the deep reinforcement learning algorithm, a suitable reward function needs to be set first. In the training process, the reward value of the reward function is obtained, the environment is observed, and the joint angle and mode of the next motion of the robot arm are adjusted for obtaining the maximum reward value until the expected training purpose is achieved. The motion of the robot arm is more complex than that of the ordinary agent. First, the robot arm is a multi-joint robot, and the motion mode of the robot arm needs to be considered when building the simulation environment for training the robot arm. Secondly, in the actual application, the motion of the robot arm is a three-dimensional motion, and the characteristics of the three-dimensional motion need to be considered when setting the reward function. Finally, the robot arm may be divided into multiple tasks in the actual application, including end tracking of the robot arm, pushing of the robot arm and grasping of the robot arm, and the reward function should be different for different tasks.

[0092] In the setting of the reward function of the reinforcement learning algorithm, the completion of the task by the agent is generally used as a condition to set, such as assigning a positive reward value to the agent for completing a task. In order to improve the efficiency of the agent, a negative reward value is often provided for the motion steps of the agent. This form of reward function is commonly used and effective in simple simulation environments and single task objectives. However, for the reward function of the deep reinforcement learning algorithm for the robot arm, this form cannot achieve good training effect, which is due to the fact that the robot arm moves in a three-dimensional space. If only the reward is assigned to the task completion state, it will result in sparse feedback, which is a relatively tricky problem in reinforcement learning. In view of the above problems, the layered reward function for end tracking motion of the robot arm is proposed by fully considering the motion mode and characteristics of the robot arm. The layered reward function mainly divides the reward function into multiple levels, so that the end of the robot arm can complete the task in as few steps as possible under the incentive of the multi-level reward function.

[0093] S21, reward rule;

[0094] Specifically, first, a reward value is given for the task completion degree, a reward value of 1 is given when the task is completed, and no reward value is given when the task is not completed, and the expression is as follows:

[0095]

[0096] In the above formula, target represents the task completion condition;

[0097] At the same time, in order to ensure that the robot arm can realize the target, when the end of the robot arm stays in the target area for more than 50 steps, the algorithm will give a reward value of 200 once, in order to improve the probability of the robot arm completing the target in the early training, the consumed steps will be subtracted after completing the target, and a coefficient of 0.5 is multiplied, and the reward value is as follows:

[0098]

[0099] In the above formula, steps represent the number of steps required for the robot arm to complete the target, and goal represents the number of steps that the end of the robot arm stays in the target area;

[0100] The above-mentioned method for ensuring that the robot arm can realize the target can prevent the robot arm from accidentally touching the target area and completing the task during training, but at this time the robot arm actually cannot realize the content of the task, so the present application sets a preset stay step number, only when the stay exceeds the predetermined step number, the robot arm can be considered to complete the task, which can effectively avoid the task completion condition caused by accidental touching of the robot arm;

[0101] Secondly, in order to stimulate the training efficiency of the robot arm, the movement cost of the robot arm will be reflected, when the number of steps of the robot arm increases by one, a certain negative reward value will be given, when it is closer to the maximum number of steps, the penalty value is larger, which ensures that the robot arm realizes the target in the least number of steps, wherein the coefficient -0.01 is determined according to the maximum number of steps of each set of the robot arm in the simulation environment and the reward value after completing the task, and the reward value expression is as follows:

[0102] r3=-0.01×steps

[0103] Finally, in order to improve the training speed of the robot arm, the reward function needs to set a positive reward value for the movement of the robot arm. In two-dimensional space, the coordinate information of the target is unchanged, for the movement of the robot arm, its coordinates (X B ,Y B ) only need to reach the X O coordinate or Y O coordinate of the target to give it a positive reward, so that the robot arm learns the orientation information of the target, and the mathematical expression is as follows:

[0104]

[0105] In the above formula, the r4 reward function fully considers the distance between the end of the robot arm and the target, the X O axis direction of the end of the robot arm reaching the target and the reward value of the Y I axis direction of the end of the robot arm reaching the target. Such a reward function can ensure that the end of the robot arm moves towards the target, and can also guide the robot arm to the target through the reward in the X O axis and Y O axis. 0.5 is set according to the reward of 1 for reaching the target, so that after reaching the target area, the robot arm will obtain rewards in two directions, and the total value is consistent with the target completion reward.

[0106] The reward function of the final deep reinforcement learning algorithm is obtained by compounding the reward functions of the above levels, and the mathematical expression is as follows:

[0107] r = r1 + r2 + r3 + r4

[0108] In the above formula, r represents the total reward function, and r1, r2, r3 and r4 represent the conditional rules of different reward functions, respectively.

[0109] S3, an experience replay strategy based on a deep deterministic policy gradient algorithm.

[0110] Specifically, in a simulation environment based on reinforcement learning, an agent mainly trains a policy by obtaining a reward value of each action. For a simulation environment with simple scenes or reasonable reward values, the agent can quickly learn the target and task. However, for a simulation environment with sparse rewards and high exploration difficulty, the agent has few opportunities to learn correct operations, and the training efficiency is low. If in some extreme simulation environment, the agent will only get a reward when it has a correct operation in the final state, which is extremely unfavorable for training. To solve this problem, the off-policy based reinforcement learning algorithm introduces an experience replay mechanism. The experience replay mechanism stores each training experience in an experience pool, and then repeatedly uses data in the experience pool for training, thereby improving sample utilization. The present application can solve the problem of sparse rewards by using experience replay, increasing the sampling probability of experiences with larger weights, strengthening the repeated training of important samples by the agent, and improving the utilization frequency of past experiences.

[0111] The off-policy based reinforcement learning algorithm is a learning process that can improve a certain policy through learning instead of improving on the basis of an existing policy, in the off-policy algorithm, the agent learns how to evaluate a set of different policies, and can automatically adjust the policy over time, unlike the on-policy algorithm, the off-policy algorithm does not need to maintain a specific policy applied to the problem in its learning process, common off-policy reinforcement learning algorithms include Q-learning and the Actor-Critic method in deep reinforcement learning;

[0112] The data stored in the initial experience pool is consistent, without weight size and priority, and the playback experience samples are obtained in a random manner, the equal-probability playback manner only repeatedly trains the experience, and the effect of the experience sample with great influence cannot be embodied, which makes the network training efficiency low and the training effect on the policy network weak, in view of the problem, the priority experience playback (PER) adopts a suitable measurement to determine the weight of the experience sample, and the weight of each experience in the experience pool is considered, so that the probability of adoption of the experience with greater weight is improved;

[0113] The main core of the PER algorithm is to define the weight of each experience, which considers that the size of the experience weight should be determined by how much the agent can learn from the experience, although this cannot be intuitively represented, but in the reinforcement learning of the value function, the TD-error is often used to update the value function, the TD-error is used to predict the next state from the current state, and the size of the TD-error can represent the difference between the current value function and the target value function, for an experience, the greater the TD-error represents the greater the learning potential of the experience, and the higher the priority, if the TD-error is smaller, it means that the experience is more ordinary, and when the experience is stored in the experience pool, its priority should be reduced, for the simulation environment of the application, the experience sample is defined as a five-tuple (s t ,a t ,r t ,s t+1 ,done), wherein s t represents the observation state of the current robot arm, a t represents the action output result of the current robot arm, r t represents the corresponding reward value, s t+1 represents the observation state of the robot arm at the next moment, and done represents the task completion condition;

[0114] Here, TD-error refers to Temporal Difference error, a crucial metric in reinforcement learning's Temporal Difference (TD) algorithm. It represents the difference between the actual reward and the predicted reward at the current time step. The TD algorithm continuously updates the estimated value of the value function to progressively approximate the true value function. At each time step, the TD algorithm estimates the current reward based on the current state and the value of the estimated value function, then compares it with the actual reward to calculate the TD-error.

[0115] Furthermore, the mathematical expression for TD-error is as follows:

[0116]

[0117] In the above formula, r t+1 This represents the cumulative reward value at time t+1. δ represents the TD-target time-series differential target. t Q(s) represents the time difference error at time t, γ represents the discount factor used to balance the importance of current and future rewards. t ) represents the estimated value of the function for the current state.

[0118] Here, TD-target refers to Temporal Difference target, a crucial concept in reinforcement learning's TD algorithm. The TD algorithm approximates the true value function by continuously updating its estimated value. At each time step, the TD algorithm estimates the current reward based on the current state and the value of the estimated value function, compares it to the true reward, calculates the TD-error, and then uses the TD-error to update the value of the estimated value function. TD-target is the objective value used by the TD algorithm when updating the estimated value function, typically defined as the reward at the current time step plus a discount factor multiplied by the estimated value of the next state.

[0119] S31. Store and process the data in the experience sample pool based on the SumTree binary tree and the weights of the experience samples.

[0120] Specifically, after determining the method for calculating the weights of the empirical samples, the PER algorithm stores the samples using a SumTree binary tree, as shown in the image. Figure 5 As shown, in a SumTree binary tree, leaf nodes store experience samples. Besides storing the quintuple of the experience sample, each leaf node also needs to carry its priority, such as... Figure 5The values in the gray nodes, and other nodes do not store experience samples, only keep the sum of the values of its child nodes;

[0121] The PER algorithm adopts the process to Figure 5 For example, first, PER will be through the (0, 38) uniform sampling, if the sampling value is 25, will start from the root node first on the left child node comparison, because 25 is greater than 15, will enter to the right child node, then subtract the left tree total sum 15, this time the sampling value left 10, the root node becomes 23, with 23 as the root node, continue to compare the size of the left child node and the sampling value, because 10 is greater than 5, then select the right child node 18 as the next round of root node, sampling value is 5, finally because 18 is a leaf node, this time the leaf node on the storage experience sample five tuple, will be the five tuple as the final training sample, this storage mode can guarantee the weight of the greater experience sample, in the experience pool sampling is selected probability is greater, because the weight of the experience sample is greater, it represents the selection interval is greater. For uniform sampling experience pool each value is selected probability is the same, the interval size of experience sample represents the range of value selection, and the range of greater experience sample, the probability of being selected is greater.

[0122] S32, introduce the binary tree data structure of the minimum heap, based on the weight of experience sample to replace the data in the experience sample pool;

[0123] Specifically, the data structure of the minimum heap is in the form of binary tree, and the sorting method only needs to meet the head node is the minimum value, so the time complexity of its insertion and deletion operation is O (log n), better than the general sorting method, in the SumTree experience sample replacement, if using the first in first out way, may lead to the weight of the experience sample is replaced. The binary heap method ensures that the weight of the experience sample is replaced every time, which is more conducive to the training of the network;

[0124] The PER algorithm realizes the priority playback of the experience samples with large weights in the experience pool by using the TD-error weight calculation method and the SumTree binary tree sample storage method, but does not limit the size of the experience pool, and adopts a full retention form. Although the diversity of the experience is ensured, the memory and the calculation efficiency are undoubtedly affected for a large number of training samples. The traditional solution is to set an upper limit for the experience pool. For the experience pool that reaches the upper limit, the experience samples need to be replaced, and the new experience samples replace the old experience samples. The replacement method generally adopts the First-In-First-Out (FIFO) idea, and the replacement is sequentially performed according to the time sequence of entering the experience pool. This way can store the latest experience samples in the experience pool, and ensures that each experience sample gets the same storage time. However, the weight of each experience sample is different. The experience samples with good learning effect may be replaced by the first-in-first-out algorithm, which is actually not conducive to training. The experience samples with good learning effect should be kept in the experience pool as much as possible, and the experience samples with small weights should be replaced by the latest experience samples, so that the experience samples in the experience pool are better. In view of this problem, the present application introduces the data structure of the minimum heap, and uses an array to maintain a minimum heap according to the weight size. The minimum heap is a complete binary tree, the value of the root node is smaller than the values of the left and right child nodes, and the root node maintains the minimum value of the entire sample space. The binary tree is as shown in Figure 6 ;

[0125] Referring to Figure 7 and Figure 8 , each node represents the weight of the experience sample, and the index of the experience sample is also stored. The root node is the minimum value in the sample space. When the experience pool reaches the maximum number of samples, the new experience sample will replace the experience sample with the smallest weight. First, the root node is deleted by using the minimum heap, and the experience sample of the root node is covered by the experience sample of the last node in the binary tree. Then, the whole binary tree is adjusted from top to bottom recursively, so that the whole binary tree satisfies the data structure of the minimum heap again. After the adjustment, the experience sample with the lowest weight is finally removed. In the next step, the new experience sample is inserted after the last node of the binary tree, and the experience sample with the smallest weight is moved to the root node by upward recursion. The gray nodes are the newly inserted experience samples, Figure 7 and Figure 8 show a round of experience sample replacement process. According to the characteristics of the minimum heap, the replacement efficiency is higher than that of the ordinary sorting method, and the data of the experience samples with large weights are retained during the replacement process, and the experience samples with small weights are replaced by the latest experience samples.

[0126] Further, the PER algorithm based on the binary heap is simulated and compared with the existing algorithm under three different indicators, as shown in Figure 9 、 Figure 10 and Figure 11 , it is obvious that the PER algorithm based on the binary heap is obviously better than the other two methods under the three different indicators. When the experience pool is full, the PER algorithm based on the binary heap can retain experience samples with larger weights when replacing the experience samples, so that the deep deterministic policy algorithm can have a greater probability of sampling experience samples with large weights when updating the actor network and critic network parameters from the experience pool. The experience samples with small weights are replaced by new experience samples through the replacement method, which can maximize the utilization of experience samples with high weights. The training effect of the PER algorithm based on FIFO is obviously better than that of the random sampling algorithm, which also proves that the experience samples sampled by the experience replay strategy using TD-error as the weight standard are more beneficial to network training. Among them, the average reward value of the PER algorithm based on FIFO in the interval from 2000 to 3000 fluctuates greatly and is higher than that of the PER algorithm based on the binary heap. According to the step three of the PER algorithm principle, although the PER algorithm classifies experience samples according to the TD-error standard and stores the samples in the SumTree binary tree, there is still a probability of sampling experience samples with small weights in the sampling process, which will produce certain randomness in the experiment. This is a normal situation. From Figure 9 、 Figure 10 and Figure 11 , it can be seen that the priority experience replay strategy based on the binary heap proposed by the present application has a good effect on the deep deterministic policy algorithm, can improve the utilization rate of training samples of the mechanical arm, and improve the training speed.

[0127] Referring to Figure 2 , the mechanical arm path planning system based on the deep deterministic policy gradient comprises:

[0128] A construction module is configured to construct a mechanical arm multi-task motion model, a deep deterministic policy gradient algorithm network model, an end tracking hierarchical reward function of the mechanical arm, and an experience sample pool.

[0129] A training module is configured to train the deep deterministic policy gradient algorithm network model based on the hierarchical reward function and the weight of the experience sample, and obtain the trained deep deterministic policy gradient algorithm network model.

[0130] A planning module is configured to deploy the trained deep deterministic policy gradient algorithm network model to the mechanical arm multi-task motion model, and plan a path for the mechanical arm.

[0131] The content in the method embodiments is applicable to the system embodiments. The system embodiments specifically implement the same functions as the method embodiments, and achieve the same beneficial effects as the method embodiments.

[0132] The above is a specific description of the preferred embodiments of the application. However, the application is not limited to the embodiments described above. Those skilled in the art can make various equivalent modifications or replacements without departing from the spirit of the application. These equivalent modifications or replacements are all included in the scope defined by the claims of the present application.

Claims

1. A method for robot arm path planning based on deep deterministic policy gradient, characterized in that, The method comprises the following steps: considering the three-dimensional space motion characteristics of the mechanical arm, a multi-task motion model of the mechanical arm is constructed, the multi-task motion of the mechanical arm includes end tracking of the mechanical arm, pushing of the mechanical arm and grabbing of the mechanical arm; based on an Actor-Critic network structure, a deep deterministic policy gradient algorithm network model is established; a preset reward rule is introduced, and a hierarchical reward function for end tracking of the mechanical arm is constructed; a priority experience replay mechanism is introduced, an experience sample pool is accumulated, and the weight of the experience sample is obtained; the deep deterministic policy gradient algorithm network model is trained based on the hierarchical reward function and the weight of the experience sample, and a trained deep deterministic policy gradient algorithm network model is obtained; the trained deep deterministic policy gradient algorithm network model is deployed to the multi-task motion model of the mechanical arm, and path planning is performed on the mechanical arm; the step of establishing the deep deterministic policy gradient algorithm network model based on the Actor-Critic network structure specifically comprises: the deep deterministic policy gradient algorithm network model updates the deterministic policy parameters through the maximum cumulative reward value and outputs the probability of the mechanical arm action space, the deep deterministic policy gradient algorithm network model comprises a main network and a target network, the main network comprises an Actor network and a Critic network, and the target network comprises a Target Actor network and a Target Critic network; the Actor network adopts a deterministic policy, integrates the input state to output a determined action information, and takes the deterministic policy gradient function as an updating method of the Actor network parameters; the Critic network is an evaluation network, which is used for calculating Q value to evaluate the policy adopted by the Actor network, and takes the state-action value function as an updating method of the Critic network parameters; the target network is used for calculating target Q value and assisting in updating the parameters of the main network; the step of introducing the priority experience replay mechanism and accumulating the experience sample pool specifically comprises: the state of the current mechanical arm is observed and input to the Actor network to obtain the corresponding mechanical arm action output result, and the mechanical arm action output result is stored in the experience sample pool; the mechanical arm action output result is executed by the multi-task motion model of the mechanical arm, the state of the mechanical arm is updated to obtain the observation state of the mechanical arm at the next moment, and the corresponding reward value is calculated; the state of the current mechanical arm and the mechanical arm action output result are input to the Critic network to obtain the Q estimation value; the observation state of the mechanical arm at the next moment is input to the Target Actor network to obtain the mechanical arm action output result at the next moment; the observation state of the mechanical arm at the next moment and the mechanical arm action output result at the next moment are input to the Target Critic network to obtain the Q target value; the Q estimation value and the Q target value are calculated by difference to obtain the TD-error value, wherein the greater the TD-error value, the greater the potential of experience learning, and the higher the priority; Circulating the execution steps of the above main network and target network, accumulating experience sample pool, and setting TD-error value as the weight of corresponding experience sample, the data stored in the experience sample pool is in the form of five-tuple wherein, represents the observation state of the current mechanical arm, represents the action output result of the current mechanical arm, represents the current corresponding reward value, represents the observation state of the mechanical arm at the next moment, represents the task completion condition; the calculation expression of the TD-error is: In the above formula, denotes the cumulative reward value at time t, denotes the TD-target temporal difference target, denotes the time difference error at time t, denotes the discount factor, which balances the importance of current and future rewards, denotes the value function estimate for the current state.

2. The method of claim 1, wherein, The step of considering the three-dimensional space motion characteristics of the mechanical arm and constructing a mechanical arm multi-task motion model specifically includes: drawing a three-dimensional motion model of the mechanical arm; initializing a virtual environment and setting the three-dimensional motion model of the mechanical arm based on a coordinate system O-XYZ in the virtual environment; defining the action space, observation state information and action information of the mechanical arm, and setting the update step of the mechanical arm.

3. The method of claim 2, wherein, The expression of the deep deterministic policy gradient algorithm network model is specifically as follows: In the above formula, denotes a deterministic policy, denotes a training objective, which maximizes the network is trained, denotes the expected value obtained by the network by executing the policy afterward, denotes the state of the environment feedback, denotes a distribution function of the state, denotes the integral over the state.

4. The method of claim 3, wherein, The preset reward rule specifically includes: pre-releasing an action training task to the mechanical arm multi-task motion model; considering the task completion of the mechanical arm, for the mechanical arm that completes the task, sending a preset target setting value reward, and for the mechanical arm that does not complete the task, not sending a reward value; considering the number of steps of the mechanical arm staying in the target area, for the mechanical arm that stays in the target area for more than a preset number of steps, sending a one-time reward minus the consumed steps multiplied by a preset coefficient reward value, and for the mechanical arm that stays in the target area for less than a preset number of steps, not sending a reward value; considering the number of steps of the mechanical arm completing the task, calculating the number of steps consumed by the mechanical arm in completing the pre-released action training task, and sending a preset proportion value multiplied by the negative reward value of the consumed steps; considering the distance between the end of the mechanical arm and the target, if the distance between the end of the mechanical arm and the target is less than a set value, a preset reward value is given to the mechanical arm.

5. The method of claim 4, wherein, Further comprising storing and processing data in the experience sample pool based on the SumTree binary tree and combining the weights of the experience samples, specifically: introducing a minimum heap binary tree data structure, the minimum heap binary tree data structure includes leaf nodes and root nodes, the leaf nodes are used to store five-tuple data of experience samples and corresponding TD-error values, and the root nodes reserve the sum of TD-error values of their leaf nodes; uniformly sampling the nodes in the minimum heap binary tree data structure, and presetting a sampling value; comparing the sampling value with the left child node of the left root node in the minimum heap binary tree data structure, if the sampling value is greater than the left child node of the root node, then the sampling value is subtracted from the value of the left child node of the root node, and the difference value is the sampling value at the next moment, and enters the right root node; comparing the sampling value at the next moment with the left child node of the right root node until the comparison result is greater than the leaf node corresponding to the root node, and storing the five-tuple data of the experience sample into the leaf node corresponding to the root node.

6. The method of claim 5, wherein, Further comprising introducing a minimum heap binary tree data structure, and replacing data in the experience sample pool based on the weights of the experience samples, specifically: when the experience pool reaches the maximum number of samples, the new experience sample will replace the experience sample with the smallest weight; deleting the root node by the minimum heap, covering the experience sample of the root node with the experience sample of the last node in the binary tree, and recursively adjusting from top to bottom, and the adjustment target is to make the entire binary tree satisfy the data structure of the minimum heap again; After inserting the new experience sample behind the last node of the binary tree, the experience sample with the minimum weight is moved to the root node by recursive upward.

7. A deep deterministic policy gradient based robotic arm path planning system for implementing the deep deterministic policy gradient based robotic arm path planning method according to any one of claims 1-6, characterized in that, The method comprises the following modules: A construction module is configured to construct a multi-task motion model of a robot arm, a deep deterministic policy gradient algorithm network model, a layered reward function for end tracking of the robot arm, and an experience sample pool; A training module is configured to train the deep deterministic policy gradient algorithm network model based on the layered reward function and the weight of the experience sample, and obtain a trained deep deterministic policy gradient algorithm network model; A planning module is configured to deploy the trained deep deterministic policy gradient algorithm network model to the multi-task motion model of the robot arm, and plan a path for the robot arm.

Images