Multi-mechanical-arm path planning method combining improved artificial potential field method and deep reinforcement learning

Abstract

The invention relates to the field of mechanical arm motion planning, relates to a multi-mechanical arm motion planning method based on an improved artificial potential field method and deep reinforcement learning, and aims to solve the problems that in a multi-mechanical arm system, early-stage rewards are sparse, the training speed is low, and effective strategies are difficult to learn in deep reinforcement learning. The invention provides an improved dynamic artificial potential field expert demonstration scheme, and in the training process, an artificial potential field reward function and a dynamic artificial potential field expert demonstration mechanism which dynamically change along with the training process are added, so that the mechanical arm obtains higher learning efficiency and more positive samples in the early-stage training process; in later training, the mechanical arm is assisted to reach a high-precision target, and then the learning speed of the mechanical arm is increased. The training process mainly comprises the following steps of constructing a simulation environment, describing and modeling a multi-mechanical-arm path planning task, generating a training sample, executing the training task and performing simulation, dynamically adjusting expert data according to a simulation result, collecting experience and updating a strategy. Compared with a traditional method, the method has the advantages that the convergence speed is higher, the training time is shorter, the precision is higher, and particularly, the early-stage convergence speed is higher, and the fitting precision is higher.

Description

Technical Field

[0001] The present invention relates to the technical field of multi-robotic arm intelligent motion planning and deep reinforcement learning, and relates to a multi-robotic arm path planning method combining an improved artificial potential field method and deep reinforcement learning. Background Art

[0002] At present, with the development of robot technology, robots are widely used in the fields of medical, industrial, national defense and home services. Based on traditional methods such as manual teaching, remote operation and complex programming, robots have certain operating skills after training, and can complete tasks quickly and accurately in a structured environment. Among them, path planning is a basic research field in robot motion control systems, which determines the movement mode and operation performance of robots. Different path planning schemes are required for different use occasions. At present, the path planning of manipulators is mainly based on traditional methods and artificial intelligence-based methods. Traditional methods mainly include artificial potential field method, graph search method, sampling-based planning method, etc. Although these traditional methods are simple in structure and easy to execute, and can easily achieve good results in simple task environments, their shortcomings are also obvious. For example, the artificial potential field method is difficult to apply to complex objects and is easy to fall into local minima; the graph search method requires accurate modeling of the environment and graph, and at the same time, the computational complexity is very high in high-dimensional environments; the sampling-based method has certain application and practical value, but with the increase in the number and degree of freedom of the manipulator, the dimension required for sampling also increases rapidly, and the computational time also increases exponentially. At the same time, sampling-based methods have disadvantages such as difficulty in effectively sampling complex environments and difficulty in effectively handling dynamic tasks and multi-target tasks.

[0003] In recent years, artificial intelligence technology has continued to innovate and break through. As one of the key technologies of artificial intelligence, deep reinforcement learning (RL) has shown broad application prospects in the field of robotic arm motion planning. Compared with traditional methods, deep reinforcement learning strategies have excellent adaptability, can effectively handle non-complete constraints, get rid of modeling difficulties, autonomously generate planning schemes according to the environment and tasks, and reduce online planning time through offline training. At present, many deep reinforcement learning methods, including PPO (Proximal Policy Optimization), SAC (Soft Actor-Critic), DDPG (Deep Deterministic Policy Gradient) methods, have shown their respective advantages. In addition to the single robotic arm environment, the situation of multi-robotic arm systems should also be considered. Multi-agent deep reinforcement learning, as an effective method to solve multi-object planning tasks, has more potential than traditional algorithms in solving multi-robotic arm planning problems. From the perspective of training and execution, multi-agent deep reinforcement learning can be divided into the following three paradigms: independent learning (IL) paradigm, centralized training-centralized execution (CTCE) paradigm, and centralized training-distributed execution (CTDE) paradigm. Compared with traditional methods, deep reinforcement learning methods have shown their superiority in scalability and low dependence on modeling.

[0004] However, in the process of training, multi-agent deep reinforcement learning also shows some of its own shortcomings. First of all, deep reinforcement learning is different from ordinary deep learning. The deep learning process relies on a pre-set training set, while the data source of the deep reinforcement learning method is mainly the interaction between the agent and the environment. In the deep reinforcement learning environment, the agent accumulates original experience by observing the environment, performing actions and obtaining rewards or penalties. Although this does not rely on the data set, when the environment is too complex or the task goal is too difficult, it is difficult for the agent to obtain positive rewards in the early stage. This is the so-called sparse reward problem. Under the influence of the sparse reward problem, in the early stage of training, the agent's experience pool accumulates a large amount of invalid experience without positive rewards, making the early training process very slow. In addition, in the multi-robot path planning task, although after a period of training, the agent can approach the target well and ensure no collision, when approaching the task target, the agent often cannot approach the target with the expected accuracy, and will wander around the target. At the same time, when improving the accuracy required for the task, the agent also needs to spend a lot of time training and exploration. Therefore, solving the problem of slow exploration speed of early intelligent agents and insufficient accuracy in completing tasks of later intelligent agents is a key technical issue. Summary of the invention

[0005] In order to solve the above-mentioned problems, the present invention proposes a multi-robot path planning method that combines an improved artificial potential field method and deep reinforcement learning. The method introduces two methods, dynamic potential field reward function and dynamic potential field expert demonstration. Through the overall training and the execution progress of the current task, the reward function is dynamically adjusted and the expert demonstration is used. While using the reward function to dynamically induce the robot arm to move away from the target point and approach the target, the distribution of the reward function is appropriately adjusted to prevent the strategy from converging to the local optimum. At the same time, a dynamic expert demonstration method is used to induce the robot arm to move away from the starting point and approach the target point, increasing the convergence accuracy of the robot arm while expanding more positive rewards for the experience pool.

[0006] To achieve the above purpose, the technical solution provided by the present invention is:

[0007] A multi-manipulator path planning method combining an improved artificial potential field method and deep reinforcement learning comprises the following steps:

[0008] Step 1: Configure the environment

[0009] Select a suitable simulation platform and the corresponding robotic arm model, and build the corresponding physical environment calculation method and robotic arm manipulation method to ensure that the joint activities of the robotic arm driven by the motor, the movement of the various components of the robotic arm and the end effector, as well as the collision with other robotic arms or obstacle entities can be simulated normally.

[0010] Step 2: Define the task

[0011] The multi-robot path planning task is considered as a type of multi-agent deep reinforcement learning task. The goal of the multi-robot path planning task is to find a collision-free path from the initial position to the target position for all the robots. In the multi-agent deep reinforcement learning problem, the environment is shared by multiple agents, and each agent learns the optimal strategy by interacting with the environment. The multi-agent deep reinforcement learning problem is a multi-agent partially observable Markov decision process, which can be expressed as a tuple M =<N,S,A,R,O,T,γ> .

[0012] in:

[0013] N is the number of agents:

[0014] S represents a set of all possible states of the agent, which is used to describe the state of the agent;

[0015] A represents a set of possible actions of the agent, which is used to describe the actions that the agent may take;

[0016] R represents a function R(s,a), which represents the reward value that the agent can obtain by executing action a in set A when it is in a state s in set S;

[0017] O represents the observation space, which is used to express the agent's observation and summary of the surrounding environment information.

[0018] T represents the time for the task to run;

[0019] γ represents the discount factor, which is used to limit the reward value obtained by the agent.

[0020] Step 3: Generate training data

[0021] According to the robot arm configuration and the task requirements of multi-robot arm path planning, task data and static expert data are generated for learning and training of the multi-robot arm path planning agent. The task data and static expert data correspond one to one, where the task data includes the number of robot arms, base position, starting configuration, end effector starting posture, target configuration, end effector target posture, etc.; static expert data includes a series of action values. Task data is used to describe the starting state and target state of a task; static expert data represents a feasible action sequence that can reach the target state from the starting state. In the multi-robot task, it is stipulated that when all the robot arms reach the target state from the starting state without collision, the task is considered to be successfully completed. If one robot arm collides during the process or the task is not completed within the specified time, the task is considered to have failed.

[0022] Step 4: Simulate and interact

[0023] According to the loaded tasks, the simulation environment and experience pool are initialized, and a strategy is configured for each robot. In a time step t, when robot i interacts with the environment, it first collects its own state as the state value s i Then, additionally collect the state values ​​s of other n robotic arms within a certain range of itself 1,……,n , and the state value of itself and the state value of other robotic arms s i,1,……,n Sort them in ascending order according to the distance from their own joint base and process them as observation values ​​o i . According to the state value and observation value, the corresponding action value a is obtained i . Then execute the action and enter the next time step t' to obtain the corresponding reward value r and the state value and observation value of the next state. After completing the simulation of the current time step t, the observation value, action value, reward value, and observation value of the current time are combined into an experience, and this experience is stored in the experience pool.

[0024] Step 5: Dynamic Demonstration and Rewards

[0025] If the simulation task of the current time step t fails, the static expert data corresponding to the current task data generated in step 3 is read, the robot performs actions according to the expert task data and obtains the state value and observation value, and the experience pool of the failed task in step 4 and the experience pool of the expert data are added to the experience replay buffer. If the task being executed in step 4 is successful, the experience pool is directly added to the experience replay buffer. If the current task has not succeeded or failed, the dynamic potential field reward function and dynamic expert demonstration are adjusted according to the current elapsed time and the current state of each robot, and the simulation process continues after the adjustment. The dynamic reward function is divided into three parts: collective reward, proximity reward and collision reward. The collision reward is a negative value as a penalty term. Different positive rewards are given according to the degree to which the robot approaches the target position, and different negative collision rewards are given according to the proximity of the robot to other entities and whether a collision occurs. These rewards are dynamically adjusted as the robot moves and the time step advances. Dynamic expert demonstration also consists of two parts, the discriminant expert and the demonstration expert. The discriminant expert is responsible for determining under what circumstances the robot arm activates the dynamic expert demonstration, and the demonstration expert is responsible for inducing the robot arm to move closer to the target state when the robot arm approaches the target point. As time passes and the robot arm moves, the effect of the expert demonstration is different. The discriminant expert is responsible for balancing the agent's own exploration behavior and the expert demonstration behavior. The demonstration expert mainly takes effect at the end and when the robot arm approaches the target point.

[0026] Step 6: Experience Collection and Strategy Update

[0027] The experience replay buffer is continuously filled according to the experience gained after the operation in step 4 and step 5. After executing a certain number of tasks and accumulating enough interaction and experience data, a certain amount of experience is randomly extracted from the experience replay buffer as the training data of the deep reinforcement learning strategy, and the deep reinforcement learning strategy is updated according to the specified method, so that the deep reinforcement learning strategy's evaluation of the environment is closer to the actual environment, and the output action value can obtain more reward value.

[0028] Step 7: Loop execution

[0029] Repeat steps 4 to 6 to continuously update the experience to update the deep reinforcement learning network and improve the performance of the deep reinforcement learning network. As the cycle progresses, the performance of the deep reinforcement learning network continues to improve. The cycle can be terminated manually or stopped when the specified time or performance is reached to ensure the learning effect.

[0030] Further features, in step 3, the action values ​​in the generated expert data are traditional global path planning methods, such as the BI-RRT* algorithm.

[0031] A further feature is that in step 5, the method used is an expert demonstration method different from that in step 3, and instead of a global path planning method, a local path planning method is used, such as an artificial potential field method.

[0032] Further features, in step 5, the reward function used is a function related to the gravitational potential field function in the artificial potential field method.

[0033] Further features, in step 6, the deep reinforcement learning method used is an algorithm based on the actor-critic architecture, such as the Deep Deterministic Policy Gradient (DDPG) algorithm and the Soft Actor-Critic (SAC) algorithm.

[0034] Compared with the prior art, the present invention has the following beneficial effects:

[0035] Compared with traditional deep reinforcement learning algorithms, this algorithm introduces two expert demonstration methods: static and dynamic. In the early stages of training, the two expert demonstrations can increase the number of positive reward samples, improve the effectiveness of early samples, and speed up the early training process. In the later stages of training, dynamic expert demonstrations can effectively improve the adaptability of samples to high-precision tasks and accelerate the adaptive training process of later high-precision tasks.

[0036] Compared with the traditional artificial potential field method, this algorithm introduces a dynamically changing artificial potential field and improves the potential field calculation method based on the characteristics of multi-robot reinforcement learning. The algorithm can dynamically adjust the scope and degree of effectiveness of the potential field according to the current task completion status, while retaining the simplicity and efficiency of the artificial potential field method, avoiding the algorithm from converging to the local optimum, and solving the problem of being too difficult to judge the effective boundary of the potential field in a multi-agent environment and difficult to construct the potential field for complex objects.

[0037] This method has good scalability. The various modules of the algorithm are independent of each other. The expert demonstration method and the reward function settings can be dynamically modified according to the needs of the current task. It can be trained for different task requirements, such as changing the number of intelligent agents, adding dynamic target points, adding environmental obstacles, etc., and has good scalability. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is a work flow chart of the present invention;

[0039] Figure 2 This is a model architecture diagram of an embodiment of the present invention.

[0040] Figure 3 This is a comparison of the time steps taken when the average task success rate of the robot arm reaches 40% before and after using this method in an embodiment of the present invention. DETAILED DESCRIPTION

[0041] The present invention is further described in detail below in conjunction with embodiments and drawings.

[0042] Step 1: Simulation environment setup

[0043] The simulation environment of this embodiment is built under the Ubuntu platform, the main programming language is python, and anaconda is used for environment management. Pybullet is selected as the physical simulation platform. Pybullet is a classic and commonly used Python library. It can be used to simulate the interaction between multiple rigid bodies, such as the connecting rod in the robot arm, the interaction between the end effector of the robot arm and other objects, etc., and can operate and calculate the movement of objects according to physical parameters such as speed, displacement, acceleration, etc. In addition, it can accurately detect collisions between objects and provide parameters such as collision points and collision forces. In addition, Pybullet can also calculate the forward kinematics and inverse kinematics of the robot.

[0044] This example uses the UR5 open source robotic arm model for testing. As a classic six-degree-of-freedom robotic arm, the UR5 robotic arm is widely used in various production and laboratory scientific research tasks due to its open source model, simple structure and suitable six degrees of freedom. In the Pybullet simulation environment, the program can simply perform operations, simulations, feedback and various detection operations on the UR5 robotic arm.

[0045] Step 2: Define the task;

[0046] The multi-arm path planning task can be regarded as a kind of multi-agent deep reinforcement learning. The goal of the multi-arm path planning task is to find a collision-free path from the initial pose configuration to the target pose configuration for all the manipulators. In a three-dimensional space, let i be a robot with an initial configuration. and a joint space configuration C i There is also a target configuration If the end effector configuration of robot arm i With target configuration The distance difference and angle difference ∈ p and ∈ r When they are less than the two specified values, the robot is considered to have reached the target posture. The collision-free configuration space of robot i is F i ∈C i , the goal is to find a continuous function σ:[0,1]→F1×F2×…×F in the presence of N robotic arms N , where σ(0) is the composite initial joint configuration of the robot, and σ(1) is the joint configuration of all the robot arms to reach the target posture.

[0047] In the multi-agent deep reinforcement learning problem, the environment is shared by N agents, and each agent learns the optimal strategy by interacting with the environment. Multi-agent deep reinforcement learning can be expressed as a partially observable Markov decision process, which can be expressed as a tuple M =<N,S,A,R,O,T,γ> Expression.

[0048] Where N represents the number of robotic arms.

[0049] S represents the set of robot arm states.

[0050] For robot arm i, its state s i Can be represented as a tuple

[0051] in, is a two-dimensional vector used to represent the position of the robot base on the ground, P i jointis a vector representing the angle values ​​of all joints of the robot. For the UR5 robot with six degrees of freedom, it is a six-dimensional vector. P i eef ,P i target is a vector describing the position and posture of the robot. When describing the position and posture, each rigid body is usually represented by a vector group containing two vectors, where the position vector is represented by a three-dimensional vector and the attitude vector is represented by a quaternion. is a vector group describing the positions of all links. i eef and P i target Describe the position and posture of the robot end effector and the target respectively.

[0052] A is the set of robot arm actions. It corresponds to the degree of freedom of the robot arm. Since the UR5 robot arm has six degrees of freedom, the action a of robot arm i i It is a six-dimensional vector that describes the changes in the joint values ​​of the six joints respectively.

[0053] R is the set of reward values ​​obtained by the robot. In general deep reinforcement learning tasks, for each agent, the sum of the reward values ​​R total It can be calculated as follows: R total =R individual +R cooperate +R punishment

[0054] Where R individual Reward for a single robot arm to reach the target; R cooperate represents the collective reward of all robotic arms reaching the target; R punishment Represents the penalty for collision with other robots, where the penalty is expressed as a negative reward value.

[0055] In the deep reinforcement learning training process, this reward distribution method will cause a sparse reward problem, that is, rewards are given only when one or all robotic arms reach the target point, which will result in too few samples with positive rewards harvested by the robotic arms in the early stage. total After improvement, the new reward function can be expressed as the following formula: R total =R force +R cooperate

[0056] In which R cooperate Still represents the reward for all robotic arms to reach the target. force is the reward of the robot arm under the artificial potential field. It can be expressed as the following formula:

[0057] Among them, R individual and R punishment It still represents the reward for a single robot arm to reach the target point and the penalty for collision. f represents the reward obtained by the robot under the influence of the artificial potential field, and γ is the attenuation coefficient mentioned above.

[0058] Artificial Potential Field (APF) is a classic algorithm widely used in robot path planning. Its basic idea is to set up an artificial potential field in the robot's workspace, where the target point corresponds to the gravitational potential well and the obstacle corresponds to the repulsive potential barrier. The robot moves in this artificial potential field, always moving in the direction corresponding to the gravitational field, while avoiding the repulsive field, so as to find a collision-free path to the target point. In general, the gravitational field can be expressed as F att = -μρ(q,q g ), where μ represents the proportional gain coefficient, and ρ is a vector representing the Euclidean distance between two points. Based on the idea of ​​gravitational potential field in artificial potential field, this embodiment sets the potential field reward function R f , R f The specific calculation method is as follows:

[0059]

[0060] where α dist and β dist Are two constants used to refine the size of the reward value. base is the base value of the potential field reward function, which is used to control the proportion of the potential field reward function. new and dist min Respectively represent the current value of the Euclidean distance between the end effector of the robot and the target point at the current time step t and the historical minimum value before reaching this time step. Obviously, at each time step, if and only if the robot is close to the target point, it can get this reward. The closer to the target point, the higher the reward value. When the robot reaches the target point completely, R f Get the maximum value R base ×β dist . R punishment It is the penalty for the robot arm when it collides, which is a large negative value.

[0061] O is the observation value of the robot arm to the surrounding environment. Since the robot arm has an active radius range work , for two manipulators i and j, when their base positions and P j baseThe Euclidean distance between them is greater than twice the radius, which means that the processing condition is met. , the two manipulators cannot collide, so only the manipulators whose base is within twice the active radius need to be processed. In order to make the observation value o a vector of constant length, when calculating the observation value o of manipulator i i When , take its own state value and the state values ​​s of all n robotic arms that meet the above processing conditions i,1,2,…,n , these state values ​​are processed according to the distance from the robot arm and arranged in order of size and output as a fixed-length sequence, which is used as the observation value o of robot arm i i .

[0062] T is the time for task execution, in steps. In each step, the robot generates an action value based on the current state observation value, moves according to the action value to obtain a reward, enters the next state and obtains the next action. When the time step exceeds the preset range, the robot is considered to have failed to execute the task.

[0063] γ is the decay coefficient. In order to prevent the potential field reward function from being too influential and causing the agent to fall into the local optimum, the attenuation coefficient γ is used to control the influence of the potential field reward function. Initially, the value of γ is 50%. As the average success rate of the task increases during the training process, the value of γ is gradually reduced until γ is reduced to 0, at which point the reward function is equal to the traditional reward function.

[0064] Step 3: Generate tasks:

[0065] In this example, each task corresponds to two sets of data, task data and static expert data. Task data is the main part of the task, including the number of manipulators in the task, the base position of each manipulator, the starting state and the end state of the manipulator, the starting joint configuration of the manipulator and the joint configuration corresponding to the end state calculated by inverse kinematics. It also contains the unique serial number of the task and the difficulty of the task. The difficulty of the task is calculated by the overlap degree of the working range of the manipulator.

[0066] Static expert data is a sequence of actions generated for the corresponding task data using a traditional global path planning algorithm. When a task fails, the corresponding expert data is read, the interaction is performed according to the expert action, and the reward value is added to the experience replay pool. In this example, the Bi-RRT* algorithm is used to generate expert data for the task.

[0067] Step 4: Simulate and interact:

[0068] In a simulation, we first read a task data, initialize the initial states of N manipulators according to the number of manipulators, and configure the environment. At the beginning of each time step t, the manipulators initialize their respective state information s 1,2,……,NAt the same time, each robot observes and obtains the base spacing less than or equal to 2*range work Status information of other robotic arms, where range work is the maximum working radius of the robot arm mentioned above. The self-state information and the observed state information are processed to obtain a fixed-length vector as the observation information o 1,2,……N Based on these N pieces of information, the N robotic arms obtain their corresponding action values ​​a 1,2,……,N After obtaining the action value, each robot arm starts to perform actions and simulations according to the action. The action execution process is divided into several small steps to facilitate collision detection and judgment of reaching the target point. After executing the action, the corresponding reward value r is obtained according to the execution result. 1,2,……,N , and obtain the state value and observation value s1 of the new state ′ ,2,……,N and o1 ′ ,2,……,N After executing an operation for a certain time, the experience of the i-th robot in the time step t is It can be expressed as At the end of a time step, all the experience of the robot arms is added to the experience pool. When the task succeeds or fails, all the experience in the experience pool is added to the experience replay buffer. In the case of failure, an additional static expert demonstration is performed, and the experience obtained by executing the expert demonstration is added to the experience replay buffer.

[0069] Step 5: Perform dynamic expert demonstration and reward function setting:

[0070] In the above execution process, although the dynamic potential field reward function is introduced to improve the reward value obtained in the early stage, it is easy to observe from the experimental proof that compared with the training process without the dynamic reward function, although the average posture difference between the robot arm and the target is reduced faster, there is still a lack of positive samples of successful tasks. Therefore, in this example, dynamically changing artificial potential field expert guidance is introduced. The artificial potential field expert guidance function consists of two parts, the expert discrimination part and the dynamic expert guidance part.

[0071] The expert judgment part is responsible for determining whether the expert guidance should be executed. Assume that in a task, the current time step and the maximum executable time step of the simulation process are t curr and t max At the same time, the initial configuration of robot arm i is The target configuration is Current time step t curr The configuration of the lower end effector is These three configurations are recorded in the form of robot arm joint values, which are six-dimensional vectors in this example. curr End effector configuration With target configuration The distance difference and angle difference are and Record the initial configuration of robot arm i and target configuration The distance difference and angle difference are and If at some point, and Are less than or equal to the given value and The task is considered completed.

[0072] When the accuracy of the task is too high, or the accuracy of the robot arm is temporarily increased, the robot arm may experience the following phenomenon: the robot arm can successfully reach the vicinity of the target point from the starting point, but after reaching the target point, the robot arm will oscillate around the target point. In a few cases, the robot arm will also oscillate around the starting point, making it difficult for the robot arm to reach the target point. When the task is not completed for a long time, these phenomena are likely to occur. Therefore, the expert judgment part is responsible for dynamically judging whether the expert demonstration should be enabled to deal with this phenomenon. Since ∈ p and ∈ r ∈ p The impact on the task is greater, so in this example, the expert judgment parameter f of robot arm i is expert Set to the following formula:

[0073]

[0074] Among them, α T is a constant value that limits the effective range, and its value is between 0 and 1. During training, the value of this parameter can be changed according to different requirements of the task. The idea of ​​this parameter is that while using the artificial potential field method as the expert demonstration, the limit of the expert demonstration is made larger and larger, just like a force field that expands over time. For robot arm i, at the current time step t curr If the result of its expert discrimination parameter is true, the dynamic expert demonstration of robot arm i is activated.

[0075] The dynamic expert demonstration part is responsible for providing dynamic expert demonstration actions for the agent. When activating dynamic expert demonstration, first collect t curr The state and action information of robot arm i at time s i and a i According to the idea of ​​artificial potential field method, the gravitational action of robot i is recorded The formula is as follows:

[0076]

[0077] where a i is the observation value o of robot arm i i The action you plan to take. Gravitational Action represents the action that the robot should make under the influence of the normalized traction force. curr When the dynamic expert demonstration of a robot arm is activated, the action a of the robot arm i at that moment is directly i Replaced with expert moves It can be expressed as:

[0078]

[0079] in When executing the expert action, the replaced action is used as the action a of robot i i , to collect subsequent experience. When performing expert judgment and expert demonstration, each robot arm is calculated separately, and the robot arm that does not meet its own conditions is calculated separately. expert The robot arm does not perform any operation. After a task is completed, the reward function is dynamically adjusted based on the average success rate of the current task.

[0080] Step 6: Experience collection and strategy update:

[0081] Based on the experience gained after the operations in step 4 and step 5, the experience replay buffer is updated. After performing a certain number of tasks and accumulating enough interaction and experience data, a certain number of experiences are randomly extracted from the experience replay buffer to update the deep reinforcement learning network. In this case, the SAC algorithm is used as the deep reinforcement learning network. The SAC (Soft Actor-Critic) algorithm is a deep deep reinforcement learning algorithm based on policy gradient. Its key innovation is the addition of an entropy term to the objective function. The entropy of a policy measures the uncertainty or degree of exploration of the policy. Therefore, the SAC algorithm can balance exploration (high entropy) and exploitation (low entropy).

[0082] The optimization goal of the SAC algorithm can be expressed as the following formula:

[0083]

[0084] in, is the entropy value of strategy π, and λ is the temperature coefficient. The SAC network consists of 1 actor network and 4 critic networks. After extracting experience, the critic network and actor network are updated in turn according to the update method of the SAC network.

[0085] Repeat steps 4-6 until the number of model iterations or the accuracy meets the requirements.

Claims

1. A multi-manipulator path planning method combining improved artificial potential field method and deep reinforcement learning, characterized by the following steps: Step 1: Configure the environment Select the simulation platform and the corresponding robot model, and build the corresponding physical environment calculation method and robot manipulation method to ensure that the joint activities of the robot driven by the motor, the movement of the various components of the robot and the end effector, as well as the collision with other robots or obstacle entities can be simulated normally; Step 2: Define the task Consider the multi-robot path planning task as a multi-agent deep reinforcement learning task; The goal of the multi-manipulator path planning task is to find a collision-free path from the initial pose to the target pose for all the manipulators; In the multi-agent deep reinforcement learning problem, the environment is shared by multiple agents, and each agent learns the optimal strategy by interacting with the environment; the multi-agent deep reinforcement learning problem is a multi-agent partially observable Markov decision process, which is expressed as a tuple M =<N,S,A,R,O,T,γ> ; in: N is the number of agents; S represents a set of all possible states of the agent, which is used to describe the state of the agent; A represents a set of possible actions of the agent, which is used to describe the actions that the agent may take; R represents a function R(s,a), which represents the reward value that the agent can obtain by executing action a in set A when it is in a state s in set S; O represents the observation space, which is used to express the observation and summary results of the agent on the surrounding environment information; T represents the time for the task to run; γ represents the discount factor, which is used to limit the reward value obtained by the agent; Step 3: Generate training data According to the task requirements of the robot arm configuration and multi-robot arm path planning, task data and static expert data are generated for the multi-robot arm path planning agent to learn and train; the task data and static expert data correspond one to one, where the task data includes the number of robot arms, base position, starting configuration, end effector starting posture, target configuration, and end effector target posture; static expert data includes a series of action values; task data is used to describe the starting state and target state of a task; static expert data represents a feasible action sequence from the starting state to the target state; in the multi-robot arm task, it is stipulated that when all the robot arms reach the target state from the starting state without collision, the task is considered to be successfully completed; if one robot arm collides during the process or the task is not completed within the specified time, the task is considered to have failed; Step 4: Simulate and interact According to the loaded task, the simulation environment and experience pool are initialized, and a strategy is configured for each robot arm; within a time step t, when robot arm i interacts with the environment, it first collects its own state as the state value s i Then, the state values ​​s1, ..., n of other manipulators within a certain range of itself are collected, and the state values ​​of itself and other manipulators si, 1, ..., n are sorted in ascending order according to the distance from its joint base and processed as observation values ​​oi; the corresponding action value ai is obtained according to the state value and the observation value; then the action is executed and the next time step t is entered ′ , get the corresponding reward value r and the state value and observation value of the next state; after completing the simulation of the current time step t, combine the observation value, action value, reward value, and observation value of the current time into an experience, and store this experience in the experience pool; Step 5: Dynamic Demonstration and Rewards If the simulation task at the current time step t fails, the static expert data corresponding to the current task data generated in step 3 is read, the robot performs actions according to the expert task data and obtains state values ​​and observation values, and the experience pool of the task that failed to be executed in step 4 and the experience pool of the expert data are added to the experience playback buffer; If the task being executed in step 4 is successful, the experience pool is directly added to the experience playback buffer; if the current task has not yet succeeded or failed, the dynamic potential field reward function and dynamic expert demonstration are adjusted according to the current elapsed time and the current state of each robot, and the simulation process continues after the adjustment; the dynamic reward function is divided into three parts: collective reward, proximity reward and collision reward; the collision reward is a negative value, which is used as a penalty item; different positive rewards are given according to the degree of proximity of the robot to the target position, and different negative collision rewards are given according to the proximity of the robot to other entities and whether a collision occurs. These rewards are dynamically adjusted with the movement of the robot and the advancement of time steps; the dynamic expert demonstration also includes two parts, the discriminant expert and the demonstration expert. The discriminant expert is responsible for determining under what circumstances the robot activates the dynamic expert demonstration, and the demonstration expert is responsible for inducing the robot to move closer to the target state when the robot approaches the target point; with the passage of time steps and the movement of the robot, the effect of the expert demonstration is different; the discriminant expert is responsible for balancing the agent's own exploration behavior and the expert demonstration behavior, and the demonstration expert mainly takes effect at the end and when the robot approaches the target point; Step 6: Experience Collection and Strategy Update Based on the experience gained after the operations in step 4 and step 5, the experience replay buffer is continuously filled; after executing a certain number of tasks and accumulating enough interaction and experience data, a certain number of experiences are randomly extracted from the experience replay buffer as training data for the deep reinforcement learning strategy, and the deep reinforcement learning strategy is updated according to the specified method, so that the evaluation results of the deep reinforcement learning strategy on the environment are closer to the actual environment, and the output action value can obtain more reward value; Step 7: Loop execution Repeat steps 4 to 6 to continuously update the experience to update the deep reinforcement learning network and improve the performance of the deep reinforcement learning network. As the cycle proceeds, the performance of the deep reinforcement learning network continues to improve. The cycle can be terminated manually or stopped when the specified time or performance is reached to ensure the learning effect.

2. The multi-manipulator path planning method combining improved artificial potential field method and deep reinforcement learning according to claim 1, characterized in that: In step 3, the method for generating action values ​​in expert data is a global path planning method.

3. The multi-manipulator path planning method combining improved artificial potential field method and deep reinforcement learning according to claim 2, characterized in that: In step 3, the global path planning method is the BI-RRT* algorithm.

4. The multi-manipulator path planning method combining improved artificial potential field method and deep reinforcement learning according to claim 1, characterized in that: In step 5, the method adopted is a local path planning method.

5. The multi-manipulator path planning method combining improved artificial potential field method and deep reinforcement learning according to claim 4, characterized in that: In step 5, the local path planning method is to adopt the artificial potential field method.

6. The multi-manipulator path planning method combining improved artificial potential field method and deep reinforcement learning according to claim 1, characterized in that: In step 5, the reward function used is a function related to the gravitational potential field function in the artificial potential field method.

7. The multi-manipulator path planning method combining improved artificial potential field method and deep reinforcement learning according to claim 1, characterized in that: In step 6, the deep reinforcement learning method used is an algorithm based on the actor-critic architecture.

8. The multi-manipulator path planning method combining improved artificial potential field method and deep reinforcement learning according to claim 7, characterized in that: In step 6, the algorithms based on the actor-critic architecture are the deep deterministic policy gradient algorithm DDPG algorithm and the soft actor-critic algorithm SAC.

Images