A multi-robot path planning method combining improved artificial potential field method and deep reinforcement learning

Abstract

The present application relates to the field of mechanical arm motion planning, and relates to a multi-robot arm motion planning method based on an improved artificial potential field method and deep reinforcement learning, aiming at the characteristics that the early reward is sparse, the training speed is slow, and it is difficult to learn effective strategies in the multi-robot arm system of deep reinforcement learning, the present application proposes an improved dynamic artificial potential field expert demonstration scheme, by adding an artificial potential field reward function and a dynamic artificial potential field expert demonstration mechanism which dynamically change with the training process in the training process, so that the robot arm obtains higher learning efficiency and more positive samples in the early training process, and assists the robot arm to reach the high-precision target in the later training, thereby speeding up the learning speed of the robot arm. The training process mainly includes the following steps: constructing a simulation environment, describing and modeling the multi-robot arm path planning task, generating training samples, executing the training task and performing simulation, dynamically adjusting the expert data according to the simulation result, experience collection and strategy updating. Compared with the traditional method, the present application has faster convergence speed, shorter training time and higher precision, especially faster early convergence speed and higher fitting precision.

Description

Technical Field

[0001] The present invention relates to the technical field of multi-robotic arm intelligent motion planning and deep reinforcement learning, and in particular to a multi-robotic arm path planning method combining an improved artificial potential field method and deep reinforcement learning. Background Art

[0002] With the advancement of robotics technology, robots are now widely used in fields such as medicine, industry, defense, and home services. Based on traditional methods such as manual instruction, teleoperation, and complex programming, robots, after training, acquire certain operational skills and can quickly and accurately complete tasks in structured environments. Path planning is a fundamental research area in robotic motion control systems, determining the robot's motion and operational performance. Different path planning schemes are required for different applications. Currently, traditional and artificial intelligence-based methods are predominantly used for manipulator path planning. Traditional methods include artificial potential field methods, graph search methods, and sampling-based planning methods. While these traditional methods are simple and easy to implement, often achieving good results in simple task environments, they also have significant drawbacks. For example, artificial potential field methods are difficult to apply to complex objects and are prone to getting stuck in local minima. Graph search methods require precise modeling of the environment and graph, and are computationally complex in high-dimensional environments. Sampling-based methods have certain application and practical value, but as the number and degree of freedom of the manipulator increase, the number of sampling dimensions increases rapidly, and the computational time also increases exponentially. At the same time, sampling-based methods have disadvantages such as difficulty in effectively sampling complex environments and difficulty in effectively handling dynamic tasks and multi-target tasks.

[0003] In recent years, artificial intelligence (AI) technology has continued to achieve innovative breakthroughs. Deep reinforcement learning (RL), as one of the key AI technologies, has shown broad application prospects in the field of robotic arm motion planning. Compared to traditional methods, deep reinforcement learning strategies offer superior adaptability, effectively handling nonholonomic constraints, overcoming modeling difficulties, autonomously generating planning solutions based on the environment and task, and reducing online planning time through offline training. Currently, numerous deep reinforcement learning methods, including Proximal Policy Optimization (PPO), Soft Actor-Critic (SAC), and Deep Deterministic Policy Gradient (DDPG), have demonstrated their respective advantages. Beyond single-manipulator environments, multi-manipulator systems must also be considered. Multi-agent deep reinforcement learning, as an effective approach for multi-object planning tasks, holds greater potential than traditional algorithms for solving multi-manipulator planning problems. From the perspective of training and execution, multi-agent deep reinforcement learning can be divided into three paradigms: independent learning (IL), centralized training / centralized execution (CTCE), and centralized training / decentralized execution (CTDE). Compared to traditional approaches, deep reinforcement learning demonstrates superiority in scalability and reduced reliance on modeling.

[0004] However, during the training process, multi-agent deep reinforcement learning also exhibits some of its own shortcomings. First, unlike ordinary deep learning, the deep learning process relies on a pre-set training set, while the data source of the deep reinforcement learning method is mainly the interaction between the agent and the environment. In the deep reinforcement learning environment, the agent accumulates raw experience by observing the environment, performing actions, and obtaining rewards or penalties. Although this does not rely on a data set, when the environment is too complex or the task goal is too difficult, the agent has difficulty obtaining positive rewards in the early stage, which is the so-called problem of sparse rewards. Under the influence of the problem of sparse rewards, in the early stage of training, the agent's experience pool accumulates a large amount of invalid experience without positive rewards, making the early training process very slow. In addition, in the multi-robot path planning task, although the agent can approach the target well and ensure no collision after a period of training, the agent often cannot approach the target with the desired accuracy and will wander around the target. At the same time, when improving the accuracy required by the task, the agent also needs to spend a lot of time training and exploring. Therefore, solving the problem of slow exploration speed of the agent in the early stage and the problem of insufficient accuracy of the agent in completing the task in the later stage is a key technical issue. SUMMARY

[0005] In order to solve the above-mentioned problems, the present application provides a multi-robot path planning method combining improved artificial potential field method and deep reinforcement learning. The method introduces dynamic potential field reward function and dynamic potential field expert demonstration, dynamically adjusts the reward function and uses dynamic expert demonstration according to the overall progress and the execution progress of the current task, dynamically induces the robot to move away from the target point and approach the target point using the reward function, and appropriately adjusts the distribution of the reward function to prevent the policy from converging to a local optimum. At the same time, the dynamic expert demonstration method is used to induce the robot to move away from the starting point and approach the target point, increase the convergence accuracy of the robot, and expand the experience pool with more positive rewards.

[0006] To achieve the above purpose, the technical scheme provided by the present application is:

[0007] A multi-robot path planning method combining improved artificial potential field method and deep reinforcement learning, comprising the following steps:

[0008] Step 1: Environment configuration

[0009] Select a suitable simulation platform and corresponding robot model, build a corresponding physical environment calculation method and robot manipulation method to ensure that the movement of each joint of the robot under motor drive and the movement of each component and end effector of the robot can be normally simulated, and the reaction of collision with other robots or obstacles, etc.

[0010] Step 2: Define the task

[0011] The multi-manipulator path planning task is considered as a type of multi-agent deep reinforcement learning task. The goal of the multi-manipulator path planning task is to find a collision-free path from the initial pose to the target pose for all manipulators. In the multi-agent deep reinforcement learning problem, the environment is shared by multiple agents, and each agent learns the optimal strategy by interacting with the environment. The multi-agent deep reinforcement learning problem is a multi-agent partially observable Markov decision process, which can be expressed as a tuple M =<N,S,A,R,O,T,γ> .

[0012] in:

[0013] N is the number of agents:

[0014] S represents a set of all possible states of the agent, which is used to describe the state of the agent;

[0015] A represents a set of possible actions of the agent, which is used to describe the actions that the agent may take;

[0016] R represents a function R(s,a), which represents the reward value that the agent can obtain by executing action a in set A when it is in a state s in set S;

[0017] O represents the observation space, which is used to express the agent's observation and summary of the surrounding environment information.

[0018] T represents the time the task runs;

[0019] γ represents a discount factor that limits the reward value obtained by the agent.

[0020] Step 3: Generate training data

[0021] Based on the robot configuration and the task requirements of multi-robot path planning, task data and static expert data are generated for learning and training of the multi-robot path planning agent. There is a one-to-one correspondence between task data and static expert data, where task data includes the number of robot arms, base position, starting configuration, end effector starting posture, target configuration, end effector target posture, etc.; static expert data includes a series of action values. Task data is used to describe the starting state and target state of a task; static expert data represents a feasible action sequence that can reach the target state from the starting state. In multi-robot tasks, it is stipulated that when all robot arms reach the target state from the starting state without collision, the task is considered to be successfully completed. If one robot arm collides during the process or the task is not completed within the specified time, the task is considered to have failed.

[0022] Step 4: Simulation and Interaction

[0023] According to the loaded task, initialize the simulation environment and experience pool, and configure a strategy for each robot arm. In a time step t, when the robot arm i interacts with the environment, it first collects its own state as the state value s i , and then additionally collects the state values s 1,……,n of its n robot arms within a certain range. i,1,……,n After sorting and processing according to the distance from its own joint base from small to large, the state values and observation values o i of other robot arms are obtained. i According to the state value and observation value, the corresponding action value a i is obtained. Then the action is executed, and the next time step t' is entered to obtain the corresponding reward value r and the next state value and observation value. After completing the simulation of the current time step t, the observation value, action value, reward value, and observation value of the next time of the current time are combined into an experience, and this experience is stored in the experience pool.

[0024] Step 5: Dynamic Demonstration and Reward

[0025] If the simulation task of the current time step t fails, read the static expert data corresponding to the current task data generated in step 3, and the robot arm executes the action according to the expert task data and obtains the state value and observation value. The experience pool of the task that failed in step 4 and the experience pool of the expert data are added to the experience replay buffer. If the task being executed in step 4 is successful, the experience pool is directly added to the experience replay buffer. If the current task has not been successful or failed, the dynamic potential field reward function and dynamic expert demonstration adjustment are performed according to the current elapsed time and the current state of each robot arm. After adjustment, the simulation process continues. The dynamic reward function is divided into three parts: collective reward, proximity reward, and collision reward. The collision reward is negative and serves as a penalty term. Different positive rewards are given according to the degree of proximity of the robot arm to the target position. Different negative collision rewards are given according to the proximity of the robot arm to other entities and whether a collision occurs. These rewards are dynamically adjusted with the movement of the robot arm and the advancement of the time step. The dynamic expert demonstration also includes two parts: the discrimination expert and the demonstration expert. The discrimination expert is responsible for determining when the robot arm activates the dynamic expert demonstration. The demonstration expert is responsible for inducing the robot arm to move towards the target state when it is close to the target point. The effect of the expert demonstration varies with the passage of time and the movement of the robot arm. The discrimination expert is responsible for balancing the exploration behavior of the agent itself and the expert demonstration behavior. The demonstration expert mainly acts in the final stage and when the robot arm is close to the target point.

[0026] Step 6: Experience Collection and Strategy Update

[0027] The experience replay buffer is continuously filled based on the experience gained from the operations in steps 4 and 5. After executing a certain number of tasks and accumulating sufficient interaction and experience data, a certain amount of experience is randomly extracted from the experience replay buffer as training data for the deep reinforcement learning policy. The deep reinforcement learning policy is updated according to the specified method, so that the deep reinforcement learning policy's assessment of the environment is closer to the actual environment and the output action value can obtain higher reward value.

[0028] Step 7: Loop execution

[0029] Repeat steps 4 through 6 to continuously update the experience and improve the performance of the deep reinforcement learning network. As the loop progresses, the performance of the deep reinforcement learning network continues to improve. The loop can be manually terminated or stopped when a specified time or performance is reached to ensure learning results.

[0030] Further features, in step 3, the action values ​​in the generated expert data are traditional global path planning methods, such as the BI-RRT* algorithm.

[0031] A further feature is that in step 5, the method adopted is an expert demonstration method different from that in step 3, and instead of a global path planning method, a local path planning method, such as an artificial potential field method, is adopted.

[0032] Further features, in step 5, the reward function used is a function related to the gravitational potential field function in the artificial potential field method.

[0033] Further features, in step 6, the deep reinforcement learning method used is an algorithm based on the actor-critic architecture, such as the Deep Deterministic Policy Gradient (DDPG) algorithm and the Soft Actor-Critic (SAC) algorithm.

[0034] Compared with the prior art, the present invention has the following beneficial effects:

[0035] Compared to traditional deep reinforcement learning algorithms, this algorithm introduces two expert demonstration methods: static and dynamic. In the early stages of training, these two types of expert demonstration can increase the number of positively rewarded samples, improve the effectiveness of these samples, and accelerate the training process. In the later stages of training, dynamic expert demonstration can effectively improve the samples' adaptability to high-precision tasks, accelerating the adaptive training process for these tasks.

[0036] Compared to traditional artificial potential field methods, this algorithm introduces a dynamically changing artificial potential field and improves its calculation method to address the unique characteristics of multi-manipulator reinforcement learning. The algorithm dynamically adjusts the scope and extent of the potential field's effectiveness based on the current task completion status. While retaining the simplicity and efficiency of the artificial potential field method, it avoids convergence to local optima. Furthermore, it addresses the challenges of determining the potential field's effectiveness boundaries and constructing potential fields for complex objects in multi-agent environments.

[0037] This method has good scalability. The various modules of the algorithm are independent of each other. The expert demonstration method and the reward function settings can be dynamically modified according to the needs of the current task. It can be trained for different task requirements, such as changing the number of intelligent agents, adding dynamic target points, adding environmental obstacles, etc., and has good scalability. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is a workflow diagram of the present invention;

[0039] Figure 2 This is a model architecture diagram of an embodiment of the present invention.

[0040] Figure 3 This is a comparison of the time steps taken for the average task success rate of the robotic arm to reach 40% before and after using this method in an embodiment of the present invention. DETAILED DESCRIPTION

[0041] The present invention will be described in further detail below with reference to the embodiments and accompanying drawings.

[0042] Step 1: Simulation environment setup

[0043] The simulation environment of this embodiment was built on the Ubuntu platform, with Python as the primary programming language and Anaconda used for environment management. Pybullet was selected as the physical simulation platform. Pybullet is a classic and commonly used Python library. It can be used to simulate the interaction between multiple rigid bodies, such as the connecting rods in a robotic arm, the interaction between the robotic arm's end effector and other objects, and can manipulate and calculate the motion of objects based on physical parameters such as velocity, displacement, and acceleration. Furthermore, it can accurately detect collisions between objects and provide parameters such as collision point and collision force. Furthermore, Pybullet can also perform forward kinematics and inverse kinematics calculations for robots.

[0044] This example uses the UR5 open-source robotic arm model for testing. As a classic six-degree-of-freedom robotic arm, the UR5 is widely used in various production and laboratory research tasks due to its open-source model, simple structure, and suitable six degrees of freedom. Within the Pybullet simulation environment, the program can easily execute operations, simulations, feedback, and various detection operations on the UR5 robotic arm.

[0045] Step 2: Define the task;

[0046] The multi-manipulator path planning task can be regarded as a kind of multi-agent deep reinforcement learning. The goal of the multi-manipulator path planning task is to find a collision-free path from the initial pose configuration to the target pose configuration for all manipulators. In a three-dimensional space, let i be an initial configuration and a joint space configuration C i There is also a target configuration If the end effector configuration of robot arm i is Target configuration The distance difference and angle difference ∈ p and ∈ r When the two values ​​are less than the specified values, the manipulator is considered to have reached the target posture. The collision-free configuration space of manipulator i is F i ∈C i , the goal is to find a continuous function σ:[0,1]→F1×F2×…×F in the presence of N robotic arms N , where σ(0) is the composite initial joint configuration of the manipulator, and σ(1) is the joint configuration of all manipulators to reach the target pose.

[0047] In the multi-agent deep reinforcement learning problem, the environment is shared by N agents, and each agent learns the optimal strategy by interacting with the environment. Multi-agent deep reinforcement learning can be expressed as a partially observable Markov decision process, which can be represented by a tuple M=<N,S,A,R,O,T,γ> Expression.

[0048] Where N represents the number of robotic arms.

[0049] S represents the set of robot arm states.

[0050] For robot arm i, its state s i Representable as a tuple

[0051] in, is a two-dimensional vector used to represent the position of the robot base on the ground, P i jointis a vector representing the angle values of all joints of the robot arm. For a UR5 robot arm with six degrees of freedom, it is a six-dimensional vector. P i eef ,P i target is a vector representing the pose of the robot arm. In representing a pose, each rigid body is usually represented by a set of two vectors, where the position vector is represented by a three-dimensional vector and the attitude vector is represented by a quaternion. is a set of vectors representing the poses of all links. P i eef and P i target respectively represent the pose of the end effector of the robot arm and the pose of the target.

[0052] A is a set of actions of the robot arm. Corresponding to the degrees of freedom of the robot arm. Since the UR5 robot arm has six degrees of freedom, the action a i of the robot arm i is a six-dimensional vector, which respectively describes the change amount of the six joint values.

[0053] R is a set of reward values obtained by the robot arm. In a general deep reinforcement learning task, for each agent, the sum of the reward values R total can be calculated as follows: R total = R individual + R cooperate + R punishment

[0054] where R individual is the reward of a single robot arm reaching the target; R cooperate represents the collective reward of all robot arms reaching the target; and R punishment represents the penalty for collision with other robot arms, where the penalty is represented by a negative reward value.

[0055] During the deep reinforcement learning training process, this reward allocation method will cause the problem of sparse rewards, that is, only when a single or all robot arms reach the target point will the reward be given, which will result in too few samples with positive rewards being obtained by the robot arm in the early stage. In this embodiment, the reward function R total is improved, and the new reward function can be represented as follows: R total = R force + R cooperate

[0056] where R cooperate still represents the reward of all robot arms reaching the target. And R force is the reward of the robot arm in the artificial potential field. Specifically, it can be represented as follows:

[0057] Among them, R individual and R punishment Still represents the reward for a single robot arm to reach the target point and the penalty for collision. f represents the reward obtained by the robotic arm under the influence of the artificial potential field, and γ is the attenuation coefficient mentioned above.

[0058] The Artificial Potential Field (APF) method is a classic algorithm widely used in robot path planning. Its basic idea is to set up an artificial potential field in the robot's workspace, where the target point corresponds to the gravitational potential well and the obstacles correspond to the repulsive potential barriers. The robot moves in this artificial potential field, always moving in the direction corresponding to the gravitational field while avoiding the repulsive field, thus finding a collision-free path to the target point. In general, the gravitational field can be expressed as F att =-μρ(q,q g ), where μ represents the proportional gain coefficient and ρ is a vector representing the Euclidean distance between two points. Based on the idea of ​​gravitational potential field in artificial potential field, this embodiment sets the potential field reward function R f , R f The specific calculation method is as follows:

[0059]

[0060] where α dist and β dist Are two constants used to refine the size of the reward value. base It is the base value of the potential field reward function, which is used to control the proportion of the potential field reward function. new and dist min Respectively represent the current value of the Euclidean distance between the end effector of the manipulator and the target point at the current time step t and the historical minimum value before reaching this time step. Obviously, at each time step, if and only if the manipulator is close to the target point, it can get this reward. The closer to the target point, the higher the reward value. When the manipulator completely reaches the target point, R f Get the maximum value R base ×β dist . R punishment It is the penalty for the robot arm to collide, which is a large negative value.

[0061] O is the observation value of the robot arm to the surrounding environment. Since the robot arm has an active radius range work , for two manipulators i and j, when their base positions and P j baseThe Euclidean distance between them is greater than twice the radius, which means that the processing condition is met. When , the two manipulators cannot collide, so only the manipulator whose base is within twice the active radius needs to be processed. In order to make the observation value o a vector of constant length, when calculating the observation value o of manipulator i i When , take its own state value and the state value s of all n manipulators that meet the above processing conditions i,1,2,…,n , these state values ​​are processed according to the distance from the robot arm and the size is arranged and a fixed-length sequence is output, which is used as the observation value o of the robot arm i i .

[0062] T is the time it takes to execute the task, measured in steps. At each step, the robot generates an action value based on the current state observation, moves according to the action value, receives a reward, enters the next state, and obtains the next action. If the time step exceeds the preset range, the robot is considered to have failed the task.

[0063] γ is the attenuation coefficient. To prevent the potential field reward function from being too influential and causing the agent to fall into a local optimum, a decay coefficient γ is used to control its influence. Initially, γ is set to 50%. As the average task success rate increases during training, the value of γ is gradually reduced until it reaches 0, at which point the reward function becomes equivalent to the traditional reward function.

[0064] Step 3: Generate tasks:

[0065] In this example, each task corresponds to two sets of data: task data and static expert data. The task data is the main component of the task and includes the number of manipulators in the task, the base position of each manipulator, the starting and ending states of the manipulators, and the starting and ending joint configurations of the manipulators calculated from these states using inverse kinematics. It also contains a unique sequence number for the task and the task difficulty. The task difficulty is calculated based on the degree of overlap between the manipulators' working ranges.

[0066] Static expert data is a sequence of actions generated for the corresponding task data using a traditional global path planning algorithm. When a task fails, the corresponding expert data is read, interactions are performed based on the expert actions, and the reward value is added to the experience replay pool. In this example, the Bi-RRT* algorithm is used to generate expert data for the task.

[0067] Step 4: Simulate and interact:

[0068] In a simulation process, we first read a task data, initialize the initial state of N manipulators according to the number of manipulators, and configure the environment. At the beginning of each time step t, the manipulators initialize their respective state information s 1,2,……,NAt the same time, each robot observes and obtains the base spacing less than or equal to 2*range work Status information of other robotic arms, where range work is the maximum working radius of the robot arm mentioned above. The self-state information and the observed state information are processed to obtain a fixed-length vector as the observation information o 1,2,……N Based on these N pieces of information, the N robotic arms each obtain their corresponding action value a 1,2,……,N After obtaining the action value, each robot arm starts to perform actions and simulates them. The action execution process is divided into several small steps to facilitate collision detection and determination of reaching the target point. After executing the action, the corresponding reward value r is obtained according to the execution result. 1,2,……,N , and obtain the state value and observation value s1 of the new state ′ ,2,……,N and o1 ′ ,2,……,N After completing an operation for a certain time, the experience of the i-th manipulator in the time step t is It can be expressed as At the end of a time step, all experiences of the robot arm are added to the experience pool. When the task succeeds or fails, all experiences in the experience pool are added to the experience replay buffer. In the case of failure, an additional static expert demonstration is performed, and the experience gained from executing the expert demonstration is added to the experience replay buffer.

[0069] Step 5: Perform dynamic expert demonstration and reward function setting:

[0070] In the above execution process, although a dynamic potential field reward function was introduced to improve the initial reward value, experimental results clearly show that while the average pose difference between the robot arm and the target decreased more rapidly compared to training without a dynamic reward function, there was still a lack of positive examples of successful tasks. Therefore, in this example, a dynamically changing artificial potential field expert guidance was introduced. The artificial potential field expert guidance function consists of two parts: an expert judgment component and a dynamic expert guidance component.

[0071] The expert judgment part is responsible for determining whether the expert guidance should be executed. Assume that in a task, the current time step and the maximum executable time step of the simulation process are t curr and t max At the same time, the initial configuration of robot arm i is The target configuration is Current time step t curr The lower end effector configuration is The three configurations are recorded in the form of robot arm joint values, which are all six-dimensional vectors in this example. curr End effector configuration Target configuration The distance difference and angle difference are and The initial configuration of robot arm i is recorded as and target configuration The distance difference and angle difference are and If at some point, and Are less than or equal to the given value and The task is considered completed.

[0072] When the accuracy of the task is too high, or the accuracy of the robot arm is temporarily increased, the robot arm may experience the following phenomenon: the robot arm can smoothly reach the target point from the starting point, but after reaching the target point, the robot arm will wander and oscillate near the target point. In a few cases, the robot arm will also oscillate near the starting point, making it difficult for the robot arm to reach the target point. When the task is not completed for a long time, there is a high possibility that these phenomena may occur. Therefore, the expert judgment part is responsible for dynamically judging whether the expert demonstration should be enabled to deal with this phenomenon. Since ∈ p and ∈ r ∈ p It has a greater impact on the task, so in this example, the expert judgment parameter f of the robot arm i is expert Set to the following formula:

[0073]

[0074] Among them, α T is a constant value that limits the effective range, ranging from 0 to 1. During training, the value of this parameter can be changed according to the different requirements of the task. The idea of ​​this parameter is to use the artificial potential field method as the expert demonstration while making the limit of the expert demonstration effective become larger and larger, just like a force field that expands over time. For robot arm i, at the current time step t curr If the result of its expert discrimination parameter is true, the dynamic expert demonstration of robot arm i is activated.

[0075] The dynamic expert demonstration part is responsible for providing dynamic expert demonstration actions for the agent. When activating dynamic expert demonstration, first collect t curr The state and action information s of the robot arm i at the moment i and a i According to the idea of ​​artificial potential field method, record the gravitational movement of robot arm i The formula is as follows:

[0076]

[0077] where a i is the action planned by the robot i at observation o i . The attractive action represents the action that the robot should make under the influence of the normalized attractive force. If at time t curr , the dynamic expert of a certain robot is in the active state, then the action a i of the robot i at time t is replaced by the expert action can be expressed as:

[0078]

[0079] where When the expert action is executed, the replaced action is taken as the action a i of the robot i to collect subsequent experience. When the expert discrimination and expert demonstration are executed, the parameters are calculated for each robot respectively. For the robots that do not meet their own conditions f expert , no operation is performed. After a task is completed, the reward function is dynamically adjusted according to the average success rate of the current task.

[0080] Step 6: Experience collection and policy update:

[0081] According to the experience obtained after the operation in step 4 and step 5, the experience replay buffer is updated. After a certain number of tasks are executed and sufficient interaction and experience data are accumulated, a certain number of experiences are randomly extracted from the experience replay buffer, and the deep reinforcement learning network is updated. In this example, the SAC algorithm is used as the deep reinforcement learning network. The SAC (Soft Actor-Critic) algorithm is a deep deep reinforcement learning algorithm based on policy gradient. Its key innovation is to add an entropy term in the objective function. The entropy of the policy measures the uncertainty or exploration degree of the policy. Therefore, the SAC algorithm can balance exploration (high entropy) and exploitation (low entropy).

[0082] The optimization objective of the SAC algorithm can be expressed as the following formula:

[0083]

[0084] where, is the entropy value of the policy π, and λ is the temperature coefficient. The SAC network contains 1 actor network and 4 critic networks. After extracting the experience, the critic network and the actor network are updated in turn according to the update method of the SAC network.

[0085] Steps 4-6 are repeated until the number of model iterations or the accuracy reaches the required value.

Claims

1. A multi-manipulator path planning method combining an improved artificial potential field method and deep reinforcement learning, characterized by the following steps: Step 1: Configure the environment Select a simulation platform and a corresponding robotic arm model, and build the corresponding physical environment calculation method and robotic arm manipulation method to ensure that the motor-driven robotic arm joint movements, the movement of the robotic arm's components and end effector, and collisions with other robotic arms or obstacles can be properly simulated; Step 2: Define the task Consider the multi-robot path planning task as a multi-agent deep reinforcement learning task; The goal of the multi-manipulator path planning task is to find a collision-free path from the initial pose to the target pose for all manipulators; In the multi-agent deep reinforcement learning problem, the environment is shared by multiple agents, and each agent learns the optimal strategy by interacting with the environment. The multi-agent deep reinforcement learning problem is a multi-agent partially observable Markov decision process, which is expressed as a tuple M=<N,S,A,R,O,T,γ> ; in: N is the number of agents; S represents a set of all possible states of the agent, which is used to describe the state of the agent; A represents a set of possible actions of the agent, which is used to describe the actions that the agent may take; R represents a function R(s,a), which represents the reward value that the agent can obtain by executing action a in set A when it is in state s in set S. O represents the observation space, which is used to express the observation and summary results of the agent's surrounding environment information. T represents the time the task runs; γ represents the discount factor, which is used to limit the reward value obtained by the agent; Step 3: Generate training data According to the task requirements of the robot arm configuration and multi-robot arm path planning, task data and static expert data are generated for the multi-robot arm path planning agent to learn and train; the task data and static expert data correspond one-to-one, where the task data includes the number of robot arms, base position, starting configuration, end effector starting posture, target configuration, and end effector target posture; the static expert data includes a series of action values; the task data is used to describe the starting state and target state of a task; the static expert data represents a feasible action sequence from the starting state to the target state; in the multi-robot arm task, it is stipulated that when all the robot arms reach the target state from the starting state without collision, the task is considered to be successfully completed; if one of the robot arms collides during the process or the task is not completed within the specified time, the task is considered to have failed; Step 4: Simulate and interact According to the loaded task, the simulation environment and experience pool are initialized, and a strategy is configured for each robot arm. In a time step t, when the robot arm i interacts with the environment, it first collects its own state as the state value s. i Then, it collects the state values ​​s1, ..., n of the other n robotic arms within a certain range of itself, sorts its own state value and the state values ​​si, 1, ..., n of other robotic arms according to the distance from its own joint base from small to large, and processes them as the observation value oi; obtains the corresponding action value ai according to the state value and the observation value; then executes the action and enters the next time step t ′ , get the corresponding reward value r and the state value and observation value of the next state; after completing the simulation of the current time step t, combine the current time observation value, action value, reward value, and the next time observation value into an experience, and store this experience in the experience pool; Step 5: Dynamic Demonstration and Rewards If the simulation task at the current time step t fails, the static expert data corresponding to the current task data generated in step 3 is read, the robot arm performs actions according to the expert task data and obtains state values ​​and observation values, and the experience pool of the task that failed in step 4 and the experience pool of the expert data are added to the experience replay buffer; If the task being executed in step 4 is successful, the experience pool is directly added to the experience replay buffer; if the current task has not yet succeeded or failed, the dynamic potential field reward function and dynamic expert demonstration are adjusted according to the current time and the current state of each manipulator, and the simulation process continues after the adjustment; the dynamic reward function is divided into three parts: collective reward, proximity reward and collision reward; the collision reward is a negative value, which serves as a penalty item; different positive rewards are given according to the degree to which the manipulator approaches the target position, and different negative collision rewards are given according to the proximity of the manipulator to other entities and whether a collision occurs. These rewards are dynamically adjusted as the manipulator moves and the time step advances; the dynamic expert demonstration also includes two parts, the discriminant expert and the demonstration expert. The discriminant expert is responsible for determining under what circumstances the manipulator activates the dynamic expert demonstration, and the demonstration expert is responsible for inducing the manipulator to move closer to the target state when the manipulator approaches the target point; as the time step passes and the manipulator moves, the effect of the expert demonstration is different; the discriminant expert is responsible for balancing the agent's own exploration behavior and the expert demonstration behavior, and the demonstration expert mainly takes effect at the end and when the manipulator approaches the target point; Step 6: Experience Collection and Strategy Update Based on the experience gained after the operations in steps 4 and 5, the experience replay buffer is continuously filled. After performing a certain number of tasks and accumulating sufficient interaction and experience data, a certain amount of experience is randomly extracted from the experience replay buffer as training data for the deep reinforcement learning strategy. The deep reinforcement learning strategy is updated according to the specified method, so that the deep reinforcement learning strategy's evaluation results of the environment are closer to the actual environment, and the output action value can obtain more reward value. Step 7: Loop execution Repeat steps 4 to 6 to continuously update the experience to update the deep reinforcement learning network and improve the performance of the deep reinforcement learning network. As the cycle progresses, the performance of the deep reinforcement learning network continues to improve. The cycle can be manually terminated or stopped when the specified time or performance is reached to ensure the learning effect.

2. The multi-manipulator path planning method combining improved artificial potential field method and deep reinforcement learning according to claim 1, characterized in that: In step 3, the method for generating action values ​​in expert data is a global path planning method.

3. The multi-manipulator path planning method combining improved artificial potential field method and deep reinforcement learning according to claim 2, characterized in that: In step 3, the global path planning method is the BI-RRT* algorithm.

4. The multi-manipulator path planning method combining improved artificial potential field method and deep reinforcement learning according to claim 1, characterized in that: In step 5, the method adopted is the local path planning method.

5. The multi-manipulator path planning method combining improved artificial potential field method and deep reinforcement learning according to claim 4, characterized in that: In step 5, the local path planning method is to adopt the artificial potential field method.

6. The multi-manipulator path planning method combining improved artificial potential field method and deep reinforcement learning according to claim 1, characterized in that: In step 5, the reward function used is a function related to the gravitational potential field function in the artificial potential field method.

7. The multi-manipulator path planning method combining improved artificial potential field method and deep reinforcement learning according to claim 1, characterized in that: In step 6, the deep reinforcement learning method used is an algorithm based on the actor-critic architecture.

8. The multi-manipulator path planning method combining improved artificial potential field method and deep reinforcement learning according to claim 7, characterized in that: In step 6, the algorithms based on the actor-critic architecture are the deep deterministic policy gradient algorithm DDPG algorithm and the soft actor-critic algorithm SAC.

Images