Three-dimensional environment multi-agent path planning method based on deep reinforcement learning

Abstract

The invention relates to the technical field of path planning, in particular to a three-dimensional environment multi-agent path planning method based on deep reinforcement learning, which adopts a near-end strategy optimization algorithm and expands the algorithm to a multi-agent system, and by designing an independent strategy network and a central value network, a multi-agent path is planned. Cooperative work and barrier-free path planning of multiple agents are realized, and the problem of slow convergence caused by complex environment is solved. Moreover, a long-short-term memory network is introduced into a strategy network, and historical state information is modeled, so that an intelligent agent can better capture a time sequence dependency relationship in an environment, and the path planning capability of the intelligent agent in a complex environment is improved; an incremental collision penalty reward mechanism based on dynamic distance perception is designed in a reward function, radar ranging is simulated through a kinematic model, and an obstacle avoidance decision is optimized in real time. According to the method, the multi-agent collaborative obstacle avoidance capability of the agents in a complex three-dimensional environment is remarkably improved, and the method has relatively high efficiency and adaptability.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of path planning, and particularly relates to a three-dimensional environment multi-agent path planning method based on deep reinforcement learning. BACKGROUND

[0002] In existing research, path planning is widely applied to various research fields such as military, agriculture, robot technology, and the like, including but not limited to search and rescue, and the like. Despite this, achieving automatic obstacle avoidance of agents is still a thorny problem for researchers. The purpose of path planning is to determine a collision-free optimal trajectory of an agent from an initial position to a final destination. This requires researchers to ensure an efficient and safe obstacle avoidance path while complying with specific constraints. Advanced path planning algorithms can maximize the efficiency of the agent to reach the destination while significantly improving the safety and reliability of unmanned aerial vehicle operation.

[0003] In the field of path planning, traditional algorithms such as A* search, Dijkstra algorithm, and rule-based heuristic methods have shown good performance in known or structured environments. However, these methods are inefficient in handling high-dimensional spaces and complex environments, and are difficult to adapt to dynamic changes in the environment. Li et al. proposed an improved particle swarm algorithm combined with the strategy of genetic algorithm to solve the local optimal problem in the problem of unmanned aerial vehicle path planning. However, this algorithm still faces problems such as only aiming at single unmanned aerial vehicle path planning, not involving conflict avoidance in multi-unmanned aerial vehicle cooperative planning, and high computational complexity. In order to improve the problem that the grey wolf optimization algorithm is easy to fall into local optimum in path planning problem, Liu et al. proposed a heuristic mutation rule composed of triangles formed by local path segments. This method improves the obstacle avoidance ability of local paths, but the complexity of the algorithm is high, and a smoothing operator is not introduced. It can be seen that the computational cost of intelligent optimization algorithms is relatively high, and the performance of the algorithm depends to a great extent on the parameter setting, so that their efficiency and stability in practical application become limiting factors.

[0004] With the development of deep learning, the modeling capability of deep reinforcement learning (DRL) in high-dimensional state space is widely applied to multi-agent path planning problems. Deep reinforcement learning interacts with the environment, constantly learning the optimal strategy through interaction with the environment, so as to adapt to the complex and changeable environment autonomously without prior knowledge. In existing research, Chen et al. proposed an intelligent ship path planning algorithm based on Q-learning, which solved the shortcomings of traditional path planning algorithms in considering the dynamic characteristics of ships and complex navigation rules. Du et al. proposed a real-time local path planning algorithm QR-DQN, which solved the local path planning and obstacle avoidance in complex environments. However, QR-DQN is a value-based reinforcement learning algorithm that can only handle discrete action space problems, so it cannot fully match the characteristics of continuous motion of mobile agents in actual applications. Liu et al. proposed a robot obstacle avoidance motion planning framework based on deep reinforcement learning, which realized efficient obstacle avoidance trajectory generation in complex environments by designing a priori guided actor-critic structure, an extended double memory sampling mechanism, and a composite obstacle avoidance reward system, significantly improving the adaptability and training efficiency of robots in new scenarios. However, this framework does not involve multi-robot obstacle avoidance and interaction. Gao et al. proposed a method called ARL-MADDPG, which only extracts key information of adjacent unmanned surface vehicles and obstacles, uses the latest experience priority replay mechanism, and introduces residual structures and preprocessing networks into the critic network, making the algorithm perform well in scalability and training efficiency for multi-UAV cooperative path planning. Xu et al. proposed an algorithm called PPDC (Path Planning and Dynamic Collision Avoidance), which follows the international maritime collision avoidance rules while solving the challenge of dynamic obstacle avoidance for unmanned surface vehicles in complex marine environments. Wang et al. proposed a cooperative obstacle avoidance decision-making method based on MADRL, which uses an improved network update mechanism to realize the extended application of the DRQN method in multi-ship cooperative obstacle avoidance. Zhang et al. improved the PPO algorithm using post-experience replay to solve the path planning problem of mobile robots. Fan et al. proposed an intelligent collision avoidance algorithm NPD3QNU based on deep reinforcement learning, which integrates noise networks, priority experience replay, and double architecture optimization to significantly improve the autonomous collision avoidance capability of unmanned surface vehicles in complex marine environments while ensuring compliance with international maritime collision avoidance rules. Chu et al. proposed an improved double deep Q-network path planning algorithm that is resistant to ocean current interference, which realizes real-time obstacle avoidance of underactuated autonomous underwater vehicles in unknown environments through dynamic reward functions and NURBS smoothing techniques.

[0005] In summary, most of the existing researches are based on a two-dimensional environment for modeling. In the two-dimensional environment, the motion control of the agent is relatively simple, the state space and action space dimensions are low, and the path planning task is easy to model and has high training efficiency. However, the two-dimensional environment has significant simplification assumptions, and it is difficult to truly restore the dynamic complexity in actual application scenarios. Taking the unmanned aerial vehicle cluster and air formation as an example, the running space is mostly three-dimensional space, involving more dimensions, more complex obstacle avoidance logic and path optimization strategies. Therefore, extending the path planning algorithm to a three-dimensional environment not only has stronger practical significance, but also puts higher requirements on the generalization ability, perception modeling ability and computing efficiency of the algorithm.

[0006] In a three-dimensional environment, due to the additional spatial dimension, the path planning of multiple agents becomes more complex. Compared with a two-dimensional plane, the path planning problem in a three-dimensional environment involves more stringent control of height and needs to face more difficult obstacle avoidance and collaborative decision-making problems. Moreover, due to the complexity of the three-dimensional environment, the state space of the agent increases, the obstacle avoidance and navigation ability of the agent is limited, and the training difficulty increases. In addition, as the number of agents increases, the size of the joint action space grows exponentially, which will greatly increase the training time.

[0007] Therefore, how to improve the collaborative obstacle avoidance ability of multiple agents in a complex three-dimensional environment and shorten the training time is a problem to be solved in the prior art. SUMMARY

[0008] To this end, the technical problem to be solved by the present application is to overcome the problem of reduced collaborative obstacle avoidance ability of multiple agents in a complex three-dimensional environment and increased training time in the prior art.

[0009] To solve the above technical problems, the present application provides a three-dimensional environment multi-agent path planning method based on deep reinforcement learning, comprising: constructing an independent policy network for each agent: the policy networks of all agents have the same structure, and both take the historical sequence of the local observation space of the current time step as input and take the execution action of the current time step as output; the structure of the policy network includes a long short-term memory network and three fully connected networks connected in turn; In the training phase, based on the proximal policy optimization algorithm, the policy networks of multiple agents are trained through multiple rounds; one round of training of the policy networks of multiple agents includes: each agent executes multiple time steps in the environment to obtain the execution action and immediate reward of each agent at each time step; Each agent constructs an interaction dataset of the agent by taking the global state space of the corresponding time step, the execution action of the agent, and the immediate reward of the agent as a sample; inputs the interaction dataset of the agent into the center value network, calculates the objective function of the agent, and updates the parameters of the policy network of the agent based on the objective function of the agent; In the execution phase, each agent respectively acquires a historical sequence of the corresponding local observation space at each time step, inputs the trained policy network, and obtains the execution action of the agent at the next time step.

[0010] Preferably, the long short-term memory network in the policy network comprises 128 neurons, and each of the three fully connected networks comprises 512 neurons.

[0011] Preferably, the global state space comprises the three-dimensional position and speed of each agent, and the minimum distance between each agent and the obstacle.

[0012] Preferably, the one round comprises that all agents reach the target point from the starting point, or there is a collision between the agent and the obstacle, or there is an agent exceeding the boundary.

[0013] Preferably, the immediate reward comprises a distance reward, a collision penalty, an out-of-boundary penalty, a task completion reward, and a cooperation reward.

[0014] Preferably, the distance reward is defined as: ; wherein, is the distance reward, is the distance of the agent from the target point at the time step t and the time step t-1, is a preset constant.

[0015] Preferably, the collision penalty is defined as: ; wherein, is the collision penalty, is the minimum distance between the agent and the obstacle, is a preset safety distance threshold between the agent and the obstacle, , , and are all preset constants.

[0016] Preferably, the out-of-boundary penalty is defined as: if the coordinate position of the agent exceeds the preset environment range, the agent is given an out-of-boundary penalty ; otherwise, the agent is given an out-of-boundary penalty ​.

[0017] Preferably, the task completion reward is defined as: if the agent reaches a neighborhood centered on the target position with a preset distance as the radius, the agent is given a task completion reward ; otherwise, the agent is given a task completion reward .

[0018] Preferably, the coordination reward is defined as: if all agents reach a neighborhood centered on the target position with a preset distance as the radius, each agent is given a coordination reward ; otherwise, each agent is given a coordination reward .

[0019] The above technical solutions of the present application have the following beneficial effects compared with the prior art: The three-dimensional environment multi-agent path planning method based on deep reinforcement learning disclosed by the present application takes the historical sequence of the local observation space at the current time step as input, and uses a long short-term memory network to build a policy network to model the historical sequence of the local observation space, so that the agent can better capture the time sequence dependence in the environment, thereby making more accurate and stable decisions under partial observable conditions, significantly improving the path planning ability of the agent in a complex environment, and especially showing stronger adaptability and robustness when facing obstacle avoidance and multi-agent coordination tasks. Furthermore, the present application extends the proximal policy optimization algorithm to multi-agent, adopts a centralized training and decentralized execution framework, so that all agents share the same central value network during the training process, thereby enabling efficient learning using global information and reducing training time.

[0020] Further, the present application further gives the agent a collision penalty and an out-of-bound penalty at the end of each round, as well as a single-agent task completion reward and an all-agent task completion reward, based on the immediate reward of the action performed at each time step, effectively improving the agent's perception and response ability to dynamic changes in the environment and strengthening its decision-making level in complex scenarios.

[0021] In summary, the multi-agent path planning method proposed by the present application not only improves the efficiency of path planning, but also significantly reduces the collision risk with obstacles when multiple agents work together, and improves the task completion rate. BRIEF DESCRIPTION OF DRAWINGS

[0022] In order to make the content of the present application easier to be clearly understood, the present application will be further described in detail below according to specific embodiments of the present application and in conjunction with the drawings, in which: Figure 1 is a flowchart of a three-dimensional environment multi-agent path planning method based on deep reinforcement learning of the present application; Figure 2 is a schematic diagram of distance calculation between agent and obstacle, wherein Figure 2 (a) in (a) is a schematic diagram of agent lower than obstacle height, Figure 2 (b) in (b) is a schematic diagram of agent higher than obstacle height; Figure 3 is a schematic diagram of collision between agent and obstacle, wherein Figure 3 (a) in (a) is a schematic diagram of collision, Figure 3 (b) in (b) is a schematic diagram of horizontal projection of collision; Figure 4 is a structural diagram of strategy network; Figure 5 is a structural diagram of long short-term memory network; Figure 6 is a structural diagram of multi-agent near-end strategy optimization algorithm; Figure 7 is a cumulative reward comparison curve diagram of four algorithms; Figure 8 is a smooth curve diagram of cumulative reward comparison of four algorithms; Figure 9 is a test scene schematic diagram of the embodiment of the application, wherein Figure 9 (a) in (a) is a schematic diagram of sparse environment, Figure 9 (b) in (b) is a schematic diagram of medium density environment, Figure 9 (c) in (c) is a schematic diagram of high density environment; Figure 10 is a path planning result of three agents in a complex obstacle environment using the method of the application, wherein Figure 10 (a) in (a) is an orthographic view, Figure 10 (b) in (b) is a plan view; Figure 11 is a path planning result of five agents in a complex obstacle environment using the method of the application, wherein Figure 11 (a) in (a) is an orthographic view, Figure 11 (b) in (b) is a plan view. DETAILED DESCRIPTION

[0023] The application will be further described below in conjunction with the drawings and specific embodiments, so that those skilled in the art can better understand the application and implement it, but the embodiments are not limiting to the application.

[0024] In multi-agent path planning research, a common challenge is that agents often cannot observe all information in the environment. This invention models the research object as a Decentralized Partially Observable Markov Decision Process (Dec-POMDP). POMDP is a mathematical framework for sequential decision-making problems in uncertain environments, effectively handling the uncertainty of environmental states caused by agents' limited observations. Dec-POMDP, in multi-agent collaborative scenarios, allows multiple agents to make independent decisions based on their respective local observations and achieve collaborative goals by sharing global rewards. This modeling approach more realistically reflects the characteristics of information-constrained and decentralized decision-making in multi-agent systems.

[0025] The problem definition of Dec-POMDP mainly consists of seven parts, namely: .in, A set of intelligent agents is usually denoted as ; It is a collection of global state spaces; This represents the joint action space, where each agent i has its own action space. ; It is the global reward function, representing the state space. All agents perform joint actions The instant reward obtained afterward; It is the state transition probability function , indicating in the global state space Next action Then, it transitions to the next state with a certain probability. ; It is a joint observation space, where each agent i can only obtain a local observation space. This reflects its local perception of the environment; It is a discount factor used to weigh short-term and long-term returns. The closer this value is to 1, the more the agent tends to consider long-term returns.

[0026] Therefore, the research problem of this invention is modeled as follows: at each time step t, each agent i, based on its local observation space... Through strategy Independently select and execute actions The joint action of all intelligent agents will be based on probability Transition to the next state and receive global rewards. Meanwhile, each agent receives a new observation local observation space . Thus, the goal of the agent is to maximize the cumulative expected return, defined as .

[0027] Referring to FIG. 1, the present application provides a three-dimensional environment multi-agent path planning method based on deep reinforcement learning, comprising: Figure 1 S1: three-dimensional modeling of the multi-agent environment; S2: defining the state space, action space and reward of the multi-agent; S3: constructing an independent policy network for each agent: the policy network structures of all agents are the same, and all take the historical sequence of the local observation space at the current time step as input and take the execution action at the current time step as output; the structure of the policy network includes a long short-term memory network and three fully connected networks connected in turn; S4: in the training phase, based on the proximal policy optimization algorithm, the policy network of the multi-agent is trained through multiple rounds; S5: in the execution phase, each agent respectively acquires the historical sequence of its corresponding local observation space at each time step, inputs the trained policy network, and obtains the execution action of the agent at the next time step.

[0028] In this embodiment, S1 specifically comprises: Deep reinforcement learning is applied to the path planning task of multi-agent, enabling it to autonomously navigate in a complex three-dimensional environment. The agent perceives environmental information by imitating a radar range finder and adjusts its own motion strategy accordingly to achieve obstacle avoidance and target point navigation. In this application, all obstacles are assumed to be cylinders and the agent is a UAV. In the three-dimensional Cartesian coordinate system, let the origin be O, and the position coordinates of the agent at the current time t be . Since it is a continuous state space, the motion direction of the agent is a vector defined in three-dimensional space. , and represent the motion classification along the x, y, z axis direction respectively. When , the agent moves along the positive square of the axis; otherwise, it moves along the negative direction.

[0029] In the three-dimensional obstacle avoidance problem, the distance calculation between the agent and the cylindrical obstacle needs to consider its spatial geometric characteristics. The complexity of the three-dimensional environment is reflected in the calculation of the distance between the agent and the obstacle. The three-dimensional coordinates of the center of the bottom surface of the obstacle cylinder are set as (this embodiment defaults to ), the bottom surface radius is , and the obstacle height is​ The obstacles include environmental obstacles and other intelligent agents.

[0030] Referring to Figure 2 , the distance between the intelligent agent and the obstacle is discussed in two cases: Referring to Figure 2 (a), if the z-coordinate of the intelligent agent at the current time step is less than or equal to the height of the obstacle , only the distance between the intelligent agent and the obstacle in the horizontal direction needs to be considered, and the distance between the intelligent agent and the obstacle is , wherein is the distance between the intelligent agent and the obstacle in the horizontal direction, and are the x-coordinate and y-coordinate of the intelligent agent at time step t, respectively; if , the horizontal projection of the intelligent agent is located within the cross-sectional circle of the obstacle, and there is a risk of collision, as shown in Figure 3 (a), and the horizontal projection is shown in Figure 3 (b).

[0031] Referring to Figure 2 (b), if the z-coordinate of the intelligent agent at the current time step is greater than or equal to the height of the obstacle , not only the distance between the intelligent agent and the obstacle in the horizontal direction needs to be considered, but also the distance of the intelligent agent in the vertical direction, and the distance between the intelligent agent and the obstacle is , wherein is the z-coordinate of the intelligent agent at time step t.

[0032] The segmented modeling method of the three-dimensional environment in this embodiment not only guarantees the calculation efficiency, but also strictly satisfies the geometric constraint conditions of three-dimensional obstacle avoidance.

[0033] In this embodiment, S2 specifically includes: S21: defining a state space.

[0034] Multi-agent path planning not only needs to interact with the environment, but also needs to share certain information with other intelligent agents, resulting in a large amount of memory occupation and time consumption. This collaborative process usually involves a large amount of state space search and information exchange, resulting in significant memory occupation and computation time consumption. In particular, in a three-dimensional environment, the complexity of path planning is further increased because the intelligent agent needs to make decisions in more degrees of freedom, which causes the dimension of the state space to expand dramatically, therefore, it is very important to reasonably design the state space.

[0035] The agent needs to receive part of the information in the environment to achieve obstacle avoidance to reach the destination, including: the state information of the agent itself and the information obtained by interacting with the environment. The information of the agent itself refers to the coordinates of the agent itself and the current action trend. First, since the environment is unknown, the agent cannot obtain the global information of the environment, so the position of the agent is defined as , which represents the coordinates of the three directions of the coordinate system established by the agent in the environment at time step t. Since it is a continuous state space, according to the relationship between distance and speed defined in the kinematic model, the velocity of the agent in three directions at a certain time step can be represented as to represent the state of all agents approaching the end point.

[0036] Most importantly, the embodiment uses the minimum distance to the surrounding obstacles (including obstacles and other agents) in the state space to represent the information of interaction with the surrounding environment as part of the state space. This design simulates a radar range finder observation method, and the agent only perceives the nearest obstacle. Simplify the complex environmental information into a key indicator, thereby reducing the complexity of the state space. By focusing on the distance of the nearest obstacle, the size of the input data is reduced, and the training speed is accelerated.

[0037] Therefore, the global state space of the embodiment includes: the three-dimensional position and velocity of each agent, and the minimum distance between each agent and the obstacle.

[0038] S22: Define the action space.

[0039] In this embodiment, for a three-dimensional space, since the proposed multi-agent motion model realizes continuous motion control of the agent based on acceleration, the action to be performed is defined as the change in acceleration of the agent in three directions, and the action to be performed is controlled between -1 and 1. Therefore, the action space of the agent represents the acceleration of the agent in three directions within the three-dimensional space, which is represented as: ;

[0040] Therefore, the design of the state space and the action space of the embodiment is based on the kinematic model, which conforms to the relationship between distance, velocity, and acceleration in the kinematic model, so the update of the agent state is represented by the following formula: ; .

[0041] This modeling method fully reflects the dynamic evolution process of the motion state of the agent in the continuous space, and uses a small number of variables to make the construction of the state space and the action space truly reflect the physical behavior of the agent in the actual three-dimensional environment.

[0042] Through the above design, the behavior strategy of the agent not only has good physical interpretability, but also can effectively adapt to continuous motion tasks based on acceleration control in real or simulated environments, thereby providing a reasonable modeling basis for subsequent strategy optimization.

[0043] S23: defining a reward.

[0044] The reward set in this embodiment includes: distance reward, collision penalty, out-of-boundary penalty, task completion reward, and cooperation reward.

[0045] The learning goal of the agent is to avoid obstacles in the existing three-dimensional environment and safely reach the destination. The key condition for reaching the target is to design a feasible and efficient reward function. The reward function defines the learning goal in the multi-agent environment, and its design directly affects the learning performance of the mobile agent studied. Specifically, the reward function is actually a quantitative description of the learning task, which can guide the network training to develop in the desired reward direction.

[0046] The distance reward is defined as: ; wherein, is the distance reward, and are the distances of the agent from the target point at the current time step t and time step t-1, respectively, is a preset constant. The distance reward represents the distance shortened by the agent after taking action in a time step from the target point. Such incremental reward mechanism based on distance change can encourage the UAV to approach the destination, while preventing the agent from moving away from the target point.

[0047] In addition, the most critical aspect of the agent exploring a complex environment is to ensure its safety. In practical applications, the consequences of collision with obstacles or neighboring UAVs are not only the loss of resources, but also the inability to guarantee safety. In addition, when the agent performs actions in a three-dimensional environment, it will be penalized when it approaches any obstacle or other agent. In order to improve the safety guarantee of the agent in a complex environment, this embodiment proposes a dynamic distance-aware incremental collision penalty reward mechanism (Dynamic Distance-aware Incremental Penalty, DDIP), which defines the collision penalty as: ;

[0048] wherein, is the collision penalty, is the minimum distance between the agent and the obstacle, a safety distance threshold preset between the agent and the obstacle, , , and are all preset constants.

[0049] When the agent approaches the obstacle to a certain critical value, the collision penalty will increase linearly; when the agent is very close to the obstacle, the collision penalty will grow exponentially rapidly; if a collision occurs, a penalty will be given.

[0050] In three-dimensional motion planning, in order to guide the agent to converge efficiently to the target position, the embodiment sets a critical boundary in each coordinate axis direction. When the agent deviates too much from the expected path direction, the system will impose a penalty term due to deviation from the target, thereby effectively constraining the exploration range of the agent, avoiding invalid divergence, and ensuring that the motion trajectory converges quickly and stably to the target area. The out-of-bound penalty is defined as: if the coordinate position of the agent exceeds the preset environment range, the agent is given an out-of-bound penalty ; otherwise, the agent is given a task completion reward . The formula is as follows: ; wherein, , , is a preset boundary coordinate.

[0051] In the reinforcement learning framework, when a single agent enters a neighborhood with a target position as the center and a radius of to complete the designated task, it will be given a significant positive reward signal, i.e., a task completion reward . The embodiment defines the task completion reward as: if the agent reaches a neighborhood with the target position as the center and a preset distance as the radius, the agent is given a task completion reward ; otherwise, the agent is given a task completion reward .

[0052] In particular, when all agents successfully arrive at the target neighborhood, an additional cooperative reward will be issued to reinforce the cooperative behavior of the multi-agent system. The embodiment defines the cooperative reward as: if all agents reach a neighborhood with the target position as the center and a preset distance as the radius, each agent is given a cooperative reward ; otherwise, each agent is given a cooperative reward . The formula is as follows: ; ​

[0053] Total reward may be expressed as: .

[0054] In the subsequent training phase, the instant reward of each time step is calculated with the total reward.

[0055] In order to adapt to the cooperative path planning of multiple agents, the proximal policy optimization algorithm is improved, and a multi-agent proximal policy optimization algorithm (Multi-agent PPO, MAPPO) is proposed. However, in a partially observable environment, the decision of an agent only depends on the local observation at the current time, which leads to the fact that the policy learning is easy to fall into an unstable or suboptimal solution. At the same time, the cooperation between agents is often modeled as independent of each other, lacking necessary information sharing and historical memory.

[0056] Considering that the behavior and observation of each agent and the state in the environment exhibit similar time series characteristics in reinforcement learning, the long short-term memory network (Long Short-Term Memory Network, LSTM) has good performance in processing time series. In order to solve the above problems, the long short-term memory network is introduced into the policy network in the MAPPO framework to enhance the time series modeling ability of the agent. Since the perception information of multiple agents in a three-dimensional complex environment is limited, the state of a single time step may not be sufficient to completely describe the environment state, thereby affecting the policy learning. LSTM can model the time series of historical state information by maintaining an internal memory unit, so that the agent not only depends on the current observation, but also integrates the information of multiple time steps in the past to make more reasonable decisions, thereby enhancing the perception ability of the agent to the environment, and improving the performance of path planning, especially in high-dimensional and dense obstacle environments.

[0057] Specifically, referring to Figure 4 , the embodiment constructs an independent policy network for each agent: the structure of the policy network of all agents is the same, and the historical sequence of the local observation space of the current time step is input, and the execution action of the current time step is output; the structure of the policy network includes a long short-term memory network and three fully connected networks connected in turn. The historical sequence of the local observation space of the current time step is a time series composed of the local observation space of the current time step and the previous 29 time steps.

[0058] In the task of agent navigation and obstacle avoidance, the observation and action of each agent exhibit a continuous time series pattern. The structural diagram of the LSTM network is shown in Figure 5As shown, the embodiment inputs the historical state sequence of each agent into LSTM as a time sequence feature extraction module in the policy network, processes the observation sequence of the past 30 time steps through the hidden state dimension of the 128-layer LSTM unit, and continuously encodes the interaction trajectory of the agent and the complex obstacle. In the environment design, the obstacle information is sorted according to the distance from the agent and input, and the rear end of the sequence often represents closer obstacles. The LSTM learns this spatial priority through its gating mechanism (such as the input gate and the forgetting gate) - the later the information is, the easier it is to retain, and therefore it can pay more attention to the current potential threat.

[0059] The time sequence features output by the LSTM layer are fused through a 3-layer 512-dimensional fully connected network, which uses a ReLU activation function to enhance feature extraction capability, and finally generates a unified spatio-temporal representation. Action decision is then output through the policy layer. LSTM integrates multi-time sequence information through the internal memory state, thereby modeling the time sequence features and making up for the information loss caused by partial observation.

[0060] In addition, under the Dec-POMDP framework, a single agent can only perceive the environment based on its own local observation space , and cannot directly access the global state. To deal with this partial observability, the policy network of the present application takes the observation sequence of the agent as input, encodes the observation of the past time steps through the hidden state of the LSTM, and thereby implicitly estimates the state transition function of the environment. This time sequence modeling mechanism enables the network to capture potential dynamic rules from historical observations and form an internal representation of the system state, which is equivalent to introducing an approximate modeling process for and in policy learning. Therefore, LSTM not only realizes time sequence feature extraction, but also reflects the internal relationship between state transition and partial observation in the Dec-POMDP framework at the structural level. This mechanism significantly enhances the response capability of the agent in performing obstacle avoidance and coordination tasks in complex environments, while improving the stability and reliability of system operation.

[0061] The MAPPO algorithm proposed in the present application extends the proximal policy optimization algorithm to multi-agent systems, and realizes the collaborative work and obstacle-free path planning of multi-agents by designing shared policy and value networks. The structural diagram of the MAPPO algorithm is as shown in Figure 6 . In the training process, all agents share a central value network to evaluate the execution actions of each agent. Each agent selects an execution action according to the local observation at the current time step and its corresponding policy network. The agent obtains the corresponding immediate reward after executing the action. The central value network receives the global state space and the execution actions of all agents at the time step as input to evaluate the execution actions of each agent.

[0062] Specifically, in S4, the policy network of multiple agents is trained in one round, including: Each agent performs multiple time steps in the environment, obtaining the execution action of each agent at each time step and the immediate reward ; Each agent takes the global state space of the corresponding time step , the execution action of the agent , and the immediate reward of the agent as a sample to construct the interaction data set of the agent; the interaction data set of the agent is input into the central value network to calculate the objective function of the agent, and the parameters of the policy network of the agent are updated based on the objective function of the agent.

[0063] Specifically, in the centralized training phase of multi-agent reinforcement learning, the system can access the global state space , joint action , and global reward . Each agent 's policy network outputs an execution action according to the historical sequence of its own local observation space , which is expressed in the formula as: ; Wherein, is the policy network of the i-th agent, is the parameter of the policy network of the i-th agent, is the execution action of the i-th agent at time step t, is the historical sequence of the local observation space of the i-th agent at time step t.

[0064] The central value network uses global information to evaluate the state-action value: ; Wherein, is the parameter of the central value network, is the discount factor.

[0065] The training target is based on the expected return maximization principle of reinforcement learning: ;

[0066] And the parameters of the policy network are updated by the policy gradient method: ; Wherein is the advantage function of the i-th agent, which is used to guide the optimization direction of the policy.

[0067] During the training process, the central value network can access the global state and joint actions to achieve centralized value evaluation and stable gradient update. The LSTM module in the policy network is used to process the local observation history sequence and extract time-related features, thereby forming a more robust policy representation in partially observable environments.

[0068] The goal of policy update is to update the policy by maximizing the expected return of the policy. However, directly using policy gradient update is prone to large policy changes. Therefore, the proximal policy optimization algorithm introduces a clipping objective function to limit the magnitude of policy changes in each update. MAPPO also follows the same principle and applies it to each agent's policy to ensure the overall learning stability and collaboration of the system.

[0069] In this embodiment, the interaction data set of the agent is input into the central value network, and the objective function of the agent is calculated, and the formula of the objective function is: ; wherein, is the ratio of the current policy to the old policy, is the estimated value of the advantage function, which is used to measure the advantage of performing action in state under the state is the clipping magnitude. The essence of the proximal policy optimization algorithm is to limit the degree of policy update to avoid substantial policy changes, which can thus make the training more stable.

[0070] To calculate the advantage function, this embodiment uses the temporal difference error to measure the difference between the predicted value of the current state value function and the actually observed reward. The mathematical expression of the temporal difference error is: ; wherein, is the temporal difference error at time step t, is the immediate reward obtained at time step t, is the discount factor, which is used to control the weight of long-term return. represents the value function of the global state space at time step t, i.e. the expected cumulative return of the state under the current policy.

[0071] is the Monte Carlo approximation of the advantage function, which is used to measure the advantage or disadvantage of a certain action relative to the average action under the current policy, which can be expressed as: .

[0072] To further stabilize the estimation of the policy gradient and improve the sample efficiency, the advantage function in the above formula is replaced by a generalized advantage estimation in the embodiment, and the formula is as follows: ; wherein, is the advantage function obtained at each time step after time step t, is a key hyperparameter that adjusts the bias-variance trade-off and is used to integrate the advantage estimation values of multiple time steps.

[0073] Specifically, in S5, in the execution phase, each agent respectively obtains the historical sequence of its corresponding local observation space at each time step, inputs the trained policy network, and obtains the execution action of the agent at the next time step, which is expressed by the formula as follows: .

[0074] In the execution phase, each agent only relies on the historical sequence of its own local observation to make decisions, and no longer accesses the global state or the private information of other agents. At this time, the hidden state of the LSTM continues to maintain the time continuity as the internal memory of the historical observation, so as to make effective action decisions without global visibility.

[0075] The MAPPO algorithm proposed in the application is under the centralized training and decentralized execution framework, for realizing the stable update of the policy network and the value network, and adopting a policy optimization method based on the timing difference error, which combines the advantage function estimation and the generalized advantage estimation to improve the sample efficiency, and constrains the policy update range through the clipping target function of the proximal policy optimization algorithm, so as to ensure the stability and collaboration of the multi-agent system in the training process.

[0076] The three-dimensional environment multi-agent path planning method based on deep reinforcement learning provided by the application takes the historical sequence of the local observation space at the current time step as input, and constructs a policy network for modeling the historical sequence of the local observation space by using a long short-term memory network, so that the agent can better capture the timing dependence relationship in the environment, thereby making more accurate and stable decisions under partial observability, and significantly improving the path planning ability of the agent in a complex environment, especially showing stronger adaptability and robustness when facing obstacle avoidance and multi-agent collaborative tasks. Moreover, the proximal policy optimization algorithm is extended to multi-agent in the application, and a centralized training and decentralized execution framework is adopted, so that all agents share the same central value network in the training process, thereby enabling efficient learning by using global information and reducing the training time.

[0077] Further, the present application further gives the agent a collision penalty and an out-of-bound penalty at the end of each round, as well as a reward for a single agent completing the task and a reward for all agents completing the task, based on the immediate reward of the action performed by the agent at each time step, effectively improving the agent's perception and response ability to dynamic changes in the environment and strengthening its decision-making level in complex scenarios.

[0078] In summary, the multi-agent path planning method proposed by the present application not only improves the efficiency of path planning, but also significantly reduces the collision risk with obstacles when multiple agents work together, and improves the task completion rate.

[0079] In order to verify the effect of the method proposed in the present application, the following experiments were carried out in this embodiment.

[0080] During the training process, Python 3.12, Pytorch, Ray and other libraries were used for training to verify the effectiveness of the DDIPL-MAPPO algorithm. The starting point and the end point of all agents were randomly generated within a certain range. Among them, the initial position will not coincide with the position of the obstacle. All agents will make decisions according to the observed local information and other agents, calculate the position and speed of the next state according to the updated acceleration of the strategy, and the task is terminated when all agents reach the target position or illegal behavior of the agent occurs. Illegal behavior includes the agent exceeding the boundary or the collision between the agent and the obstacle. All training is run on Windows 11 with 16 GB RAM and Intel(R) Core(TM) i7-14650HX CPU.

[0081] The setting of hyperparameters in the network is adjusted through a series of repeated experiments, and the best hyperparameters are used to deploy the model. The hyperparameters used during training are shown in Table 1. The agent completes the training in a 500x500x200 three-dimensional environment. The parameter settings of the reward function part are as follows: , , , , , .

[0082] Table 1, parameter settings

[0083] To verify the effectiveness of this invention in multi-agent path planning and obstacle avoidance tasks, this embodiment compares the proposed multi-agent proximal optimization algorithms based on dynamic distance-aware incremental penalty and LSTM (DDIPL-MAPPO), DDIP-MAPPO (DDIP based on MAPPO), L-MAPPO (LSTM based on MAPPO), and the traditional MAPPO algorithm in a simulated environment.

[0084] This embodiment employs two reward design methods: the incremental collision penalty reward mechanism based on dynamic distance perception proposed in this invention, and the collision fixed reward function R. base Set to: .

[0085] To verify the effectiveness of the incremental collision penalty-reward mechanism based on dynamic distance perception proposed in this invention, this embodiment applies the two reward mechanisms to two different strategy optimization frameworks, L-MAPPO and MAPPO, respectively, forming four sets of control experiments. The aim is to verify the effectiveness of the DDIP mechanism itself from multiple dimensions and provide basic support for subsequent ablation experiments.

[0086] The cumulative reward of the four algorithms is compared to... Figure 7 As shown, Figure 8 For smoothing Figure 7 The result is determined by the reward value. In multi-agent cooperative obstacle avoidance and navigation tasks, the L-MAPPO algorithm with the DDIP reward mechanism can effectively learn from historical trajectories, conduct efficient exploration, and improve navigation efficiency. Compared with other combinations, DDIPL-MAPPO has a better convergence speed during training and eventually stabilizes in a higher reward range, demonstrating stronger learning efficiency and policy stability. The training process exhibits smaller fluctuations, and the agent can continuously and steadily improve policy quality.

[0087] Compared with MAPPO: In both the case of using collision fixed reward function and the case of introducing DDIP-based reward mechanism, the convergence speed of the LSTM-based strategy optimization method proposed in the present application is significantly better than that of the traditional MAPPO algorithm in the early stage of training. Specifically, the number of iterations required for L-MAPPO to reach the convergence state in the early stage is reduced by about 50 to 100 rounds compared with MAPPO, showing higher training efficiency. In addition, after convergence, the L-MAPPO algorithm is also better than MAPPO in the stability of strategy output, and can maintain a more stable strategy updating process, thereby reducing training shock, improving the repeatability and robustness of overall performance.

[0088] Compared with collision fixed reward function: In terms of performance, the agent strategy trained after introducing the DDIP reward mechanism performs better in multiple indicators compared with the traditional collision fixed reward function. Specifically, in terms of average cumulative return, the agent using the DDIP mechanism can stably maintain a high return interval of more than 75, while the agent using the collision fixed reward function has an average return of about 65. When facing more complex and strict obstacle avoidance penalty settings, DDIP can still maintain high adaptability and learning ability, with cumulative return improved by about 15% compared with the fixed reward scheme. This shows that the DDIP mechanism can effectively improve the agent's perception and response ability to environmental dynamics and enhance its decision-making level in complex scenarios.

[0089] In summary, the DDIPL-MAPPO algorithm proposed in the present application combines the high-performance advantages of the DDIP reward mechanism and the efficiency of the L-MAPPO strategy optimization method, and not only realizes faster convergence speed and more stable strategy performance in the multi-agent path planning task, but also shows strong generalization ability and environmental adaptability, with good practical application potential.

[0090] The present embodiment also selects the following indicators to evaluate the scalability, generalizability and quality of the navigation algorithm, and the specific explanations of the indicators are as follows:

[0091] a) Cumulative success rate: The proportion of individuals that have not collided and successfully reached the target point among all agents during the period from the start of training to the current round. This indicator not only reflects the overall performance of the agent in long-term training, but also intuitively reflects the speed of algorithm convergence. Higher success rate usually means that the algorithm has better stability and learning efficiency.

[0092] b) Cumulative collision rate: The percentage of the number of agents that failed the task due to collision with the environment or other agents during the period from the start of training to the current round. This indicator directly reflects the safety and coordination problems of the agent in the process of navigation or decision-making, and its numerical change can reveal the performance of the algorithm in obstacle avoidance and multi-agent collaboration. A higher collision rate usually indicates that the motion planning strategy of the agent has defects or the environmental adaptability is insufficient, and the algorithm needs to be further optimized to improve the robustness and reliability of the system. A lower collision rate usually means that the algorithm has better learning efficiency.

[0093] c) Cumulative out-of-bound rate: The proportion of individuals that failed the task due to deviating from the preset working area boundary among all agents during the period from the start of training to the current evaluation round. This indicator directly reflects the navigation accuracy problems of the agent in the process of path planning or motion control, and its numerical change can effectively evaluate the degree of compliance of the agent to spatial constraints. Generally, a higher out-of-bound rate will occur in the early stage, usually indicating that the agent is still in the learning stage and the boundary perception ability is defective, and the motion control strategy needs to be optimized. This indicator has important reference value for evaluating the reliability and environmental adaptability of the agent in a restricted environment.

[0094] d) Average reward: The mean value of the reward signal obtained by the agent during the training process after it tends to be stable. Specifically, the embodiment calculates the average reward value of the agent in the last 200 rounds of iteration after the reward curve tends to be stable. The high or low of this indicator directly reflects the performance level of the agent in navigation. A smaller average reward means a lower performance level.

[0095] The simulation environment used in this embodiment is randomly generated, and the size and position of the obstacles are completely different. These environments are used to simulate obstacles. In addition to the random scene, this embodiment also constructs three new test scenes to evaluate the performance of the trained model in unseen scenes, from easy to difficult, as shown in FIG. 9, where Figure 9 (a) in FIG. 9 is a schematic diagram of a sparse environment (large obstacle spacing, wide channel), Figure 9 (b) in FIG. 9 is a schematic diagram of a medium-density environment (obstacles are locally clustered and need to avoid obstacles frequently), Figure 9 (c) in FIG. 9 is a schematic diagram of a high-density environment (obstacles are closely arranged and the navigation space is extremely limited).

[0096] In each scenario, this embodiment takes the number of agents as an example. Under different numbers of agents and different multi-agent scenarios, the success rate, collision rate and failure rate proposed above are compared respectively, and the results are shown in Table 2. It is worth noting that in addition to the above three types of states, there are also individual agents that fail to complete the task within the specified number of steps during the experiment, which are not counted as a separate indicator in this embodiment.

[0097] Table 2, comparison of methods in different scenarios

[0098] According to the comparative experiment results shown in Table 2, under the same network structure conditions, the incremental collision penalty reward mechanism based on dynamic distance perception proposed in the application shows a significant advantage compared with the fixed reward function (R base ). Specifically, the safety of the agent navigation process is improved: the DDIPL-MAPPO algorithm proposed in the application shows the best obstacle avoidance performance in all test scenarios. In the sparse environment, the collision rate of the DDIPL-MAPPO method is 5.38%, which is reduced by 9.7% and 16.6% compared with the collision rate of 5.96% of the L-MAPPO using Rbase and the collision rate of 6.45% of the DDIP-MAPPO, respectively; in the medium density environment, the collision rate is as low as 1.51%. Using the L-MAPPO algorithm to optimize the strategy, the collision rate of the method using R base is 6.11%, which is reduced by 75.3%, verifying the improvement effect of the dynamic reward function on safety performance.

[0099] Since the evaluation range of the success rate indicator in this embodiment is from the start of training to the current round, the high success rate achieved by the agent in the complex environment not only means that the agent can achieve higher success rate and thus higher safety, but also can converge faster and more efficiently in the complex environment to complete the obstacle avoidance and navigation task. The spatiotemporal correlation modeling of the DDIP reward enhances the adaptability of the agent to the complex environment. And the addition of LSTM also makes the agent have a higher overall success rate.

[0100] The DDIPL-MAPPO algorithm proposed in the application shows the best overall success rate in all test scenarios. And has the ability to adapt to randomly different complex environments. The baseline algorithm performs poorly in gradually complex obstacle environments, with a gradually decreasing success rate. The success rate of the DDIPL-MAPPO method is as high as 89.45% in the medium density environment, and reaches 82.2% in the complex scenario in the high density environment.

[0101] In addition, the embodiment also designs comparative experiments under different numbers of agents to evaluate the adaptability and stability of the proposed algorithm when the multi-agent scale expands. The experimental results are shown in Table 3. With the number of agents increasing from 1 to 5, the complexity of the overall environment increases significantly, and the path conflict and obstacle avoidance challenge also intensify. Under this background, the performance of the traditional MAPPO algorithm in the multi-agent environment decreases significantly, especially when the number of agents reaches 5, the success rate is only 39.22%, which is much lower than the DDIPL-MAPPO (88.94%) proposed in the application. From the success rate, DDIPL-MAPPO achieves the optimal result in all multi-agent environment number settings, indicating that it has stronger path planning and collaboration ability. In addition, the collision rate and the out-of-bound rate further verify the robustness of DDIPL-MAPPO. In the 5-agent scenario, the collision rate of DDIPL-MAPPO is the lowest, only 2.76%, which is significantly better than MAPPO (11.24%) and other improved methods. In terms of out-of-bound rate, DDIPL-MAPPO also performs best, only 1.00%, which is more than half less than MAPPO (2.53%), indicating that it has higher trajectory controllability and safety in complex environments.

[0102] In summary, as the number of agents increases, the baseline algorithm is more likely to fall into local optimum, while the DDIPL-MAPPO algorithm proposed in the application still maintains stable and superior performance in the multi-agent high-density scenario, fully embodying its advantages in path planning and obstacle avoidance in complex dynamic environments.

[0103] Table 3, performance comparison of methods under different numbers of agents

[0104] To more clearly verify the effectiveness of each module in the proposed method, Table 4 presents the comparison of the average reward and the number of convergence iterations of each module. The DDIPL-MAPPO method shows significant improvement in average reward, reaching 230.81, which is about 12.5% higher than the version without using the DDIP mechanism and about 3.1% higher than the version without using the LSTM module. In addition, the convergence speed of DDIPL-MAPPO is also significantly faster than the other two methods, only 340 iterations are needed to converge, while the L-MAPPO algorithm without using the DDIP mechanism and the DDIP-MAPPO algorithm without adding the LSTM module need 400 and 480 iterations to achieve convergence, respectively.

[0105] Table 4, ablation experiment results: influence of DDIP mechanism and LSTM module on algorithm performance

[0106] Figure 10 The performance results of three agents using the method of the application to navigate in a complex obstacle environment are shown, wherein Figure 10 (a) in (a) is a front view, Figure 10 (b) in (b) is a top view. Figure 11 The performance results of five agents using the method of the application to navigate in a complex obstacle environment are shown, wherein Figure 11 (a) in (a) is a front view, Figure 11 (b) in (b) is a top view. It can be observed that each agent can autonomously avoid three-dimensional obstacles in the environment under the action of the guidance mechanism, and eventually successfully reaches the vicinity of the respective target area, embodying good collaborative obstacle avoidance and target convergence capability. The visualization results of the navigation process are shown in the figure, and due to the complexity of the environment, an additional top view is added to intuitively show the overall process of the agent starting from the starting point, dynamically adjusting the track, and finally reaching the vicinity of the target point, further verifying the effectiveness and stability of the proposed method in complex environments.

[0107] The application proposes a multi-agent path planning method in a three-dimensional environment based on deep reinforcement learning, which is used to solve multi-agent navigation and obstacle avoidance. By using a deep reinforcement learning strategy, the actions of each agent are determined to achieve the tasks of navigation and obstacle avoidance in complex environments. The application uses an improved MAPPO algorithm to optimize the strategy, and designs a reward function based on incremental collision of distance change. Compared with the traditional MAPPO algorithm and the fixed collision reward, the algorithm proposed in the application not only has faster convergence speed, but also has higher global success rate and lower global collision rate. The method of the application is systematically verified in a variety of differentiated scenarios, and the results show that the proposed method exhibits excellent cross-scene adaptability and generalization performance.

[0108] Those skilled in the art will appreciate that embodiments of the application can be provided as methods, systems, or computer program products. Accordingly, the application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) having computer-usable program code embodied therein.

[0109] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flow or blocks. Figure 1 one or more flow or blocks.

[0110] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart block or blocks. Figure 1 one or more flow or blocks. Figure 1 one or more flow or blocks.

[0111] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flow or blocks. Figure 1 Figure 1 one or more flow or blocks.

[0112] Obviously, the above-described embodiments are only examples for clarity of description and are not limiting on the embodiments. Based on the above description, one of ordinary skill in the art can further make other different forms of changes or modifications. Here, it is not necessary or possible to enumerate all the embodiments. The obvious changes or modifications derived therefrom are still within the protection scope of the present application.

Claims

1. A method for multi-agent path planning in a three-dimensional environment based on deep reinforcement learning, characterized in that, The application relates to a method for training a plurality of agents in an environment, comprising: constructing an independent policy network for each agent: the policy network of all agents has the same structure, and the policy network takes a historical sequence of a local observation space at a current time step as input and takes an execution action at the current time step as output; the structure of the policy network comprises a long short-term memory network and three fully connected networks connected in sequence; in a training phase, based on a proximal policy optimization algorithm, the policy network of the plurality of agents is trained through multiple rounds; training the policy network of the plurality of agents in one round, the process comprising: each agent performs a plurality of time steps in the environment to obtain an execution action and an immediate reward of each agent at each time step; each agent takes a global state space at the corresponding time step, the execution action of the agent and the immediate reward of the agent as a sample to construct an interaction dataset of the agent; the interaction dataset of the agent is input into the central value network to calculate the objective function of the agent, and the parameters of the policy network of the agent are updated based on the objective function of the agent; in an execution phase, each agent respectively obtains a historical sequence of a corresponding local observation space at each time step, inputs the trained policy network to obtain an execution action of the agent at the next time step.

2. The method of claim 1, wherein, The long short-term memory network in the policy network comprises 128 neurons, and each of the three fully connected networks comprises 512 neurons.

3. The method of claim 1, wherein, The global state space comprises three-dimensional positions and velocities of each agent and minimum distances between each agent and obstacles.

4. The method of claim 1, wherein, The one round comprises that all agents reach a target point from a starting point, or an agent collides with an obstacle, or an agent exceeds a boundary.

5. The method of claim 1, wherein, The immediate reward comprises a distance reward, a collision penalty, an out-of-boundary penalty, a task completion reward and a cooperation reward.

6. The method of claim 5, wherein, The distance reward is defined as: ； wherein, is a distance reward, and are distances of the agent from the goal point at time step t and time step t-1, respectively, is a preset constant.

7. The method of claim 5, wherein, The collision penalty is defined as: ； wherein, is a collision penalty, is a minimum distance between the agent and the obstacle, is a predetermined safety distance threshold between the agent and the obstacle, , , and are predetermined constants.

8. The method of claim 5, wherein, The out-of-bound punishment is defined as: if the coordinate position of the agent exceeds the preset environment range, the agent is given an out-of-bound punishment ; otherwise, the agent is given an out-of-bound punishment .

9. The method of claim 5, wherein, The task completion reward is defined as: if the agent reaches a neighborhood with the target position as the center and a preset distance as the radius, the agent is given a task completion reward ; otherwise, the agent is given a task completion reward .

10. The method of claim 5, wherein, The cooperative reward is defined as: if all agents reach the neighborhood centered on the target position with a preset distance as the radius, each agent is given a cooperative reward ; otherwise, each agent is given a cooperative reward .

Images