Layered collaborative deep reinforcement learning decision-making method for multi-agent path planning

Abstract

The invention discloses a hierarchical collaborative deep reinforcement learning decision-making method for multi-agent path planning, and belongs to the technical field of multi-agent path planning. The method comprises the following steps: constructing a PPO-based hierarchical collaborative decision-making model through a bottom-layer task allocation decision-making module and an upper-layer behavior decision-making module taking PPO as a framework; taking environment RGB image data, a task request and observation information as input of a layered collaborative decision model based on PPO; wherein the observation information comprises bottom layer observation information and upper layer observation information; and the optimal control strategy of the multi-agent system is obtained by continuously training the PPO-based hierarchical collaborative decision model. The method has the advantages that task resource allocation is efficient, the system state is stable, the execution time is shortened, and the robustness of the system is enhanced.

Description

Technical Field

[0001] The present invention relates to a hierarchical collaborative deep reinforcement learning decision-making method for multi-agent path planning, belonging to the technical field of multi-agent path planning. Background Art

[0002] Deep reinforcement learning (DRL) is an algorithm that combines deep learning with reinforcement learning to achieve end-to-end control from raw input to output. It optimizes strategies based on the current state and behavior of the agent to achieve the best performance while completing the task. Therefore, DRL is considered to be an effective method to solve the multi-agent path finding (MAPF) problem. MAPF involves multiple agents being able to find the best path to the target in the same environment while performing tasks. However, most of the existing MAPF research focuses on path planning itself, while ignoring the requirements related to task resource allocation, especially in warehousing environments. In such an environment, the agent first needs to accept the resource task allocation made by the system, and then plan the path to meet actual needs and improve overall efficiency. Current research often fails to solve this problem well, resulting in resource waste and inefficiency in practical applications. In addition, research based on multi-agent DRL also faces the challenge of unstable system state. Each agent has only a local field of view, and other agents are in a dynamic situation. The local information of each agent will change at the same time, which will affect the accuracy and consistency of individual decisions. As the number of agents increases, the complexity of the system increases, and the interaction between agents will cause the behavior of the entire system to become unstable. The instability of the system further makes it difficult for multiple agents to collaborate, leading to problems such as low system resource utilization and low execution efficiency.

[0003] In view of this, the present invention is proposed. Summary of the invention

[0004] The present invention provides a hierarchical collaborative deep reinforcement learning decision-making method for multi-agent path planning, which is used to construct a PPO-based hierarchical collaborative decision-making model to solve the challenges faced by existing deep reinforcement learning methods when performing path planning tasks in an environment.

[0005] The technical solution of the present invention is:

[0006] A hierarchical collaborative deep reinforcement learning decision-making method for multi-agent path planning, comprising: constructing a PPO-based hierarchical collaborative decision-making model through a bottom-level task allocation decision module and an upper-level behavior decision-making module based on the PPO framework; using environmental RGB image data, task requests and observation information as inputs of the PPO-based hierarchical collaborative decision-making model; wherein the observation information includes bottom-level observation information and upper-level observation information; and obtaining the optimal control strategy of the multi-agent system by continuously training the PPO-based hierarchical collaborative decision-making model.

[0007] The underlying observation information of each agent is converted into the Dec-POMDP process, which is expressed as: in, Represents the underlying state at time t; Represents the underlying decision action of the agent to decide whether to accept the task request; The transition function that changes the state after the agent executes the underlying decision action; represents the field of view of the agent under the underlying task allocation decision, V represents the visible environment map, and E represents the distance weight set; is the expected reward after executing the underlying decision at time t.

[0008] The upper-level observation information of each agent is converted into the Dec-POMDP process, which is expressed as: in, The state set representing the upper state at time t; The action set representing the upper-level decision-making actions that can be executed by the agent at time t; The state transition probability distribution of the agent after executing the upper-level decision action; Represents the field of view of the agent in the upper-level behavior decision module; is the expected reward after executing the upper-level decision at time t.

[0009] The PPO-based hierarchical collaborative decision-making model is based on a loss function constructed according to an entropy function, a centralized strategy network objective function with hierarchical decision rewards, and a centralized value network objective function with hierarchical decision rewards.

[0010] The loss function is expressed as:

[0011]

[0012] Among them, L P (θ) represents the objective function of the centralized policy network with hierarchical decision rewards, represents the objective function of the centralized value network with hierarchical decision rewards, c1 and c2 are the objective functions of the centralized value network with hierarchical decision rewards. And the entropy function LS The weight coefficient of (θ).

[0013] The objective function of the centralized strategy network with hierarchical decision rewards is expressed as:

[0014]

[0015] Among them, L P (θ) represents the objective function of the centralized policy network with hierarchical decision rewards, θ is the network parameter of the centralized policy network; J represents the number of agents; T represents the total number of discrete time steps; min represents the minimum function; ε represents the clipping factor, ξ represents the importance sampling ratio; clip() represents the clipping function, represents the advantage estimator of agent j with hierarchical decision rewards at time t.

[0016] The advantage estimator with hierarchical decision rewards is expressed as:

[0017]

[0018] Where L represents the estimation step size of the advantage estimator, represents the estimated value error of the value function with hierarchical decision rewards when the estimation step size is l for agent j at time t; λ is the smoothing factor; γ∈[0,1] is the discount factor.

[0019] The estimated value error of the value function with hierarchical decision rewards is expressed as:

[0020]

[0021] in, is the hierarchical decision reward of agent j at time t; The upper state of agent j at time t+1 is 2 s t+1 The estimated value when the old value function is used; The upper state of agent j at time t is 2 s t The estimated value when the old value function is used; is the predicted value of the hierarchical decision reward of agent j at time t for time t+1.

[0022] The objective function of the centralized value network with hierarchical decision rewards is expressed as:

[0023]

[0024] in, represents the objective function of a centralized value network with hierarchical decision rewards; is the network parameter of the centralized value network; J represents the number of agents; T represents the total number of discrete time steps; min represents the minimum function; clip() represents the clipping function; The upper state of agent j at time t is 2 s t The value function value when ; is the predicted value of the hierarchical decision reward of agent j at time t for time t+1; The upper state of agent j at time t is 2 s t is the estimated value when the old value function is used; ε represents the clipping factor.

[0025] The training of the PPO-based hierarchical collaborative decision-making model includes: initializing the underlying task allocation decision modules and upper-level behavior decision modules of all intelligent agents; all intelligent agents interact with the environment to generate a preset number of underlying task allocation decision module data and upper-level behavior decision module data, and store them in the experience pool respectively; based on the current underlying state, extracting the underlying task allocation decision module data from the experience pool to determine whether each intelligent agent accepts the current task request and calculates the expected rewards after the underlying decision of all intelligent agents, thereby obtaining intelligent agents with tasks and intelligent agents without tasks; intelligent agents without tasks do not enter the upper-level behavior decision module, and intelligent agents with tasks enter the upper-level behavior decision module; based on the current upper-level state of the intelligent agent with tasks, extracting the upper-level behavior decision module data from the experience pool to determine whether each intelligent agent accepts the current task request and calculates the expected rewards after the underlying decision of all intelligent agents, thereby obtaining intelligent agents with tasks and intelligent agents without tasks; intelligent agents without tasks do not enter the upper-level behavior decision module, and intelligent agents with tasks enter the upper-level behavior decision module; based on the current upper-level state of the intelligent agent with tasks, extracting the upper-level behavior decision module data from the experience pool to determine whether each intelligent agent accepts the current task request and calculates the expected rewards after the underlying decision of all intelligent agents The execution actions of the agents are calculated, and then the advantage function of each agent with hierarchical decision-making rewards is calculated; based on the advantage function of each agent with hierarchical decision-making rewards, the objective function of the centralized strategy network with hierarchical decision-making rewards of the multi-agent system is calculated; based on the entropy function of the multi-agent system, the objective function of the centralized strategy network with hierarchical decision-making rewards and the objective function of the centralized value network with hierarchical decision-making rewards, the loss function of the multi-agent system is obtained; based on the loss function of the multi-agent system, the parameters of the centralized strategy network and the centralized value network are updated, the updated parameters of the centralized value network are fixed, all agents using the updated centralized strategy network continue to interact with the environment, and new underlying task allocation decision module data and upper-level behavior decision module data are generated and stored in the experience pool until the termination condition is reached and the training is completed.

[0026] The beneficial effects of the present invention are:

[0027] The decision-making method of hierarchical collaborative deep reinforcement learning for multi-agent path planning proposed in the present invention aims to solve the challenges faced by existing deep reinforcement learning methods when performing path planning tasks in the environment, including disconnection from system task resource allocation, difficulty in multi-agent collaboration, system instability, poor system resource utilization, and low efficiency during task execution. The following are the main advantages of the present invention:

[0028] Efficient task resource allocation: The present invention fully considers the task resource allocation requirements in the real environment, so that the intelligent agent can obtain task resource allocation information before planning the path, avoiding resource waste, improving resource utilization, and thus reducing operating costs.

[0029] Stable system state: By introducing a centralized strategy network and a centralized value network, the present invention enables each agent to observe the state and behavior of other agents, making decisions more stably and avoiding frequent fluctuations in the system state. This advantage enables the entire system to maintain stable operation in a dynamic environment and improves the reliability of the system.

[0030] Reduce execution time: The present invention proposes a joint decision-making of joint resource on-demand allocation and path planning, so that all intelligent agents can perform tasks more efficiently, reduce task execution time, and improve the response speed of the entire system.

[0031] Enhanced system robustness: The multi-agent hierarchical decision-making mechanism of the present invention enhances the robustness of the system in an unstable environment. Each agent can make stable decisions based on local information, reducing the impact of environmental changes on system stability and improving system reliability. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is a framework structure diagram of the present invention;

[0033] Figure 2 It is a framework diagram of the underlying task allocation decision module of the present invention;

[0034] Figure 3 It is a storage environment diagram adopted by the present invention;

[0035] Figure 4 This is the task success rate curve of all methods when the number of agents is 3;

[0036] Figure 5 This is the average reward curve of all methods when the number of agents is 3. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical scheme and advantages of the embodiments of the present invention clearer, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. It should be noted that the embodiments in this application and the features in the embodiments can be combined with each other arbitrarily without conflict.

[0038] Example 1: Figure 1-5 As shown, according to the first aspect of an embodiment of the present invention, a hierarchical collaborative deep reinforcement learning decision method for multi-agent path planning is provided, including: constructing a hierarchical collaborative decision model based on PPO through a bottom-level task allocation decision module and an upper-level behavior decision module based on the PPO framework; using RGB image data of a warehouse environment with a global view, task requests (i.e., resources to be allocated), and observation information as inputs of the hierarchical collaborative decision model based on PPO; wherein the observation information includes bottom-level observation information and upper-level observation information; and obtaining the optimal control strategy of the multi-agent system by continuously training the hierarchical collaborative decision model based on PPO. In an embodiment of the present invention, a warehouse environment is taken as an example, and the size of the environment RGB image is uniformly processed to 32*20*3.

[0039] Furthermore, the underlying observation information of each agent can be transformed into a partially observable Markov decision Dec-POMDP process, expressed as: in, represents the underlying state at time t, , each discrete time step T represents the total number of discrete time steps, and 0 represents the instant; Represents the underlying decision action of the agent to decide whether to accept the task request; The transition function that changes the state after the agent executes the underlying decision action; represents the field of view of the agent under the underlying task allocation decision, V represents the visible environment map, and E represents the distance weight set e ω, ω represents the distance weight, There are real numbers greater than 0. indivual; is the expected reward after executing the underlying decision at time t.

[0040] Said in, t r i It means that the task request r is assigned to i at time t. It means that at time t, agent n is assigned value j, and task request r = (s i ,gi ) from the current position s i ∈V and the corresponding target position g i ∈V; after the agent accepts the task request, the estimated Euclidean distance d between the target position and the current position is set to the weight n = (v, τ, r) indicates that the agent n accepts the task request r, and the time step required to reach the position v∈V is τ; v indicates the node position to be reached in the next time step (the agent is in a waiting state / moving state. If it is in a waiting state, the position in the next time step is the current node position); I indicates the total number of task request numbers; J indicates the total number of agent numbers; for the sake of simplicity, the present invention adopts j v t To represent the position of agent j at time t.

[0041] Said The expression is:

[0042]

[0043] In the formula, Represents the agent's request for a task t r i The decision variable for making a decision takes values ​​of 0 and 1, where 0 means rejecting the request and 1 means accepting the request; Indicates a task request t r i Make an acceptance decision and assign it to agent j; It means that each agent cannot accept new task requests before completing a request, that is, each agent can only accept one task request. Based on the above, it can be seen that for an agent, it either does not accept a task request or can only accept one task request.

[0044] Said Specifically:

[0045] When the agent rejects a task request, the agent state does not change;

[0046] when , add the request state to the agent state, that is, when When j ← t r i ;in, Indicates that the task request accepted by agent j is non-empty; when the agent initially accepts the task request, that is, when At this time When the agent has completed the previous task request before accepting the next task request, that is, when , new requests can be added to the agent (regardless of the agent state), that is, (t r i ) i∈{1,...,I} ←( t+1 r i ) i∈{1,...,I+1} , update time step t←t+1; where s(r) refers to the starting position of the agent when accepting task request r; g(r) refers to the target position of the agent when accepting task request r.

[0047] Said , specifically: when the agent does not accept the task request, it will receive a penalty value of -(1 / ω); when the agent accepts the task request, it will receive -(1 / 2ω) as an incentive value; the constructed expression is:

[0048]

[0049] In the formula, k represents the number of rejected requests; j r t represents the task request of agent j at time t, j τ t =0 means instantaneous moment, s( j r t ) represents the position of agent j at time t.

[0050] Furthermore, the upper-level observation information of each agent can be transformed into a partially observable Markov decision Dec-POMDP process, which is expressed as: in, express The upper state of the moment 2 s t The state set of each discrete time step , T represents the total number of discrete time steps, and 0 represents the instant; Represents the upper-level decision-making action that can be executed by the agent at time t 2 a t The action set; The state transition probability distribution of the agent after executing the upper-level decision action; Represents the field of view of the agent in the upper-level behavior decision module; is the expected reward after executing the upper-level decision at time t.

[0051] Said The expression is:

[0052]

[0053] in, They respectively mean wait, up, right, down, and left.

[0054] Said The expression is:

[0055]

[0056] In the formula, They represent rewards for waiting, moving, colliding, being too close to the boundary, and reaching the target point. Table 1 shows the specific setting rules for the upper-level rewards:

[0057] Table 1

[0058]

[0059] Furthermore, the PPO-based hierarchical collaborative decision-making model is based on a loss function constructed according to an entropy function, a centralized strategy network objective function with hierarchical decision rewards, and a centralized value network objective function with hierarchical decision rewards.

[0060] Furthermore, the loss function , the expression is:

[0061]

[0062] Among them, L P (θ) represents the objective function of the centralized policy network with hierarchical decision rewards, represents the objective function of the centralized value network with hierarchical decision rewards, c1 and c2 are the objective functions of the centralized value network with hierarchical decision rewards. And the entropy function L S The weight coefficient of (θ).

[0063] Furthermore, the objective function of the centralized strategy network with hierarchical decision rewards is expressed as:

[0064]

[0065] Among them, L P (θ) represents the objective function of the centralized strategy network with hierarchical decision rewards, θ is the network parameter of the centralized strategy network; min represents the minimum function, which is used to control the strategy update step; J represents the number of agents, clip() represents the clipping function, represents the advantage estimator of agent j with hierarchical decision reward at time t, ε represents the clipping factor, and ξ represents the importance sampling ratio.

[0066] Furthermore, the advantage estimator with hierarchical decision rewards is expressed as:

[0067]

[0068] Where L represents the estimation step size of the advantage estimator, It represents the estimated value error of the value function with hierarchical decision rewards of agent j at time t when the estimated step length is l. λ is the smoothing factor. γ∈[0,1] is the discount factor, which is used to control the influence of the long-term reward estimate of the agent. A small γ value will encourage the agent to maximize short-term rewards, while a large γ value will encourage the agent to maximize rewards in a longer time frame, which is more forward-looking.

[0069] Furthermore, the value function with hierarchical decision reward estimates the value error, expressed as:

[0070]

[0071] in, is the hierarchical decision reward of agent j at time t; γ∈[0,1] is the discount factor; The upper state of agent j at time t+1 is 2 s t+1 The estimated value when the old value function is used; The upper state of agent j at time t is 2 s t The estimated value when the old value function is used; is the predicted value of the hierarchical decision reward of agent j at time t for time t+1.

[0072] Furthermore, the objective function of the centralized value network with hierarchical decision rewards is expressed as:

[0073]

[0074] in, represents the objective function of the centralized value network with hierarchical decision rewards, which is used to reduce the estimation error of the value function with hierarchical decision rewards; The network parameters for the centralized value network; The upper state of agent j at time t is 2 s t The value function value when ; ε represents the clipping factor.

[0075] Furthermore, the entropy function is expressed as:

[0076]

[0077] Among them, L S (θ) represents the entropy function, It represents the average value of the function of all agents within time T, which is used to promote multi-agent collaboration and achieve the same goal; α is the entropy regularization coefficient, Represents the centralized strategy gradient value of agent j at time t; θ is the network parameter of the centralized strategy network.

[0078] Furthermore, all hierarchical decision rewards are expressed as:

[0079]

[0080] Among them, γ∈[0,1] is a discount factor, which is used to control the influence of the agent's long-term reward estimate. A small γ value will encourage the agent to maximize short-term rewards, while a large γ value will encourage the agent to maximize rewards in a longer time frame, which is more forward-looking. is the expected reward after executing the underlying decision at time t; is the expected reward after executing the upper-level decision at time t; is the hierarchical decision reward of agent j at time t; It is the joint hierarchical decision reward of all agents after executing the model decision.

[0081] Furthermore, the training of the PPO-based hierarchical collaborative decision-making model includes: initializing the underlying task allocation decision modules and upper-level behavior decision modules of all intelligent agents; wherein, (the upper-level behavior decision module includes the strategy network, value network and experience pool under the PPO framework, and also includes a centralized strategy network and a centralized value network); all intelligent agents interact with the environment to generate a preset number of underlying task allocation decision module data and upper-level behavior decision module data, and store them in the experience pool respectively; based on the current underlying state, extract the underlying task allocation decision module data from the experience pool to determine whether each intelligent agent accepts the current task request and calculate the expected rewards of all intelligent agents at the bottom level, thereby obtaining intelligent agents with tasks and intelligent agents without tasks; intelligent agents without tasks do not enter the upper-level behavior decision module, and intelligent agents with tasks enter the upper-level behavior decision module; based on the current The upper state extracts the upper behavior decision module data from the experience pool to determine the execution actions of each agent, and then calculates the advantage function of each agent with hierarchical decision rewards; based on the advantage function of each agent with hierarchical decision rewards, calculates the centralized strategy network objective function of the multi-agent system with hierarchical decision rewards; based on the entropy function of the multi-agent system, the centralized strategy network objective function with hierarchical decision rewards and the centralized value network objective function with hierarchical decision rewards, obtains the loss function of the multi-agent system; based on the loss function of the multi-agent system, updates the parameters of the centralized strategy network and the centralized value network, fixes the updated centralized value network parameters, and continues to interact with the environment for all agents using the updated centralized strategy network, and generates new underlying task allocation decision module data and upper behavior decision module data and stores them in the experience pool until the termination condition is reached and the training is completed.

[0082] According to a second aspect of an embodiment of the present invention, a decision-making system of hierarchical collaborative deep reinforcement learning for multi-agent path planning is provided, comprising: a construction module for constructing a hierarchical collaborative decision-making model based on PPO through a bottom-level task allocation decision module and an upper-level behavior decision-making module based on the PPO framework; an acquisition module for taking environmental RGB image data, task requests and observation information as inputs of the hierarchical collaborative decision-making model based on PPO; and for obtaining the optimal control strategy of the multi-agent system by continuously training the hierarchical collaborative decision-making model based on PPO; wherein the observation information comprises bottom-level observation information and upper-level observation information.

[0083] According to a third aspect of an embodiment of the present invention, a processor is provided, which is used to execute operations. When the processor executes the operations, the following steps are implemented: a hierarchical collaborative decision-making model based on PPO is constructed through a bottom-level task allocation decision module and an upper-level behavior decision-making module based on PPO framework; environmental RGB image data, task requests and observation information are used as inputs of the hierarchical collaborative decision-making model based on PPO; wherein the observation information includes bottom-level observation information and upper-level observation information; and the optimal control strategy of the multi-agent system is obtained by continuously training the hierarchical collaborative decision-making model based on PPO.

[0084] Furthermore, in this embodiment, the present invention and its modified algorithm MARPF are experimentally compared with the existing MADDPG, MATD3, and DEMADDPG, wherein MARPF is an algorithm that does not use the underlying task allocation decision in the present invention. Figure 3 As shown, 18 rows of storage cabinets with a size of 10×1.2 are included as static obstacles, and only one agent is allowed to pass between every two rows of storage cabinets in the vertical direction. This setting increases the risk of motion deadlock of the agent during movement, and better reflects the algorithm's flexible processing capabilities in dealing with extreme situations. In addition, the storage environment setting is an end-to-end reinforcement learning with visual settings, which includes difficulties such as delayed reward distribution, partial observability, and high-dimensional observation states. In this embodiment, the number of agents is set to 3 and 6 respectively. The reason is that, in the same research task, the maximum number of agents is set to 3. In addition, the present invention has good generalization and does not require hyperparameter adjustment for specific problems. The specific settings are shown in Table 2.

[0085] Table 2 Parameter settings

[0086] parameter Numeric Discount factor γ 0.99 Smoothing factor λ 0.95 Advantage estimation step length L 15 Clipping factor ε 0.10 Entropy regularization coefficient α 0.20 <![CDATA[Central value network weight coefficient c1]]> 0.50 <![CDATA[Entropy function weight coefficient c2]]> 0.01 Learning rate lr <![CDATA[5×10 -4 ]]> Experience buffer segmentation length L 15 Minimum batch size mini_batch_size 8 Batch sample number batch_size 32 Total training rounds 200,000

[0087] The present invention is a method for solving the multi-agent collaborative path planning problem by using DRL. Therefore, the success rate index of MAPF and the reward value index of DRL are used as collaborative optimization indexes; the optimal control strategy is evaluated based on the collaborative optimization index.

[0088] The collaborative optimization index is expressed as:

[0089]

[0090] in, It means maximizing the MAPF success rate within the preset time; Indicates maximizing the hierarchical decision reward of all agents within a preset time represents the MAPF success rate, which is the number of agents that complete the task N C The ratio to the number of all agents N is calculated as:

[0091]

[0092] It should be noted that the present invention adopts the DRL method for the MAPF problem, and the evaluation index should take the success rate in the MAPF problem as the primary evaluation index.

[0093] As shown in Table 3, the hierarchical collaborative deep reinforcement learning decision-making method for multi-agent path planning proposed in this invention ( Figure 1 , Figure 2 )In the warehouse environment( Figure 3 ), the algorithm converges quickly and performs best. For example, the present invention exhibits the best performance when the number of agents is 3; Figure 4 As shown, in terms of the success rate index, the method of the present invention far exceeds other comparative algorithms. Specifically, the success rate of the present invention reaches 71.61%, which is 48.97% higher than MADDPG, 46.67% higher than DEMADDPG, and 48.98% higher than MATD3. The MADDPG algorithm has a centralized value network and can handle collaborative tasks between agents, but each agent tends to optimize its own rewards, which may lead to poor collaboration. DEMADDPG enhances the collaborative ability of agents by introducing adversarial strategies and improves the success rate of MAPF tasks. However, DEMADDPG will face local optimal problems because the agents are still limited by local information and individual interest optimization. In contrast, the method of the present invention successfully compensates for this defect by introducing centralized policy networks and centralized value networks. The MATD3 algorithm relies on policy networks and value networks to make decisions, but the state and action distribution of agents change in the face of dynamic environments, which increases the difficulty of policy learning and may also lead to instability and low success rate of training; in terms of DRL measurement indicators, that is, average reward, such as Figure 5As shown, MARPF performs best, while the present invention performs second. This is because the present invention introduces the underlying task allocation decision that will punish the agent that does not accept the resource allocation. Although the reward value of the present invention performs second best among all methods, its success rate is the best, which further reflects that the present invention is conducive to all agents learning strategies to collaboratively achieve goals, optimize system resource utilization efficiency and improve task success rate. Compared with other algorithms (MADDPG, DEMADDPG, MATD3), the high reward value of MARPF shows that centralized strategy networks and centralized value networks are helpful to quickly learn effective collaborative strategies. In terms of execution time, the execution time of the present invention can ensure that the success rate remains at a high level with very little increase. Therefore, the hierarchical collaborative deep reinforcement learning decision-making method for multi-agent path planning proposed in the present invention can improve the utilization of system resources, efficiently complete tasks, significantly improve the stability of the multi-agent algorithm, the algorithm has strong generalization, and the measurement standards for response types can obtain high-performance indicators, and have the characteristics of high efficiency and stability.

[0094] Table 3 Performance comparison of different algorithms

[0095]

[0096] The specific implementation modes of the present invention are described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above implementation modes, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.

Claims

1. A hierarchical collaborative deep reinforcement learning decision-making method for multi-agent path planning, characterized by: include: A hierarchical collaborative decision-making model based on PPO is constructed through the underlying task allocation decision module and the upper-level behavior decision module based on PPO framework; The environmental RGB image data, task request and observation information are used as inputs of the PPO-based hierarchical collaborative decision-making model; the observation information includes bottom-level observation information and upper-level observation information; By continuously training the PPO-based hierarchical collaborative decision-making model, the optimal control strategy of the multi-agent system is obtained.

2. The hierarchical collaborative deep reinforcement learning decision-making method for multi-agent path planning according to claim 1 is characterized in that: The underlying observation information of each agent is converted into the Dec-POMDP process, which is expressed as: in, represents the underlying state at time t; Represents the underlying decision action of the agent to decide whether to accept the task request; A transition function that changes the state after the agent executes the underlying decision action; represents the agent's field of view under the underlying task allocation decision, V represents the visible environment map, and E represents the distance weight set; is the expected reward after executing the underlying decision at time t.

3. The hierarchical collaborative deep reinforcement learning decision-making method for multi-agent path planning according to claim 1 is characterized in that: The upper-level observation information of each agent is converted into the Dec-POMDP process, which is expressed as: in, The state set representing the upper state at time t; The action set representing the upper-level decision-making actions that can be executed by the agent at time t; The state transition probability distribution of the agent after executing the upper-level decision action; Represents the field of view of the agent in the upper-level behavior decision module, and V represents the visible environment map; is the expected reward after executing the upper-level decision at time t.

4. The hierarchical collaborative deep reinforcement learning decision-making method for multi-agent path planning according to claim 1 is characterized in that: The PPO-based hierarchical collaborative decision-making model is based on a loss function constructed according to an entropy function, a centralized strategy network objective function with hierarchical decision rewards, and a centralized value network objective function with hierarchical decision rewards.

5. The hierarchical collaborative deep reinforcement learning decision-making method for multi-agent path planning according to claim 4 is characterized in that: The loss function is expressed as: Among them, L P (θ) represents the objective function of the centralized policy network with hierarchical decision rewards, represents the centralized value network objective function with hierarchical decision rewards, c1 and c2 are the centralized value network objective functions with hierarchical decision rewards and entropy function L S The weight coefficient of (θ).

6. The hierarchical collaborative deep reinforcement learning decision-making method for multi-agent path planning according to claim 4 is characterized in that: The objective function of the centralized strategy network with hierarchical decision rewards is expressed as: Among them, L P (θ) represents the objective function of the centralized policy network with hierarchical decision rewards, θ is the network parameter of the centralized policy network; J represents the number of agents; T represents the total number of discrete time steps; min represents the minimum function; ε represents the clipping factor, ξ represents the importance sampling ratio; clip() represents the clipping function, represents the advantage estimator of agent j at time t with hierarchical decision rewards.

7. The hierarchical collaborative deep reinforcement learning decision-making method for multi-agent path planning according to claim 6 is characterized in that: The advantage estimator with hierarchical decision rewards is expressed as: Where L represents the estimation step size of the advantage estimator, represents the estimated value error of the value function with hierarchical decision rewards when the agent j estimates the step size l at time t; λ is the smoothing factor; γ∈[0,1] is the discount factor.

8. The hierarchical collaborative deep reinforcement learning decision-making method for multi-agent path planning according to claim 7 is characterized in that: The estimated value error of the value function with hierarchical decision rewards is expressed as: in, is the hierarchical decision reward of agent j at time t; The upper state of agent j at time t+1 is 2 s t+1 The estimated value when the old value function is used; For agent j at time t, the upper state is 2 s t The estimated value when the old value function is used; is the predicted value of the hierarchical decision reward of agent j at time t for time t+1.

9. The hierarchical collaborative deep reinforcement learning decision-making method for multi-agent path planning according to claim 4 is characterized in that: The objective function of the centralized value network with hierarchical decision rewards is expressed as: in, represents the objective function of a centralized value network with hierarchical decision rewards; is the network parameter of the centralized value network; J represents the number of agents; T represents the total number of discrete time steps; min represents the minimum function; clip() represents the clipping function; For agent j at time t, the upper state is 2 s t The value function value when ; is the predicted value of the hierarchical decision reward of agent j at time t for time t+1; For agent j at time t, the upper state is 2 s t is the estimated value when the old value function is used; ε is the clipping factor.

10. The hierarchical collaborative deep reinforcement learning decision-making method for multi-agent path planning according to claim 1, characterized in that: The training of the PPO-based hierarchical collaborative decision-making model includes: Initialize the underlying task allocation decision modules and upper-level behavior decision modules of all intelligent agents; All agents interact with the environment, generating a preset amount of data for the underlying task allocation decision module and the upper-level behavior decision module, and storing them in the experience pool respectively; Based on the current underlying state, the underlying task allocation decision module data is extracted from the experience pool to determine whether each agent accepts the current task request and calculate the expected reward after the underlying decision of all agents, thereby obtaining agents with tasks and agents without tasks; agents without tasks do not enter the upper-level behavior decision module, while agents with tasks enter the upper-level behavior decision module; Based on the current upper-level state of the agent with the task, the upper-level behavior decision module data is extracted from the experience pool to determine the execution action of each agent, and then the advantage function of each agent with hierarchical decision rewards is calculated; Based on the advantage function of each agent with hierarchical decision-making rewards, the objective function of the centralized strategy network with hierarchical decision-making rewards of the multi-agent system is calculated; based on the entropy function of the multi-agent system, the objective function of the centralized strategy network with hierarchical decision-making rewards, and the objective function of the centralized value network with hierarchical decision-making rewards, the loss function of the multi-agent system is obtained; According to the loss function of the multi-agent system, the parameters of the centralized policy network and the centralized value network are updated, the updated centralized value network parameters are fixed, and all agents using the updated centralized policy network continue to interact with the environment, and new underlying task allocation decision module data and upper-level behavior decision module data are generated and stored in the experience pool until the termination condition is reached and the training is completed.

Images