Multi-agent path planning method, device and equipment and computer storage medium

Abstract

The invention discloses a multi-agent path planning method, device and equipment and a computer storage medium. The method comprises the following steps: constructing a multi-agent path planning model comprising a subtask network, an execution strategy network and a multi-commentator network; generating sub-task targets of multiple agents in a fixed time span through a sub-task network based on the current global state of the multiple agents; according to the sub-task target, through a sub-task network, controlling a plurality of agents to execute an action sequence of a preset step number; calculating a reward value according to the execution result of the preset step number; updating parameters of the multiple reviewer networks and the strategy network and parameters of each reviewer network in the multiple reviewer networks based on the environment rewards and the internal rewards; updating parameters of a sub-task network through a state transition strategy gradient module according to a relationship between a new state of the multi-agent after the action is executed according to the preset step number and a sub-task target state; and repeating the steps until training convergence to obtain a multi-agent path planning target model.

Description

Technical Field

[0001] This application relates to the field of multi-agent path planning technology, and in particular to a multi-agent path planning method, apparatus, device, and computer storage medium in an environment without a fixed road network. Background Technology

[0002] Multi-agent path planning is a key technology in fields such as autonomous driving, drone swarm control, and robot team collaboration. Traditional path planning methods typically rely on predefined fixed road networks or environmental maps, using graph search or heuristic algorithms to find the optimal path. However, in environments without fixed road networks, such as field operations, disaster relief, and complex terrain exploration, agents cannot rely on pre-defined roads or clear maps for planning, facing greater planning difficulty and challenges. This type of problem is called the multi-agent path planning problem in environments without fixed road networks. Its characteristic is that agents move freely in a continuous state and action space, requiring real-time environmental perception and autonomous decision-making capabilities to complete navigation tasks.

[0003] Reinforcement learning, through the interaction between agents and the environment, autonomously learns optimal strategies, becoming a new direction for solving dynamic pathless network planning problems. However, existing reinforcement learning methods suffer from a severe sparse reward problem: agents only receive effective feedback when they reach the final goal or when a collision occurs. In multi-agent high-dimensional spaces, the extreme scarcity of reward signals leads to low training efficiency, difficulty in policy convergence, and even the inability to obtain feasible paths.

[0004] To alleviate sparse rewards, hierarchical reinforcement learning decomposes long tasks into sub-tasks and provides dense rewards. However, existing hierarchical reinforcement learning methods require manual design of sub-task objectives or pre-defined decomposition rules, lacking the ability to adapt to complex environments. Therefore, it remains difficult to achieve efficient and stable path planning performance in real-world, complex, and variable roadless environments. Summary of the Invention

[0005] This application addresses the shortcomings of the prior art by providing a multi-agent path planning method in an environment without a fixed road network, including:

[0006] Step S1: Construct a multi-agent path planning model with a hierarchical architecture including a sub-task network, an execution policy network, and a multiple critic network;

[0007] Step S2: Based on the current global state of the multi-agent network, the sub-task objectives of multiple agents are generated within a fixed time span.

[0008] Step S3: According to the sub-task objective, control the multiple agents to execute a sequence of actions with a preset number of steps through the sub-task network, wherein the state of each agent changes according to the environmental dynamics model;

[0009] Step S4: Calculate the reward value based on the execution result of the preset number of steps. The reward value includes environmental rewards and intrinsic rewards based on the multi-critic network.

[0010] Step S5: Based on the environmental reward and the intrinsic reward, update the parameters of the multi-critic network, the policy network, and the parameters of each critic network in the multi-critic network;

[0011] Step S6: The parameters of the subtask network are updated by the state transition strategy gradient module according to the relationship between the new state of the multi-agent after the preset number of steps and the subtask target state;

[0012] Step S7: Repeat steps S2 to S6 until training converges to obtain a multi-agent path planning target model. The multi-agent path planning target model includes the trained policy network and the trained sub-task network model.

[0013] Optionally, in this embodiment of the application, the subtask objective is a unit vector in a preset dimension space; the fixed time span includes multiple decision cycles with the preset number of steps, and the subtask network outputs the corresponding subtask objective at the beginning of each decision cycle.

[0014] Optionally, in this embodiment, the environmental reward is calculated in real time by the environment module based on the distance change between each agent and the final target location or the task completion event; the intrinsic reward is determined by the multi-critic network evaluating the achievement degree of the sub-task target.

[0015] Optionally, in this embodiment, the intrinsic reward is determined by evaluating the achievement of the sub-task objective through the multi-critic network, including: evaluating the value of the current joint state and joint actions of the multi-agent through a dual action-state value critic network, outputting at least two action-state value estimates, and calculating the average of each state value estimate; using the deviation between each action-state value estimate and the average as an error metric, adjusting the estimation results of the critic network, generating an intrinsic reward signal based on the sub-task objective, and updating the policy network after weighted fusion of the intrinsic reward signal and the environmental reward.

[0016] Optionally, in this embodiment of the application, updating the parameters of the policy network and the parameters of each critic network in the multiple critic network includes: updating the parameters of the policy network using a policy gradient algorithm; and updating the parameters of each critic network in the multiple critic network using temporal difference error.

[0017] Optionally, in this embodiment, updating the parameters of the subtask network by the state transition policy gradient module based on the relationship between the new state of the multi-agent after performing the action at the preset number of steps and the subtask target state includes: calculating the relative value of the current subtask target based on the degree of deviation of the state of the multi-agent relative to the subtask target state at the end of the preset number of steps; evaluating the state change caused by the preset number of steps through a state transition value critic network to obtain the corresponding state transition advantage value; and combining the relative value with the state transition advantage value to calculate the state transition policy gradient of the subtask network to update the parameters of the subtask network.

[0018] On another aspect of this application, a multi-agent path planning device is also provided in an environment without a fixed road network, comprising:

[0019] A multi-agent path planning model building unit is used to construct a multi-agent path planning model with a hierarchical architecture including a sub-task network, an execution policy network, and a multiple critic network.

[0020] The subtask target generation unit is used to generate subtask targets for multiple agents within a fixed time span based on the current global state of the multiple agents through the subtask network.

[0021] The task execution control unit is used to control the multiple agents to execute a sequence of actions with a preset number of steps through the sub-task network according to the sub-task objective, wherein the state of each agent changes according to the environmental dynamics model;

[0022] The reward value calculation unit is used to calculate the reward value based on the execution result of the preset number of steps. The reward value includes environmental rewards and intrinsic rewards based on the multi-critic network.

[0023] The policy network update unit is used to update the parameters of the multi-critic network, the policy network, and the parameters of each critic network in the multi-critic network based on the environmental reward and the intrinsic reward.

[0024] The subtask adaptive optimization unit is used to update the parameters of the subtask network by means of the state transition strategy gradient module based on the relationship between the new state of the multi-agent after the preset number of steps and the subtask target state;

[0025] The target model building unit is used to repeatedly execute steps S2 to S6 until training converges to obtain a multi-agent path planning target model, which includes the trained policy network and the trained sub-task network model.

[0026] Optionally, in this embodiment of the application, the reward value calculation unit includes: an environmental reward calculation unit, used to calculate the environmental reward in real time by the environmental module based on the distance change between each agent and the final target position or the task completion event; and an intrinsic reward calculation unit, used to determine the intrinsic reward by evaluating the achievement degree of the sub-task target through the multi-critic network.

[0027] In another aspect of this application, a multi-agent path planning device is also provided in an environment without a fixed road network, comprising:

[0028] Memory, used to store computer programs;

[0029] A processor is configured to invoke and execute the computer program to implement the steps of the multi-agent path planning method as described in any of the preceding claims.

[0030] In another aspect of the embodiments of this application, a storage medium is also provided, including a software program adapted for a processor to execute the steps of the multi-agent path planning method as described in any of the preceding claims.

[0031] The multi-agent path planning device in the environment without a fixed road network includes a computer program stored on a medium. The computer program includes program instructions. When the program instructions are executed by the computer, the computer performs the methods described in the above aspects and achieves the same technical effect.

[0032] The embodiments described in this application have the following beneficial effects:

[0033] This application provides a multi-agent path planning method in environments without a fixed road network. It utilizes a subtask-commentator architecture to decompose long-cycle tasks into short-cycle subtasks, alleviating the sparse reward problem. A multi-commentator network is introduced to evaluate the value of joint agent actions, reducing the value estimation error that might occur with a single commentator and improving the accuracy and robustness of the value function estimation. An intrinsic reward based on the subtask objective is designed to provide additional guidance signals to the agent at each subtask stage, encouraging the execution policy to optimize towards the subtask objective, thereby avoiding direct reliance on sparse global rewards for policy updates and preventing the amplification of value estimation errors. A state transition policy gradient module is introduced, breaking the limitation that the subtask network relies on the reverse gradient update of the execution policy network. Instead, it directly calculates gradients based on the subtask execution results to update the subtask policy, improving the stability of high-level policy updates.

[0034] Furthermore, this application utilizes a reinforcement learning algorithm that combines multiple critics and hierarchical policy gradients to achieve multi-agent path planning in environments without fixed road networks, enabling more stable convergence and obtaining superior planning results. Attached Figure Description

[0035] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. The accompanying drawings in the following description are merely exemplary embodiments of this application.

[0036] Figure 1 This is a flowchart illustrating the multi-agent path planning method in an environment without a fixed road network provided in this application.

[0037] Figure 2 This is a schematic diagram of the structure of the multi-agent path planning simulation platform provided in this application;

[0038] Figure 3 This is a schematic diagram of the algorithm architecture of the multi-agent path planning method provided in this application;

[0039] Figure 4 This is a comparison chart of the scores of the multi-agent path planning simulation platform provided in this application with the scores of classic algorithms in the same field.

[0040] Figure 5 This is a schematic diagram of the structure of the multi-agent path planning device in an environment without a fixed road network provided in this application;

[0041] Figure 6 This is a schematic diagram of the structure of the multi-agent path planning device in an environment without a fixed road network provided in this application. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0043] Example 1

[0044] The following is combined Figures 1 to 4 This application provides a detailed description of the multi-agent path planning method 100 in an environment without a fixed road network. Figure 1 This is a flowchart illustrating the multi-agent path planning method provided in this application for an environment without a fixed road network. Figure 2 This is a schematic diagram of the structure of the multi-agent path planning simulation platform provided in this application. Figure 3 This is a schematic diagram of the algorithm architecture of the multi-agent path planning method provided in this application. Figure 4 This is a comparison chart of the scores provided in this application on the simulation platform with the scores of classic algorithms in the same field. For example... Figure 1As shown, the multi-agent path planning method 100 includes steps S1 to S7.

[0045] In step S1, a multi-agent path planning model with a hierarchical architecture including a sub-task network, an execution policy network, and a multiple critic network is constructed.

[0046] In this embodiment of the application, a multi-agent path planning model is constructed in an environment without a fixed road network. Figure 2 In the diagram, the G-shaped circle represents the target point, the black outlined circle represents the agent, and the star-shaped, airplane-shaped, solid circle, hourglass-shaped, and leaf-shaped obstacles represent them. This environment is a continuous two-dimensional planar space without pre-defined road constraints, allowing multiple agents to move freely. The environment contains several static obstacles that the agents must avoid to reach the target point. Each agent is a small robot using a front-engine, rear-drive model. Its motion state is determined by parameters such as position, orientation, and velocity, conforming to non-holonomic dynamics constraints. The initial states of multiple agents (e.g., starting position, orientation) are randomly set or pre-set by the environment creation module. Next, based on a reinforcement learning framework, the neural network modules required for the algorithm are initialized, including a subtask network, an execution policy network, and a multiple critic network. The subtask network generates high-level subtask objectives based on the current global state, the execution policy network outputs low-level action control instructions for each agent, and the multiple critic network includes several evaluation subnetworks used to assess the value of the policy and subtasks.

[0047] In this embodiment, the multi-criterion network includes two types of critics: an action-state value critic network, used to evaluate the value of a joint action in a given state; and a state value critic network, used to evaluate the value of a given state itself. The action-state value critic network employs a dual-network redundancy architecture, constructing two structurally identical but independently trained networks (e.g., Q) to output two sets of action-state value estimates, Q1(S, A) and Q2(S, A), to mitigate the impact of a single estimation bias. The state value critic network outputs a state value estimate V(S), which can be used as a baseline to reduce variance or combined with the Q value to calculate the advantage function. Furthermore, a state transition value critic network is included to assist in updating the sub-task network. Its input can be state change information before and after the sub-task execution, and its output is an evaluation of the value of the state transition. The network parameters are initially randomized or determined through a small amount of pre-training.

[0048] In step S2, the subtask network generates subtask objectives for multiple agents within a fixed time span based on the current global state of the multi-agent network.

[0049] In this embodiment, when performing the path planning task, a sub-task network is first activated. This network takes the global state of all agents as input. The global state comprehensively covers key information such as the position, speed, target point, and relative positions of each agent, providing the sub-task network with a complete picture of the overall environment. Based on this information, the sub-task network comprehensively analyzes the complex environment faced by multiple agents, identifying key path planning elements, such as obstacle areas that agents need to avoid and potential cooperation points between agents.

[0050] The sub-task network integrates complex analysis algorithms and planning logic to rationally break down the entire path planning task. It predicts the phased goals that each agent should achieve within a set fixed time span—the sub-task goals. These sub-task goals are not arbitrarily set but closely aligned with the ultimate global path planning goal, such as all agents successfully reaching their respective destinations or the entire team completing the exploration of a specific area. Each sub-task goal aims to guide the agents step-by-step towards the final objective while ensuring collaboration and coordination among the agents.

[0051] When generating subtask objectives, the subtask network also fully considers the characteristics of each agent, such as movement speed and payload capacity, to ensure that the generated subtask objectives are both challenging and feasible.

[0052] Furthermore, the subtask objective is a unit vector in a preset dimension space; the fixed time span includes multiple decision cycles with the preset number of steps, and the subtask network outputs the corresponding subtask objective at the beginning of each decision cycle.

[0053] Figure 3 The schematic diagram of the algorithm architecture of this application is shown, which divides the core parts such as the agent local policy module, the intrinsic reward mechanism module, and the state transition policy gradient module. Figure 3 middle, This represents the observation of car n at time t. This represents the encoded hidden state obtained after the observation passes through the coding layer. Let represent the subtask acquired by car n at time t in the previous time step. This represents the subtask acquired by car n at time t. Let represent the sub-task objective obtained by car n at time t. Let represent the variable obtained by linear transformation of the sub-task objective acquired by car n at time t. This represents the basic strategy acquired by car n at time t. Let represent the basic strategy acquired by car n at time t in the previous time step. This represents the final movement strategy obtained by car n at time t. Let A represent the observation of car 1 at time t. t S represents an action sequence. t Indicates the current state. V represents the output of the value critic network for the two action states at time t. t The output of the state-value commentator network is represented by the following sub-task network modules: Encoder is the network used for encoding, GRU is a gated recurrent unit, and Norm represents the normalization operation; in the execution policy network, Linear represents linear transformation, MLP represents feedforward neural network, and GRU represents gated recurrent unit; in the action-state-value commentator network: Linear represents linear transformation, and ReLU represents the execution activation function ReLU. Figure 3 The illustrated algorithm architecture diagram shows that a subtask network generates subtask objectives, an execution policy network generates movement policies based on the subtask objectives, an intrinsic reward mechanism module evaluates the policies through a critic network and generates intrinsic reward signals, and a state transition policy gradient module updates the parameters of the subtask network based on the new state and the subtask objectives. This algorithm architecture enables the agent to perform path planning in complex environments without fixed road networks.

[0054] Reference Figure 3 For example, at the beginning of each decision loop, the agents collectively face the current global state S. t The subtask network uses this global state S t As input, a subtask objective G is output after forward computation. t G t This represents a target point or direction vector in a low-dimensional feature space. In this embodiment, G... t Defined as a unit direction vector on a two-dimensional plane, originating from the current agent's position, representing the direction of the agent's expected movement within a preset number of steps, *c*. It should be noted that G... t It's not a specific coordinate location, but rather a subtask guidance signal. For example, in an outdoor environment without fixed roads, G... t This can correspond to the subtask of "moving a certain distance in the northeast direction". Decomposing the long-term ultimate goal into several directional sub-goals can reduce the decision depth. The subtask goal G is output in the subtask network. t Then, the sub-task target G t It is sent to the execution policy network to guide the selection of subsequent actions.

[0055] In step S3, according to the sub-task objective, the multiple agents are controlled to execute a sequence of actions with a preset number of steps through the sub-task network, wherein the state of each agent undergoes state transition based on the environmental dynamics model.

[0056] In this embodiment, reference is continued. Figure 3 The execution policy network is based on the current state S. t Sub-task objective G t Generate action sequence A t The action sequence A t It contains the control commands corresponding to each agent. In a centralized training scheme, the actions of multiple agents can be viewed as a combination of actions into an action sequence vector. (Where N is the number of agents), the execution policy network directly outputs the action sequence. Alternatively, a decentralized execution architecture can be adopted, where each agent determines its actions based on the sub-task objective and its own local observations. However, during the training phase, for ease of evaluation, a centralized execution policy network is used to manage global information. Action sequence A t The specific form depends on the motion model of the agent. For example, for a vehicle-type mobile robot, the action can include control variables for speed and steering angle. Multiple agents in the environment execute their respective actions simultaneously. After each step, the environmental state is updated. After continuously executing a preset number of steps (e.g., c steps), time reaches t+c, and the set of agents reaches a new state S. t+c .

[0057] During state evolution, interactions between agents and between agents and the environment need to be considered. For example, if multiple agents exist, collisions must be avoided; if obstacles exist in the environment, the agent's movement path must not cross them. These constraints are automatically guaranteed by environmental dynamics simulation: after each action, the environment update module calculates the next position based on each agent's action and current position, detects collisions or boundary violations, and corrects the agent's state if necessary (e.g., setting the velocity to zero upon collision). It should be noted that this application does not depend on a specific motion model or environment type; this method can be applied to any multi-agent environment with continuous or discrete states, requiring only adjustments to the agent dynamics model and reward function definitions.

[0058] In step S4, a reward value is calculated based on the execution result of the preset number of steps. The reward value includes environmental rewards and intrinsic rewards based on the multi-critic network.

[0059] In this embodiment, after each agent completes a preset number of steps according to the sub-task objective, the system calculates a comprehensive reward value that includes environmental rewards and intrinsic rewards from the multi-critic network. The environmental reward is determined based on the direct interaction between the agent and the environment; a positive reward is given for successfully approaching the target, while a negative reward is given for a collision. The intrinsic reward is generated by the multi-critic network after evaluating the agent's behavior from multiple perspectives, such as the agent's action continuity and the closeness of teamwork.

[0060] Furthermore, the environmental reward is calculated in real time by the environment module based on the change in distance between each agent and the final target location or the task completion event; the intrinsic reward is determined by the multi-critic network evaluating the achievement degree of the sub-task target.

[0061] Environmental rewards R env The settings are typically based on the overall task completion status. In path planning tasks, progressive environmental rewards can be given based on the changes in distance between multiple agents and their respective target points. For example, small positive rewards are given for each step based on the decrease in distance, large reward values ​​are given when the target is reached, and punitive negative rewards are given when a collision occurs or the boundary is exceeded. This environmental reward design ensures that only the final successful (or unsuccessful) result receives significant feedback, but the agents can still obtain some guidance through distance changes during the process.

[0062] Intrinsic reward R int Used to evaluate the completion of sub-task objectives, providing additional optimization signals to the agent at the sub-task level. The intrinsic reward is calculated by the multi-critic network through the intrinsic reward mechanism module. Specifically, after the policy execution network performs c steps, the trajectory τ of that sequence (including the initial state S) can be obtained. t Subtask objective G t Action sequence A t and the final state S t+c The intrinsic reward mechanism module calculates G. t With S t+c and S t The relationship between them. If the agent follows G... t If the agent successfully moves in the expected direction, a positive intrinsic reward is given; conversely, if the agent's behavior deviates from G, a negative intrinsic reward is given. t If guidance (such as moving in the wrong direction due to environmental interference or improper strategy) is provided, the intrinsic reward is relatively low or even negative, indicating that the subtask has not been effectively completed.

[0063] To quantitatively assess the above situation, cosine similarity is used to calculate the intrinsic reward, and the calculation formula is as follows:

[0064]

[0065] Among them, R int As an intrinsic reward, d cos (x, y) is the cosine similarity function used to calculate the similarity between x and y, and j is the j-th action starting from time t. This represents the local observation results of car i at time t. This represents the local observation results of car i at step j, starting from time t. Let be the sub-task objective obtained by car i at time t.

[0066] Finally, the total reward value for the current subtask phase is obtained by combining environmental and intrinsic rewards:

[0067] R = R env +α·R int

[0068] Where R is the total reward value and α is the weighting coefficient.

[0069] The influence of intrinsic reward on total reward can be controlled by adjusting the weighting coefficient α. Total reward not only reflects the progress of the overall goal but also the achievement of sub-task goals, making it richer than sparse rewards that rely solely on the environment, and thus helping to guide the gradual improvement of the strategy.

[0070] Furthermore, the intrinsic reward is determined by evaluating the achievement of the sub-task objective through the multi-critic network, including: evaluating the value of the current joint state and joint actions of the multi-agent through a dual action-state value critic network, outputting at least two action-state value estimates, and calculating the average of each state value estimate; using the deviation between each action-state value estimate and the average as an error metric, adjusting the estimation results of the critic network, generating an intrinsic reward signal based on the sub-task objective, and updating the policy network after weighted fusion of the intrinsic reward signal and the environmental reward.

[0071] In this embodiment, the generation process of intrinsic rewards involves a comprehensive evaluation of the achievement of sub-task objectives by a multi-critic network. First, a dual-action-state-value critic network is used, which receives the joint state and joint actions of multiple agents as input. The joint state reflects information such as the position and velocity of all agents in the environment, while the joint actions represent a series of actions taken by each agent.

[0072] Two critic networks independently evaluate the value of the current joint state and joint action, outputting at least two distinct action-state value estimates. These estimates reflect the potential value of taking specific joint actions in the current state, i.e., how these actions contribute to achieving the sub-task objective. Next, the average of these two value estimates is calculated; this average serves as a benchmark to measure the relative deviation of each critic network's estimation results.

[0073] An error metric is obtained by calculating the deviation between each action-state value estimate and the average. This error metric reflects the dispersion or uncertainty of the critic network's estimation results. Based on this error metric, the system adjusts the critic network's estimation results to generate a more accurate intrinsic reward signal. This adjustment mechanism helps improve the stability and reliability of the intrinsic reward signal, enabling it to better reflect the agent's behavior's actual contribution to the sub-task objective.

[0074] Finally, the generated intrinsic reward signal and the environmental reward are weighted and fused. This weighted fusion takes into account the different importance and reliability of the two reward signals, thus forming a comprehensive reward signal. This comprehensive reward signal will be used to update the policy network, guiding the agent to make better decisions in future path planning.

[0075] In step S5, based on the environmental reward and the intrinsic reward, the parameters of the multi-critic network, the policy network, and the parameters of each critic network in the multi-critic network are updated.

[0076] In this embodiment, after receiving environmental and intrinsic reward feedback, the actor-critic method in reinforcement learning is used to update the execution policy network (actor) and the critic network, as well as the parameters of the execution policy network and each critic network. Based on the estimates from the critic networks, the policy gradient direction is calculated, and the gradient equation of the execution policy network can be obtained:

[0077]

[0078] Where, π θ To execute the joint action probability distribution output by the policy network, A t For an action sequence, S t For the initial state, G t For sub-task objectives.

[0079] Furthermore, updating the parameters of the policy network and the parameters of each critic network in the multiple critic network includes: updating the parameters of the policy network using a policy gradient algorithm; and updating the parameters of each critic network in the multiple critic network using temporal difference error.

[0080] The critic network parameters are updated using temporal difference (TD) learning. This utilizes the state S after c steps. t+c Value estimation to update the initial state S t The value estimation is improved to reduce the difference between the two, i.e., the TD error. In specific implementations, a dual-critic network can be used to reduce overestimation of value, for example, by taking a certain value when calculating the target value Q. To avoid a single network overestimating future returns, two commentator networks update their parameters independently, while the state-value commentator network provides a benchmark and also minimizes the loss (V(S) - V). T (S)) 2 To train, where V(S) is the target value obtained using the updated network, V T (S) represents the target value obtained using the target network. Through the above updates, the policy execution network will gradually learn to select better action sequences under the guidance of sub-task objectives. Simultaneously, the multiple critic network continuously optimizes value estimation, ensuring more accurate and stable evaluation of the policy.

[0081] In multi-agent scenarios, a paradigm of centralized training and decentralized execution can be adopted. For example, during training, the critic network uses global state and joint actions to evaluate, thereby fully capturing the mutual influence between agents (such as the coordination value of avoiding collisions). During execution and deployment, each agent can execute the policy based solely on its own observations and shared sub-task objectives, without the need for real-time centralized control. This training method has been proven in the field of reinforcement learning to effectively improve the performance of multi-agent collaborative tasks.

[0082] In step S6, the parameters of the subtask network are updated by the state transition strategy gradient module according to the relationship between the new state of the multi-agent after the preset number of steps and the subtask target state.

[0083] In this embodiment, the parameters of the high-level sub-task network are directly updated through the state transition policy gradient module, reducing the dependence of the high-level policy on the value evaluation error of the low-level network. The optimization goal of the sub-task network is to generate high-value sub-task objectives, that is, to guide the agent towards a better intermediate state. Therefore, it is necessary to measure the current sub-task objective G. t The "good" or "bad" state. In other words, if the agent enters state S after executing a preset number of steps c, then... t+c If it possesses higher value (e.g., closer to the ultimate goal, higher future returns), then G t A good subtask should be considered good, and vice versa. In this embodiment, G is evaluated using a state value critic network and a state transition value critic network. t The value contribution of state S. State value critic network can evaluate state S. t+c Value V(S) t+c ) and initial state S t Value V(S) t The difference between the two is ΔV = V(S). t+c )-V(S t This reflects, to some extent, the increase in value during this period.

[0084] However, directly using ΔV as the optimization signal for the subtask network may result in significant noise, and the change in V is not entirely determined by the subtask objective G. t The decision is also related to the optimality of the execution strategy. Therefore, the State Transition Value Commentator Network is specifically used to evaluate "given subtask objective G". t "The value that can be achieved by state transition within c steps." It can be viewed as a simulation model or value network V. n (G t Output a pair of subtask objectives G. t Estimation of potential value. The value network V... n (G t This is compared with the actual value gain ΔV to determine the update direction of the sub-task network. For example, if ΔV is lower than V... n (G t This indicates that the execution strategy failed to fully realize G. t The potential value of G, but G t If the subtask itself is promising, the cause should be sought more from the execution strategy level; conversely, if ΔV is also not high and V n (G t If the evaluation itself is low, then it may be G. t This is not a valid subtask objective; the output direction of the subtask network needs to be adjusted.

[0085] Furthermore, the step of updating the parameters of the subtask network by means of the state transition policy gradient module based on the relationship between the new state of the multi-agent after performing the action at the preset number of steps and the subtask target state includes: calculating the relative value of the current subtask target based on the degree of deviation of the state of the multi-agent relative to the subtask target state at the end of the action at the preset number of steps; evaluating the state change caused by the action at the preset number of steps through a state transition value critic network to obtain the corresponding state transition advantage value; and combining the relative value with the state transition advantage value to calculate the state transition policy gradient of the subtask network to update the parameters of the subtask network.

[0086] Based on the factors mentioned above, this embodiment constructs a state transition strategy gradient module for subtasks, first calculating the relative value r of the subtask objective. g For example, take V(S) t+c G is characterized by the difference between ΔV and an expected baseline. t The actual effect; then calculate the state transition advantage Adv. n (This indicates the discrepancy between the expected and actual goals of a sub-task):

[0087] Adv n =ΔV-V n (Gt )

[0088] Among them, Adv n For the state transition advantage, ΔV is the actual value gain, V n (G t (This is a value network.)

[0089] Finally, the subtask network parameters are updated along the gradient direction, and their gradient equation can be expressed as:

[0090]

[0091] Where, θ g Let L be the probability distribution parameter, and r be the optimization objective. g For rewards derived from the environment, Adv n For the advantage of state transition, The subtask network outputs a probability distribution of the subtask objective based on the state, G. t For the sub-task objective, S t This is the initial state.

[0092] When Adv n It is positive, and r g When it is large, it indicates that G t This brings unexpected value gains, and there should be an increased tendency to generate similar sub-task objectives; conversely, when Adv... n When it is negative, then reduce things like G. t The generation probability is calculated. In this way, the sub-task network can adaptively adjust based on the sub-task execution results, gradually tending to generate sub-task objectives that contribute more to the final goal, greatly improving the stability of high-level decisions. It should be noted that because this update method does not rely on gradient backpropagation of low-level execution policies, it avoids interference with high-level policies when low-level policies are unstable, thus maintaining continuous improvement of sub-task policies in complex multi-agent tasks.

[0093] In step S7, steps S2 to S6 are repeated until training converges to obtain a multi-agent path planning target model, which includes the trained policy network and the trained subtask network model.

[0094] Steps S2 to S6 described above constitute a complete training loop, which will be executed repeatedly throughout the training process. Generally, a maximum number of iterations or convergence criteria are set, such as considering convergence as when the average reward reaches a certain threshold and fluctuates little. The new strategy generated in each loop gradually replaces the old strategy, continuously improving the agent's path planning ability. In the early stages of training, due to inaccurate policy and value estimations, the agents may behave randomly. The internal reward mechanism ensures they continue to strive towards the sub-task goal, thus gradually accumulating positive experience. After a period of training, the multi-agent cooperative strategy matures, and each agent can cooperate to select actions according to the sub-task goal, avoid obstacles, and move towards the final destination. At this point, training can be stopped and the network parameters fixed to obtain the final path planning strategy model.

[0095] In practical applications, the trained model is deployed into a multi-agent system, where each agent makes online decisions using a sub-task network and an execution policy network. Since coordination among the agents is already included during training, centralized control is unnecessary during deployment; agents only need to share the current global state or appropriate local information to make their own decisions. In each decision cycle, all agents obtain a unified sub-task objective for the next stage through the sub-task network and each calculates the required action through the execution policy network. After executing the action, the agent updates its state and enters the next decision cycle until the task is completed. This approach guides multiple agents to autonomously plan paths in complex environments without road networks, reaching their respective goals in stages. Compared to single-layer reinforcement learning algorithms, this application significantly improves training stability and sample efficiency; compared to traditional rule-based algorithms, the method in this application enables agents to autonomously learn and adapt to their environment, coping with dynamic and complex scenarios.

[0096] The effectiveness of the multi-agent path planning method provided in this application will be explained in detail below with specific applications. Figure 4 This is a comparison chart of the scores of the multi-agent path planning simulation platform provided in this application with the scores of classic algorithms in the same field. Figure 4In the figure, the horizontal axis represents the number of steps, M represents millions, and the vertical axis represents the average reward score. The SS-MAPF (red curve) algorithm (based on this application) represents the algorithm constructed in this application; MADDPG (yellow curve) represents the multi-agent deep deterministic policy gradient algorithm; MAPPO (purple curve) represents the multi-agent proximal policy optimization algorithm; QMIX (blue curve) represents the action-state-value hybrid network algorithm; and FACMAC (green curve) represents the factorization multi-agent central policy gradient algorithm. As can be seen from the figure, SS-MAPF (red curve) exhibits a significant advantage throughout the training process. In the early stages of training, the score of SS-MAPF rises rapidly, quickly surpassing other algorithms. As the number of training steps increases, the score of SS-MAPF continues to steadily improve, eventually reaching a high score close to -30 at 1.0M steps, significantly outperforming other algorithms. In contrast, the performance of classic algorithms such as MADDPG (yellow curve), MAPPO (purple curve), QMIX (blue curve), and FACMAC (green curve) is relatively weak. The scores of these algorithms also increased to varying degrees in the early stages of training, but then stabilized, and even fluctuated and declined. At 1.0M steps, the scores of these algorithms were mostly between -60 and -80, significantly lower than SS-MAPF. Specifically, MADDPG scored around -60, showing relatively stable performance, but with limited improvement; MAPPO scored around -70, with large fluctuations, indicating that its strategy was not stable enough in different training stages; QMIX scored around -80, with the worst overall performance, indicating certain limitations in handling multi-agent path planning problems; FACMAC scored around -70, performing well in the early stages of training, but then showing a significant downward trend. In contrast, the SS-MAPF algorithm provided in this application showed a significant performance advantage in multi-agent path planning tasks, with its score continuously improving during training and ultimately reaching a level far exceeding that of other classic algorithms. This indicates that the SS-MAPF algorithm has stronger learning ability and better policy optimization effect when dealing with complex multi-agent path planning problems, and can provide agents with more efficient and reliable path planning solutions.

[0097] In summary, this application provides a multi-agent path planning method in an environment without a fixed road network. This application utilizes a subtask-commentator architecture to decompose long-cycle tasks into short-cycle subtasks, alleviating the sparse reward problem. It introduces a multi-commentator network to evaluate the value of joint agent actions, reducing the value estimation error that may occur with a single commentator and improving the accuracy and robustness of the value function estimation. It designs an intrinsic reward based on the subtask objective, providing additional guidance signals to the agent at each subtask stage, encouraging the execution policy to optimize towards the subtask objective, thereby avoiding direct reliance on sparse global rewards for policy updates and preventing the amplification of value estimation errors. Furthermore, it introduces a state transition policy gradient module, breaking the limitation that the subtask network relies on the reverse gradient update of the execution policy network, instead directly calculating gradient updates for subtask policies based on the subtask execution results, thus improving the stability of high-level policy updates.

[0098] Furthermore, this application utilizes a reinforcement learning algorithm that combines multiple critics and hierarchical policy gradients to achieve multi-agent path planning in environments without fixed road networks, enabling more stable convergence and obtaining superior planning results.

[0099] Example 2

[0100] Corresponding to the method embodiments, another aspect of the embodiments of this application also provides a multi-agent path planning device in an environment without a fixed road network. Figure 5 This diagram illustrates the structure of a multi-agent path planning device in a non-fixed road network environment provided in an embodiment of this application. The multi-agent path planning device in a non-fixed road network environment is a... Figure 1 The device corresponding to the multi-agent path planning method in the non-fixed road network environment described in the corresponding embodiment is implemented through a virtual device. Figure 1 The multi-agent path planning method in the corresponding embodiment for an environment without a fixed road network can be executed by electronic devices, such as network devices, terminal devices, or servers. Specifically, the multi-agent path planning device in an environment without a fixed road network in this application embodiment includes:

[0101] Multi-agent path planning model building unit 01 is used to build a multi-agent path planning model with a hierarchical architecture including a sub-task network, an execution policy network, and a multiple critic network.

[0102] The subtask target generation unit 02 is used to generate subtask targets for multiple agents within a fixed time span based on the current global state of the multiple agents through the subtask network.

[0103] Furthermore, the subtask objective is a unit vector in a preset dimension space; the fixed time span includes multiple decision cycles with the preset number of steps, and the subtask network outputs the corresponding subtask objective at the beginning of each decision cycle.

[0104] The task execution control unit 03 is used to control the multiple intelligent agents to execute a sequence of actions with a preset number of steps through the sub-task network according to the sub-task objective, wherein the state of each intelligent agent undergoes state transition according to the environmental dynamics model;

[0105] The reward value calculation unit 04 is used to calculate the reward value based on the execution result of the preset number of steps. The reward value includes environmental rewards and intrinsic rewards based on the multi-critic network.

[0106] Furthermore, the intrinsic reward is determined by evaluating the achievement of the sub-task objective through the multi-critic network, including: evaluating the value of the current joint state and joint actions of the multi-agent through a dual action-state value critic network, outputting at least two action-state value estimates, and calculating the average of each state value estimate; using the deviation between each action-state value estimate and the average as an error metric, adjusting the estimation results of the critic network, generating an intrinsic reward signal based on the sub-task objective, and updating the policy network after weighted fusion of the intrinsic reward signal and the environmental reward.

[0107] The policy network update unit 05 is used to update the parameters of the multi-critic network, the policy network, and each critic network in the multi-critic network based on the environmental reward and the intrinsic reward.

[0108] Furthermore, updating the parameters of the policy network and the parameters of each critic network in the multiple critic network includes: updating the parameters of the policy network using a policy gradient algorithm; and updating the parameters of each critic network in the multiple critic network using temporal difference error.

[0109] The subtask adaptive optimization unit 06 is used to update the parameters of the subtask network by means of the state transition strategy gradient module according to the relationship between the new state of the multi-agent after the action is performed by the preset number of steps and the subtask target state;

[0110] Furthermore, the step of updating the parameters of the subtask network by means of the state transition policy gradient module based on the relationship between the new state of the multi-agent after performing the action at the preset number of steps and the subtask target state includes: calculating the relative value of the current subtask target based on the degree of deviation of the state of the multi-agent relative to the subtask target state at the end of the action at the preset number of steps; evaluating the state change caused by the action at the preset number of steps through a state transition value critic network to obtain the corresponding state transition advantage value; and combining the relative value with the state transition advantage value to calculate the state transition policy gradient of the subtask network to update the parameters of the subtask network.

[0111] The target model construction unit 07 is used to repeatedly execute steps S2 to S6 until training converges to obtain a multi-agent path planning target model, which includes the trained policy network and the trained sub-task network model.

[0112] Furthermore, the reward value calculation unit 04 includes:

[0113] An environmental reward calculation unit is used to calculate the environmental reward in real time by the environmental module based on the change in distance between each agent and the final target location or the task completion event.

[0114] An intrinsic reward calculation unit is used to determine the intrinsic reward by evaluating the achievement of the sub-task objective through the multi-critic network.

[0115] It should be noted that the specific implementation and technical effects of the multi-agent path planning device in the non-fixed road network environment in this application embodiment can be referred to Figure 1 The corresponding multi-agent path planning methods in environments without fixed road networks will not be elaborated here.

[0116] Example 3

[0117] Corresponding to the method embodiments, this application also provides a multi-agent path planning device in an environment without a fixed road network, such as a terminal or a server. The server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal can be a smartphone, tablet, laptop, desktop computer, etc., but is not limited to these.

[0118] An example diagram of the hardware structure block diagram of the multi-agent path planning device in an environment without a fixed road network provided in this application is shown below. Figure 6 As shown, it may include:

[0119] Processor 1, communication interface 2, memory 3, and communication bus 4;

[0120] The processor 1, communication interface 2, and memory 3 communicate with each other via communication bus 4.

[0121] Optionally, communication interface 2 can be an interface of a communication module, such as the interface of a GSM module;

[0122] Processor 1 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application.

[0123] Memory 3 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0124] Specifically, processor 1 is used to execute the computer program stored in memory 3 to perform the following steps:

[0125] Step S1: Construct a multi-agent path planning model with a hierarchical architecture including a sub-task network, an execution policy network, and a multiple critic network;

[0126] Step S2: Based on the current global state of the multi-agent network, the sub-task objectives of multiple agents are generated within a fixed time span.

[0127] Step S3: According to the sub-task objective, control the multiple agents to execute a sequence of actions with a preset number of steps through the sub-task network, wherein the state of each agent changes according to the environmental dynamics model;

[0128] Step S4: Calculate the reward value based on the execution result of the preset number of steps. The reward value includes environmental rewards and intrinsic rewards based on the multi-critic network.

[0129] Step S5: Based on the environmental reward and the intrinsic reward, update the parameters of the multi-critic network, the policy network, and the parameters of each critic network in the multi-critic network;

[0130] Step S6: The parameters of the subtask network are updated by the state transition strategy gradient module according to the relationship between the new state of the multi-agent after the preset number of steps and the subtask target state;

[0131] Step S7: Repeat steps S2 to S6 until training converges to obtain a multi-agent path planning target model. The multi-agent path planning target model includes the trained policy network and the trained sub-task network model.

[0132] The above-mentioned products can execute the methods provided in the embodiments of this application, and have the corresponding functional modules and beneficial effects for executing the methods. Technical details that are not described in detail in this embodiment can be found in the request processing method based on the large language model provided in the embodiments of this application.

[0133] Example 4

[0134] In this embodiment of the application, a storage medium is also provided, which can store a program suitable for execution by a processor, the program being used for:

[0135] Step S1: Construct a multi-agent path planning model with a hierarchical architecture including a sub-task network, an execution policy network, and a multiple critic network;

[0136] Step S2: Based on the current global state of the multi-agent network, the sub-task objectives of multiple agents are generated within a fixed time span.

[0137] Step S3: According to the sub-task objective, control the multiple agents to execute a sequence of actions with a preset number of steps through the sub-task network, wherein the state of each agent changes according to the environmental dynamics model;

[0138] Step S4: Calculate the reward value based on the execution result of the preset number of steps. The reward value includes environmental rewards and intrinsic rewards based on the multi-critic network.

[0139] Step S5: Based on the environmental reward and the intrinsic reward, update the parameters of the multi-critic network, the policy network, and the parameters of each critic network in the multi-critic network;

[0140] Step S6: The parameters of the subtask network are updated by the state transition strategy gradient module according to the relationship between the new state of the multi-agent after the preset number of steps and the subtask target state;

[0141] Step S7: Repeat steps S2 to S6 until training converges to obtain a multi-agent path planning target model. The multi-agent path planning target model includes the trained policy network and the trained sub-task network model.

[0142] The above-mentioned products can execute the methods provided in the embodiments of this application, and have the corresponding functional modules and beneficial effects for executing the methods. Technical details that are not described in detail in this embodiment can be found in the request processing method based on the large language model provided in the embodiments of this application.

[0143] Optionally, the refined and extended functions of the program can be found in the description above.

[0144] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0145] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0146] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of those different embodiments or examples.

[0147] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "a plurality of" means two or more, unless otherwise explicitly specified.

[0148] The above description is merely an exemplary embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various variations or substitutions within the technical scope described in this application, and these should all be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A multi-agent path planning method in an environment without a fixed road network, characterized in that, include: Step S1: Construct a multi-agent path planning model with a hierarchical architecture including a sub-task network, an execution policy network, and a multiple critic network; Step S2: Based on the current global state of the multi-agent network, the sub-task objectives of multiple agents are generated within a fixed time span. Step S3: According to the sub-task objective, control the multiple agents to execute a sequence of actions with a preset number of steps through the sub-task network, wherein the state of each agent changes according to the environmental dynamics model; Step S4: Calculate the reward value based on the execution result of the preset number of steps. The reward value includes environmental rewards and intrinsic rewards based on the multi-critic network. Step S5: Based on the environmental reward and the intrinsic reward, update the parameters of the multi-critic network, the policy network, and the parameters of each critic network in the multi-critic network; Step S6: The parameters of the subtask network are updated by the state transition strategy gradient module according to the relationship between the new state of the multi-agent after the preset number of steps and the subtask target state; Step S7: Repeat steps S2 to S6 until training converges to obtain a multi-agent path planning target model. The multi-agent path planning target model includes the trained policy network and the trained sub-task network model.

2. The multi-agent path planning method according to claim 1, characterized in that, The subtask objective is a unit vector in a preset dimensional space; The fixed time span includes multiple decision cycles with the preset number of steps, and at the beginning of each decision cycle, the subtask network outputs the corresponding subtask objective.

3. The multi-agent path planning method according to claim 2, characterized in that, The environmental reward is calculated in real time by the environment module based on the change in distance between each agent and the final target location or the task completion event; The intrinsic reward is determined by evaluating the achievement of the sub-task objective through the multi-critic network.

4. The multi-agent path planning method according to claim 3, characterized in that, The intrinsic reward is determined by evaluating the achievement of the sub-task objective through the multi-critic network, including: The joint state and joint actions of the current multi-agent are evaluated by a dual action-state value critic network, outputting at least two action-state value estimates, and the average of the state value estimates is calculated. The deviation between each action-state value estimate and the average value is used as an error metric to adjust the estimation results of the critic network, generate an intrinsic reward signal based on the sub-task objective, and update the policy network by weighted fusion of the intrinsic reward signal and the environmental reward.

5. The multi-agent path planning method according to claim 1, characterized in that, Updating the parameters of the policy network and the parameters of each critic network in the multi-critic network includes: The parameters of the policy network are updated using the policy gradient algorithm; The parameters of each critic network in the multi-critic network are updated using time difference error.

6. The multi-agent path planning method according to claim 1, characterized in that, The step of updating the parameters of the sub-task network by the state transition strategy gradient module according to the relationship between the new state of the multi-agent after performing the action at the preset number of steps and the sub-task target state includes: The relative value of the current subtask target is calculated based on the degree of deviation of the state of the multi-agent at the end of the preset number of steps relative to the state of the subtask target. The state transition advantage value is obtained by evaluating the state changes caused by the preset number of steps through a state transition value critic network. The relative value is combined with the state transition advantage value to calculate the state transition policy gradient of the sub-task network, thereby updating the parameters of the sub-task network.

7. A multi-agent path planning device in an environment without a fixed road network, characterized in that, include: A multi-agent path planning model building unit is used to construct a multi-agent path planning model with a hierarchical architecture including a sub-task network, an execution policy network, and a multiple critic network. The subtask target generation unit is used to generate subtask targets for multiple agents within a fixed time span based on the current global state of the multiple agents through the subtask network. The task execution control unit is used to control the multiple agents to execute a sequence of actions with a preset number of steps through the sub-task network according to the sub-task objective, wherein the state of each agent changes according to the environmental dynamics model; The reward value calculation unit is used to calculate the reward value based on the execution result of the preset number of steps. The reward value includes environmental rewards and intrinsic rewards based on the multi-critic network. The policy network update unit is used to update the parameters of the multi-critic network, the policy network, and the parameters of each critic network in the multi-critic network based on the environmental reward and the intrinsic reward. The subtask adaptive optimization unit is used to update the parameters of the subtask network by means of the state transition strategy gradient module based on the relationship between the new state of the multi-agent after the preset number of steps and the subtask target state; The target model building unit is used to repeatedly execute steps S2 to S6 until training converges to obtain a multi-agent path planning target model, which includes the trained policy network and the trained sub-task network model.

8. The multi-agent path planning device according to claim 7, characterized in that, The reward value calculation unit includes: An environmental reward calculation unit is used to calculate the environmental reward in real time by the environmental module based on the change in distance between each agent and the final target location or the task completion event. An intrinsic reward calculation unit is used to determine the intrinsic reward by evaluating the achievement of the sub-task objective through the multi-critic network.

9. A multi-agent path planning device in an environment without a fixed road network, characterized in that, include: Memory, used to store computer programs; A processor is configured to invoke and execute the computer program to implement the steps of the multi-agent path planning method as described in any one of claims 1 to 6.

10. A storage medium, characterized in that, Includes a software program adapted for a processor to execute the steps of the multi-agent path planning method as described in any one of claims 1 to 6.

Images