A learning multi-agent path planning method based on local traffic pressure perception

Abstract

This invention discloses a learning-based multi-agent path planning method based on local traffic pressure perception, belonging to the field of multi-agent path planning technology. It includes adding a local traffic pressure calculation unit, a traffic pressure reward adjustment unit, and a selective communication gating unit to a basic learning-based path planning strategy network to form a multi-agent path planning model. The model is then trained by combining the current position, target position, local observation data, and communication observation data of each agent. The data to be detected is input into the trained multi-agent path planning model to generate collision-free executable action signals. This invention can reduce invalid waiting and local conflicts in high-pressure areas, suppress the activation of redundant communication features in low-pressure states, and improve the long-term task throughput efficiency of the lifetime multi-agent path planning system.

Description

Technical Field

[0001] This invention relates to the field of multi-agent path planning technology, and in particular to a learning-based multi-agent path planning method based on local traffic pressure perception. Background Technology

[0002] Multi-agent path planning aims to plan collision-free paths for multiple agents, enabling each agent to reach the target location from its starting position. Unlike one-time path planning, lifetime multi-agent path planning allows agents to continue receiving new targets after completing their current task, thus enabling the system to operate continuously over a long period. This type of problem is widely found in automated warehousing, smart factories, and logistics sorting systems.

[0003] Existing technologies mainly include search-based path planning methods and learning-based path planning methods. Search-based methods obtain collision-free paths through conflict search, priority planning, local replanning, or heuristic search, achieving high-quality planning results when the number of agents is small. However, as the number of agents increases, the search space and online computational overhead increase significantly. Learning-based methods typically employ a centralized training and distributed execution framework, mapping local agent observations to action probabilities through neural networks and combining neighborhood communication and execution coordination modules to complete online decision-making. These methods have advantages such as fast inference speed and good scalability, making them suitable for continuous task scenarios with hundreds or even more agents.

[0004] Existing learning-based multi-agent path planning methods generally rely on a fixed local field of view to acquire information such as obstacles, neighboring agents, and target direction, and employ fixed or near-fixed communication mechanisms to achieve neighborhood information sharing. However, they still have the following drawbacks: 1. Lack of explicit modeling of local traffic conditions. Existing policy networks mainly rely on raw local observations for feature extraction, making it difficult to directly distinguish between normal high-density traffic and high-coordination pressure states that may lead to congestion propagation. This results in the inability to alleviate waiting, conflict, and backtracking behaviors in bottleneck areas in a timely manner.

[0005] 2. Lack of adaptive adjustment in communication feature contribution. Existing methods typically use neighborhood communication information in a fixed manner, making it difficult to dynamically adjust the contribution of communication features to policy input based on local traffic conditions. This results in redundant communication features being fully activated under low-pressure conditions, while insufficient neighborhood coordination information may be lacking under high-pressure conditions.

[0006] 3. The reward mechanism and communication mechanism are independent of each other. Some existing solutions only apply congestion penalties to the reward function or only control communication through a gating network. The two modules lack a unified state signal, which can easily lead to inconsistent optimization objectives and prevent the formation of coordinated regulation oriented towards local traffic pressure.

[0007] 4. Long-term transportation efficiency in high-density structured scenarios still needs improvement. When a large number of agents repeatedly pass through aisle walkways, intersections, and near sorting workstations, local conflicts will continue to propagate, causing a decrease in throughput, an increase in accumulated waiting time, and the continuous activation of redundant communication features.

[0008] Therefore, how to provide a learning-based multi-agent path planning method based on local traffic pressure perception, while maintaining distributed execution and lightweight network structure, and improving local coordination capabilities in high-density scenarios, is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0009] In view of this, the present invention provides a learning-based multi-agent path planning method based on local traffic pressure perception. By fusing local occupancy density ratio and neighborhood activity, local traffic pressure is constructed. A unified traffic pressure signal is used to drive reward adjustment and selective communication gating, enabling agents to dynamically change learning constraints and communication feature contributions according to the coordination needs of different areas. This reduces ineffective waiting and local conflicts in high-pressure areas, suppresses the activation of redundant communication features in low-pressure states, and improves the long-term task throughput efficiency of the lifelong multi-agent path planning system.

[0010] To achieve the above objectives, the present invention adopts the following technical solution: A learning-based multi-agent path planning method based on local traffic pressure perception includes the following steps: S1. Obtain the current position, target position, local observation data, and communication observation data of each agent, and preprocess them to form an initial dataset; S2. Add a local traffic pressure calculation unit, a traffic pressure reward adjustment unit, and a selective communication gating unit to the basic learning-based path planning strategy network to construct a learning-based multi-agent path planning model, wherein the traffic pressure reward adjustment unit and the selective communication gating unit are connected in parallel. S3. Input the initial dataset into the local traffic pressure calculation unit to calculate the local traffic pressure corresponding to each agent. S4. During the training phase, the pressure perception reward is updated through the traffic pressure reward adjustment unit, and the communication feature gating coefficient is updated through the selective communication gating unit to train the multi-agent path planning model. During the inference phase, the communication feature gating coefficient is updated according to the local traffic pressure through the selective communication gating unit, and the trained multi-agent path planning model is used for inference. S5. Input the data to be detected into the trained multi-agent path planning model to generate collision-free executable action signals. The agent executes the action according to the signal content and updates the agent's position. S6. Using the updated agent position as new input data, repeat steps S3-S5 to achieve continuous collision-free movement of multiple agents within a shared limited space.

[0011] Optionally, local traffic pressure is calculated in S3. It is obtained by weighted fusion of local occupancy density ratio and neighborhood motion activity, and the corresponding expression is:

[0012] Where α and β are the fusion weights, satisfying , and , For intelligent agents In the local observation area The local density percentage For intelligent agents In the local observation area The activity level of the neighborhood movement.

[0013] Optionally, the local occupancy density ratio is used to describe the degree of space occupancy caused by obstacles and other agents within the agent's local field of vision, allowing the agent to... The local observation area is The total number of valid grids is The number of grids occupied by static obstacles is The number of grids occupied by other agents is Then the local occupancy density ratio can be expressed as: ; Neighborhood activity level is used to describe the activity level of local neighboring agents. At any moment The number of observed neighboring agents is From time to The number of neighboring agents whose locations changed is intelligent agent At any moment The neighborhood activity level is represented as: .

[0014] Optionally, in S4, updating the stress perception reward through the traffic stress reward adjustment unit includes: Set a stress trigger threshold and construct a stress reward system by combining local traffic pressure and penalty coefficient. ; Stress Rewards With basic rewards The updated stress perception reward is obtained by stacking it, and the corresponding expression is: .

[0015] Optionally, updating the communication characteristic gating coefficient in S4 via the selective communication gating unit includes: determining the communication characteristic gating coefficient based on the interval to which the local traffic pressure belongs. The corresponding expression is: , in, The first pressure threshold, The second pressure threshold, The preset communication feature gating coefficients are set for low, medium, and high pressure states, respectively, and satisfy the following conditions: .

[0016] Optionally, generating collision-free executable action signals in S5 includes: Scaling communication observations based on communication feature gating coefficients forms scaled communication observation data, which is then stitched together with local observation data to form gated observations. During the training phase, stress-aware rewards are used to optimize the basic learning path planning policy network; gated observations are input into the basic learning path planning policy network to generate action proposals for multiple agents. By utilizing the existing conflict resolution process in the basic learning-based path planning strategy network, action proposals from multiple agents are processed to generate collision-free executable actions.

[0017] Optionally, communication observations can be scaled based on communication feature gating coefficients. During scaling, only the activation level of communication features in the policy input is adjusted, without directly reducing the number of messages, communication bandwidth, or communication edges, resulting in gated observations. The expression is: , in, For intelligent agents At any moment The communication characteristic gating coefficient, For local observation, For communication observation.

[0018] As can be seen from the above technical solution, compared with the prior art, the present invention provides a learning-based multi-agent path planning method based on local traffic pressure perception, which has the following beneficial effects: 1) This invention overcomes the problem of insufficient description by a single density index by integrating the local occupancy density ratio and the activity of neighborhood movement, and simultaneously characterizing the degree of static passage space restriction and dynamic interaction needs; 2) This invention unifies traffic pressure signals to simultaneously drive reward adjustment and communication gating, so that the policy learning objective and the adjustment of communication feature contribution are optimized around the same local coordination needs; under low pressure, the contribution of communication features to policy input is weakened, while under high pressure, sufficient neighborhood communication features are retained, realizing adaptive adjustment of the degree of contribution of communication information, and suppressing the activation of redundant communication features without directly changing the number of messages, communication bandwidth or communication connection relationship. 3) The stress-aware reward in this invention can reduce the tendency of the strategy to remain in an inefficient and high-pressure local state for a long time, and improve the throughput of long-term tasks in scenarios with narrow channels and continuous bottlenecks, such as Warehouse, and improve the passage efficiency in structured high-density scenarios. 4) The online execution of this invention only adds local pressure calculation and gating operation, without global search or significant modification to the basic policy network, making it easy to integrate into existing learning-based path planning frameworks while maintaining distributed and lightweight characteristics; 5) This invention relies only on local observation and neighborhood communication. The pressure index, threshold, communication level and reward coefficient can be configured according to the map structure, robot density and communication conditions, adapting to different hardware platforms and business scenarios. It can be applied to warehousing, logistics and intelligent manufacturing systems with hundreds or even more mobile intelligent agents running continuously, and has good scalability and applicability. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0020] Figure 1 This is a flowchart of a learning-based multi-agent path planning method based on local traffic pressure perception disclosed in this invention. Figure 2 This is a schematic diagram of local traffic pressure calculation disclosed in this invention; Figure 3 This is a schematic diagram of the local traffic pressure-driven reward regulation and communication gating disclosed in this invention. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] Reference Figure 1 As shown, this invention discloses a learning-based multi-agent path planning method based on local traffic pressure perception, comprising the following steps: S1. Obtain the current position, target position, local observation data, and communication observation data of each agent, and preprocess them to form an initial dataset; S2. Add a local traffic pressure calculation unit, a traffic pressure reward adjustment unit, and a selective communication gating unit to the basic learning-based path planning strategy network to construct a learning-based multi-agent path planning model, wherein the traffic pressure reward adjustment unit and the selective communication gating unit are connected in parallel. S3. Input the initial dataset into the local traffic pressure calculation unit to calculate the local traffic pressure corresponding to each agent. S4. During the training phase, the pressure perception reward is updated through the traffic pressure reward adjustment unit, and the communication feature gating coefficient is updated through the selective communication gating unit to train the multi-agent path planning model. During the inference phase, the communication feature gating coefficient is updated according to the local traffic pressure through the selective communication gating unit, and the trained multi-agent path planning model is used for inference. S5. Input the data to be detected into the trained multi-agent path planning model to generate collision-free executable action signals. The agent executes the action according to the signal content and updates the agent's position. S6. Using the updated agent position as new input data, repeat steps S3-S5 to achieve continuous collision-free movement of multiple agents within a shared limited space.

[0023] Furthermore, the multi-agent operating environment is represented as an undirected graph. ,in Represents the set of passable locations. This indicates the connection relationships between adjacent passable locations. The system includes... A smart agent Each intelligent agent at any time With current location and current goals .

[0024] Within each time step, an agent can choose to move to an adjacent passable location or wait in place. The system must satisfy vertex conflict constraints and edge conflict constraints, meaning that no two agents can occupy the same location at the same time, nor can they exchange positions in adjacent time steps.

[0025] Each agent constructs a local observation area centered on its current location. The size of the local field of view can be selected as 11×11. The local observation includes at least the distribution of obstacles, passable areas, the locations of neighboring agents, the motion status of neighboring agents, the direction of its own target, and optional historical state information.

[0026] Furthermore, refer to Figure 2 As shown, S3 calculates local traffic pressure. It is obtained by weighted fusion of local occupancy density ratio and neighborhood motion activity, and the corresponding expression is:

[0027] Where α and β are the fusion weights, satisfying , and , For intelligent agents In the local observation area The local density percentage For intelligent agents In the local observation area The activity level of traffic in the neighborhood is considered, while the local occupancy density ratio reflects the degree of occupancy of local passage space, and the activity level of traffic in the neighborhood reflects the degree of local movement activity. The local traffic pressure obtained by combining the two is used to describe local coordination needs, but is not directly equivalent to the severity of congestion or deadlock.

[0028] Furthermore, the local occupancy density ratio is used to describe the degree of spatial occupancy within an agent's local field of vision caused by obstacles and other agents, allowing the agent to... The local observation area is The total number of valid grids is The number of grids occupied by static obstacles is The number of grids occupied by other agents is Then the local occupancy density ratio can be expressed as: ; In addition, different weights can be set for static obstacle occupation and dynamic agent occupation, using only passable grids as the denominator.

[0029] Neighborhood activity level is used to describe the activity level of local neighboring agents. At any moment The number of observed neighboring agents is From time to The number of neighboring agents whose locations changed is intelligent agent At any moment The neighborhood activity level is represented as: , Among them, the higher This indicates that many neighboring agents are moving simultaneously within the local interaction area, which may increase potential action conflicts and coordination needs. Therefore, neighborhood motion activity is used to reflect the degree of local motion activity, rather than directly measuring the degree of waiting, congestion, or deadlock.

[0030] Furthermore, in S4, the stress perception reward is updated through the traffic stress reward adjustment unit, including: Set a stress trigger threshold and construct a stress reward system by combining local traffic pressure and penalty coefficient. This allows the strategy to explicitly perceive local high-stress states during training; Stress Rewards With basic rewards The updated stress perception reward is obtained by stacking it, and the corresponding expression is: , ; in, The pressure trigger threshold, This is the pressure penalty coefficient. When At that time, no additional pressure or punishment is imposed; when At that time, the penalty increases linearly with the degree to which local traffic pressure exceeds the threshold.

[0031] The reward adjustment only affects policy parameter updates during the training phase. This additional reward is not calculated during the online inference phase, nor is it introduced into the additional search process.

[0032] Furthermore, in S4, updating the communication characteristic gating coefficient through the selective communication gating unit includes: determining the communication characteristic gating coefficient based on the interval to which the local traffic pressure belongs. This allows the neighborhood communication characteristics under high pressure to be more fully preserved in the policy input. The corresponding expression is: , in, The first pressure threshold, The second pressure threshold, The preset communication feature gating coefficients are set for low, medium, and high pressure states, respectively, and satisfy the following conditions: Under low pressure, communication characteristics are weakened; under medium pressure, some communication characteristics are retained; and under high pressure, complete communication characteristics are retained.

[0033] Furthermore, the generation of collision-free executable action signals in S5 includes: Scaling communication observations based on communication feature gating coefficients forms scaled communication observation data, which is then stitched together with local observation data to form gated observations. During the training phase, stress-perceived rewards are used to optimize the basic learning-based path planning policy network; gated observations are input into the basic learning-based path planning policy network, such as... Figure 3 As shown, action proposals are generated for multiple agents; By utilizing the existing conflict resolution process in the basic learning-based path planning strategy network, action proposals from multiple agents are processed to generate collision-free executable actions.

[0034] Furthermore, communication observations are scaled based on communication feature gating coefficients. During the scaling process, only the activation level of communication features in the policy input is adjusted, without directly reducing the number of messages, communication bandwidth, or communication edges, resulting in gated observations. The expression is: , in, For intelligent agents At any moment The communication characteristic gating coefficient, For local observation, For communication observation. Under low-pressure conditions, the contribution of communication features is weakened, while under high-pressure conditions, neighborhood communication features are more fully preserved.

[0035] Furthermore, the basic learning-based path planning policy network includes a policy decision-making unit, a conflict resolution unit, and an agent execution unit connected in sequence. Action proposals are generated based on the policy decision-making unit, specifically by outputting the action probability distribution based on gated observations. The corresponding expression is: , in, Represented as parameters The basic learning-based path planning strategy function is used to output the probability distribution of candidate actions based on gated observations. Candidate actions include moving up, down, left, right, and waiting in place.

[0036] In one specific embodiment, the method of this application was used to conduct experiments with 600 agents on four types of maps: Berlin, Paris, Sortation, and Warehouse. Each method used the same map, number of agents, task generation method, and 500-step evaluation time, and used 8 sets of paired random seeds for evaluation.

[0037] Experimental results show that, with 600 agents, the throughput of this invention is higher than the basic method SILLM on Sortation, Warehouse, and Berlin maps. Specifically, the throughput of the Sortation map increased from 16.378 to 16.465, a relative improvement of 0.533%; the throughput of the Warehouse map increased from 13.559 to 14.105, a relative improvement of 4.023%; and the throughput of the Berlin map increased from 7.052 to 7.102, a relative improvement of 0.702%. On the Paris map, the throughput decreased from 8.791 to 8.772, a relative change of -0.219%. The average relative throughput improvement across the four maps is approximately 1.26%.

[0038] In a large-scale experiment with 10,000 agents, the throughput of this invention on the Sortation map was 38.882, while that of the basic method was 38.875, with the two showing essentially the same performance. On the Warehouse map, the throughput of this invention increased from 36.452 to 36.751, a relative improvement of about 0.82%, indicating that the method is still applicable under large-scale agent conditions.

[0039] Ablation experiments further demonstrate that on the Warehouse map, using stress-aware reward regulation alone improves throughput by 2.832%, using communication feature gating alone improves throughput by 4.333%, and the complete method improves throughput by 5.091%. On the Sortation map, the throughput changes brought by each module are relatively small, with communication gating alone achieving a 0.647% improvement. These results indicate that communication feature gating contributes more significantly to the throughput improvement, while the synergistic effect between stress-reward regulation and communication gating is map-dependent.

[0040] Without altering the core idea of ​​this invention, the local field of view size, and The fusion weights, pressure thresholds, and communication feature gating levels can be adjusted according to the map structure, agent size, and task density.

[0041] The foregoing description enables those skilled in the art to implement or use this invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A learning-based multi-agent path planning method based on local traffic pressure perception, characterized in that, Includes the following steps: S1. Obtain the current position, target position, local observation data, and communication observation data of each agent, and preprocess them to form an initial dataset; S2. Add a local traffic pressure calculation unit, a traffic pressure reward adjustment unit, and a selective communication gating unit to the basic learning-based path planning strategy network to construct a learning-based multi-agent path planning model, wherein the traffic pressure reward adjustment unit and the selective communication gating unit are connected in parallel. S3. Input the initial dataset into the local traffic pressure calculation unit to calculate the local traffic pressure corresponding to each agent. S4. During the training phase, the pressure perception reward is updated through the traffic pressure reward adjustment unit, and the communication feature gating coefficient is updated through the selective communication gating unit to train the multi-agent path planning model. During the inference phase, the communication feature gating coefficient is updated according to the local traffic pressure through the selective communication gating unit, and the trained multi-agent path planning model is used for inference. S5. Input the data to be detected into the trained multi-agent path planning model to generate collision-free executable action signals. The agent executes the action according to the signal content and updates the agent's position. S6. Using the updated agent position as new input data, repeat steps S3-S5 to achieve continuous collision-free movement of multiple agents within a shared limited space.

2. The learning-based multi-agent path planning method based on local traffic pressure perception according to claim 1, characterized in that, Calculating local traffic pressure in S3 It is obtained by weighted fusion of local occupancy density ratio and neighborhood motion activity, and the corresponding expression is: Where α and β are the fusion weights, satisfying , and , For intelligent agents In the local observation area The local density percentage For intelligent agents In the local observation area The activity level of the neighborhood movement.

3. The learning-based multi-agent path planning method based on local traffic pressure perception according to claim 2, characterized in that, Local occupancy density ratio is used to describe the degree of space occupancy within an agent's local field of view caused by obstacles and other agents, allowing the agent to... The local observation area is The total number of valid grids is The number of grids occupied by static obstacles is The number of grids occupied by other agents is Then the local occupancy density ratio can be expressed as: ； Neighborhood activity level is used to describe the activity level of local neighboring agents. At any moment The number of observed neighboring agents is From time to The number of neighboring agents whose locations changed is intelligent agent At any moment The neighborhood activity level is represented as: 。 4. The learning-based multi-agent path planning method based on local traffic pressure perception according to claim 1, characterized in that, In S4, the stress perception reward is updated through the traffic stress reward adjustment unit, including: Set a stress trigger threshold and construct a stress reward system by combining local traffic pressure and penalty coefficient. ; Stress Rewards With basic rewards The updated stress perception reward is obtained by stacking it, and the corresponding expression is: 。 5. The learning-based multi-agent path planning method based on local traffic pressure perception according to claim 1, characterized in that, In S4, updating the communication characteristic gating coefficient through the selective communication gating unit includes: determining the communication characteristic gating coefficient based on the local traffic pressure interval. The corresponding expression is: ， in, The first pressure threshold, The second pressure threshold, The preset communication feature gating coefficients are set for low, medium, and high pressure states, respectively, and satisfy the following conditions: .

6. The learning-based multi-agent path planning method based on local traffic pressure perception according to claim 1, characterized in that, The generation of collision-free executable action signals in S5 includes: Scaling communication observations based on communication feature gating coefficients forms scaled communication observation data, which is then stitched together with local observation data to form gated observations. During the training phase, stress-aware rewards are used to optimize the basic learning path planning policy network; gated observations are input into the basic learning path planning policy network to generate action proposals for multiple agents. By utilizing the existing conflict resolution process in the basic learning-based path planning strategy network, action proposals from multiple agents are processed to generate collision-free executable actions.

7. The learning-based multi-agent path planning method based on local traffic pressure perception according to claim 6, characterized in that, Communication observations are scaled based on communication feature gating coefficients. During scaling, only the activation level of communication features in the policy input is adjusted; the number of messages, communication bandwidth, or communication edges are not directly reduced, resulting in gated observations. The expression is: ， in, For intelligent agents At any moment The communication characteristic gating coefficient, For local observation, For communication observation.

Images