An adaptive distributed multi-robot path planning method

Abstract

The application discloses a kind of self-adapting distributed multi-robot path planning method, solve the problem that multiple agents are easily trapped in deadlock and the problem of high computing cost under partially observable and communication limited environment.This method is based on the partially observable multiple agent path planning model under global map prior, constructs a kind of dynamic strategy switching framework that fuses deep reinforcement learning and heuristic search.The framework adaptively adjusts the decision mode through environment density evaluation (PSS): in low-density scene, heuristic search is used to ensure global navigation efficiency, and in high-density area, it is switched to reinforcement learning to handle real-time dynamic obstacle avoidance and cooperation.Meanwhile, a fault monitoring mechanism (FSS) is introduced to ensure the continuity of the solution when the search is trapped in local deadlock.The application realizes the complementary advantages of the two types of algorithms, with low computing overhead, precise dynamic obstacle avoidance, etc., significantly improving the system robustness and engineering landing value of the robot cluster under non-ideal network.

Description

Technical Field

[0001] This invention relates to the field of intelligent robot pathfinding technology, and in particular to a distributed path planning and decision-making framework that integrates reinforcement learning and heuristic rules. It features low computational overhead, accurate dynamic obstacle avoidance, effective avoidance of local livelock, and consideration of global navigation efficiency. Background Technology

[0002] With the profound evolution of industrial automation and intelligence, the efficient collaboration of large-scale robot swarms in dynamic and complex environments has become a core element in improving industrial efficiency. As a key underlying technology for multi-agent systems, Multi-Agent Path Finding (MAPF) aims to provide swarms with a collaborative navigation solution that balances absolute safety and global optimization, holding significant application value in areas such as intelligent warehouse scheduling and autonomous driving game theory. However, the effective operation of traditional MAPF algorithms highly depends on stable, real-time global information interaction between agents. In real-world industrial scenarios, non-ideal network environments (such as weak networks and communication blind spots) are common due to occlusion, interference, and physical environmental limitations. Communication delays, data loss, or connection interruptions can easily induce the paralysis of highly coupled centralized systems, seriously threatening the operational reliability of multi-robot systems. Especially under partially observable constraints, ensuring that agents can still perform robust autonomous decision-making and path replanning based solely on local observations under extreme conditions of missing perception information and communication interruptions has become a key technological barrier restricting the transition of multi-agent systems from ideal experimental environments to large-scale industrial deployment. Summary of the Invention

[0003] Objective: To address the computational complexity and system cost issues arising from the global state sharing assumption in centralized planning, and the problems of incomplete decision information and local deadlock caused by local observations in current distributed planning, this invention first proposes the MI-POMAPF problem, which is more in line with real-world scenarios. Furthermore, based on the MI-POMAPF problem, this invention constructs a strategy switching algorithm that considers local observations, static map priors, and collaborative decision-making and heuristic search, proposing an adaptive distributed multi-robot path planning method.

[0004] Technical Solution: To achieve the objectives of this invention, the technical solution adopted is: an adaptive distributed multi-robot path planning method, comprising the following steps:

[0005] Distributed decision-making CT-HA* is implemented based on static maps and local observation information, and distributed obstacle avoidance is achieved by combining global static maps with local dynamic observations.

[0006] Simultaneously, the CT-EPOM framework, an evolutionary strategy optimization algorithm based on temporal communication and memory mechanisms, is constructed, and a training strategy is executed.

[0007] Finally, dynamic policy switching is performed based on two types of policy switchers: agent density CT-PSS and cyclic detection CT-FSS. By evaluating the environmental state of the agent in real time, the optimal decision policy is dynamically selected.

[0008] Furthermore, the local observation information is designed to be represented as a sparse matrix, which includes a heuristic cost matrix, an agent position matrix, and an agent target matrix.

[0009] The heuristic cost matrix, or Cost-to-go matrix, represents the spatial cost information related to the greedy movement strategy in the local environment; this channel integrates two types of spatial information:

[0010] First, the distribution of static obstacles within the local observation range, with The first is to indicate the location of an impassable obstacle; the second is to show the change in time step cost when performing greedy movement in different directions with the current agent's position as a reference.

[0011] For feasible locations, in intervals The integer or real value within represents the change in time step relative to the current state, where a negative value indicates that the remaining shortest path length decreases after moving toward the target, and a positive value indicates that the path length increases due to deviation from the target.

[0012] Meanwhile, by encoding information on obstacle constraints and greedy cost changes, this matrix provides directional guidance for action selection similar to a potential field, rather than direct deterministic path guidance, while preserving local obstacle constraints.

[0013] The agent position matrix represents the spatial distribution of all agents within the current local observation range; this channel is represented by a binary occupancy grid of the same size as the observation window: grids occupied by agents are marked as 1, otherwise as 0; at the same time, its spatial structure is consistent with other observation channels.

[0014] The agent's target matrix, also known as the endpoint matrix, represents the local relative orientation of the agent's global target. It is the same size as the local observation window, with the target position marked as 1 and the other positions marked as 0.

[0015] When the target is outside the observation range, it is mapped to the range boundary. This target matrix provides target-oriented cues for action selection while maintaining local spatial consistency. It is used to fuse with the heuristic cost matrix and agent position matrix channels.

[0016] Finally, these three matrices are overlapped and used as the overall input to the decision-making algorithm based on the local observation information.

[0017] Furthermore, a static map is introduced as a global prior and fused with local observation information to realize a distributed decision-making algorithm;

[0018] The algorithm is based on the A* algorithm for single agents. It utilizes the global environment structure and local dynamic changes in the A* search framework to achieve online adaptive update of the path.

[0019] All agents acquire a global static map, meaning that the agents acquire the static obstacle positions of the entire scene; in addition, each agent acquires the positions of other agents within its local observation range centered on itself, which is equivalent to dynamic obstacle changes within a local range.

[0020] Based on the positions of obstacles in the static map and the positions of agents within a local area, the A* algorithm performs corresponding heuristic path planning to determine the agent's decision action at the current time step.

[0021] Specifically, the algorithm not only sets obstacles in the static map as hard constraints, but also introduces a local perception range and models the positions of other agents within that range as hard constraints.

[0022] During the heuristic search process, the algorithm still follows the evaluation function of the original algorithm. Provide heuristic guidance based on the time already spent. Based on this, calculate the heuristic value of the movable point after removing all hard constraints. And select the final and smallest point as the next point to move to;

[0023] At the execution level, each time a complete safe path to the destination is generated, only the first discrete action of that path at the current time step is output and executed.

[0024] Meanwhile, based on the local observation information at each time step, the positions of other agents are obtained in real time, changes in local dynamic obstacles are processed, and action decisions for the current time step are made in real time.

[0025] Furthermore, in a local observation environment, a loop detection and action selection strategy is introduced, the specific process of which is as follows:

[0026] Predict the agent's next position before executing a decision. and with the past A set of historical states at each time step Perform a comparison;

[0027] like This means that the system is determined to be trapped in a cyclical oscillation, at which point... If the probability triggers an action intervention, the current action is changed to wait in place. This wait in place is executed only once, and the next action prediction will continue to repeat the above action verification and execution logic; otherwise, the predicted action is executed normally.

[0028] Furthermore, an evolutionary strategy optimization model CT-EPOM based on temporal communication and memory mechanisms is constructed as follows:

[0029] This model adopts an Actor-Critic architecture, where the Actor and Critic share the feature extraction backbone network but are independent in the final output layer. The Actor outputs the action probability distribution. Critic outputs state value ;

[0030] The backbone network consists of a heterogeneous encoder and a temporal memory fusion module, including: a local spatial coding branch, a communication intent coding branch, and a temporal and feature fusion module;

[0031] The local spatial encoding branch utilizes a CNN architecture to deeply extract local spatial topological features, which are used to guide the agent to avoid obstacles and approach the target; the communication intent encoding branch is used to extract the cooperative movement trends and potential interaction intentions of neighboring agents.

[0032] The aforementioned time series and feature fusion module concatenates the spatial features and communication features extracted from the dual branches, and performs dimensionality reduction fusion through a fully connected layer FC 512 with ReLU activation;

[0033] The input is then a gated recurrent network (GRU) 512 with 512 hidden units; the GRU incorporates the hidden state from the previous time step. Iterative updates are performed, historical memory is used to make up for the incomplete information caused by local observations, and finally the confidence state is output to the Actor and Critic prediction heads to complete the end-to-end online decision-making.

[0034] Ultimately, the algorithm uses the time-series confidence state output by the GRU and a dual-head network to perform end-to-end distributed decision-making.

[0035] The Actor maps the probability distribution of discrete actions based on a fully connected layer. The Critic guides the agent in making action decisions; it evaluates the value of the current state based on the output of a single-output fully connected layer. .

[0036] Furthermore, the input to the local spatial coding branch is a three-channel local observation matrix, which are as follows:

[0037] Obstacle distribution matrix, agent position matrix, and target point relative projection matrix; the matrices are successively normalized, Convolutional layer and three consecutive residual blocks.

[0038] Furthermore, the input to the communication intent encoding branch is the state sequence of other agents within the local communication range, including position, target, and greedy action based on the prior map;

[0039] After the sequence text is embedded and positionally encoded, classification feature labels are concatenated, and finally input into the attention module for processing.

[0040] Furthermore, rewards are set for algorithm training to guide the agent in completing the navigation task. The rewards are set as follows:

[0041] Apply a value at each time step A small penalty can drive the agent toward the goal;

[0042] When the agent collides with an obstacle or other agent, a value is assigned. Collision penalties are used to enhance the obstacle avoidance capabilities of intelligent agents;

[0043] In addition, agents that wait in place for longer than a threshold are also subject to a numerical value. The penalty is used to prevent intelligent agents from passively avoiding collision risks by waiting in place for a long time.

[0044] Positive rewards are only given when the agent successfully reaches the target location.

[0045] Furthermore, a dynamic policy switcher is designed to dynamically select the optimal decision policy by evaluating the environmental state of the agent in real time. The method is as follows:

[0046] In the distributed decision-making process, the switcher selects the corresponding strategy based on the current state or local observations. The strategy switching methods are divided into the following two types of mechanisms:

[0047] Local observation sensing strategy switcher CT-PSS:

[0048] The switcher adapts to switching based on environmental density; the switcher counts the number of neighboring agents within the local field of view in real time;

[0049] If the preset threshold is exceeded If it is identified as a high-risk conflict zone, the DRL strategy is activated to resolve the conflict by leveraging its collaborative capabilities.

[0050] Conversely, if the density is low, it is determined to be a low-density region. In a low-density region, heuristic search is switched to ensure the global optimum of the path and computational efficiency.

[0051] Planning Fault Awareness Strategy Switch CT-FSS:

[0052] The strategy switcher intervenes forcibly based on anomaly detection; it continuously monitors the effectiveness of heuristic planning, and determines that the current search strategy is invalid when it detects that the agent's trajectory is cyclic or trapped in a local deadlock.

[0053] At this point, CT-FSS switches control to the DRL strategy, using its learned nonlinear cooperative experience to break the heuristic local optimization trap.

[0054] Beneficial effects: Compared with the prior art, the technical solution of the present invention has the following beneficial technical effects:

[0055] This invention focuses on developing distributed path planning strategies with highly autonomous decision-making capabilities. By systematically modeling the partially observable multi-agent path planning (MI-POMAPF) problem under global map priors, it aims to improve the system's robustness and response speed under conditions of limited observation and non-ideal communication.

[0056] This invention not only provides theoretical support for the continuous operation of large-scale clusters in complex real-world environments, but also explores feasible engineering paths for the unmanned evolution of fields such as intelligent manufacturing and smart logistics.

[0057] The adaptive strategy switching scheme proposed in this invention demonstrates extremely high system integrity and innovation in multi-agent path planning.

[0058] This invention achieves a complementary advantage between deep reinforcement learning and heuristic search. It utilizes CT-EPOM to handle real-time conflicts in high-density areas, leveraging its cooperative obstacle avoidance potential; simultaneously, it employs CT-HA* to maintain long-range navigation performance in low-density scenarios, overcoming the performance bottleneck of single planning algorithms in complex environments.

[0059] This invention establishes a dynamic policy switching mechanism that balances efficiency and robustness. By evaluating environment density through the PSS (Plan-Side System) and adaptively adjusting the decision-making mode, it solves the performance degradation problem of reinforcement learning in sparse environments. Utilizing the FSS (Fixed-Side System) to monitor the planning state, it intervenes through learned experience when the search process gets stuck in deadlock or loop failure, ensuring the continuity of path solving.

[0060] This invention enhances collaborative capabilities in partially observable environments. By introducing a communication mechanism into local perception constraints, agents can share movement intentions and state information, compensating for the interference of perception blind spots on group decision-making and enhancing the depth of cooperation among agents.

[0061] This invention possesses strong engineering applicability. The research framework closely adheres to real-world physical constraints, combining static map priors with dynamic local observations. Through multi-scenario map testing and verification, the system demonstrates stable planning performance under limited observation conditions, successfully transforming the theoretical model into a practical application scenario. Attached Figure Description

[0062] Figure 1 This diagram illustrates various local observation representations that can be used by MI-POMAPF in embodiments of the present invention.

[0063] Figure 2 This is a diagram illustrating the data generation process of the MI-POMAPF local observation matrix in an embodiment of the present invention.

[0064] Figure 3 This is a demonstration example of distributed decision-making CT-HA* based on static maps and local observation information in an embodiment of the present invention.

[0065] Figure 4 This is a flowchart of the algorithm for resolving inter-agent loops in CT-HA* in an embodiment of the present invention.

[0066] Figure 5 This is a model framework diagram of the evolutionary strategy optimization algorithm CT-EPOM based on time-series communication and memory mechanism in an embodiment of the present invention.

[0067] Figure 6 This is a diagram illustrating the reward settings for reinforcement learning in an embodiment of the present invention.

[0068] Figure 7 This is a framework diagram of the strategy switcher in an embodiment of the present invention.

[0069] Figure 8 This is a framework diagram of the local agent density and cyclic detection strategy switcher in an embodiment of the present invention.

[0070] Figure 9 These are example images of various maps used in the training of schemes in embodiments of the present invention.

[0071] Figure 10 These are example maps of various types used in the testing of solutions in embodiments of the present invention.

[0072] Figure 11 This diagram illustrates a comparison of the success rates of the present invention with other solutions.

[0073] Figure 12 This is a schematic diagram comparing the costs of the present invention with those of other solutions. Detailed Implementation

[0074] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0075] This invention presents an adaptive distributed multi-robot path planning method. First, it defines the MI-POMAPF problem. Next, it explains distributed decision-making based on static maps and local observations (CT-HA*), which combines global static maps with local dynamic observations for distributed obstacle avoidance. Simultaneously, it introduces the evolutionary policy optimization algorithm framework (CT-EPOM) based on temporal communication and memory mechanisms, along with related training strategies. Finally, it introduces two types of policy switchers: local agent density-based CT-PSS and cyclic detection CT-FSS, effectively solving the MI-POMAPF problem.

[0076] 1. MI-POMAPF.

[0077] To address the discrepancy between the global observability of the classic MAPF and the limitations of real-world communication and perception, and the decision-making difficulties caused by PO-MAPF's neglect of the global map, this invention proposes a more realistic partially observable MAPF (MI-POMAPF) problem with a global map prior. The core of MI-POMAPF lies in effectively combining dynamic and static information while maintaining fully distributed execution. First, agents primarily rely on local observations and limited communication for online decision-making, without needing to acquire the global real-time state of other agents in the system. This reduces the communication overhead of global state synchronization, ultimately improving system scalability. Second, unlike PO-MAPF, MI-POMAPF pre-shares a global static map among all agents. Agents can utilize this prior information for path calculation and direction guidance during local decision-making, compensating for information gaps caused by local perspectives. In summary, MI-POMAPF combines the global navigation advantages of classic MAPF with the distributed high scalability of PO-MAPF, making it a more realistic problem definition for multi-agent scenarios.

[0078] To adapt to different algorithm architectures and meet communication bandwidth constraints, multi-agent systems require efficient and scalable information encoding methods. This invention summarizes different methods for representing local observation information and communication information to support cooperation and decision-making in partially observable environments. Specific representation methods are as follows: Figure 1 As shown.

[0079] Vector representation: Mapping state information into a fixed-length low-dimensional vector. This method has low computational overhead and high transmission efficiency, but its expressive power is limited, making it difficult to characterize complex spatial dependencies and conflict relationships.

[0080] Token Sequence: This method encodes location, historical actions, and interaction information into discrete token sequences. While it can model temporal relationships, the sequence length increases with the number of agents, leading to additional communication and computational overhead in high-density scenarios.

[0081] Graph structure representation: Constructing a dynamic topology graph based on the communication relationships between agents. This method can express spatial associations and conflict structures, but it requires frequent topology updates in dynamic environments, resulting in relatively high computational costs.

[0082] Sparse matrix: Spatial relationships within a local observation range are represented using sparse matrices. This method requires less data, has higher computational efficiency, and is suitable for distributed conflict prediction and local cooperative control.

[0083] Semantic text: Representing high-level strategies or interaction intentions as structured symbols or natural language. This approach is helpful for expressing complex collaborative information, but the parsing and reasoning processes are costly, limiting real-time performance.

[0084] Based on the above representation methods of local observation and communication information, this invention designs a more effective local observation format according to the definition of MI-POMAPF, the specific representation of which is as follows: Figure 2 As shown. The most important form is a sparse matrix, which mainly includes three parts: the Cost-to-go matrix (heuristic cost matrix), the agent position matrix, and the endpoint matrix (agent goal matrix).

[0085] Cost-to-go matrix: Represents spatial cost information related to the greedy movement strategy in the local environment. This channel deeply integrates two types of spatial information: one is the distribution of static obstacles within the local observation range, and the other is... The first represents the location of impassable obstacles; the second represents the change in time step cost when performing greedy movement in different directions with the current agent's position as a reference. For feasible locations, the interval is used as the basis for the calculation. The integer or real value within the matrix represents the change in time step relative to the current state, where negative values ​​indicate a decrease in the remaining shortest path length after moving towards the target, and positive values ​​indicate an increase in path length due to deviation from the target. Furthermore, by encoding obstacle constraints and greedy cost changes, this matrix can provide potential-field-like directional guidance for action selection, rather than direct deterministic path guidance, while preserving local obstacle constraints.

[0086] Agent position matrix: Represents the spatial distribution of all agents within the current local observation range. This channel uses a binary occupancy grid of the same size as the observation window: grids occupied by agents are marked as 1, otherwise as 0. This encoding method can explicitly reflect the risk of occupancy conflicts and the degree of spatial congestion within the local area, providing geometric constraint information for collision detection and conflict prediction. Simultaneously, its spatial structure is consistent with other observation channels, facilitating multi-channel fusion and convolutional feature extraction, thereby enhancing the model's ability to model spatial interaction relationships.

[0087] Endpoint matrix: Represents the local relative orientation of the agent's global target. It is the same size as the local observation window, with the target position marked as 1 and other positions marked as 0. When the target is outside the observation range, it is mapped to the range boundary to ensure the effectiveness of the direction guidance signal. This target matrix, while maintaining local spatial consistency, provides target-oriented cues for action selection, facilitating fusion with heuristic cost matrices, agent position matrices, and other channels to ultimately achieve efficient path planning and conflict avoidance.

[0088] Finally, these three matrices are overlapped and used as the overall input to the decision-making algorithm based on the local observation information.

[0089] 2. A distributed decision-making algorithm based on static maps and local observation information.

[0090] In the PO-MAPF problem, to address the incomplete planning issue caused by limited local observation information, this invention introduces a static map as a global prior in the MI-POMAPF problem and fuses it with local observation information to realize the distributed decision-making algorithm. The algorithm is based on the single-agent A* algorithm for a single robot in a grid scenario that needs to move from a starting point to an end point. Within the A* search framework, online adaptive path updates are achieved by utilizing the global environment structure and local dynamic changes. Under distributed decision-making conditions, this method enables the agent to independently complete path planning and action selection, reducing the planning risks caused by local loops, path blocking, and conflicts, thereby improving the feasibility and stability of the overall path planning.

[0091] like Figure 3 As shown, all agents can acquire a global static map, thus enabling them to determine the static obstacle positions across the entire scene. Furthermore, each agent can acquire the positions of other agents within its own local observation range, which corresponds to dynamic obstacle changes within a local area.

[0092] Finally, based on the obstacle positions in the static map and the agent positions within a local area, the A* algorithm performs heuristic path planning to determine the agent's action at the current time step. Specifically, the algorithm not only sets obstacles in the static map as hard constraints but also introduces a local perception range, modeling the positions of other agents within that range as hard constraints as well. During the heuristic search process, the algorithm still follows the original algorithm's evaluation function. Provide heuristic guidance based on the time already spent. Based on this, calculate the heuristic value of the movable point after removing all hard constraints. The system selects the final and smallest point as the next point to move to. At the execution level, while each planning step generates a complete safe path to the destination, it only outputs and executes the first discrete action (such as moving or waiting) of that path at the current time step. Simultaneously, to promptly address changes in local dynamic obstacles, the system acquires the positions of other agents in real time based on local observations at each time step, thereby determining the action decision for the current time step in real time.

[0093] In local observation environments lacking global communication, agents relying on reactive obstacle avoidance are highly susceptible to periodic oscillations (i.e., livelock). In this state, the agent loops between repeated obstacle avoidance and returning to the original path, unable to negotiate priorities, leading to local deadlock and task stagnation. To address this oscillating behavior, this invention, based on the concept of loop detection and action selection, introduces a loop detection and action selection strategy. The specific algorithm process is as follows: Figure 4 As shown. The core logic of this algorithm is: before executing a decision action, the agent's next position is predicted using the reconstructed A* algorithm. and with the past A set of historical states at each time step Perform a comparison. If... That is, it is determined that the system may be trapped in a cyclical oscillation, and at this time... The probability triggers an action intervention, forcibly changing the current action to a wait, thus breaking the symmetric deadlock. This wait is executed only once, and the next action prediction will continue to repeat the above action verification and execution logic; otherwise, the predicted action is executed normally. Experiments show that under the MI-POMAPF framework, when the historical window... And the probability of intervention When the time is right, the solution efficiency of this mechanism reaches its optimal level.

[0094] 3. Optimization algorithm for grid memory strategy based on time-series communication information.

[0095] For the MI-POMAPF problem, this invention constructs an evolutionary strategy optimization model (CT-EPOM) based on the classic PPO algorithm and a time-series communication and memory mechanism. For example... Figure 5 As shown, this model employs an Actor-Critic architecture, reusing the feature extraction backbone network and operating independently only in the final output layer. The Actor outputs the action probability distribution. Critic outputs state value .

[0096] The backbone network mainly consists of a heterogeneous encoder and a temporal memory fusion module, with the following specific structure:

[0097] The local spatial encoding branch input is a three-channel local observation matrix (representing obstacle distribution, agent position, and target point relative projection). The matrix is ​​then normalized sequentially. The encoder consists of convolutional layers and three consecutive residual blocks. This part of the encoder uses the CNN architecture to deeply extract local spatial topological features to guide the agent in obstacle avoidance and target approach.

[0098] The communication intent encoding branch takes as input the state sequence of other agents within the local communication range (including position, target, and greedy actions based on the prior map). The sequence text is embedded and encoded using PosEmbedding, then concatenated with classification feature labels, and finally input into the attention module for processing. This encoder effectively extracts the cooperative movement trends and potential interaction intentions of neighboring agents.

[0099] The temporal and feature fusion module concatenates the spatial and communication features extracted from the dual branches, performs dimensionality reduction fusion through a fully connected layer with ReLU activation (FC 512), and then inputs it into a gated recurrent network (GRU 512) with 512 hidden units. The GRU combines the hidden state from the previous time step. Iterative updates are performed, historical memory is used to compensate for incomplete information caused by local observations, and finally the confidence state is output to the Actor and Critic prediction heads to complete end-to-end online decision-making.

[0100] Ultimately, the algorithm uses a dual-head network to perform end-to-end distributed decision-making based on the temporal confidence states output by the GRU. The Actor maps the probability distribution of discrete actions using a fully connected layer. The Critic guides the agent in making action decisions; while the Critic evaluates the value of the current state based on the output of a single-output fully connected layer. In multi-timestep continuous decision-making, the two work together to optimize, effectively ensuring training stability and policy convergence efficiency in complex multi-agent interaction environments.

[0101] The reward settings for training this algorithm are as follows: Figure 6 As shown, this guides the agent to complete the navigation task efficiently and safely. The specific reward settings are as follows: At each time step, a reward is applied... A small penalty is imposed to encourage the agent to move toward the goal as quickly as possible; when a collision occurs (including obstacles and other agents), a penalty is imposed. Collision penalties are applied to enhance obstacle avoidance capabilities; furthermore, to prevent agents from passively avoiding collision risks by waiting in place, penalties are also imposed on agents that wait in place. The penalty is that the agent can only receive a positive reward if it successfully reaches the target location.

[0102] 4. Strategy switching.

[0103] In the highly dynamic and locally observed environment of MI-POMAPF, a single decision-making algorithm often struggles to balance computational efficiency, generalization ability, and global optimality. Specifically, deep reinforcement learning (DRL) excels at handling high-density dynamic conflicts, but is prone to policy degradation in sparse scenarios or ultra-long-range tasks; while heuristic search can guarantee path optimality in low-conflict scenarios, its efficiency drops sharply when dealing with large-scale complex cooperative conflicts.

[0104] To overcome the limitations of a single algorithm, this invention designs a method such as... Figure 7 The dynamic policy switcher shown aims to dynamically select the optimal decision-making policy by evaluating the agent's environmental state in real time, thereby achieving complementary advantages. The main policies involved are the distributed decision-making algorithm CT-HA* based on static maps and local observation information, and the evolutionary policy optimization algorithm CT-EPOM based on temporal communication and memory mechanisms, as mentioned earlier. During the distributed decision-making process, the switcher selects the corresponding policy based on the current state or local observations, ultimately achieving effective execution.

[0105] The specific strategy switching methods are mainly divided into the following two types of mechanisms, such as Figure 8 As shown:

[0106] Local Observation Perception Strategy Switcher (CT-PSS): This switcher adaptively switches based on environmental density. The switcher continuously counts the number of neighboring agents within the local field of view; if the number exceeds a preset threshold... If a region is characterized by high conflict risk, the DRL strategy is activated, leveraging its powerful collaborative capabilities to resolve complex conflicts. Conversely, in low-density regions, heuristic search is switched to ensure the global optimum of the path and computational efficiency. This mechanism effectively overcomes the generalization bottleneck of DRL in sparse scenarios and avoids the failure of heuristic rules in dense interactions.

[0107] Fault-Aware Strategy Switcher (CT-FSS): This strategy switcher intervenes forcefully based on anomaly detection. The system continuously monitors the effectiveness of heuristic planning (such as the A* algorithm). Once it detects a loop in the agent's trajectory or that it is trapped in a local deadlock, it determines that the current search strategy has failed. At this point, CT-FSS quickly switches control to the DRL strategy, using its learned nonlinear cooperative experience to break the heuristic local optimization trap, thereby ensuring the continuity of path planning and the overall robustness of the system in complex topology environments.

[0108] To train an effective CT-EPOM decision model, this invention is based on real / game scene maps (WC3, SC1, Street) from MovingAI, a leading benchmark for multi-agent path planning, and procedurally generated maps (Random, Maze) automatically constructed using the PRIMAL2 generator. Typical examples of each map type are detailed below. Figure 9 As shown. The game-derived maps WC3 and SC1 feature numerous narrow passages, easily leading to spatial congestion and deadlock among multiple agents. The Street map recreates realistic outdoor geographical scenes. The Random and Maze maps in the training maps are primarily used to control navigation difficulty through varying obstacle densities.

[0109] To effectively verify the performance of different MAPF solvers, the comparative experiments of this invention were conducted on four typical maps of the MAPF Benchmark: Empty, Maze, Room, and Random. The specific map styles are shown below. Figure 10 As shown.

[0110] To verify the effectiveness of the proposed CT-EPOM, CT-HA*, and two policy switchers (CT-PSS and CT-FSS) in complex environments, comparative experiments were conducted with several representative MAPF solvers, including DCC, DHC, and SCRIMP. The above methods can comprehensively reflect the performance level of current mainstream algorithms. The final experimental comparison results are as follows: Figure 11 and Figure 12 As shown.

[0111] In relatively open maps like Empty and Random, all algorithms maintained high overall success rates. DCC, DHC, CT-EPOM, and SCRIMP on Random maps performed poorly, while only CT-HA*, CT-PSS, and CT-FSS consistently maintained extremely high overall success rates. This indicates that even in open environments, reinforcement learning methods based on local observations, lacking global guidance, are prone to unnecessary path exploration, causing some agents to fail to reach the goal within a finite number of steps, resulting in a low overall success rate. In contrast, CT-HA*, CT-PSS, and CT-FSS utilize a global prior map to provide clear navigation directions, effectively reducing ineffective exploration, improving path planning efficiency, and demonstrating higher success rates and robustness.

[0112] In maps like Maze and Room, which have many narrow, bottlenecked sections, the overall success rate of most algorithms drops sharply as the number of agents increases. In the Maze map, CT-EPOM struggles to solve the problem when the number of agents is small. CT-HA* maintains a high overall success rate with 32 agents, surpassing many other algorithms, but it also fails to solve the problem completely as the number of agents increases. CT-PSS and CT-FSS, through effective algorithm switching strategies, maintain significantly higher overall success rates than other algorithms with 48 and 64 agents, respectively. In the Room map, while CT-EPOM maintains a high overall solution rate with a small number of agents, it is slightly lower than other algorithms. However, when the number of agents exceeds 64, its overall success rate surpasses most algorithms. This indicates that communication information can facilitate effective cooperative movement with a large number of agents. CT-HA* maintains a high overall solution rate, but it drops sharply after the number of agents reaches 96. CT-PSS and CT-FSS, on the other hand, maintain extremely high overall solution rates. Compared to the Hybrid-TSR algorithm, the CT-PSS and CT-FSS methods have higher success rates in Empty, Random, and Room maps. However, in Maze maps, because Hybrid-TSR learns the conflict resolution strategy of a centralized algorithm, although its success rate is lower than the policy switching algorithm when the number of agents is less than 64, it is higher when the number of agents exceeds 64.

[0113] Considering the overall time-step cost, the proposed CT-PSS and CT-FSS improve the instance solution rate across different maps with a slight sacrifice in time-step cost. Compared to Hybrid-TSR, the policy switching algorithm has a lower overall time-step cost and demonstrates stronger solution and adaptability on most maps. This effectively addresses the issue of decreased policy success rate in Hybrid-TSR when facing extremely complex maps outside the training distribution.

Claims

1. An adaptive distributed multi-robot path planning method, characterized in that, Includes the following steps: Distributed decision-making CT-HA* is implemented based on static maps and local observation information, and distributed obstacle avoidance is achieved by combining global static maps with local dynamic observations. Simultaneously, the CT-EPOM framework, an evolutionary strategy optimization algorithm based on temporal communication and memory mechanisms, is constructed, and a training strategy is executed. Finally, dynamic policy switching is performed based on two types of policy switchers: agent density CT-PSS and cyclic detection CT-FSS. By evaluating the environmental state of the agent in real time, the optimal decision policy is dynamically selected.

2. The adaptive distributed multi-robot path planning method according to claim 1, characterized in that, The local observation information is designed to be represented as a sparse matrix, which includes a heuristic cost matrix, an agent position matrix, and an agent target matrix. The heuristic cost matrix, or Cost-to-go matrix, represents the spatial cost information related to the greedy movement strategy in the local environment; this channel integrates two types of spatial information: First, the distribution of static obstacles within the local observation range, with The first is to indicate the location of an impassable obstacle; the second is to show the change in time step cost when performing greedy movement in different directions with the current agent's position as a reference. For feasible locations, in intervals The integer or real value within represents the change in time step relative to the current state, where a negative value indicates that the remaining shortest path length decreases after moving toward the target, and a positive value indicates that the path length increases due to deviation from the target. Meanwhile, by encoding information on obstacle constraints and greedy cost changes, this matrix provides directional guidance for action selection similar to a potential field, rather than direct deterministic path guidance, while preserving local obstacle constraints. The agent position matrix represents the spatial distribution of all agents within the current local observation range; this channel is represented by a binary occupancy grid of the same size as the observation window: grids occupied by agents are marked as 1, otherwise as 0; at the same time, its spatial structure is consistent with other observation channels. The agent's target matrix, also known as the endpoint matrix, represents the local relative orientation of the agent's global target. It is the same size as the local observation window, with the target position marked as 1 and the other positions marked as 0. When the target is outside the observation range, it is mapped to the range boundary. This target matrix provides target-oriented cues for action selection while maintaining local spatial consistency. It is used to fuse with the heuristic cost matrix and agent position matrix channels. Finally, these three matrices are overlapped and used as the overall input to the decision-making algorithm based on the local observation information.

3. The adaptive distributed multi-robot path planning method according to claim 1, characterized in that, A static map is introduced as a global prior and fused with local observation information to realize a distributed decision-making algorithm; The algorithm is based on the A* algorithm for single agents. It utilizes the global environment structure and local dynamic changes in the A* search framework to achieve online adaptive update of the path. All agents acquire a global static map, meaning that agents acquire the static obstacle positions of the entire scene; in addition, each agent acquires the positions of other agents within its local observation range centered on itself, which is equivalent to dynamic obstacle changes within a local range. Based on the positions of obstacles in the static map and the positions of agents within a local area, the A* algorithm performs corresponding heuristic path planning to determine the agent's decision action at the current time step. Specifically, the algorithm not only sets obstacles in the static map as hard constraints, but also introduces a local perception range and models the positions of other agents within that range as hard constraints. During the heuristic search process, the algorithm still follows the evaluation function of the original algorithm. Provide heuristic guidance based on the time already spent. Based on this, calculate the heuristic value of the movable point after removing all hard constraints. And select the final and smallest point as the next point to move to; At the execution level, each time a complete safe path to the destination is generated, only the first discrete action of that path at the current time step is output and executed; Meanwhile, based on the local observation information at each time step, the positions of other agents are obtained in real time, changes in local dynamic obstacles are processed, and action decisions for the current time step are made in real time.

4. The adaptive distributed multi-robot path planning method according to claim 1, characterized in that, In a local observation environment, a loop detection and action selection strategy is introduced, and the specific process is as follows: Before executing a decision, the agent's next position is predicted using the reconstructed A* algorithm. and with the past A set of historical states at each time step Perform a comparison; like This means that the system is determined to be trapped in a cyclical oscillation, at which point... If the probability triggers an action intervention, the current action is changed to wait in place. This wait in place is executed only once, and the next action prediction will continue to repeat the above action verification and execution logic; otherwise, the predicted action is executed normally.

5. The adaptive distributed multi-robot path planning method according to claim 1, characterized in that, The evolutionary strategy optimization model CT-EPOM based on temporal communication and memory mechanisms is constructed as follows: This model adopts an Actor-Critic architecture, where the Actor and Critic share the feature extraction backbone network but are independent in the final output layer. The Actor outputs the action probability distribution. Critic outputs state value ; The backbone network consists of a heterogeneous encoder and a temporal memory fusion module, including: a local spatial coding branch, a communication intent coding branch, and a temporal and feature fusion module; The local spatial encoding branch utilizes a CNN architecture to deeply extract local spatial topological features, which are used to guide the agent to avoid obstacles and approach the target; the communication intent encoding branch is used to extract the cooperative movement trends and potential interaction intentions of neighboring agents. The aforementioned time series and feature fusion module concatenates the spatial features and communication features extracted from the dual branches, and performs dimensionality reduction fusion through a fully connected layer FC 512 with ReLU activation; The input is then a gated recurrent network (GRU) 512 with 512 hidden units; the GRU incorporates the hidden state from the previous time step. Iterative updates are performed, historical memory is used to make up for the incomplete information caused by local observations, and finally the confidence state is output to the Actor and Critic prediction heads to complete the end-to-end online decision-making. Ultimately, the algorithm uses the time-series confidence state output by the GRU and a dual-head network to perform end-to-end distributed decision-making. The Actor maps the probability distribution of discrete actions based on a fully connected layer. The Critic guides the agent in making action decisions; it evaluates the value of the current state based on the output of a single-output fully connected layer. .

6. The adaptive distributed multi-robot path planning method according to claim 5, characterized in that, The input to the local spatial coding branch is a three-channel local observation matrix, which are as follows: Obstacle distribution matrix, agent position matrix, and target point relative projection matrix; the matrices are successively normalized, Convolutional layer and three consecutive residual blocks.

7. The adaptive distributed multi-robot path planning method according to claim 5, characterized in that, The input to the communication intent encoding branch is the state sequence of other agents within the local communication range, including position, target, and greedy action based on the prior map; After the sequence text is embedded and positionally encoded, classification feature labels are concatenated, and finally input into the attention module for processing.

8. An adaptive distributed multi-robot path planning method according to any one of claims 5-7, characterized in that, Rewards are set for algorithm training to guide the agent in completing the navigation task. The rewards are set as follows: Apply a value at each time step A small penalty can drive the agent toward the goal; When the agent collides with an obstacle or other agent, a value is assigned. Collision penalties are used to enhance the obstacle avoidance capabilities of intelligent agents; In addition, agents that wait in place for longer than a threshold are also subject to a numerical value. The penalty is used to prevent intelligent agents from passively avoiding collision risks by waiting in place for a long time. Positive rewards are only given when the agent successfully reaches the target location.

9. An adaptive distributed multi-robot path planning method according to any one of claims 1-7, characterized in that, Design a dynamic policy switcher that dynamically selects the optimal decision policy by evaluating the agent's environmental state in real time. The method is as follows: In the distributed decision-making process, the switcher selects the corresponding strategy based on the current state or local observations. The strategy switching methods are divided into the following two types of mechanisms: Local observation sensing strategy switcher CT-PSS: The switcher adapts to switching based on environmental density; the switcher counts the number of neighboring agents within the local field of view in real time; If the preset threshold is exceeded If it is identified as a high-risk conflict zone, the DRL strategy is activated to resolve the conflict by leveraging its collaborative capabilities. Conversely, if the density is low, it is determined to be a low-density region. In a low-density region, heuristic search is switched to ensure the global optimum of the path and computational efficiency. Planning Fault Awareness Strategy Switch CT-FSS: The strategy switcher intervenes forcibly based on anomaly detection; it continuously monitors the effectiveness of heuristic planning, and determines that the current search strategy is invalid when it detects that the agent's trajectory is cyclic or trapped in a local deadlock. At this point, CT-FSS switches control to the DRL strategy, using its learned nonlinear cooperative experience to break the heuristic local optimization trap.

Images