Multi-agent collaborative coverage path planning method based on deep reinforcement learning

Abstract

The invention discloses a multi-agent collaborative coverage path planning method based on deep reinforcement learning. The method comprises the following steps: modeling a multi-agent collaborative coverage path planning problem as a decentralized partially observable Markov decision process (De-POMDP); for an Actor-Critic architecture, a local observation space containing a local view and an action history, a discrete action space with a mask mechanism and a multi-component award function fused with a heuristic strategy are designed; and a multi-agent near-end strategy optimization (NV-MAPPO) framework based on noise value is combined with course learning to carry out strategy optimization, and finally, the trained actor network is deployed to each agent to carry out decentralized real-time decision making. The invention aims to solve the problems of low exploration efficiency, high coordination complexity, unstable training, insufficient generalization ability and the like of the existing method in an unknown environment, and provides an efficient, stable and robust solution.

Description

TECHNICAL FIELD

[0001] The application relates to a multi-agent collaborative coverage path planning method based on deep reinforcement learning and belongs to the technical field of robot path planning. BACKGROUND

[0002] Coverage path planning is a core task in the field of mobile robots, aiming to efficiently plan paths to enable agents to traverse target areas without omission. This technology has important application value in autonomous cleaning, precision agriculture, post-disaster search and mapping, etc., and is a key link to improve the autonomy and intelligence level of robot systems. Unlike conventional path planning, coverage tasks not only require the optimality of the path, but also emphasize the completeness of traversal. Especially in unknown or dynamic environments, how to achieve fast and complete autonomous coverage puts high requirements on the environmental perception, decision planning and collaboration ability of robots, and it is of great significance to realize efficient coverage in such scenarios.

[0003] Multi-agent reinforcement learning (MARL) provides an effective way to cope with the above challenges and is gradually applied to the field of path planning. The MARL-based method allows multiple agents to learn collaborative strategies through direct trial-and-error interaction with the environment without relying on accurate prior environmental models. This method enables the agent group to adaptively cope with environmental uncertainty through decentralized decision execution and experience sharing between agents, showing superior robustness and scalability. Compared with traditional single-agent or rule-based heuristic algorithms, multi-agent reinforcement learning can better handle collaboration and conflict problems in complex environments and can better cope with dynamics and complexity in path planning. Therefore, applying multi-agent reinforcement learning methods to coverage path planning problems in unknown environments has important application value. SUMMARY

[0004] The technical problem to be solved by the application is to overcome the defects of the existing multi-agent collaborative coverage path planning method, such as low exploration efficiency in unknown environments, high coordination complexity, unstable training, and insufficient generalization ability, and to provide a multi-agent coverage path planning method that can achieve efficient, stable and robust coverage path planning.

[0005] The purpose of the application is achieved by the following technical solutions:

[0006] A multi-agent collaborative coverage path planning method based on deep reinforcement learning, comprising the following steps:

[0007] S1. Construct an environmental map model and a multi-agent model. The multi-agent model consists of N isomorphic agents. The entire multi-agent collaborative task is formalized as a decentralized partially observable Markov decision process.

[0008] S2. Design the state and observation space for the Actor-Critic architecture. The global state includes obstacles in the environment, the covered area, and the positions of all agents. Generate a local observation for each agent, which is composed of a local environmental perception representation and temporal state information.

[0009] S3. Construct a discrete action space with a masking mechanism; all agents share a discrete action space and apply an action masking mechanism to enable agents to dynamically filter invalid or dangerous actions before making decisions.

[0010] S4. Design a multi-component reward function, including basic rewards and heuristic rewards; basic rewards ensure the effectiveness and safety of the task, while heuristic rewards guide the agent to learn better exploration and collaborative strategies;

[0011] S5. Define the Actor-Critic dual-network model structure. Each agent has an independent actor network, and the agent's policy is obtained based on the agent's local observations. All agents share a critic network during the centralized training phase. The critic network generates state value estimates based on the global state and a fixed parameterized noise vector bound to an agent.

[0012] S6. Optimizing actor and critic networks based on the noise-value multi-agent proximal strategy NV-MAPPO and curriculum learning strategy;

[0013] S7, Network Model Deployment and Decentralized Execution.

[0014] The local environmental perception is represented as a local view cropped from the belief map maintained by the agent itself, and the temporal state information is the action history sequence of its past preset number of steps; the belief map includes Boolean matrix channels of obstacles known to agent i at time t, covered areas, positions of other agents, unknown areas and its own position; when multiple agents enter each other's communication range, they can establish communication and aggregate the obstacle information and covered area information in the belief map.

[0015] The discrete action space includes five actions: moving north, south, west, east, and stopping; the invalid or dangerous actions include actions that cause the agent to move outside the map boundary or collide with known obstacles or cells currently occupied by other agents, and intention conflict actions that occur when multiple agents attempt to move to the same target cell at the same time.

[0016] The basic rewards include rewards and penalties for coverage and repetition that incentivize agents to explore unknown areas, time penalties, rewards and penalties for efficiency and consistency, and collision penalties; the heuristic rewards include boundary coverage rewards, trap escape guidance rewards that help agents escape from traps, and proximity penalties between agents.

[0017] The actor network includes a convolutional neural network module and a multilayer perceptron module. The convolutional neural network module extracts spatial features from local observations, and then the multilayer perceptron module receives the features extracted by the convolutional neural network module and the non-spatial features of the action history. Finally, it outputs a probability distribution to represent the agent's policy.

[0018] The critic network comprises a convolutional neural network module and a multilayer perceptron module. The convolutional neural network module extracts global spatial features from the global state and then modulates them with a noise vector. The data is then concatenated and input into the multilayer perceptron module, ultimately outputting a state value estimate.

[0019] S6 is specifically as follows:

[0020] S61. Setting the training framework and strategy: Following the centralized training and decentralized execution paradigm, the noise value multi-agent proximal strategy specifically involves randomly generating a fixed parameterized noise vector for each agent during the centralized training phase, and concatenating the noise vector with the global state as the input to the critic network. The course learning strategy involves training by gradually increasing the obstacle density in the environment from low to high.

[0021] S62. Trajectory Data Acquisition and Processing: During the training loop, trajectory data is acquired based on the current agent's policy and environment interaction, and the advantage function is calculated using the generalized advantage estimation method.

[0022] S63. Optimize network parameters: Optimize the loss function of the actor and critic networks using gradient descent. Repeat steps S62 and S63 to form a training iteration loop until the preset training termination condition is met.

[0023] The specific details of S7 are as follows:

[0024] S71. Deploy the actor network trained in S6 to each agent;

[0025] After each agent is deployed, it enters a real-time decision execution loop, repeatedly executing the following steps S72 to S74 until the coverage task is completed;

[0026] S72. When each intelligent agent performs a task, it generates its own local observations based on real-time environmental information.

[0027] S73. Each agent inputs its local observations into the deployed actor network to generate decision-making actions;

[0028] S74. Each agent executes decision-making actions to complete the coverage task.

[0029] Compared with the prior art, the present invention has the following beneficial effects:

[0030] (1) Improved exploration efficiency and safety: By introducing an action masking mechanism, invalid or dangerous actions of the agent are avoided from the source of decision-making, which significantly reduces invalid exploration and improves sample efficiency and training safety.

[0031] (2) Enhanced the intelligence and efficiency of the strategy: By designing heuristic rewards that include strategies such as boundary exploration and escape guidance, domain expert knowledge is effectively integrated into the training process, guiding the agent to learn better exploration strategies and significantly improving the overall coverage efficiency.

[0032] (3) Improved training stability and generalization ability: Based on the NV-MAPPO framework, parameterized noise is introduced to encourage policy diversification and avoid getting trapped in local optima; and the generalization ability of the model in different complex environments is further improved through course learning. Attached Figure Description

[0033] Figure 1 This is a schematic diagram of the overall process of a multi-agent cooperative coverage path planning method based on deep reinforcement learning as described in this invention.

[0034] Figure 2 This is a schematic diagram of the environmental grid map model for a multi-agent collaborative coverage path planning task in an embodiment of the present invention.

[0035] Figure 3 This is a schematic diagram of a belief map maintained by an agent in an embodiment of the present invention.

[0036] Figure 4 This is a schematic diagram of the mechanism for information fusion among intelligent agents to update the belief map in an embodiment of the present invention.

[0037] Figure 5 This is a schematic diagram of a discrete action space with a masking mechanism in an embodiment of the present invention.

[0038] Figure 6 This is a schematic diagram of the actor network structure in an embodiment of the present invention.

[0039] Figure 7 This is a schematic diagram of the critic network structure in an embodiment of the present invention.

[0040] Figure 8 This is a comparison curve of the coverage of the method of this invention and various benchmark models during the training process.

[0041] Figure 9 This is a graph comparing the average number of steps completed during the training process of the method of this invention with that of various benchmark models.

[0042] Figure 10 This is a bar chart comparing the task success rates of the method of this invention with various benchmark models on the test set.

[0043] Figure 11 This is a bar chart comparing the average number of steps completed by the method of this invention with that of various benchmark models on the test set. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0045] This invention provides a multi-agent cooperative coverage path planning method based on deep reinforcement learning, the overall flowchart of which is shown below. Figure 1 As shown, the specific implementation steps are as follows:

[0046] S1. Constructing the Environmental Map Model and Multi-Agent Model. First, the task environment is modeled as a two-dimensional discrete grid map of size H×W, where H and W represent the number of rows and columns of the grid map, respectively. This map physically consists of two types of regions: passable regions and impassable regions (i.e., obstacles). Second, a multi-agent model is constructed. The multi-agent system consists of N homogeneous agents that possess the same perception range, communication capabilities, coverage area, and decision network structure (i.e., actor network), and share the same action space. These agents can be entities such as a swarm of drones performing collaborative reconnaissance and search and rescue missions, ground robots responsible for large-area cleaning or disinfection, or automated guided vehicles (AGVs) for warehouse management. This key setting means that all agents can adopt the same policy learning paradigm, thus simplifying the training process and facilitating system expansion. The entire multi-agent collaborative coverage task is rigorously formalized as a decentralized partially observable Markov decision process (Dec-POMDP). Specifically, this process can be defined by a tuple g. Where: N is the number of agents; S is the global state space; For joint action space, in which Let represent the action space of the Nth agent; P is the state transition function; R is the joint reward function; Ω is the joint observation space; O is the observation function; and γ is the discount factor.

[0047] S2. Designing the state and observation space for an Actor-Critic architecture:

[0048] S21. Define the global state. Global state s tComposed of three Boolean matrices of size H×W, it can be represented as follows: , representing obstacles in the environment at time t respectively. Covered areas and the location of all intelligent agents .

[0049] S22. Generate a local observation for each agent, which is composed of a local environment perception representation and temporal state information. The local environment perception representation is a local view cropped from the belief map maintained by the agent itself, and the temporal state information is its past action history sequence of a preset number of steps; the decision of each agent i is entirely based on its locally available information, i.e., the local observation. This local observation was meticulously constructed in the following way:

[0050] First, each agent independently maintains a belief map. This belief map consists of multiple H×W Boolean matrix channels of the same size as the environment map, formally represented as follows: , respectively represent the obstacles known to agent i at time t. Covered areas Other agent locations Unknown areas and one's own position .

[0051] When multiple agents enter each other's communication range, they can establish communication and merge their respective belief maps. Specifically, this information fusion process is a channel-level aggregation, and the information fusion update includes aggregating obstacle information and covered area information in the belief map. For example, it performs an element-wise maximum operation on covered area channels and obstacle channels, and updates the position information of other agents. This fusion mechanism ensures that each agent can make decisions using the most comprehensive environmental information shared by the team, which is the foundation for achieving efficient collaboration.

[0052] Then, to generate a fixed-size, agent-centric observation, clipping and padding operations are performed on the five belief map channels. Specifically, a local view of size d×d is clipped, centered on the agent's current position. An asymmetric padding strategy is used when handling map boundaries: for the obstacle channel, the area outside the map boundary is filled with 1s (treating the area outside the map boundary as an obstacle), while for the other four channels, the area outside the boundary is filled with 0s.

[0053] Finally, to provide the policy network with the contextual information needed to form time-dependent policies, a non-spatial feature, namely the agent's action history, is added to the local observation. This feature is a feature vector formed by flattening the discrete actions of the past L steps after one-hot encoding. The cropped and padded d×d multi-channel local view is concatenated with the action history vector to constitute the complete local observation of agent i at time t. .

[0054] S3. Construct a discrete action space with a masking mechanism:

[0055] S31. Define the action space: All agents share a discrete action space A. i The discrete action space includes five actions: moving north, south, west, east, and stopping.

[0056] S32. Apply the action masking mechanism to enable the agent to dynamically filter invalid or dangerous actions before making a decision.

[0057] The first layer mechanism is a static environment mask before decision-making. Before the policy network outputs the action probabilities, the system generates a binary mask vector for each agent i. This mask will pre-block all actions that would cause the agent to move outside the map boundary or collide with known obstacles or cells currently occupied by other agents.

[0058] The second layer of mechanism is dynamic conflict resolution after the decision. This mechanism is specifically designed to handle intent conflicts that arise when multiple agents attempt to move to the same target cell simultaneously.

[0059] Each agent first generates a decision action based on the valid action set filtered by the first layer of masking. Before executing these actions, a conflict detection is performed to determine if multiple agents have selected the same target cell. If a conflict is detected, the system forcibly overrides the decision actions of all agents involved in the conflict with a stop action and executes the stop action, thus avoiding a collision at the current time t. As direct feedback to the conflict at time t, at the next time t+1, the system applies an additional action mask to all agents involved in the conflict, which specifically masks the particular action that caused the conflict in the previous time step.

[0060] S4. Design a multi-component reward function;

[0061] This step effectively guides policy learning by designing a multi-component reward function that decomposes the task objective into immediate and dense numerical feedback for each action of the agent, thereby improving the learning process. The individual reward for each agent i at time t is... It consists of two main parts: basic rewards and heuristic rewards. Its complete formal expression is as follows:

[0062]

[0063] The specific definitions and technical objectives of each award are as follows:

[0064] S41, Basic Rewards:

[0065] This part of the reward is designed to establish the most basic code of conduct to ensure the effectiveness and safety of the task.

[0066] Coverage and duplicate rewards and punishments This is the core reward for the task. A significant positive reward is given when the agent moves to a previously uncovered grid cell. Conversely, if moving to an already covered grid cell, a negative penalty is applied. This component directly incentivizes agents to explore unknown regions.

[0067] Time penalty At each decision time step t, regardless of the action the agent takes, it will be subject to a fixed penalty. This component is designed to incentivize the agent to complete the coverage task in the fewest possible total steps.

[0068] Efficiency and Consistency of Rewards and Punishments : The action of agent i at the current time t Rather than the action at the previous time t − 1 When they are the same, give a positive reward. A key prerequisite is that the reward is only available to those who have already received it. This action is not active when the "stop" action is executed, to prevent the agent from remaining stationary for extended periods. Furthermore, the "stop" action is typically inefficient and should only be used for necessary obstacle avoidance or conflict resolution. Therefore, when the agent executes the "stop" action, it is subject to a negative penalty— .

[0069] Dynamic collision penalty While the safety mechanisms in S3 can avoid most collisions, to handle extreme cases of dynamic conflicts, this component sets up a strong negative penalty for two agents i and j when their planned paths overlap in the next time step. .

[0070] S42, Heuristic Rewards

[0071] This part of the reward incorporates domain expert knowledge for solving coverage problems, aiming to accelerate the learning process and guide agents to learn more advanced and intelligent exploration strategies.

[0072] Boundary Coverage Rewards When an agent moves to an uncovered grid cell adjacent to a map boundary or a known obstacle, the system awards an additional positive reward. This design precisely encourages a systematic "edge-cleaning" strategy, a pattern proven to be highly efficient in coverage problems.

[0073] Trap Escape Guidance This is a dynamic guidance mechanism to help agents escape local optima. When the system detects that an agent is "trapped" by a covered or impassable area, it uses algorithms such as breadth-first search (BFS) to find the nearest uncovered grid as an "escape point." During this process, the agent receives a large positive reward for each action it takes that brings it closer to the escape point.

[0074] Proximity penalty between agents To avoid exploration redundancy and task conflicts caused by excessive aggregation of agent teams, when the distance between any two agents is less than a preset threshold... At that time, a negative penalty is imposed on them. Additionally, this threshold... It should be set to be less than the communication distance between agents, so as to encourage agents to explore in a decentralized manner while ensuring that agents can still maintain an effective communication link for information fusion.

[0075] S5. Define the Actor-Critic dual-network model structure:

[0076] S51. Actor Network: Each agent i has its own independent actor network. The input to this network is the agent's local observations. The observation information first passes through a convolutional neural network module containing at least one convolutional layer to extract spatial features from local observations. Subsequently, the extracted features are concatenated with non-spatial features such as action history, and then fed into a multilayer perceptron module containing at least one fully connected layer. This perceptron receives the features extracted by the convolutional neural network module along with the non-spatial features such as action history. Finally, a softmax layer outputs a probability distribution in the action space, which represents the agent's policy. .

[0077] S52. Critic Network: All agents share a single critic network during centralized training. The input to this network is the global state. and a fixed-parameterized noise vector bound to agent i Global state Global spatial features are extracted using a convolutional neural network module with at least one convolutional layer, and these features are then compared with a noise vector. The data are concatenated and input into a multilayer perceptron (MLP) module, which receives the features and parameterized noise vector extracted by the convolutional neural network module. The final output is a single scalar value, representing the assessment of the current state. , Represents the global state at time t. Noise vector specific to agent i The estimated state value is equal to the output value of the critic network. ,in This indicates that the network input is derived from the global state. The noise vector unique to agent i It is pieced together, and φ represents the parameters of the critic network.

[0078] S6. Optimize strategies based on NV-MAPPO and course learning:

[0079] This step is the core process of policy learning, and its goal is to iteratively optimize the actor network parameters θ and the critic network parameters φ defined in S5 through extensive interaction with the environment. The training algorithm uses NV-MAPPO.

[0080] S61. Setting the Training Framework and Strategy: This method follows the Centralized Training Decentralized Execution (CTDE) paradigm. During the centralized training phase, a fixed parameterized noise vector is randomly generated for each agent, and this noise vector is concatenated with the global state as input to the critic network to promote diversified policy exploration;

[0081] To improve the model's generalization ability and training stability, a course-based training strategy is employed. Training is conducted by gradually increasing the obstacle density in the environment from low to high. That is, training begins in a simple environment with low obstacle density, and after the model initially converges, it is gradually transferred to more complex environments for training. The core idea of ​​the NV-MAPPO framework is to assign a fixed noise vector to each agent i during centralized training. This is used as one of the inputs for the critics' network to encourage strategic diversity.

[0082] S62. Acquiring and Processing Trajectory Data: At the beginning of each training iteration, the actor network strategy of all current agents is used. It interacts with the environment to collect a batch of trajectory data. Based on the collected data, it first utilizes a centralized network of commentators. Calculate the state value estimate for each time step. Then, the generalized advantage estimation (GAE) method is used to calculate the advantage function estimate for each agent at each time step. and value goals .

[0083] S63. Optimize network parameters: Utilize the advantage function estimates pre-calculated for each time step in S62. and value goals The loss function of the actor and critic networks is optimized using gradient descent.

[0084] To enhance training stability, the critic network employs a pruning technique for updates. The critic network's loss function L(φ) aims to minimize its value predictions. Compared with the value target calculated in S62 Mean square error between:

[0085]

[0086] in, It is a tailored value forecast that limits the difference between the new and old value forecasts to a preset range. Within this, ε is the value pruning coefficient, a hyperparameter used to limit the update magnitude. This mechanism prevents training instability caused by drastic fluctuations in value estimation.

[0087] Loss function of actor network The aim is to maximize a combined objective function that incorporates surrogate pruning and entropy regularization. Its specific form is as follows:

[0088]

[0089] in:

[0090] It's a new strategy Compared to the old strategy The probability ratio.

[0091] The function will ratio of probability Limited to Within the interval, this is used to prevent the strategy from being broken by a single update step being too large. It is the entropy of the strategy. It is its weighting coefficient. The entropy regularization term is used to encourage the policy to maintain a certain degree of randomness, thereby promoting the agent's exploratory ability.

[0092] Subsequently, steps S62 and S63 are repeated to form a training iteration loop until the preset training termination condition is met.

[0093] S7, Network Model Deployment and Decentralized Execution:

[0094] S71. Deploy the actor network trained in S6 to each agent;

[0095] After deployment, each agent enters a real-time decision execution loop, repeatedly executing the following steps S72 to S74 until the coverage task is completed:

[0096] S72. When each intelligent agent performs a task, it generates its own local observations based on real-time environmental information.

[0097] S73. Each agent inputs its local observations into the deployed actor network to generate decision-making actions;

[0098] S74. Each agent executes decision-making actions to complete the coverage task.

[0099] To further illustrate the present invention, a specific embodiment is provided herein.

[0100] This embodiment aims to solve the task of collaborative coverage path planning by N=3 homogeneous intelligent agents in an unknown grid map with H=10 and W=10.

[0101] S1. Construct an environmental map model and a multi-agent model. In this embodiment, as shown... Figure 2 As shown, the task environment is modeled as a 10×10 two-dimensional discrete grid map. The multi-agent system consists of N=3 homogeneous agents. The communication range between agents is defined as a 5×5 square area centered on the agent, covering the grid where the agent's own position is located. The entire task is formalized as a Dec-POMDP.

[0102] S2. Designing the state and observation space for an Actor-Critic architecture:

[0103] S21. Define the global state: global state s t It consists of three 10×10 Boolean matrices, which accurately represent the real-time positions of obstacles, covered areas, and three agents, respectively.

[0104] S22. Generate local observations: such as Figure 3 As shown, each agent maintains a 10×10 five-channel belief map, where the five channels represent its own position, the positions of neighboring agents, the covered area, the obstacle area, and the unknown area, respectively. When agents enter the communication range, they communicate via... Figure 4The mechanism shown performs information fusion. Each agent acquires the other's detected map environment information (white areas represent free areas, black areas represent obstacles), covered area information (blue areas), and the other agent's current position information (yellow areas). Specifically, it performs element-wise maximum value operation on the covered area channel and obstacle area channel of the two agents, and element-wise minimum value operation on the unknown area channel. At the same time, it updates the position information of neighboring agents for each agent, resulting in a fused belief map. Then, the fused belief map is cropped and filled to generate a multi-channel local view of size d = 11, i.e., 11×11. Simultaneously, the action history of the past L = 5 steps is recorded and encoded as a feature vector. The local view is concatenated with the action history vector to form the final local observation. .

[0105] S3. Construct a discrete action space with a masking mechanism:

[0106] S31. Define the action space: Define a shared discrete action space for the three agents. .

[0107] S32. Applying action masking mechanisms: such as... Figure 5 As shown, before making a decision, the system generates an action mask for each agent to shield invalid actions that could lead to collisions or boundary violations. Figure 5 In the diagram, green arrows indicate valid actions, while white arrows indicate invalid actions.

[0108] S4. Design a multi-component reward function:

[0109] Based on the specific task objectives of this embodiment, the specific values ​​of each reward component in S4 are set as follows:

[0110] Coverage Rewards ;

[0111] Duplicate Coverage Penalty - = −20;

[0112] Time penalty ;

[0113] Action consistency reward ;

[0114] Stop the punishment - = −5;

[0115] Collision penalty - = −5;

[0116] Boundary Coverage Rewards ;

[0117] Reward for those trapped and escaping ;

[0118] The trigger distance of the proximity penalty Each grid unit corresponds to a penalty value. = −10.

[0119] S5. Define the Actor-Critic dual-network model structure:

[0120] S51, The network structure of the actor network is as follows: Figure 6 As shown, the network input consists of the local observations defined in S2, specifically an 11×11 five-channel local view and a five-step action history vector. The local view first passes through a CNN module consisting of two convolutional layers. The first convolutional layer transforms the 11×11 input feature map into a 32-channel map, and the second convolutional layer further processes this 32-channel feature map, with ReLU activation applied after each layer. After processing by the CNN module, the output 32×11×11 feature map is flattened into a high-dimensional vector. This vector is concatenated with the five-step action history vector to form a combined feature vector that integrates spatial and temporal information. This combined feature vector passes through three fully connected layers, each containing 512 neurons, and then through a fully connected output layer and a Softmax activation function to generate a probability distribution vector containing five elements, corresponding to the probabilities of the five discrete actions.

[0121] S52, The network structure of the critic network is as follows: Figure 7 As shown. The input consists of the global state defined in S2 and a noise vector bound to the agent. Specifically, it is a 10×10 three-channel global state and a noise vector of length 10. The global state is first processed through a CNN module similar in structure to the actor's network. This module also consists of two convolutional layers, each followed by a ReLU activation function. The output of the CNN module is flattened into a high-dimensional vector. This vector is then passed through a fully connected layer with 512 neurons and combined with a noise vector of length 10. The features are then concatenated to form a new feature vector. This feature vector is then passed through a fully connected layer with 512 neurons and an output layer to generate a single scalar value, which is the value assessment of the current state.

[0122] S6. Optimize strategies based on NV-MAPPO and course learning:

[0123] S61. Setting the training framework and strategies

[0124] This embodiment employs the NV-MAPPO framework for centralized training and a decentralized execution paradigm, utilizing a noise-value multi-agent proximal policy optimization framework, combined with a curriculum learning strategy for training. To enhance the model's generalization ability and training stability, a three-stage curriculum learning strategy is implemented, totaling 5000 training rounds. Stage 1 (rounds 1-1500): Training is conducted on a set of 1000 simple 10×10 maps with an obstacle density of 5%. Stage 2 (rounds 1501-3000): Training is conducted on a set of 1000 medium-difficulty 10×10 maps with an obstacle density of 10%. Stage 3 (rounds 3001-5000): Training is conducted on a set of 1000 hard 10×10 maps with an obstacle density of 15%.

[0125] S62. Acquisition and Processing of Trajectory Data

[0126] In each training iteration, the system collects a batch of trajectory data. Subsequently, it collects trajectory data based on the interaction between the current policy and the environment, and calculates the estimated value of the advantage function using the generalized advantage estimation method. and value goals :

[0127] The discount factor γ was set to 0.99, and the smoothing parameter λ for generalized advantage estimation was set to 0.98.

[0128] S63, Optimize network parameters

[0129] Calculated using S62 and The actor and critic networks are updated according to the loss function formula defined in S63. The Adam optimizer is used for network optimization, with the actor network learning rate set to 0.0003, the critic network learning rate set to 0.001, the pruning factor ε set to 0.1, and the entropy regularization coefficient set to... The value is set to 0.01, and the training process is limited to a maximum of 100 steps per round.

[0130] The effectiveness of this training process is determined by Figure 8 and Figure 9 The performance curves shown demonstrate this. For comparison, several control group models were set up: ablation models with some or all of the heuristic reward removed (labeled w / o escapereward and w / o escape & boundary reward, respectively), and a standard baseline model without a parameterized noise value function (MAPPO). (See reference...) Figure 8The method described in this invention (NV-MAPPO curve in the figure) exhibits the fastest convergence speed and the highest final coverage rate during training compared to all control groups. Correspondingly, refer to... Figure 9 The NV-MAPPO model consistently requires significantly fewer average steps to complete the coverage task throughout the training process compared to other models. These two figures together demonstrate that the training method described in S6 of this invention can efficiently and stably learn high-quality cooperative strategies.

[0131] S7, Network Model Deployment and Decentralized Execution:

[0132] According to step S71, the three actor network models trained in S6 are deployed to the corresponding three agents. After deployment, each agent, when performing the coverage task, will independently and in parallel enter a real-time decision-making loop of "perception-decision-action" until the task is completed. The specific execution process of this loop strictly corresponds to steps S72 to S74: at each decision moment, each agent first generates its own local observation based on its real-time environmental information, then inputs the observation into the deployed actor network to generate a decision action, and finally executes the decision action.

[0133] The test set consists of 500 brand-new maps, completely independent of the training set. Multiple control groups were also included for comparison, including: ablation models with some or all heuristic rewards removed (labeled w / o escape reward and w / o escape & boundary reward, respectively), and a standard baseline model without a parameterized noise value function (MAPPO).

[0134] Depend on Figure 10 As shown in the bar chart comparing task success rates, the model of this invention achieved the highest success rates on the easy, medium, and hard difficulty test sets, reaching 95.8%, 93.8%, and 90.4% respectively, demonstrating that its strategy has extremely high reliability and strong generalization ability. A task is considered successful when the agent completes full coverage of the idle area within one hundred steps. Further... Figure 11 As shown in the average number of steps completed, the model of this invention completed the coverage task with the fewest average steps under all difficulty levels, namely 55.81 steps, 56.27 steps and 56.32 steps, respectively, and its coverage efficiency far exceeded that of all control groups.

[0135] In summary, through a comprehensive analysis of the training process and the final test results, this embodiment fully demonstrates that the technical solution described in this invention has significant advantages in terms of learning efficiency, task success rate, and coverage efficiency.

[0136] The contents not described in detail in this specification are common knowledge to those skilled in the art.

[0137] Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make possible changes and modifications to the technical solutions of the present invention by utilizing the methods and techniques disclosed above without departing from the spirit and scope of the present invention. Therefore, any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solutions of the present invention shall fall within the protection scope of the technical solutions of the present invention.

Claims

1. A multi-agent cooperative coverage path planning method based on deep reinforcement learning, characterized in that, Includes the following steps: S1. Construct an environmental map model and a multi-agent model. The multi-agent model consists of N isomorphic agents. The entire multi-agent collaborative task is formalized as a decentralized partially observable Markov decision process. S2. Design the state and observation space for the Actor-Critic architecture. The global state includes obstacles in the environment, the covered area, and the positions of all agents. Generate a local observation for each agent, which is composed of a local environmental perception representation and temporal state information. S3. Construct a discrete action space with a masking mechanism; all agents share a discrete action space and apply an action masking mechanism to enable agents to dynamically filter invalid or dangerous actions before making decisions. S4. Design a multi-component reward function, including basic rewards and heuristic rewards; basic rewards ensure the effectiveness and safety of the task, while heuristic rewards guide the agent to learn better exploration and collaborative strategies; S5. Define the Actor-Critic dual-network model structure. Each agent has an independent actor network, and the agent's policy is obtained based on the agent's local observations. All agents share a critic network during the centralized training phase. The critic network generates state value estimates based on the global state and a fixed parameterized noise vector bound to an agent. S6. Optimize actor and critic networks based on noise-valued multi-agent proximal strategies and curriculum learning strategies; S7, Network Model Deployment and Decentralized Execution.

2. The multi-agent cooperative coverage path planning method based on deep reinforcement learning according to claim 1, characterized in that, The local environment perception is represented as a local view cropped from the belief map maintained by the agent itself, and the temporal state information is its past action history sequence of a preset number of steps; the belief map includes Boolean matrix channels of obstacles known to agent i at time t, covered areas, positions of other agents, unknown areas and its own position. When multiple agents enter each other's communication range, they can establish communication and aggregate information on obstacles and covered areas in the belief map.

3. The multi-agent cooperative coverage path planning method based on deep reinforcement learning according to claim 2, characterized in that, The discrete action space includes five actions: moving north, south, west, east, and stopping; the invalid or dangerous actions include actions that cause the agent to move outside the map boundary or collide with known obstacles or cells currently occupied by other agents, and intention conflict actions that occur when multiple agents attempt to move to the same target cell at the same time.

4. The multi-agent cooperative coverage path planning method based on deep reinforcement learning according to claim 3, characterized in that, The basic rewards include rewards and penalties for coverage and repetition that incentivize agents to explore unknown areas, time penalties, rewards and penalties for efficiency and consistency, and collision penalties; the heuristic rewards include boundary coverage rewards, trap escape guidance rewards that help agents escape from traps, and proximity penalties between agents.

5. The multi-agent cooperative coverage path planning method based on deep reinforcement learning according to claim 4, characterized in that, The actor network includes a convolutional neural network module and a multilayer perceptron module. The convolutional neural network module extracts spatial features from local observations, and then the multilayer perceptron module receives the features extracted by the convolutional neural network module and the non-spatial features of the action history. Finally, it outputs a probability distribution to represent the agent's policy. The critic network includes a convolutional neural network module and a multilayer perceptron module. The convolutional neural network module extracts global spatial features from the global state, then concatenates them with a noise vector, and inputs them into the multilayer perceptron module to finally output a state value estimate.

6. The multi-agent cooperative coverage path planning method based on deep reinforcement learning according to claim 5, characterized in that, S6 is specifically as follows: S61. Setting the training framework and strategy: Following the centralized training and decentralized execution paradigm, the noise value multi-agent proximal strategy specifically involves randomly generating a fixed parameterized noise vector for each agent during the centralized training phase, and concatenating the noise vector with the global state as the input to the critic network. The course learning strategy involves training by gradually increasing the obstacle density in the environment from low to high. S62. Trajectory Data Acquisition and Processing: During the training loop, trajectory data is acquired based on the current agent's policy and environment interaction, and the advantage function is calculated using the generalized advantage estimation method. S63. Optimize network parameters: Optimize the loss function of the actor and critic networks using gradient descent. Repeat steps S62 and S63 to form a training iteration loop until the preset training termination condition is met.

7. The multi-agent cooperative coverage path planning method based on deep reinforcement learning according to claim 6, characterized in that, The specific details of S7 are as follows: S71. Deploy the actor network trained in S6 to each agent; After each agent is deployed, it enters a real-time decision execution loop, repeatedly executing the following steps S72 to S74 until the coverage task is completed; S72. When each intelligent agent performs a task, it generates its own local observations based on real-time environmental information. S73. Each agent inputs its local observations into the deployed actor network to generate decision-making actions; S74. Each agent executes decision-making actions to complete the coverage task.

Images