Method for planning coverage path based on multi-unmanned aerial vehicle cooperation and related equipment

Abstract

The application belongs to the technical field of coverage path planning, and discloses a coverage path planning method based on multi-unmanned aerial vehicle cooperation and related equipment; wherein the coverage path planning method based on multi-unmanned aerial vehicle cooperation comprises inputting geographic information data into a constructed cooperative coverage path planning model, outputting a plurality of decision actions, forming a coverage path through the plurality of decision actions, and the processing method in the cooperative coverage path planning model comprises constructing a global continuous time variable and an agent state lock, constructing a space-time joint observation vector based on the global continuous time variable, obtaining a dynamic action mask based on the agent state lock, and obtaining a decision action based on the space-time joint observation vector and the dynamic action mask; the application not only improves the real physical operation efficiency of multi-unmanned aerial vehicle cooperation in task execution, but also solves the redundancy and collision problems in the coverage task through an improved reward mechanism.

Description

Technical Field

[0001] This invention relates to the field of coverage path planning technology, specifically to a coverage path planning method and related equipment based on multi-UAV collaboration. Background Technology

[0002] Coverage Path Planning (CPP) is a core technology for drone swarms to perform large-scale tasks such as disaster relief, agricultural plant protection, and geographic mapping. It aims to plan a collision-free path so that mobile robots can completely cover all accessible points in the target area.

[0003] Traditional CPP methods (such as ox-plowing and spiral overlay) rely heavily on prior maps and struggle to adapt to dynamic and complex environments. In recent years, methods based on multi-agent deep reinforcement learning (MARL) have attracted widespread attention due to their powerful adaptive capabilities; however, existing solutions still face significant bottlenecks and limitations in real-world physical deployments. Existing technologies, such as Chinese invention application CN112000128A, "A Method and System for Collaborative Tasks of Unmanned Aerial Vehicle (UAV) Swarms for Disaster Relief," employ a layered architecture, utilizing Particle Swarm Optimization (PSO) for route planning at the operational level; and Chinese invention application CN117420849A, "A Learning-Based Variable-Granularity Collaborative Search and Rescue Method for Maritime UAV Swarms," ​​use the MADDPG algorithm for action decision-making. These mainstream frameworks are generally based on the assumption of synchronous execution at discrete time steps, meaning the system must wait for all agents to complete their current actions and report their status before proceeding to the next time step. In real physical environments, due to factors such as mechanical structure and wind resistance, the physical time consumed by different UAVs performing different actions (such as straight flight versus sharp turns) varies significantly (heterogeneity). The synchronization mechanism forces "faster" UAVs to hover and wait for "slower" UAVs; this "weakest link" effect results in a large amount of idle waiting time, severely reducing the efficiency of parallel operations within the swarm. Summary of the Invention

[0004] The purpose of this invention is to provide a coverage path planning method and related equipment based on multi-UAV collaboration to overcome the problems existing in the prior art. This invention not only improves the real physical operation efficiency of multi-UAV collaborative task execution, but also solves the redundancy and collision problems in coverage tasks through an improved reward mechanism.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows: In a first aspect, the present invention provides a coverage path planning method based on multi-UAV cooperation, comprising the following steps: Step 1: Obtain geographic information data of the area to be covered, including map size and obstacle distribution data; Step 2: Input geographic information data into the constructed collaborative coverage path planning model, output several decision actions, and form a coverage path through these decision actions; wherein, the processing method in the collaborative coverage path planning model includes constructing global continuous time variables and agent state locks, constructing spatiotemporal joint observation vectors based on global continuous time variables, obtaining dynamic action masks based on agent state locks, and obtaining decision actions based on spatiotemporal joint observation vectors and dynamic action masks.

[0006] In some embodiments, the method for constructing the collaborative coverage path planning model includes the following steps: Step 2.1: Obtain historical geographic information data and historical decision-making actions for the historical area to be covered; perform environmental initialization based on the historical geographic information data to obtain the initialized global raster state matrix and action-time mapping table. Step 2.2: Construct and initialize the Actor network and Critic network based on the MAPPO model to obtain the initialized Actor network parameters and initialized Critic network parameters; Step 2.3: Define and initialize global continuous time variables and agent state locks, where the agent state locks include busy deadline locks and ready flags; Step 2.4: Construct a spatiotemporal joint two-layer observation vector based on the initialization of the global raster state matrix and the initialization of global continuous time variables; Step 2.5: Obtain the dynamic action mask based on the state of the ready flag, where the state of the ready flag includes either a busy state or a ready state. Step 2.6: Use the two-layer observation tensor as the input of the Actor network and the historical decision action as the output of the Actor network to obtain the original log probability vector of the action. Apply the dynamic action mask to the original log probability vector to obtain the corrected log probability vector. Perform Softmax operation on the corrected log probability vector to obtain the probability distribution and realize the training of the Actor network. Step 2.7: Query the action-time mapping table based on the historical target decision to obtain the time elapsed. Update the busy deadline lock based on the time elapsed to obtain the updated busy deadline lock. Update the ready flag based on the updated busy deadline lock to obtain the updated ready flag. Use the updated ready flag as the target position. Step 2.8: Obtain the time set of all historical decision actions implemented in the drone cluster, select the smallest timestamp in the time set that is greater than the current global continuous time variable, update the current global continuous time variable based on the smallest timestamp, and obtain the updated global continuous time variable; Step 2.9: Obtain the updated proportion of drones with the readiness flag set to ready, and apply utilization penalty or utilization reward based on the drone proportion to obtain parallel reward; Step 2.10: Obtain the cumulative number of visits to the target location, obtain a quadratic penalty function based on the cumulative number of visits, and obtain the total reward value based on the parallel reward and penalty function; Step 2.11: Store the current two-layer observation vector, historical decision actions, total reward value, the next two-layer observation vector, dynamic action mask, and ready flag as tuples into the Rollout Buffer. If the Buffer is full, perform GAE calculation through the Critic network based on the tuples to obtain the generalized advantage value and reward target value. Step 2.12: Iteratively update the MAPPO model based on the generalized advantage value and the reward objective value until the training is completed, and obtain the collaborative coverage path planning model.

[0007] In some embodiments, the environmental initialization based on historical geographic information data to obtain an initialized global raster state matrix and an action-time mapping table specifically includes: Historical geographic information data is divided into two-dimensional raster maps. Environmental discretization modeling is performed based on the two-dimensional raster maps, and an initial global raster state matrix is ​​output. The historical geographic information data includes historical map size and historical obstacle distribution data, and the two-dimensional raster map includes the physical side length of the raster. The discrete action space of the UAV swarm is defined based on the initialized global grid state matrix. An action-time mapping table is established based on the discrete action space, the UAV's cruising speed, and the grid physical side length.

[0008] In some embodiments, obtaining the dynamic action mask based on the state of the ready flag specifically includes: When the ready flag is busy, a zero mask is generated and the drone does not perform any new actions; When the ready flag is in the ready state, the discrete action space is traversed to obtain the predicted position. The predicted position is then subject to boundary constraints, obstacle constraints, and collision constraints to obtain the dynamic action mask.

[0009] In some embodiments, defining and initializing the global continuous-time variable and the agent state lock specifically includes: Define and initialize the global continuous-time variable to obtain the initialized global continuous-time variable; Set and initialize the busy deadline lock and ready flag for each drone in the drone cluster, and obtain the initialized busy deadline lock and initialized ready flag.

[0010] In some embodiments, the construction of a spatiotemporally joint two-layer observation vector based on initializing the global raster state matrix and initializing global continuous-time variables specifically includes: The spatial observation vector of the UAV is constructed based on the initialized global grid state matrix, and the temporal state vector of the UAV is constructed based on the initialized global continuous time variables. The spatial observation vector and the temporal state vector are concatenated to obtain a spatiotemporal joint two-layer observation vector.

[0011] In some embodiments, the MAPPO model is iteratively updated based on the generalized advantage value and the reward objective value until training is complete, specifically including: Construct the PPO loss function, update the Actor network parameters based on the generalized advantage value to obtain the updated Actor network parameters, update the Critic network parameters based on the reward target value to obtain the updated Critic network parameters, and execute step 2.6 again until the training ends.

[0012] Secondly, the present invention provides a coverage path planning system based on multi-UAV collaboration, comprising: A geographic information data acquisition module is used to acquire geographic information data of the area to be covered, including map size and obstacle distribution data. The coverage path acquisition module is used to input geographic information data into the constructed collaborative coverage path planning model, output several decision actions, and form a coverage path through these decision actions. The processing methods in the collaborative coverage path planning model include constructing global continuous time variables and agent state locks, constructing spatiotemporal joint observation vectors based on global continuous time variables, obtaining dynamic action masks based on agent state locks, and obtaining decision actions based on spatiotemporal joint observation vectors and dynamic action masks.

[0013] Thirdly, the present invention provides a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described method.

[0014] Fourthly, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described method.

[0015] The above technical solution has the following advantages or beneficial effects: Firstly, this invention provides a coverage path planning method based on multi-UAV collaboration. By introducing global continuous time variables and agent state locks, a spatiotemporal joint observation vector and dynamic action masking mechanism are constructed. This not only enables efficient collaboration among multiple UAVs in complex environments but also ensures real-time sharing of spatiotemporal states among UAVs. By dynamically restricting invalid actions through agent state locks, collisions and path conflicts are effectively avoided. Furthermore, experiments show that this technology significantly reduces regional overlapping coverage and improves coverage efficiency. At the same time, the dynamic masking mechanism enhances the system's adaptability in obstacle environments. This invention provides reliable technical support for automated coverage tasks of UAV swarms in surveying, inspection, agriculture, and other fields, and has significant practical application value.

[0016] In some embodiments, this invention achieves efficient collaboration and adaptive optimization of multi-UAV systems in complex environments by constructing a collaborative coverage path planning model that includes spatiotemporal joint observation and dynamic action masking. The model utilizes an agent state lock mechanism to manage the UAV state in real time and ensures time synchronization through a global continuous time variable, effectively avoiding collisions and path conflicts. By introducing a utilization-based parallel reward and repeated access penalty function, the regional duplicate coverage rate is significantly reduced, improving coverage efficiency and resource utilization. This training method enables the model to learn the optimal collaborative strategy, providing a reliable and efficient autonomous coverage solution for multi-UAV systems in practical tasks such as surveying and inspection.

[0017] In some embodiments, the present invention converts continuous geographic space into a global grid state matrix through environmental discretization modeling, thereby realizing a structured representation of complex environments. By combining the defined discrete action space and grid physical side lengths, an accurate action-time mapping table is established, providing a unified spatiotemporal benchmark for UAV collaborative planning. This method enhances the model's adaptability to the actual environment, lays an accurate time cost foundation for subsequent path optimization, and improves the execution efficiency and predictability of the covered tasks.

[0018] In some embodiments, the present invention dynamically generates action masks through agent state locks, thereby achieving fine-grained action control; the all-zero mask in the busy state forces the UAV to wait, ensuring the continuity of task execution; the multiple constraint checks (boundaries, obstacles, collisions) in the ready state ensure the safety and feasibility of flight; this mechanism effectively prevents the execution of invalid or dangerous actions, significantly improves the stability and safety of multi-UAV collaboration, and provides an important guarantee for reliable coverage in complex environments.

[0019] In some embodiments, the present invention provides a unified spatiotemporal coordination benchmark for multi-UAV systems by establishing a global continuous time variable and an agent state lock. The global time variable ensures that all UAVs make decisions within the same time frame, avoiding asynchronous conflicts. The individual state lock (busy deadline and ready flag) enables precise management of the UAV task status, ensuring the timing of action execution is reasonable. This mechanism effectively improves the system's time consistency and coordination efficiency, laying a solid foundation for real-time coordination under complex tasks.

[0020] In some embodiments, the present invention effectively solves the problem of short-sighted decision-making caused by the lack of temporal information in traditional path planning by constructing a spatiotemporally integrated two-layer observation vector. The spatial observation vector provides environmental structure information, while the temporal state vector synchronizes task progress and UAV status. The combination of the two enables the model to make decisions based on historical and current temporal contexts, enhances the system's ability to perceive the dynamic evolution of the task, avoids action conflicts and inefficiencies caused by ignoring the time dimension, and significantly improves the temporal consistency and long-term efficiency of collaborative planning.

[0021] In some embodiments, this invention achieves stable optimization of the policy and value functions by constructing a PPO loss function and collaboratively updating the Actor-Critic network using generalized advantage value and reward target value. This method guides the policy to update towards higher rewards through the advantage function, while using the reward target value to improve the accuracy of value estimation, thus ensuring the convergence and stability of the training process. This mechanism effectively improves the learning efficiency and final performance of the model in complex collaborative tasks, providing a reliable and efficient reinforcement learning training framework for multi-UAV coverage path planning.

[0022] Secondly, this invention provides a coverage path planning system based on multi-UAV collaboration. By organically combining geographic information acquisition and collaborative planning modules, it achieves efficient collaborative coverage of multiple UAVs in complex environments. The system adopts a global time synchronization and agent state lock mechanism, combined with spatiotemporal joint observation and dynamic action masking, to ensure that UAVs significantly reduce path repetition rate while avoiding obstacles and collisions. This invention not only improves the execution efficiency and safety of coverage tasks, but also achieves collision-free and low-repetition-rate coverage.

[0023] Thirdly, the present invention provides a computer device that, through a processor executing a specific computer program, can efficiently implement the steps of the method of the present invention. When performing data processing tasks, the computer device can accurately perform numerical calculations and logical judgments, avoiding errors caused by human factors. At the same time, since the computer program has high stability and reliability, it can ensure the accuracy and consistency of the data processing results.

[0024] Fourthly, the present invention provides a computer-readable storage medium. By programming the steps of the method of the present invention into a computer program and storing it on the computer-readable storage medium, users can easily load these programs onto any compatible computer device and execute them without rewriting or converting the code, which greatly improves the convenience and flexibility of program execution. Attached Figure Description

[0025] Figure 1 This is a schematic diagram of the coverage path planning method based on multi-UAV cooperation according to Embodiment 1 of the present invention; Figure 2 This is a schematic diagram of the coverage path planning method based on multi-UAV collaboration according to Embodiment 2 of the present invention; Figure 3 This is a schematic diagram of the coverage path planning system based on multi-UAV cooperation according to Embodiment 3 of the present invention; Figure 4 This is a schematic diagram of the total reward value curve for Experiment 1 according to an embodiment of the present invention; Figure 5 This is a schematic diagram of the repeatability curve of Experiment 1 according to an embodiment of the present invention; Figure 6 This is a schematic diagram of the coverage curve for Experiment 1 according to an embodiment of the present invention; Figure 7 This is a schematic diagram of the time curve used in Experiment 1 according to an embodiment of the present invention; Figure 8 This is a schematic diagram of the total reward value curve for Experiment 2 according to an embodiment of the present invention; Figure 9 This is a schematic diagram of the repeatability curve of Experiment 2 according to an embodiment of the present invention; Figure 10 This is a schematic diagram of the coverage curve for Experiment 2 according to an embodiment of the present invention; Figure 11 This is a schematic diagram of the time curve used in Experiment 2 according to an embodiment of the present invention; Figure 12 This is a schematic diagram of the total reward value curve for Experiment 3 according to an embodiment of the present invention; Figure 13 This is a schematic diagram of the repeatability curve of Experiment 3 according to an embodiment of the present invention; Figure 14 This is a schematic diagram of the coverage curve for Experiment 3 according to an embodiment of the present invention; Figure 15 This is a schematic diagram of the time curve used in Experiment 3 according to an embodiment of the present invention; Figure 16 This is a schematic diagram of the structure of a computer device according to some embodiments of this specification. Detailed Implementation

[0026] The present invention will be further described in detail below with reference to specific embodiments. These descriptions are for explanation purposes only and are not intended to limit the scope of the invention. To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention. It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0027] This invention provides a coverage path planning method and related equipment based on multi-UAV collaboration. The method and equipment include the following steps: Step 1: Obtain geographic information data of the area to be covered, including map size and obstacle distribution data; Step 2: Input geographic information data into the constructed collaborative coverage path planning model, output several decision actions, and form a coverage path through these decision actions; wherein, the collaborative coverage path planning model includes an agent state lock, a two-layer state perception module, a dynamic action mask, and an iteratively updated MAPPO model connected in sequence, and the two-layer state perception module includes a spatial observation vector and a temporal state vector.

[0028] The core of this solution lies in reconstructing traditional modular functions into a strictly sequential logical closed loop. This process achieves efficient, collision-free, and low-repetition coverage of multiple UAVs in complex environments by constructing an event-driven mechanism on a continuous timeline, combining deep neural networks for joint encoding of spatiotemporal features, and rule-based dynamic action masks. The overall logical flow of this invention strictly follows the following steps in sequence, with the output data of preceding steps serving as the input for subsequent steps, forming a complete data flow closed loop.

[0029] Example 1: This embodiment provides a coverage path planning method based on multi-UAV cooperation, including the following steps: Step 1: Obtain geographic information data of the area to be covered, including map size and obstacle distribution data.

[0030] Specifically, the obstacle distribution data is the obstacle rate.

[0031] Step 2: Input geographic information data into the constructed collaborative coverage path planning model, output several decision actions, and form a coverage path through these decision actions; wherein, the collaborative coverage path planning model includes an agent state lock, a two-layer state perception module, a dynamic action mask, and an iteratively updated MAPPO model connected in sequence, and the two-layer state perception module includes a spatial observation vector and a temporal state vector.

[0032] In some embodiments, the method for constructing the collaborative coverage path planning model includes the following steps: Step 2.1: Obtain historical geographic information data and historical decision actions for the historical area to be covered. Perform environmental initialization based on the historical geographic information data to obtain the initialized global raster state matrix and action-time mapping table.

[0033] Specifically, the environmental initialization based on historical geographic information data to obtain an initialized global raster state matrix and an action-time mapping table includes: Historical geographic information data is divided into two-dimensional raster maps. Environmental discretization modeling is performed based on the two-dimensional raster maps, and an initial global raster state matrix is ​​output. The historical geographic information data includes historical map size and historical obstacle distribution data, and the two-dimensional raster map includes the physical side length of the raster. The discrete action space of the UAV swarm is defined based on the initialized global grid state matrix. An action-time mapping table is established based on the discrete action space, the UAV's cruising speed, and the grid physical side length.

[0034] Specifically, the historical obstacle distribution data is the historical obstacle rate.

[0035] Step 2.2: Construct and initialize the Actor network and Critic network based on the MAPPO model to obtain the initialized Actor network parameters and initialized Critic network parameters.

[0036] Step 2.3: Define and initialize global continuous time variables and agent state locks, where the agent state locks include busy deadline locks and ready flags.

[0037] Step 2.4: Construct a spatiotemporal joint two-layer observation vector based on the initialization of the global raster state matrix and the initialization of global continuous time variables.

[0038] Specifically, the construction of a spatiotemporally joint two-layer observation vector based on initializing the global raster state matrix and initializing global continuous time variables includes: The spatial observation vector of the UAV is constructed based on the initialized global grid state matrix, and the temporal state vector of the UAV is constructed based on the initialized global continuous time variables. The spatial observation vector and the temporal state vector are concatenated to obtain a spatiotemporal joint two-layer observation vector.

[0039] Specifically, the construction of the UAV's time state vector based on the initialized global grid state matrix and the construction of the UAV's time state vector based on the initialized global continuous time variables specifically include: Centered on the position of one of the drones, extract the grid state of a preset size around the drone in the initialized global grid state matrix. The grid state includes coverage, obstacle status and boundary data. A region coverage state map is generated based on the grid state, and the region coverage state map is flattened into a one-dimensional vector. Calculate the normalized distance vector of other UAVs relative to one of the UAVs, and concatenate the one-dimensional vector with the normalized distance vector to obtain the spatial observation vector; The time vector of each UAV is constructed based on the initialization of global continuous time variables, resulting in several time vectors. These time vectors are then concatenated to obtain the time state vector.

[0040] Step 2.5: Obtain the dynamic action mask based on the state of the ready flag, where the state of the ready flag includes either a busy state or a ready state.

[0041] Specifically, obtaining the dynamic action mask based on the state of the ready flag bit includes: When the ready flag is busy, a zero mask is generated and the drone does not perform any new actions; When the ready flag is in the ready state, the discrete action space is traversed to obtain the predicted position. The predicted position is then subject to boundary constraints, obstacle constraints, and collision constraints to obtain the dynamic action mask.

[0042] Specifically, the boundary constraints are Does it exceed map boundaries? Obstacle constraints: Does it fall on an obstacle grid? Collision constraints: Does it conflict with the current position of other agents? If the action... a k If all constraints are satisfied, then (Valid, indicating that the action is physically feasible and sampling is allowed); if any constraint is violated, then... (Invalid, indicating that the action violates physical constraints, such as going out of bounds, hitting a wall, or colliding with a teammate, and will be forcibly ignored by the neural network in subsequent steps).

[0043] Step 2.6: Use the two-layer observation tensor as the input to the Actor network and the historical decision actions as the output of the Actor network to obtain the original log probability vector of actions. Apply the dynamic action mask to the original log probability vector to obtain the modified log probability vector. Perform Softmax operation on the modified log probability vector to obtain the probability distribution.

[0044] Step 2.7: Query the action-time mapping table based on the historical target decision to obtain the time elapsed. Update the busy deadline lock based on the time elapsed to obtain the updated busy deadline lock. Update the ready flag based on the updated busy deadline lock to obtain the updated ready flag. Use the updated ready flag as the target position.

[0045] Step 2.8: Obtain the time set of all historical decision actions implemented in the drone cluster, select the smallest timestamp in the time set that is greater than the current global continuous time variable, update the current global continuous time variable based on the smallest timestamp, and obtain the updated global continuous time variable.

[0046] Step 2.9: Obtain the updated proportion of drones with the readiness flag set to ready, and apply utilization penalty or utilization reward based on the drone proportion to obtain parallel reward.

[0047] Step 2.10: Obtain the cumulative number of visits to the target location, obtain a quadratic penalty function based on the cumulative number of visits, and obtain the total reward value based on the parallel reward and penalty function.

[0048] Step 2.11: Store the current two-layer observation vector, historical decision actions, total reward value, the next two-layer observation vector, dynamic action mask, and ready flag as tuples into the Rollout Buffer. If the Buffer is full, perform GAE calculation through the Critic network based on the tuples to obtain the generalized advantage value and reward target value.

[0049] Step 2.12: Iteratively update the MAPPO model based on the generalized advantage value and the return target value until the training ends, and obtain the cooperative coverage path planning model.

[0050] Specifically, construct a PPO loss function, update the initialized Actor network parameters based on the generalized advantage value to obtain the updated Actor network parameters, update the initialized Critic network parameters based on the return target value to obtain the updated Critic network parameters, and then execute Step 26 until the training ends.

[0051] Embodiment 2: This embodiment provides a coverage path planning method based on multi - UAV cooperation, including the following steps: Step 1: Obtain the geographical information data of the area to be covered, where the geographical information data includes map size and obstacle distribution data.

[0052] Step 2: Input the geographical information data into the constructed cooperative coverage path planning model, output a number of decision actions, and form a coverage path through the number of decision actions; among them, the cooperative coverage path planning model includes an agent state lock, a double - layer state perception module, a dynamic action mask, and an iteratively updated MAPPO model connected in sequence, and the double - layer state perception module includes a spatial observation vector and a temporal state vector.

[0053] In some embodiments, the construction method of the cooperative coverage path planning model includes the following steps: Step 2.1: Construct a discretized grid environment and a heterogeneous kinematic model.

[0054] This step aims to establish the physical and mathematical basis for the algorithm to run, map the continuous geographical space into discrete states that can be processed by a computer, and define the physical action attributes of the UAVs.

[0055] Environmental discretization modeling: Obtain the geographical information of the area to be covered and divide it into a two - dimensional grid map G of H × W. State definition: For any grid (r, c) in the map (where 0 ≤ r < H, 0 ≤ c < W, H represents the height / number of rows of the two - dimensional grid map G, W represents the width / number of columns of the two - dimensional grid map G, r represents the abscissa of any grid, and c represents the ordinate of any grid), define the state of any grid Sgrid(r, c). At the initial moment, the Sgrid of all passable areas is set to 0 (uncovered); the obstacle area is set to 2 (non - passable); the area outside the boundary is set to - 1. Input: The boundary coordinates of the target area, obstacle distribution data; Output: Initialized global grid state matrix M global .

[0056] Action space definition and physical semantic mapping: Define drone swarm U = { u 1, u 2,..., u N Discrete action space A N represents the total number of drones. To simulate realistic flight dynamics, this scheme abandons the traditional 4-connectivity setting and adopts an 8-DOF motion space. The motion set includes: orthogonal motions (Index0... 3): Top, bottom, left, right, corresponding coordinate increments ( r, c) are respectively ( 1,0),(1,0),(0, 1), (0,1). Diagonal movements (Index4) 7): The coordinate increments for top left, top right, bottom left, and bottom right are ( 1, 1),( 1,1),(1, 1),(1,1). Wherein, r represents the coordinate increment on the horizontal axis (row direction); c represents the coordinate increment on the vertical axis (vertical direction).

[0057] Heterogeneous time consumption calculation and time lookup table generation: Based on the drone's cruising speed and grid physical side length L Calculate the physical time consumption of each action and establish the action. Time mapping table. Physical distance calculation: For orthogonal actions, physical distance... d orth = L For diagonal movements, physical distance d diag = · L ≈ 1.414 L Time-consuming model: Define the action. a k Execution time , k Indicates the number of actions. This represents the physical distance of the action; the model establishes the heterogeneity of the actions: diagonal flight takes approximately 1.414 times longer than straight flight. Output: Action time lookup table (containing lookup tables for the times of all 8 actions). This table will serve as the basis for advancing the timeline in subsequent steps.

[0058] Step 2.2, Deep Neural Network Initialization.

[0059] Based on the MAPPO architecture, the Actor (policy) network and Critic (value) network are initialized. Network structure: A multilayer perceptron (MLP) is used. The Actor network structure is Input-FC(256, ReLU)-FC(128, ReLU)-Output(8); where FC represents a fully connected layer and Output represents the output layer; the Critic network structure is Input-FC(256, ReLU)-FC(128, ReLU)-Output(1). Orthogonal initialization: To prevent gradient vanishing or exploding in the early stages of training, the network weights are orthogonally initialized.

[0060] Specifically, the gain of the Actor output layer is set to 0.01 to ensure a high entropy value (nearly uniform distribution) for the initial policy, promoting exploration and encouraging the agent to explore the environment in the early stages of training; the gain of the Critic output layer is set to 1.0. Output: Initialized Actor network parameters (initialized Actor neural network parameters). θ actor Initialize the Critic network parameters (initialize the Critic neural network parameters). critic .

[0061] Step 2.3: Initialize the global continuous timeline and agent state lock.

[0062] This step aims to establish the "clock heart" of the asynchronous system, replacing the traditional synchronous time step mechanism.

[0063] Global clock initialization: Define global continuous time variables It is initialized to 0.0. This variable will no longer increment by integers, but will instead change based on event triggers. Its value changes depending on discrete events occurring in the system (such as a drone completing an action), and is used to synchronize the passage of time in the physical world.

[0064] Agent state lock initialization: For intelligent agents u i (No. i Individual agent / drone u Assign two key time-state variables: Busy Until Lock ): Records the absolute timestamp of the agent's current action completion (the absolute point in time to which the agent is "locked," before which the agent cannot accept new instructions), initialized to 0.0. Wherein, It can also be called the remaining busy time of the agent at the current moment, which is the difference between the release time and the global clock and is not less than 0.

[0065] Ready Flag ): Indicates whether the agent can accept a new instruction. Logically defined as: if ≥ ,but = 1 (True); otherwise = 0 (False), initialized to 1.

[0066] Status Output: Outputs the initial time state set. These variables will flow in real time to subsequent observation construction steps.

[0067] Step 2.4: Construct a spatiotemporal joint two-layer observation vector.

[0068] This step utilizes the geographic information from step 21 and the temporal status from step 23 to construct a composite observation vector containing a "spatial overlay layer" and a "temporal status layer," thus solving the temporal blind spot problem of traditional methods.

[0069] Spatial feature extraction (Spatial Layer): For agents u N Construct spatial observation vectors Local perception: Extracts the grid state within an S×S (e.g., 3×3) area centered on the agent, where S represents the size of the local perception field of view, including coverage (0 / 1), obstacles (2), and boundaries (-1). Global perception (Critic-specific): Generates a Region Coverage Status Map (RCSM), a matrix of the same dimension as the map, which records the coverage progress of the entire map in real time and flattens it into a one-dimensional vector. Relative position encoding: Calculates the relative position encoding of other agents. u j Relative to this machine u i Normalized distance vector , Represent other intelligent agents u j Relative to this machine u i The normalized relative x-axis, Represent other intelligent agents u j Relative to this machine u i The normalized relative ordinate.

[0070] Temporal Layer: Using the global continuous time variables output in step 23, explicitly calculate the "Remaining Time" for each agent.

[0071] Remaining time calculation: For any agent, its remaining time ,in, This ensures the value is non-negative. Local time state: Construct its own time state vector. Teammate Status: Show all teammates Time state splicing: In the formula, This is a concatenation vector representing the time states of all teammates.

[0072] Feature fusion: The spatial observation vector is concatenated with the temporal state vector to obtain a spatiotemporally joint two-layer observation vector. , In the formula, This indicates concatenation, a two-layer observation vector. It is directly used as the input to the neural network in step 2.5.

[0073] Step 2.5: Generate a dynamic action mask based on physical constraints.

[0074] Before making a decision, a dynamic action mask must be generated based on the current physical state. (ActionMask) is used to fundamentally eliminate invalid actions. Status bit filtering: Check the output of step 23. ,like ==0 (Busy): Generates an all-zero mask (or a specific No-Op mask), forcing the agent not to perform a new action and skipping the decision-making phase directly (meaning the agent is currently in a locked period of action execution and cannot physically accept new control commands). Geometry and collision prediction: If ==1 (Ready), then traverse the discrete action space defined in step 21. A Every action in a k Calculate the predicted location In the formula, Indicates the predicted location; This represents the agent's current grid coordinates; Indicates action a k The corresponding coordinate increment vector, such as (0, 1) or ( 1, 1) Perform the following triple constraint checks: Boundary constraints: Does it exceed map boundaries? Obstacle constraints: Does it fall on an obstacle grid? Collision constraints: Does it conflict with the current position of other agents? If the action... a k If all constraints are satisfied, then (Valid, indicating that the action is physically feasible and sampling is allowed); if any constraint is violated, then... (Invalid, indicating that the action violates physical constraints, such as going out of bounds, hitting a wall, or colliding with a teammate, and will be forcibly ignored by the neural network in subsequent steps).

[0075] Step 2.6, Neural network decision-making and sampling based on mask correction.

[0076] Using the two-layer observation vector from step 2.4 and the dynamic action mask in step 2.5 The optimal action is generated through the Actor network.

[0077] Network forward propagation: transferring two-layer observation vectors Input the Actor network and output the raw (unnormalized) action log probability vector. L raw Masking Logic: Before performing Softmax normalization, the dynamic action mask is adjusted. Applied to L raw For all index k Perform the following operation: In the formula, k This represents the action index, designed to utilize the properties of the Softmax function (e ∞ (≈ 0), mathematically forcing the sampling probability of invalid actions to be reduced to zero. The mathematical principle is that in the Softmax function, as the input approaches negative infinity, its exponent term e... ∞ The probability of sampling an invalid action is mathematically guaranteed to be strictly zero, thus completely eliminating invalid exploration noise during the training process.

[0078] Action sampling: Perform a softmax operation on the corrected Logits to obtain the legal probability distribution (the legal action probability distribution after mask correction). And sample the actions from them Output: Physically feasible decision actions (target actions) .

[0079] Step 2.7, Asynchronous physical execution and discrete event-driven time advancement.

[0080] This step simulates the operation of the physical world, updates the system state, and drives the timeline forward; this is the core component for achieving the "asynchronous" characteristic. Locking and time-consuming calculation: based on the decision action... Query the time consumption table generated in step 21. Acquire time consumption (decision action) (physical execution time) Update the agent's state lock: ,in, Once it has determined when it can act again, the agent... Immediately flip to 0 (Busy), and update its position to the target position.

[0081] Step 2.8, Discrete Event Simulation (DES) logic.

[0082] The system is no longer simple In the formula, T Instead of representing time, it searches for "the next earliest event". It iterates through all agents, collecting their unlocking time set (the set of times when all agents in the cluster are expected to complete their current action). Find the one greater than the current one. T global Minimum timestamp: In the formula, Indicates the time point at which the next decision event in the system will occur. This means finding the minimum value among all unlock times greater than the current time, i.e., "the next earliest time to complete the action". Time Advancement: Update the global clock. State wake-up: In a new state T global At any moment, all is satisfied The intelligent agent is awakened Prepare to proceed to step 24 of the next round. Output: Updated global state (a completely new global state after time advances and positions are updated, serving as the basis for the next round of the loop). And the new system time.

[0083] Step 2.9, Mixed reward calculation and feedback.

[0084] To guide the agent in learning efficient, low-repetition policies, a mixed reward value is calculated. Calculate the parallel utilization reward. : Calculate the proportion of agents that are currently in the "ready" state. In the formula, This represents the total number of agents currently in the READY state. Idle penalty: If... ρ >0.5 (more than half of the drones are idle), apply a penalty (penalty value for high idle rate). = 0.3 × ( ρ 0.5), at this point, the parallel utilization bonus for This forces agents to act at off-peak times, reducing collective waiting time. High-load rewards: If ρ <0.3 (most drones are busy), give a reward (reward value for high load rate). = 0.2 × (0.3 ρ At this point, the parallel utilization bonus... for .

[0085] Step 2.10, calculate the nonlinear repetition penalty .

[0086] Get the cumulative number of visits to the target location (target grid). V ( x , y A quadratic penalty function is used: As the number of visits increases, the penalty increases rapidly and non-linearly, forming a strong "repulsive potential field" that effectively prevents agents from becoming entangled repeatedly in the same area.

[0087] Total Rewards Aggregation: Output the total reward value (also known as the mixed reward value or scalar reward signal). In the formula, This indicates a fixed bonus (+1) for covering a new grid. This represents the energy cost incurred during movement (fixed at -0.1).

[0088] Step 2.11, Experience Replay and Generalized Advantage Estimation (GAE).

[0089] The interaction data is stored in a buffer and its value is estimated. Data storage: tuples (the current two-layer observation vector) are stored in the buffer. Decision-making actions Total reward value The next time step's two-layer observation vector Dynamic action mask Ready flag Store in the Rollout Buffer. GAE Calculation: Evaluate the value of the current state using the Critic network. Calculate the Time Difference Error (TD Error). : In the formula, This represents the time difference error, which measures the "unexpected surprise" brought about by the current action. Indicates an immediate reward. The discount factor is used to balance short-term and long-term returns. Indicates the value of the current state. Represents the value of the next state. The generalized dominance value is calculated recursively: In the formula, express t The generalized advantage estimate at time 1. This represents the GAE smoothing coefficient (between 0 and 1), used to balance bias and variance. This represents the generalized dominance estimate for the next time step. Output: Generalized dominance value And the target return value. The target reward value, which is the "advantage value + state value benchmark", is used for training and will serve as the "correct answer" (Label) when the Critic network is updated.

[0090] Step 2.12, MAPPO model iterative update.

[0091] The collected data is used to update the network parameters, completing the learning loop. Actor network update: Construct the PPO loss function and utilize dynamic action masks. Ensure that policy updates only occur within the valid action space: ; ; In the formula, This represents the objective loss function of the Actor network; Represents the expectation operator; This represents the probability ratio of the new and old strategies in terms of actions; This represents the parameters of the new policy, which are optimized through gradient descent to learn a better policy. This represents the old policy parameters, used in the denominator of the probability ratio calculation, providing a stable baseline; This represents the cutoff range hyperparameter (usually set to 0.2), used to limit the step size of policy updates and prevent policy collapse; This represents the policy entropy regularization term, used to increase the randomness of action distribution and prevent the algorithm from getting trapped in local optima too early. This represents the current action and is used as output. This represents the current state and serves as input.

[0092] Critic network update: Calculating mean squared error loss In the formula, Represents the loss function of the Critic network; This indicates the current state of the Critic network. The estimated value; This represents the target return value, which serves as the "label" for training.

[0093] Closed-loop iteration: The updated network parameters are directly used in the next round of step 26. As the number of iterations increases, the agent gradually masters the optimal strategy of asynchronous collaboration and efficient coverage.

[0094] Example 3: This embodiment provides a coverage path planning system based on multi-UAV collaboration, including: A geographic information data acquisition module is used to acquire geographic information data of the area to be covered, including map size and obstacle distribution data. The coverage path acquisition module is used to input geographic information data into the constructed collaborative coverage path planning model, output several decision actions, and form a coverage path through these decision actions. The processing methods in the collaborative coverage path planning model include constructing global continuous time variables and agent state locks, constructing spatiotemporal joint observation vectors based on global continuous time variables, obtaining dynamic action masks based on agent state locks, and obtaining decision actions based on spatiotemporal joint observation vectors and dynamic action masks.

[0095] For details, see Figure 3 The overall system architecture of this invention and the logical interaction relationships between its internal modules are shown in the figure. This architecture diagram intuitively illustrates the closed-loop control process from environmental state perception, spatiotemporal feature encoding, to decision sampling masking and reward feedback. The collaborative coverage path planning model includes: The Asynchronous Environment & Kinematics Module is responsible for maintaining a global continuous clock, calculating the physical time of heterogeneous actions, and managing the "busy / ready" state locks of each agent. The Dual-Layer Perception Module is responsible for constructing a composite observation vector containing a "spatial overlay layer" and a "temporal state layer" to solve the problem of missing temporal information. Dynamic Action Masking Module: Responsible for generating a set of legal actions in real time based on geometric boundaries, obstacle information, and the agent's readiness state, and physically masking invalid decisions; Hybrid Reward Shaping module; includes a hybrid reward function that combines non-linear penalties and parallelism incentives; The Deep Reinforcement Learning Decision Module (DRL) is based on an improved MAPPO architecture and includes an orthogonally initialized Actor-Critic network and a hybrid reward shaping mechanism. It is responsible for outputting the optimal control policy.

[0096] Specifically, the asynchronous environment interaction and kinematic modeling module: To address the problem that existing technologies neglect the description of the action execution process, this invention establishes a refined kinematic model based on physical distance and cruising speed, and introduces a continuous time scheduler in a reinforcement learning environment.

[0097] Physical semantic definition of discrete action space: This invention defines a discrete action space with 8 degrees of freedom in the ConfigurableGridEnv environment. A This covers all neighborhood movement directions within the two-dimensional plane. Actions are encoded using the IntEnum enumeration class, and their physical semantics are mapped to specific grid coordinate increments (Deltas). The index, code, and corresponding coordinate transformation of each action are shown in Table 1.

[0098] Table 1. Definition of Discrete Action Space for Unmanned Aerial Vehicles

[0099] Heterogeneous action time calculation model: To simulate real physical laws, this invention abandons the simplistic assumption that all actions take a time of 1. Action execution time (duration) Based on the physical distance of the action and the cruising speed of drones Decide.

[0100] Let the physical side length of the grid cell be... L (Default value is 1.0), the drone's set cruise speed is For any selected action The time consumption calculation formula is as follows: ; Among them, physical distance Calculated based on the Euclidean norm: ; In the formula, This indicates the increment of the grid row corresponding to the action; This indicates the increment of the grid column corresponding to the action; Orthogonal actions (ID 0–3): ; Diagonal moves (ID 4–7): ; Therefore, performing a diagonal flight takes approximately 1.414 times longer than a straight flight. This difference in time cost forces reinforcement learning algorithms to make trade-offs between "path length" and "time cost." For example, while diagonal flight can cover a greater Euclidean distance in one step, its longer passive (busy) state may cause the agent to miss a critical cooperation window.

[0101] Continuous timeline and state locking mechanism: The system maintains a global continuous clock (a global continuous time variable). And a list of state locks for each agent (agent state locks), state evolution logic: First, an action is triggered if and only if the agent is in the ready state (i.e. ≥ When the policy network outputs an action, the lock is updated to receive the environment action, and the computation time is calculated. And update the agent's release time (busy deadline): In the formula, This represents the state lock update logic, which adds the current global time to the action's duration and assigns the sum to the agent's release time variable. Simultaneously, the synchronization update logic: temporarily stores the agent's logical position, marking `agent_in_motion = True`; records the target coordinates `agent_pending_targets`; where `agent_in_motion` is a boolean flag, set to True when the agent's action is locked; `agent_pending_targets` represents the targets the agent needs to cover, used to record the target coordinates for this action. Furthermore, remaining time awareness is performed, and the system calculates the remaining time for each agent in real time. As an important input to the two-layer sensing module: ; Finally, environmental clock synchronization and environmental advancement are performed. To ensure the logical continuity of the physical process up to the next nearest discrete event point (such as the moment when any agent completes its action).

[0102] Specifically, the two-layer state awareness module: To endow UAVs with comprehensive perception capabilities of the spatiotemporal environment, this invention sets up a two-layer observation model of "spatial layer + temporal layer". This model extends traditional pure spatial observation into a spatiotemporal joint vector, which serves as the input to the neural network.

[0103] The dual-layer state perception module includes a spatial perception layer, primarily responsible for extracting the geometric structure, coverage state, and spatial positional relationships of the environment. It contains the following feature components: self-position encoding, local / global terrain map, and relative positions of teammates. Self-position encoding includes the agent's current normalized coordinates (position code). The dimension is 2, and in the formula, This represents the agent's current (absolute) x-coordinate. This represents the agent's current (absolute) ordinate. The local / global terrain map includes a local view: for the Actor network, the input is a local perception map centered on the agent (3 × 3 area), where each grid cell is numerically encoded: 0 - uncovered, 1 - covered, 2 - obstacle, -1 - boundary; it also includes a global view: for the Critic network, a global coverage state graph (RCSM) is maintained. The RCSM is a matrix of the same dimension as the grid map. Initially, obstacle positions are marked as 2, and all others as 0. When any drone visits the coordinates, the corresponding value in the RCSM is updated to 1 in real time. This graph reflects the coverage progress and obstacle distribution of the entire area in real time. At input, it is flattened into a one-dimensional vector and input into the Critic network. Teammate relative positions include calculating the normalized distance vectors of all other agents relative to the local agent. The dimension is 2 × (N) 1).

[0104] The dual-layer state perception module also includes a time layer: temporal feature embedding. To enable the neural network to perceive "time," the time states of all agents are explicitly embedded in the observation vector, creating a dual-layer observation vector. The construction combines global spatial information with multi-agent temporal information, and the specific formula is as follows: ; In the formula, the spatial observation vector Obtained by flattening the Region Coverage State Map (RCSM), it contains the coverage state and obstacle information of all grids in the environment; its own temporal state vector. Includes the agent's current readiness status and remaining busy time. , Indicates whether the agent is in an idle, decision-making state; This represents the physical time required for the agent to complete its action.

[0105] Specifically, the dynamic action mask module: To prevent the policy network from outputting invalid actions when agents are busy, this invention integrates an action mask generator into the step function. A dynamic action mask is generated for each agent. (Mask vector).

[0106] like ==0 (Busy), then ;like If == 1 (ready), then iterate through each action. a k Calculate the predicted location , Indicates the predicted location; This represents the agent's current grid coordinates; Indicates action a k The corresponding coordinate increment vector, such as (0, 1) or ( 1, 1) If If it crosses the boundary or is an obstacle, then ;otherwise .

[0107] Network integration: During the action sampling phase of the PPO algorithm, the Logits output by the network (the raw, unnormalized probability values ​​output by the policy network, which, when combined with a mask, can eliminate invalid exploration noise) and... Combine (invalid position set) ∞), ensuring that the sampled actions are necessarily valid.

[0108] Specifically, the hybrid reward shaping module: A hybrid reward function that incorporates nonlinear penalties and parallelism incentives: ; In the formula, Indicates the total reward value; This indicates a fixed bonus (+1) for covering a new grid. This represents the energy cost incurred during movement (fixed at -0.1). This represents a reward for parallel utilization. This indicates a non-linear repetition penalty; Parallel utilization reward The system calculates the current readiness ratio. In the formula, This represents the total number of agents currently in the READY state. High idle penalty (when ρ > 0.5): = 0.3 × ( ρ 0.5); High load bonus (when ρ < 0.3): = 0.2 × (0.3 ρ ).

[0109] Nonlinear repetitive penalty Based on the current number of times the grid has been accessed. V ( x , y Applying a nonlinear penalty creates an increasing "repulsive potential field": .

[0110] Specifically, this invention is based on an improved MAPPO algorithm, and the algorithm training architecture (Modified MAPPO) is as follows: Network structure: Includes an Actor network and a Critic network. The Actor network consists of a fully connected layer (256, ReLU) - a fully connected layer (128, ReLU) - an output layer (Softmax), with weights initialized orthogonally (gain=0.01). The Critic network consists of an input layer (including global RCSM) - a fully connected layer (256, ReLU) - a fully connected layer (128, ReLU) - an output layer (Linear), with weights initialized orthogonally (gain=1.0). Here, gain represents the gain coefficient of orthogonal initialization of network weights; Actor is 0.01; Critic is 1.0.

[0111] Data flow: Dynamic normalization is performed using RunningMeanStd, and the advantage function is calculated using Generalized Advantage Estimation (GAE).

[0112] Loss function: The truncated surrogate loss function of PPO is adopted, combined with value loss and entropy regularization term.

[0113] Specifically, the core algorithm—the improved MAPPO—adopts the Multi-Agent Proximity Policy Optimization (MAPPO) framework and makes the following core improvements: Automatic Action Masking: Utilizes the set of legal actions provided by environmental feedback to forcibly mask invalid actions (such as colliding with obstacles or flying out of bounds) and pseudo-actions in busy states at the Actor network output layer, greatly reducing the search space.

[0114] Non-linear repeated access penalty: Define the penalty term Prepeat = 0.5 × (count 1), where count is the cumulative number of visits to that grid point. By increasing the penalty at a quadratic level, the agent is forced to explore uncovered areas.

[0115] Asynchronous efficiency reward function: Idle penalty: when the readiness rate is >50%, a penalty is imposed to reduce collective waiting; Movement reward: positive feedback is given to actions that successfully perform physical displacement to suppress stationary actions.

[0116] Specifically, the algorithm processing flow is as follows: (1) Initialization: Reset the environment, generate the initial global map, set all agent states to READY, and set the global time T = 0, where T represents the global continuous clock, used to record the absolute physical time of the system operation, and the initial value is set to 0.

[0117] (2) The asynchronous decision-making loop (Step Function) includes: Input check: The system receives joint action instructions (the set of joint action instructions output by all agents in the current decision step).

[0118] Mask filtering: Traversal ready flag If the agent is in the BUSY state (i.e., T < 0) If the action is ignored, the trajectory from the previous moment will be maintained; if the action is in the READY state, the action will be executed.

[0119] Physics calculation: Calculates time consumption based on action type. (The line is 1.0, and the diagonal is 1.414), update the lock time of this agent.

[0120] State update: Advance the global clock T, update the remaining_time (physical time required for the action to be completed) of all agents, and generate the Next State containing new spatiotemporal features.

[0121] Reward Calculation: Calculate coverage rewards, movement costs, and duplicate penalties, and return the result. , ,Done, where Done represents a Boolean flag indicating whether the current training round has ended.

[0122] (3) Model update: The PPO algorithm calculates the gradient and updates the Actor / Critic network parameters based on the collected trajectory data using a masked probability distribution.

[0123] To more clearly illustrate the overall operating logic of the algorithm described in this invention, Figure 3 The training and decision processing flow based on the improved MAPPO algorithm is demonstrated. This flow covers the complete loop from environment initialization to asynchronous interaction and model updates.

[0124] This invention proposes a decentralized asynchronous scheduling mechanism. Unlike the rigid framework based on discrete synchronous time steps in existing technologies, this invention utilizes independent state locks for each agent and heterogeneous action time consumption models to break the limitation of forced synchronization. This architecture allows UAVs to independently release and make decisions based on their own physical states (such as time consumption differences caused by wind resistance and load), achieving "pipeline-level" parallelism of the cluster in the time dimension and eliminating the "idle waiting" phenomenon of the overall system caused by single-machine latency. By constructing a composite input model containing spatial grid features and temporal state vectors, this invention addresses the deficiency in traditional coverage tasks where agents cannot perceive the progress of teammates' actions. It explicitly embeds the remaining busy time and ready flags of all teammates in the observation space, giving the neural network the ability to perceive "cooperation opportunities" and enabling it to learn advanced cooperative strategies based on time differences. Through a physical constraint-driven dynamic action masking mechanism, a hard constraint filter is designed to be generated in real time at the output of the policy network. This mechanism dynamically calculates the set of legal actions based on the current position, geometric boundaries, and busy state of the UAV, and directly performs the calculation during the forward propagation stage of the neural network. By shielding invalid actions (such as wall collisions and boundary crossings), the physical security of the output strategy is guaranteed from the underlying logic. Furthermore, by significantly reducing the effective search space, the convergence speed of reinforcement learning algorithms is greatly accelerated. A reward shaping mechanism combining "dynamic reward and punishment based on cluster readiness rate" and "nonlinear potential field repetition punishment" is proposed through a hybrid reward function based on parallelism incentive. This invention introduces an explicit reward term for parallel efficiency: when the system detects a high idle rate (i.e., low parallelism), a penalty is imposed, and otherwise a reward is given. Combined with a nonlinear repulsive potential field based on increasing access frequency, this mechanism forces the cluster to spontaneously exhibit load-balanced and low-overlap cooperative coverage behavior.

[0125] To verify the effectiveness of the invention, three sets of preliminary experiments were set up, covering different map sizes and obstacle densities.

[0126] Experiment 1: Experimental setup: Map size: 6 × 6 (36 grids); Obstacle rate: 5% (approximately 2 random obstacles); Number of agents: 2 drones; Algorithm: Asynchronous Mask MAPPO of this invention.

[0127] Analysis of experimental results: See Figure 4 The figure shows the total reward curve for Experiment 1. The total reward curve rises rapidly in the early stage of training (approximately 0-2000 Episodes), climbing from a negative value (approximately -100) to a positive value (approximately 140), and then remains stable. This indicates that the agent quickly learns to avoid obstacles and cover open ground.

[0128] See Figure 5 The figure shows a schematic diagram of the repeat ratio curve in Experiment 1. The repeat ratio curve drops sharply from an initial value of about 60% to about 15%, and the decline is smooth, demonstrating the nonlinear penalty mechanism. It effectively suppressed ineffective reciprocating motion.

[0129] See Figure 6 The figure shows the coverage ratio curve of Experiment 1. The coverage ratio curve reached 100% in a very short time (about 500 Episodes) and remained at a high level close to 100%, which proves that the RCSM mechanism successfully achieved the sharing of global coverage information and eliminated the blind spots of missing coverage.

[0130] See Figure 7 The graph shows the elapsed time curve for Experiment 1. The average number of time steps required to complete the task decreased significantly, from approximately 35 seconds to around 20 seconds. This is directly attributed to the asynchronous scheduling mechanism (…). The agent learned to work in parallel, reducing idle waiting time.

[0131] Experiment 2: Experimental setup: Map size: 8 × 8 (64 grids, state space increases exponentially); obstacle rate: 5%; number of agents: 2 drones; algorithm: the asynchronous mask MAPPO of this invention.

[0132] Analysis of experimental results: See Figure 8 The figure shows the total reward curve for Experiment 2. Despite the increased state space, the reward curve still exhibits a steady upward trend, eventually converging to a score of approximately 200. Compared to the 6 × 6 scenario, the number of episodes required for convergence has increased slightly, which is in line with expectations.

[0133] See Figure 9 The diagram shows the repeat ratio curve for Experiment 2, which illustrates the average ready (idle) percentage of agents. The curve shows an upward trend (approximately 55% - 75%), which may seem counterintuitive in the later stages. However, considering the "Elapsed Time," this indicates that in the later stages of the task, since most areas have been covered, agents are more in a state of "searching for remaining scattered points," or one agent has completed its task and is waiting for another. This suggests that the algorithm has learned to perform load balancing at the end of the task; although the idle rate increases, the total time is still decreasing.

[0134] See Figure 10 The figure shows the coverage ratio curve of Experiment 2. The coverage ratio curve reached 100% in a short time (about 1000 Episodes) and remained at a high level close to 100%, which proves that the RCSM mechanism successfully achieved the sharing of global coverage information and eliminated the blind spots of missing coverage.

[0135] See Figure 11 The figure shows the Elapsed Time curve for Experiment 2. The average number of time steps required to complete the task decreased significantly, from about 60 seconds to about 40 seconds. This is directly attributed to the asynchronous scheduling mechanism, which enabled the agent to learn to work in parallel and reduce idle waiting.

[0136] Experiment 3: Experimental setup: Map size: 6 × 6 (36 grids); Obstacle rate: 10% (number of obstacles doubles, risk of dead ends increases); Number of agents: 2 drones; Algorithm: Asynchronous mask MAPPO of this invention.

[0137] Analysis of experimental results: See Figure 12 The figure shows the total reward curve for Experiment 3. It demonstrates that the total reward can still converge stably even in environments with higher obstacle density.

[0138] See Figure 13 The diagram shows the repeat ratio curve for Experiment 3. The repeat ratio decreased from approximately 65% ​​initially to around 25%. The curve was smooth, demonstrating that the nonlinear penalty mechanism effectively suppressed ineffective reciprocating motion.

[0139] See Figure 14 The figure shows the coverage ratio curve for Experiment 3. The figure shows that it can still reach 100%, which means that the algorithm did not give up on some hard-to-reach areas due to obstacles, demonstrating good exploratory capabilities.

[0140] See Figure 15 The figure shows the Elapsed Time curve for Experiment 3. The average number of time steps required to complete the task decreased significantly, from about 35 seconds to about 23 seconds. This is directly attributed to the asynchronous scheduling mechanism, which enabled the agent to learn to work in parallel and reduce idle waiting.

[0141] The three sets of experiments described above fully demonstrate the versatility of this invention under scenarios of varying complexity. In particular, the significant decrease in "ElapsedTime" and the rapid convergence of "Coverage Ratio" directly verify the technical effectiveness of the asynchronous mechanism and two-layer information fusion in improving efficiency.

[0142] This invention achieves complete decoupling of the agent's decision-making cycle and eliminates synchronization waiting time by maintaining a global continuous time axis and independent agent state locks, and by calculating the time consumption of heterogeneous actions and searching for the minimum completion time. It constructs a composite observation tensor containing a spatial grid state (RCSM) and a time state vector, where the time state vector explicitly includes the "readiness flag" and "remaining busy time" of all agents, giving agents the ability to perceive opportunities for cooperation. Before the neural network output layer, a binary mask vector is generated based on the agent's real-time position, boundary conditions, static obstacles, and dynamic locking state, and applied to the Softmax sampling process to mathematically force the masking of physically infeasible actions. It combines a parallelism reward and punishment function based on the cluster readiness rate with a nonlinear repetition penalty function based on the quadratic of the number of visits, simultaneously incentivizing the cluster to maximize parallel operations and minimize path redundancy.

[0143] This invention, by combining an asynchronous scheduling mechanism with action masking technology, not only improves the real physical efficiency of multi-UAV collaborative task execution, but also solves the redundancy and collision problems in coverage tasks through an improved reward mechanism, and has extremely high practical application value.

[0144] See Figure 16In one embodiment of the present invention, a computer device is provided, comprising a processor and a memory. The memory stores a computer program, which includes program instructions. The processor executes the program instructions stored in the computer storage medium. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, and is suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions in the computer storage medium to realize a corresponding method flow or corresponding function. The processor described in this embodiment of the present invention can be used for the operation of a coverage path planning method based on multi-UAV collaboration.

[0145] In one embodiment of the present invention, a computer-readable storage medium is provided, specifically a computer-readable storage medium (Memory), which is a memory device in a computer device used to store programs and data. It is understood that the computer-readable storage medium here can include both the built-in storage medium in the computer device and extended storage media supported by the computer device. The computer-readable storage medium provides storage space that stores the terminal's operating system; and the storage space also stores one or more instructions suitable for loading and execution by a processor. These instructions can be one or more computer programs (including program code). It should be noted that the computer-readable storage medium here can be a high-speed RAM memory or a non-volatile memory, such as at least one disk storage device. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the coverage path planning method based on multi-UAV cooperation in the embodiment.

[0146] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0147] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0148] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0149] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0150] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A coverage path planning method based on multi-UAV collaboration, characterized in that, Includes the following steps: S1, Obtain geographic information data of the area to be covered, the geographic information data including map size and obstacle distribution data; S2, inputting geographic information data into the constructed collaborative coverage path planning model, outputting several decision actions, and forming a coverage path through these decision actions; wherein, the processing method in the collaborative coverage path planning model includes constructing global continuous time variables and agent state locks, constructing spatiotemporal joint observation vectors based on global continuous time variables, obtaining dynamic action masks based on agent state locks, and obtaining decision actions based on spatiotemporal joint observation vectors and dynamic action masks.

2. The coverage path planning method based on multi-UAV collaboration according to claim 1, characterized in that, The method for constructing the collaborative coverage path planning model includes the following steps: S2.1, Obtain historical geographic information data and historical decision-making actions of the historical area to be covered, perform environmental initialization based on the historical geographic information data, and obtain the initialized global raster state matrix and action-time mapping table; S2.2, Construct and initialize the Actor network and Critic network based on the MAPPO model to obtain the initialized Actor network parameters and initialized Critic network parameters; S2.3 Define and initialize global continuous time variables and agent state locks, where the agent state locks include busy deadline locks and ready flags; S2.4, construct a spatiotemporal joint two-layer observation vector based on initializing the global grid state matrix and initializing the global continuous time variables; S2.5, obtain the dynamic action mask based on the state of the ready flag, wherein the state of the ready flag includes busy state or ready state; S2.6, the two-layer observation tensor is used as the input of the Actor network, and the historical decision action is used as the output of the Actor network to obtain the original action log probability vector. The dynamic action mask is applied to the original log probability vector to obtain the corrected log probability vector. The corrected log probability vector is subjected to Softmax operation to obtain the probability distribution, and the Actor network is trained. S2.7, Query the action-time mapping table according to the historical target decision to get the time elapsed, update the busy deadline lock based on the time elapsed to get the updated busy deadline lock, update the ready flag based on the updated busy deadline lock to get the updated ready flag, and use the updated ready flag as the target position. S2.8, obtain the time set of all historical decision actions implemented in the drone cluster, select the smallest timestamp in the time set that is greater than the current global continuous time variable, update the current global continuous time variable based on the smallest timestamp, and obtain the updated global continuous time variable; S2.9, obtain the updated readiness flag as the proportion of drones in the ready state, apply utilization penalty or utilization reward based on the drone proportion, and obtain parallel reward; S2.10, obtain the cumulative number of visits to the target location, obtain a quadratic penalty function based on the cumulative number of visits, and obtain the total reward value based on the parallel reward and penalty function; S2.11 Store the current two-layer observation vector, historical decision actions, total reward value, next-time two-layer observation vector, dynamic action mask, and ready flag as tuples into the Rollout Buffer. If the Buffer is full, perform GAE calculation through the Critic network based on the tuples to obtain the generalized advantage value and reward target value. S2.12 Iteratively update the MAPPO model based on the generalized advantage value and the reward objective value until the training ends, and obtain the collaborative coverage path planning model.

3. The coverage path planning method based on multi-UAV collaboration according to claim 2, characterized in that, The environmental initialization based on historical geographic information data, which yields an initialized global raster state matrix and an action-time mapping table, specifically includes: Historical geographic information data is divided into two-dimensional raster maps. Environmental discretization modeling is performed based on the two-dimensional raster maps, and an initial global raster state matrix is ​​output. The historical geographic information data includes historical map size and historical obstacle distribution data, and the two-dimensional raster map includes the physical side length of the raster. The discrete action space of the UAV swarm is defined based on the initialized global grid state matrix. An action-time mapping table is established based on the discrete action space, the UAV's cruising speed, and the grid physical side length.

4. The coverage path planning method based on multi-UAV collaboration according to claim 3, characterized in that, The dynamic action mask obtained based on the state of the ready flag bit specifically includes: When the ready flag is busy, a zero mask is generated and the drone does not perform any new actions; When the ready flag is in the ready state, the discrete action space is traversed to obtain the predicted position. The predicted position is then subject to boundary constraints, obstacle constraints, and collision constraints to obtain the dynamic action mask.

5. A coverage path planning method based on multi-UAV collaboration according to claim 2, characterized in that, The definition and initialization of the global continuous-time variables and agent state lock specifically includes: Define and initialize the global continuous-time variable to obtain the initialized global continuous-time variable; Set and initialize the busy deadline lock and ready flag for each drone in the drone cluster, and obtain the initialized busy deadline lock and initialized ready flag.

6. A coverage path planning method based on multi-UAV collaboration according to claim 2, characterized in that, The construction of a spatiotemporally joint two-layer observation vector based on initializing the global raster state matrix and initializing global continuous-time variables specifically includes: The spatial observation vector of the UAV is constructed based on the initialized global grid state matrix, and the temporal state vector of the UAV is constructed based on the initialized global continuous time variables. The spatial observation vector and the temporal state vector are concatenated to obtain a spatiotemporal joint two-layer observation vector.

7. A coverage path planning method based on multi-UAV collaboration according to claim 2, characterized in that, The MAPPO model is iteratively updated based on the generalized advantage value and the reward objective value until training is complete, specifically including: Construct the PPO loss function, update the Actor network parameters based on the generalized advantage value, and obtain the updated Actor network parameters. Update the Critic network parameters based on the reward target value, and obtain the updated Critic network parameters. Execute S2.6 again until training ends.

8. A coverage path planning system based on multi-UAV collaboration, characterized in that, include: A geographic information data acquisition module is used to acquire geographic information data of the area to be covered, including map size and obstacle distribution data. The coverage path acquisition module is used to input geographic information data into the constructed collaborative coverage path planning model, output several decision actions, and form a coverage path through these decision actions. The processing methods in the collaborative coverage path planning model include constructing global continuous time variables and agent state locks, constructing spatiotemporal joint observation vectors based on global continuous time variables, obtaining dynamic action masks based on agent state locks, and obtaining decision actions based on spatiotemporal joint observation vectors and dynamic action masks.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1-7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1-7.

Images