Intelligent coverage method and system in complex environment based on Voronoi diagram and multi-agent reinforcement learning, and storage medium

Abstract

The invention discloses an intelligent coverage method and system in a complex environment based on a Voronoi graph and multi-agent reinforcement learning, and a storage medium, and solves the problems of low coverage efficiency and the like easily caused in the prior art. The method comprises the steps of performing grid division on a to-be-covered area based on a coverage radius of an intelligent agent, and constructing a global coverage matrix; randomly initializing the positions of a plurality of agents in the grid region, constructing a corresponding Voronoi diagram, and dynamically updating the Voronoi diagram according to the position change of the agents in the execution process of the coverage task; a forbidden zone matrix is introduced and marked based on the to-be-covered area and the global coverage matrix, and a target point matrix and a Voronoi graph before or after dynamic updating are constructed; and defining a global state space, an action space and a reward function required by multi-agent reinforcement learning to construct a global network, and training to obtain a global Q value of target point navigation and repeated coverage in the irregular region for coverage performance evaluation. The method is used for regional monitoring of the multi-agent system in the unstructured environment.

Description

Technical Field

[0001] A method, system, and storage medium for intelligent coverage in complex environments based on Voronoi diagrams and multi-agent reinforcement learning are proposed. This method is used for multi-agent systems in area monitoring, environmental perception, search and rescue, and inspection tasks in unstructured environments. It involves multi-agent cooperative control, spatial partitioning, autonomous coverage, and policy learning, and belongs to the technical field of artificial intelligence and robot autonomous control. Background Technology

[0002] Coverage control is a crucial technique for achieving perception and information gathering in traditional tasks such as environmental monitoring, area search, map building, and target tracking. With the development of robotics and artificial intelligence, the use of multi-agent systems to achieve efficient coverage of complex environments has led to widespread attention in recent years for multi-agent reinforcement learning (MARL) methods. These methods learn optimal behavioral strategies from the interactions between agents and their environment, achieving cooperative control under unsupervised or weakly supervised conditions. Compared to traditional heuristic or model-based coverage methods, MARL exhibits stronger robustness and adaptability in dynamic, partially observable, and complexly constrained scenarios.

[0003] Existing MARL algorithms, such as Independent Q-Learning (IQL), Value Decomposition Networks (VDN), and QMIX, have been applied to research on multi-agent coverage tasks and have achieved initial success. These methods realize cooperative policy learning among agents through a centralized training and distributed execution framework (CTDE), effectively mitigating problems such as policy instability and environmental non-stationarity.

[0004] Although existing methods have applied techniques such as Voronoi diagrams and multi-agent reinforcement learning to multi-agent region coverage tasks and have achieved task partitioning and collaborative decision-making to some extent, they still face the following key challenges in complex, dynamic, or constrained environments:

[0005] 1. In complex or irregular environments, intelligent agents need to consider multiple factors such as obstacle avoidance, repeated coverage, coverage redundancy and resource constraints at the same time. Existing methods are difficult to effectively model such task requirements, which can easily lead to problems such as low coverage efficiency, unreasonable path planning, weak cooperation ability, poor algorithm generalization and serious waste of resources.

[0006] 2. Existing Q-network structures are relatively simple and lack the ability to model local spatial structures, which limits the precision and generalization of agent policies;

[0007] 3. When faced with practical constraints such as target point allocation and action feasibility limitations, the agent is prone to policy collapse or unstable training.

[0008] 4. Traditional state representation and reward design methods lack consideration for fine-grained factors such as "repeated coverage penalty" and "goal point arrival" in coverage tasks, making it difficult to support multi-objective collaborative behavior learning driven by tasks;

[0009] 5. Existing Voronoi diagrams cannot flexibly integrate the needs of complex tasks in the scenario, which limits their adaptability in dynamic scenarios and affects the quality of agent behavior.

[0010] Therefore, there is an urgent need to propose a novel approach that combines dynamic region partitioning, autonomous collaborative decision-making, and complex task modeling capabilities to address the aforementioned technical challenges and improve the region coverage efficiency and stability of multi-agent systems in complex environments. Summary of the Invention

[0011] To address the problems mentioned above, the present invention aims to provide an intelligent coverage method, system, and storage medium for complex environments based on Voronoi diagrams and multi-agent reinforcement learning. This solves the problem that existing technologies struggle to effectively model such task requirements in complex or irregular environments where agents need to simultaneously consider multiple factors such as obstacle avoidance, redundant coverage, coverage redundancy, and resource constraints. This often leads to problems such as low coverage efficiency, unreasonable path planning, weak collaborative capabilities, poor algorithm generalization, and severe resource waste.

[0012] To achieve the above objectives, the present invention adopts the following technical solution:

[0013] A method for intelligent overlay in complex environments based on Voronoi diagrams and multi-agent reinforcement learning includes the following steps:

[0014] Step S1: Divide the area to be covered in the actual scene into grids based on the coverage radius of the intelligent agent, and construct a global coverage matrix with the same size as the gridded area;

[0015] Step S2: Randomly initialize the positions of multiple agents in the grid area divided into the area to be covered in the actual scene, construct the corresponding Voronoi diagram based on the current position of the agent, and dynamically update the Voronoi diagram according to the position change of the agent during the execution of the coverage task.

[0016] Step S3: Based on the actual scene's area to be covered and the global coverage matrix, introduce a no-go zone matrix to penalize repeated coverage of the target area, mark the relevant areas in the global coverage matrix, and at the same time, select the target point matrix for priority coverage of the target point to be reached.

[0017] Step S4: Based on the global coverage matrix obtained in S3 and the Voronoi diagram before or after dynamic update, define the global state space, action space and reward function required for multi-agent reinforcement learning;

[0018] Step S5: Based on the reinforcement learning elements constructed in S4, construct a global network and train it;

[0019] Step S6: Based on the trained global network, obtain the target point navigation and global Q value of repeated coverage in the irregular area to evaluate the coverage performance.

[0020] Furthermore, the specific steps in step S1 are as follows:

[0021] Step S1.1: Based on any irregular area to be covered in the actual scene, place it into a space of size [size missing]. Within the rectangle, and based on the coverage radius of the agent. Mesh the rectangular area as Each agent covers a small rectangular patch, thus creating a gridded region. An agent reaching the center of any small rectangular patch within the gridded region is considered to have completed coverage of that region. The center of the small rectangular patch matrix serves as the agent's navigation point. and It is obtained from the following formula:

[0022]

[0023] Step S1.2: Construct a global coverage information map based on the grid region composed of small rectangular blocks. This yields the global coverage matrix. Represents the first in the global coverage matrix line, number The elements of the column represent the coverage status of each navigation point in the actual coverage area, and satisfy the following conditions: ,in, Indicates coordinates as The grid has been covered or the area is a restricted zone. Indicates coordinates as The grid needs to be overlaid again. Indicates coordinates as The grid needs to be covered twice. Indicates coordinates as The grid requires three more coverage operations. When an agent covers a certain area, the corresponding... The value is updated based on the number of times it has been covered.

[0024] Furthermore, the specific steps in step S2 are as follows:

[0025] Step S2.1: Randomly initialize the positions (i.e., two-dimensional coordinates) of multiple agents in the grid area divided into the area to be covered in the actual scene, and design the cost matrix. This is used to represent the movement cost between the agent and various navigation points in the gridded region, specifically expressed as:

[0026]

[0027] in, Represents Euclidean distance. Indicates the first Two-dimensional coordinates of an agent, Represents the first in the gridded region Two-dimensional coordinates of a navigation point Cost matrix Elements in;

[0028] Based on this cost matrix, the Hungarian algorithm is used to search for the minimum cost matching scheme in the full permutation space, that is, to assign a navigation point with the shortest distance to each agent.

[0029] Step S2.2: Based on the Voronoi diagram algorithm and the navigation points of each agent in the full permutation space, perform region segmentation, and assign navigation points within the Voronoi diagram region to the navigation points of each agent in the full permutation space, thus constructing the corresponding Voronoi diagram, which is the first... The formula for assigning navigation points to an agent within a Voronoi diagram region is:

[0030]

[0031] in, Represented as the first Each agent assigns its navigation point within the Voronoi diagram region to navigation points within the full permutation space. Indicates the first Navigation points for an intelligent agent within the entire permutation space. Represented as in the permutation space A finite set of navigation points in the middle, "For any" is a general quantifier used to indicate that a condition holds true for the entire set. Represented as the permutation space Any navigation point within the area express and The Euclidean distance;

[0032] Step S2.3: During the execution of the coverage task, the Voronoi diagram is dynamically updated every fixed time step according to the position changes of each agent, following the method in step S2.2.

[0033] Furthermore, the specific steps of step S3 are as follows:

[0034] Step S3.1: Considering the irregularity of the area to be covered in the actual scenario, design a no-go zone matrix. Used to represent The rectangle indicates whether the location is within the area to be covered. Represents the first in the forbidden zone matrix line, number The elements of the column, and satisfying ,in, Represented as the first line, number The navigation point of the column is within the area to be covered. Represented as the first line, number The navigation point of the column is outside the area to be covered, which is a restricted area, and the value of the restricted area is marked as 1 in the global coverage matrix, indicating that it does not need to be covered.

[0035] Step S3.2: Design the target point matrix This is used to represent the position of the navigation point that an agent in a gridded area should be at when it finishes performing a coverage task. Represents the first point in the target point matrix line, number The elements of the column, and satisfying ,in, This indicates that the waypoint is not the final destination the agent intends to reach. This indicates that the navigation point is the final destination that the agent must reach.

[0036] Furthermore, the specific steps in step S4 are as follows:

[0037] Step S4.1: Construct an intelligent agent based on the global coverage matrix after the restricted area marking. Global state at time step The global state of the agent is:

[0038]

[0039] in, Represents the global coverage matrix exist The value at time;

[0040] Step S4.2: Design the agent's action space based on the global coverage matrix after the restricted area marking. The agent's actions are manifested as the agent's selection of surrounding navigation points. At each moment, the agent can only choose to stay at the current position or one of the 8 neighboring navigation points adjacent to the current position. This represents the 9 navigation points chosen by the agent, the first... The action space of an agent is defined as:

[0041]

[0042] Step S4.3: Based on the forbidden zone matrix Target point matrix Intelligent agents Global state at time 1, the first The action space of each agent, the Voronoi diagram before or after dynamic updates, and the requirements of the task in the real-world scenario are used to design the reward function for reinforcement learning. The formula is as follows:

[0043]

[0044]

[0045]

[0046] in, The AND operator is used to connect multiple logical conditions. A compound condition is true if and only if all the connected conditions are true. To control the weighting parameters of directional rewards, Represented as an intelligent agent In global state Take action below Instant rewards received Represented as an intelligent agent Current state and local state Distance from the target point Represented as an intelligent agent The coordinates of the matched target point This is the distance penalty coefficient. The threshold for reaching the target point, Indicates based on intelligent agent Current state and local state The goal point navigation reward obtained from the goal point Representing the current time The global reward is the sum of the immediate rewards for all agents and the reward for navigation to the target point. Let be the set of all actions taken by agents at time t. This represents the total number of intelligent agents.

[0047] Furthermore, the global network in step S5 includes a global state. Forbidden Zone Matrix and target point matrix The super network is processed to obtain weight parameters and offset parameters; the individual Q network obtains the local Q value of each agent based on the global state and action space of each agent; and the hybrid network transforms the local Q value into a global vector based on the output of the super network and the individual Q network.

[0048] The hypernetwork includes sequentially connected global states. Forbidden Zone Matrix and target point matrix The first CNN network, the second CNN network, the third CNN network and the fourth CNN network are processed, and the output of the fourth CNN network is processed to obtain the first weight parameter and the second weight parameter of the first multilayer perceptron MLP and the third multilayer perceptron MLP, and the output of the fourth CNN network is processed to obtain the first bias parameter and the second bias parameter of the second multilayer perceptron MLP and the fourth multilayer perceptron MLP.

[0049] The individual Q-network includes a fifth CNN network that extracts local spatial features of each agent based on their global state and action space. The local spatial features extracted by the fifth CNN network are used to estimate the state-action advantage value of a fifth multilayer perceptron (MLP) and to estimate the state value of a sixth multilayer perceptron (MLP). The outputs of the fifth and sixth MLPs are then fused to obtain a fusion layer for the local Q-value. The formula for the fusion layer is as follows:

[0050]

[0051] in, The state-action advantage value is the output of the fifth multilayer perceptron (MLP). This represents the state value output by the sixth-layer perceptron (MLP). The local Q-value obtained after fusion. This represents the size of the action space of the i-th agent. This indicates the global state of the i-th agent. Sum the advantage values ​​of all actions. This represents any action in the action space of the i-th agent;

[0052] The hybrid network adopts the additivity hybrid structure of QMIX, which first concatenates the local Q values ​​of each agent into a local Q vector. Then, a two-layer mapping network is used to generate the global Q-value: the first layer of the mapping network uses the first weight parameter and the first bias parameter dynamically generated by the supernetwork to monotonically weight the local Q-value and obtain intermediate features through non-linear activation; the second layer uses the second weight parameter and the second bias parameter of the mapping network to map the intermediate features to the global Q-value. In this context, all weight parameters and bias parameters are determined by the hypernetwork based on the global state. Forbidden Zone Matrix and target point matrix Dynamic generation is used to ensure centralized training and monotonicity constraints. The specific formula is as follows:

[0053]

[0054]

[0055]

[0056]

[0057]

[0058] in, This represents the global Q-value, which is an estimate of the maximum expected reward in the current state. It is an exponential linear unit activation function, and the final output is... For the global state-action value function used for optimization, Represents the local Q-vector. Indicates the first The local Q-value of an agent and Let represent the first weight parameter and the second weight parameter, respectively. The first weight parameter controls the importance of the local Q-values ​​of each agent in the global mixture, while the second weight parameter compresses the hidden features into a global Q-value. and This represents the first bias parameter and the second bias parameter. The first bias parameter is a dynamic bias of the hidden features, which is a nonlinear combination that adjusts the local Q-values. The second bias parameter is the global Q-value. The final bias, that is, the global correction of the global Q-value for all agents, , , and These represent the first multilayer perceptron (MLP), the third multilayer perceptron (MLP), the second multilayer perceptron (MLP), and the fourth multilayer perceptron (MLP), respectively.

[0059] A smart overlay system for complex environments based on Voronoi diagrams and multi-agent reinforcement learning includes a memory processor and a computer program stored in the memory. The processor executes the computer program to implement the steps of a smart overlay method for complex environments based on Voronoi diagrams and multi-agent reinforcement learning.

[0060] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of an intelligent covering method for complex environments based on Voronoi diagrams and multi-agent reinforcement learning.

[0061] Compared with the prior art, the beneficial effects of this invention are as follows:

[0062] This invention was compared with existing solutions (fixed region allocation strategy, MARL method based on a single reward function) in various simulation environments. It achieved approximately 15% performance improvement in key indicators such as coverage efficiency, repetition rate reduction, path optimization, collaborative ability, algorithm generalization, and resource utilization. Specifically, the improvements are as follows:

[0063] First, in terms of coverage efficiency, this invention introduces a spatial dynamic partitioning method based on Voronoi diagrams, which enables autonomous allocation of areas and division of responsibilities among multiple agents in coverage tasks. This avoids redundant access to the same area by multiple agents and effectively improves the spatiotemporal distribution balance of the overall coverage. At the same time, combined with a target point-based guidance mechanism and an illegal action blocking strategy, agents can complete coverage tasks more efficiently while avoiding environmental constraints. This solves the problem that existing Voronoi diagrams cannot flexibly integrate the needs of complex tasks in coverage scenarios, which limits their adaptability in dynamic scenarios and affects the quality of agent behavior.

[0064] Second, the global state, action space, and reward function constructed in this invention fully consider various constraints in the environment (such as restricted areas, repeated coverage, and goal achievement), enabling the trained global network to possess strong adaptability and robustness. Specifically, by introducing a coverage matrix and Voronoi region constraints into the reward function, repeated coverage regions are penalized, effectively suppressing ineffective repeated access behavior. Simultaneously, through the target matrix T and distance function... Explicitly construct target point navigation rewards This approach ensures that agents can efficiently converge toward the task objective during collaboration. The joint design of repeated coverage penalty and target reward enables the reward function to express task-driven multi-objective collaborative behavior, thereby overcoming the shortcomings of insufficient granularity in traditional state representation and reward design. It also solves the problem that traditional methods are difficult to cope with the sparse and dynamically changing distribution of repeated coverage areas. It can be transferred to coverage environments of different scales and obstacle distributions, and has good versatility and scalability. In other words, it solves the problem that the existing Q-network structure is relatively simple and lacks the ability to model local spatial structures, which limits the fineness and generalization of agent policies. Furthermore, by introducing target point navigation and action masking mechanisms, it also solves the problem that existing technologies are prone to policy collapse or training instability when faced with practical constraints such as target point allocation and action feasibility limitations.

[0065] Third, in terms of path planning and resource consumption, the reward function of this invention significantly reduces the invalid walking distance by introducing repeated coverage penalty and optimal path guidance, and achieves dual optimization of energy consumption and time. At the same time, the dynamic weighted hybrid network adaptively adjusts the contribution of individual Q value to the global Q function, which improves the efficiency of multi-agent collaborative behavior under complex tasks.

[0066] IV. In terms of practical application prospects, the multi-agent intelligent coverage method proposed in this invention is not only applicable to typical gridded environments such as ground inspection and regional monitoring, but can also be extended to aerial drone collaborative coverage, underwater robot detection and regional perception tasks in complex industrial environments, and has broad engineering application potential and market promotion value.

[0067] V. This invention solves this problem through fine-grained design of the reward function. On the one hand, by introducing a coverage matrix and Voronoi region constraints into the reward function, duplicate coverage regions are penalized, effectively suppressing invalid repeated visits; on the other hand, the target matrix T and distance function are used... Explicitly construct target point navigation rewards This ensures that agents can efficiently converge toward the task objective during collaboration. The joint design of repetition coverage penalty and objective reward enables the reward function to express task-driven multi-objective collaborative behavior, thereby overcoming the shortcomings of insufficient granularity in traditional state representation and reward design. Attached Figure Description

[0068] Figure 1 This is a flowchart of the present invention;

[0069] Figure 2 This is a diagram of the global network structure. Detailed Implementation

[0070] The present invention will now be further described in conjunction with the accompanying drawings and specific embodiments.

[0071] The objective of this invention is to propose a multi-agent cooperative coverage method that can adapt to complex environmental constraints. It integrates the spatial partitioning capabilities of Voronoi diagrams with the policy optimization advantages of multi-agent reinforcement learning, achieving efficient and stable coverage control under various complex conditions such as overlapping coverage, target point constraints, and impassable regions. This method not only enhances the agents' perception and response capabilities to local environmental structures but also effectively addresses problems such as policy divergence, coverage redundancy, and cooperative failure in traditional methods through network structure optimization and action space constraint mechanisms. It possesses good generalization ability and practical application value.

[0072] like Figure 1 As shown, an intelligent covering method for complex environments based on Voronoi diagrams and multi-agent reinforcement learning includes the following steps:

[0073] Step S1: Divide the area to be covered in the actual scene into grids based on the coverage radius of the intelligent agent, and construct a global coverage matrix with the same size as the gridded area;

[0074] The specific steps are as follows:

[0075] Step S1.1: Based on any irregular area to be covered in the actual scene, place it into a space of size [size missing]. Within the rectangle, and based on the coverage radius of the agent. Mesh the rectangular area as Each agent covers a small rectangular patch, thus creating a gridded region. An agent reaching the center of any small rectangular patch within the gridded region is considered to have completed coverage of that region. The center of the small rectangular patch matrix serves as the agent's navigation point. and It is obtained from the following formula:

[0076]

[0077] Step S1.2: Construct a global coverage information map based on the grid region composed of small rectangular blocks. This yields the global coverage matrix. Represents the first in the global coverage matrix line, number The elements of the column represent the coverage status of each navigation point in the actual coverage area, and satisfy the following conditions: ,in, Indicates coordinates as The grid has been covered or the area is a restricted zone. Indicates coordinates as The grid needs to be overlaid again. Indicates coordinates as The grid needs to be covered twice. Indicates coordinates as The grid requires three more coverage operations. When an agent covers a certain area, the corresponding... The value is updated based on the number of times it has been covered.

[0078] Step S2: Randomly initialize the positions of multiple agents in the grid area divided into the area to be covered in the actual scene, construct the corresponding Voronoi diagram based on the current position of the agent, and dynamically update the Voronoi diagram according to the position change of the agent during the execution of the coverage task.

[0079] The specific steps are as follows:

[0080] Step S2.1: Randomly initialize the positions (i.e., two-dimensional coordinates) of multiple agents in the grid area divided into the area to be covered in the actual scene, and design the cost matrix. This is used to represent the movement cost between the agent and various navigation points in the gridded region, specifically expressed as:

[0081]

[0082] in, Represents Euclidean distance. Indicates the first Two-dimensional coordinates of an agent, Represents the first in the gridded region Two-dimensional coordinates of a navigation point Cost matrix Elements in;

[0083] Based on this cost matrix, the Hungarian algorithm is used to search for the minimum cost matching scheme in the full permutation space, that is, to assign a navigation point with the shortest distance to each agent.

[0084] Step S2.2: Based on the Voronoi diagram algorithm and the navigation points of each agent in the full permutation space, perform region segmentation, and assign navigation points within the Voronoi diagram region to the navigation points of each agent in the full permutation space, thus constructing the corresponding Voronoi diagram, which is the first... The formula for assigning navigation points to an agent within a Voronoi diagram region is:

[0085]

[0086] in, Represented as the first Each agent assigns its navigation point within the Voronoi diagram region to navigation points within the full permutation space. Indicates the first Navigation points for an intelligent agent within the entire permutation space. Represented as in the permutation space A finite set of navigation points in the middle, "For any" is a general quantifier used to indicate that a condition holds true for the entire set. Represented as the permutation space Any navigation point within the area express and The Euclidean distance;

[0087] Step S2.3: During the execution of the coverage task, the Voronoi diagram is dynamically updated every fixed time step according to the position changes of each agent, following the method in step S2.2.

[0088] Step S3: Based on the actual scene's area to be covered and the global coverage matrix, introduce a no-go zone matrix to penalize repeated coverage of the target area, mark the relevant areas in the global coverage matrix, and at the same time, select the target point matrix for priority coverage of the target point to be reached.

[0089] The specific steps are as follows:

[0090] Step S3.1: Considering the irregularity of the area to be covered in the actual scenario, design a no-go zone matrix. Used to represent The rectangle indicates whether the location is within the area to be covered. Represents the first in the forbidden zone matrix line, number The elements of the column, and satisfying ,in, Represented as the first line, number The navigation point of the column is within the area to be covered. Represented as the first line, number The navigation point of the column is outside the area to be covered, which is a restricted area, and the value of the restricted area is marked as 1 in the global coverage matrix, indicating that it does not need to be covered.

[0091] Step S3.2: Design the target point matrix This is used to represent the position of the navigation point that an agent in a gridded area should be at when it finishes performing a coverage task. Represents the first point in the target point matrix line, number The elements of the column, and satisfying ,in, This indicates that the waypoint is not the final destination the agent intends to reach. This indicates that the navigation point is the final destination that the agent must reach.

[0092] Step S4: Based on the global coverage matrix obtained in S3 and the Voronoi diagram before or after dynamic update, define the global state space, action space and reward function required for multi-agent reinforcement learning;

[0093] The specific steps are as follows:

[0094] Step S4.1: Construct an intelligent agent based on the global coverage matrix after the restricted area marking. Global state at time step The global state of the agent is:

[0095]

[0096] in, Represents the global coverage matrix exist The value at time;

[0097] Step S4.2: Design the agent's action space based on the global coverage matrix after the restricted area marking. The agent's actions are manifested as the agent's selection of surrounding navigation points. At each moment, the agent can only choose to stay at the current position or one of the 8 neighboring navigation points adjacent to the current position. This represents the 9 navigation points chosen by the agent, the first... The action space of an agent is defined as:

[0098]

[0099] Step S4.3: Based on the forbidden zone matrix Target point matrix Intelligent agents Global state at time 1, the first The action space of each agent, the Voronoi diagram before or after dynamic updates, and the requirements of the task in the real-world scenario are used to design the reward function for reinforcement learning. The formula is as follows:

[0100]

[0101]

[0102]

[0103] in, The AND operator is used to connect multiple logical conditions. A compound condition is true if and only if all the connected conditions are true. To control the weighting parameters of directional rewards, Represented as an intelligent agent In global state Take action below Instant rewards received Represented as an intelligent agent Current state and local state Distance from the target point Represented as an intelligent agent The coordinates of the matched target point This is the distance penalty coefficient. The threshold for reaching the target point, Indicates based on intelligent agent Current state and local state The goal point navigation reward obtained from the goal point Representing the current time The global reward is the sum of the immediate rewards for all agents and the reward for navigation to the target point. Let be the set of all actions taken by agents at time t. This represents the total number of intelligent agents.

[0104] The global state is encoded by a global coverage matrix. The action space corresponds one-to-one with the grid division of the area to be covered. The reward function is generated with coverage efficiency, coverage accuracy and overlap penalty as constraints, and is used to guide the training of subsequent policy models.

[0105] Step S5: Based on the reinforcement learning elements constructed in S4, construct a global network and train it;

[0106] The global network includes a super network that processes the action space to obtain weight parameters and offset parameters, an individual Q network that obtains the local Q values ​​of each agent based on the global state and action space of each agent, and a hybrid network that transforms the local Q values ​​into global vectors based on the outputs of the super network and the individual Q network.

[0107] The hypernetwork includes sequentially connected global states. Forbidden Zone Matrix and target point matrix The first, second, third, and fourth CNN networks are processed. The output of the fourth CNN network is processed to obtain the first and second weight parameters of the first and third multilayer perceptron MLPs. The output of the fourth CNN network is processed to obtain the first and second bias parameters of the second and fourth multilayer perceptron MLPs. The super network uses the global state containing information such as global coverage state, target point position, and repeated coverage marker as input. At the same time, it adopts the multilayer perceptron (MLP) structure to output the weight parameters and bias parameters of the hybrid network, thereby enhancing the adaptability of the hybrid network to changes in task scenarios.

[0108] The individual Q-network includes a fifth CNN network that extracts local spatial features of each agent based on the global state and action space of each agent. The local spatial features extracted by the fifth CNN network are used to estimate the state-action advantage value (i.e., using the state-action advantage function) of a fifth multilayer perceptron (MLP) and to estimate the state value (i.e., using the value function). A fusion layer is used to fuse the results of the fifth and sixth multilayer perceptrons to obtain the local Q-value. To cope with action constraints in complex environments, an action masking mechanism is introduced in the output layer of the individual Q-network. The Q-values ​​corresponding to inexecutable actions are set to negative infinity or minimum values ​​to avoid the policy network from selecting invalid or collision-causing actions during the action selection process, thereby significantly improving the stability and efficiency of the agent in the training and execution phases.

[0109] The formula for the fusion layer is:

[0110]

[0111] in, The state-action advantage value is the output of the fifth multilayer perceptron (MLP). This represents the state value output by the sixth-layer perceptron (MLP). The local Q-value obtained after fusion. This represents the size of the action space of the i-th agent. This indicates the global state of the i-th agent. Sum the advantage values ​​of all actions. This represents any action in the action space of the i-th agent;

[0112] The hybrid network adopts the additivity hybrid structure of QMIX, which first concatenates the local Q values ​​of each agent into a local Q vector. Then, a two-layer mapping network is used to generate the global Q-value: the first layer of the mapping network uses the first weight parameter and the first bias parameter dynamically generated by the supernetwork to monotonically weight the local Q-value and obtain intermediate features through non-linear activation; the second layer uses the second weight parameter and the second bias parameter of the mapping network to map the intermediate features to the global Q-value. In this context, all weight parameters and bias parameters are determined by the hypernetwork based on the global state. Forbidden Zone Matrix and target point matrix Dynamic generation is used to ensure centralized training and monotonicity constraints. The specific formula is as follows:

[0113]

[0114]

[0115]

[0116]

[0117]

[0118] in, This represents the global Q-value, which is an estimate of the maximum expected reward in the current state. It is an exponential linear unit activation function, and the final output is... For the global state-action value function used for optimization, Represents the local Q-vector. Indicates the first The local Q-value of an agent and Let represent the first weight parameter and the second weight parameter, respectively. The first weight parameter controls the importance of the local Q-values ​​of each agent in the global mixture, while the second weight parameter compresses the hidden features into a global Q-value. and This represents the first bias parameter and the second bias parameter. The first bias parameter is a dynamic bias of the hidden features, which is a nonlinear combination that adjusts the local Q-values. The second bias parameter is the global Q-value. The final bias, that is, the global correction of the global Q-value for all agents, , , and These represent the first, third, second, and fourth multilayer perceptron MLPs, respectively, with the weight parameters using absolute values. Ensure monotony.

[0119] Hybrid networks can ensure global Q-value Regarding individual Q value The additivity and monotonicity of satisfy:

[0120]

[0121] Ultimately, the global Q-value is optimized by minimizing the TD (Time Difference) error to ensure effective learning of the cooperative strategy.

[0122] The training mechanism is still based on the Centralized Training with Decentralized Execution (CTDE) framework, and adopts a parameter sharing strategy to improve training efficiency and reduce model redundancy.

[0123] The training process is as follows:

[0124] Initialize local Q-network (i.e., individual Q-network) parameters Hypernetwork parameters Target network parameters ;

[0125] Initialize the experience pool Maximum number of training rounds Maximum step size in a round Parameter update interval Learning rate , Soft update ratio Parameters;

[0126] Iteration loop ,implement:

[0127] Initialize agent position and obtain state ;

[0128] For each time step arrive ,implement:

[0129] Each agent is based on its current state. Using local Q-networks Strategy selection action Time-based intelligent agent Action space ;

[0130] All agents perform a joint action. The environment returns to the next state. , to receive a reward Termination mark .Will Store in experience pool middle;

[0131] If the update condition (current step number) is met If a batch of data is randomly sampled from the experience pool, the following update is performed:

[0132] Calculate the local Q-value of each agent using the current local Q-network;

[0133] The weights of the hybrid network are generated based on the global state using a hypernetwork, and then combined to obtain the global Q-value. ;

[0134] Generate targets using the Target Q-Network And calculate TD error If the requirements are met, the grid parameters are fixed after training and used for policy decisions during the execution phase; otherwise, the local Q-network parameters ϕ and the supernetwork parameters ψ are updated.

[0135]

[0136]

[0137]

[0138] Update target network hyperparameters:

[0139] .

[0140] Step S6: Based on the trained global network, obtain the global Q-values ​​for target point navigation and repeated coverage in irregular regions to evaluate coverage performance. Coverage performance evaluation includes coverage time, coverage accuracy, and target satisfaction. Coverage time characterizes the number of time steps required for the agent to complete all necessary coverage units in the task area; coverage accuracy characterizes the overlap ratio between the agent's actual coverage area and the target coverage area; target satisfaction characterizes the degree of matching between the actual number of coverage layers and the target number of coverage layers in areas with multiple coverage requirements.

[0141] In summary, this invention, by combining Voronoi graph partitioning with a multi-agent reinforcement learning framework, achieves a systematic improvement in coverage efficiency, policy optimization capability, model generalization, environmental adaptability, and practical deployment feasibility. On one hand, dynamic Voronoi partitioning enables adaptive allocation of task load among agents, significantly improving overall coverage speed and resource utilization. On the other hand, the introduction of a dominance function-based dual-branch network structure and an illegal action shielding mechanism effectively enhances the stability and convergence efficiency of policy learning. Furthermore, the state representation and reward mechanism designed in this invention possess high environmental awareness and flexibility, enabling the trained model to maintain good performance even in complex scenarios with varying sizes, obstacle densities, and task objectives. In addition, the overall structure of this method has good scalability and deployability, is compatible with various reinforcement learning frameworks, adapts to deployment constraints and communication conditions in real-world scenarios, and has broad engineering application prospects. Therefore, this invention provides a practical and effective technical solution for intelligent coverage tasks.

[0142] The above are merely representative embodiments among the many specific applications of this invention, and do not constitute any limitation on the scope of protection of this invention. All technical solutions formed by transformation or equivalent substitution fall within the scope of protection of this invention.

Claims

1. A method for intelligent covering in complex environments based on Voronoi diagrams and multi-agent reinforcement learning, characterized in that, Includes the following steps: Step S1: Divide the area to be covered in the actual scene into grids based on the coverage radius of the intelligent agent, and construct a global coverage matrix with the same size as the gridded area; Step S2: Randomly initialize the positions of multiple agents in the grid area divided into the area to be covered in the actual scene, construct the corresponding Voronoi diagram based on the current position of the agent, and dynamically update the Voronoi diagram according to the position change of the agent during the execution of the coverage task. Step S3: Based on the actual scene's area to be covered and the global coverage matrix, introduce a no-go zone matrix to penalize repeated coverage of the target area, mark the relevant areas in the global coverage matrix, and at the same time, select the target point matrix for priority coverage of the target point to be reached. Step S4: Based on the global coverage matrix obtained in S3 and the Voronoi diagram before or after dynamic update, define the global state space, action space and reward function required for multi-agent reinforcement learning; Step S5: Based on the reinforcement learning elements constructed in S4, construct a global network and train it; Step S6: Based on the trained global network, obtain the target point navigation and global Q value of repeated coverage in the irregular area to evaluate the coverage performance.

2. The intelligent coverage method for complex environments based on Voronoi diagrams and multi-agent reinforcement learning according to claim 1, characterized in that, The specific steps in step S1 are as follows: Step S1.1: Based on any irregular area to be covered in the actual scene, place it into a space of size [size missing]. Within the rectangle, and based on the coverage radius of the agent. Mesh the rectangular area as Each agent covers a small rectangular patch, thus creating a gridded region. An agent reaching the center of any small rectangular patch within the gridded region is considered to have completed coverage of that region. The center of the small rectangular patch matrix serves as the agent's navigation point. and It is obtained from the following formula: Step S1.2: Construct a global coverage information map based on the grid region composed of small rectangular blocks. This yields the global coverage matrix. Represents the first in the global coverage matrix line, number The elements of the column represent the coverage status of each navigation point in the actual coverage area, and satisfy the following conditions: ,in, Indicates coordinates as The grid has been covered or the area is a restricted zone. Indicates coordinates as The grid needs to be overlaid again. Indicates coordinates as The grid needs to be covered twice. Indicates coordinates as The grid requires three more coverage operations. When an agent covers a certain area, the corresponding... The value is updated based on the number of times it has been covered.

3. The intelligent covering method for complex environments based on Voronoi diagrams and multi-agent reinforcement learning according to claim 1, characterized in that, The specific steps in step S2 are as follows: Step S2.1: Randomly initialize the positions (i.e., two-dimensional coordinates) of multiple agents in the grid area divided into the area to be covered in the actual scene, and design the cost matrix. This is used to represent the movement cost between the agent and various navigation points in the gridded region, specifically expressed as: in, Represents Euclidean distance. Indicates the first Two-dimensional coordinates of an agent, Represents the first in the gridded region Two-dimensional coordinates of a navigation point Cost matrix Elements in; Based on this cost matrix, the Hungarian algorithm is used to search for the minimum cost matching scheme in the full permutation space, that is, to assign a navigation point with the shortest distance to each agent. Step S2.2: Based on the Voronoi diagram algorithm and the navigation points of each agent in the full permutation space, perform region segmentation, and assign navigation points within the Voronoi diagram region to the navigation points of each agent in the full permutation space, thus constructing the corresponding Voronoi diagram, which is the first... The formula for assigning navigation points to an agent within a Voronoi diagram region is: in, Represented as the first Each agent assigns its navigation point within the Voronoi diagram region to navigation points within the full permutation space. Indicates the first Navigation points for an intelligent agent within the entire permutation space. Represented as in the permutation space A finite set of navigation points in the middle, "For any" is a general quantifier used to indicate that a condition holds true for the entire set. Represented as the permutation space Any navigation point within the area express and The Euclidean distance; Step S2.3: During the execution of the coverage task, the Voronoi diagram is dynamically updated every fixed time step according to the position changes of each agent, following the method in step S2.

2.

4. The intelligent covering method for complex environments based on Voronoi diagrams and multi-agent reinforcement learning according to claim 1, characterized in that, The specific steps of step S3 are as follows: Step S3.1: Considering the irregularity of the area to be covered in the actual scenario, design a no-go zone matrix. Used to represent The rectangle indicates whether the location is within the area to be covered. Represents the first in the forbidden zone matrix line, number The elements of the column, and satisfying ,in, Represented as the first line, number The navigation point of the column is within the area to be covered. Represented as the first line, number The navigation point of the column is outside the area to be covered, which is a restricted area, and the value of the restricted area is marked as 1 in the global coverage matrix, indicating that it does not need to be covered. Step S3.2: Design the target point matrix This is used to represent the position of the navigation point that an agent in a gridded area should be at when it finishes performing a coverage task. Represents the first point in the target point matrix line, number The elements of the column, and satisfying ,in, This indicates that the waypoint is not the final destination the agent intends to reach. This indicates that the navigation point is the final destination that the agent must reach.

5. The intelligent coverage method for complex environments based on Voronoi diagrams and multi-agent reinforcement learning according to claim 1, characterized in that, The specific steps in step S4 are as follows: Step S4.1: Construct an intelligent agent based on the global coverage matrix after the restricted area marking. Global state at time step The global state of the agent is: in, Represents the global coverage matrix exist The value at time; Step S4.2: Design the agent's action space based on the global coverage matrix after the restricted area marking. The agent's actions are manifested as the agent's selection of surrounding navigation points. At each moment, the agent can only choose to stay at the current position or one of the 8 neighboring navigation points adjacent to the current position. This represents the 9 navigation points chosen by the agent, the first... The action space of an agent is defined as: Step S4.3: Based on the forbidden zone matrix Target point matrix Intelligent agents Global state at time 1, the first The action space of each agent, the Voronoi diagram before or after dynamic updates, and the requirements of the task in the real-world scenario are used to design the reward function for reinforcement learning. The formula is as follows: in, The AND operator is used to connect multiple logical conditions. A compound condition is true if and only if all the connected conditions are true. To control the weighting parameters of directional rewards, Represented as an intelligent agent In global state Take action below Instant rewards received Represented as an intelligent agent Current state and local state Distance from the target point Represented as an intelligent agent The coordinates of the matched target point This is the distance penalty coefficient. The threshold for reaching the target point, Indicates based on intelligent agent Current state and local state The goal point navigation reward obtained from the goal point Representing the current time The global reward is the sum of the immediate rewards for all agents and the reward for navigation to the target point. Let be the set of all actions taken by agents at time t. This represents the total number of intelligent agents. Furthermore, the global network in step S5 includes a global state. Forbidden Zone Matrix and target point matrix The super network is processed to obtain weight parameters and offset parameters; the individual Q network obtains the local Q value of each agent based on the global state and action space of each agent; and the hybrid network transforms the local Q value into a global vector based on the output of the super network and the individual Q network. The hypernetwork includes sequentially connected global states. Forbidden Zone Matrix and target point matrix The first CNN network, the second CNN network, the third CNN network and the fourth CNN network are processed, and the output of the fourth CNN network is processed to obtain the first weight parameter and the second weight parameter of the first multilayer perceptron MLP and the third multilayer perceptron MLP, and the output of the fourth CNN network is processed to obtain the first bias parameter and the second bias parameter of the second multilayer perceptron MLP and the fourth multilayer perceptron MLP. The individual Q-network includes a fifth CNN network that extracts local spatial features of each agent based on their global state and action space. The local spatial features extracted by the fifth CNN network are used to estimate the state-action advantage value of a fifth multilayer perceptron (MLP) and to estimate the state value of a sixth multilayer perceptron (MLP). The outputs of the fifth and sixth MLPs are then fused to obtain a fusion layer for the local Q-value. The formula for the fusion layer is as follows: in, The state-action advantage value is the output of the fifth multilayer perceptron (MLP). This represents the state value output by the sixth-layer perceptron (MLP). The local Q-value obtained after fusion. This represents the size of the action space of the i-th agent. This indicates the global state of the i-th agent. Sum the advantage values ​​of all actions. This represents any action in the action space of the i-th agent; The hybrid network adopts the additivity hybrid structure of QMIX, which first concatenates the local Q values ​​of each agent into a local Q vector. Then, a two-layer mapping network is used to generate the global Q-value: the first layer of the mapping network uses the first weight parameter and the first bias parameter dynamically generated by the supernetwork to monotonically weight the local Q-value and obtain intermediate features through non-linear activation; the second layer uses the second weight parameter and the second bias parameter of the mapping network to map the intermediate features to the global Q-value. In this context, all weight parameters and bias parameters are determined by the hypernetwork based on the global state. Forbidden Zone Matrix and target point matrix Dynamic generation is used to ensure centralized training and monotonicity constraints. The specific formula is as follows: in, This represents the global Q-value, which is an estimate of the maximum expected reward in the current state. It is an exponential linear unit activation function, and the final output is... For the global state-action value function used for optimization, Represents the local Q-vector. Indicates the first The local Q-value of an agent and Let represent the first weight parameter and the second weight parameter, respectively. The first weight parameter controls the importance of the local Q-values ​​of each agent in the global mixture, while the second weight parameter compresses the hidden features into a global Q-value. and This represents the first bias parameter and the second bias parameter. The first bias parameter is a dynamic bias of the hidden features, which is a nonlinear combination that adjusts the local Q-values. The second bias parameter is the global Q-value. The final bias, that is, the global correction of the global Q-value for all agents, , , and These represent the first multilayer perceptron (MLP), the third multilayer perceptron (MLP), the second multilayer perceptron (MLP), and the fourth multilayer perceptron (MLP), respectively. A smart overlay system for complex environments based on Voronoi diagrams and multi-agent reinforcement learning includes a memory processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method of claim 1. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method of claim 1.

Images