Multi-layer game reinforcement learning method for swarm intelligence regional coverage control

Abstract

The application discloses a kind of multi-layer game reinforcement learning methods for intelligent group area coverage control, comprising: each intelligent agent in intelligent group is mapped as game player, and the task target of intelligent agent is combined to construct revenue evaluation mechanism, and form group game model;Based on the physical capability constraint of intelligent agent, construct perception model and communication model, design coverage efficiency, running safety, energy efficiency and communication collaboration evaluation index, and integrate as the vector revenue function of intelligent agent;TD3 model is used as the game player role of intelligent agent, and the association of reinforcement learning and group game is established;Multi-layer semi-distributed game reinforcement learning framework is constructed, and environment interaction is carried out, to solve the intelligent group area coverage control strategy satisfying Pareto Nash equilibrium.The application realizes the balance of individual and collective benefits, improves the calculation efficiency through semi-distributed iterative algorithm, and adapts to the solution demand of large-scale complex system.

Description

Technical Field

[0001] This invention belongs to the field of next-generation artificial intelligence technology in the Internet of Things, and relates to a multi-layer game-theoretic reinforcement learning method for regional coverage control of intelligent groups. Background Technology

[0002] With the rapid development of swarm intelligence technology, swarm intelligence agents have demonstrated enormous potential in various application fields, especially in area coverage control. Area coverage control refers to achieving effective coverage of a target area through the distribution and interaction of intelligent agents in space to meet specific mission requirements. This technology is not limited to traditional industrial applications but is also widely used in military, environmental monitoring, emergency rescue, and many other fields.

[0003] Traditional region coverage methods often rely on centralized algorithms or manual planning, resulting in low information processing efficiency, insufficient adaptability, and slow response to environmental changes. Currently, numerous solutions exist for the coverage control problem. However, most research focuses on scenarios where the agent dynamics model and system parameters are known, with a few relaxing one or two conditions. With advancements in learning algorithms, coverage control methods based on multi-agent reinforcement learning have been developed, effectively addressing challenges posed by inaccurate models and unknown environments. However, when the coverage control problem lacks an optimal solution or multiple evaluation criteria conflict and their relationships are unclear, multi-agent reinforcement learning strategies may struggle to balance individual and collective interests, and may even fail to converge. This is because, whether centralized or distributed training, or entirely centralized or distributed, these methods can be viewed as weighted methods with concentrated benefits. In real-world environments, the diversity among agents—including differences in cognitive abilities, behavioral patterns, and goal orientations—is naturally present. This diversity allows agents to fully utilize their unique strengths, thereby improving system performance and adaptability. Therefore, in addressing the challenges of collaborative coverage control, fully embracing and utilizing this diversity to ensure the rational allocation of tasks and resources is crucial. However, current research mainly focuses on interest-centered weighted approaches, while neglecting the complex dynamics of cooperation and competition among stakeholders. Summary of the Invention

[0004] To address the aforementioned problems, this invention provides a multi-layered game-theoretic reinforcement learning method for intelligent swarm area coverage control. This invention aims to provide an innovative method for intelligent swarm area coverage control, achieving effective coverage of target areas through efficient organization and management of the intelligent swarm, and adapting to various environmental conditions and task requirements. This method not only considers the balance between coverage area and effectiveness but also emphasizes energy reduction, collision avoidance, and communication maintenance, thereby improving the system's practicality and reliability. The practical scenarios addressed by this invention's coverage control encompass cross-domain collaboration, including maritime, land, and air-to-sea operations, as well as game-theoretic and intelligent collaboration in fields such as economics and agriculture.

[0005] The technical solution adopted in this invention is as follows:

[0006] A multi-layered game-theoretic reinforcement learning method for intelligent swarm regional coverage control includes the following steps:

[0007] S1. For an intelligent group consisting of several intelligent agents, where the intelligent agents are drones, unmanned boats or other unmanned intelligent devices, each intelligent agent in the intelligent group is mapped to a game player, the strategy space and strategy combination of the intelligent agents are defined, and a benefit evaluation mechanism is constructed in combination with the task objectives of the intelligent agents to form a group game model.

[0008] S2. Based on the physical capability constraints of the agent, construct a perception model and a communication model, design evaluation indicators covering performance, operational safety, energy efficiency and communication coordination, and integrate the evaluation indicators into a vector reward function for the agent;

[0009] S3. The TD3 model is used as the game player role of the agent to realize the agent's policy learning, defining the learning state, action and reward, and establishing the connection between reinforcement learning and group game;

[0010] S4. Construct a multi-layer semi-distributed game reinforcement learning framework, realize the dynamic interaction between the agent and the environment and between the agents through the environment interaction model, perform policy iterative optimization based on reinforcement learning, and combine the convergence judgment mechanism to solve the intelligent group regional coverage space control strategy that satisfies Pareto Nash equilibrium.

[0011] Furthermore, step S1 specifically includes:

[0012] Setting by A group game system consisting of 10 players, where each player corresponds to an agent in the intelligent swarm. , ;

[0013] Define each intelligent agent Limited strategy space and hybrid strategy space ,in It is the cardinality of the strategy space;

[0014] Define the Cartesian product of the mixed policy space of all agents as: By each intelligent agent Hybrid strategy Combining to form a game situation ;

[0015] Define each intelligent agent target set ,in It is the cardinality of the target set, each target This corresponds to a task requirement of the intelligent agent;

[0016] Building intelligent agents Vector profit function Each element of the vector profit function corresponds to the target set. The payoff value of one of the objectives, and the payoff value varies with the game situation. Dynamically adjust to changes;

[0017] The group game model is composed of the number of players, the strategy space, the mixed strategy space, the vector payoff function, and the objective set. .

[0018] Furthermore, the physical capability constraints in step S2 include: speed not exceeding the rated speed, acceleration not exceeding the rated acceleration, heading angle within the rated effective range, and having fixed energy reserves, maximum sensing range, and maximum communication range.

[0019] Furthermore, the perception model construction process in step S2 includes: defining the perception area of ​​the agent as a circular area with its own position as the center and the maximum perception distance as the radius. Within the perception area, the probability of the agent detecting any detection point decreases linearly with the increase of distance. Based on the radius constraint and the linear decay law, a perception model is constructed to quantify the agent's ability to perceive detection points within the area.

[0020] The process of constructing the communication model includes: defining the communication region of an agent as a circular region with its own position as the center and the maximum communication distance as the radius. Within the communication region, the level of communication interaction between the agent and other agents decreases linearly with increasing distance. Based on the radius constraint and the linear decay law, a communication model is constructed to quantify the communication capabilities between agents.

[0021] Furthermore, the evaluation indicators for design coverage performance, operational safety, energy efficiency, and communication coordination in step S2 specifically include:

[0022] Design coverage effectiveness evaluation index: For any monitoring point within the agent's perception area, calculate the cumulative coverage level of the monitoring point by the agent from the initial time to the current time. If the cumulative coverage level exceeds a preset effective coverage threshold, the monitoring point is determined to have achieved effective coverage. Calculate the area of ​​monitoring points within the agent's perception range that have achieved effective coverage at the current time, and use this as the individual effective coverage evaluation function. ;Calculate the area of ​​monitoring points in the target region of the task within the task time that are effectively covered by at least one agent, and use this as the group effective coverage evaluation function. ;

[0023] Design and implement security evaluation metrics: Construct a density evaluation function based on kernel function, bandwidth parameter, and security thresholds between agents. The density evaluation function quantifies the crowding level among agents to avoid collisions and excessive aggregation.

[0024] Design an energy efficiency evaluation index: Construct an individual energy consumption evaluation function based on the agent's step size over a period of time. Construct a residual energy evaluation function based on the agent's total energy reserves and individual cumulative energy consumption. When the remaining energy is lower than a preset energy reserve threshold, the agent stops moving and disconnects from the communication network.

[0025] Design a communication coordination evaluation index: An undirected graph is used to describe the communication topology of the group system. For any decision time, the undirected graph contains a set of nodes consisting of the positions of all agents and a set of edges formed by pairs of agents capable of communication. The connectivity of the undirected graph is determined by calculating the second smallest eigenvalue of its Laplacian matrix. A communication evaluation function is constructed based on the connectivity of the undirected graph. It is used to quantify the communication cooperation of intelligent agents.

[0026] Furthermore, the individual effective coverage evaluation function Effective Coverage Evaluation Function Density evaluation function Individual energy consumption evaluation function Residual energy evaluation function and communication evaluation function Together they form a vector benefit function .

[0027] Furthermore, step S3 specifically includes:

[0028] Using the TD3 model as the player role of the agent in the game, the group game model constructed in step S1 and the vector payoff function obtained in step S2 are integrated to establish the correspondence between the TD3 model and the group game, and the state of the agent in each decision stage is defined. ,action and rewards ;

[0029] The state It consists of the individual strategies of the agent itself and the collective strategies of all other agents, and the state also serves as the game situation in the decision-making stage.

[0030] The action It consists of two parts: a hybrid strategy for the agent regarding step size and a hybrid strategy regarding heading angle.

[0031] The reward It consists of the agent's vector reward function and the target preference weight vector of the vector reward function, and the target preference weight vector is selected according to the task requirements.

[0032] Furthermore, the multi-layer semi-distributed game reinforcement learning process in step S4 includes t learning rounds, each learning round includes e learning cycles, and each learning cycle includes k decision stages.

[0033] The specific steps of the multi-layer semi-distributed game reinforcement learning process include:

[0034] Initialize the network parameters and network structure of the TD3 model for each agent, including the parameters of the critic evaluation network and the actor decision network, and set the target policy delay frequency, local cutoff condition, global cutoff condition, action noise, and target preference weight vector for each agent.

[0035] Perform learning round iterations: In each round, reset or adjust the environmental scene parameters, and then perform learning cycle iterations;

[0036] Execute learning cycle iteration: In each cycle, the decision-making stage iteration is executed first. After all agents have completed the decision-making stage iteration, the similarity between the current cycle and the previous cycle's group game situation is calculated. If the similarity satisfies the global cutoff condition, the group strategy is determined to be globally converged, the current cycle iteration is terminated and the next round is entered; otherwise, the decision-making stage iteration is executed again.

[0037] Decision-making phase iteration: In each decision-making phase, the agent selects actions with exploratory noise based on the TD3 model, obtains immediate rewards, the next state, and remaining energy through the environmental interaction model, and stores the experience tuple composed of the current state, action, reward, next state, and remaining energy into the experience pool; samples are randomly drawn from the experience pool to update the network parameters of the TD3 model; when the similarity of the states of two consecutive decision-making phases meets the local cutoff condition, the agent's policy is determined to be locally converged, and the current decision-making phase iteration is terminated; after all agents have completed the decision-making phase iteration, the process returns to the periodic iteration level for global convergence judgment.

[0038] Furthermore, each agent constructs an environment interaction model, which can update the state and position of the next decision stage based on the individual strategy of the agent and the collective strategy of other agents, calculate the vector reward of the agent, obtain the immediate reward by weighted summation of the vector rewards based on preference weights, and calculate the similarity between the state of the current decision stage and the previous decision stage.

[0039] Furthermore, after all learning rounds have been iterated, the output group policy is the Pareto Nash equilibrium solution, which satisfies the following: each agent cannot improve its individual reward by adjusting its own policy alone, nor can it increase any of its own objective rewards without reducing the rewards of other objective rewards.

[0040] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0041] 1. The method of this invention excels in handling continuous actions and state spaces, and is particularly suitable for decision-making in complex dynamic environments. Based on a multi-layer semi-distributed reinforcement learning architecture, it effectively overcomes environmental instability and the curse of dimensionality, not only compensating for the shortcomings of fully distributed reinforcement learning in terms of environmental stability, but also solving the curse of dimensionality problem in centralized learning.

[0042] 2. The method of this invention can dynamically adapt to environmental changes and other intelligent agent decisions. It not only considers individual goals but also balances individual preferences with group interests, maintaining the principle of fairness and effectively improving system performance and the realization of intelligent decision-making. By balancing individual and collective interests, it significantly improves the overall performance of the system and ensures the fairness of decision-making, making it particularly suitable for multi-objective decision-making scenarios.

[0043] 3. Compared to traditional centralized training and distributed execution frameworks, the unique feature of the multi-layered semi-distributed reinforcement learning framework is that it does not require centralized training. Its policy iteration process incorporates game interaction, particularly the interaction between policies, effectively solving the equilibrium problem in complex dynamic games by integrating game theory and reinforcement learning.

[0044] In summary, the method of the present invention has significant advantages in handling continuous actions and state space problems, and can flexibly respond to environmental changes and balance individual and collective interests, which is of great significance to the field of intelligent decision-making. Attached Figure Description

[0045] Figure 1 This is a flowchart of a multi-layer game-theoretic reinforcement learning method for regional coverage control of intelligent groups, as described in an embodiment of the present invention.

[0046] Figure 2 This is a schematic diagram illustrating the collaborative detection and alarm coverage of an unknown sea area along the typhoon's movement trajectory in this embodiment of the invention.

[0047] Figure 3 This is a graph showing the payoff of P2 during the game interaction process when P2 executes a hybrid strategy in an embodiment of the present invention.

[0048] Figure 4 This is a multi-objective hybrid strategy payoff diagram for all agents in the second round of the game in this embodiment of the invention. Detailed Implementation

[0049] The technical solution of the present invention will be further described clearly and in detail below with reference to the accompanying drawings and specific examples.

[0050] like Figure 1 As shown, a multi-layered game-theoretic reinforcement learning method for intelligent group regional coverage control includes the following steps:

[0051] S1. For an intelligent swarm consisting of several agents, where the agents are drones, unmanned surface vessels, or other unmanned intelligent devices, each agent in the swarm is mapped to a player in a game. The strategy space and strategy combinations of the agents are defined, and a payoff evaluation mechanism is constructed based on the agents' task objectives, forming a swarm game model. Each agent in the intelligent swarm is mapped to a player in a game, constructing a swarm game model with a finite number of... A group game system consisting of multiple players, in which Representing intelligent agents The index. The available strategy space is a finite set. ,in It is the cardinality of the policy space. Corresponding to... ,definition The hybrid strategy space is Define the intelligent agent. The target set is ,in It is the cardinality of the target set. Each target Corresponding to a specific outcome, the agent By selecting a hybrid strategy To optimize these results. The Cartesian product of the mixed policy space of all agents is represented as: Each player From Choose one strategy This creates a game situation. Each player The vector profit function is expressed as Multi-objective group game can be defined as... .

[0052] S2. Based on the physical capability constraints of the agent, construct a perception model and a communication model, design evaluation indicators covering performance, operational safety, energy efficiency and communication coordination, and integrate the evaluation indicators into a vector reward function for the agent.

[0053] In real-world operating conditions, the behavior of intelligent agents is limited by communication and sensing capabilities, and also varies due to limitations in mobility and energy resources. This invention considers agent performance including rated speed and acceleration, effective range of heading angle, energy reserves, detection accuracy, and communication coverage. This ensures the intelligent agent's performance during mission execution. For safety and controllability to be achieved, the following conditions must be met: speed acceleration and heading angle Each must satisfy , and ,in , and These represent the rated values ​​for velocity, acceleration, and heading angle, respectively.

[0054] Will The sensory area is defined as the center. ,radius circle ,radius This indicates its maximum sensing range. Considering... The ability to perceive decreases linearly with distance, therefore In the sensing area Event detected internally The probability is denoted as ,in express The position (two-dimensional plane). The perception model is designed as follows:

[0055] ,

[0056] in express Peak perceived quality, and It is a function that decays linearly with increasing distance, defined as .

[0057] Will The communication area is defined as... With the center, and a circle with radius , express The maximum communication range. Considering that communication capability decreases linearly with distance, and The communication interaction model can be designed as follows:

[0058] ,

[0059] in express The peak propagation intensity.

[0060] If the interaction level Above the threshold ,but and Communication can be established, and Recorded as an undirected graph One of the edges in, where Represents a node. Representation diagram The edges in.

[0061] consider Any point within the sensing area The cumulative coverage level it receives is within a time interval The inner meaning is as follows:

[0062] ,

[0063] If in time Cumulative coverage level Exceeding the threshold Then the point is considered Effective coverage has been achieved. The individual effective coverage rate assessment function is defined as follows:

[0064]

[0065] Define function ,in Indicates in Always in the sensing area Points in The set. If for any point within the region In the time interval If effective coverage is achieved at least once, it is called... Region in time interval Effective coverage was achieved within the population. The population effective coverage evaluation function is defined as follows:

[0066] ,

[0067] This invention aims to derive and optimize control strategies to achieve swarm control in the task space. Effectively achieving the designated area coverage level. Population density and coverage area are inversely proportional: higher density results in smaller coverage area, leading to an increase in the number of encounters between individuals. Based on this, a proposed... The density evaluation function is shown below:

[0068]

[0069] in, Represents the kernel function. Indicates bandwidth, defines parameters: ,in express and The distance between them Indicates the safety threshold. This is a hyperparameter. Considering the limited energy capacity of an individual, reducing its travel distance can effectively reduce energy consumption. Therefore, a time interval is defined. The individual energy consumption evaluation function within is defined as:

[0070]

[0071] in yes exist Step size of time, These are hyperparameters. Extending this concept to measuring residual energy, we get:

[0072]

[0073] in, express Total energy reserves, It's a hyperparameter. If at time... , Energy reserves are lower than That is, the conditions are met. ,but Stop moving and automatically disconnect from the communication network.

[0074] Effective communication is crucial for fostering collaboration within a collective system. Considering bidirectional communication, this invention employs an undirected graph. To describe the topology of a group system. For any decision time , The set of nodes including the locations of all individuals and the edge set formed between intelligent individuals capable of communication. If the diagram Laplace matrix The second smallest eigenvalue A value greater than 0 indicates that the graph is connected, which means that in the graph... There exists a path connecting all vertices. Based on the above inference, the communication evaluation function is as follows:

[0075] ,

[0076] Among them, hyperparameters By integrating the above objective evaluation function, we can obtain... Vector profit function The evaluation function can be selected based on the actual operational situation and individual needs, and includes, but is not limited to, various functions such as linear and nonlinear functions. Vector benefit functions weight the sum of individual and group benefits to achieve benefit equilibrium, and include, but are not limited to, the weighted vector method, the weighted Wilson global criterion, the Pareto minimum criterion, the weighted sum Chebyshev function, the weighted distributed function, and the ordered weighted function.

[0077] S3. The TD3 model is used as the game player role of the agent to realize the agent's policy learning, defining the learning state, action and reward, and establishing the connection between reinforcement learning and group game.

[0078] This invention introduces the existing, mature, and widely used dual-delay deep deterministic policy gradient (TD3) single-agent reinforcement algorithm, and establishes the connection between its key elements and multi-objective swarm game. It then executes a multi-layer semi-distributed game reinforcement learning algorithm to seek the swarm Pareto Nash equilibrium solution, i.e., the Pareto optimal policy of swarm coverage control. TD3 is considered as an alternative agent. The player character, through the integration of models Sum of vector benefits To define its in the first Round decision phase status ,action and rewards The details are as follows (the first part) (in the round):

[0079] state It consists of two parts: the intelligent agent Individual strategies Other intelligent agents The collective strategy, in the During the cycle, that is From a game theory perspective, Also known as the first The game situation at each stage. At each decision-making stage. In the middle, intelligent agents Optimize your strategies through learning This is to ensure that we maintain our advantage in the current situation until a balance is reached. Represents a periodic index, where This represents the maximum number of cycles.

[0080] action It consists of two parts: one part is a hybrid strategy Its upper and lower limits are Step size is The other part is about the heading angle. Hybrid strategy Its upper and lower limits are Therefore, the action It can be represented as .

[0081] award The definition is as follows: ,in It is an intelligent agent The weighted vector of the vector revenue function. Select according to actual needs. To standardize .

[0082] The reason for choosing the TD3 single-unit reinforcement learning algorithm in this invention is that it has enhanced stability, fast convergence, strong anti-overfitting properties, and good applicability, and can be used in continuous and complex environments.

[0083] S4. Construct a multi-layer semi-distributed game reinforcement learning framework, realize the dynamic interaction between the agent and the environment and between the agents through the environment interaction model, perform policy iterative optimization based on reinforcement learning, and combine the convergence judgment mechanism to solve the intelligent group regional coverage space control strategy that satisfies Pareto Nash equilibrium.

[0084] In group games, the outcome of interactions between agents can be described by the game situation. In the... Each round During the cycle, the intelligent agent Based on other players' current strategies They adjust their strategies to increase their gains. After a period of interaction within the game, the situation transforms into... This iterative process continues until the game has progressed to a certain cycle. In this period In this scenario, each agent cannot improve its individual reward by adjusting its strategy, nor can it increase the reward of any objective without harming the rewards of other objectives. This situation is called a model... The Pareto Nash equilibrium solution. Based on this, this invention combines reinforcement learning methods to design a multi-layer semi-distributed game reinforcement learning framework to solve the model. The Pareto Nash equilibrium solution.

[0085] The multi-layered semi-distributed game reinforcement learning process includes t learning rounds, each learning round includes e learning cycles, and each learning cycle includes k decision stages. The specific steps include:

[0086] Initialize the network parameters and network structure of the TD3 model for each agent, including the parameters of the critic evaluation network and the actor decision network, and set the target policy delay frequency, local cutoff condition, global cutoff condition, action noise, and target preference weight vector for each agent.

[0087] Perform learning round iterations: In each round, reset or adjust the environmental scene parameters, and then perform learning cycle iterations;

[0088] Execute learning cycle iteration: In each cycle, the decision-making stage iteration is executed first. After all agents have completed the decision-making stage iteration, the similarity between the current cycle and the previous cycle's group game situation is calculated. If the similarity satisfies the global cutoff condition, the group strategy is determined to be globally converged, the current cycle iteration is terminated and the next round is entered; otherwise, the decision-making stage iteration is executed again.

[0089] Decision-making phase iteration: In each decision-making phase, the agent selects actions with exploratory noise based on the TD3 model, obtains immediate rewards, the next state, and remaining energy through the environmental interaction model, and stores the experience tuple composed of the current state, action, reward, next state, and remaining energy into the experience pool; samples are randomly drawn from the experience pool to update the network parameters of the TD3 model; when the similarity of the states of two consecutive decision-making phases meets the local cutoff condition, the agent's policy is determined to be locally converged, and the current decision-making phase iteration is terminated; after all agents have completed the decision-making phase iteration, the process returns to the periodic iteration level for global convergence judgment.

[0090] Each agent constructs an environment interaction model, which can update the state and position of the next decision stage based on the individual strategy of the agent and the collective strategy of other agents, calculate the vector reward of the agent, obtain the immediate reward by weighted summation of the vector rewards based on preference weights, and calculate the similarity between the state of the current decision stage and the previous decision stage.

[0091] The pseudocode for the training process of multi-layer semi-distributed game reinforcement learning is shown in Algorithm 1. First, through initialization... The meta-TD3 model acts as a participant in the game. Subsequently, an environmental interaction model is established to facilitate individual strategy learning. Next, a double-loop game framework is designed to evolve the game. The game scenario is formed by individual players. Ultimately, the DLGIF game theory framework is used to seek the PNES of MOGG.

[0092] Algorithm 1: Multi-layer Semi-distributed Game Theory Reinforcement Learning Algorithm

[0093] 1. Initialize network parameters and network structure for each player:

[0094] 1) Critical network , and its target network , network parameters of critic1 critic2 network parameters ; Target critic1 network parameters Target critic2 network parameters ;

[0095] 2) Actor Network and its corresponding target network Target actor network parameters actor network parameters ;

[0096] 3) Target policy delay frequency Local cutoff condition and global cutoff condition motion noise , It follows a standard normal distribution;

[0097] 4) Initialize the preference weights of each player for each target. ;

[0098] 2: Environment Model Establishment: For each player Design the environment interaction model, see Algorithm 2;

[0099] 3: Iterative Loop Rounds ;

[0100] 4: Iteration cycle ;

[0101] 5: Iterative Decision-Making Phase Find the dominant strategy under the current game situation, see Algorithm 3;

[0102] 6: Judgment and Decision-Making Stage Is the desired strategy stable, convergent, or has it reached a set threshold number of iterations? If yes, then begin the next iteration cycle; otherwise, continue iterating through the decision-making phase. This involves calculating the similarity between the player state in the current decision-making phase and the previous decision-making phase, i.e., calculating... Then determine whether the local cutoff condition has been met. ;

[0103] 7: Determine the cycle Whether the game situation is stable, convergent, or has reached a set threshold number of cycles. If yes, then start the next round; otherwise, input the current situation to enter the decision-making phase and continue the iteration cycle; during this process, it is necessary to calculate the similarity between the current cycle and the previous cycle's situation, that is, calculate... Then determine whether the global cutoff condition has been met. ;

[0104] 8: After the round of iteration ends, output the game situation.

[0105] Based on the framework of Algorithm 1, the development of the game situation can be guided by adjusting the player's preferences and goals; and further iteration of strategies can be promoted by updating the situation and optimizing the strategy space.

[0106] Algorithm 2: Player-Environment Interaction Algorithm

[0107] 1: Input Player Index and status ;

[0108] 2: Action-based Update players The state of the next decision-making stage and location;

[0109] 3: Calculate Players Each target return value, i.e., calculation ;

[0110] 4: Calculate Players Instant rewards ;

[0111] 5: Calculate the similarity between the player's state in the current decision-making stage and the previous decision-making stage. ;

[0112] 6: Determine if the local iteration has converged, i.e., if ,but, ;like ,but, ;

[0113] 7: Output: , , , .

[0114] Algorithm 2 describes the interaction process between the player and the environment, including state transitions, evaluation metric calculation, reward decision-making, and state similarity calculation. As a similarity measure, quantification is used in Players in turn state and The differences between them.

[0115] Algorithm 3: Group Coverage Control Algorithm Based on TD3

[0116] 1: Enter the game situation for the current round. ;

[0117] 2: The decision-making phase cycle begins ( ):

[0118] a) Select an action with exploration noise. And execute Algorithm 2 to obtain the reward. Next stage status Energy consumption and local state similarity Calculate remaining energy consumption ;

[0119] b) Determine if the local cutoff condition is met. If it is met, i.e. Then, record the local convergence time as... (Decision-making stage threshold), store experience tuples In the experience pool middle;

[0120] c) From the experience pool Small to medium batches of randomly extracted empirical tuples, with a cardinality of . ;

[0121] d) Target actor network selection action

[0122] ;

[0123] e) Calculate the target value Update the two critic network parameters based on the formula. ;

[0124] f) Determine if the remainder of the quotient of the decision time base and the target policy delay frequency is zero. If yes, update the actor network parameters according to the following formula:

[0125] ;

[0126] g) Update the target network parameters as follows:

[0127] ;

[0128] 3: Output the dominance matrix and .

[0129] Algorithm 3 is pseudocode designed according to the content and application background of this invention. Based on the extended TD3 framework, this algorithm integrates the generation, utilization, and storage of energy reserves. The study introduces an assessment of game situation similarity and strengthens the environmental interaction model to maintain consistency with the present invention.

[0130] like Figure 2 As shown, in one specific embodiment of the present invention, starting with the unknown sea area of ​​the typhoon's movement trajectory, multiple groups of unmanned surface vessels (USVs) work together to detect and provide early warning coverage, the simulation example is first described and then the execution flow of the method is introduced in detail.

[0131] Taking an unknown sea area with a typhoon's trajectory as an example, this invention uses simulation to evaluate the performance of the proposed method. Multi-layer semi-distributed game-based reinforcement learning is implemented on a Windows 11 platform using PyTorch 1.13.1 and Python 3.7.12. The simulation scenario involves five unmanned surface vessels collaboratively covering a 20×20 area on a sea surface. The hyperparameters selected for this simulation are as follows: , , , , In summary, all characteristics of each unmanned surface vessel are randomly selected, including its effective sensing range. Communication distance Peak sensing quality Peak communication intensity Safety threshold Energy storage reserves .in, , , , Features in Random selection of intervals Features in Random selection of intervals Features in The interval is randomly selected. The remaining parameters are set as follows: , , , , , , When vector gains , The weights are set to For clarity, we chose to show the first 6 rounds (randomly selected). The simulation results, such as Figure 3 As shown.

[0132] according to Figure 3 It is evident that the generation strategy of the multi-layer semi-distributed reinforcement learning method enables... The rewards steadily increase as the rounds progress. Meanwhile, from Figure 4 It can be observed that within the same round, the payoff error of each individual remains within the TD3 objective smoothing constraint range, indicating that the system has reached an equilibrium state. Combined with... Figure 3 and Figure 4 The results lead to the conclusion that, based on the proposed method, the unmanned surface vessel (USV) swarm achieves Pareto optimal equilibrium in each round under the guidance of a multi-objective group game model. Adjusting the characteristics and parameters of the USVs... It can be used to analyze and explain the cooperative and competitive relationships of unmanned surface vessels in swarm coverage, providing important guidance for practical applications.

[0133] The simulation examples above effectively demonstrate the effectiveness and accuracy of this invention. The following will describe in detail the method execution process, specifically the process of calculating the Pareto Nash equilibrium solution in a group game. The method execution steps are as follows:

[0134] Step 1: Based on the optional pure strategies of the unmanned surface vessels (USVs), namely speed and angle, initialize the probability vector of each USV selecting the elements of the pure strategy set to obtain the hybrid strategy space. Each unmanned surface vessel selects a strategy from its hybrid strategy space. Forming a game situation .

[0135] Step 2: Establish a one-to-one correspondence between the game situation and state, strategy and action; design the action-state value function; design the individual and collective benefit equilibrium reward function; design the strategy iteration and evaluation mechanism; design the action-state value network and policy network structure, as well as the corresponding optimizer and activation function.

[0136] Step 3: Execute the multi-layer semi-distributed reinforcement learning Pareto Nash equilibrium solution algorithm, i.e., Algorithm 1. Using the TD3 model as an independent unmanned surface vessel (USV), each USV interacts with its environment in a distributed manner during the decision-making phase loop. This involves executing Algorithm 3 to update its own policy until the local cutoff condition is met or the interaction reaches a certain number of iterations. The game situation is iterated cyclically. After the decision-making phase loop ends under the current game situation, the output result is used to update the game situation at the next moment. This involves centralized recording and calculation of the group policy, which is then fed back to individual USVs for policy updates. This iteration continues until the game situation stabilizes, i.e., the global cutoff condition is met or the maximum number of iterations is reached.

[0137] The cyclical process of executing Algorithm 1 is as follows:

[0138] a) Initialization phase: Each unmanned surface vessel randomly selects a strategy from its available mixed strategy space to engage in game interaction and records the interaction results.

[0139] b) Design a global game situation similarity index, which is the end marker of the current game interaction round.

[0140] c) In the current game cycle, until the end of the game interaction cycle is reached, the game situation is updated after each decision phase cycle, and the global similarity of the game situation is calculated. If the global end condition is met, the next game round begins; otherwise, the iteration continues.

[0141] d) At the end of the current cycle, update the game situation again and pass it on to the next game interaction round.

[0142] The decision-making phase loop of Algorithm 3 is as follows:

[0143] a) Initialize the parameters of the policy iteration algorithm, including the target policy delay update frequency, target network parameters, two evaluation network parameters, action selection network parameters, sample replay space, maximum number of interactions with the environment, target policy smoothing parameters, learning rate, and number of game interactions.

[0144] (b) A random game situation is introduced in the initial stage; the game situation from the previous time step is introduced in the non-initial stage. In the current strategy selection stage, the unmanned surface vessel (USV) selects an action based on its current behavioral strategy. Specifically, the USV executes its current hybrid strategy to interact with the environment, that is, to engage in a game with other prisoners, obtaining the reward for the current time step and the state for the next time step. Subsequently, the USV's hybrid strategy from the previous time step is replaced with its current hybrid strategy, while keeping the hybrid strategies of other USVs unchanged.

[0145] c) Randomly sample small batches of data from the sample replay space to learn and update the parameters of the evaluation network and the action selection network, in order to further evaluate and iterate the behavioral strategy.

[0146] d) Determine whether a Pareto Nash equilibrium or the maximum number of game interactions has been reached. If the conditions are not met, repeat steps a) to c); otherwise, output the game situation for the current round.

[0147] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0148] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0149] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0150] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0151] The above description is merely a preferred embodiment of the present invention. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make many possible variations and modifications to the technical solutions of the present invention using the methods and techniques disclosed above, or modify them into equivalent embodiments with equivalent changes, without departing from the scope of the technical solutions of the present invention. Therefore, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solutions of the present invention shall still fall within the protection scope of the technical solutions of the present invention.

Claims

1. A multi-layered game-theoretic reinforcement learning method for regional coverage control of intelligent groups, characterized in that, Includes the following steps: S1. For an intelligent group consisting of several intelligent agents, where the intelligent agents are drones, unmanned boats or other unmanned intelligent devices, each intelligent agent in the intelligent group is mapped to a game player, the strategy space and strategy combination of the intelligent agents are defined, and a benefit evaluation mechanism is constructed in combination with the task objectives of the intelligent agents to form a group game model. S2. Based on the physical capability constraints of the agent, construct a perception model and a communication model, design evaluation indicators covering performance, operational safety, energy efficiency and communication coordination, and integrate the evaluation indicators into a vector reward function for the agent; S3. The TD3 model is used as the game player role of the agent to realize the agent's policy learning, defining the learning state, action and reward, and establishing the connection between reinforcement learning and group game; S4. Construct a multi-layer semi-distributed game reinforcement learning framework, realize the dynamic interaction between the agent and the environment and between the agents through the environment interaction model, perform policy iterative optimization based on reinforcement learning, and combine the convergence judgment mechanism to solve the intelligent group regional coverage space control strategy that satisfies Pareto Nash equilibrium.

2. The multi-layer game-theoretic reinforcement learning method for intelligent group regional coverage control according to claim 1, characterized in that, Step S1 specifically includes: Setting by A group game system consisting of 10 players, where each player corresponds to an agent in the intelligent swarm. , ; Define each intelligent agent Limited strategy space and hybrid strategy space ,in It is the cardinality of the strategy space; Define the Cartesian product of the mixed strategy space for all players as: By each intelligent agent Hybrid strategy Combining to form a game situation ; Define each intelligent agent target set ,in It is the cardinality of the target set, each target This corresponds to a task requirement of the intelligent agent; Building intelligent agents Vector profit function Each element of the vector profit function corresponds to the target set. The payoff value of one of the objectives, and the payoff value varies with the game situation. Dynamically adjust to changes; The group game model is composed of the number of players, the strategy space, the mixed strategy space, the vector payoff function, and the objective set. .

3. The multi-layer game-theoretic reinforcement learning method for intelligent group regional coverage control according to claim 1, characterized in that, The physical capability constraints in step S2 include: speed not exceeding the rated speed, acceleration not exceeding the rated acceleration, heading angle within the rated effective range, and having fixed energy reserves, maximum sensing range, and maximum communication range.

4. The multi-layer game-theoretic reinforcement learning method for intelligent group regional coverage control according to claim 1, characterized in that, The perception model construction process in step S2 includes: defining the perception area of ​​the agent as a circular area with its own position as the center and the maximum perception distance as the radius. Within the perception area, the probability of the agent detecting any detection point decreases linearly with the increase of distance. Based on the radius constraint and the linear decay law, a perception model is constructed to quantify the agent's ability to perceive detection points within the area. The process of constructing the communication model includes: defining the communication region of an agent as a circular region with its own position as the center and the maximum communication distance as the radius. Within the communication region, the level of communication interaction between the agent and other agents decreases linearly with increasing distance. Based on the radius constraint and the linear decay law, a communication model is constructed to quantify the communication capabilities between agents.

5. The multi-layer game-theoretic reinforcement learning method for intelligent group regional coverage control according to claim 1, characterized in that, The evaluation indicators for design coverage performance, operational safety, energy efficiency, and communication coordination in step S2 specifically include: Design coverage effectiveness evaluation index: For any monitoring point within the agent's perception area, calculate the cumulative coverage level of the monitoring point by the agent from the initial time to the current time. If the cumulative coverage level exceeds a preset effective coverage threshold, the monitoring point is determined to have achieved effective coverage. Calculate the area of ​​monitoring points within the agent's perception range that have achieved effective coverage at the current time, and use this as the individual effective coverage evaluation function. ;Calculate the area of ​​monitoring points in the target region of the task within the task time that are effectively covered by at least one agent, and use this as the group effective coverage evaluation function. ; Design and implement security evaluation metrics: Construct a density evaluation function based on kernel function, bandwidth parameter, and security thresholds between agents. The density evaluation function quantifies the crowding level among agents to avoid collisions and excessive aggregation. Design an energy efficiency evaluation index: Construct an individual energy consumption evaluation function based on the agent's step size over a period of time. Construct a residual energy evaluation function based on the agent's total energy reserves and individual cumulative energy consumption. When the remaining energy is lower than a preset energy reserve threshold, the agent stops moving and disconnects from the communication network. Design a communication coordination evaluation index: An undirected graph is used to describe the communication topology of the group system. For any decision time, the undirected graph contains a set of nodes consisting of the positions of all agents and a set of edges formed by pairs of agents capable of communication. The connectivity of the undirected graph is determined by calculating the second smallest eigenvalue of its Laplacian matrix. A communication evaluation function is constructed based on the connectivity of the undirected graph. It is used to quantify the communication cooperation of intelligent agents.

6. The multi-layer game-theoretic reinforcement learning method for intelligent group regional coverage control according to claim 5, characterized in that, The individual effective coverage evaluation function Effective Coverage Evaluation Function Density evaluation function Individual energy consumption evaluation function Residual energy evaluation function and communication evaluation function Together they form a vector benefit function .

7. The multi-layer game-theoretic reinforcement learning method for intelligent group regional coverage control according to claim 1, characterized in that, Step S3 specifically includes: Using the TD3 model as the player role of the agent in the game, the group game model constructed in step S1 and the vector payoff function obtained in step S2 are integrated to establish the correspondence between the TD3 model and the group game, and the state of the agent in each decision stage is defined. ,action and rewards ; The state It consists of the individual strategies of the agent itself and the collective strategies of all other agents, and the state also serves as the game situation in the decision-making stage. The action It consists of two parts: a hybrid strategy for the agent regarding step size and a hybrid strategy regarding heading angle. The reward It consists of the agent's vector reward function and the target preference weight vector of the vector reward function, and the target preference weight vector is selected according to the task requirements.

8. The multi-layer game-theoretic reinforcement learning method for intelligent group regional coverage control according to claim 1, characterized in that, The multi-layer semi-distributed game reinforcement learning process in step S4 includes t learning rounds, each learning round includes e learning cycles, and each learning cycle includes k decision stages. The specific steps of the multi-layer semi-distributed game reinforcement learning process include: Initialize the network parameters and network structure of the TD3 model for each agent, including the parameters of the critic evaluation network and the actor decision network, and set the target policy delay frequency, local cutoff condition, global cutoff condition, action noise, and target preference weight vector for each agent. Perform learning round iterations: In each round, reset or adjust the environmental scene parameters, and then perform learning cycle iterations; Execute learning cycle iteration: In each cycle, the decision-making stage iteration is executed first. After all agents have completed the decision-making stage iteration, the similarity between the current cycle and the previous cycle's group game situation is calculated. If the similarity satisfies the global cutoff condition, the group strategy is determined to be globally converged, the current cycle iteration is terminated and the next round is entered; otherwise, the decision-making stage iteration is executed again. Decision-making phase iteration: In each decision-making phase, the agent selects actions with exploratory noise based on the TD3 model, obtains immediate rewards, the next state, and remaining energy through the environmental interaction model, and stores the experience tuple composed of the current state, action, reward, next state, and remaining energy into the experience pool; samples are randomly drawn from the experience pool to update the network parameters of the TD3 model; when the similarity of the states of two consecutive decision-making phases meets the local cutoff condition, the agent's policy is determined to be locally converged, and the current decision-making phase iteration is terminated; after all agents have completed the decision-making phase iteration, the process returns to the periodic iteration level for global convergence judgment.

9. The multi-layer game-theoretic reinforcement learning method for intelligent group regional coverage control according to claim 8, characterized in that, Each agent constructs an environment interaction model, which can update the state and position of the next decision stage based on the individual strategy of the agent and the collective strategy of other agents, calculate the vector reward of the agent, obtain the immediate reward by weighted summation of the vector rewards based on preference weights, and calculate the similarity between the state of the current decision stage and the previous decision stage.

10. The multi-layer game-theoretic reinforcement learning method for intelligent group regional coverage control according to claim 8, characterized in that, After all learning rounds have been iterated, the output group policy is the Pareto Nash equilibrium solution, which satisfies the following: each agent cannot improve its individual reward by adjusting its own policy alone, nor can it increase any of its own objective rewards without reducing the rewards of other objective rewards.

Images