An agent cluster pursuit-evasion game method based on hierarchical reinforcement learning
By employing a hierarchical reinforcement learning framework and adaptive methods, the problems of decision space complexity and uncertainty in intelligent agent swarm pursuit are solved, achieving efficient and robust decision-making in complex environments.
Patent Information
- Application Number
- CN202610484178.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-04-14
- Publication Date
- 2026-07-24
AI Technical Summary
Traditional agent swarm pursuit methods struggle to guarantee optimal decision-making in complex adversarial environments. Deep reinforcement learning suffers from insufficient interpretability and robustness in decision space during training in large-scale swarms, while hierarchical reinforcement learning exhibits unstable network training in high-uncertainty scenarios.
A hierarchical reinforcement learning framework is adopted, which maps target assignment and path planning to discrete and continuous networks respectively. A probabilistic ensemble network is embedded to quantify uncertainty, and independent pre-training and cross-training are used to promote network adaptive interaction and enhance robustness.
It maintains high robustness in high uncertainty scenarios, enables effective decision-making in intelligent agent cluster pursuit and escape game, and improves the decision-making efficiency and accuracy of cluster pursuit and escape tasks.
Smart Images

Figure CN122452745A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of swarm intelligence control technology, and in particular relates to a method for intelligent agent swarm pursuit and escape game based on hierarchical reinforcement learning. Background Technology
[0002] With the rise of artificial intelligence, agent control technology is receiving increasing attention. Swarm pursuit is a significant application of agent swarms, where agents are expected to achieve victory through AI-driven decision-making. Typical application scenarios include pursuit-escape and defense-attack games. Traditional expert system methods utilize prior knowledge from human experts to build system models and select strategies from a knowledge base using fuzzy matching. This approach relies heavily on rules established by numerous human experts, failing to guarantee optimal decision-making in complex adversarial environments. In game theory, the swarm pursuit problem is often modeled as a differential game. However, due to the large number of state variables and the complexity of differential equations, game theory is difficult to apply to complex multi-agent environments. Furthermore, heuristic methods model swarms as bio-inspired networks to simulate the dynamics of swarm pursuit, which holds greater potential for solving large-scale adversarial problems. However, the simulation and testing of this method require substantial computational resources and time. Due to the large number of agents involved and the high degree of conflict uncertainty, the intrinsic mechanism of swarm pursuit is an NP-hard complex problem. As the number of agents and actions increases, traditional algorithms struggle in terms of computational complexity and resource costs.
[0003] Deep reinforcement learning is a viable solution to this problem. It employs an end-to-end framework to approach optimal decisions rather than performing numerous iterations, thus achieving significant success in fields such as game theory, natural language processing, and robotics. Driven by maximizing the cumulative value of the reward function, deep reinforcement learning generates ideal policies by optimizing the decision network. This method learns rules from a large number of problem instances rather than manually designing them. However, directly using an end-to-end approach in swarm chase scenarios leads to the uninterpretability of the mixed decision space, hindering the training of deep reinforcement learning in large-scale swarms.
[0004] Hierarchical reinforcement learning methods can map swarm pursuit into a divide-and-conquer framework, treating instructions and actions as discrete and continuous two-layer decision networks, respectively. The upper layer divides the timeline into several non-uniform parts, each with a unique sub-objective and reward designed for the lower layer network during training, thereby maximizing the global reward. However, this open-loop structure, once the upper layer designs the sub-objective and reward, does not consider the performance of the lower layer, resulting in limited applicability. To achieve closed-loop functionality, many improved methods introduce feedback rewards, where the upper layer adjusts its policy based on the distance between the lower layer and the sub-objective. However, due to the inherent characteristics of hierarchical networks, especially in high-uncertainty scenarios like swarm pursuit, their robustness is difficult to guarantee when training two layers simultaneously. Summary of the Invention
[0005] To address the shortcomings of existing technologies, this invention proposes a hierarchical reinforcement learning-based method for agent swarm pursuit and escape game theory. This method addresses the problem of a complex decision space in swarm pursuit scenarios by mapping discrete instructions and continuous actions to target allocation and path planning, respectively. A novel hierarchical reinforcement learning framework is then proposed, including instruction networks and action networks for target allocation and path planning. To address the uncertainty caused by adversary policies, moving obstacles, and insufficient training, a robust adaptive method is proposed. This method embeds a probabilistic ensemble network between the instruction and action networks to quantify uncertainty and adaptively adjusts the interaction frequency of the two networks based on the uncertainty. To address the instability of traditional training methods, a fusion training method is proposed. This method performs independent pre-training of the instruction and action networks and cross-trains these two networks to promote an adaptive interaction mechanism.
[0006] The hierarchical reinforcement learning-based intelligent agent cluster chase-escape game method of the present invention enables the hierarchical reinforcement learning method to be applied to the cluster chase-escape problem and maintains high robustness in high uncertainty scenarios.
[0007] The specific technical solution of the present invention is as follows: A hierarchical reinforcement learning-based agent swarm chase game method includes the following steps: Step S1: Establish an intelligent agent cluster pursuit and escape scenario model and an intelligent agent cluster pursuit and escape decision model. The intelligent agent cluster pursuit and escape decision model includes a target allocation network, a path planning network, and an environmental situation network. Step S2: The target allocation network is used to allocate targets based on the current adversarial pursuit state of the pursuer. The reward function provides a static allocation reward based on the current adversarial pursuit state of the pursuer and the target allocation action. Step S3: The current chaser's adversarial pursuit state, target assignment action, and static reward assignment constitute a sample tuple, and the sample tuple is stored in the first experience replay pool; Step S4: Select the next pursuer as the current pursuer, and determine whether the number of sample tuples in the first experience replay pool has reached the preset threshold. If it has not reached the threshold, return to step S2; if it has reached the threshold, sample a portion of the sample tuples from the first experience replay pool, calculate the cost function of the target allocation network, and then update the gradient of the target allocation network according to the cost function. Step S5: Return to step S2 until the training of the target assignment network is completed; Step S6: The path planning network performs path planning based on the current pursuer's local observations and random target points. The reward function provides a static planning reward for the current pursuer's local observations and path planning actions. Step S7: The sample tuple is composed of the local observations, path planning actions and static planning rewards of the current chaser, and the sample tuple is stored in the second experience replay pool; Step S8: Select the next chaser as the current chaser, and determine whether the number of sample tuples in the second experience replay pool has reached the preset threshold. If not, return to step S6; if so, sample some sample tuples from the second experience replay pool, calculate the cost function of the path planning network, and then update the gradient of the path planning network according to the cost function. Step S9: Return to step S6 until the path planning network has been trained for the specified number of rounds. Step S10: The environmental situation network quantifies the uncertainty in the adversarial environment based on the current adversarial pursuit state of the pursuer, and calculates the interval step size of the path planning based on the uncertainty. Step S11: The target allocation network allocates targets based on the current adversarial pursuit state of the pursuer, and the reward function provides a static allocation reward based on the current adversarial pursuit state and the target allocation action. Step S12: The path planning network performs path planning based on the local observations of the current pursuer and the target allocation action obtained in step S11. The reward function provides a static planning reward for the local observations and path planning action of the current pursuer. Step S13: The sample tuple is composed of the local observations, path planning actions and static planning rewards of the current pursuer, and the sample tuple is stored in the third experience replay pool; Step S14: Select the next chaser as the current chaser, and determine whether the number of sample tuples in the third experience replay pool has reached the preset threshold. If it has not reached the threshold, return to step S12; if it has reached the threshold, sample some sample tuples from the third experience replay pool, calculate the cost function of the path planning network, and then update the gradient of the path planning network according to the cost function. Step S15: Determine whether the number of training iterations for the path planning network has reached the interval step size. If not, accumulate the static planning reward from step S12 to obtain the cumulative path reward and return to step S12. If yes, calculate the total allocation reward of the target allocation network based on the static allocation reward, cumulative path reward, and escape reward obtained in step S11. The escape reward is the number of escapees captured at present, and the cumulative path reward is cleared to zero. Step S16: The current chaser's adversarial state, target allocation action, and total allocation reward constitute a sample tuple, and the sample tuple is stored in the fourth experience replay pool. Step S17: Select the next pursuer as the current pursuer, and determine whether the number of sample tuples in the fourth experience replay pool has reached the preset threshold. If it has not reached the threshold, return to step S10. If it has reached the threshold, sample some sample tuples from the fourth experience replay pool, calculate the cost function of the target allocation network and the environmental situation network, and then update the gradient of the target allocation network and the environmental situation network according to the cost function. Step S18: Return to step S10 until the training of the environmental situation network reaches the specified number of rounds. Step S19: Apply the trained agent cluster pursuit and escape decision model to the agent cluster pursuit and escape decision.
[0008] Preferably, the intelligent agent cluster pursuit and escape scenario model in step S1 is a pursuit-escape game scenario with dynamic obstacles. Multiple pursuers jointly guard the target point and capture multiple escapees without colliding with each other or with the dynamic obstacles. Multiple escapees reach the target point without being captured, and the maximum speed amplitude of the escapees is greater than that of the pursuers.
[0009] Preferably, the counter-pursuit state is represented as follows:
[0010] in, Assign a matrix to the target of all pursuers. For the pursuers i Location coordinates, For the pursuers i speed, For the pursuers i The capture radius, For the pursuers i The maximum speed amplitude value.
[0011] Preferably, the target allocation action is represented as:
[0012] in, For the pursuers jLocation coordinates, For the pursuers j position velocity, For the escapee j The maximum speed amplitude value.
[0013] Preferably, the statically allocated reward is represented as follows:
[0014] in, This is the weighting factor between local and global rewards. For local rewards, This is a global reward.
[0015] Preferably, the local observation is represented as:
[0016] in, For the pursuers j Location coordinates, For the pursuers i The coordinates of the nearest neighbor. The coordinates of the nearest obstacle. For the pursuers i The speed of the nearest neighbor, For the pursuers j position velocity, The speed of the nearest obstacle.
[0017] Preferably, the path planning action is represented as:
[0018] in, For the pursuers i The direction of velocity.
[0019] Preferably, the static programming reward is represented as:
[0020] in, This is the weighting coefficient between intrinsic reward and collision avoidance reward. As an intrinsic reward, A reward for avoiding collisions.
[0021] Preferably, step S10 specifically includes: Step S10-1: A virtual environment model was constructed. Its expression is:
[0022] in, and These are the predicted values of the virtual environment model for state transitions and rewards, respectively. Step S10-2: Using a ensemble neural network To quantify cognitive uncertainty and arbitrary uncertainty in virtual environments, The expression is as follows:
[0023] in and These are the mean and variance of a Gaussian distribution, respectively. The current state. The allocation result at the current moment Step S10-3: Train the ensemble neural network to maximize the expected likelihood value.
[0024] in For network output, To find the expected value of the function, For experience replay pool, Let V be the variance of the Gaussian distribution. The mean of a Gaussian distribution is given. Step S10-4: Virtual Environment Model The mean and variance are expressed as follows:
[0025] Step S10-5: Virtual Environment Model The uncertainty is set as the variance of the ensemble neural network, expressed as:
[0026] Step S10-6: Interval step size Represented as:
[0027] in, These are weighting coefficients. A baseline value set by humans.
[0028] Preferably, the total allocation reward in step S15 is represented as follows:
[0029] in, It is an extra reward equal to the number of escapees captured at the current time.
[0030] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. This invention proposes a hierarchical reinforcement learning-based method for intelligent agent swarm pursuit and escape game. Addressing the problem of complex decision spaces in swarm pursuit and escape scenarios, it proposes a hierarchical reinforcement learning framework, including discrete and continuous networks for target allocation and path planning. This framework maps discrete instructions and continuous actions to the target allocation network and path planning network, respectively.
[0031] 2. This invention proposes a hierarchical reinforcement learning-based agent swarm pursuit game method. Addressing the uncertainty caused by enemy strategies, moving obstacles, and insufficient training, it proposes a robust adaptive approach. This method embeds an environmental situation network between the target assignment network and the path planning network to quantify uncertainty, and adaptively adjusts the interaction frequency of the two networks based on the uncertainty.
[0032] 3. This invention proposes a hierarchical reinforcement learning-based method for agent swarm pursuit and escape game theory. Addressing the instability of traditional training methods, it proposes a fusion training approach. This method independently pre-trains the target allocation network and the path planning network, and then cross-trains these two networks to promote an adaptive interaction mechanism. This approach enables hierarchical reinforcement learning to be applied to swarm pursuit and escape problems while maintaining high robustness in high-uncertainty scenarios. Attached Figure Description
[0033] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the embodiments will be briefly introduced below. The features and advantages of the present invention can be more clearly understood by referring to the accompanying drawings. The accompanying drawings are schematic and should not be construed as limiting the present invention in any way. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0034] Figure 1 This is a flowchart of an intelligent agent cluster chase-escape game method based on hierarchical reinforcement learning proposed in this invention.
[0035] Figure 2 This is a schematic diagram of a scenario model of intelligent agent cluster pursuit and escape. Detailed Implementation
[0036] To better understand the above-mentioned objectives, features, and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments of the present invention and the features thereof can be combined with each other.
[0037] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and therefore the scope of protection of the invention is not limited to the specific embodiments disclosed below.
[0038] This invention addresses the problem of complex decision spaces in swarm pursuit scenarios, including uncertainties caused by enemy strategies, moving obstacles, and insufficient training, as well as the instability of traditional training methods. It proposes a hierarchical reinforcement learning-based agent swarm pursuit game approach. Figure 1 As shown, the specific steps include: Step S1: Establish an intelligent agent cluster pursuit and escape scenario model and an intelligent agent cluster pursuit and escape decision model. The intelligent agent cluster pursuit and escape decision model includes a target allocation network, a path planning network, and an environmental situation network.
[0039] This invention models the intelligent agent swarm pursuit scenario as a pursuit-escape game with dynamic obstacles. Multiple pursuers, each with varying abilities, jointly defend a pre-defined target point. The pursuers must cooperate to capture the escapee as quickly as possible while avoiding collisions with obstacles and neighbors. The escapee must reach the pre-defined target point as quickly as possible without being caught by the pursuers, such as... Figure 2 As shown in the image. The left side represents the pursuer, and the right side represents the escapee. , and Let represent the length and width of the battlefield area, and the straight-line distance between the starting areas of both sides, respectively. To simplify the movement of the agents, a chase-escape game is considered in a two-dimensional scene. The position coordinates and velocities in the two dimensions are represented by ... and This indicates that neither the pursuer nor the escapee can exceed the boundaries of the scene or the maximum speed limit. . This represents the Euclidean standard.
[0040] and These represent a group consisting of pursuers and escapees. Both pursuers and escapees must avoid dynamic obstacles. For any pursuer , For the pursuers i Location coordinates, For the pursuers i speed, For the pursuers i The capture radius, For the pursuers i The maximum velocity amplitude. For any escapee. , For the pursuersj Location coordinates, For the pursuers j position velocity, For the escapee j Maximum speed amplitude value, Treat the agent and obstacles as having radii of... and The invention employs a circular model. A hierarchical reinforcement learning method is designed for the pursuer, while an artificial potential field method is used for the escapee. The hierarchical reinforcement learning method, through a divide-and-conquer framework, reflects the instructions and actions of the pursuit-escape game separately in target allocation and path planning. This can be used... To represent the result of target allocation, it indicates whether or not the target will be allocated. Assigned to Therefore, we can conclude that...
[0041] This represents the target assignment matrix for all pursuers. Subsequently, the pursuers chase the assigned targets through real-time path planning. Throughout the process, target assignment and path planning are dynamically alternated.
[0042] The following rules were designed for both the pursuer and the escapee: (1) Each pursuer can only choose one escapee, and an escapee can be assigned to multiple pursuers.
[0043] (2) If an escapee enters the capture radius of one of the pursuers, then it is captured.
[0044] (3) Neither the escapee nor the pursuer that satisfies rule (2) can continue to move.
[0045] (4) The conditions for the pursuer to win: the escapee fails to reach the preset target point within the specified time, or more than half of the escapees are captured.
[0046] (5) The conditions for the escapees to win: more than half of the escapees can reach the preset target point within the specified time without being caught by the pursuers.
[0047] Rule (3) states that a chaser can catch at most one escapee. In some embodiments, the number of chasers and escapees can be set to be the same. If there are fewer chasers than escapees, the chaser cannot catch all the escapees. Otherwise, if there are more chasers than escapees, the chaser does not need to devise a strategy to catch as many escapees as possible.
[0048] Markov decision process: It represents the set of real numbers. The mathematical expectation is represented. A Markov Decision Process (MDP) is used to simulate reinforcement learning. and These represent the environmental state and the agent's actions, respectively. Represents the probability of state transition. Represents the reward function, This represents the discount factor. MDP refers to the case where only a single agent or the system is considered a centralized agent. A fully cooperative multi-agent reinforcement learning task can be described as a decentralized partially observable Markov decision process (Dec-POMDP). A Dec-POMDP is represented as a six-tuple... The state space State transition function Reward function and discount factor The representation method is the same as that of MDP. For each agent... , and These represent the actions of each agent and the set of joint actions of all agents, respectively. and These represent the sets of observations for each agent and the sets of joint observations from all agents, respectively. It represents the trajectory of each intelligent agent after interacting with the environment. and Each represents a step in the decision-making process. The actions chosen and the observed values achieved. The goal of each agent is to optimize the policy network. Make cumulative rewards The maximum value is reached under this strategy.
[0049] Step S2: The target allocation network is used to allocate targets based on the current chaser's adversarial state. The reward function provides a static allocation reward based on the current chaser's adversarial state and target allocation action.
[0050] The current pursuit and escape status of the pursuers mainly includes the target allocation results of all pursuers. And the current pursuers that need to be assigned The information. It can be represented as
[0051] The target assignment action for the current pursuer is represented as:
[0052] The target allocation process involves each pursuer selecting an escapee. Once one pursuer has made their selection, the next pursuer will reselect based on the current state.
[0053] in, This indicates interaction with the environment.
[0054] The design process for statically allocated rewards is as follows. (The Chaser) Compared to the escapee The capture probability can be evaluated by the distance between them and the capture radius:
[0055] If multiple pursuers are assigned to the same escapee Then the joint capture probability is ,
[0056] The capture value of an escapee is related to its maximum speed; therefore, the objective of target assignment can be set to maximize the following efficiency function:
[0057] Indicates the pursuer Assigned to escapees The target allocation matrix is then obtained. Local rewards within the allocation can be calculated as follows:
[0058] Once all pursuers have completed their allocation, they receive the global reward from that allocation, calculated as follows:
[0059] in This represents the final target allocation matrix. The static allocation reward, a linear operation consisting of local and global allocation rewards, can be represented as follows:
[0060] in, It is the weighting coefficient between local rewards and global rewards.
[0061] Step S3: The current pursuer's adversarial pursuit state, target allocation action, and static allocation reward constitute a sample tuple, and the sample tuple is stored in the first experience replay pool.
[0062]
[0063] in, It is the first experience replay pool.
[0064] Step S4: Select the next pursuer as the current pursuer, and determine whether the number of sample tuples in the first experience replay pool has reached the preset threshold. If it has not reached the threshold, return to step S2; if it has reached the threshold, sample some sample tuples from the first experience replay pool, calculate the cost function of the target allocation network, and then update the gradient of the target allocation network according to the cost function.
[0065] Based on the above MDP settings, the DQN method is used to train the target assignment network. This method utilizes a parameterized neural network. To estimate the optimal state-action value function Subscript It is a weighting factor in the neural network. For , The discount payoff of the optimal strategy over an infinite range was estimated. This method uses... To approximate The loss function is set in the following form:
[0066] in yes —Target. Subscript It is a slowly moving online average, which avoids overestimation. During each iteration, it updates according to the following rules:
[0067] in It is a constant factor.
[0068] Step S5: Return to step S2 until the training of the target assignment network is completed.
[0069] Step S6: The path planning network performs path planning based on the current pursuer's local observations and random target points. The reward function provides a static planning reward for the current pursuer's local observations and path planning actions.
[0070] Considering the agent's rapid movement, instead of setting a fixed distance threshold for constructing the observation vector, it is assumed that the agent can observe its nearest neighbor. The current pursuer's local observations include information about the assigned escapee, its nearest neighbor, and obstacles. Therefore:
[0071] in, For the pursuers j Location coordinates, For the pursuers i The coordinates of the nearest neighbor. The coordinates of the nearest obstacle. For the pursuers i The speed of the nearest neighbor, For the pursuers j position velocity, The speed of the nearest obstacle.
[0072] The current pursuer's path planning actions include speed magnitude. and velocity direction , represented as:
[0073] Based on the current pursuit and escape situation and decisions of the pursuer, the position at the next moment can be calculated using a first-order dynamics model:
[0074] in It is the time step.
[0075]
[0076] in, This indicates interaction with the environment.
[0077] In order to intercept the assigned escapee, the pursuers During path planning, intrinsic rewards are derived from higher levels. These intrinsic rewards can be described as:
[0078] In path planning, the chaser, while pursuing its assigned target, needs to avoid collisions with neighbors and obstacles. The collision avoidance reward is set as follows:
[0079] If the pursuer Enter its nearest obstacle Threat areas ( Then there is
[0080] otherwise, .
[0081] If the pursuer Entering the threat zone of its nearest neighbor ( Then there is
[0082] otherwise, .
[0083] The reward is a constant. A set threat distance is used to avoid pursuers. Collisions with neighbors and obstacles occur. The final reward for path planning is a linear calculation consisting of intrinsic reward and collision avoidance reward, and its formula is as follows:
[0084] in, It is the weighting coefficient between intrinsic reward and collision avoidance reward.
[0085] Step S7: The sample tuple is composed of the local observations, path planning actions and static planning rewards of the current pursuer, and the sample tuple is stored in the second experience replay pool.
[0086]
[0087] in, It is the second experience replay pool.
[0088] Step S8: Select the next chaser as the current chaser, and determine whether the number of sample tuples in the second experience replay pool has reached the preset threshold. If it has not reached the threshold, return to step S6; if it has reached the threshold, sample some sample tuples from the second experience replay pool, calculate the cost function of the path planning network, and then update the gradient of the path planning network according to the cost function.
[0089] To enable the chasers to cooperate in interception while also making decisions based on local observations, the MADDPG algorithm is used. This is a method of centralized training and distributed execution. It designs a separate value network for each agent. Its update method is similar to DQN, as follows:
[0090] in In addition, each agent maintains a policy network. Its loss function is as follows:
[0091] Step S9: Return to step S6 until the path planning network has been trained for the specified number of rounds.
[0092] Step S10: The environmental situation network quantifies the uncertainties in the adversarial environment based on the current adversarial pursuit state of the pursuer, and calculates the interval step size for path planning based on the uncertainties. .
[0093] The goal assignment and path planning are decoupled into a two-layer network and unified into a dynamic cyclic process. During path planning, after... After a certain number of time steps, the target needs to be reassigned based on the current pursuit and escape status of the pursuer, where the interval step size is... It is a variable related to the current pursuer's adversarial pursuit state and target assignment actions. In this dynamic process, the upper layer assigns targets and provides intrinsic rewards to the lower layer, while the lower layer pursues the assigned target while avoiding obstacles through real-time path planning. To quantify uncertainty, including mutating opponent strategies and transient battlefield environments, a virtual environment model incorporating state transitions and reward functions is constructed. Its expression is
[0094] in, These represent the environmental model pairs. and The predicted value.
[0095] Using ensemble neural networks This invention quantifies cognitive uncertainty and arbitrary uncertainty in the environment. Cognitive uncertainty stems from insufficient training at lower levels, while arbitrary uncertainty refers to unknown enemy strategies and moving obstacles. The model takes state-action pairs as input and outputs a Gaussian distribution of the next state and reward. . It can be expressed in the following form:
[0096] in and Let be the mean and variance of a Gaussian distribution, respectively. The current state. This represents the allocation result at the current moment.
[0097] This transition dynamics model is trained to maximize the expected likelihood value.
[0098] in This represents the model output. For simplicity, it has been omitted. and $ Input Then the output of the sub-model is used to represent... The mean and variance are calculated using the following formulas:
[0099] The uncertainty can be set as the variance of the ensemble neural network. It is expressed as:
[0100] Unlike traditional fixed-step or infinite-iteration methods, an adaptive truncation method is used, which calculates the following linear operations. Expected value:
[0101] in, These are weighting coefficients. It is an integer, restricted to... Subscript , and These represent the base value, minimum value, and maximum value, respectively. ,in and Represents a set of integers. Arbitrary uncertainty cannot be reduced through training, but it can be quantized, thus adaptively adjusting the interaction frequency. When uncertainty... At higher levels, The smaller the value, the more frequently the upper level needs to allocate targets. When adapting to the environment, the cognitive uncertainty of the lower layers decreases, and the frequency of goal allocation also decreases.
[0102] Step S11: The target allocation network allocates targets based on the current chaser's adversarial state. The reward function provides a static allocation reward based on the current chaser's adversarial state and target allocation action.
[0103] This step is consistent with the process of step S2.
[0104] Step S12: The path planning network performs path planning based on the local observations of the current pursuer and the target allocation action obtained in step S11. The reward function provides a static planning reward for the local observations and path planning actions of the current pursuer.
[0105] This step is consistent with the process of step S6.
[0106] Step S13: The sample tuple is composed of the local observations, path planning actions and static planning rewards of the current pursuer, and the sample tuple is stored in the third experience replay pool.
[0107] Step S14: Select the next pursuer as the current pursuer, and determine whether the number of sample tuples in the third experience replay pool has reached the preset threshold. If it has not reached the threshold, return to step S12; if it has reached the threshold, sample some sample tuples from the third experience replay pool, calculate the cost function of the path planning network, and then update the gradient of the path planning network according to the cost function.
[0108] This step is consistent with the process of step S8.
[0109] Step S15: Determine whether the number of training iterations for the path planning network has reached the interval step size. If not, accumulate the static planning reward from step S12 to obtain the cumulative path reward and return to step S12. If yes, calculate the total allocation reward of the target allocation network based on the static allocation reward, cumulative path reward, and pursuit reward obtained in step S11. The pursuit reward is the number of escapees captured at present, and the cumulative path reward is cleared to zero.
[0110] Execute interval step size in the lower layer. Then, the accumulated rewards are fed back to the upper layer. Static allocation rewards and accumulated path rewards respectively encourage the agent to allocate reasonable goals and plan feasible paths. Therefore, the upper layer linearly weights these rewards as the total allocation reward for goal allocation, which can be expressed as:
[0111] in, This is an additional reward equal to the number of escapees captured in the current time. Based on the total reward allocation, the upper level will simultaneously weigh and update the rewards for the above three objectives.
[0112] Step S16: The current chaser's adversarial pursuit status, target allocation action, and total allocation reward constitute a sample tuple, and the sample tuple is stored in the fourth experience replay pool.
[0113]
[0114] in, It is an experience replay pool.
[0115] Step S17: Select the next pursuer as the current pursuer, and determine whether the number of sample tuples in the fourth experience replay pool has reached the preset threshold. If it has not reached the threshold, return to step S10; if it has reached the threshold, sample some sample tuples from the fourth experience replay pool, calculate the cost function of the target allocation network and the environmental situation network, and then update the gradient of the target allocation network and the environmental situation network according to the cost function.
[0116] This step is consistent with the process of step S4.
[0117] Step S18: Return to step S10 until the training of the environmental situation network reaches the specified number of rounds, and end the cross-training process of the hierarchical network.
[0118] Step S19: Apply the trained agent cluster pursuit and escape decision model to the agent cluster pursuit and escape decision.
[0119] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.
[0120] In this invention, unless otherwise explicitly specified and limited, "above" or "below" the second feature can include direct contact between the first and second features, or contact between the first and second features through another feature between them. Furthermore, "above," "over," and "on top" of the second feature includes the first feature directly above or diagonally above the second feature, or simply indicates that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature includes the first feature directly below or diagonally below the second feature, or simply indicates that the first feature is at a lower horizontal level than the second feature.
[0121] In this invention, the terms "first," "second," "third," and "fourth" are used for descriptive purposes only and should not be construed as indicating or implying relative importance. The term "multiple" refers to two or more unless otherwise expressly defined.
[0122] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A method for intelligent agent swarm pursuit and escape game based on hierarchical reinforcement learning, characterized in that, Includes the following steps: Step S1: Establish an intelligent agent cluster pursuit and escape scenario model and an intelligent agent cluster pursuit and escape decision model. The intelligent agent cluster pursuit and escape decision model includes a target allocation network, a path planning network, and an environmental situation network. Step S2: The target allocation network is used to allocate targets based on the current adversarial pursuit state of the pursuer. The reward function provides a static allocation reward based on the current adversarial pursuit state of the pursuer and the target allocation action. Step S3: The current chaser's adversarial pursuit state, target assignment action, and static reward assignment constitute a sample tuple, and the sample tuple is stored in the first experience replay pool; Step S4: Select the next pursuer as the current pursuer, and determine whether the number of sample tuples in the first experience replay pool has reached the preset threshold. If it has not reached the threshold, return to step S2; if it has reached the threshold, sample a portion of the sample tuples from the first experience replay pool, calculate the cost function of the target allocation network, and then update the gradient of the target allocation network according to the cost function. Step S5: Return to step S2 until the training of the target assignment network is completed; Step S6: The path planning network performs path planning based on the current pursuer's local observations and random target points. The reward function provides a static planning reward for the current pursuer's local observations and path planning actions. Step S7: The sample tuple is composed of the local observations, path planning actions and static planning rewards of the current chaser, and the sample tuple is stored in the second experience replay pool; Step S8: Select the next chaser as the current chaser, and determine whether the number of sample tuples in the second experience replay pool has reached the preset threshold. If not, return to step S6; if so, sample some sample tuples from the second experience replay pool, calculate the cost function of the path planning network, and then update the gradient of the path planning network according to the cost function. Step S9: Return to step S6 until the path planning network has been trained for the specified number of rounds. Step S10: The environmental situation network quantifies the uncertainty in the adversarial environment based on the current adversarial pursuit state of the pursuer, and calculates the interval step size of the path planning based on the uncertainty. Step S11: The target allocation network allocates targets based on the current adversarial pursuit state of the pursuer, and the reward function provides a static allocation reward based on the current adversarial pursuit state and the target allocation action. Step S12: The path planning network performs path planning based on the local observations of the current pursuer and the target allocation action obtained in step S11. The reward function provides a static planning reward for the local observations and path planning action of the current pursuer. Step S13: The sample tuple is composed of the local observations, path planning actions and static planning rewards of the current pursuer, and the sample tuple is stored in the third experience replay pool; Step S14: Select the next chaser as the current chaser, and determine whether the number of sample tuples in the third experience replay pool has reached the preset threshold. If it has not reached the threshold, return to step S12; if it has reached the threshold, sample some sample tuples from the third experience replay pool, calculate the cost function of the path planning network, and then update the gradient of the path planning network according to the cost function. Step S15: Determine whether the number of training iterations for the path planning network has reached the interval step size. If not, accumulate the static planning reward from step S12 to obtain the accumulated path reward, and return to step S12. If so, the total allocation reward of the target allocation network is calculated based on the static allocation reward, cumulative path reward and escape reward obtained in step S11, wherein the escape reward is the number of escapees captured at present, and the cumulative path reward is cleared to zero. Step S16: The current chaser's adversarial state, target allocation action, and total allocation reward constitute a sample tuple, and the sample tuple is stored in the fourth experience replay pool. Step S17: Select the next pursuer as the current pursuer, and determine whether the number of sample tuples in the fourth experience replay pool has reached the preset threshold. If it has not reached the threshold, return to step S10. If it has reached the threshold, sample some sample tuples from the fourth experience replay pool, calculate the cost function of the target allocation network and the environmental situation network, and then update the gradient of the target allocation network and the environmental situation network according to the cost function. Step S18: Return to step S10 until the training of the environmental situation network reaches the specified number of rounds. Step S19: Apply the trained agent cluster pursuit and escape decision model to the agent cluster pursuit and escape decision.
2. The agent swarm pursuit and escape game method based on hierarchical reinforcement learning as described in claim 1, characterized in that, The intelligent agent cluster pursuit and escape scenario model in step S1 is a pursuit-escape game scenario with dynamic obstacles. Multiple pursuers jointly guard the target point and capture multiple escapees without colliding with each other or with the dynamic obstacles. Multiple escapees reach the target point without being captured, with the escapee's maximum speed amplitude being greater than that of the pursuer.
3. The agent swarm pursuit and escape game method based on hierarchical reinforcement learning as described in claim 1, characterized in that, The state of resisting pursuit and escape is represented as follows: in, Assign a matrix to the target of all pursuers. For the pursuers i Location coordinates, For the pursuers i speed, For the pursuers i The capture radius, For the pursuers i The maximum speed amplitude value.
4. The agent swarm pursuit and escape game method based on hierarchical reinforcement learning as described in claim 1, characterized in that, The target allocation action is represented as follows: in, For the pursuers j Location coordinates, For the pursuers j position velocity, For the escapee j The maximum speed amplitude value.
5. The agent swarm pursuit and escape game method based on hierarchical reinforcement learning as described in claim 1, characterized in that, The statically allocated reward is represented as follows: in, This is the weighting factor between local and global rewards. For local rewards, This is a global reward.
6. The agent swarm pursuit and escape game method based on hierarchical reinforcement learning as described in claim 1, characterized in that, The local observation is represented as: in, For the pursuers j Location coordinates, For the pursuers i The coordinates of the nearest neighbor. The coordinates of the nearest obstacle. For the pursuers i The speed of the nearest neighbor, For the pursuers j position velocity, The speed of the nearest obstacle.
7. The agent swarm pursuit and escape game method based on hierarchical reinforcement learning as described in claim 1, characterized in that, The path planning action is represented as follows: in, For the pursuers i The direction of velocity.
8. The agent-based cluster chase-and-escape game method based on hierarchical reinforcement learning as described in claim 1, characterized in that, The static programming reward is represented as follows: in, This is the weighting coefficient between intrinsic reward and collision avoidance reward. As an intrinsic reward, A reward for avoiding collisions.
9. The agent swarm pursuit and escape game method based on hierarchical reinforcement learning as described in claim 1, characterized in that, Step S10 specifically includes: Step S10-1: A virtual environment model was constructed. Its expression is: in, and These are the predicted values of the virtual environment model for state transitions and rewards, respectively. Step S10-2: Using a ensemble neural network To quantify cognitive uncertainty and arbitrary uncertainty in virtual environments, The expression is as follows: in and These are the mean and variance of a Gaussian distribution, respectively. The current state. This represents the allocation result at the current moment; Step S10-3: Train the ensemble neural network to maximize the expected likelihood value. in For network output, To find the expected value of the function, For experience replay pool, Let V be the variance of the Gaussian distribution. The mean of a Gaussian distribution is given. Step S10-4: Virtual Environment Model The mean and variance are expressed as follows: Step S10-5: Virtual Environment Model The uncertainty is set as the variance of the ensemble neural network, expressed as: Step S10-6: Interval step size Represented as: in, These are weighting coefficients. A baseline value set by humans.
10. The agent swarm pursuit and escape game method based on hierarchical reinforcement learning as described in claim 1, characterized in that, The total allocated reward in step S15 is represented as follows: in, It is an extra reward equal to the number of escapees captured at the current time.
