Reinforcement learning based swarm cooperative hunting method

Abstract

The application belongs to the technical field of reinforcement learning, and specifically discloses a cluster cooperative hunting method based on reinforcement learning, which comprises the following steps: initializing a policy neural network of a hunting agent and an initial state of an attacking agent; after the hunting agent and the attacking agent respectively perform selected actions, determining a reward obtained by the hunting agent and updating an action value function and a state value function of the policy neural network based on the reward, so as to train the policy neural network; in the case that a target loss function of the policy neural network converges, migrating the policy neural network to a game confrontation scene for testing; in the case that the hunting agent based on the policy neural network meets the cluster cooperative hunting requirement in the scene testing, determining that the policy neural network realizes the cluster cooperative hunting of the hunting agent. The scheme solves the technical problem of solving the mixed Nash equilibrium strategy of both hunting parties under asymmetric conditions.

Description

Technical Field

[0001] This invention belongs to the technical field of reinforcement learning, specifically relating to a cluster collaborative hunting method based on reinforcement learning. Background Technology

[0002] A swarm is a collection of real objects or organisms, typically forming in nature or artificial environments. Common swarms include bee colonies, ant colonies, and cattle herds. By observing their behavioral patterns, researchers have found that individuals within a swarm can accomplish many complex tasks through collaborative cooperation. In recent years, swarm applications have become increasingly widespread, achieving remarkable results in numerous fields such as search and rescue, ground exploration, agricultural planting, and specialized logistics. In these complex scenarios, swarm agents, through division of labor and cooperation, have accomplished many tasks that single agents could not complete or could not accomplish effectively.

[0003] Regional target encirclement is a typical application scenario of swarm collaboration, originating from the prevalent predation and hunting behavior in nature. Over the years, researchers have extensively expanded the forms and objectives of encirclement. Among these, multi-target encirclement is an important aspect of confrontation and a hot topic in swarm collaborative encirclement. It can serve as a powerful means of capturing key enemy personnel or suppressing important enemy targets in confrontation.

[0004] When swarms execute cooperative hunting missions, the process can be summarized into two main phases: dispersion and encirclement. By dispersing to bypass the main battlefield and creating a favorable position, they then encircle the enemy, achieving a targeted strike against key enemy targets with minimal losses. In terms of the forms of cooperative action, swarm cooperative hunting missions can be categorized into problems such as cooperative formation, state consistency judgment, aggregation, envelope control, and cooperative detour hunting. Through the cooperation and collaboration among multiple agents within the swarm, the swarm can accomplish complex problems that are difficult for a single agent to handle. Most existing research on swarm cooperative hunting is based on distributed control, which transforms the swarm hunting problem into a consistency problem, and then designs distributed algorithms to make the swarm approach and converge towards the target location, achieving the effect of encircling and capturing the enemy target.

[0005] Recent research has focused on formation control, using methods such as virtual structure, behavioral methods, artificial potential fields, and graph theory to coordinate multiple agents in target hunting tasks. However, research on multi-target hunting based on these methods is still in its early stages, often requiring complex computations and control strategies. Moreover, many studies analyze multiple targets as a whole, with relatively few studies focusing on hunting multiple targets separately.

[0006] Some studies have optimized hunting tasks by decomposing hunting behavior, such as using loose preference rules to enable individual agents to spontaneously form ideal hunting formations, setting dynamic hunting points, calculating the optimal hunting angle, and proposing multi-layered ring ambush hunting models. However, these hunting methods are procedural hunting methods. In other words, although these methods do not require manual control, they require a lot of resources to remodel and solve the external environment model when faced with environmental changes and emergencies. This greatly affects the real-time performance and effectiveness of the agent's decision-making and makes it difficult to apply to multi-agent hunting tasks in unknown and complex environments.

[0007] The aforementioned algorithms generally do not consider the asymmetry in resources, detection information, and strategy levels between the hunting swarm and the enemy agent. They are also weak in handling dynamic environmental changes and unexpected situations, fail to utilize intelligent algorithms to optimize strategies, and struggle to achieve hunting effects when rewards are sparse. Therefore, they cannot promote the continuous improvement and enhancement of multi-agent collaborative hunting strategies. Summary of the Invention

[0008] The purpose of this invention is to provide a cluster cooperative hunting method based on reinforcement learning to solve the technical problem of solving the hybrid Nash equilibrium strategy of the hunting parties under asymmetric conditions.

[0009] To achieve the above objectives, the present invention adopts the following technical solution:

[0010] A cluster-based cooperative hunting method based on reinforcement learning includes: constructing a hunting agent and an attacking agent in a game-like adversarial scenario, and initializing the policy neural network of the hunting agent and the initial state of the attacking agent, wherein both the hunting agent and the attacking agent perform the task of searching for and eliminating the enemy; in Markov decision-making, using the policy function of the policy neural network to select actions for the hunting agent and selecting actions for the attacking agent based on a preset combat strategy; after the hunting agent and the attacking agent each execute their selected actions, determining the reward obtained by the hunting agent and updating the action value function and state value function of the policy neural network based on the reward to train the policy neural network; if the target loss function of the policy neural network converges, transferring the policy neural network to a game-like adversarial scenario for testing, wherein the target loss function is determined based on the policy function, the action value function, and the state value function; if the hunting agent meets the requirements of cluster-based cooperative hunting in the scenario test based on the policy neural network, it is determined that the policy neural network has achieved cluster-based cooperative hunting of the hunting agent.

[0011] Preferably, constructing the hunting agent includes: constructing a Markov model of the hunting agent, including the input state, output action, reward function, and discount coefficient of the hunting agent.

[0012] Preferably, constructing an offensive agent includes: constructing an offensive agent based on a preset combat strategy, and determining the input state of the offensive agent, wherein the input state of the offensive agent includes the position information of each offensive agent, the distance matrix between each offensive agent, the position information of all hunting agents detected by the offensive agent detector, and the distance matrix between each offensive agent and all hunting agents.

[0013] Preferably, initializing the policy neural network of the hunting agent includes: constructing a policy neural network including an input layer, a feature encoding module, an action generation module, and an output layer; and initializing the policy function, action value function, state value function, and target loss function of the policy neural network.

[0014] Preferably, the action selection of the hunting agent using the policy function of the aforementioned policy neural network and the action selection of the attacking agent based on a preset combat strategy include: determining the output action of the hunting agent using the aforementioned policy function based on the input state of the hunting agent, wherein the output action of the hunting agent indicates the target position of the hunting agent at the next moment; and determining the output action of the attacking agent using a preset comparison strategy based on the input state of the attacking agent, wherein the output action of the attacking agent indicates the target position of the attacking agent at the next moment.

[0015] Preferably, training the above-mentioned policy neural network includes: storing the current state, actions performed, traction rewards, and next state of the hunting agent as experience in a shared experience pool; when the amount of experience in the shared experience pool reaches a preset training amount, the hunting agent draws samples from the shared experience pool in batches for learning and updates the policy function, action value function, and state value function of the policy neural network.

[0016] Preferably, updating the policy function, action value function, and state value function of the policy neural network includes: determining the update magnitude of the policy neural network based on the current training effect; and updating the policy function, action value function, and state value function of the policy neural network according to the update magnitude.

[0017] Preferably, the hunting agent based on the above-mentioned strategy neural network meets the requirements of cluster collaborative hunting in the scenario test, including: when the hunting win rate of the hunting agent exceeds the win rate threshold, the loss rate of the hunting agent is lower than the loss threshold, and the hit rate of the hunting agent is higher than the hit threshold, it is determined that the hunting agent based on the above-mentioned strategy neural network meets the requirements of cluster collaborative hunting in the scenario test.

[0018] Preferably, after the hunting agent of the above-mentioned strategy neural network meets the requirements of cluster collaborative hunting in the scenario test, the method further includes: calculating the single-step calculation time of the hunting agent; and determining the hardware carrier parameters of the above-mentioned strategy neural network based on the single-step calculation time and the preset time requirement.

[0019] A cluster-based cooperative hunting system based on reinforcement learning includes: a construction initialization module, used to construct hunting agents and attacking agents in a game-like adversarial scenario, and initialize the policy neural network of the hunting agents and the initial state of the attacking agents, wherein both the hunting agents and the attacking agents perform the task of searching for and eliminating the enemy. The training module is used in Markov decision-making to select actions for the hunting agent using the policy function of the aforementioned policy neural network and to select actions for the attacking agent based on a preset combat strategy. After the hunting agent and the attacking agent each execute their selected actions, the reward obtained by the hunting agent is determined, and the action value function and state value function of the aforementioned policy neural network are updated based on the reward to train the aforementioned policy neural network. The testing module is used to transfer the aforementioned policy neural network to a game adversarial scenario for testing when the target loss function of the aforementioned policy neural network converges, wherein the target loss function is determined based on the aforementioned policy function, the aforementioned action value function, and the aforementioned state value function. The determination module is used to determine that the aforementioned policy neural network has achieved the cluster cooperative hunting of the aforementioned hunting agent when the hunting agent meets the requirements of cluster cooperative hunting in the scenario test.

[0020] A computer-readable storage medium storing a computer program configured to execute the aforementioned reinforcement learning-based cluster cooperative hunting method at runtime.

[0021] An electronic device includes a memory and a processor, wherein the memory stores a computer program and the processor is configured to execute the aforementioned reinforcement learning-based cluster cooperative hunting method through the computer program.

[0022] In this invention, a hunting agent and an attacking agent are constructed in a game-theoretic scenario, and the initial states of the hunting agent's policy neural network and the attacking agent's initial states are initialized. In Markov decision-making, the policy function of the policy neural network is used to select actions for the hunting agent, and the attacking agent's actions are selected based on a preset battle strategy. After each of the hunting and attacking agents executes its selected actions, the reward obtained by the hunting agent is determined, and the action value function and state value function of the policy neural network are updated based on the reward to train the policy neural network. When the objective loss function of the policy neural network converges, the policy neural network is transferred to a game-theoretic scenario for testing. Based on the fact that the hunting agent in the policy neural network meets the requirements of clustered collaborative hunting in the scenario test, it is determined that the policy neural network has achieved clustered collaborative hunting of the hunting agent. Thus, based on the Markov decision process, a formal representation of a zero-sum game model with a finite time-domain and finite state-action space under variable player sizes is provided. The focus is on reward changes during state transitions, effectively solving the reward sparsity problem in model training. This solves the technical problem of solving the mixed Nash equilibrium strategy for both hunting parties under asymmetric conditions, achieving the technical effect of ensuring the optimality of the decisions of both players. Attached Figure Description

[0023] Figure 1 This is a flowchart illustrating a cluster collaborative hunting method based on reinforcement learning in an embodiment of the present invention.

[0024] Figure 2 This is a flowchart of a cluster collaborative hunting method based on reinforcement learning in an embodiment of the present invention.

[0025] Figure 3 This is a simulation test scenario for a cluster collaborative hunting method based on reinforcement learning in an embodiment of the present invention;

[0026] Figure 4 This is a schematic diagram of a blue team rule example in an embodiment of a cluster collaborative hunting method based on reinforcement learning in this invention.

[0027] Figure 5 This is a diagram of the strategy neural network structure in an embodiment of a cluster collaborative hunting method based on reinforcement learning in this invention.

[0028] Figure 6 This is an embodiment of a reinforcement learning-based cluster collaborative hunting method, which describes the interaction and training process of the red and blue intelligence agents.

[0029] Figure 7 is a graph of key training parameters in an embodiment of a cluster collaborative hunting method based on reinforcement learning in this invention; actor_grad_norm is shown in Figure 7(a), average_episode_rewards is shown in Figure 7(b), value_loss is shown in Figure 7(c), and critic_grad_norm is shown in Figure 7(d).

[0030] Figure 8 This is a schematic diagram of the structure of a cluster collaborative hunting system based on reinforcement learning in an embodiment of the present invention. Detailed Implementation

[0031] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. The advantages and features of the present invention will become clearer from the following description and claims. It should be noted that the drawings are all in a very simplified form and are not to a precise scale, and are only used to facilitate and clarify the illustration of the embodiments of the present invention.

[0032] It should be noted that, in order to clearly illustrate the content of this invention, several embodiments are provided to further explain different implementations of the invention. These embodiments are enumerated rather than exhaustive. Furthermore, for the sake of brevity, content mentioned in the preceding embodiments is often omitted in the following embodiments. Therefore, content not mentioned in the later embodiments can be referred to in the preceding embodiments. Example 1

[0033] A cluster-based cooperative hunting method based on reinforcement learning, such as Figure 1 As shown, the method includes:

[0034] S102, Construct hunting agents and attacking agents in a game-playing scenario, and initialize the policy neural network of the hunting agent and the initial state of the attacking agent. Both the hunting agent and the attacking agent perform the task of searching for and destroying the enemy.

[0035] S104 In Markov decision-making, the policy function of the policy neural network is used to select the action of the hunting agent and the action of the attacking agent is selected based on the preset battle strategy. After the hunting agent and the attacking agent each execute their selected actions, the reward obtained by the hunting agent is determined and the action value function and state value function of the policy neural network are updated based on the reward in order to train the policy neural network.

[0036] S106, If the objective loss function of the policy neural network converges, the policy neural network is transferred to a game adversarial scenario for testing. The objective loss function is determined based on the policy function, action value function, and state value function.

[0037] S108, when the hunting agent based on the policy neural network meets the requirements of clustered collaborative hunting in the scenario test, it is determined that the policy neural network has realized the clustered collaborative hunting of the hunting agent.

[0038] As an optional implementation, constructing a hunting agent includes: constructing a Markov model of the hunting agent, including the input state, output action, reward function, and discount coefficient of the hunting agent.

[0039] Optionally, a Markov model of the hunting agent is constructed. ),in, The input state for the hunting agent. The output actions of the hunting agent. For the reward function of the hunting agent, This is the discount factor.

[0040] Combining information on the hunting agent and weapons, and information on the attacking agent and weapons, the input state of the hunting agent is not limited to: ;

[0041] in, The location information of each hunting agent is usually represented by latitude and longitude. This represents the distance matrix between the hunting agents. This indicates the number of weapons carried by each hunting agent. This represents the location information of all attacking agents that can be detected by the hunting agent detector. This represents the distance matrix between all hunting agents and all attacking agents. This indicates the information of the two weapons closest to the current hunting agent. The above information is not limited to the specific description: ;

[0042] ;

[0043] ;

[0044] ;

[0045] ;

[0046] .

[0047] Define each hunting agent. Guided reward function , represented as:

[0048]

[0049] in, Indicates hunting intelligent agents Rewards when in a hunting state This represents the reward for detecting an attacking enemy agent. This indicates the reward received when the device is destroyed. Indicates whether the reward is within the central area. Indicates a reward for destroying the enemy. This refers to the reward for approaching the enemy while in a hunting situation. This indicates the reward for the enemy entering the attack range. This represents the reward for being far from a specified dispersion reference point when in a dispersed state. This indicates a reward for dodging enemy weapons.

[0050] Not limited to setting a learning decay factor: This is used to accumulate the reward value throughout the entire game process. The higher the value, the more emphasis is placed on long-term returns.

[0051] As an optional implementation method, constructing an offensive agent includes: constructing an offensive agent based on a preset combat strategy, and determining the input state of the offensive agent, wherein the input state of the offensive agent includes the position information of each offensive agent, the distance matrix between each offensive agent, the position information of all hunting agents detected by the offensive agent detector, and the distance matrix between each offensive agent and all hunting agents.

[0052] Optionally, an offensive agent can be constructed based on a preset combat strategy. The offensive agent is not limited to having strong offensive capabilities, the ability to evade enemy weapons, and the ability to accelerate escape.

[0053] The situational input information for the attacking agent is set as follows: Combining information from both the attacking and hunting agents, the input state representation of the attacking agent is defined as follows:

[0054] in, This represents the latitude and longitude information of each attacking agent. This represents the distance matrix between attacking agents. This represents the latitude and longitude information of all hunting agents that the attacking agent can detect. This represents the distance matrix between all attacking agents and all hunting agents. The above information is not limited to the specific description: ;

[0055] ;

[0056] ;

[0057] .

[0058] As an optional implementation, initializing the policy neural network of the hunting agent includes: constructing a policy neural network including an input layer, a feature encoding module, an action generation module, and an output layer; and initializing the policy function, action value function, state value function, and target loss function of the policy neural network.

[0059] Optionally, a policy neural network is constructed for the red hunting agent. Specifically, it includes, but is not limited to, the input layer, feature encoding module, action generation module, and output layer, as well as the input state of the policy neural network. and output action The relationship is represented as:

[0060] in, It is Gaussian white noise, used to enhance the exploratory ability of decision-making during training.

[0061] action Based on the game-playing scenario, the specific action space is not limited to the following representation:

[0062] The action space is not limited to four basic actions: following the target, moving left relative to the target, moving right relative to the target, and moving in the opposite direction relative to the target. The size of the action space is determined by the number of agents on both sides. Each agent's actions include actions relative to each teammate. Relative to each enemy target and the closest weapon .

[0063] The action value function of the hunting agent is not limited to being expressed as:

[0064] in, For mathematical expectation, This represents the initial state of the agent. Indicates the initial action. It indicates that it follows a probability distribution Action value function, Let P represent the state generation function that follows a probability distribution P. Let Q represent the discounted reward function that satisfies the initial conditions. The action value function Q is the joint action value of all agents in each state s. Therefore, it is affected by the action chosen by each agent and requires comprehensive consideration of the action value and discounted reward at the target time.

[0065] The state-value function of the hunting agent is not limited to being expressed as:

[0066] in, For mathematical expectation, It indicates that it follows a probability distribution Action value function, Let P represent the state generation function that follows a probability distribution P. Indicates the discount rate The reward function, This represents the initial state of the agent.

[0067] The state value function comprehensively considers the state of each decision agent at the target time. Each agent assumes an order when making a decision. The agent making the current decision considers the actions of the agents making previous decisions, thereby generating coordination and cooperation among agents. Moreover, the order of each decision is random, ensuring that the agent's decision satisfies Nash equilibrium. This realizes the modeling of the state value of a single agent in a multi-agent group, and improves the state value of the overall multi-agent group by optimizing the single agent.

[0068] The target loss function of the hunting agent is not limited to being expressed as:

[0069]

[0070] in, The corresponding parameters of the policy network and the objective loss function are represented. In fact, it is not limited to minimizing the empirical Bellman error; This function defines the probability that the current agent will perform action 'a' given that the previous agent has performed an action in the policy neural network. This refers to a policy pruning technique that ensures policy updates remain within a defined range, preventing large variance after policy updates. The default value is 0.05.

[0071] As an optional implementation, the action selection of the hunting agent is performed using the policy function of the policy neural network, and the action selection of the attacking agent is performed based on a preset combat strategy. This includes: determining the output action of the hunting agent based on the input state of the hunting agent using the policy function, wherein the output action of the hunting agent indicates the target position of the hunting agent at the next moment; and determining the output action of the attacking agent based on the input state of the attacking agent using a preset comparison strategy, wherein the output action of the attacking agent indicates the target position of the attacking agent at the next moment.

[0072] Assume the output action of the hunting agent is Output action The output action is the action taken by the hunting agent after receiving the input state. Not limited to:

[0073] in, This indicates the target longitude of the hunting agent in the next moment. This represents the target dimension of the hunting agent in the next moment. By controlling the expected target point at each moment, the collaborative hunting path planning of each hunting agent can be achieved.

[0074] Assume the output action of the attacking agent is Output action The output action is the action taken by the attacking agent after receiving the input state. Represented as:

[0075] in, This indicates the target longitude of the attacking agent in the next moment. This represents the target dimension of the attacking agent in the next moment. By controlling the expected target point at each moment, the collaborative hunting path planning of each attacking agent can be achieved.

[0076] As an optional implementation, training the policy neural network includes: storing the current state, actions performed, traction rewards, and next state of the hunting agent as experience in a shared experience pool; when the amount of experience in the shared experience pool reaches a preset training amount, the hunting agent draws samples from the shared experience pool in batches for learning and updates the policy function, action value function, and state value function of the policy neural network.

[0077] As an optional implementation, updating the policy function, action value function, and state value function of the policy neural network includes: determining the update magnitude of the policy neural network based on the current training effect; and updating the policy function, action value function, and state value function of the policy neural network according to the update magnitude.

[0078] Optionally, the specific method for training the policy neural network is as follows:

[0079] Step 1: Assume the maximum number of training rounds is E, the maximum step size per round is T, the shared experience pool capacity is M, and the learning rates of the policy neural network and the objective loss function are respectively... and The parameter update scaling factor for the target policy neural network and the target loss function is: ;

[0080] Step 2: Randomly initialize the state space and and in the state dimension where the initial position of the hunting agent is fixed, while the initial position of the attacking agent is random within a certain area, and the neural network weight parameters are initialized;

[0081] Step 3: Select the action of the hunting agent in the state : :

[0082]

[0083] where is Gaussian white noise, which is used to increase the exploration ability of decision-making during training.

[0084] Step 4: Select the action of the attacking agent in the state : :

[0085]

[0086] where is Gaussian white noise, which is used to increase the exploration ability of decision-making during training.

[0087] Step 5: The hunting agent executes the action , the attacking agent executes the action , obtain the traction reward of the hunting agent at time t according to Step 1 , then obtain the states of the hunting agent and the attacking agent at time t + 1 according to Step 5, and then save the experience in the shared experience pool;

[0088] Step 6: Judge whether the number of experiences N currently stored in the shared experience pool reaches the batch training amount M. If N < M, continue to execute Step 3, Step 4 and Step 5; if the number of currently stored experiences is equal to the capacity of the shared experience pool, the new experience will overwrite the oldest experience in the shared experience pool. At this time, enter Step 7, Step 8 and Step 9;

[0089] Step 7: The hunting agent shuffles all the samples in the shared experience pool and randomly extracts samples in multiple batches for learning. The sample size is M, and update the policy neural network according to the minimum loss function, which is expressed as:

[0090]

[0091] where L is the overall loss function.

[0092] Step 8: Update the policy gradient:

[0093]

[0094] in, To evaluate the gradient of the state-behavior value function of a policy neural network, The gradient of the policy neural network;

[0095] Step 9: Update the parameters of the state-value network and policy neural network of the hunting agent:

[0096] in, To update the scaling factor, the update magnitude of the policy network is determined based on the training effect.

[0097] As an optional implementation method, the hunting agent based on the policy neural network meets the requirements of cluster collaborative hunting in the scenario test, including: when the hunting agent's hunting win rate exceeds the win rate threshold, the hunting agent's loss rate is lower than the loss threshold, and the hunting agent's hit rate is higher than the hit threshold, it is determined that the hunting agent based on the policy neural network meets the requirements of cluster collaborative hunting in the scenario test.

[0098] The determination of the effectiveness of the coordinated hunting of hunting agents is not limited to the calculation of the following three key indicators: hunting success rate, red team weapon hit rate, and red team agent attrition rate.

[0099] The specific calculation method for the success rate of a hunting agent is not limited to:

[0100] Where K is the total number of test rounds, This represents the success rate of the hunt in the kth round, with 1 for success and 0 for failure. The default value for K is 100000.

[0101] The specific calculation method for weapon hit rate of hunting agents is not limited to:

[0102] Where K is the total number of test rounds, This represents the number of attacking agents destroyed in the k-th round. This represents the number of weapons fired by the hunting agent in the kth round, with K having a default value of 100000.

[0103] The specific calculation method for the loss rate of the hunting agent is not limited to:

[0104] Where K is the total number of test rounds, This represents the number of attacking agents destroyed in the k-th round. This represents the initial total number of attacking agents in the k-th round, with K defaulting to 100000.

[0105] Determining the outcome of a game is not limited to setting a threshold. ,when , If the target is met, the hunting effect is considered satisfactory; otherwise, if any relation is not satisfied, the hunting effect is considered unsatisfactory, and the training and optimization of the policy neural network continues.

[0106] As an optional implementation, after the strategy neural network hunting agent meets the requirements of clustered collaborative hunting in the scenario test, it also includes: calculating the single-step calculation time of the hunting agent; and determining the hardware carrier parameters of the strategy neural network based on the single-step calculation time and the preset time requirement.

[0107] The assessment of the effectiveness of the coordinated hunting operation by hunting agents is not limited to including the single-step computation time of the hunting agents. Specifically, it is not limited to determining the hardware parameters of the neural network supporting the strategy based on the single-step computation time, assuming that the hunting success rate, weapon hit rate, and agent attrition rate all meet the requirements. The specific calculation method for the single-step computation time is as follows:

[0108] in, Indicates the training time The computation time of step-by-step reasoning This represents the total number of calculation steps. The default value is 100000.

[0109] If the single-step computation time exceeds a preset threshold, adjust the parameters of the hardware supporting the strategy neural network so that the single-step computation time of the strategy neural network during cooperative hunting meets the preset requirements.

[0110] Furthermore, the overall implementation process of the reinforcement learning-based cluster collaborative hunting method is not limited to, for example... Figure 2As shown, a hunting game adversarial training scenario is first constructed, and training is conducted within this scenario. Specifically, the policy function and action value function of each combat unit are initialized, the state s and target loss function are initialized, and an experience pool is constructed. In state s1, action a1 is selected and executed, yielding a reward value r1 and the next state s1'. Experience samples are saved to the experience pool, and samples are randomly selected to train the agent. Is the experience pool full? If no, the experience pool is not full, and the process returns to execute the action; if yes, the experience pool is full, the neural networks are updated, and a hunting game evaluation scenario is constructed. In this evaluation scenario, the agent engages in combat against the opponent according to the rules. Is the number of evaluation attempts reached? If no, the number of evaluation attempts has not been reached, and the process returns to continue. If yes, the number of evaluation attempts has been reached, and the evaluation win rate has been reached. If no, the evaluation win rate has not been reached, and the process returns to reinitialize the state, target loss function, and construct the experience pool. If yes, the evaluation win rate has been reached, and the process ends, confirming that the current policy neural network can achieve cluster-based collaborative hunting.

[0111] Taking the hunting agent as the red side and the attacking agent as the blue side as the blue side, the above-mentioned cluster cooperative hunting method based on reinforcement learning will be further explained.

[0112] The reinforcement learning-based cluster cooperative hunting method includes the following steps:

[0113] Step 1: Construct a game-playing training scenario that includes a red hunting agent and a blue attacking agent. Both the red and blue sides perform the task of searching for and eliminating the enemy. Construct a Markov model of the red hunting agent.

[0114] Step 2: Construct the Blue Force offensive intelligent agent based on preset combat rules;

[0115] Step 3: Construct the policy function, action value function, state value function, and target loss function of the red hunting agent, and set initial values;

[0116] Step 4: In the Markov decision process, the policy function is used to select the actions of the red hunting agent, and the rule policy is used to select the actions of the blue attacking agent.

[0117] Step 5: Execute the action selected in Step 4. Assume a state space transition occurs, changing from state s to state s. The red team's hunting agent received a reward R;

[0118] Step 6: The red team's hunting agent updates the action value function based on the reward value obtained in Step 5;

[0119] Step 7: Repeat steps 4 to 6 to train the policy neural network of the red hunting agent until the target loss function converges, thus completing the drone swarm cooperative hunting decision based on reinforcement learning.

[0120] Step 8: Transfer the policy function, action value function, state value function, and target loss function of the red hunting agent to the adversarial game test scenario;

[0121] Step 9: Conduct multiple rounds of testing to verify the effectiveness of the hunting strategy. If the red side's hunting win rate exceeds the threshold... The loss rate of the red agent is lower than the threshold. The red team's weapon hit rate is above the threshold. If the cluster's coordinated hunting is achieved, and if any of the three indicators does not meet the threshold, then return to step 7 to continue training.

[0122] The simulation environment involved in verifying the above algorithm is as follows: Figure 3 As shown in the figure, triangles indicate the red hunting agents and circles indicate the blue attacking agents. The distribution of the red hunting agents and the blue attacking agents is shown in the figure.

[0123] based on Figure 3 The scenario depicted is a hunting game involving multiple hunters and attackers. We are the hunters, with 12 combat units, all drones, each carrying 4 weapons. Our objective is to search for and eliminate enemies in a designated airspace before our limited fuel runs out, and to ensure that at least one of our combat units survives. The enemy is the attacker, with 2 combat units, each equipped with 6 weapons. Their objective is to eliminate all our units or to deplete our resources through escape and maneuvering to reach our base.

[0124] The algorithm consists of two parts: the training process and the testing process.

[0125] Step 1: Construct a game-like adversarial training scenario involving a red hunting agent and a blue attacking agent. Both sides perform the task of searching for and eliminating the enemy. Construct a Markov model for the red hunting agent; construct a Markov model for the red hunting agent ( ),in This represents the input state for the red team's hunting agent. The output actions of the red team's hunting agent. For the reward function of the red side hunting the intelligent agent, This is the discount factor, with a default value of 0.99; Initialization is as follows:

[0126]

[0127] Specifically, it is expressed as follows:

[0128]

[0129] Specifically, it is expressed as follows:

[0130]

[0131] Step 2: Construct a rule-based blue team attacking agent. In this example, the blue team's rules are designed as follows: Figure 4 As shown, the rules are briefly described below:

[0132] T1: Firing Moment. When the Blue team's attacking unit's fire control radar detects the Red team's attacking unit, and the Red team is within the Blue team's frontal fan-shaped firing range, the Blue team can automatically fire when its firing conditions are met.

[0133] T2: Offset Guidance. After the Blue Team fires a weapon, it uses offset guidance, deflecting towards the direction with fewer enemy aircraft, making it easier to retreat in a safe direction after striking.

[0134] T3: Retreat after attacking. In this scenario, the blue team's attacking units have a fire cooldown period after firing, during which they cannot fire. Therefore, they need to retreat temporarily to prevent the blue team from being destroyed due to being too close to the red team and unable to fire.

[0135] T4: Retreat. After firing all weapons or destroying all Red Team targets, the Blue Team performs a retreat, completing the mission objective of searching for and eliminating the enemy, and returns to base.

[0136] Step 3: Construct the policy function, action value function, state value function, and target loss function of the red hunting agent, and set initial values; first, for all states... Actions of the hunting party Attacking actions Set the learning decay factor Learning decay factor This is used to accumulate the reward value throughout the entire game process. The higher the value, the more emphasis is placed on long-term returns;

[0137] Policy Neural Network like Figure 5 As shown, it consists of an input layer, a feature encoding module, an action generation module, and an output layer. The policy neural network takes the input state as its input. Output action The process is represented as:

[0138] in, It is Gaussian white noise, used to enhance the exploratory ability of decision-making during training.

[0139] The action value function is initialized as follows:

[0140] in, Take 0.9;

[0141] The state value function is initialized as follows:

[0142]

[0143] in, Take 0.9;

[0144] The target loss function is initialized as follows:

[0145]

[0146]

[0147] in, Take 0.05.

[0148] Step 4: In the Markov decision process, the policy function is used to select the actions of the red hunting agent, and the rule policy is used to select the actions of the blue attacking agent.

[0149] The action selection process considers the contribution relationship between individual combat units and combat clusters, and aims to achieve a balance between solution space exploration and utilization. Therefore, during training, a strategy model is used to randomly select actions for our hunting units with a certain probability. The status of the red team's hunting units changed from Transfer to and receive a reward R;

[0150] The enemy attacking unit selects an action appropriate to the scenario based on the attack rules. And prepare to make the next action selection according to the rules.

[0151] Step 5: Execute the action selected in Step 4. At this point, the state space transitions from state s to state s. The red team's hunting agent received a reward R;

[0152] Step 6: The red team's hunting agent updates the action value function based on the reward value obtained in Step 5;

[0153] Step 7: Repeat steps 4 to 6 to train the relevant neural network of the red hunting agent until the target loss function converges, such as... Figure 6 As shown, a cluster-based collaborative hunting decision is achieved based on reinforcement learning;

[0154] Step 8: Transfer the policy function, action value function, state value function, and target loss function of the red hunting agent to the adversarial game test scenario;

[0155] Step 9: Conduct multiple rounds of testing to verify the training effect, assuming that the red side's hunting win rate exceeds the threshold. The loss rate of the red agent is lower than the threshold. The red team's weapon hit rate is above the threshold. If the cluster's coordinated hunting is achieved, and if any of the three indicators does not meet the threshold, then return to step 7 to continue training.

[0156] The training process is completed by repeating the above steps, and the optimal balanced hunting strategy under the current finite state space is obtained.

[0157] The training results are not limited to those shown in Figure 7. It can be seen that the errors of the key training parameters actor_grad_norm and critic_grad_norm of the policy network stabilize around 0.2 after multiple rounds of training, indicating that the policy neural network has converged. The key parameter, average_episode_rewards, fluctuates around the peak value of 200 after reaching its highest point. Since the reward value per step is around 10 points, the current fluctuation value does not exceed 5 points, which also indicates that the policy neural network has converged. The value_loss of the action is within ±0.1 for each calculation, and the final fluctuation range of the curve in the figure is around 0.05, indicating that the action value function has converged and can be used to test the hunting effect in adversarial game testing scenarios.

[0158] To verify the accuracy and feasibility of the results, the evaluation system will not be limited to testing each indicator 10,000 times. Each time, the combat units of both sides will randomly appear at any location in the designated area.

[0159] In the simulation results, due to the limited fuel of our hunting agents, the enemy attackers could use the escape mechanism to consume all our fuel before circling back to approach. Therefore, actions involving more than 300 steps were considered a victory for the enemy attackers. The final results showed that when the initial maximum distance between the two sides was 230 kilometers, with one enemy combat unit and six friendly combat units, the win rate of our multiple units conducting coordinated flanking maneuvers and hunting attacks was the highest, reaching 86%. When the initial maximum distance between the two sides was 230 kilometers, with two enemy combat units and twelve friendly combat units, the win rate of our multiple units conducting coordinated flanking maneuvers and hunting attacks was the highest, reaching 72%. The red agent's attrition rate was 25%, and the red weapon's hit rate was 63%, meeting the threshold requirements. This indicates that the coordinated hunting strategy obtained based on the reinforcement learning algorithm is effective and meets expectations, because the more attackers there are, the greater the difficulty of our hunting and the greater the proportion of resources consumed.

[0160] In this embodiment, for the decision-making problem in cluster cooperative hunting, a strategy generation for cluster cooperative hunting is achieved through reinforcement learning and rule-based adversarial game. Considering the asymmetric adversarial game between the two parties in real-world scenarios, a zero-sum game model with a finite time domain and finite state-action space is designed under the condition that the scale of the two parties can be changed. The model is formally represented by a Markov decision process, focusing on the change of reward value during the state transition process, which effectively solves the reward sparsity problem in the model training process.

[0161] By combining the random decision-making order of agents, a comprehensive advantage function that satisfies Nash equilibrium is constructed to solve the game problem. A reinforcement learning method based on the Encoder-Decoder objective function is designed to learn the value of actions and states through interaction with the environment, fully explore the game process between the hunter and the attacker, and ensure the optimality of the decisions of both parties in the rational game. Example 2

[0162] A cluster-based cooperative hunting system based on reinforcement learning, such as Figure 8 As shown, the system includes:

[0163] An initialization module 802 is constructed to build hunting agents and attacking agents in a game-playing scenario, and to initialize the policy neural network of the hunting agent and the initial state of the attacking agent. Both the hunting agent and the attacking agent perform the task of searching for and destroying the enemy.

[0164] Training module 804 is used in Markov decision-making to select actions for the hunting agent using the policy function of the policy neural network and select actions for the attacking agent based on a preset combat strategy. After the hunting agent and the attacking agent each execute their selected actions, the reward obtained by the hunting agent is determined and the action value function and state value function of the policy neural network are updated based on the reward in order to train the policy neural network.

[0165] Test module 806 is used to transfer the policy neural network to a game adversarial scenario for testing when the target loss function of the policy neural network converges. The target loss function is determined based on the policy function, action value function, and state value function.

[0166] The determination module 808 is used to determine the cluster collaborative hunting of the hunting agent based on the policy neural network when the hunting agent meets the requirements of cluster collaborative hunting in the scenario test.

[0167] Optionally, the above-mentioned initialization module 802 constructs the hunting agent, including: constructing a Markov model of the hunting agent, including the input state, output action and reward function, and discount coefficient of the hunting agent.

[0168] Optionally, the above-mentioned initialization module 802 constructs an offensive agent, including: constructing an offensive agent based on a preset combat strategy, and determining the input state of the offensive agent, wherein the input state of the offensive agent includes the position information of each offensive agent, the distance matrix between each offensive agent, the position information of all hunting agents detected by the offensive agent detector, and the distance matrix between each offensive agent and all hunting agents.

[0169] Optionally, the above-mentioned initialization module 802 initializes the policy neural network of the hunting agent, including: constructing a policy neural network including an input layer, a feature encoding module, an action generation module and an output layer; and initializing the policy function, action value function, state value function and target loss function of the policy neural network.

[0170] Optionally, the training module 804 uses the policy function of the policy neural network to select actions for the hunting agent and select actions for the attacking agent based on a preset combat strategy, including: determining the output action of the hunting agent based on the input state of the hunting agent using the policy function, wherein the output action of the hunting agent indicates the target position of the hunting agent at the next moment; and determining the output action of the attacking agent based on the input state of the attacking agent using a preset comparison strategy, wherein the output action of the attacking agent indicates the target position of the attacking agent at the next moment.

[0171] Optionally, the training module 804 trains the policy neural network by: storing the current state, actions, traction rewards, and next state of the hunting agent as experience in a shared experience pool; when the amount of experience in the shared experience pool reaches a preset training amount, the hunting agent draws samples from the shared experience pool in batches for learning and updates the policy function, action value function, and state value function of the policy neural network.

[0172] Optionally, the training module 804 updates the policy function, action value function, and state value function of the policy neural network, including: determining the update magnitude of the policy neural network based on the current training effect; and updating the policy function, action value function, and state value function of the policy neural network according to the update magnitude.

[0173] Optionally, the determination module 808 determines that the hunting agent based on the policy neural network meets the requirements of cluster collaborative hunting in the scenario test, including: when the hunting agent's hunting win rate exceeds the win rate threshold, the hunting agent's loss rate is lower than the loss threshold, and the hunting agent's hit rate is higher than the hit threshold, the hunting agent is determined to meet the requirements of cluster collaborative hunting in the scenario test.

[0174] Optionally, the above-mentioned reinforcement learning-based cluster cooperative hunting system also includes a parameter module, which is used to calculate the single-step computation time of the hunting agent after the policy neural network hunting agent meets the cluster cooperative hunting requirements in the scenario test; and to determine the hardware carrier parameters of the policy neural network based on the single-step computation time and the preset time requirement.

[0175] In this embodiment, a hunting agent and an attacking agent are constructed in a game-theoretic scenario, and the initial states of the hunting agent's policy neural network and the attacking agent's initial states are initialized. In Markov decision-making, the policy function of the policy neural network is used to select actions for the hunting agent, and the attacking agent's actions are selected based on a preset battle strategy. After the hunting agent and the attacking agent each execute their selected actions, the reward obtained by the hunting agent is determined, and the action value function and state value function of the policy neural network are updated based on the reward to train the policy neural network. When the objective loss function of the policy neural network converges, the policy neural network is transferred to the game-theoretic scenario for testing. Based on the fact that the hunting agent of the policy neural network meets the requirements of clustered collaborative hunting in the scenario test, it is determined that the policy neural network has realized the clustered collaborative hunting of the hunting agent. Thus, based on the Markov decision process, a zero-sum game model with finite time domain and finite state action space under variable scale of both sides is formally represented. The focus is on the reward change during the state transition process, which effectively solves the reward sparsity problem in model training and solves the technical problem of solving the mixed Nash equilibrium strategy of the hunting parties under asymmetric conditions, achieving the technical effect of ensuring the optimality of the decisions of both sides of the game. Example 3

[0176] In another aspect, the present invention provides an electronic device for implementing the above-described reinforcement learning-based cluster cooperative hunting method. This electronic device is not limited to a terminal device or server in a radar system. The electronic device includes, but is not limited to, a memory and a processor. The memory stores a computer program, and the processor is configured to execute the steps of any of the above method embodiments via the computer program. Example 4

[0177] In another aspect, the present invention provides a computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in various optional embodiments of the reinforcement learning-based cluster cooperative hunting method described above. The computer program is configured to execute the steps in any of the above method embodiments at runtime.

Claims

1. A method for swarm cooperative hunting based on reinforcement learning, characterized in that, The method comprises the following steps: In the game confrontation scene, a hunting agent and an attacking agent are constructed, and a strategy neural network of the hunting agent and an initial state of the attacking agent are initialized, wherein the hunting agent and the attacking agent both perform a search and kill task; the attacking agent is constructed, including: constructing an attacking agent based on a preset combat strategy, and determining an input state of the attacking agent, wherein the input state of the attacking agent comprises position information of each attacking agent, a distance matrix between the attacking agents, position information of all hunting agents detected by an attacking agent detector, and a distance matrix between each attacking agent and all hunting agents; In the Markov decision, the strategy function of the strategy neural network is used for action selection of the hunting agent, and the action selection of the attacking agent is based on a preset battle strategy; after the hunting agent and the attacking agent perform the selected actions respectively, the reward obtained by the hunting agent is determined, and the action value function and the state value function of the strategy neural network are updated based on the reward to train the strategy neural network; the strategy neural network is trained, including: saving the current time state, the executed action, the attractive reward and the next time state of the hunting agent in a shared experience pool as experience; when the amount of experience in the shared experience pool reaches a preset training amount, the hunting agent extracts samples from the shared experience pool in batches for learning, and the update amplitude of the strategy neural network is determined according to the current training effect; the strategy function, the action value function and the state value function of the strategy neural network are updated according to the update amplitude; In the case that the target loss function of the strategy neural network converges, the strategy neural network is migrated to the game confrontation scene for testing, wherein the target loss function is determined according to the strategy function, the action value function and the state value function; In the case that the hunting agent reaches the requirement of cluster cooperative hunting in the scene test based on the strategy neural network, it is determined that the strategy neural network realizes the cluster cooperative hunting of the hunting agent. 2.The method of claim 1, wherein, The hunting agent is constructed, including: A Markov model of the hunting agent is constructed, including an input state, an output action and a reward function, and a discount coefficient of the hunting agent. 3.The method of claim 1, wherein, The strategy neural network of the hunting agent is initialized, including: The strategy neural network comprising an input layer, a feature encoding module, an action generation module and an output layer is constructed; The strategy function, the action value function, the state value function and the target loss function of the strategy neural network are initialized. 4.The method of claim 1, wherein, The strategy function of the strategy neural network is used for action selection of the hunting agent, and the action selection of the attacking agent is based on a preset battle strategy, including: Based on the input state of the hunting agent, the output action of the hunting agent is determined by using the strategy function, wherein the output action of the hunting agent indicates the target position of the hunting agent at the next time. Determine an output action of the attacking agent based on the input state of the attacking agent and a preset comparison strategy, wherein the output action of the attacking agent indicates a target position of the attacking agent at a next time point. 5.The method of claim 1, wherein, Determine that the hunting agent reaches the requirement of the swarm cooperative hunting in the scene test based on the policy neural network, including: Determine that the hunting agent reaches the requirement of the swarm cooperative hunting in the scene test based on the policy neural network in a case that a hunting success rate of the hunting agent is higher than a success rate threshold, a loss rate of the hunting agent is lower than a loss threshold, and a hit rate of the hunting agent is higher than a hit threshold.

6. The reinforcement learning-based coordinated hunting method of clusters as claimed in claim 5, wherein, After the hunting agent reaches the requirement of the swarm cooperative hunting in the scene test based on the policy neural network, further include: Calculate a single-step calculation time consumption of the hunting agent; Determine a hardware carrier parameter of the policy neural network based on the single-step calculation time consumption and a preset time consumption requirement.

7. A reinforcement learning-based coordinated hunting system of a swarm, applying the reinforcement learning-based coordinated hunting method of any one of claims 1 to 6, characterized in that, Include: Construct an initialization module, configured to construct a hunting agent and an attacking agent in a game confrontation scene, and initialize a policy neural network of the hunting agent and an initial state of the attacking agent, wherein the hunting agent and the attacking agent both perform a search and destroy enemy task; A training module configured to, in a Markov decision, use a policy function of the policy neural network to select an action of the hunting agent and use a preset comparison strategy to select an action of the attacking agent, determine a reward obtained by the hunting agent after the hunting agent and the attacking agent perform the selected actions respectively, and update an action value function and a state value function of the policy neural network based on the reward, so as to train the policy neural network; A test module configured to, in a case that a target loss function of the policy neural network converges, migrate the policy neural network to the game confrontation scene for testing, wherein the target loss function is determined according to the policy function, the action value function and the state value function; A determination module configured to, in a case that the hunting agent reaches the requirement of the swarm cooperative hunting in the scene test based on the policy neural network, determine that the policy neural network realizes the swarm cooperative hunting of the hunting agent.

Images