Cluster collaborative hunting method based on reinforcement learning

Abstract

The invention belongs to the technical field of reinforcement learning, and particularly discloses a reinforcement learning-based cluster collaborative hunting method, which comprises the steps of initializing a strategy neural network of a hunting agent and an initial state of an attack agent; after the hunting agents and the attack agents execute the selected actions respectively, rewards obtained by the hunting agents are determined, and an action value function and a state value function of the strategy neural network are updated based on the rewards, so that the strategy neural network is trained; under the condition that the target loss function of the strategy neural network is converged, migrating the strategy neural network to a game confrontation scene for testing; under the condition that the hunting agent based on the strategy neural network meets the cluster collaborative hunting requirement in the scene test, the strategy neural network is determined to realize cluster collaborative hunting of the hunting agent. According to the scheme, the technical problem of solving the mixed Nash equilibrium strategy of both hunting parties under the asymmetric condition is solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of reinforcement learning, and specifically relates to a cluster collaborative hunting method based on reinforcement learning. Background Art

[0002] A cluster is a collection of real objects or organisms, usually formed in nature or artificial environments. Common clusters include bee colonies, ant colonies, cattle herds, etc. By observing their behavior patterns, researchers have found that individuals in a cluster can accomplish many complex tasks based on collaboration. In recent years, clusters have been used more and more widely, and have achieved good results in many fields such as search and rescue, ground exploration, agricultural planting, special logistics, etc. In these complex scenarios, cluster agents have completed many tasks that single agents cannot complete or complete poorly through division of labor, cooperation, and collaboration.

[0003] Regional target hunting is a typical application scenario of cluster coordination. It originates from the predatory hunting behavior that is prevalent in nature. In the years of research and development, researchers have made rich expansions on the hunting forms and purposes. Among them, multi-target hunting is an important confrontation link and a hot issue in cluster coordinated hunting. It can be used as a powerful means to capture important enemy personnel or suppress important enemy targets in confrontation.

[0004] When a cluster performs a collaborative hunting mission, it can be summarized into two stages: dispersion and hunting. By dispersing and bypassing the front battlefield, forming a favorable situation, and then surrounding the enemy, it is possible to achieve a hunting attack on the enemy's key targets with less loss. In terms of the form of collaborative action, cluster collaborative hunting tasks can be divided into problems such as collaborative formation, state consistency judgment, aggregation, envelope control, and collaborative detour hunting. Through the cooperation and collaboration between multiple agents in the cluster, the cluster can complete complex problems that a single agent cannot handle. Most of the existing research on cluster collaborative hunting is based on distributed control, that is, by converting the cluster hunting problem into a consistency problem, and then designing a distributed algorithm to make the cluster approach and converge to the target position, so as to achieve the encirclement and capture effect of the enemy target.

[0005] Some recent studies have been conducted from the perspective of formation control, using virtual structure methods, behavioral methods, artificial potential field methods, and graph theory methods to coordinate multiple agents to perform target hunting tasks. However, the research on multi-target hunting based on the above methods is still immature, often relying on complex calculations and control strategies, and many studies consider multiple targets as a whole for analysis, and there are few studies on hunting multiple targets separately.

[0006] Some other studies have optimized hunting tasks by decomposing hunting behaviors, such as using loose preference rules to enable individual agents to spontaneously form ideal hunting formations, setting dynamic hunting points, calculating the optimal hunting angles, and proposing multi-layer circular ambush hunting models. However, this type of hunting method is a procedural hunting method. In other words, although this type of method does not require manual control, it requires a lot of resources to remodel and solve the external environment model in the face of environmental changes and emergencies, which greatly affects the real-time and effectiveness of the agent's decision-making, and is difficult to apply to multi-agent hunting tasks in unknown and complex environments.

[0007] The above algorithms generally do not take into account the asymmetry of resources, detection information, and strategy levels between the hunting cluster and the enemy agents, and have weak capabilities in handling dynamic environmental changes and emergencies. They do not use intelligent algorithms to optimize strategies, and it is difficult to achieve hunting effects when rewards are sparse. Therefore, they cannot promote the continuous improvement and enhancement of multi-agent collaborative hunting strategies. Summary of the invention

[0008] The purpose of the present invention is to provide a cluster collaborative hunting method based on reinforcement learning to solve the technical problem of solving the mixed Nash equilibrium strategy of the hunting parties under asymmetric conditions.

[0009] To achieve the above object, the present invention adopts the following technical solution: A cluster collaborative hunting method based on reinforcement learning, comprising: constructing a hunting agent and an attacking agent in a game confrontation scenario, and initializing the strategy neural network of the hunting agent and the initial state of the attacking agent, wherein the hunting agent and the attacking agent both perform the task of searching and destroying the enemy; in Markov decision making, using the strategy function of the strategy neural network to select actions for the hunting agent and selecting actions for the attacking agent based on a preset combat strategy, after the hunting agent and the attacking agent each perform the selected actions, determining the reward obtained by the hunting agent and updating the action value function and state value function of the strategy neural network based on the reward to train the strategy neural network; when the target loss function of the strategy neural network converges, migrating the strategy neural network to a game confrontation scenario for testing, wherein the target loss function is determined based on the strategy function, the action value function, and the state value function; and when the hunting agent meets the cluster collaborative hunting requirements in the scenario test based on the strategy neural network, determining that the strategy neural network has achieved the cluster collaborative hunting of the hunting agent.

[0010] Preferably, constructing a hunting agent includes: constructing a Markov model of the hunting agent, including the input state, output action and reward function, and discount coefficient of the hunting agent.

[0011] Preferably, constructing an offensive intelligent agent includes: constructing an offensive intelligent agent based on a preset combat strategy, and determining the input state of the offensive intelligent agent, wherein the input state of the offensive intelligent agent includes the position information of each offensive intelligent agent, the distance matrix between each offensive intelligent agent, the position information of all hunting intelligent agents detected by the offensive intelligent agent detector, and the distance matrix between each offensive intelligent agent and all hunting intelligent agents.

[0012] Preferably, initializing the strategy neural network of the above-mentioned hunting agent includes: constructing a strategy neural network including an input layer, a feature encoding module, an action generation module and an output layer; initializing the strategy function, action value function, state value function and target loss function of the above-mentioned strategy neural network.

[0013] Preferably, the strategy function of the above-mentioned strategy neural network is used to select the action of the hunting agent and the action of the attacking agent is selected based on the preset combat strategy, including: based on the input state of the above-mentioned hunting agent, the output action of the hunting agent is determined by the above-mentioned strategy function, wherein the output action of the above-mentioned hunting agent indicates the target position of the above-mentioned hunting agent at the next moment; based on the input state of the above-mentioned attacking agent, the output action of the attacking agent is determined by the preset control strategy, wherein the output action of the above-mentioned attacking agent indicates the target position of the above-mentioned attacking agent at the next moment.

[0014] Preferably, the strategy neural network is trained, including: saving the current state, execution action, traction reward and next state of the hunting agent as experience in a shared experience pool; when the amount of experience in the shared experience pool reaches a preset training amount, the hunting agent extracts samples from the shared experience pool in batches for learning, and updates the strategy function, action value function and state value function of the strategy neural network.

[0015] Preferably, updating the policy function, action value function and state value function of the policy neural network includes: determining the update amplitude of the above-mentioned policy neural network according to the current training effect; and updating the policy function, action value function and state value function of the policy neural network according to the above-mentioned update amplitude.

[0016] Preferably, the above-mentioned hunting agent based on the above-mentioned strategy neural network meets the requirements of cluster collaborative hunting in the scenario test, including: when the hunting win rate of the above-mentioned hunting agent exceeds the win rate threshold, the loss rate of the above-mentioned hunting agent is lower than the loss threshold, and the hit rate of the above-mentioned hunting agent is higher than the hit threshold, it is determined that the above-mentioned hunting agent based on the above-mentioned strategy neural network meets the requirements of cluster collaborative hunting in the scenario test.

[0017] Preferably, after the above-mentioned strategy neural network and the above-mentioned hunting agent meet the cluster collaborative hunting requirements in the scenario test, it also includes: calculating the single-step calculation time of the hunting agent; determining the hardware carrier parameters of the above-mentioned strategy neural network based on the above-mentioned single-step calculation time and the preset time requirements.

[0018] A cluster cooperative hunting system based on reinforcement learning, comprising: constructing an initialization module, for constructing a hunting agent and an attacking agent in a game confrontation scenario, and initializing the strategy neural network of the hunting agent and the initial state of the attacking agent, wherein the hunting agent and the attacking agent both perform the task of searching and destroying the enemy; A training module is used to use the strategy function of the above-mentioned strategy neural network to select actions for the hunting agent and to select actions for the attacking agent based on a preset battle strategy in Markov decision making. After the above-mentioned hunting agent and the above-mentioned attacking agent each execute the selected actions, the reward obtained by the above-mentioned hunting agent is determined and the action value function and state value function of the above-mentioned strategy neural network are updated based on the above-mentioned reward to train the above-mentioned strategy neural network; a testing module is used to migrate the above-mentioned strategy neural network to a game confrontation scenario for testing when the target loss function of the above-mentioned strategy neural network converges, wherein the above-mentioned target loss function is determined according to the above-mentioned strategy function, the above-mentioned action value function and the above-mentioned state value function; a determination module is used to determine that the above-mentioned strategy neural network has realized the cluster collaborative hunting of the above-mentioned hunting agent when the above-mentioned hunting agent meets the requirements of cluster collaborative hunting in the scenario test based on the above-mentioned strategy neural network.

[0019] A computer-readable storage medium having a computer program stored therein, wherein the computer program is configured to execute the above-mentioned cluster collaborative hunting method based on reinforcement learning when running.

[0020] An electronic device includes a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the cluster collaborative hunting method based on reinforcement learning through the computer program.

[0021] In the present invention, a hunting agent and an attacking agent are constructed in a game confrontation scenario, and the strategy neural network of the hunting agent and the initial state of the attacking agent are initialized; in Markov decision making, the strategy function of the strategy neural network is used to select the action of the hunting agent and the action of the attacking agent is selected based on the preset battle strategy. After the hunting agent and the attacking agent each execute the selected action, the reward obtained by the hunting agent is determined and the action value function and state value function of the strategy neural network are updated based on the reward to train the strategy neural network; when the target loss function of the strategy neural network converges, the strategy neural network is migrated to the game confrontation scenario for testing, and when the hunting agent based on the strategy neural network meets the cluster collaborative hunting requirements in the scenario test, it is determined that the strategy neural network realizes the cluster collaborative hunting of the hunting agent, thereby formally representing the zero-sum game model of the finite time domain and finite state action space under the condition of variable scale of the two game parties based on the Markov decision process, focusing on the reward change during the state transfer process, effectively solving the reward sparseness problem in model training, solving the technical problem of solving the mixed Nash equilibrium strategy of the hunting two game parties under asymmetric conditions, and achieving the technical effect of ensuring the optimality of the decision of the two game parties. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It is a flow chart of a cluster collaborative hunting method based on reinforcement learning in an embodiment of the present invention; Figure 2 It is a flow chart of a cluster collaborative hunting method based on reinforcement learning in an embodiment of the present invention; Figure 3 It is a simulation test scenario of a cluster collaborative hunting method based on reinforcement learning in an embodiment of the present invention; Figure 4 It is a schematic diagram of a blue team rule example in an embodiment of a cluster collaborative hunting method based on reinforcement learning in an embodiment of the present invention; Figure 5 It is a strategy neural network structure diagram in an embodiment of a cluster collaborative hunting method based on reinforcement learning in an embodiment of the present invention; Figure 6 It is the interaction and training process of the red and blue intelligent agents in a cluster collaborative hunting method based on reinforcement learning in an embodiment of the present invention; FIG7 is a graph of key training parameters in an embodiment of a cluster collaborative hunting method based on reinforcement learning according to an embodiment of the present invention; actor_grad_norm is shown in FIG7(a), average_episode_rewards is shown in FIG7(b), value_loss is shown in FIG7(c), and critic_grad_norm is shown in FIG7(d); Figure 8 It is a structural diagram of a cluster collaborative hunting system based on reinforcement learning in an embodiment of the present invention. DETAILED DESCRIPTION

[0023] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. The advantages and features of the present invention will become more apparent from the following description and claims. It should be noted that the drawings are all in very simplified form and are not in precise proportions, and are only used to conveniently and clearly assist in explaining the purpose of the embodiments of the present invention.

[0024] It should be noted that, in order to clearly explain the content of the present invention, the present invention specifically cites multiple embodiments to further illustrate different implementations of the present invention, wherein the multiple embodiments are enumerated rather than exhaustive. In addition, for the sake of brevity of explanation, the contents mentioned in the previous embodiments are often omitted in the subsequent embodiments. Therefore, the contents not mentioned in the subsequent embodiments can refer to the previous embodiments accordingly. Example 1

[0025] A cluster collaborative hunting method based on reinforcement learning, such as Figure 1 As shown, the method includes: S102, constructing a hunting agent and an attacking agent in a game confrontation scenario, and initializing a strategy neural network of the hunting agent and an initial state of the attacking agent, wherein both the hunting agent and the attacking agent perform a task of searching and destroying the enemy; S104, in Markov decision making, using the policy function of the policy neural network to select actions for the hunting agent and selecting actions for the attacking agent based on a preset combat strategy, after the hunting agent and the attacking agent each perform the selected actions, determining the reward obtained by the hunting agent and updating the action value function and state value function of the policy neural network based on the reward, so as to train the policy neural network; S106, when the target loss function of the policy neural network converges, migrating the policy neural network to a game confrontation scenario for testing, wherein the target loss function is determined according to the policy function, the action value function, and the state value function; S108, when the strategy neural network-based hunting agent meets the cluster collaborative hunting requirements in the scenario test, it is determined that the strategy neural network has achieved the cluster collaborative hunting of the hunting agent.

[0026] As an optional implementation, constructing a hunting agent includes: constructing a Markov model of the hunting agent, including the input state, output action and reward function, and discount coefficient of the hunting agent.

[0027] Optionally, construct a Markov model of the hunting agent ( ),in, is the input state of the hunting agent, is the output action of the hunting agent, is the reward function of the hunting agent, is the discount factor.

[0028] Combine the hunting agent and weapon information, the attacking agent and weapon information, and do not limit the input state of the hunting agent to: ; in, Represents the location information of each hunting agent, usually expressed in longitude and latitude. represents the distance matrix between hunting agents, Indicates the number of weapons carried by each hunting agent, represents the position information of all attacking agents that can be detected by the hunting agent detector. represents the distance matrix between all hunting agents and all attacking agents, Indicates the information of the two weapons closest to the current hunting agent. The above information is not limited to the specific description: ; ; ; ; ; .

[0029] Define each hunting agent The guiding reward function , expressed as:

[0030] in, Represents a hunting agent Rewards for being in the hunting state, represents the reward for detecting an enemy attacking the agent, Indicates the reward obtained when being destroyed. Indicates whether the reward is in the center area. Rewards for destroying the enemy. Indicates the reward for approaching the enemy while in the hunting state. Indicates the reward for the enemy entering the attack range. Represents the reward for being away from the specified dispersion reference point when in a dispersed state. Represents a bonus for dodging enemy weapons.

[0031] Not limited to setting learning decay factors: , used to accumulate the reward value during the entire game process, The larger the value, the more emphasis is placed on long-term benefits.

[0032] As an optional implementation, constructing an offensive intelligent agent includes: constructing an offensive intelligent agent based on a preset combat strategy, and determining the input state of the offensive intelligent agent, wherein the input state of the offensive intelligent agent includes the position information of each offensive intelligent agent, the distance matrix between each offensive intelligent agent, the position information of all hunting intelligent agents detected by the offensive intelligent agent detector, and the distance matrix between each offensive intelligent agent and all hunting intelligent agents.

[0033] Optionally, an offensive agent is constructed based on a preset combat strategy, and the offensive agent is not limited to having a strong offensive capability, an ability to evade enemy weapons, and an ability to accelerate and escape.

[0034] Set the attacking agent’s situation input information to , combined with the information of the attacking agent and the hunting agent, the input state of the attacking agent is set as:

[0035] in, Represents the latitude and longitude information of each attacking agent, represents the distance matrix between attacking agents, represents the latitude and longitude information of all hunting agents that the attacking agent can detect, Represents the distance matrix between all attacking agents and all hunting agents. The above information is not limited to the specific description: ; ; ; .

[0036] As an optional implementation, initializing the strategy neural network of the hunting agent includes: constructing a strategy neural network including an input layer, a feature encoding module, an action generation module and an output layer; initializing the strategy function, action value function, state value function and target loss function of the strategy neural network.

[0037] Optionally, build a policy neural network for the Red Hunting Agent , specifically but not limited to including input layer, feature encoding module, action generation module and output layer, policy neural network input state and output actions The relationship is expressed as:

[0038] in, It is Gaussian white noise, which is used to increase the exploration ability of decision-making during training.

[0039] action According to the game confrontation scenario, the specific action space is preset and is not limited to the following:

[0040] The action space is not limited to the four basic actions, which are follow the target, move left relative to the target, move right relative to the target, and move in the opposite direction of the target line. The size of the action space is determined by the number of agents in both camps. Each agent's action includes the relative movement of each teammate. , relative to each enemy target , and the relative distance to the nearest weapon .

[0041] The action value function of the hunting agent is not limited to being expressed as:

[0042] in, is the mathematical expectation, represents the initial state of the agent, Indicates the initial action, Indicates that it obeys the probability distribution The action value function is represents the state generating function that obeys the probability distribution P, It represents the discounted reward function that satisfies the initial conditions. The action value function Q is the joint action value of all agents in each state s. Therefore, it is affected by the action chosen by each agent. It is necessary to comprehensively consider the action value at the target moment and the discounted return.

[0043] The state value function of the hunting agent is not limited to being expressed as:

[0044] in, is the mathematical expectation, Indicates that it obeys the probability distribution The action value function is represents the state generating function that obeys the probability distribution P, The discount rate is The reward function is Represents the initial state of the agent.

[0045] The state value function comprehensively considers the state of each decision-making agent at the target moment. Each agent assumes an order at each decision. The agent making the current decision will consider the actions of the agent making the previous decision, thereby generating coordination and cooperation between the agents. The order of each decision is random, ensuring that the agent's decision satisfies the Nash equilibrium, and realizing the state value modeling of a single agent in a multi-agent group, so as to improve the state value of the overall multi-agent population by optimizing a single agent.

[0046] The objective loss function of the hunting agent is not limited to being expressed as:

[0047]

[0048] in, Represents the corresponding parameters of the policy network, the target loss function In fact, it is not limited to being based on minimizing the empirical Bellman error; ,This function defines the policy neural network as the probability of the current agent performing action a when the previous agent performs an action; It represents the strategy clipping technology, which is used to ensure that the strategy update is within a limited range to prevent large variance after the strategy update. The default value is 0.05.

[0049] As an optional implementation, the strategy function of the strategy neural network is used to select the action of the hunting agent and the action of the attacking agent is selected based on the preset combat strategy, including: based on the input state of the hunting agent, the strategy function is used to determine the output action of the hunting agent, wherein the output action of the hunting agent indicates the target position of the hunting agent at the next moment; based on the input state of the attacking agent, the output action of the attacking agent is determined using a preset control strategy, wherein the output action of the attacking agent indicates the target position of the attacking agent at the next moment.

[0050] Assume that the output action of the hunting agent is , output action The action taken by the hunting agent after receiving the input state, output action Not limited to:

[0051] in, represents the target longitude of the hunting agent at the next moment, It represents the target latitude of the hunting agent at the next moment, and realizes the collaborative hunting path planning of each hunting agent by controlling the expected target point at each moment.

[0052] Assume that the output action of the attacking agent is , output action is the action taken by the attacking agent after receiving the input state, and the output action It is expressed as:

[0053] in, represents the target longitude of the attacking agent at the next moment, It represents the target latitude of the attacking agent at the next moment, and realizes the collaborative hunting path planning of each attacking agent by controlling the expected target point at each moment.

[0054] As an optional implementation, the strategy neural network is trained, including: saving the current state of the hunting agent, executed actions, traction rewards and next state as experience in a shared experience pool; when the amount of experience in the shared experience pool reaches a preset training amount, the hunting agent extracts samples from the shared experience pool in batches for learning, and updates the strategy function, action value function and state value function of the strategy neural network.

[0055] As an optional implementation, updating the policy function, action value function and state value function of the policy neural network includes: determining the update amplitude of the policy neural network according to the current training effect; updating the policy function, action value function and state value function of the policy neural network according to the update amplitude.

[0056] Optionally, the specific method of training the policy neural network is: Step 1: Assume that the maximum number of training rounds is E, the maximum step size per round is T, the capacity of the shared experience pool is M, and the learning rates of the policy neural network and the target loss function are and , the parameter update ratio of the target policy neural network and the target loss function is ; Step 2: Randomly initialize the state space and , and in the state dimension , where the initial position of the hunting agent is fixed, while the initial position of the attacking agent is random within a certain area, and the neural network weight parameters are initialized; Step 3: In the state Next, select the action of the hunting agent :

[0057] in, It is Gaussian white noise, which is used to increase the exploration ability of decision-making during training.

[0058] Step 4: In the state Select the action of the attacking agent :

[0059] wherein, is Gaussian white noise, which is used to increase the exploration ability of decision-making during training.

[0060] Step 5: The hunting agent executes the action , and the attacking agent executes the action . According to Step 1, obtain the traction reward of the hunting agent at time t . Then, according to Step 5, obtain the states of the hunting agent and the attacking agent at time t + 1, and then save the experience in the shared experience pool; Step 6: Determine whether the number of experiences N currently stored in the shared experience pool reaches the batch training volume M. If N < M, continue to execute Step 3, Step 4, and Step 5; if the currently stored number of experiences is equal to the capacity of the shared experience pool, the new experience will overwrite the oldest experience in the shared experience pool. At this time, enter Step 7, Step 8, and Step 9; Step 7: The hunting agent shuffles all the samples in the shared experience pool and randomly extracts samples in multiple batches for learning. The sample size is M. Update the policy neural network according to the minimum loss function, which is expressed as:

[0061]

[0062] where L is the overall loss function.

[0063] Step 8: Update the policy gradient:

[0064] wherein, is the gradient of the evaluation policy neural network state-action value function, is the gradient of the policy neural network; Step 9: Update the parameters of the state value network and the policy neural network of the hunting agent:

[0065] wherein, is the update proportionality coefficient, which determines the update amplitude of the policy network according to the training effect.

[0066] As an optional implementation, the strategy-based neural network hunting agent meets the cluster collaborative hunting requirements in the scenario test, including: when the hunting win rate of the hunting agent exceeds the win rate threshold, the hunting agent loss rate is lower than the loss threshold, and the hunting agent's hit rate is higher than the hit threshold, it is determined that the strategy-based neural network hunting agent meets the cluster collaborative hunting requirements in the scenario test.

[0067] The determination of the effectiveness of the clustered collaborative hunting of hunting agents is not limited to the determination by calculating the following three key indicators, namely: hunting success rate, red team weapon hit rate, and red team agent loss rate.

[0068] The specific calculation method of the hunting success rate of the hunting agent is not limited to:

[0069] Where K is the total number of test innings, It is the success rate of hunting in the kth round, success is 1, failure is 0, and the default value of K is 100000.

[0070] The specific calculation method of the hunting agent's weapon hit rate is not limited to:

[0071] Where K is the total number of test innings, represents the number of attacking agents destroyed in the kth round, Represents the number of weapons fired by the hunting agent in the kth round. The default value of K is 100000.

[0072] The specific calculation method of the loss rate of the hunting agent is not limited to:

[0073] Where K is the total number of test innings, represents the number of attacking agents destroyed in the kth round, Represents the total number of attacking agents at the beginning of the kth round. The default value of K is 100000.

[0074] Determining the game result is not limited to setting a threshold ,when , , then the hunting effect is judged to be up to standard. If any relationship is not satisfied, then the hunting effect is considered to be unsatisfactory, and the training and optimization of the strategy neural network will continue.

[0075] As an optional implementation, after the strategy neural network hunting agent meets the cluster collaborative hunting requirements in the scenario test, it also includes: calculating the single-step calculation time of the hunting agent; determining the strategy neural network hardware carrier parameters based on the single-step calculation time and the preset time requirements.

[0076] The determination of the hunting effect of the hunting agent's cluster cooperative hunting includes but is not limited to the single-step calculation time of the hunting agent. Specifically, it is not limited to determining the hardware parameters of the carrying strategy neural network based on the single-step calculation time when the hunting success rate, weapon hit rate and agent loss rate all meet the requirements. The specific calculation method of the single-step calculation time is:

[0077] in, Indicates the training time The computation time of step reasoning is Represents the total number of calculation steps, The default value is 100000.

[0078] When the single-step calculation time is greater than the preset threshold, the parameters of the hardware carrying the strategy neural network are adjusted so that the single-step calculation time of the strategy neural network during collaborative hunting meets the preset requirements.

[0079] Furthermore, the overall process of implementing the cluster collaborative hunting method based on reinforcement learning is not limited to the following: Figure 2 As shown, first construct a hunting game confrontation training scenario, and conduct training in the hunting game confrontation training scenario. Specifically, initialize the strategy function and action value function of each combat unit, initialize the state s, the target loss function, and build an experience pool. Select and execute action a1 in state s1, obtain the reward value r1 and the next state s1', save the experience sample to the experience pool, and randomly extract samples to train the agent. Determine whether the experience pool is full? If the judgment is no, the experience pool is not full, then return to execute the action; if the judgment is yes, the experience pool is full, then update each neural network, construct a hunting game evaluation scenario, and fight against the rain rule opponent in the hunting game evaluation scenario. Determine whether the evaluation number has been reached? If the judgment is no, the evaluation number has not been reached, then return to continue the battle. If the judgment is yes, the evaluation number has been reached, then determine whether the evaluation win rate has been reached. If the judgment is no, the evaluation win rate has not been reached, then return to reinitialize the state, target loss function, and build an experience pool. If the judgment is yes, the evaluation win rate has been reached, then the process ends, and it is determined that the current strategy neural network can achieve cluster collaborative hunting.

[0080] Taking the hunting agent as the red side and the attacking agent as the blue side as an example, the above-mentioned cluster collaborative hunting method based on reinforcement learning is further explained.

[0081] The cluster collaborative hunting method based on reinforcement learning includes the following steps: Step 1: Construct a game confrontation training scenario including the red hunting agent and the blue attacking agent. Both the red and blue agents perform the task of searching and destroying the enemy, and construct a Markov model of the red hunting agent. Step 2: Build a blue attack agent based on preset combat rules; Step 3: Construct the strategy function, action value function, state value function, target loss function, etc. of the Red Team hunting agent, and set the initial value; Step 4: In the Markov decision process, the strategy function is used to select the actions of the red hunting agent, and the rule strategy is used to select the actions of the blue attacking agent; Step 5: Execute the action selected in step 4. Assume that the state space is transferred from state s to state , the red hunting agent received a reward R; Step 6: The Red Hunting Agent updates the action value function based on the reward value obtained in step 5; Step 7: Repeat steps 4 to 6 to train the red hunting agent’s strategy neural network until the target loss function converges, completing the drone swarm collaborative hunting decision based on reinforcement learning; Step 8: Migrate the strategy function, action value function, state value function, and target loss function of the Red Team hunting agent to the adversarial game test scenario; Step 9: Conduct multiple rounds of testing to verify the hunting effect. If the Red team's hunting win rate exceeds the threshold ,,Red agent loss rate is lower than the threshold , the hit rate of the Red side's weapons is higher than the threshold , then the collaborative hunting of the cluster is realized. If any of the three indicators does not meet the threshold, return to step 7 to continue training.

[0082] The simulation environment involved in verifying the above algorithm is as follows Figure 3 As shown, the red hunting agent is indicated by a triangle, the blue attacking agent is indicated by a circle, and the distribution of the red hunting agent and the blue attacking agent is shown in the figure.

[0083] based on Figure 3 The scenario shown sets up a hunting game scenario with multiple hunters and multiple attackers. It is assumed that our side is the hunter and has 12 combat units, all of which are drones. Each combat unit carries 4 weapons. Our goal is to search and destroy the enemy in the designated airspace before the limited fuel is consumed, and ensure that at least one combat unit of our side survives; the enemy is the attacker and has 2 combat units, each equipped with 6 weapons. Its goal is to eliminate all our units, or consume our resources through escape and detour, and approach our base.

[0084] The algorithm is divided into two parts: training process and testing process.

[0085] Step 1: Construct a game confrontation training scenario including the red hunting agent and the blue attacking agent. Both the red and blue agents perform the task of searching and destroying the enemy. Construct a Markov model of the red hunting agent. Construct a Markov model of the red hunting agent ( ),in is the input state of the Red Hunting Agent, is the output action of the Red Hunting Agent, is the reward function of the red hunting agent, is the discount factor, the default value is 0.99; Initialization is as follows:

[0086] The specific expressions are as follows:

[0087] The specific expressions are as follows:

[0088] Step 2: Build a rule-based blue attack agent. In this example, the blue rule design is as follows: Figure 4 The rules are briefly described as follows: T 1 : The moment of firing. The fire control radar of the blue attacking unit detects the red attacking unit, and the red unit is within the fan-shaped firing range in front of the blue unit, and the blue unit's firing conditions are met, and the red unit can automatically fire.

[0089] T 2 : Offset guidance. After the blue team launches a weapon, it will conduct offset guidance, deviating to the direction with fewer enemy aircraft, so as to facilitate retreat to a safe direction after the strike.

[0090] T 3 : Retreat after attacking. In this scenario, the blue attacking unit has a period of firing cooldown after firing, during which it cannot fire, so it needs to retreat temporarily to prevent the blue unit from being destroyed due to being too close to the red unit and unable to fire.

[0091] T 4 : Retreat. After all weapons have been deployed or all Red targets have been destroyed, the Blue team will retreat, complete the mission objective of searching for and destroying the enemy, and withdraw to the base.

[0092] Step 3: Construct the strategy function, action value function, state value function, target loss function, etc. of the Red Team hunting agent, and set the initial value; first, for all states , hunting action , the attacking side moves , set the learning attenuation factor , learning attenuation factor , used to accumulate the reward value during the entire game process, The larger the value, the more emphasis is placed on long-term benefits; Policy Neural Network like Figure 5 As shown, it consists of an input layer, a feature encoding module, an action generation module, and an output layer. The policy neural network input state , output action The process is expressed as:

[0093] in, It is Gaussian white noise, which is used to increase the exploration ability of decision-making during training.

[0094] The action value function is initialized as follows:

[0095] in, Take 0.9; The state value function is initialized as follows:

[0096] in, Take 0.9; The target loss function is initialized as follows:

[0097]

[0098] in, Take 0.05.

[0099] Step 4: In the Markov decision process, the strategy function is used to select the actions of the red hunting agent, and the rule strategy is used to select the actions of the blue attacking agent; The contribution relationship between individual combat units and combat clusters is considered in the action selection process, and a balance between exploration and utilization of the solution space is achieved. Therefore, during the training process, the strategy model is used to randomly select actions for our hunting units with a certain probability. , the Red Hunting Unit's status changes from Transfer to , and get reward R; The enemy attacking unit chooses an action that is appropriate to the scenario according to the attack rules. , and prepare to select the next action according to the rules.

[0100] Step 5: Execute the action selected in step 4. At this time, the state space is transferred from state s to state , the red hunting agent received a reward R; Step 6: The Red Hunting Agent updates the action value function based on the reward value obtained in step 5; Step 7: Repeat steps 4 to 6 to train the relevant neural network of the Red Hunting Agent until the target loss function converges, such as Figure 6 As shown, the cluster collaborative hunting decision-making based on reinforcement learning is completed; Step 8: Migrate the strategy function, action value function, state value function, and target loss function of the Red Team hunting agent to the adversarial game test scenario; Step 9: Conduct multiple rounds of testing to verify the training effect. Assume that the red team's hunting win rate exceeds the threshold. , the loss rate of the red agent is lower than the threshold , the hit rate of the Red side's weapons is higher than the threshold , then the collaborative hunting of the cluster is realized. If any of the three indicators does not meet the threshold, return to step 7 to continue training.

[0101] The training process is completed by repeatedly executing the above steps to obtain the optimal balanced hunting strategy in the current finite state space.

[0102] The training results are not limited to those shown in FIG7 . It can be seen that the key training parameters of the policy network, actor_grad_norm and critic_grad_norm, fluctuate around 0.2 after multiple rounds of training, indicating that the policy neural network has converged. The key parameter indicator, the single-step average reward value, average_episode_rewards, fluctuates around the highest point after reaching the highest point of 200 points. Since the single-step reward value is around 10 points, the current fluctuation value does not exceed 5 points, which also indicates that the policy neural network has converged. The action value loss value_loss fluctuates at ±0.1 each time it is calculated. The final fluctuation range of the curve in the figure fluctuates around 0.05, indicating that the action value function converges and the hunting effect test can be carried out in the adversarial game test scenario.

[0103] In order to verify the accuracy and feasibility of the results, each indicator in the evaluation system is not limited to being tested 10,000 times. Each time, the combat units of both sides will randomly appear at any location in the designated area.

[0104] In the simulation results, since the fuel of our hunting agent is limited, the enemy attacker can use the escape mechanism to consume all our fuel and then circle back to approach. Therefore, the enemy attacker who executes more than 300 steps is considered to have won. The final results show that when the initial maximum distance between the enemy and our side is 230 kilometers, the enemy has 1 combat unit and our side has 6 combat units, the winning rate of our multiple units conducting coordinated roundabout hunting strikes is the highest, reaching 86%. When the initial maximum distance between the enemy and our side is 230 kilometers, the enemy has 2 combat units and our side has 12 combat units, the winning rate of our multiple units conducting coordinated roundabout hunting strikes is the highest, reaching 72%. The loss rate of the red agent is 25%, and the hit rate of the red weapon is 63%, which meets the threshold requirements. This shows that the coordinated hunting strategy obtained based on the reinforcement learning algorithm is effective and in line with expectations, because the more attackers there are, the more difficult it is for us to hunt and the greater the proportion of resources consumed.

[0105] In an embodiment of the present application, in order to solve the decision-making problem in cluster collaborative hunting, the strategy generation of cluster collaborative hunting is realized through reinforcement learning and rule-based adversarial game. Taking into account the asymmetric adversarial game between the two parties in real scenarios, a zero-sum game model with a finite time domain and finite state action space under the condition of variable scale of the two parties is designed, and it is formally represented by the Markov decision process. The change of reward value during state transition is emphasized, which effectively solves the problem of reward sparsity in the model training process.

[0106] The method of combining the random decision-making order of intelligent agents is adopted to construct a comprehensive advantage function that satisfies the Nash equilibrium to solve the game problem. A reinforcement learning method based on the Encoder-Decoder objective function is designed. Through interaction with the environment, the action value and state value are learned, and the game process between the hunter and the attacker is fully explored, ensuring the optimality of the decision-making of both rational game parties. Example 2

[0107] A cluster collaborative hunting system based on reinforcement learning, such as Figure 8 As shown, the system includes: Constructing an initialization module 802, which is used to construct a hunting agent and an attacking agent in a game confrontation scenario, and initialize the strategy neural network of the hunting agent and the initial state of the attacking agent, wherein both the hunting agent and the attacking agent perform the task of searching and destroying the enemy; A training module 804 is used to select actions of the hunting agent using the policy function of the policy neural network in the Markov decision making process and select actions of the attacking agent based on the preset combat strategy. After the hunting agent and the attacking agent respectively perform the selected actions, the reward obtained by the hunting agent is determined and the action value function and state value function of the policy neural network are updated based on the reward to train the policy neural network. A testing module 806 is used to migrate the policy neural network to a game confrontation scenario for testing when the target loss function of the policy neural network converges, wherein the target loss function is determined according to the policy function, the action value function, and the state value function; The determination module 808 is used to determine whether the strategy neural network has achieved the cluster collaborative hunting of the hunting agents when the strategy neural network hunting agents meet the cluster collaborative hunting requirements in the scenario test.

[0108] Optionally, the above-mentioned construction initialization module 802 constructs a hunting agent, including: constructing a Markov model of the hunting agent, including the input state, output action and reward function, and discount coefficient of the hunting agent.

[0109] Optionally, the above-mentioned construction initialization module 802 constructs an offensive intelligent agent, including: constructing an offensive intelligent agent based on a preset combat strategy, determining the input state of the offensive intelligent agent, wherein the input state of the offensive intelligent agent includes the position information of each offensive intelligent agent, the distance matrix between each offensive intelligent agent, the position information of all hunting intelligent agents detected by the offensive intelligent agent detector, and the distance matrix between each offensive intelligent agent and all hunting intelligent agents.

[0110] Optionally, the above-mentioned construction initialization module 802 initializes the strategy neural network of the hunting agent, including: constructing a strategy neural network including an input layer, a feature encoding module, an action generation module and an output layer; initializing the strategy function, action value function, state value function and target loss function of the strategy neural network.

[0111] Optionally, the above-mentioned training module 804 uses the strategy function of the strategy neural network to select the action of the hunting agent and selects the action of the attacking agent based on the preset combat strategy, including: based on the input state of the hunting agent, using the strategy function to determine the output action of the hunting agent, wherein the output action of the hunting agent indicates the target position of the hunting agent at the next moment; based on the input state of the attacking agent, using the preset control strategy to determine the output action of the attacking agent, wherein the output action of the attacking agent indicates the target position of the attacking agent at the next moment.

[0112] Optionally, the above-mentioned training module 804 trains the strategy neural network, including: saving the current state, execution action, traction reward and next state of the hunting agent as experience in a shared experience pool; when the amount of experience in the shared experience pool reaches a preset training amount, the hunting agent extracts samples from the shared experience pool in batches for learning, and updates the strategy function, action value function and state value function of the strategy neural network.

[0113] Optionally, the above-mentioned training module 804 updates the policy function, action value function and state value function of the policy neural network, including: determining the update amplitude of the policy neural network according to the current training effect; updating the policy function, action value function and state value function of the policy neural network according to the update amplitude.

[0114] Optionally, the above-mentioned determination module 808 determines whether the strategy neural network hunting agent meets the cluster collaborative hunting requirements in the scenario test, including: when the hunting win rate of the hunting agent exceeds the win rate threshold, the loss rate of the hunting agent is lower than the loss threshold, and the hit rate of the hunting agent is higher than the hit threshold, determining whether the strategy neural network hunting agent meets the cluster collaborative hunting requirements in the scenario test.

[0115] Optionally, the above-mentioned reinforcement learning-based cluster collaborative hunting system also includes a parameter module, which is used to calculate the single-step calculation time of the hunting agent after the strategy neural network hunting agent meets the cluster collaborative hunting requirements in the scenario test; and determine the strategy neural network hardware carrier parameters based on the single-step calculation time and the preset time requirements.

[0116] In an embodiment of the present application, a hunting agent and an attacking agent are constructed in a game confrontation scenario, and the strategy neural network of the hunting agent and the initial state of the attacking agent are initialized; in Markov decision making, the strategy function of the strategy neural network is used to select the action of the hunting agent and the action of the attacking agent is selected based on the preset battle strategy. After the hunting agent and the attacking agent each execute the selected action, the reward obtained by the hunting agent is determined and the action value function and state value function of the strategy neural network are updated based on the reward to train the strategy neural network; when the target loss function of the strategy neural network converges, the strategy neural network is migrated to the game confrontation scenario for testing, and when the hunting agent based on the strategy neural network meets the cluster collaborative hunting requirements in the scenario test, it is determined that the strategy neural network realizes the cluster collaborative hunting of the hunting agent, thereby formally representing the zero-sum game model of the finite time domain and finite state action space under the condition of variable scale of the two game parties based on the Markov decision process, focusing on the reward change during the state transfer process, effectively solving the reward sparseness problem in model training, solving the technical problem of solving the mixed Nash equilibrium strategy of the hunting parties under asymmetric conditions, and achieving the technical effect of ensuring the optimality of the decision-making of the two game parties. Example 3

[0117] Another aspect of the embodiments of the present invention further provides an electronic device for implementing the above-mentioned cluster collaborative hunting method based on reinforcement learning, and the electronic device is not limited to being a terminal device or a server in a radar system. The electronic device is not limited to including a memory and a processor, the memory stores a computer program, and the processor is configured to execute the steps in any of the above-mentioned method embodiments through the computer program. Example 4

[0118] Another aspect of the embodiments of the present invention provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method provided in various optional embodiments of the cluster collaborative hunting method based on reinforcement learning. The computer program is configured to execute the steps of any of the above method embodiments when running.

Claims

1. A cluster collaborative hunting method based on reinforcement learning, characterized in that: include: Constructing a hunting agent and an attacking agent in a game confrontation scenario, and initializing the strategy neural network of the hunting agent and the initial state of the attacking agent, wherein both the hunting agent and the attacking agent perform the task of searching and destroying the enemy; In Markov decision making, the action selection of the hunting agent is performed using the strategy function of the strategy neural network, and the action selection of the attacking agent is performed based on the preset combat strategy. After the hunting agent and the attacking agent each perform the selected action, the reward obtained by the hunting agent is determined, and the action value function and the state value function of the strategy neural network are updated based on the reward, so as to train the strategy neural network. When the target loss function of the strategy neural network converges, the strategy neural network is migrated to a game confrontation scenario for testing, wherein the target loss function is determined according to the strategy function, the action value function, and the state value function; When the hunting agent meets the cluster collaborative hunting requirements in the scenario test based on the strategy neural network, it is determined that the strategy neural network realizes the cluster collaborative hunting of the hunting agent.

2. The cluster collaborative hunting method based on reinforcement learning as claimed in claim 1, characterized in that: Build a hunting agent, including: Construct a Markov model of the hunting agent, including the hunting agent's input state, output action, reward function, and discount coefficient.

3. The cluster collaborative hunting method based on reinforcement learning as claimed in claim 1, characterized in that: Build an offensive agent, including: An offensive agent based on a preset combat strategy is constructed, and the input state of the offensive agent is determined, wherein the input state of the offensive agent includes the position information of each offensive agent, the distance matrix between each offensive agent, the position information of all hunting agents detected by the offensive agent detector, and the distance matrix between each offensive agent and all hunting agents.

4. The cluster collaborative hunting method based on reinforcement learning as claimed in claim 1, characterized in that: Initializing the strategy neural network of the hunting agent includes: Construct a policy neural network including an input layer, a feature encoding module, an action generation module, and an output layer; Initialize the policy function, action value function, state value function and target loss function of the policy neural network.

5. The cluster collaborative hunting method based on reinforcement learning as claimed in claim 1, characterized in that: The strategy function of the strategy neural network is used to select the action of the hunting agent and the action of the attacking agent is selected based on the preset combat strategy, including: Based on the input state of the hunting agent, the output action of the hunting agent is determined by using the strategy function, wherein the output action of the hunting agent indicates the target position of the hunting agent at the next moment; The output action of the offensive agent is determined based on the input state of the offensive agent using a preset comparison strategy, wherein the output action of the offensive agent indicates the target position of the offensive agent at the next moment.

6. The cluster collaborative hunting method based on reinforcement learning as claimed in claim 1, characterized in that: Training the strategy neural network includes: The current state, execution action, traction reward and next state of the hunting agent are stored as experience in a shared experience pool; When the amount of experience in the shared experience pool reaches the preset training amount, the hunting agent extracts samples from the shared experience pool in batches for learning, and updates the policy function, action value function and state value function of the policy neural network.

7. The cluster collaborative hunting method based on reinforcement learning as claimed in claim 6, characterized in that: Update the policy function, action value function, and state value function of the policy neural network, including: Determine the update amplitude of the strategy neural network according to the current training effect; The policy function, action value function and state value function of the policy neural network are updated according to the update amplitude.

8. The cluster collaborative hunting method based on reinforcement learning as claimed in claim 1, characterized in that: Based on the strategy neural network, the hunting agent meets the cluster collaborative hunting requirements in the scenario test, including: When the hunting win rate of the hunting agent exceeds the win rate threshold, the loss rate of the hunting agent is lower than the loss threshold, and the hit rate of the hunting agent is higher than the hit threshold, it is determined that the hunting agent meets the cluster collaborative hunting requirements in the scenario test based on the strategy neural network.

9. The cluster collaborative hunting method based on reinforcement learning as claimed in claim 8, characterized in that: After the hunting agent in the strategy neural network meets the cluster collaborative hunting requirements in the scenario test, the method further includes: Calculate the time required for a single step of the hunting agent; The strategy neural network hardware carrier parameters are determined based on the single-step calculation time and the preset time requirement.

10. A cluster collaborative hunting system based on reinforcement learning, characterized in that: include: Constructing an initialization module, which is used to construct a hunting agent and an attacking agent in a game confrontation scenario, and initialize the strategy neural network of the hunting agent and the initial state of the attacking agent, wherein both the hunting agent and the attacking agent perform the task of searching and destroying the enemy; A training module, for selecting actions of a hunting agent using the strategy function of the strategy neural network and selecting actions of an attacking agent based on a preset combat strategy in a Markov decision, and after the hunting agent and the attacking agent each perform the selected actions, determining a reward obtained by the hunting agent and updating an action value function and a state value function of the strategy neural network based on the reward, so as to train the strategy neural network; A testing module, used for migrating the strategy neural network to a game confrontation scenario for testing when the target loss function of the strategy neural network converges, wherein the target loss function is determined according to the strategy function, the action value function, and the state value function; A determination module is used to determine that the strategy neural network has achieved the cluster collaborative hunting of the hunting agent when the hunting agent meets the cluster collaborative hunting requirements in the scenario test based on the strategy neural network.

Images