A multi-unmanned agent-oriented cooperative fire attack strategy generation method

Abstract

The application provides a multi-unmanned agent-oriented cooperative firepower attack strategy generation method, and belongs to the field of intelligent game strategy generation. The method comprises the following steps: constructing an adversarial environment, generating an initial strategy population, performing parallel calculation on the fitness of the strategy population, performing crossover and mutation on the strategy population, and iteratively evolving the strategy population. The application solves the problems of large calculation amount and long training time of the intelligent game strategy generation method, represents the strategy as an action sequence, constructs fitness based on game winning rate, uses parallel multi-opening evolutionary calculation method, and intelligently, automatically and efficiently generates a cooperative firepower attack strategy with the fitness as the optimization target.

Description

Technical Field

[0001] This invention relates to a method for generating collaborative firepower attack strategies for multiple unmanned agents, belonging to the field of intelligent game strategy generation. Background Technology

[0002] Intelligent game strategy generation uses artificial intelligence to generate action control. Compared to traditional methods that construct logical rules based on expert knowledge, it can automatically generate control actions through interaction with the environment, effectively reducing bias caused by human intervention and improving performance. In complex game scenarios, especially in multi-player games, the high-dimensional action space directly leads to a significant increase in the length of action sequences, thereby increasing the difficulty of game strategy generation and optimization.

[0003] Patent application CN202110728217.4, entitled "A Satellite Communication Anti-interference Method Based on Zero-Sum Game Theory," uses a zero-sum game method to generate anti-interference strategies. This method is based on static game theory, and its theoretical foundation is geared towards static environments with complete information. However, the effectiveness of the strategy is limited in real-world dynamic scenarios with incomplete information.

[0004] The Deep Q-Learning framework used in the patent application CN202110728217.4, entitled "A Routing Method for Wireless Ad Hoc Network Devices Based on Deep Q-Learning", is a typical intelligent method for solving the policy generation problem. However, this method has the problem of overestimating the value of actions.

[0005] "QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning" is a representative method for solving the partially observable dynamic cooperative firepower problem, utilizing value decomposition to address the cooperation between individuals and the whole. However, compared to Deep Q-Learning, this method produces a more complex network result.

[0006] "Evolution Strategies as a Scalable Alternative to Reinforcement Learning" utilizes deep reinforcement learning and evolutionary computation methods to generate parameterized strategies represented by neural network parameters through interaction, effectively producing game strategies in a perfectly information-based steady-state environment. However, the poor interpretability of neural networks themselves makes it difficult to choose the appropriate parameter dimensions. Selecting too many neuron parameters can hinder strategy generation. Summary of the Invention

[0007] To address the problems existing in the prior art, this invention provides a method for generating cooperative firepower strike strategies for multiple unmanned agents. This method addresses the issues of high computational cost and long training time in intelligent game strategy generation methods. It represents strategies as action sequences, constructs fitness based on game win rates, and utilizes parallel multi-opening modified evolutionary computation methods. With fitness as the optimization objective, it generates game strategies intelligently, automatically, and efficiently.

[0008] The technical solution adopted in this invention is as follows:

[0009] A method for generating cooperative fire strike strategies for multiple unmanned agents includes the following steps:

[0010] Step 1: Constructing the adversarial environment;

[0011] Step 2: Generate the initial policy population;

[0012] Step 3: Parallel computation of strategy population fitness;

[0013] Step 4: Strategy population crossover and mutation;

[0014] Step 5: Iterative evolution of the strategy population yields the final coordinated firepower strike strategy.

[0015] Furthermore, in step 1, the specific method for constructing the adversarial environment is as follows:

[0016] Step 1-1: Set up a scenario where each side has K adversarial entities;

[0017] Steps 1-2: Each adversary entity can perform Num_A dimensional actions, including movement actions and firepower actions; movement actions are "stationary", "up", "down", "left" and "right", a total of 5 actions; firepower actions are to attack the k-th enemy target, a total of K actions;

[0018] Steps 1-3: Load the enemy's preset firepower strike strategy;

[0019] Steps 1-4: The longest duration of the confrontation is Max_Step, and the win rate is returned based on the survival of the unmanned agent.

[0020] Furthermore, in step 2, the strategy consists of actions, and the strategy population X consists of N strategy individuals I. Each strategy individual I consists of an action sequence with a length of Max_Step, where Max_Step is the maximum running step size of the game scenario. The value range of each action is [0, Num_A-1]. The initial population strategy will randomly select values ​​for the actions in each action sequence of each population within the range. The strategy population X is mathematically represented as a two-dimensional matrix with dimension N×Max_Step.

[0021] Furthermore, in step 3, the specific method for parallel computation of the strategy population fitness is as follows:

[0022] Step 3-1: Constructing a Parallel Multi-Open Game Environment

[0023] Construct multiple threads L, each thread is independent of the others, and each thread runs a game program based on different individual strategies independently;

[0024] Step 3-2: Construct a game theory program based on the Monte Carlo experiment

[0025] Invoke the game simulation scenario, each of our unmanned agents executes actions according to the action sequence of strategy individual I; set the number of experiments Num_Episode, count the number of wins Num_Win, calculate the game win rate h = Num_Win / Num_Episode, and define the win rate as fitness; in the parallel multi-open game process, record the longest number of times the action is executed;

[0026] Step 3-3: Strategy Population Ranking Update

[0027] After parallel execution is completed, the fitness of the action sequence corresponding to each population is collected, and the policy population X is sorted from high to low fitness.

[0028] Furthermore, in step 4, the specific method of crossover mutation in the strategy population is as follows:

[0029] Step 4-1: Calculation of population selection probability

[0030] Normalize the individual win rates based on the overall win rate, and the population selection probability of the nth strategy individual I is:

[0031]

[0032] Step 4-2: Select two individuals I1 and I2 from the population according to the population selection probability;

[0033] Step 4-3: Strategy Individual Crossover

[0034] Randomly select an integer m from [0, Max_Step-1], and concatenate I1 and I2 at position m to form two new individuals I′1 and I′2:

[0035] I′1=[I1[0:m-1],I2[m:Max_Step-1]]

[0036] I′2=[I2[0:m-1],I1[m:Max_Step-1]]

[0037] Step 4-4: Strategy Individual Variation

[0038] Mutation operations are performed sequentially at each position along the action sequence of individuals I1 and I2; a random number is drawn from [0,1] and it is determined whether it is greater than the threshold value e. If it is greater than e, an integer is randomly selected from [0,Num_A-1] to replace the original action value; if it is less than ε, no operation is performed.

[0039] Steps 4-5: New Strategy Population Generation

[0040] Add individuals I1 and I2 to the new population, and repeat steps 4-2, 4-3 and 4-4 until the number of individuals in the new strategy population X′ reaches N, and then replace the original strategy population X.

[0041] Furthermore, in step 5, the specific method of iterative evolution of the strategy population is as follows:

[0042] Step 5-1: Set the adaptation threshold and the maximum number of evolutions, Num_Evolution;

[0043] Step 5-2: Repeat steps 3 and 4 to obtain the maximum game win rate for each cycle; if the number of cycles is less than the maximum number of evolutions Num_Evolution, and it is greater than the fitness threshold... Then stop evolution and output the maximum fitness value; if it is less than or equal to the fitness threshold... Then continue to evolve. When the number of iterations equals the maximum number of evolutions Num_Evolution, the evolution stops and the maximum fitness value is output. The strategy individual corresponding to the maximum fitness value is the final cooperative firepower strike strategy.

[0044] The beneficial effects of this invention are as follows:

[0045] 1. This invention addresses the problems of large computational load and long training time in intelligent game strategy generation methods. It represents the strategy as an action sequence, constructs fitness based on the game win rate, and uses parallel multi-opening to modify the evolutionary computation method. With fitness as the optimization objective, it generates game strategies intelligently, automatically, and efficiently.

[0046] 2. Compared with typical methods, the present invention has significant advantages and can realize the intelligent generation of adversarial strategies in dynamic scenarios. Attached Figure Description

[0047] Figure 1 This is a flowchart of the present invention.

[0048] Figure 2 The curve shows the effect of the countermeasure. Detailed Implementation

[0049] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0050] A method for generating cooperative firepower strike strategies for multiple unmanned agents, the overall process of which is as follows: Figure 1 As shown.

[0051] Specifically, the steps include the following:

[0052] In step 1, the adversarial environment is constructed:

[0053] Step 1-1: In the "3m" adversarial map scenario of "StarCraft II", set each side to have K=3 adversarial entities.

[0054] Steps 1-2: Each adversary entity can perform 3D actions, including movement actions and attack actions. Movement actions consist of five actions: "Stay," "Up," "Down," "Left," and "Right." Attack actions involve targeting the k-th enemy target, with a total of K actions. Therefore, Num_A = 5 + K.

[0055] Steps 1-3: Load the enemy's preset firepower attack strategy.

[0056] Steps 1-4: The maximum duration of the confrontation is Max_Step = 60 steps, and the win rate is returned based on the survival of the unmanned agent.

[0057] Step 2: Initial strategy population generation:

[0058] A strategy consists of actions. The strategy population X comprises N policy individuals I, where N = 10. Each policy individual I consists of a sequence of actions of length Max_Step, where Max_Step is the maximum step size in the game. The value of each action ranges from [0, Num_A-1], where Num_A = 8. The initial population policy will randomly select values ​​from the range for each action in each action sequence within each population. The strategy population X is mathematically represented as a two-dimensional matrix of dimension N × Max_Step.

[0059] Step 3: Parallel computation of strategy population fitness

[0060] Step 3-1: Constructing a Parallel Multi-Open Game Environment

[0061] Construct multiple threads L, N=L, each thread is independent of the others, and each thread runs a game program based on different individual strategies independently;

[0062] Step 3-2: Construct a game theory program based on the Monte Carlo experiment

[0063] A game simulation scenario is invoked, with each of our unmanned agents following the action sequence of individual I under strategy. The number of experiments is set to Num_Episode = 20, the number of wins is recorded as Num_Win, and the game win rate h = Num_Win / Num_Episode is calculated, defined as the fitness. Furthermore, during the parallel multi-instance game process, the longest number of times an action is executed is recorded.

[0064] Step 3-3: Strategy Population Ranking Update

[0065] After parallel execution is completed, the fitness of the action sequence corresponding to each population is collected, and the policy population X is sorted from high to low fitness.

[0066] Step 4: Strategy Population Crossover and Mutation

[0067] Step 4-1: Calculation of population selection probability

[0068] Normalize the individual win rate based on the overall win rate. For the nth individual I with the correct strategy, its population selection probability is:

[0069]

[0070] Step 4-2: Select two individuals, I1 and I2, from the population according to the population selection probability;

[0071] Step 4-3: Strategy Individual Crossover

[0072] Randomly select an integer m from [0, Max_Step-1], and concatenate I1 and I2 at position m to form two new individuals I′1 and I′2:

[0073] I′1=[I1[0:m-1],I2[m:Max_Step-1]]

[0074] I′2=[I2[0:m-1],I1[m:Max_Step-1]]

[0075] Step 4-4: Strategy Individual Variation

[0076] Mutation operations are performed sequentially at each position along the action sequence of individuals I1 and I2. A random number is drawn from [0,1] and checked against a threshold ε = 0.9. If the number is greater than ε, an integer is randomly selected from [0,Num_A-1] to replace the original action value; otherwise, no operation is performed.

[0077] Steps 4-5: New Strategy Population Generation

[0078] Add individuals I1 and I2 to the new population, and repeat steps 4-2, 4-3 and 4-4 until the number of individuals in the new strategy population X′ reaches N, and then replace the original strategy population X.

[0079] Step 5: Iterative Evolution of the Strategy Population

[0080] Step 5-1: Set the adaptation threshold The maximum number of evolutions, Num_Evolution, is 20.

[0081] Step 5-2: Repeat steps 3 and 4 to obtain the maximum win rate for each cycle. If the number of cycles is less than the maximum number of evolutions (Num_Evolution), then if it exceeds the fitness threshold... Then stop evolution and output the maximum fitness value; if it is less than or equal to the fitness threshold... Then continue evolving. When the number of iterations equals the maximum number of evolutions Num_Evolution, evolution stops, and the maximum fitness value is output.

[0082] Figure 2 The adversarial effect curve shows that this method has significant advantages over typical methods, indicating that the present invention can achieve intelligent generation of adversarial strategies in dynamic scenarios.

Claims

1. A multi-unmanned agent oriented cooperative fire attack strategy generation method, characterized in that, Includes the following steps: Step 1: Constructing the adversarial environment; the specific method is as follows: Step 1-1: Set up two opposing entities in the scene, one for each side K of the conflict. Step 1-2: Each opposing entity can perform movement actions including movement type actions and fire attack type actions; the movement type actions are "still", "up", "down", "left" and "right", a total of 5 actions; the fire attack type actions are attacks on the k first, second, third, fourth and fifth enemy targets respectively, a total of K actions; Steps 1-3: Load the enemy's preset firepower strike strategy; Steps 1-4: The longest duration of the confrontation is Max_Step, and the win rate is returned based on the survival of the unmanned agent; Step 2: generate an initial strategy population; a strategy is composed of actions, the strategy population X is composed of N strategy individuals I , each strategy individual I is composed of an action sequence, the length of the action sequence is Max_Step, where Max_Step is the maximum running step length of the game scene; the value range of each action is ; the initial population strategy will randomly take values in the range for the actions in each action sequence in each population; the strategy population X is expressed mathematically as a two-dimensional matrix with a dimension of ; Step 3: Parallel computation of strategy population fitness; the specific method is as follows: Step 3-1: Constructing a Parallel Multi-Open Game Environment Constructing multiple threads L Each thread is independent of each other, and each thread runs the game program based on different strategies of individuals independently. Step 3-2: Construct a game theory program based on the Monte Carlo experiment Call the game experiment simulation scene, the agents of our side execute actions according to the action sequence of the strategy individual I ; Set the number of experiments Num_Episode, count the number of wins Num_Win, and calculate the game winning rate And the winning rate is defined as the fitness; In the process of parallel multi-opening game, record the longest action execution times; Step 3-3: Strategy Population Ranking Update After parallel execution is complete, the fitness of the action sequence corresponding to each population is collected, and the policy populations are ranked from high to low fitness. X Sort; Step 4: Strategy population crossover and mutation; The specific method is as follows: Step 4-1: Calculation of population selection probability Normalize the individual win rates based on the overall win rate, the th n individual strategies I The population selection probability is ； Step 4-2: Select two individuals from the population based on the population selection probability. and ; Step 4-3: Strategy Individual Crossover from Randomly select an integer m from the sample, and then... and The pieces are joined at position m to form... and Two new individuals: ； ； Step 4-4: Strategy Individual Variation Along individual and The action sequence direction sequentially performs mutation operations on each position; Draw a random number from [0,1] and determine if it is greater than the threshold value. If greater than Then from Randomly select an integer to replace the original action value; if it is less than If so, no action will be taken; Steps 4-5: New Strategy Population Generation Individual and Add to the new population, and repeat steps 4-2, 4-3, and 4-4 until a new strategy population is reached. The number of individuals reached N and replace the original strategy population. X ; Step 5: Iterative evolution of the strategy population yields the final coordinated firepower strike strategy; the specific method is as follows: Step 5-1: Set the adaptation threshold and the maximum number of evolutions, Num_Evolution; Step 5-2: Repeat steps 3 and 4 to obtain the maximum game win rate for each cycle; if the number of cycles is less than the maximum number of evolutions Num_Evolution, and it is greater than the fitness threshold... If the fitness value is less than or equal to the fitness threshold, then evolution stops and the maximum fitness value is output; otherwise, evolution stops. If the number of iterations equals the maximum number of iterations Num_Evolution, then evolution continues. When the number of iterations equals the maximum number of iterations Num_Evolution, evolution stops and the maximum fitness value is output. The strategy individual corresponding to the maximum fitness value is the final cooperative firepower strategy.

Images