Policy model of instant policy game and training method thereof

Abstract

The invention discloses a strategy model of a real-time strategy game. The strategy model comprises the following steps: describing an RTS game as a Markov game form; the environment size # imgabs0 #, the environment size # imgabs1 # and the environment size # imgabs2 # respectively represent the height and the width of the grid; the state space is # imgabs3 #; a global observation space is # imgabs4 #, # imgabs5 # represents a real number space, and # imgabs6 # represents a feature channel dimension; the joint action space is # imgabs7 # and # imgabs8 # and represents the action dimension of the control unit; the state transfer function is # imgabs9 #; the reward function is # imgabs10 #; the strategy is # imgabs11 #; the strategy model is # imgabs12 #. According to the method, the problems existing in the prior art are solved, training can be carried out with lower computing resource expenditure through innovation of a strategy model architecture and a training method, and meanwhile the method can adapt to setting of various RTS games.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology applied to electronic games, and in particular to a strategy model for real-time strategy games and a training method thereof. Background Art

[0002] Real-Time Strategy (RTS) games are a type of video game that is based on real-time operations (rather than turn-based) and combines strategic and tactical decision-making. In this type of game, players usually control resource collection, unit production, technology research and development, and command individual units or entire armies to fight against hostile forces.

[0003] Deep Reinforcement Learning (DRL) is a subfield of machine learning that combines the methods of deep learning and reinforcement learning. Deep learning enables computers to recognize complex patterns and features from data through neural networks, while reinforcement learning enables computers to learn how to make optimal decisions through interaction with the environment. The purpose of DRL is to allow computers to autonomously learn how to perform better in specific tasks, that is, to make optimal decisions, through a mechanism of rewards and penalties without explicit instructions.

[0004] Due to its complexity and high uncertainty, RTS games have always been an important challenge for the research process of artificial intelligence. RTS games usually involve multiple players in a dynamic and continuous environment to manage resources, control units, and plan tactics. Therefore, applying DRL in RTS games requires dealing with huge state spaces and action spaces, as well as multi-objective decision-making problems.

[0005] In the prior art, AlphaStar is an artificial intelligence program developed by DeepMind specifically to play the video game StarCraft II (a real-time strategy game produced and published by Blizzard Entertainment). AlphaStar uses a special deep reinforcement learning architecture that is capable of learning and making decisions in very complex game environments. It learns and improves rapidly by playing against different versions of itself. In 2019, AlphaStar successfully defeated professional players in a closed environment and achieved grandmaster status in the StarCraft II leaderboard.

[0006] Although AlphaStar has made impressive achievements, showing a high level of strategy and execution in matches against human professional players, it also has some shortcomings, mainly including: (1) Huge demand for computing resources: AlphaStar training requires a large amount of computing resources. Thousands of CPUs and GPUs / TPUs are used to train strategies for a long time. This resource-intensive training makes the technology difficult to popularize and apply in resource-constrained situations.

[0007] (2) Limited adaptability: The strategies used during AlphaStar training are optimized for specific game versions and maps, and therefore may not adapt well to changes in game rules or the emergence of new maps.

[0008] (3) Overspecialization: AlphaStar’s focus on training and optimizing in one game, StarCraft II, may have led to its overspecialization in that game and its lack of generalization ability to adapt to different types of tasks or environments. Summary of the invention

[0009] The purpose of the present invention is to provide a strategy model for real-time strategy games and a training method thereof, so as to solve the problems existing in the prior art. Through the innovation of the strategy model architecture and the training method, training can be performed with lower computing resource overhead, and at the same time, it can adapt to the settings of various RTS games.

[0010] In order to achieve the above object, one of the solutions of the present invention is: A strategy model for real-time strategy games, describing RTS games in the form of Markov games; the environment size is expressed as ,in and Represent the height and width of the grid respectively; the state space is represented as ; The global observation space is expressed as ,in represents the real number space, represents the feature channel dimension; the joint action space is represented as ,in represents the action dimension of the control unit; the state transfer function is expressed as ; The reward function is expressed as ; The strategy is expressed as ; The policy model is expressed as ; The time step Global observation As input, the policy model The output time step The joint action distribution of , sampling is performed on the joint action distribution, and the result obtained is at this time step Joint action ; Define the game world as A two-dimensional grid; The global observation is represented as Each feature channel represents the game world at any two-dimensional grid position. A game attribute on and Respectively represent the horizontal and vertical coordinates of the game world; The joint action distribution is expressed as Action diagram, Representation strategy model The output is located in a two-dimensional grid The action distribution of the control units on Calculated at time step The joint action mask ; For any 2D grid position :If the 2D grid position There is no controlled unit belonging to this faction, action mask is an all-zero vector, indicating that all actions cannot be performed; if the two-dimensional grid position There is a controlled unit belonging to this faction, action mask The position with 0 in the byte indicates that the control unit cannot perform the corresponding action, and the position with 1 indicates that the control unit can perform the corresponding action; For policy models The artificial neural network includes encoder-decoder network, long short-term memory network and multi-layer fully connected network.

[0011] In some embodiments, the policy model In the above example, the encoder is a convolutional neural network composed of several convolutional layers and pooling layers; the global observation It is fed into the encoder and after passing through the convolutional layer and the pooling layer, a high-level feature map representation is generated; The long short-term memory network takes the high-level feature map representation of the current time step and the context feature map representation of the previous time step as input, and outputs the context feature map representation of the current time step; The decoder consists of several transposed convolutional layers; the transposed convolutional layers transform the context feature map representation back to the environment size The size of the output is The number of channels is calculated to obtain the joint action distribution ; The context feature map representation is used as the input of the multi-layer fully connected network, and the multi-layer fully connected network generates a real number as the estimated value of the environment state; The artificial neural network instantiates a parameterized policy function With value function ,in Represents the parameters of the artificial neural network.

[0012] In some embodiments, in the global observation, the game attributes defining the feature channel are: (1) For each combination of faction and each control unit category in the game, define a feature channel containing binary values. Each two-dimensional grid position of this feature channel The value of is 0 or 1; 1 represents the two-dimensional grid position There are control units of this type belonging to this faction on the grid. 0 indicates the two-dimensional grid position. There are no controlled units of this type belonging to this faction; (2) For the terrain attributes in the game, define a feature channel containing binary values. Each two-dimensional grid position of the feature channel The value of is 0 or 1; 1 represents the two-dimensional grid position Not passable, 0 indicates a two-dimensional grid position Passable; (3) For the health attributes of the control units of each faction in the game, a feature channel is defined. The value is a real number; if the 2D grid position If there is a control unit on the grid, the real number is the health of the control unit; if the two-dimensional grid position If there is no control unit, this real number is 0.

[0013] The second solution of the present invention is: A strategy model training method adopts a deep reinforcement learning method to train the strategy model of the real-time strategy game, including a reinforcement learning algorithm and a self-playing training process. The reinforcement learning algorithm is specifically: Set the value loss to ,in and Respectively represent when the input is a global observation and The value of the model output is estimated when Represents the time step Environmental rewards, represents the discount factor; Set the policy gradient to ,in Represents the time step The joint action Represents the time step The conditional distribution of the joint action of represents the gradient operator; During the training process, the data generated by the simulated battle over time is represented as follows: ; Under each batch of data, the stochastic gradient ascent optimization algorithm is used to update the artificial neural network parameters, and its calculation formula is expressed as: ; in , represents the learning rate, A coefficient representing the weight of the control value loss.

[0014] In some embodiments, during the self-playing training process, the strategy model in training is called a learner. The learner continuously updates its own strategy model parameters through an algorithm based on the data generated in the game. The strategy model that plays against the learner to generate the data required for the learner training is called a trainer. The trainer, as the opponent of the learner, has fixed strategy model parameters. During the self-play training process, the learner's strategy model and its parameters are saved periodically according to the preset duration, and the regularly saved strategy models constitute a trainer set; before the start of each round of competition, a number of trainers are selected from the trainer set to compete with the learner based on the number of players set in the game, and the data generated by the competition is used to train the learner; when selecting trainers, the upper confidence bound algorithm is used to balance the exploration and utilization of trainer selection.

[0015] Preferably, in the self-playing training process, the specific steps of selecting a trainer are: For every new trainer ,set up Indicates trainer The historical winning rate against the learner, Indicates trainer The number of times a trainer is selected is accumulated by 1 each time he is selected; each time a trainer is selected from the trainer set, it is done in the following way: ; in It means to select the trainer with the highest historical winning rate against the learner. is the total number of times a trainer is selected during the self-playing training process, It is a hyperparameter that controls the exploration ratio.

[0016] In some embodiments, the training method adopts a multi-threaded architecture, including a main thread for updating learner model parameters and multiple worker threads for collecting required training data. The workflow of each training cycle is as follows: (1) Each worker thread wakes up; the main thread is blocked, waiting for the worker thread to submit data; (2) Each worker thread selects a trainer from the trainer set according to the self-playing training process; (3) Each worker thread initializes the game environment, uses the learner and trainer to simulate and collect battle data in parallel until the end of this round of battle; at this stage, the model parameters of the global trainer are read-only and shared by each worker thread; (4) Each worker thread submits the battle data to the battle data pool; (5) The main thread wakes up; each worker thread is blocked; (6) The main thread updates the learner's model parameters using the data in the battle data pool according to the reinforcement learning algorithm; (7) The main thread saves the learner’s model parameters at a fixed period and adds new trainers to the trainer set; (8) Jump to step (1).

[0017] After adopting the above technical solution, the present invention has the following technical effects: The present invention adopts a strategy model with an encoder-decoder as the main architecture. The encoder implemented based on the convolutional network layer shares parameters in space, which helps the control units distributed in space to share the knowledge learned from exploration. In addition, the simpler artificial neural network design can achieve faster training, reducing the hyperparameter settings that need to be adjusted, which is conducive to reducing the cost of training strategy models. The centralized reinforcement learning method helps to learn strategic behaviors with the communication and collaboration capabilities of control units. The game world is normalized in a two-dimensional grid manner, and the joint action output of the control units is represented as a regular action graph, which can adapt to various RTS game settings, such as a variable number of control units, so as to achieve rapid development of a unified RTS game strategy model training framework. At the same time, the LSTM network layer is adopted, which is conducive to learning long-term dependencies, and the special gating mechanism in the LSTM can avoid the gradient explosion and gradient disappearance problems in long sequences. The training method of the present invention introduces an upper confidence bound algorithm in the self-playing learning process to balance the exploration and utilization in the problems selected by the trainer, so that the learner can learn stronger strategies by playing against stronger opponents and can also be exposed to diverse opponents to avoid overfitting. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 A schematic diagram of the system architecture of a strategy model according to a specific embodiment of the present invention; Figure 2 The training flow chart of a specific embodiment of the present invention. DETAILED DESCRIPTION

[0019] In order to further explain the technical solution of the present invention, the present invention is described in detail below through specific embodiments.

[0020] By establishing a mathematical model, the present invention describes the RTS game as a Markov game form, including: (I) Environmental size: ,in and Represents the height and width of the grid respectively.

[0021] (II) State space: .

[0022] (III) Global observation space: ,in represents the real number space, Represents the feature channel dimension.

[0023] (IV) Joint action space: ,in Represents the action dimension of the control unit.

[0024] (V) State transfer function: .

[0025] (VI) Reward function: .

[0026] (VII) Strategy: .

[0027] refer to Figure 1 As shown, the present invention discloses a strategy model for a real-time strategy game. , Strategy Model The input is the global observation , the output is the joint action distribution. Global observation As input, the policy model The output time step The joint action distribution of , sampling on the joint action distribution, we can obtain Joint action .

[0028] Define the game world as A two-dimensional grid.

[0029] The global observation is represented as Each feature channel represents the game world at any two-dimensional grid position. A game attribute on and Respectively represent the horizontal and vertical coordinates of the game world.

[0030] The joint action distribution is expressed as The action diagram of the action dimension is the size of the action set of the control unit (such as silence, movement, attack, mining, and other actions that may occur in the game). Representation strategy model The output is located in a two-dimensional grid The action distribution of the control units on .

[0031] Sampling directly from the joint action distribution may produce illegal actions, for example: , if the 2D grid position If there are no controlled units belonging to this faction, then is meaningless; or, at time step , located at the two-dimensional grid position Certain actions of the units controlled by this faction cannot be performed. In order to ensure that the sampling of the joint action distribution always produces legal actions, the joint action mask is needed. Calculated at time step The joint action mask ; For any 2D grid position :If the 2D grid position There is no controlled unit belonging to this faction, action mask is an all-zero vector, indicating that all actions cannot be performed; if the two-dimensional grid position There is a controlled unit belonging to this faction, action mask The position of 0 in the byte indicates that the control unit cannot perform the corresponding action, and the position of 1 indicates that the control unit can perform the corresponding action.

[0032] An example of the calculation of the joint action mask is as follows: When the two-dimensional grid position If there is a controlled unit belonging to this faction, The calculation process is as follows, assuming that the integer represents the action "move one unit distance north", then when the two-dimensional grid position When walking and no other controlled units ,otherwise ; Assume integer represents the action "attack the target at offset (1,3)", then when the two-dimensional grid position When there is an enemy control unit ,otherwise ; In the strategy model Output joint action distribution Then, using the joint action mask , we can ignore illegal actions and only sample legal actions.

[0033] The artificial neural network used for the policy model includes an encoder-decoder network, a long short-term memory (LSTM) network, and a multi-layer fully connected network, with EDN as the main body. Specifically: a. The encoder is a convolutional neural network (CNN) consisting of several convolution layers and pooling layers; global observation After being fed into the encoder and passing through the convolutional and pooling layers, a high-level feature map representation with more channels and smaller size is generated. According to the representation learning theory of artificial neural networks, the generated high-level feature map representation captures the spatial correlation of the game world, thereby generating a stronger feature representation, which is conducive to learning better strategic behaviors, such as communication and collaboration between control units.

[0034] b. Due to the complexity of RTS games, the state of the environment depends not only on the current global observation, but also on past strategic behaviors. In order to learn this long-term dependency, a Long Short-Term Memory (LSTM) network is added to the policy model. LSTM takes the high-level feature map representation of the current time step and the context feature map representation of the previous time step as input, and outputs the context feature map representation of the current time step. Since the context feature map representation is continuously transmitted with time steps, the context feature map representation of the current time step contains the information of all historical time steps, so that the policy model can better understand and predict the dynamic changes of the environment.

[0035] c. The decoder consists of several transposed convolution layers; the transposed convolution layer transforms the context feature map representation back to the environment size The size of the output is The number of channels, thus obtaining the joint action distribution .

[0036] In the current mainstream Actor-Critic reinforcement learning algorithm, in addition to the policy model, it is also necessary to estimate the value corresponding to the environment state. Since the aforementioned context feature graph representation contains information of all historical time steps, the context feature graph representation is used as the input of the multi-layer fully connected network, and the multi-layer fully connected network generates a real number as the estimated value of the environment state.

[0037] The above artificial neural network instantiates a parameterized policy function With value function ,in Represents the parameters of the artificial neural network.

[0038] Through the above scheme, the present invention adopts a strategy model with an encoder-decoder as the main architecture. The encoder implemented based on the convolutional network layer shares parameters in space, which helps the control units distributed in space to share the knowledge learned from exploration. Moreover, the simpler artificial neural network design can achieve faster training, reducing the hyperparameter settings that need to be adjusted, which is conducive to reducing the cost of training strategy models; the centralized reinforcement learning method helps to learn strategic behaviors with control unit communication and collaboration capabilities; the game world is normalized in a two-dimensional grid manner, and the joint action output of the control unit is represented as a regular action graph, which can adapt to various RTS game settings, such as a variable number of control units, thereby realizing the rapid development of a unified RTS game strategy model training framework; at the same time, the use of the LSTM network layer is conducive to learning long-term dependencies, and the special gating mechanism in the LSTM can avoid the gradient explosion and gradient disappearance problems in long sequences.

[0039] A specific embodiment of the above-mentioned policy model is shown below.

[0040] In the above global observation, the game attributes that define the feature channel are: (1) For each combination of faction and each control unit category in the game, define a feature channel containing binary values. Each two-dimensional grid position of this feature channel The value of is 0 or 1; 1 represents the two-dimensional grid position There are control units of this type belonging to this faction on the grid. 0 indicates the two-dimensional grid position. There are no controlled units of this type belonging to this faction; (2) For the terrain attributes in the game, define a feature channel containing binary values. Each two-dimensional grid position of the feature channel The value of is 0 or 1; 1 represents the two-dimensional grid position Not passable, 0 indicates a two-dimensional grid position Passable; (3) For the health attributes of the control units of each faction in the game, a feature channel is defined. The value is a real number; if the 2D grid position If there is a control unit on the grid, the real number is the health of the control unit; if the two-dimensional grid position If there is no control unit, this real number is 0.

[0041] In addition, according to different game settings, more attributes that are helpful for strategic decision-making can be discovered and similar feature channels can be defined.

[0042] See also Figure 2 The present invention also discloses a method for training a strategy model, which uses a deep reinforcement learning method to train the above-mentioned strategy model, including a reinforcement learning algorithm and a self-playing training process: The specific reinforcement learning algorithm is: Set the value loss to ,in and Respectively represent when the input is a global observation and The value of the model output is estimated when Represents the time step Environmental rewards, Represents the discount factor, whose value range is [0,1), which discounts the forward reward; Set the policy gradient to ,in Represents the time step The joint action Represents the time step The conditional distribution of the joint action of represents the gradient operator; During the training process, the data generated by the simulated battle over time is represented as follows: ; In each batch of data, the Stochastic Gradient Descent (SGD) optimization algorithm is used to update the parameters of the artificial neural network. The calculation formula is expressed as: ; in , represents the learning rate, A coefficient representing the weight of the control value loss.

[0043] In RTS games, there are usually multiple players playing against each other in a game world at the same time. Self-play is a popular technique for training game-playing AI using reinforcement learning. In a self-play setting, the AI ​​system plays against a copy of itself instead of against a human or a preset opponent. This approach can generate a large amount of training data without external data or guidance, and enables AI to surpass existing human game theory. The specific process of self-play training is: In the process of self-playing training, the strategy model in training is called a learner. The learner continuously updates its own strategy model parameters through algorithms based on the data generated in the game. The strategy model that plays against the learner to generate the data required for the learner training is called a trainer. As the learner's opponent, the trainer has fixed strategy model parameters (i.e., fixed game strategy). During the self-play training process, the learner's strategy model and its parameters are saved periodically according to the preset duration, and the regularly saved strategy models constitute a trainer set; before the start of each round of competition, a number of trainers are selected from the trainer set to compete with the learner according to the number of players set in the game, and the data generated by the competition is used to train the learner; when selecting trainers, the upper confidence bound (UCB) algorithm is used to balance the exploration and utilization of trainer selection.

[0044] Through the above scheme, the training method of the present invention introduces an upper confidence bound algorithm in the self-playing learning process to balance the exploration and utilization in the problems selected by the trainer, so that the learner can learn stronger strategies by playing against stronger opponents and can also be exposed to diverse opponents to avoid overfitting.

[0045] A specific embodiment of the above training method is shown below.

[0046] In practice, the current mainstream Proximal Policy Optimization (PPO) algorithm can also be used, which can ensure the monotonicity of policy learning performance to a certain extent.

[0047] The selection of trainers will affect the distribution of battle data, which in turn affects the effect and efficiency of learner training. The present invention improves the selection of trainers from the following two aspects: (1) In order to avoid overfitting, trainers should be selected as randomly as possible to maintain the diversity of trainer strategies; (2) In order to enable learners to continuously improve, more trainers who have performed well in historical battles should be selected. Therefore, this is a typical problem of balancing exploration and exploitation. During the self-playing training process, the specific steps for selecting trainers are: For every new trainer ,set up Indicates trainer The historical winning rate against the learner, Indicates trainer The number of times a trainer is selected is accumulated by 1 each time he is selected; each time a trainer is selected from the trainer set, it is done in the following way: ; in It means to select the trainer with the highest historical winning rate against the learner. is the total number of times a trainer is selected during the self-playing training process, It is a hyperparameter that controls the exploration ratio.

[0048] In order to speed up the training, the above training method adopts a multi-threaded architecture, including a main thread for updating the learner model parameters and multiple worker threads for collecting the required training data. The workflow of each training cycle is as follows: (1) Each worker thread wakes up; the main thread blocks and waits for the worker thread to submit data.

[0049] (2) Each worker thread selects a trainer from the trainer set according to the self-playing training process.

[0050] (3) Each worker thread initializes the game environment, uses the learner and trainer to simulate and collect battle data in parallel until the end of this round of battle. At this stage, the model parameters of the global trainer are read-only and can therefore be shared by each worker thread.

[0051] (4) Each worker thread submits the battle data to the battle data pool.

[0052] (5) The main thread wakes up; each worker thread is blocked.

[0053] (6) The main thread updates the learner’s model parameters using the data in the battle data pool according to the reinforcement learning algorithm.

[0054] (7) The main thread saves the learner’s model parameters at a fixed period and adds new trainers to the trainer set.

[0055] (8) Jump to step (1).

[0056] The above embodiments and drawings do not limit the product form and style of the present invention. Any appropriate changes or modifications made by ordinary technicians in the relevant technical field should be deemed to be within the patent scope of the present invention.

Claims

1. A strategy model for a real-time strategy game, characterized in that: The RTS game is described as a Markov game; the environment size is expressed as ,in and Represent the height and width of the grid respectively; the state space is represented as ; The global observation space is expressed as ,in represents the real number space, Represents the feature channel dimension; The joint action space is expressed as ,in represents the action dimension of the control unit; the state transfer function is expressed as ; The reward function is expressed as ; The strategy is expressed as ; The policy model is expressed as ; The time step Global observation As input, the policy model The output time step The joint action distribution of , sampling is performed on the joint action distribution, and the result obtained is at this time step Joint action ; Define the game world as A two-dimensional grid; The global observation is represented as Each feature channel represents the game world at any two-dimensional grid position. A game attribute on and Respectively represent the horizontal and vertical coordinates of the game world; The joint action distribution is expressed as Action diagram, Representation strategy model The output is located in a two-dimensional grid The action distribution of the control units on Calculated at time step The joint action mask ; For any 2D grid position : If the 2D grid position There are no controlled units belonging to this faction, action mask is an all-zero vector, indicating that all actions cannot be performed; if the two-dimensional grid position There is a controlled unit belonging to this faction, action mask The position with 0 in the byte indicates that the control unit cannot perform the corresponding action, and the position with 1 indicates that the control unit can perform the corresponding action; For policy models The artificial neural network includes encoder-decoder network, long short-term memory network and multi-layer fully connected network.

2. The strategy model for a real-time strategy game as claimed in claim 1, characterized in that: The policy model In the example, the encoder is a convolutional neural network composed of several convolutional layers and pooling layers; the global observation It is fed into the encoder and after passing through the convolutional layer and the pooling layer, a high-level feature map representation is generated; The long short-term memory network takes the high-level feature map representation of the current time step and the context feature map representation of the previous time step as input, and outputs the context feature map representation of the current time step; The decoder consists of several transposed convolutional layers; the transposed convolutional layers transform the context feature map representation back to the environment size The size of the output is The number of channels is calculated to obtain the joint action distribution ; The context feature map representation is used as the input of the multi-layer fully connected network, and the multi-layer fully connected network generates a real number as the estimated value of the environment state; The artificial neural network instantiates a parameterized policy function With value function ,in Represents the parameters of the artificial neural network.

3. The strategy model for a real-time strategy game as claimed in claim 1, characterized in that: In the global observation, the game attributes of the feature channel are defined as: (1) For each combination of faction and each control unit category in the game, define a feature channel containing binary values. Each two-dimensional grid position of this feature channel The value of is 0 or 1; 1 represents the two-dimensional grid position There are control units of this type belonging to this faction on the grid. 0 indicates the two-dimensional grid position. There are no controlled units of this type belonging to this faction; (2) For the terrain attributes in the game, define a feature channel containing binary values. Each two-dimensional grid position of the feature channel The value of is 0 or 1; 1 represents the two-dimensional grid position Not passable, 0 indicates a two-dimensional grid position Passable; (3) For the health attributes of the control units of each faction in the game, a feature channel is defined. The value is a real number; if the 2D grid position If there is a control unit on the grid, the real number is the health of the control unit; if the two-dimensional grid position If there is no control unit, this real number is 0.

4. A method for training a strategy model, using a deep reinforcement learning method to train the strategy model of a real-time strategy game as claimed in any one of claims 1 to 3, including a reinforcement learning algorithm and a self-playing training process, characterized in that The specific reinforcement learning algorithm is: Set the value loss to ,in and Respectively represent when the input is a global observation and The value of the model output is estimated when Represents the time step Environmental rewards, represents the discount factor; Set the policy gradient to ,in Represents the time step The joint action Represents the time step The conditional distribution of the joint action of represents the gradient operator; During the training process, the data generated by the simulated battle over time is represented as follows: ； Under each batch of data, the stochastic gradient ascent optimization algorithm is used to update the artificial neural network parameters, and its calculation formula is expressed as: ； in , represents the learning rate, A coefficient representing the weight of the control value loss.

5. The training method of the strategy model as claimed in claim 4, characterized in that The specific process of self-playing training is: In the process of self-playing training, the strategy model in training is called the learner. The learner continuously updates its own strategy model parameters through algorithms based on the data generated in the game. The strategy model that plays against the learner to generate the data required for the learner training is called the trainer. As the opponent of the learner, the trainer has fixed strategy model parameters. During the self-play training process, the learner's strategy model and its parameters are saved periodically according to the preset duration, and the regularly saved strategy models constitute a trainer set; before the start of each round of competition, a number of trainers are selected from the trainer set to compete with the learner based on the number of players set in the game, and the data generated by the competition is used to train the learner; when selecting trainers, the upper confidence bound algorithm is used to balance the exploration and utilization of trainer selection.

6. The training method of the strategy model as claimed in claim 5, characterized in that During the self-playing training process, the specific steps of selecting a trainer are as follows: For every new trainer ,set up Indicates trainer The historical winning rate against the learner, Indicates trainer The number of times a trainer is selected is accumulated by 1 each time he is selected; each time a trainer is selected from the trainer set, it is done in the following way: ； in It means to select the trainer with the highest historical winning rate against the learner. is the total number of times a trainer is selected during the self-playing training process, It is a hyperparameter that controls the exploration ratio.

7. A method for training a strategy model as described in any one of claims 4 to 6, characterized in that The training method adopts a multi-threaded architecture, including a main thread for updating the learner model parameters and multiple worker threads for collecting the required training data. The workflow of each training cycle is as follows: (1) Each worker thread wakes up; the main thread is blocked, waiting for the worker thread to submit data; (2) Each worker thread selects a trainer from the trainer set according to the self-playing training process; (3) Each worker thread initializes the game environment, uses learners and trainers to simulate and collect battle data in parallel until the current round of battle ends; At this stage, the model parameters of the global trainer are read-only and shared by all worker threads; (4) Each worker thread submits the battle data to the battle data pool; (5) The main thread wakes up; each worker thread is blocked; (6) The main thread updates the learner's model parameters using the data in the battle data pool according to the reinforcement learning algorithm; (7) The main thread saves the learner’s model parameters at a fixed period and adds new trainers to the trainer set; (8) Jump to step (1).

Images