Strategy model and training method for real-time strategy games

Abstract

The present invention discloses a strategy model for real-time strategy games, describing RTS games as Markov games. The environment size is , and represents the height and width of the grid, respectively. The state space is ; the global observation space is , represents the real number space, and represents the feature channel dimension. The joint action space is , represents the action dimension of the control unit. The state transition function is ; the reward function is ; the strategy is ; and the strategy model is . The present invention solves the problems existing in the prior art. Through innovations in the strategy model architecture and training methods, it can be trained with lower computing resource overhead and is adaptable to various RTS game settings.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology applied to electronic games, and in particular to a strategy model for real-time strategy games and a training method thereof. Background Art

[0002] Real-Time Strategy (RTS) games are a genre of video games that primarily utilize real-time actions (rather than turn-based gameplay) and combine strategic and tactical decision-making. Players typically control resource collection, unit production, technological development, and command individual units or entire armies in battle against opposing forces.

[0003] Deep reinforcement learning (DRL) is a subfield of machine learning that combines deep learning and reinforcement learning methods. Deep learning enables computers to identify complex patterns and features from data through neural networks, while reinforcement learning enables computers to learn how to make optimal decisions through interaction with their environment. The goal of DRL is to enable computers to autonomously learn how to perform better at specific tasks—that is, to make optimal decisions—through a mechanism of rewards and penalties, without explicit instructions.

[0004] Due to their complexity and high uncertainty, RTS games have always been a major challenge for AI research. RTS games typically involve multiple players managing resources, controlling units, and planning tactics in a dynamic, continuous environment. Therefore, applying DRL to RTS games requires dealing with large state and action spaces, as well as multi-objective decision-making.

[0005] AlphaStar is an artificial intelligence program developed by DeepMind specifically to play the video game StarCraft II (a real-time strategy game produced and published by Blizzard Entertainment). AlphaStar utilizes a unique deep reinforcement learning architecture that enables learning and decision-making within highly complex game environments. It rapidly learns and improves by playing against different versions of itself. In 2019, AlphaStar successfully defeated professional players in a closed environment and achieved Grandmaster status on the StarCraft II leaderboard.

[0006] Although AlphaStar has achieved impressive results, demonstrating high-level strategy and execution capabilities in matches against human professional players, it also has some shortcomings, including:

[0007] (1) Huge demand for computing resources: AlphaStar training requires a large amount of computing resources. Thousands of CPUs and GPUs / TPUs are used to train strategies for a long time. This resource-intensive training makes the technology difficult to popularize and apply in resource-constrained situations.

[0008] (2) Limited adaptability: The strategies used during AlphaStar training are optimized for specific game versions and maps, and therefore may not adapt well to changes in game rules or the emergence of new maps.

[0009] (3) Over-specialization: AlphaStar’s focus on training and optimizing in the game StarCraft II may have led to its over-specialization in the game and lack of generalization ability to adapt to different types of tasks or environments. Summary of the Invention

[0010] The purpose of the present invention is to provide a strategy model for real-time strategy games and a training method thereof to solve the problems existing in the prior art. Through the innovation of the strategy model architecture and training method, training can be performed with lower computing resource overhead and can adapt to the settings of various RTS games.

[0011] In order to achieve the above object, one of the solutions of the present invention is:

[0012] A real-time strategy game strategy model that describes the RTS game as a Markov game; the environment size is expressed as ,in and Represent the height and width of the grid respectively; the state space is represented as ; The global observation space is expressed as ,in represents the real number space, represents the feature channel dimension; the joint action space is represented as ,in represents the action dimension of the control unit; the state transfer function is expressed as ; The reward function is expressed as ; The strategy is expressed as ; The policy model is expressed as ;

[0013] The time step Global observation As input, the policy model Will output the time step The joint action distribution of , sampling is performed on the joint action distribution, and the result obtained is at this time step Joint action ;

[0014] Define the game world as A two-dimensional grid;

[0015] The global observation is represented as Each feature channel represents the game world at any two-dimensional grid position. A game attribute on and Respectively represent the horizontal and vertical coordinates of the game world;

[0016] The joint action distribution is expressed as Action diagram, Representation Policy Model The output is located in a two-dimensional grid The action distribution of the control unit on the state Calculated at time step Joint action mask ; For any 2D grid position :If the 2D grid position There are no controlled units belonging to this faction, action mask is an all-zero vector, indicating that all actions cannot be executed; if the two-dimensional grid position There is a controlled unit belonging to this faction, action mask The position with 0 in the byte indicates that the control unit cannot perform the corresponding action, and the position with 1 indicates that the control unit can perform the corresponding action;

[0017] For policy models The artificial neural network includes encoder-decoder network, long short-term memory network and multi-layer fully connected network.

[0018] In some embodiments, the policy model In the above example, the encoder is a convolutional neural network composed of several convolutional layers and pooling layers; the global observation It is fed into the encoder and passes through the convolutional layer and pooling layer to produce a high-level feature map representation;

[0019] The long short-term memory network takes the high-level feature map representation of the current time step and the context feature map representation of the previous time step as input, and outputs the context feature map representation of the current time step;

[0020] The decoder consists of several transposed convolutional layers; the transposed convolutional layers transform the context feature map representation back to the environment size The output size is The number of channels is used to obtain the joint action distribution ;

[0021] The context feature map representation is used as the input of the multi-layer fully connected network, and the multi-layer fully connected network generates a real number as the estimated value of the environment state;

[0022] The artificial neural network instantiates a parameterized policy function and value function ,in Represents the parameters of the artificial neural network.

[0023] In some embodiments, in the global observation, the game attributes defining the feature channel are:

[0024] (1) For each combination of faction and each control unit category in the game, define a feature channel containing binary values. Each two-dimensional grid position of the feature channel The value is 0 or 1; 1 represents the two-dimensional grid position There are control units of this type belonging to this faction on the 2D grid. 0 represents the position of the control unit on the 2D grid. There are no controlled units of this type belonging to this faction;

[0025] (2) For the terrain attributes in the game, define a feature channel containing binary values. Each two-dimensional grid position of the feature channel The value is 0 or 1; 1 represents the two-dimensional grid position Not passable, 0 indicates a two-dimensional grid position Passable;

[0026] (3) For the HP attributes of the control units of each faction in the game, a feature channel is defined. Each two-dimensional grid position of the feature channel The value is a real number; if the 2D grid position If there is a control unit on the grid, the real number is the health of the control unit; if the two-dimensional grid position If there is no control unit, this real number is 0.

[0027] The second solution of the present invention is:

[0028] A strategy model training method employs a deep reinforcement learning method to train the strategy model for a real-time strategy game, including a reinforcement learning algorithm and a self-play training process. Specifically, the reinforcement learning algorithm is:

[0029] Set the value loss to ,in and Respectively represent when the input is a global observation and The value of the model output is estimated when Represents the time step Environmental rewards, represents the discount factor;

[0030] Set the policy gradient to ,in Represents the time step The joint action Represents the time step The conditional distribution of the joint action of represents the gradient operator;

[0031] During the training process, the data generated by the simulated battle over time is represented as follows:

[0032] ;

[0033] Under each batch of data, the stochastic gradient ascent optimization algorithm is used to update the artificial neural network parameters, and its calculation formula is expressed as:

[0034] ;

[0035] in , represents the learning rate, The coefficient representing the weight of the control value loss.

[0036] In some embodiments, during self-play training, the strategy model in training is called a learner. The learner continuously updates its strategy model parameters through an algorithm based on the data generated during the game. The strategy model that plays against the learner to generate the data required for the learner's training is called a trainer. The trainer, as the learner's opponent, has fixed strategy model parameters.

[0037] During the self-play training process, the learner's strategy model and its parameters are saved periodically according to the preset duration. The regularly saved strategy models constitute a trainer set. Before the start of each round of competition, several trainers are selected from the trainer set according to the number of players set in the game to compete with the learner. The data generated by the competition is used to train the learner. When selecting trainers, the upper confidence bound algorithm is used to balance the exploration and utilization of trainer selection.

[0038] Preferably, during the self-playing game training process, the specific steps of selecting a trainer are:

[0039] For every new trainer ,set up Indicates trainer The historical win rate against the learner, Indicates trainer The number of times a trainer is selected is accumulated by 1 each time he is selected. Each time a trainer is selected from the trainer set, the following method is used:

[0040] ;

[0041] in Indicates selecting the trainer with the highest historical winning rate against the learner. is the total number of times a trainer is selected during the self-playing training process, It is a hyperparameter that controls the exploration ratio.

[0042] In some embodiments, the training method adopts a multi-threaded architecture, including a main thread for updating learner model parameters and multiple worker threads for collecting required training data. The workflow of each training cycle is as follows:

[0043] (1) Each worker thread wakes up; the main thread blocks and waits for the worker thread to submit data;

[0044] (2) Each worker thread selects a trainer from the trainer set according to the self-play training process;

[0045] (3) Each worker thread initializes the game environment, uses the learner and trainer to simulate and collect game data in parallel until the end of this round of games; at this stage, the model parameters of the global trainer are read-only and shared by all worker threads;

[0046] (4) Each worker thread submits the battle data to the battle data pool;

[0047] (5) The main thread wakes up; each worker thread is blocked;

[0048] (6) The main thread updates the learner's model parameters using the data in the battle data pool according to the reinforcement learning algorithm;

[0049] (7) The main thread saves the learner's model parameters in a fixed period and adds new trainers to the trainer set;

[0050] (8) Jump to step (1).

[0051] After adopting the above technical solution, the present invention has the following technical effects:

[0052] The present invention adopts a strategy model with an encoder-decoder as the main architecture. The encoder implemented based on the convolutional network layer shares parameters in space, which helps the spatially distributed control units share the knowledge learned from exploration. In addition, the simpler artificial neural network design can achieve faster training, reducing the need for hyperparameter settings to be adjusted, which is conducive to reducing the cost of training the strategy model. The centralized reinforcement learning method helps to learn strategic behaviors with the ability of control units to communicate and cooperate. The game world is normalized in a two-dimensional grid manner, and the joint action output of the control units is represented as a regular action graph. It can adapt to various RTS game settings, such as a variable number of control units, thereby realizing the rapid development of a unified RTS game strategy model training framework. At the same time, the use of the LSTM network layer helps to learn long-term dependencies, and the special gating mechanism in the LSTM can avoid the gradient explosion and gradient vanishing problems in long sequences.

[0053] The training method of the present invention introduces an upper confidence bound algorithm in the self-play learning process to balance the exploration and utilization in the problems selected by the trainer, so that the learner can learn stronger strategies by playing against stronger opponents and avoid overfitting by being exposed to diverse opponents. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 A schematic diagram of the system architecture of a strategy model according to a specific embodiment of the present invention;

[0055] Figure 2 This is a training flow chart of a specific embodiment of the present invention. DETAILED DESCRIPTION

[0056] In order to further explain the technical solution of the present invention, the present invention is described in detail below through specific embodiments.

[0057] By establishing a mathematical model, the present invention describes the RTS game as a Markov game, including:

[0058] (I) Environment size: ,in and Represents the height and width of the grid respectively.

[0059] (II) State space: .

[0060] (III) Global observation space: ,in represents the real number space, Represents the feature channel dimension.

[0061] (IV) Joint action space: ,in Represents the action dimension of the control unit.

[0062] (V) State transfer function: .

[0063] (VI) Reward function: .

[0064] (VII) Strategy: .

[0065] refer to Figure 1 As shown, the present invention discloses a strategy model for real-time strategy games. , Strategy Model The input is the global observation , the output is the joint action distribution. Global observation As input, the policy model Will output the time step The joint action distribution of , sampling on the joint action distribution, we can obtain Joint action .

[0066] Define the game world as A two-dimensional grid.

[0067] The global observation is represented as Each feature channel represents the game world at any two-dimensional grid position. A game attribute on and Respectively represent the horizontal and vertical coordinates of the game world.

[0068] The joint action distribution is expressed as The action diagram, where the action dimension is the size of the action set of the control unit (such as silence, movement, attack, mining, etc.) Representation Policy Model The output is located in a two-dimensional grid The action distribution of the control units on .

[0069] Directly sampling from the joint action distribution may produce illegal actions, for example: , if the 2D grid position If there is no controlled unit belonging to this faction, then is meaningless; or, at time step , located at the two-dimensional grid position Certain actions of the units controlled by this faction cannot be executed. In order to ensure that the sampling of the joint action distribution always produces legal actions, we need to use the joint action mask. Calculated at time step Joint action mask ; For any 2D grid position :If the 2D grid position There are no controlled units belonging to this faction, action mask is an all-zero vector, indicating that all actions cannot be executed; if the two-dimensional grid position There is a controlled unit belonging to this faction, action mask The position with 0 in the byte indicates that the control unit cannot perform the corresponding action, and the position with 1 indicates that the control unit can perform the corresponding action.

[0070] An example of calculating the joint action mask is as follows:

[0071] When the two-dimensional grid position When there is a controlled unit belonging to this faction, The calculation process is as follows, assuming that the integer Indicates the action "move one unit distance north", then when the two-dimensional grid position When walking and no other controlled units ,otherwise ; Assuming integer Indicates the action "attack the target at offset (1,3)", then when the two-dimensional grid position When there are enemy control units ,otherwise ;

[0072] In the strategy model Output joint action distribution Then, using the joint action mask , we can ignore illegal actions and only sample legal actions.

[0073] The artificial neural network used for the policy model includes an encoder-decoder network (Encoder-Decoder Network), a long short-term memory (LSTM) network, and a multi-layer fully connected network, with EDN as the main body. Specifically:

[0074] a. The encoder is a convolutional neural network (CNN) consisting of several convolution layers and pooling layers; global observation This is fed into the encoder, where it passes through convolutional and pooling layers to produce a higher-level feature map representation with a larger number of channels and smaller size. Based on the representation learning theory of artificial neural networks, this higher-level feature map representation captures the spatial correlations of the game world, resulting in a stronger feature representation that facilitates learning better strategic behaviors, such as controlling communication and collaboration between units.

[0075] b. Due to the complexity of RTS games, the state of the environment depends not only on the current global observation but also on past policy actions. To learn this long-term dependency, a Long Short-Term Memory (LSTM) network is incorporated into the policy model. The LSTM takes as input the high-level feature map representation of the current time step and the contextual feature map representation of the previous time step, and outputs the contextual feature map representation of the current time step. Because the contextual feature map representation is continuously propagated over time steps, the contextual feature map representation of the current time step contains information from all previous time steps, enabling the policy model to better understand and predict dynamic changes in the environment.

[0076] c. The decoder consists of several transposed convolution layers; the transposed convolution layer transforms the context feature map representation back to the environment size The output size is The number of channels, thus obtaining the joint action distribution .

[0077] In current mainstream actor-critic reinforcement learning algorithms, in addition to the policy model, an estimate of the value corresponding to the environment state is also required. Because the aforementioned context feature graph representation contains information from all historical time steps, it is used as the input to a multi-layer fully connected network, which then generates a real number as the estimated value of the environment state.

[0078] The above artificial neural network instantiates a parameterized policy function and value function ,in Represents the parameters of the artificial neural network.

[0079] Through the above scheme, the present invention adopts a strategy model with an encoder-decoder as the main architecture. The encoder implemented based on the convolutional network layer shares parameters in space, which helps the spatially distributed control units to share the knowledge learned from exploration. Moreover, the simpler artificial neural network design can achieve faster training, reducing the hyperparameter settings that need to be adjusted, which is conducive to reducing the cost of training the strategy model; the centralized reinforcement learning method helps to learn strategic behaviors with the ability of control units to communicate and cooperate; the game world is normalized in a two-dimensional grid manner, and the joint action output of the control units is represented as a regular action graph, which can adapt to various RTS game settings, such as a variable number of control units, thereby realizing the rapid development of a unified RTS game strategy model training framework; at the same time, the use of the LSTM network layer is conducive to learning long-term dependencies, and the special gating mechanism in the LSTM can avoid the problems of gradient explosion and gradient disappearance in long sequences.

[0080] The following shows a specific embodiment of the above-mentioned policy model.

[0081] In the above global observation, the game attributes that define the feature channel are:

[0082] (1) For each combination of faction and each control unit category in the game, define a feature channel containing binary values. Each two-dimensional grid position of the feature channel The value is 0 or 1; 1 represents the two-dimensional grid position There are control units of this type belonging to this faction on the 2D grid. 0 represents the position of the control unit on the 2D grid. There are no controlled units of this type belonging to this faction;

[0083] (2) For the terrain attributes in the game, define a feature channel containing binary values. Each two-dimensional grid position of the feature channel The value is 0 or 1; 1 represents the two-dimensional grid position Not passable, 0 indicates a two-dimensional grid position Passable;

[0084] (3) For the HP attributes of the control units of each faction in the game, a feature channel is defined. Each two-dimensional grid position of the feature channel The value is a real number; if the 2D grid position If there is a control unit on the grid, the real number is the health of the control unit; if the two-dimensional grid position If there is no control unit, this real number is 0.

[0085] In addition, according to different game settings, more attributes that are helpful for strategic decision-making can be discovered and similar feature channels can be defined.

[0086] See also Figure 2 The present invention also discloses a training method for a strategy model, which uses a deep reinforcement learning method to train the above-mentioned strategy model, including a reinforcement learning algorithm and a self-game training process:

[0087] The specific reinforcement learning algorithm is:

[0088] Set the value loss to ,in and Respectively represent when the input is a global observation and The value of the model output is estimated when Represents the time step Environmental rewards, Represents the discount factor, which ranges from [0,1] and applies some discount to the long-term reward;

[0089] Set the policy gradient to ,in Represents the time step The joint action Represents the time step The conditional distribution of the joint action of represents the gradient operator;

[0090] During the training process, the data generated by the simulated battle over time is represented as follows:

[0091] ;

[0092] In each batch of data, the Stochastic Gradient Descent (SGD) optimization algorithm is used to update the artificial neural network parameters. Its calculation formula is expressed as:

[0093] ;

[0094] in , represents the learning rate, The coefficient representing the weight of the control value loss.

[0095] In RTS games, multiple players typically compete simultaneously in a single game world. Self-play is a popular technique for training game-playing AI using reinforcement learning. In a self-play setting, the AI ​​system plays against a copy of itself, rather than against a human or pre-defined opponent. This approach can generate large amounts of training data without external data or guidance, and enables the AI ​​to surpass existing human game theory. The self-play training process is as follows:

[0096] During self-play training, the strategy model in training is called the learner. Based on the data generated during the game, the learner continuously updates its strategy model parameters through an algorithm. The strategy model that plays against the learner to generate the data needed for the learner's training is called the trainer. The trainer, acting as the learner's opponent, has fixed strategy model parameters (i.e., a fixed game strategy).

[0097] During self-play training, the learner's strategy model and its parameters are periodically saved according to a preset duration. The regularly saved strategy models constitute a trainer set. Before each round of competition, several trainers are selected from the trainer set based on the number of players set in the game to compete with the learner. The data generated by the competition is used to train the learner. When selecting trainers, the upper confidence bound (UCB) algorithm is used to balance exploration and utilization in trainer selection.

[0098] Through the above scheme, the training method of the present invention introduces an upper confidence bound algorithm in the self-play learning process, balancing the exploration and utilization in the problems selected by the trainer, so that the learner can not only learn stronger strategies by playing against stronger opponents, but also be exposed to diverse opponents to avoid overfitting.

[0099] The following shows a specific embodiment of the above training method.

[0100] In practice, the current mainstream Proximal Policy Optimization (PPO) algorithm can also be used, which can ensure the monotonicity of policy learning performance to a certain extent.

[0101] Since the selection of trainers will affect the distribution of battle data, and thus affect the effectiveness and efficiency of learner training, the present invention improves the selection of trainers from the following two aspects: (1) In order to avoid overfitting, trainers should be selected as randomly as possible to maintain the diversity of trainer strategies; (2) In order to enable learners to continuously improve, more trainers with good performance in historical battles should be selected. Therefore, this is a typical problem of balancing exploration and exploitation. During the self-game training process, the specific steps for selecting trainers are:

[0102] For every new trainer ,set up Indicates trainer The historical win rate against the learner, Indicates trainer The number of times a trainer is selected is accumulated by 1 each time he is selected. Each time a trainer is selected from the trainer set, the following method is used:

[0103] ;

[0104] in Indicates selecting the trainer with the highest historical winning rate against the learner. is the total number of times a trainer is selected during the self-playing training process, It is a hyperparameter that controls the exploration ratio.

[0105] To speed up training, the above training method adopts a multi-threaded architecture, including a main thread for updating the learner model parameters and multiple worker threads for collecting the required training data. The workflow of each training cycle is as follows:

[0106] (1) Each worker thread wakes up; the main thread blocks and waits for the worker thread to submit data.

[0107] (2) Each worker thread selects a trainer from the trainer set according to the self-play training process.

[0108] (3) Each worker thread initializes the game environment, uses the learner and trainer to simulate and collect game data in parallel until the end of this round of games. During this stage, the model parameters of the global trainer are read-only and can therefore be shared by all worker threads.

[0109] (4) Each worker thread submits the battle data to the battle data pool.

[0110] (5) The main thread wakes up; each worker thread is blocked.

[0111] (6) The main thread updates the learner’s model parameters using the data in the battle data pool according to the reinforcement learning algorithm.

[0112] (7) The main thread saves the learner's model parameters in a fixed period and adds new trainers to the trainer set.

[0113] (8) Jump to step (1).

[0114] The above embodiments and drawings do not limit the product form and style of the present invention. Any appropriate changes or modifications made by ordinary technicians in the relevant technical field should be deemed to be within the patent scope of the present invention.

Claims

1. A strategy model for a real-time strategy game, characterized by: The RTS game is described as a Markov game; the environment size is expressed as ,in and Represent the height and width of the grid respectively; the state space is represented as ; The global observation space is expressed as ,in represents the real number space, Represents the feature channel dimension; The joint action space is expressed as ,in represents the action dimension of the control unit; the state transfer function is expressed as ; The reward function is expressed as ; The strategy is expressed as ; The policy model is expressed as ; The time step Global observation As input, the policy model The output time step The joint action distribution of , sampling is performed on the joint action distribution, and the result obtained is at this time step Joint action ; Define the game world as A two-dimensional grid; The global observation is represented as Each feature channel represents the game world at any two-dimensional grid position. A game attribute on and Respectively represent the horizontal and vertical coordinates of the game world; The joint action distribution is expressed as Action diagram, Representation Policy Model The output is located in a two-dimensional grid The action distribution of the control unit on the state Calculated at time step Joint action mask ; For any 2D grid position :If the 2D grid position There are no controlled units belonging to this faction, action mask is an all-zero vector, indicating that all actions cannot be executed; if the two-dimensional grid position There is a controlled unit belonging to this faction, action mask The position with 0 in the byte indicates that the control unit cannot perform the corresponding action, and the position with 1 indicates that the control unit can perform the corresponding action; For policy models The artificial neural network includes encoder-decoder network, long short-term memory network and multi-layer fully connected network; In the global observation, the game attributes of the feature channel are defined as: (1) For each combination of faction and each control unit category in the game, define a feature channel containing binary values. Each two-dimensional grid position of the feature channel The value is 0 or 1; 1 represents the two-dimensional grid position There are control units of this type belonging to this faction on the 2D grid. 0 represents the position of the control unit on the 2D grid. There are no controlled units of this type belonging to this faction; (2) For the terrain attributes in the game, define a feature channel containing binary values. Each two-dimensional grid position of the feature channel The value is 0 or 1; 1 represents the two-dimensional grid position Not passable, 0 indicates a two-dimensional grid position Passable; (3) For the HP attributes of the control units of each faction in the game, a feature channel is defined. Each two-dimensional grid position of the feature channel The value is a real number; if the 2D grid position If there is a control unit on the grid, the real number is the health of the control unit; if the two-dimensional grid position If there is no control unit, this real number is 0.

2. The strategy model for a real-time strategy game according to claim 1, wherein: The policy model In the above example, the encoder is a convolutional neural network composed of several convolutional layers and pooling layers; the global observation It is fed into the encoder and passes through the convolutional layer and pooling layer to produce a high-level feature map representation; The long short-term memory network takes the high-level feature map representation of the current time step and the context feature map representation of the previous time step as input, and outputs the context feature map representation of the current time step; The decoder consists of several transposed convolutional layers; the transposed convolutional layers transform the context feature map representation back to the environment size The output size is The number of channels is used to obtain the joint action distribution ; The context feature map representation is used as the input of the multi-layer fully connected network, and the multi-layer fully connected network generates a real number as the estimated value of the environment state; The artificial neural network instantiates a parameterized policy function and value function ,in Represents the parameters of the artificial neural network.

3. A method for training a strategy model, using a deep reinforcement learning method to train the strategy model of the real-time strategy game according to claim 1 or 2, comprising a reinforcement learning algorithm and a self-playing training process, characterized in that The specific reinforcement learning algorithm is: Set the value loss to ,in and Respectively represent when the input is a global observation and The value of the model output is estimated when Represents the time step Environmental rewards, represents the discount factor; Set the policy gradient to ,in Represents the time step The joint action Represents the time step The conditional distribution of the joint action of represents the gradient operator; During the training process, the data generated by the simulated battle over time is represented as follows: ； Under each batch of data, the stochastic gradient ascent optimization algorithm is used to update the artificial neural network parameters, and its calculation formula is expressed as: ； in , represents the learning rate, The coefficient representing the weight of the control value loss.

4. The training method of the strategy model according to claim 3, characterized in that The specific process of self-playing training is: During self-play training, the strategy model in training is called the learner. The learner continuously updates its strategy model parameters through an algorithm based on the data generated during the game. The strategy model that plays against the learner to generate the data needed for the learner's training is called the trainer. The trainer, as the learner's opponent, has fixed strategy model parameters. During the self-play training process, the learner's strategy model and its parameters are saved periodically according to the preset duration. The regularly saved strategy models constitute a trainer set. Before the start of each round of competition, several trainers are selected from the trainer set according to the number of players set in the game to compete with the learner. The data generated by the competition is used to train the learner. When selecting trainers, the upper confidence bound algorithm is used to balance the exploration and utilization of trainer selection.

5. The training method of the strategy model according to claim 4, characterized in that During the self-playing training process, the specific steps for selecting a trainer are as follows: For every new trainer ,set up Indicates trainer The historical win rate against the learner, Indicates trainer The number of times a trainer is selected is accumulated by 1 each time he is selected. Each time a trainer is selected from the trainer set, the following method is used: ； in Indicates selecting the trainer with the highest historical winning rate against the learner. is the total number of times a trainer is selected during the self-playing training process, It is a hyperparameter that controls the exploration ratio.

6. The training method of the strategy model according to any one of claims 3 to 5, characterized in that The training method adopts a multi-threaded architecture, including a main thread for updating the learner model parameters and multiple worker threads for collecting the required training data. The workflow of each training cycle is as follows: (1) Each worker thread wakes up; the main thread blocks and waits for the worker thread to submit data; (2) Each worker thread selects a trainer from the trainer set according to the self-play training process; (3) Each worker thread initializes the game environment, uses learners and trainers to simulate and collect game data in parallel until the end of this round of games; At this stage, the model parameters of the global trainer are read-only and shared by all worker threads; (4) Each worker thread submits the battle data to the battle data pool; (5) The main thread wakes up; each worker thread is blocked; (6) The main thread updates the learner's model parameters using the data in the battle data pool according to the reinforcement learning algorithm; (7) The main thread saves the learner's model parameters in a fixed period and adds new trainers to the trainer set; (8) Jump to step (1).

Images