A game AI training method and system for different players
By building sampling strategies and reward models, combining the extremely large and extremely small Q learning algorithm, learning Nash equilibrium strategy, the problem that game AI cannot meet the needs of multiple player types is solved, and satisfaction with different types of players is improved.
Patent Information
- Application Number
- CN202210965612.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-08-12
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2042-08-12
AI Technical Summary
The existing technology is difficult to meet the satisfaction needs of different types of players, because the game AI only uses one battle strategy and cannot effectively distinguish and cope with the needs of various player types such as competitive, entertainment and hybrid.
By constructing the coping strategy sampling strategy and reward model of game AI, using the extremely large and extremely small Q learning algorithm and game model, we learn the Nash equilibrium strategy of each type of opponent, judge the opponent type based on the interaction data and historical records, and use the corresponding strategy to interact with the game , improve player satisfaction.
The game AI is realized to adjust strategies according to different types of players, improve the satisfaction of all players, and meet the diverse player needs.
Smart Images

Figure CN115430154B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence, and in particular to a game AI training method and system for different players. Background Art
[0002] In recent years, due to the full development of artificial intelligence technology, human-computer interactive game problems have attracted widespread attention and research. Among them, electronic games (such as StarCraft, Honor of Kings, etc.) as a classic form of human-computer interactive game have become the focus of research. However, previous studies mainly focused on how to solve the optimal solution or Nash equilibrium solution of a game, ignoring the fact that there is another problem in real-life electronic games - there are many and unknown types of players. For example, in many games, players can be roughly divided into multiple types: competitive, entertainment, mixed, and other major categories, and each major category can also be subdivided into several subcategories according to different characteristics. The type of player determines that they have different expectations for the intelligence shown by the game AI. Appropriate AI intelligence can effectively improve the satisfaction of players. From the perspective of game manufacturers, the game AI developed should maximize the satisfaction of all players. However, if the game AI only uses one combat strategy, it will inevitably be difficult to satisfy all types of players. Therefore, it is necessary to train AI so that it can distinguish the types of combat players and adopt strategies that maximize player satisfaction. Summary of the invention
[0003] In order to solve the above technical problems, the present invention provides a game AI training method and system for different players.
[0004] The technical solution of the present invention is: a game AI training method for different players, comprising:
[0005] Step S1: Construct the game AI's response strategy based on the opponent type M Sampling strategy π s and reward model
[0006] Step S2: Game AI uses the sampling strategy π s Interact with opponents and collect interaction data of various types of opponents;
[0007] Step S3: Calculate M opponent reward models using the interaction data in, represents the reward model of the mth type of opponent;
[0008] Step S4: Use the minimax Q learning algorithm and game model and reward model Learn the Nash equilibrium strategy of the mth opponent as a response strategy Repeat step S4 to learn the response strategies for all types of opponents.
[0009] Compared with the prior art, the present invention has the following advantages:
[0010] The present invention discloses a game AI training method for different players, which can cope with situations with different opponent types. For example, a game has multiple types of game players, such as competitive, entertainment, and mixed types. Each type of opponent player has different satisfaction with the same game. The present invention trains AI, takes the opponent player's satisfaction as its own reward, and establishes a corresponding reward model for each type of opponent. In the game, the opponent type is distinguished through the game interaction history and the reward model, and the corresponding response strategy is used to interact with the opponent type to improve the player's satisfaction. BRIEF DESCRIPTION OF THE DRAWINGS
[0011] Figure 1 This is a flowchart of a game AI training method for different players in an embodiment of the present invention;
[0012] Figure 2 This is a schematic diagram of the implementation process of the method of the present invention in a real game battle;
[0013] Figure 3 The structure diagram of a game AI training system for different players in an embodiment of the present invention. DETAILED DESCRIPTION
[0014] The present invention provides a game AI training method for different players, which can cope with situations with different types of opponents and improve the satisfaction of all players.
[0015] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention is further described in detail below through specific implementations and in conjunction with the accompanying drawings.
[0016] Embodiment 1
[0017] like Figure 1 As shown, an embodiment of the present invention provides a game AI training method for different players, comprising the following steps:
[0018] Step S1: Construct the game AI's response strategy based on the opponent type M Sampling strategy π s and reward model
[0019] Step S2: Game AI uses sampling strategy π s Interact with opponents and collect interaction data of various types of opponents;
[0020] Step S3: Calculate M opponent reward models using interaction data in, represents the reward model of the mth type of opponent;
[0021] Step S4: Use the minimax Q learning algorithm and game model and reward model Learn the Nash equilibrium strategy of the mth opponent as a response strategy Repeat step S4 to learn the response strategies for all types of opponents.
[0022] It is very common in the gaming field to have various and unknown opponent types. Most games on the market now are both competitive and entertaining, so players can be roughly divided into competitive, entertaining, and mixed types. These different types of players may not have the same satisfaction when facing the same game, so if you want to develop game AI to play against players, how to maximize player satisfaction is a challenging problem.
[0023] From the perspective of human-computer interaction games, the interaction between the game AI and the opponent player can be modeled as a two-person zero-sum Markov game. Since the types of players are diverse and the AI cannot know the opponent type information at the beginning of the game, the embodiment of the present invention models the problem as a two-person zero-sum Markov game with unknown opponent type.
[0024] In one embodiment, the above step S1: constructing a game AI response strategy according to the opponent type M Sampling strategy π s and reward model Specifically include:
[0025] Step S11: Construct a two-player zero-sum Markov game model with M types of opponents, using an octet Indicates; among them, is the state space, They are the action spaces of our own game AI and the enemy's opponent. is the transition probability matrix, is the AI reward function for the game, γ∈[0,1) is the discount factor, Indicates the existence of M types of opponents, represents the probability of each opponent appearing at the beginning of the game;
[0026] At time t, the state of the game AI is s t , your action is a t , the opponent's manual action is b t , so that in state s t Both players use actions (a t ,b t )Transfer to the next state st+1 The probability of is the reward function of your own game AI, and the enemy's reward is the opposite of your own game AI reward -
[0027] The goal of your own game AI is to maximize its expected cumulative reward (also called return).
[0028] Reward function in Markov games with unknown opponent type is a mapping: t ,a t ,b t )→Δ(R), where R is the reward space of the own reward, and Δ(R) is a probability distribution on R. No matter which opponent you face, the reward space is the same, but the probability distribution is different. In particular, when the probability distribution is degraded (i.e., the reward function is a fixed value related to the opponent), you can judge the type of opponent by looking at the reward;
[0029] Building a game AI response strategy based on octets As shown in formula (1):
[0030]
[0031] in, is the Nash equilibrium strategy, and p(a) represents the strategy used by the game AI when interacting with the opponent m. The probability of taking action a in state s;
[0032] In a general two-player game environment, if the opponent's strategy is fixed, then the goal is to find the best response; if the opponent also has learning ability, the opponent will change the strategy according to the expected reward, then the goal is to find the Nash equilibrium strategy; when there are M types of opponents in the environment, the Nash equilibrium strategy is Use strategies when facing opponent m The probability of taking action a in state s is p(a);
[0033] The best response is relatively easy to obtain, because once the strategy is fixed, it means that the environment is not dynamic, and the game environment degenerates into a single-agent environment, and the best response can be solved using conventional reinforcement learning algorithms. When the opponent also has the ability to learn, it becomes very important to find the Nash equilibrium strategy, because this can guarantee the lower limit of one's own return. Since all players in the game have the ability to learn, the game AI needs to learn the Nash equilibrium strategy;
[0034] Since the game AI needs to learn the Nash equilibrium strategy for each opponent, and the game AI cannot determine the type of opponent before each round of game starts, the method of the present invention needs to first establish an opponent-related model, and then use the model to learn the Nash equilibrium strategy for the opponent. The model generally contains two parts, the transition probability and the reward function. As a manufacturer developing game AI, the transition probability can be considered as known, so the algorithm only needs to establish M opponent-related reward function models. When learning Nash equilibrium strategies, use the corresponding models for training. In actual game combat, you can use the models to determine the type of opponent and use the corresponding Nash equilibrium strategies to deal with them. Before building the model, you need to build a sampling model to collect enough data;
[0035] Step S12: Construct sampling strategy π s , as shown in formula (2):
[0036]
[0037] Among them, s t is the state; t ,b t The states s t Next, your game AI and the opponent use action a respectively. t ,b t ;count(s t ,a t ,b t ) indicates state s t Next, the game AI uses action a t When the enemy opponent uses action b t Total times:
[0038] The sampling result is shown in formula (3):
[0039]
[0040] in, is the game model, formula (3) means that in the game model In state s t Next, the game AI and the enemy opponent use action a respectively. t ,b t , the game AI gets r t+1 Reward, state transfer to s t+1 ; At the same time, an interaction data (s t ,a t ,b t ,r t+1 ,s t+1 );
[0041] Step S13: Constructing the opponent-related reward function As shown in formula (4):
[0042]
[0043] Among them, the reward r is the player’s satisfaction; It indicates the probability that the game AI and the enemy opponent use actions a and b respectively to obtain reward r in state s; count(r,s,a,b) indicates the number of times the game AI and the enemy opponent use actions a and b respectively to obtain reward r in state s.
[0044] In one embodiment, the above step S2: the game AI uses the sampling strategy π s Interact with opponents and collect interaction data of various types of opponents, including:
[0045] Game AI uses sampling strategy π s Interact with players, collect enough data from various types of players, and store it in offline datasets And classified according to the player type, that is, if there are M types of opponents, the data set Divided into
[0046] In one embodiment, the above step S3: using the interaction data to calculate the M opponent reward models in, Represents the reward model for the mth type of opponent, including:
[0047] In this step, formula (4) is used based on , and calculate the reward model for each type of opponent
[0048] In one embodiment, the above step S4: using the minimax Q learning algorithm, game model and reward model Learn the Nash equilibrium strategy of the mth opponent as a response strategy Repeat step S4 to learn the response strategies for all types of opponents, including:
[0049] The minimax Q learning algorithm is used to iteratively calculate the Nash equilibrium Q under each state-action tuple (s, a, b). * Value, and then solve the linear programming shown in formula (5):
[0050]
[0051] Where c is the Nash equilibrium V in state s* value; are the action sets of our own game AI and the enemy opponent, Q * (s,a,b) is the Nash equilibrium Q under the state-action tuple (s,a,b) * Value, p(a) = π(a|s) is the response strategy of the game AI in state s.
[0052] After getting the Nash equilibrium strategy facing all opponents After that, the AI of the game needs to distinguish the type of opponent it faces as soon as possible after each game starts, so as to use the corresponding strategy. According to the obtained opponent-related reward model, by analyzing the history of interaction with the opponent, it can be determined which type of opponent it is playing against. Specifically, for games with limited states and actions, the algorithm records the history h of interaction with the opponent. A history of length n is represented as h = {(s0, a0, b0, r1, s1), (s1, a1, b1, r2, s2), …, (s n-1 ,a n-1 ,b n-1 ,r n ,s n )}, then the probability of generating h under the condition that the opponent is of the mth type can be calculated as shown in formula (5):
[0053]
[0054] Then, the opponent type that maximizes this probability is the most likely opponent type, that is, the opponent type
[0055] like Figure 2 As shown, a schematic diagram of the implementation process of the method of the present invention in a real game battle is shown, wherein steps 1 to 4 are the training process, and steps 5 to 7 are the real game battle process:
[0056] Step 1: Initialize the game AI strategy according to the opponent type M Reward Model Sampling strategy π s ;
[0057] Step 2: Game AI uses sampling strategy π s Interact with players, collect enough data from various types of players, and store it in offline datasets And classified according to the player type, that is, if there are M types of opponents, the data set Divided into in Represents the data interacting with the mth type of adversary;
[0058] Step 3: Use The data in the calculation of M opponents related reward model in represents the reward model for the mth type of opponent;
[0059] Step 4: Use the minimax Q-learning algorithm, game model, and corresponding reward model Learn the Nash equilibrium strategy for dealing with the mth type of opponent Repeat step 4 until you have learned the strategies for all types of opponents;
[0060] Step 5: When the real game starts, initialize the interaction history h and load the reward model
[0061] Step 6: In the real game, according to the interaction history h and reward model Determine the type of opponent i. After determining the type of opponent, use the strategy learned in step 4 Interact with opponents;
[0062] Step 7: Record the game in this step (s t ,a t ,b t ,r t+1 ,s t+1 ) is stored in the interaction history h, and steps 6 and 7 are repeated until the game ends.
[0063] The present invention discloses a game AI training method for different players, which can cope with situations with different opponent types. For example, a game has multiple types of game players, such as competitive, entertainment, and mixed types. The satisfaction of each type of opponent player with the same game is different. The present invention trains AI, takes the satisfaction of the opponent player as its own reward, and establishes a corresponding reward model for each type of opponent. In the game, the opponent type is distinguished through the game interaction history and the reward model, and the coping strategy corresponding to the opponent type is used to interact with the opponent in the game, thereby improving the satisfaction of all players.
[0064] Embodiment 2
[0065] like Figure 3 As shown, an embodiment of the present invention provides a game AI training system for different players, including the following modules:
[0066] Model building module 51, used to build the game AI's response strategy according to the opponent type M Sampling strategy π s and reward model
[0067] Collect game interaction data module 52, used by game AI to use sampling strategy π sInteract with opponents and collect interaction data of various types of opponents;
[0068] The reward model calculation module 53 is used to calculate the M opponent reward models using the interaction data in, represents the reward model of the mth type of opponent;
[0069] Learning response strategy module 54, used to use the minimax Q learning algorithm and game model and reward model Learn the Nash equilibrium strategy of the mth opponent as a response strategy The response strategy module 54 is repeatedly learned to obtain response strategies for all types of opponents.
[0070] The above embodiments are provided only for the purpose of describing the present invention, and are not intended to limit the scope of the present invention. The scope of the present invention is defined by the appended claims. Various equivalent substitutions and modifications made without departing from the spirit and principles of the present invention should all be included within the scope of the present invention.
Claims
1. A game AI training method for different players, It is characterized in that include: Step S1: Construct the game AI's response strategy based on the opponent type M Sampling strategy π s and reward model Step S2: Game AI uses the sampling strategy π s Interact with opponents and collect interaction data of various types of opponents; Step S3: Calculate M opponent reward models using the interaction data in, represents the reward model of the mth type of opponent; Step S4: Use the minimax Q learning algorithm and game model and reward model Learn the Nash equilibrium strategy of the mth opponent as a response strategy Repeat step S4 to learn the response strategies for all types of opponents.
2. The game AI training method for different players according to claim 1, It is characterized in that Step S1: Constructing a game AI response strategy based on the opponent type M Reward Model and sampling strategy π s , specifically including: Step S11: Construct a two-player zero-sum Markov game model with M types of opponents, using an octet Indicates; among them, is the state space, They are the action spaces of our own game AI and the enemy's opponent. is the transition probability matrix, is the AI reward function for the game, γ∈[0,1) is the discount factor, Indicates the existence of M types of opponents, represents the probability of each opponent appearing at the beginning of the game; Based on the octet, build a game AI response strategy As shown in formula (1): in, is the Nash equilibrium strategy, and p(a) represents the strategy used by the game AI when interacting with the opponent m. The probability of taking action a in state s; Step S12: Construct sampling strategy π s , as shown in formula (2): Among them, s t is the state; t , b t The states s t Next, your game AI and the opponent use action a respectively. t , b t ;count(s t , a t , b t ) indicates state s t Next, the game AI uses action a t , the enemy opponent uses action b t Total times: The sampling result is shown in formula (3): in, is the game model, formula (3) means that in the game model In state s t Next, the game AI and the enemy opponent use action a respectively. t , b t , the game AI gets r t+1 Reward, state transfer to s t+1 ; At the same time, an interaction data (s t , a t , b t , r t+1 ,s t+1 ); Step S13: Constructing the opponent-related reward function As shown in formula (4): Among them, the reward r is the player’s satisfaction; It indicates the probability that the game AI and the enemy opponent use actions a and b respectively to obtain reward r in state s; count(r, s, a, b) indicates the number of times the game AI and the enemy opponent use actions a and b respectively to obtain reward r in state s.
3. The game AI training method for different players according to claim 2, It is characterized in that Step S4: Using the Minimax Q learning algorithm and the game model and reward model Learn the Nash equilibrium strategy of the mth opponent as a response strategy Repeat step S4 to learn the response strategies for all types of opponents, including: The maximum and minimum Q learning algorithm is used to iteratively calculate the Nash equilibrium Q under each state-action tuple. * Value, and then solve the linear programming shown in formula (5): Where c is the Nash equilibrium V in state s * value; are the action sets of our own game AI and the enemy opponent, Q * (s, a, b) is the Nash equilibrium Q under the state-action tuple (s, a, b) * Value, p(a) = π(a|s) is the response strategy of the game AI in state s.
4. A game AI training system for different players, It is characterized in that Includes the following modules: Model building module, used to build game AI response strategies based on opponent type M Sampling strategy π s and reward model Collect game interaction data module for game AI to use the sampling strategy π s Interact with opponents and collect interaction data of various types of opponents; A reward model calculation module is used to calculate M opponent reward models using the interaction data in, represents the reward model of the mth type of opponent; Learning response strategy module, used to use the minimax Q learning algorithm and game model and reward model Learn the Nash equilibrium strategy of the mth opponent as a response strategy Repeat step S4 to learn the response strategies for all types of opponents.
Citation Information
Patent Citations
Method for carrying out opponent modeling based on non-complete information game playing
CN106469317A
Training method and device for intelligent game agent
CN110141867A
Reinforcement learning battle game AI training method based on information bottleneck theory
CN112717415A
Intersection joint signal control method based on Nash Q learning algorithm
CN113487891A
Game strategy model generation method and control method of intelligent agent in game
CN114344889A
