A bidirectional distillation method and device suitable for game AI

Abstract

The present invention discloses a two-way distillation method and device suitable for game AI, including: in each iterative training, randomly separating some agents from the training group of game AI agents as virtual groups, and assigning virtual group strategies; deploying the training group and the virtual group in the game for training, the training group is updated using the learning strategy network, and the virtual group is updated using the distillation strategy network; the distillation strategy network includes forward distillation and reverse distillation; when performing forward distillation, the KL divergence between the distillation strategy and the learning strategy is minimized; when performing reverse distillation, the KL divergence between the distillation strategy and the learning strategy is maximized; repeating the above steps for multiple iterative training, after the training is completed, the updated training group is used as the final game AI agent, and deployed in the game. The present invention enables game AI to optimize its strategy distribution in complex game scenes.

Description

Technical Field

[0001] The present invention belongs to the field of game AI in multi-agent reinforcement learning, and in particular to a two-way distillation method and device suitable for game AI. Background Art

[0002] Game AI has always been an important application scenario for reinforcement learning (RL) algorithms, especially in multiplayer cooperative or confrontational games (such as StarCraft and MOBA games). Game AI not only needs to make autonomous decisions, but also needs to be able to collaborate with other agents or even human players. However, due to the unpredictability of teammates' behavior, game AI faces huge challenges. For example, human-controlled agents may exhibit a variety of behaviors such as extreme defense, reckless offense, or random exploration, which may not have appeared during the training phase. Game AI needs to quickly adjust its strategies based on these dynamically changing behaviors to ensure the overall collaborative efficiency of the team. However, traditional multi-agent learning methods often lack this adaptability, especially when the behavior in the test environment is completely different from the training data, the generalization ability of the agent is particularly insufficient.

[0003] In order to solve the problem of collaborative uncertainty in game AI, traditional self-game methods have become an important tool for multi-agent reinforcement learning. In multiplayer games (such as StarCraft and MOBA games), AI not only needs to fight against the enemy, but also needs to collaborate with teammates to complete complex tasks. The self-game method simulates a variety of possible interaction modes through autonomous confrontation or collaboration between agents, and is an effective way to improve the diversity and competitiveness of AI strategies. Its core goal is to generate high-quality strategies suitable for training scenarios through self-learning and confrontation without relying on external opponents or teammates.

[0004] Specifically, in the training of game AI, the self-game method maintains a historical strategy library to record the strategies learned by AI at different training stages. During the training process, AI repeatedly interacts with these historical strategies, and through adversarial or collaborative game simulations, gradually expands its interaction distribution and improves its adaptability under known behavior patterns. For example, in MOBA games, AI can gradually learn how to cooperate with these behavior patterns through multiple rounds of simulations with defensive or offensive teammates in historical strategies, thereby achieving better performance in the training environment. In this way, the self-game method can effectively cover some under-explored strategy patterns and enhance the generalization ability of game AI in training scenarios.

[0005] However, the advantages of self-game methods are mainly limited to generalization in the inside-space, that is, simulating the known interaction distribution during the training process through the historical strategy library. When faced with outside-space strategies in the test environment, such as players or opponents showing new behavior patterns that are completely different from the training data, the effectiveness of self-game methods is often limited. This is because the generation ability of self-game methods is limited by the historical strategy library and cannot go beyond the boundaries of its known strategy distribution. For example, when AI is accustomed to collaborating with defensive teammates in StarCraft training, but encounters overly aggressive teammates in the test environment, traditional self-game methods may not be able to adjust their own strategies, resulting in the failure of teamwork.

[0006] This limitation directly affects the performance of game AI in zero-sample scenarios. When AI faces strategic behaviors that it has never seen before, traditional self-game methods cannot provide sufficient exploration capabilities, nor can they break through the limitations of the historical library in strategy generation, resulting in insufficient generalization in key scenarios. Therefore, although the self-game method can solve some of the collaboration problems in training, its application in game AI still faces obvious bottlenecks, and new methods are urgently needed to enhance AI's adaptability in external space strategies. Summary of the invention

[0007] The present invention provides a two-way distillation method and device suitable for game AI. Through forward distillation and backward distillation mechanisms, the game AI can optimize its strategy distribution in complex game scenarios, retaining historical strategy experience and having the ability to explore new strategies.

[0008] A bidirectional distillation method suitable for game AI includes the following steps:

[0009] In each iteration of training, some agents are randomly separated from the training group of game AI agents as virtual groups, and virtual group strategies are assigned; the virtual groups are used to simulate the behavior of zero-sample agents;

[0010] The training group and the virtual group are deployed in the game for training. The training group is updated using the learning strategy network, and the virtual group is updated using the distillation strategy network.

[0011] The distillation strategy network includes forward distillation and reverse distillation. When performing forward distillation, the KL divergence between the distillation strategy network and the learning strategy network is minimized to achieve implicit self-game, so as to retain historical strategy knowledge and increase the diversity of interaction patterns. When performing reverse distillation, the KL divergence between the distillation strategy network and the learning strategy network is maximized to promote the preference transfer of virtual group strategies to explore new strategy distributions.

[0012] Repeat the above steps for multiple iterative training. After the training is completed, use the updated training population as the final game AI agent and deploy it into the game.

[0013] Furthermore, the virtual group adopts a different strategy distribution from the training group, and simulates game scenarios that have never been seen before (such as cooperative, aggressive or defensive teammates) by randomly generating opponent or teammate behavior patterns. This virtual group mechanism enables the training process to have a higher dynamic interaction diversity.

[0014] Furthermore, in each training iteration, the training population and the virtual population are randomly divided using a binary vector. When dividing the training population and the virtual population, it is ensured that at least one agent remains in the training population.

[0015] Preferably, in each iterative training, updates are performed by alternating optimization of forward distillation and backward distillation.

[0016] The loss function for the KL divergence between the distillation policy network and the learning policy network is defined as follows:

[0017] ;

[0018] In the formula, represents the KL divergence loss function, represents the learning policy network, represents the distillation strategy network, Indicates that in the state Select Action , For intelligent agents The allocation status, represents the KL divergence between the distillation policy network and the learning policy network, N represents the number of agents, and E represents the average of all divergences.

[0019] The gradient update formula for forward distillation is:

[0020] ;

[0021] in, represents the distillation strategy network, represents the learning rate of forward distillation, represents the updated gradient, It represents the KL divergence loss function.

[0022] The gradient update formula for reverse distillation is:

[0023] ;

[0024] in, represents the distillation strategy network, represents the learning rate of back distillation, represents the updated gradient, It represents the KL divergence loss function.

[0025] A two-way distillation device suitable for game AI comprises a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the above-mentioned two-way distillation method suitable for game AI.

[0026] Compared with the prior art, the present invention has the following beneficial effects:

[0027] 1. Improve generalization ability: The two-way distillation method effectively enhances the adaptability of game AI in zero-sample scenarios, enabling it to cope with the behavior patterns of dynamic teammates and enemies.

[0028] 2. Simplify storage requirements: No complex historical policy storage system is required, which significantly reduces the consumption of computing resources.

[0029] 3. Wide applicability: It performs well in a variety of tasks such as collaboration, competition, and social dilemmas, and is especially suitable for strategy optimization in RTS and MOBA games. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 Schematic diagram of the training and testing scenarios in the embodiments of the present invention.

[0031] Figure 2 Schematic diagram of using learning strategy and distillation strategy update in an embodiment of the present invention.

[0032] Figure 3 The figure is a schematic diagram of a verification scenario in an embodiment of the present invention.

[0033] Figure 4 A schematic diagram showing the comparison between different methods. DETAILED DESCRIPTION

[0034] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be pointed out that the embodiments described below are intended to facilitate the understanding of the present invention and do not have any limiting effect on the present invention.

[0035] The present invention provides a bidirectional distillation method suitable for game AI, which includes the following core steps:

[0036] 1. Construction of virtual groups.

[0037] Virtual Groups Acts as a virtual background group during the training phase. Figure 2As shown, the orange agents are virtual background groups, while the blue ones are training groups. From the training group Specifically, in order to generate , the present invention uses a binary vector To determine the agent Is it a virtual group? or training group .vector is sampled from the following probability distribution:

[0038] ;

[0039] ;

[0040] ;

[0041] in, is the indicator function, Represents a length of A vector with all elements set to 0. In most cases, when hour, is a standard Bernoulli distribution, indicating that each agent There is a probability Belong to the training group However, when agents 1, ..., N-1 all belong to the virtual group There is one exception. In this case, Degenerates into a deterministic distribution, ensuring that the last agent N definitely belongs to the training population This safeguard prevents all agents from being assigned to virtual groups. , because it would be meaningless if all agents acted as background agents. Virtual Group The agents in the matrix do not use learned strategies when interacting with each other. , but using the distillation strategy ,like Figure 2 As shown in the virtual group The distillation strategy is used while the training population interacts with the environment using the learned strategy.

[0042] 2. Forward distillation.

[0043] Virtual Groups The strategies or behaviors are expected to show differentiation to diversify multi-agent interactions during the game in the matrix. Therefore, self-playing methods such as RPM explicitly store the historical learned policies of multiple agents and allow the agents to load these stored policies to cover as many decision-making modes as possible. This approach is effective in addressing the internal space challenge, as discussed in Figure 2. In this study, the present invention proposes a more concise solution by leveraging knowledge distillation to implicitly preserve the space of historical learned policies and avoid excessive resource consumption. Specifically, this involves using a distilled policy network To imitate the learning strategy network during the entire training process The training goal is to minimize the KL divergence between the distillation policy network and the learning policy network, as shown below:

[0044] ;

[0045] ;

[0046] In the formula, represents the KL divergence loss function, represents the learning policy network, represents the distillation strategy network, For intelligent agents The allocation status, It represents the KL divergence between the distillation strategy network and the learning strategy network. It represents the index of the agent, N represents the number of agents, and E represents the average of all divergences. represents the learning rate of forward distillation, represents the gradient of the update. The equation ensures that the distillation strategy closely mimics the learning strategy and makes similar decisions, hence the name forward distillation. It is worth noting that the learning strategy network is constantly updated in each iteration, while the forward distillation is periodically updated in Iterations are executed at the specified interval. This enables the distilled policy network to serve as a lagged representation of the continuously updated learned policy network until the next distillation update.

[0047] 3. Reverse distillation.

[0048] Forward distillation is inherently limited to the domain of historical strategies, just like self-playing methods. In some scenarios, such as the game of chicken in Google’s melting pot environment, trained teammates may have learned to prefer “dove” strategies, while unseen teammates may prefer “hawk” strategies. This emphasizes the importance of providing virtual groups with The importance of preference transitions is introduced to diversify interactions during training. Formally, given a state, the present invention hopes is achievable, where and are the preferred actions of the training group and the virtual group, respectively.

[0049] Based on this, the present invention proposes to use reverse distillation as follows:

[0050] ;

[0051] in, represents the distillation strategy network, represents the learning rate of back distillation, represents the updated gradient, represents the KL divergence loss function. This process aims to push the distilled policy away from the distribution of historical policies by increasing the KL divergence. In this way, the distilled policies can capture action preferences that are different from the learned policies, allowing them to explore and generalize outside the internal space distribution. Throughout the training process, forward distillation and backward distillation are performed alternately at regular intervals, each It is iterated once, hence the name bidirectional distillation. BiDist strikes a balance between preservation and innovation, ensuring that the distillation strategy inherits the strengths of its predecessor (the self-game strategy) while being given the ability to explore and adapt to new external space possibilities.

[0052] 4. Workflow.

[0053] In each policy iteration At the beginning, the present invention randomizes the index of the agent and then selects the agent from the distribution The vector extracted from , divide them into training groups or virtual groups . Training the population using a learning strategy network , while the virtual group uses the distillation strategy network Then, the training population and the fictitious population are deployed to run in the matrix and generate a diverse set of multi-agent trajectories that are designed to mimic those generated by interacting with the background population in the test scenario. , we instantiate the BiDist of the present invention on the MAPPO algorithm, which is an efficient online MARL algorithm based on the CTDE paradigm. The objective function can be formally defined as:

[0054] ;

[0055] in, and is the behavioral strategy for collecting trajectories, The action probability output by the current policy network; The action probabilities stored for the old policy network are used to limit the update range of the policy; represents the advantage function, which is used to measure the quality of an action; Indicates the clipping range, limiting the strategy update offset that is too large.

[0056] Parameters of the distillation strategy By alternating forward distillation and reverse distillation, each Iterates once, forming a hybrid self-game framework. In the forward phase, the virtual group follows the historical strategy space and forms an implicit self-game. On the other hand, in the reverse phase, the virtual group moves away from the historical strategy space and forms a non-self-game strategy.

[0057] In order to verify the effect of the algorithm in the game AI scenario, the embodiment of the present invention selected the MeltingPot environment proposed by Google, especially in the Coop Mining scenario for experiment and analysis, such as Figure 3 As shown in the figure. This scenario is highly collaborative and has complex dynamic changes, which is suitable for evaluating the generalization performance of multi-agent algorithms. In the Coop Mining scenario, multiple agents need to collaborate to mine in an environment with limited resources. There are multiple types of ores in the scenario, and each ore has different values, but the collection of ores requires the cooperation of multiple agents. For example, some high-value ores require two agents to stand next to the ore at the same time and work together to collect them successfully. At the same time, the agents also need to reasonably allocate the collection tasks to avoid inefficiency due to competition for the same ore resources. In addition, the scenario designs multiple dynamic variables to interfere with the behavior of the agents, including random refresh of ores, changes in task priorities, and unpredictable behavior of background groups (such as obstruction or resource grabbing). These dynamic characteristics make the Coop Mining scenario an ideal test scenario for evaluating the generalization ability of algorithms. In the training phase of the Coop Mining scenario, all agents are arranged in a unified environment for joint training, with the aim of learning how to maximize mining efficiency through collaboration. During the training process, the agents optimize their strategies in continuous interaction to cope with challenges such as random refresh of ore resources and dynamic task allocation. In the testing phase, in order to evaluate the generalization performance of the algorithm, some training agents are replaced with pre-trained background groups (agents that have not appeared in training), such as Figure 1 As shown in the figure, the yellow agents use pre-trained agent strategies. These background group agents may exhibit unknown behavior patterns, such as interfering with the mining process or uncooperative behavior. The remaining training agents (focus groups) need to quickly adjust their strategies based on the dynamic behavior of the background groups in the test scenario to ensure efficient completion of the task.

[0058] At the beginning of each strategy iteration, the present invention randomizes the index of the agent and calculates the index of the agent according to the distribution. The vector extracted from , the agents are divided into training groups and virtual groups. The training group uses the learning strategy network , while the virtual group uses the distillation strategy network Next, the training group and the virtual group interact in the environment to generate a diverse set of multi-agent trajectories. , the environment returns observation information for all agents and current status The agent follows the policy network or Select Action . Environment according to action Update Status , and feedback reward To optimize the learning strategy , this paper applies bidirectional distillation to the MAPPO algorithm, which is an effective on-policy multi-agent reinforcement learning method based on the centralized training decentralized execution (CTDE) paradigm.

[0059] In the Coop Mining scenario, all agents receive local observation information of the environment. , and formulate mining strategies based on this information. In the specific implementation: local observation This includes the encoding of environmental states, such as the location of mining sites, the amount of remaining resources at mining sites, the distribution of tool stations, etc. This includes the state of the agent itself, such as the amount of resources currently held, health points, equipment status, and the distance to the target mining site. These observations are processed by a convolutional neural network (CNN) to extract high-dimensional feature representations. High-dimensional features are mapped to the action space through a fully connected layer to generate a probability distribution for each optional action. The training group and the virtual group will execute strategies in the Coop Mining scenario to generate multi-agent interaction trajectories. For example, in a mining task, a trajectory might include the following events: Detect the nearest mining point and move to the target point. Select the action of mining or transporting resources and obtain the corresponding rewards. These trajectory data will be stored in the experience replay pool D for subsequent policy updates. In order to guide the optimization of the policy network, the present invention estimates the advantage function based on the time difference (TD) method: In the mining mission: Indicates the selected action in the current state The expected benefits that can be obtained, such as mining volume or transportation efficiency per unit time. It represents the benchmark value of the current state, such as the expected profit of the agent at the current mining point. In the Coop Mining scenario, in order to optimize the mining efficiency of the agent, the present invention will continuously update the policy network of the agent. The specific process is as follows: The present invention will maintain two learning policy networks, one is the distillation policy network , and the other is a learning strategy network , the present invention each passes The strategy is updated iteratively, and a bidirectional distillation step is performed to improve the generalization ability of the policy network: when the number of iterations is yes When the number is a multiple of , the knowledge of the learning strategy network is transferred to the distillation strategy network through the gradient descent method , the updated distillation strategy network retains the latest strategy knowledge in the current learning strategy network, and the formula is as follows:

[0060] ;

[0061] In the formula, represents the KL divergence loss function, represents the distillation strategy network, represents the learning rate of forward distillation, It represents the updated gradient. yes When the value is an even multiple of , reverse distillation is performed, and the formula is as follows:

[0062] ;

[0063] in, represents the distillation strategy network, represents the learning rate of forward distillation, represents the updated gradient, It represents the KL divergence loss function.

[0064] Distillation interval of the method Set to 5 to ensure a time lag between the distillation strategy and the learning strategy. For tasks involving fewer agents, it is 0.4, and for tasks involving more agents, it is 0.2. The learning rate in the forward phase The learning rate is 0.001, while the back distillation stage uses a smaller learning rate of 0.00001 to avoid infeasible or invalid distillation strategies.

[0065] After a specified number of training iterations, the present invention designs a systematic testing process to evaluate the performance of the algorithm, especially the generalization ability of the focus group when dealing with unseen background groups. In the testing phase, the present invention follows the following steps:

[0066] During the testing phase, some of the training agents (the focus group) are replaced with pre-trained background groups that did not participate in the training. The behavior patterns of the background group agents are unpredictable, such as interfering with resource collection, being completely uncooperative, or randomly exploring. The main difference between the test environment and the training environment is that all agents in the training collaborate, while the focus group in the test environment needs to independently deal with the dynamic interaction with the background group.

[0067] To ensure the stability and reliability of the test results, the present invention conducted 10 repeated tests for each test scenario. In each test:

[0068] Initial state randomization: The initial ore distribution, agent positions, and task priorities of the test scenario are randomly initialized to ensure the diversity of test conditions.

[0069] Fixed strategy: The strategy network of the focus group is fixed during the test and no updates are performed, while the behavior strategy of the background group remains dynamically changing.

[0070] Observation feedback: During the test, each agent’s local observations, action choices, and reward values ​​are recorded to evaluate the behavioral adaptability of the focus group.

[0071] In each test, the present invention records the single-round reward of each agent in the focus group. Then the average reward of 10 tests is calculated to eliminate the random influence in a single test. In addition, in order to further reduce statistical errors, the present invention tests three different training seeds:

[0072] Training seed differences: By adjusting the initial parameter configuration of training, multiple seeds are generated to ensure that the model can cope with different initial conditions.

[0073] Average result: The test results generated by all seeds are averaged to get the final focus group return value.

[0074] In order to intuitively compare the test performance in different scenarios, the present invention scales the focus group return value of each scenario to the range of 0 to 1 and uses the following formula for normalization:

[0075] ;

[0076] This normalization process eliminates the impact of the difference in return range between scenarios on the comparison results, making the results more comparable. The present invention verifies the algorithm in the Melting Pot environment proposed by Google, which involves five environments:

[0077] 1. Pure Coordination: Players need to cooperate and coordinate to collect resources of the same color to reach a consensus and get rewards.

[0078] 2. Coop Mining: Players need to cooperate to mine gold, but they can also mine iron alone. Coop mining will get higher rewards.

[0079] 3. Chicken: Players adopt a "hawk" or "dove" strategy when collecting resources of different colors, trying to exploit each other, but this may lead to suboptimal results for both parties.

[0080] 4. Coins: Two players collect coins of different colors. Collecting coins of the same color will result in negative rewards, so you need to carefully balance your collection strategy.

[0081] 5. Prisoners Dilemma: Players encounter a prisoner's dilemma when collecting resources and need to balance the conflict between individual rewards and group rewards.

[0082] like Figure 4 As shown, the BiDist method of the present invention consistently achieves higher normalized focus per capita returns under different dynamic conditions.

[0083] The embodiments described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above are only specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A bidirectional distillation method suitable for game AI, characterized in that: The following steps are involved: In each iteration of training, some agents are randomly separated from the training group of game AI agents as virtual groups, and the distilled strategy network is assigned to them. The virtual groups are used to simulate the behavior of zero-shot agents. The virtual groups use a different strategy distribution from the training group and simulate game scenarios that have never been seen before by randomly generating opponent or teammate behavior patterns. The training group and the virtual group are deployed in the game for training. The training group is updated using the learning strategy network, and the virtual group is updated using the distillation strategy network. The distillation strategy network includes forward distillation and reverse distillation. When performing forward distillation, the KL divergence between the distillation strategy network and the learning strategy network is minimized to achieve implicit self-game, so as to retain historical strategy knowledge and increase the diversity of interaction patterns. When performing reverse distillation, the KL divergence between the distillation strategy network and the learning strategy network is maximized to promote the preference transfer of the strategy of the virtual group to explore new strategy distributions. Repeat the above steps for multiple iterative training. After the training is completed, use the updated training population as the final game AI agent and deploy it into the game.

2. The bidirectional distillation method for game AI according to claim 1, characterized in that: In each training iteration, the training population and the dummy population are randomly divided using a binary vector.

3. The bidirectional distillation method for game AI according to claim 2, characterized in that: When dividing the training population and the virtual population, ensure that at least one agent remains in the training population.

4. The bidirectional distillation method for game AI according to claim 1, characterized in that: In each iterative training, updates are performed through alternating optimization of forward distillation and backward distillation.

5. The bidirectional distillation method for game AI according to claim 1, characterized in that: The loss function for the KL divergence between the distillation policy network and the learning policy network is defined as follows: ； In the formula, represents the KL divergence loss function, represents the learning policy network, represents the distillation strategy network, Indicates that in the state Select Action , For intelligent agents The allocation status, represents the KL divergence between the distillation policy network and the learning policy network, N represents the number of agents, and E represents the average of all divergences.

6. The bidirectional distillation method for game AI according to claim 1, characterized in that: The gradient update formula for forward distillation is: ； in, represents the distillation strategy network, represents the learning rate of forward distillation, represents the updated gradient, It represents the KL divergence loss function.

7. The bidirectional distillation method for game AI according to claim 1, characterized in that: The gradient update formula for reverse distillation is: ； in, represents the distillation strategy network, represents the learning rate of back distillation, represents the updated gradient, It represents the KL divergence loss function.

8. A two-way distillation device suitable for game AI, characterized in that: It comprises a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, it is used to implement the bidirectional distillation method suitable for game AI described in any one of claims 1-7.

Images