A game AI training method and device, electronic equipment and storage medium

Abstract

The application provides a game AI training method and device, electronic equipment and a storage medium. By introducing attention to all players in a game scene and predicting the macro decision of all players in a game, the perception of the game AI to the behaviors of other players can be enhanced, the data utilization rate in model training can be effectively improved by fully utilizing the data of all players in each game, the humanization of the game AI can be improved, the game AI is more realistic, and the application can be effectively applied to different application scenarios of various competitive games to improve the user experience of game players.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence, and more specifically, to a game AI training method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the rapid development of artificial intelligence (AI), there are now many cases of AI being applied to games, such as the famous Google AlphaGo AI and Tencent's King of Glory AI.

[0003] For game AI, the biggest variable in a game is always the other human players. Therefore, improving its perception of other players' behavior is crucial. Making appropriate actions based on the current game state is relatively simple; the challenge lies in how the AI ​​adjusts itself according to the actions of teammates and opponents. Furthermore, it's even more difficult to predict the strategic goals of teammates and opponents in the coming period and to coordinate and evade them appropriately. Simply put, current game AIs are often better at quick reactions and subtle maneuvers, but they tend to be less adept at perceiving others and understanding overall strategy.

[0004] In competitive games, both skillful execution and strategic thinking are indispensable for skilled players. This is especially true in turn-based games, where strategic thinking is paramount. However, skillful execution alone is insufficient for training high-level game AI. Therefore, researching and developing high-level, human-like AI systems with a high degree of awareness of others is a pressing issue for competitive game AI.

[0005] Currently, the technical solution adopted to address the above-mentioned needs is a MOBA game AI solution based on deep neural networks and supervised learning. This solution is based on a hierarchical macro-strategy model, using fully connected networks and convolutional networks to abstract backbone features, and then outputting macro-level and micro-level decisions through a hierarchical network. However, this solution currently has the following drawbacks: low data utilization efficiency, as there are usually multiple players in a game scenario, but this solution only utilizes the label of one player; weak local perception capability, as it is unclear what actions teammates will take to coordinate with them, and unable to predict what actions enemies will take to avoid them in advance; and a lack of overall perception capability, as it is unclear about the strategic objectives of teammates and enemies in the coming period, lacking a holistic view. Summary of the Invention

[0006] To address the aforementioned issues, this invention provides a game AI training method, apparatus, electronic device, and storage medium.

[0007] A first aspect of this invention provides a game AI training method, the method comprising:

[0008] Extract feature information of all game characters in the game environment. The feature information includes image features and vector features. The game characters include game characters controlled by game AI and game characters controlled by human players.

[0009] The feature information is input into the strategy model;

[0010] The strategy model outputs the current decision-making strategy of all game characters based on the input content. The decision-making strategy includes macro-decision and micro-action.

[0011] The game character, controlled by the game AI, performs interactive actions corresponding to the decision-making strategy and interacts with the game environment.

[0012] Optionally, the extraction of feature information of the agent in the game environment specifically includes:

[0013] Image features are extracted based on the position data of the game character in the game environment;

[0014] Vector features are extracted based on the game character's basic game data in the game environment.

[0015] Optionally, the steps by which the strategy model outputs the current decision-making strategies of all game characters based on the input content specifically include:

[0016] The vector features are abstracted and transformed into vectors through multiple fully connected layers;

[0017] The image features are processed through multiple convolutional layers, then encoded by another convolutional layer, and then flattened into vectors.

[0018] The processed vector features and image features are merged into the backbone features through a fully connected layer;

[0019] The core features are input into a multi-layer, multi-supervised macro-decision network, which outputs the macro-decision of all game characters.

[0020] After concatenating the macro-level decisions with the core features, the data is input into a multi-layered, multi-supervised micro-level decision network to obtain the micro-level actions of all game characters.

[0021] Optionally, the macro-decision network is composed of an MLP network with a small number of parameters. After passing through multiple fully connected layers, it outputs N macro-decision results through N fully connected layers, where N is the number of game characters in the game environment.

[0022] Optionally, the micro-decision network is composed of an MLP network with a large number of parameters. After passing through multiple fully connected layers, the results of N micro-decisions are output through N fully connected layers respectively.

[0023] Optionally, the training method of the policy model includes:

[0024] Acquire the game data for the entire game, and read the game data of the game character controlled by the human player. The game data contains all the information of every frame of the entire game.

[0025] According to a certain time period or preset strategy, extract the feature information and corresponding action tags of each game character in sequence;

[0026] The feature information and action tags are stored in a specified dataset, and the feature information and action tags are saved in two different sets respectively. The mapping relationship between features and tags is established by using a unique game match ID and a unique game frame number.

[0027] Train a policy model on the dataset until the model converges, save the model parameter file, and determine whether the model has converged by judging whether the loss decrease has reached a plateau.

[0028] Optionally, the tags include macro-decision tags and micro-behavioral tags. When extracting action tags of game characters, macro-decision tags are extracted first, and then corresponding micro-behavioral tags are extracted based on the content of the macro-decision tags.

[0029] A second aspect of the present invention provides a game AI training device, the device comprising:

[0030] The feature extraction unit is used to extract feature information of all game characters in the game environment. The feature information includes image features and vector features. The game characters include game characters controlled by game AI and game characters controlled by human players.

[0031] An action decision unit is used to input the feature information into a strategy model. The strategy model outputs the current decision strategy for all game characters based on the input. The decision strategy includes macro-level decisions and micro-level actions.

[0032] The action interaction unit is used to control the game character controlled by the game AI to perform interactive actions corresponding to the decision-making strategy and interact with the game environment.

[0033] Optionally, the apparatus further includes a model training unit for training a policy model, wherein the training method of the policy model includes:

[0034] Acquire the game data for the entire game, and read the game data of the game character controlled by the human player. The game data contains all the information of every frame of the entire game.

[0035] According to a certain time period or preset strategy, extract the feature information and corresponding action tags of each game character in sequence;

[0036] The feature information and action tags are stored in a specified dataset, and the feature information and action tags are saved in two different sets respectively. The mapping relationship between features and tags is established by using a unique game match ID and a unique game frame number.

[0037] Train the policy model on the dataset until the model converges, save the model parameter file, and determine whether the model has converged by judging whether the loss decrease has reached a plateau.

[0038] A third aspect of the present invention provides an electronic device, characterized in that it comprises:

[0039] One or more processors; memory; one or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, the one or more applications being configured to perform the method as described in the first aspect.

[0040] A fourth aspect of the present invention provides a computer-readable storage medium, characterized in that the computer-readable storage medium stores program code that can be invoked by a processor to execute the method described in the first aspect.

[0041] In summary, this invention provides a game AI training method, apparatus, electronic device, and storage medium. By incorporating consideration of all players in a game scenario and predicting the macro-level decisions of all players in a match, it enhances the game AI's perception of other players' behaviors, fully utilizes data from all players in each game, effectively improves data utilization during model training, enhances the human-likeness of the game AI, and makes the game AI more realistic. It can be effectively applied to various application scenarios in competitive games, improving the user experience for gamers. Attached Figure Description

[0042] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1This is a schematic diagram illustrating an application scenario of the game AI training method and apparatus according to an embodiment of the present invention;

[0044] Figure 2 This is a flowchart of the game AI training method according to an embodiment of the present invention;

[0045] Figure 3 This is a flowchart illustrating a method for the strategy model of this invention to output the current decision-making strategies of all game characters based on the input content.

[0046] Figure 4 This is a flowchart illustrating the training method for the strategy model provided in an embodiment of the present invention.

[0047] Figure 5 This is a functional block diagram of the game AI training device according to an embodiment of the present invention;

[0048] Figure 6 This is a structural block diagram of an electronic device for executing a game AI training method according to an embodiment of the present invention;

[0049] Figure 7 This is a structural block diagram of a computer-readable storage medium for storing or carrying program code implementing the game AI training method according to an embodiment of the present invention.

[0050] icon:

[0051] AI server 100; game environment server 200; feature extraction unit 110; action decision unit 120; action interaction unit 130; model training unit 140; electronic device 300; processor 310; memory 320; computer-readable storage medium 400; program code 410. Detailed Implementation

[0052] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided to make the invention more comprehensive and complete, and to fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a full understanding of embodiments of the invention. However, those skilled in the art will recognize that the technical solutions of the invention may be practiced with one or more of these specific details omitted, or other methods, components, apparatus, steps, etc., may be employed. In other instances, well-known technical solutions are not shown or described in detail to avoid obscuring various aspects of the invention.

[0053] Furthermore, the accompanying drawings are merely illustrative of the invention and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0054] Before providing a further detailed description of the embodiments of the present invention, the nouns and terms involved in the embodiments of the present invention will be explained, and the nouns and terms involved in the embodiments of the present invention shall be interpreted as follows.

[0055] (1) In response to, used to indicate the conditions or states on which the operation is performed depends. When the conditions or states on which the operation is performed are met, one or more operations may be performed in real time or with a set delay. Unless otherwise specified, there is no restriction on the order in which the multiple operations are performed.

[0056] (2) Based on, used to indicate the conditions or states on which the operation is performed depends. When the conditions or states on which it depends are met, one or more operations can be performed in real time or with a set delay. Unless otherwise specified, there is no restriction on the order of execution of the multiple operations.

[0057] (3) Model training: Multi-class classification learning on the image dataset. This model can be constructed using deep learning frameworks such as TensorFlow and Torch, employing a combination of multiple neural network layers, such as CNN, to form a multi-class classification model. The model input is a three-channel or original-channel matrix formed by reading images using tools such as OpenCV. The model output is the multi-class probability, ultimately outputting the webpage category through algorithms such as soft max. During training, the model approximates the correct trend using objective functions such as cross-entropy.

[0058] (4) Neural Network (NN): Artificial Neural Network (ANN), also known as neural network or neural network-like network, is a mathematical or computational model in the fields of machine learning and cognitive science that imitates the structure and function of biological neural networks (the central nervous system of animals, especially the brain) and is used to estimate or approximate functions.

[0059] (5) Game Environment: This is the game environment displayed (or provided) by the application when it runs on the terminal. The game environment can be a simulation of the real world, a semi-simulated / semi-fictional three-dimensional environment, or a purely fictional three-dimensional environment. The game environment can be any of a two-dimensional game environment, a 2.5-dimensional game environment, or a three-dimensional game environment. The following embodiments use a three-dimensional game environment as an example, but this is not a limitation. Optionally, the game environment is also used for game environment battles between at least two virtual objects. Optionally, the game environment is also used for battles between at least two virtual objects through the rolling of a virtual sphere. Optionally, the game environment can also be, but is not limited to, shooting games, parkour games, racing games, multiplayer online battle arena (MOBA), racing games (RCG), and sports games (SPG). The trained game model provided in this application can be deployed in game servers corresponding to the aforementioned game scenarios to generate real-time game strategies, execute corresponding action information, simulate the operation of virtual users, and jointly complete different types of games in the game environment with actual users participating in the game.

[0060] (6) Action information: Taking the game user's first-person or third-person participation in speed competition, including racing, flying and other racing games as an example, action information refers to the operation instructions such as the directional keys that control the moving object as actions. For role-playing games, action information refers to virtual weapons that attack by firing bullets in the game environment, or virtual bows and arrows that fire arrows. Virtual objects in the game environment can pick up virtual guns and attack with the virtual guns they have picked up.

[0061] The game AI can be a user-controlled virtual object that can be operated through client-side actions, or it can be artificial intelligence (AI) trained and set up for combat in the game environment, or it can be a non-user virtual object (NPC / Non-Player Character) set up for interaction in the game environment. Optionally, the virtual object can be a virtual character competing in the game environment. Optionally, the number of virtual objects participating in the interaction in the game environment can be preset or dynamically determined according to the number of clients joining the interaction.

[0062] Example

[0063] Please see Figure 1 This embodiment provides an application scenario diagram of a game AI training method and device.

[0064] like Figure 1 As shown, this invention provides a game AI training method and apparatus, applied to an AI server 100 and a game environment server 200. The AI ​​server 100 is used to train the algorithm model, and the game environment server 200 is used to implement the game environment. In actual operation, the AI ​​server 100 connects to the game environment server 200. The game environment server 200 sends the current state of the game environment to the AI ​​server 100. The AI ​​server 100 extracts useful information from the game environment and transforms it into features that the model can recognize. These features are then passed to the strategy model for prediction. Finally, the prediction result is sent to the game environment server 200 for specific execution.

[0065] It should be noted that the AI ​​server 100 and the game environment server 200 can be either cloud servers located in the cloud or local servers located on a local machine. The AI ​​server 100 and the game environment server 200 can be functionally differentiated servers based on the same physical device, or they can be functionally differentiated servers based on different physical devices.

[0066] Based on the above, such as Figure 2 As shown, this is a game AI training method provided in an embodiment of the present invention, the method comprising:

[0067] Step S101: Extract feature information of all game characters in the game environment, including image features and vector features; the game characters include game characters controlled by game AI and game characters controlled by human players.

[0068] Image features and vector features correspond to different dimensions of information in the game environment. Specifically, image features are mainly based on the image content corresponding to the game character's location in the environment; therefore, image features are extracted based on the game character's position data in the game environment. Vector features are mainly based on attribute-related content in the environment; vector features are extracted based on the game character's basic game data in the game environment. Vector features consist of 1-dimensional vectors and mainly contain basic game data. Image features consist of 2-dimensional vectors and contain spatial information.

[0069] Taking a specific game environment as an example, image features are used to express spatial information within the game, such as the distribution of different characters on the minimap in a multiplayer online battle arena (MOBA) game. In a two-dimensional matrix, a value of 1 at a certain position represents the presence of an object at that position, while a value of 0 indicates that the position is empty. The vector features of the model include: current character features, which in competitive games often include the character's health, level, attack, and defense; other character features, including the game status of teammates and enemies; features of other units in the game, such as monsters; and global features, such as game duration.

[0070] Game characters in the game environment are divided into two categories based on their controllers: those controlled by game AI and those controlled by human players. Feature extraction is performed on all game characters within the game environment.

[0071] Step S102: Input the feature information into the strategy model.

[0072] In the game environment, based on the number of players required and the number of human players involved, multiple game characters controlled by the game AI are usually generated to play the game. The game AI determines the decision-making strategy through a pre-configurable strategy model. The decision-making strategy made by the strategy model should fully consider its better perception of teammates and enemies, and its ability to cooperate more with players in the game. Therefore, it is necessary to input the feature information of all game characters into the strategy model.

[0073] The strategy model configuration for the game AI is performed during game environment deployment, using previously trained and saved historical models. As a preferred embodiment of the invention, the model can be selected based on the historical game performance of human players in the game environment.

[0074] For example, in a "warm-up" match: for players experiencing a losing streak, multiple AIs capable of sensing each other can be introduced to play together, thus providing a more supportive experience. Traditional game AI often leads players to victory by playing with all their might and striving for win. However, for players on a losing streak, this approach isn't necessarily beneficial and may even make them feel like outsiders. This new game AI, with its better perception of teammates and enemies, can cooperate more effectively with the player, assisting them in achieving better performance and resulting in a more enjoyable gaming experience.

[0075] For example, in tutorials: AI can be used to teach and guide new players. This game's AI not only provides its own best decision but also predicts the future goals of other players based on their current behavior. Therefore, it can be used for tutorials, evaluating the intentions behind a new player's actions, scoring their decisions, and suggesting the best course of action. For instance, if other teammates are attacking while the player chooses to defend, the AI ​​will score the player's decision and provide the correct advice, helping teammates attack a specific target.

[0076] Step S103: The strategy model outputs the current decision-making strategy of all game characters based on the input content. The decision-making strategy includes macro-decision and micro-action.

[0077] After feature information is input into the policy model, the policy model outputs a corresponding decision policy based on the input feature information. For different game characters, due to their different specific locations and attributes, the corresponding decision policies also differ. For different game AIs in different game scenarios, a decision policy corresponding to themselves is determined based on the output results.

[0078] Decision-making strategies comprise two parts of information: macro-level decisions, which correspond to the actions that should be taken at the macro level under the current circumstances, such as moving to a certain location or attacking a certain target; and micro-level actions, which correspond to the specific operational actions based on the macro-level decisions.

[0079] As a preferred embodiment of the present invention, based on the complexity of the actions performed by the game character in the game environment, the micro-actions can be further divided into multiple levels, each corresponding to a different dimension of the specific operation. Taking two levels as an example, based on the above macro-decision, the first level of the micro-actions represents the current behavior, including movement, attack, no action, releasing skills, etc. The second level represents the sub-tags under the current action, such as the position of movement, the target of attack, the direction, position, and target of releasing skills, etc.

[0080] Step S104: Control the game character controlled by the game AI to perform the interactive actions corresponding to the decision strategy and interact with the game environment.

[0081] The strategy model outputs the current decision-making strategies for all game characters based on the input. The game AI then determines its own corresponding decision-making strategy based on the output. It then controls the corresponding game character to perform corresponding interactive actions within the game environment—the micro-actions—to interact with the game environment (including other players, fixed NPCs, and scene props).

[0082] As a preferred embodiment of the present invention, the specific method by which the strategy model outputs the current decision-making strategies of all game characters based on the input content is as follows: Figure 3 As shown, it includes:

[0083] Step S201: The vector features are abstracted into vectors through multiple fully connected layers;

[0084] Step S202: The image features are processed through multiple convolutional layers, then encoded by another convolutional layer, and then flattened into vectors.

[0085] Step S203: The processed vector features and image features are merged into backbone features through a fully connected layer;

[0086] Step S204: Input the backbone features into a multi-layer, multi-supervised macro decision network and output the macro decisions for all game characters.

[0087] Step S205: After concatenating the macro-decision with the backbone features, input the data into a multi-layer, multi-supervised micro-decision network to obtain the micro-actions of all game characters.

[0088] In this embodiment, vector features consist of 1-dimensional vectors, mainly containing basic game data, and are processed primarily using MLP (Multilayer Perceptron) in the model. Image features consist of 2-dimensional vectors, containing spatial information, and are processed primarily using CNN in the model.

[0089] After the vector features and image features are processed through the two networks described above, they are transformed into two features containing different high-dimensional information. These two high-dimensional features are concatenated to obtain the backbone feature, which is then flattened. The backbone feature is also a vector. The flattening operation is simply rearranging a two-dimensional matrix into a one-dimensional vector, a single-step operation. The backbone feature is obtained by concatenating the two parts.

[0090] The macro-decision network consists of an MLP network with a small number of parameters. After passing through multiple fully connected layers, it outputs N macro-decision results through N fully connected layers, where N is the number of game characters in the game environment.

[0091] The micro-decision network consists of an MLP network with a large number of parameters. After passing through multiple fully connected layers, it outputs N micro-decision results through N fully connected layers respectively.

[0092] In the embodiments provided by this invention, the strategy model is a neural network model, which can be trained offline using supervised learning based on a large amount of human player interaction and game data. The specific training method is as follows: Figure 4 As shown, it includes:

[0093] Step S206: Obtain the game data for the entire game, and read the game data of the game character controlled by the human player. The game data contains all the information of every frame of the entire game.

[0094] Step S207: Extract the feature information and corresponding action tags of each game character sequentially according to a certain time period or preset strategy;

[0095] As a preferred implementation method in this embodiment, feature information and corresponding action tags can be extracted frame-by-frame or sequentially according to time nodes, extracting the features and corresponding tags of each player in turn. This process selects important time nodes, such as retaining more actions like attacking and releasing skills, and reducing actions like long-distance movement, thereby improving the efficiency of training data utilization.

[0096] In this implementation, the strategy model employs a multi-supervised strategy model to extract game-related features and corresponding action labels from the data of all human players in each game. Features include vector features and image features. Supervised training is performed using all player data, with hierarchical training of macro-level and micro-level decisions, ultimately yielding the AI ​​strategy model. The AI ​​strategy model incorporates CNN and MLP (Multilayer Perceptron).

[0097] The action labels used to train the strategy model include macro-decision labels and micro-behavioral labels. When extracting action labels for game characters, macro-decision labels are extracted first, and then the corresponding micro-behavioral labels are extracted based on the content of the macro-decision labels. The specific content of the action labels corresponds to the decision-making strategies output by the strategy model mentioned above.

[0098] The macro-decision label corresponds to the actions that the current game character should take at the macro level.

[0099] For example, a macro-level decision is a character's goal after a certain period of time, such as moving to a certain location or attacking a certain target. Macro-level decision labels include two tags: future movement and attack, each represented by a 2D matrix. Each position in the 2D matrix corresponds to a region on the game map. If, after a certain period of time, the game character moves to a region on the game map, the corresponding movement tag has a value of 1 for that position and 0 for other positions. If, after a certain period of time, the game character performs an attack in a region on the game map, the corresponding attack tag has a value of 1 for that position and 0 for other positions.

[0100] Micro-behavioral tags, based on the content of the micro-actions, can also be designed as multi-level, multi-classification questions. Again, using a two-level example, the first level represents the current action, including movement, attack, no action, and skill activation. The second level represents sub-tags under the current action, such as the position of movement, the target of an attack, and the direction, position, and target of skill activation. When no information is sent, there are no sub-tags. Different actions use different encoding methods as sub-tags; for example, when the main tag result is "movement," the sub-tag would be the coordinates of the corresponding position.

[0101] To better understand other players, the strategy model extracts action tags for each role. For example, in a competitive game with 10 players, action tags are extracted for the actions of each of the 10 players.

[0102] Step S208: Store the feature information and action tags in a designated dataset. The feature information and tags are saved in two different sets. A mapping relationship between features and tags is established through a unique game match ID and a unique game frame number.

[0103] Unique identifiers enable convenient and rapid location and use of data with mapping relationships. As a preferred implementation, features and tags can be stored in TFRecords.

[0104] Step S209: Train the policy model on the dataset until the model converges, and save the model parameter file; determine whether the model has converged by judging whether the loss decrease has reached a plateau.

[0105] During training, the extracted feature information is input into the decision model, and the output decision strategy is compared with the extracted action labels. The model training loss uses the cross-entropy formula shown below:

[0106] -(y log(p)+(1-y)log(1-p))

[0107] Where y refers to the label during training, and p refers to the probability that the model predicts that label.

[0108] According to the above method, a policy model is trained on the dataset. The model convergence is determined by whether the loss decreases to a plateau. When the model converges, the model parameter file is saved. During training, multiple different historical policy models are generated, each with varying performance. In practical applications, a suitable historical policy model can be selected and loaded into the game AI based on specific needs.

[0109] As a preferred embodiment of this example, based on the existing strategy model structure, more complex network designs, such as Residual Network (ResNet) and Long Short-Term Memory (LSTM) networks, can be added according to the needs of the actual scenario.

[0110] As a preferred embodiment of this example, based on the hierarchical decision-making structure proposed above, macro-decision and micro-decision can also be decoupled by training two separate models through model splitting.

[0111] In summary, the game AI training method provided in this embodiment, by incorporating attention to all players in the game scene and predicting the macro-level decisions of all players in a match, can enhance the game AI's perception of other players' behaviors, fully utilize the data from all players in each game, effectively improve data utilization during model training, enhance the anthropomorphism of the game AI, and make the game AI more realistic. It can be effectively applied to various application scenarios in competitive games, improving the user experience for gamers.

[0112] like Figure 5 As shown, the game AI training device provided by this invention includes:

[0113] The feature extraction unit 110 is used to extract feature information of all game characters in the game environment. The feature information includes image features and vector features. The game characters include game characters controlled by game AI and game characters controlled by human players.

[0114] Action decision unit 120 is used to input the feature information into the strategy model. The strategy model outputs the current decision strategy of all game characters based on the input content. The decision strategy includes macro-decision and micro-action.

[0115] The action interaction unit 130 is used to control the game character controlled by the game AI to perform interactive actions corresponding to the decision strategy and interact with the game environment.

[0116] In a preferred embodiment of the present invention, the apparatus further includes a model training unit 140 for training a policy model, wherein the training method of the policy model includes:

[0117] Acquire the game data for the entire game, and read the game data of the game character controlled by the human player. The game data contains all the information of every frame of the entire game.

[0118] According to a certain time period or preset strategy, extract the feature information and corresponding action tags of each game character in sequence;

[0119] The feature information and action tags are stored in a specified dataset, and the feature information and action tags are saved in two different sets respectively. The mapping relationship between features and tags is established by using a unique game match ID and a unique game frame number.

[0120] Train a policy model on the dataset until the model converges, save the model parameter file, and determine whether the model has converged by judging whether the loss decrease has reached a plateau.

[0121] The game AI training device provided in this embodiment of the invention is used to implement the above-described game AI training method. Therefore, the specific implementation method is the same as the above method and will not be repeated here.

[0122] like Figure 6 The diagram shows a structural block diagram of an electronic device 300 provided in an embodiment of the present invention. This electronic device 300 can be a smartphone, tablet computer, e-reader, or other electronic device capable of running applications. The electronic device 300 in this application may include one or more of the following components: a processor 310, a memory 320, and one or more applications, wherein the one or more applications can be stored in the memory 320 and configured to be executed by one or more processors 310, and the one or more applications are configured to perform the methods described in the foregoing method embodiments.

[0123] Processor 310 may include one or more processing cores. Processor 310 connects to various parts within the electronic device 300 using various interfaces and lines, and performs various functions and processes data of the electronic device 300 by running or executing instructions, programs, code sets, or instruction sets stored in memory 320, and by calling data stored in memory 320. Optionally, processor 310 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). Processor 310 may integrate one or a combination of several of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the displayed content; and the modem handles wireless communication. It is understood that the modem may also not be integrated into processor 310 and may be implemented separately using a communication chip.

[0124] The memory 320 may include random access memory (RAM) or read-only memory (ROM). The memory 320 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 320 may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as touch functionality, sound playback functionality, image playback functionality, etc.), and instructions for implementing the various method embodiments described below. The data storage area may also store data created by the terminal during use (such as phonebook data, audio and video data, chat log data, etc.).

[0125] like Figure 7 The diagram shows a structural block diagram of a computer-readable storage medium 400 provided in an embodiment of the present invention. The computer-readable medium stores program code 410, which can be called by a processor to execute the methods described in the above method embodiments.

[0126] The computer-readable storage medium 400 may be an electronic memory such as flash memory, EEPROM (Electrically Erasable Programmable Read-Only Memory), EPROM, hard disk, or ROM. Optionally, the computer-readable storage medium 400 includes a non-transitory computer-readable storage medium. The computer-readable storage medium 400 has storage space for program code 410 that performs any of the method steps described above. This program code 410 can be read from or written to one or more computer program products. The program code 410 may, for example, be compressed in a suitable form.

[0127] In summary, this invention provides a game AI training method, apparatus, electronic device, and storage medium. By incorporating consideration of all players in a game scenario and predicting the macro-level decisions of all players in a match, it enhances the game AI's perception of other players' behaviors, fully utilizes data from all players in each game, effectively improves data utilization during model training, enhances the human-likeness of the game AI, and makes the game AI more realistic. It can be effectively applied to various application scenarios in competitive games, improving the user experience for gamers.

[0128] In the embodiments disclosed in this application, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0129] In addition, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0130] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A method for training game AI, characterized in that, The method includes: Extract feature information of all game characters in the game environment. The feature information includes image features and vector features. The game characters include game characters controlled by game AI and game characters controlled by human players. The feature information is input into the strategy model; The strategy model outputs the current decision-making strategy of all game characters based on the input content. The decision-making strategy includes macro-decision and micro-action. Control the game character, controlled by the game AI, to perform interactive actions corresponding to the decision-making strategy and interact with the game environment; The steps of the strategy model outputting the current decision-making strategies of all game characters based on the input content specifically include: The vector features are abstracted and transformed into vectors through multiple fully connected layers; The image features are processed through multiple convolutional layers, then encoded by another convolutional layer, and then flattened into vectors. The processed vector features and image features are merged into backbone features through a fully connected layer; The core features are input into a multi-layer, multi-supervised macro-decision network, which outputs the macro-decision of all game characters. After concatenating the macro-level decisions with the core features, the data is input into a multi-layered, multi-supervised micro-level decision network to obtain the micro-level actions of all game characters.

2. The game AI training method according to claim 1, characterized in that, The extraction of feature information of all game characters in the game environment specifically includes: Image features are extracted based on the position data of the game character in the game environment; Vector features are extracted based on the game character's basic game data in the game environment.

3. The game AI training method according to claim 2, characterized in that, The macro-decision network consists of an MLP network with a small number of parameters. After passing through multiple fully connected layers, it outputs N macro-decision results through N fully connected layers, where N is the number of game characters in the game environment.

4. The game AI training method according to claim 3, characterized in that, The micro-decision network consists of an MLP network with a large number of parameters. After passing through multiple fully connected layers, it outputs N micro-decision results through N fully connected layers respectively.

5. The game AI training method according to claim 4, characterized in that, The training method for the policy model includes: Acquire the game data for the entire game, and read the game data of the game character controlled by the human player. The game data contains all the information of every frame of the entire game. According to a certain time period or preset strategy, extract the feature information and corresponding action tags of each game character in sequence; The feature information and action tags are stored in a specified dataset, and the feature information and action tags are saved in two different sets respectively. The mapping relationship between features and tags is established through a unique game match ID and a unique game frame number. Train the policy model on the dataset until the model converges, and then save the model parameter file.

6. The game AI training method according to claim 5, characterized in that, The tags include macro-decision tags and micro-behavioral tags. When extracting action tags for game characters, macro-decision tags are extracted first, and then the corresponding micro-behavioral tags are extracted based on the content of the macro-decision tags.

7. A game AI training device, characterized in that, The device includes: The feature extraction unit is used to extract feature information of all game characters in the game environment. The feature information includes image features and vector features. The game characters include game characters controlled by game AI and game characters controlled by human players. An action decision unit is used to input the feature information into a strategy model. The strategy model outputs the current decision strategy for all game characters based on the input. The decision strategy includes macro-level decisions and micro-level actions. The action interaction unit is used to control the game character controlled by the game AI to perform interactive actions corresponding to the decision-making strategy and interact with the game environment; The action decision unit is further configured to abstract the vector features into vectors through multiple fully connected layers; process the image features through multiple convolutional layers, encode them through another convolutional layer, and then flatten them into vectors; merge the processed vector features and image features through a fully connected layer to form backbone features; input the backbone features into a multi-layer, multi-supervised macro-decision network to output the macro-decision of all game characters; and concatenate the macro-decision with the backbone features and input them into a multi-layer, multi-supervised micro-decision network to obtain the micro-actions of all game characters.

8. An electronic device, characterized in that, include: One or more processors; Memory; One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, the one or more applications being configured to perform the method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores program code that can be invoked by a processor to execute the method as described in any one of claims 1-6.

Images