Game agent determination, game running method, device, medium and electronic equipment

Abstract

The present disclosure provides a game agent determination method, a game running method, a device, a storage medium and an electronic device, and relates to the technical field of artificial intelligence. The game agent determination method comprises: training an evaluation network model in an initial game agent based on perfect features and imperfect features in a training sample, and training a decision network model in the initial game agent based on the imperfect features; obtaining a pretreatment game agent by updating model parameters of the initial game agent based on trained model parameters; continuing to train the evaluation network model and the decision network model by taking the pretreatment game agent as the initial game agent, and determining the pretreatment game agent as a target game agent when a preset training target is reached, thereby solving the technical problem of low overall training efficiency of the current imperfect information game agent and achieving the technical effect of improving the overall training efficiency of the current imperfect information game agent.

Description

Game agent identification, game operation methods, devices, media and electronic devices Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to a method for determining a game agent, a game operation method, an apparatus, a storage medium, and an electronic device. Background Technology

[0002] Game bots have always been an important part of game design, and AI (Artificial Intelligence) technology is being increasingly widely used in building game bots. However, a significant portion of games involve imperfect information, such as Dou Dizhu (a popular Chinese card game) and Mahjong. Despite their popularity, the imperfect information nature of these games makes it difficult to build high-level bots. Current game agents (a reinforcement learning term referring to models capable of making decisions based on given tasks and environments) are trained on imperfect information, primarily using tree search or rule-based methods to mitigate the potentially huge search space problem caused by imperfect information.

[0003] However, due to the inherent strategic nature and high complexity of this type of game, simply adopting rules can only reduce the search space to a certain extent. Humanly considering all possible card situations and providing corresponding rules is extremely difficult, requiring enormous computing power to enumerate all potential card-playing possibilities, thus resulting in the low overall training efficiency of current imperfect information game agents. Summary of the Invention

[0004] This disclosure provides a method for determining game agents, a game operation method, a device, a storage medium, and an electronic device, thereby improving the overall training efficiency of currently imperfect information game agents.

[0005] In a first aspect, one embodiment of this disclosure provides a method for determining a game agent, including:

[0006] The initial game agent's valuation network model is trained based on perfect and imperfect features from the training samples, and the initial game agent's decision network model is trained based on imperfect features. Perfect features refer to the event information corresponding to all known events for the current account in the virtual game; imperfect features refer to the event information corresponding to events for which the current account is not fully aware of all events in the virtual game.

[0007] The model parameters of the initial game agent are updated based on the model parameters of the trained evaluation network model and decision network model to obtain the updated preprocessed game agent.

[0008] The preprocessed game agent is used as the initial game agent to continue training the valuation network model and the decision network model until the current preprocessed game agent reaches the preset training target. Then, the current preprocessed game agent is determined as the target game agent.

[0009] Secondly, one embodiment of this disclosure provides a game running method, including:

[0010] Obtain the input characteristics of the current account in the virtual game scene;

[0011] The decision network model in the target game agent is invoked to determine the action probability of each game action corresponding to the input features; wherein, the target game agent is determined according to any of the above game agent determination methods;

[0012] The game action with the highest probability among all actions is determined as the target game action to be performed by the current account.

[0013] Thirdly, one embodiment of this disclosure provides a game agent determination device, the device comprising:

[0014] The first training module is used to train the valuation network model in the initial game agent based on perfect and imperfect features in the training samples, and to train the decision network model in the initial game agent based on imperfect features. Among them, perfect features refer to the event information corresponding to all known events for the current account in the virtual game; imperfect features refer to the event information corresponding to the event information corresponding to the event information corresponding to the event information corresponding to the event information for the current account in the virtual game that is not fully known for all events.

[0015] The update module is used to update the model parameters of the initial game agent based on the model parameters of the trained evaluation network model and decision network model, so as to obtain the updated preprocessed game agent.

[0016] The second training module is used to continue training the valuation network model and decision network model with the preprocessed game agent as the initial game agent until the current preprocessed game agent reaches the preset training target, and then the current preprocessed game agent is determined as the target game agent.

[0017] Fourthly, one embodiment of this disclosure provides a game running device, including:

[0018] The acquisition module is used to acquire the input features of the current account in the virtual game scene;

[0019] The first determination module is used to call the decision network model in the target game agent to determine the action probability of each game action corresponding to the input features; wherein, the target game agent is determined according to any of the above game agent determination methods;

[0020] The second determination module is used to determine the game action with the highest probability among all actions as the target game action to be performed by the current account.

[0021] Fifthly, one embodiment of this disclosure provides a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the method described above.

[0022] In a sixth aspect, one embodiment of this disclosure provides an electronic device, including: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform the above-described method by executing the executable instructions.

[0023] The technical solution disclosed herein has the following beneficial effects:

[0024] The aforementioned method for determining the game agent trains the valuation network model in the initial game agent using both perfect features (i.e., imperfect information) and imperfect features (i.e., perfect information). By introducing perfect information, the accuracy of the action probabilities output by the valuation network model is improved. By using imperfect features to train the decision network model in the initial game agent, the problem of game cheating can be avoided. At the same time, the accuracy of the decision network model can be indirectly improved by the high accuracy of the action probabilities output by the valuation network model, resulting in a more accurate and reliable target game agent.

[0025] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0026] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0027] Figure 1 shows a flowchart of a game agent determination method in this exemplary embodiment;

[0028] Figure 2 shows a flowchart of a game agent determination method in this exemplary embodiment;

[0029] Figure 3 shows a schematic diagram of the encoding in a game agent determination method according to this exemplary embodiment;

[0030] Figure 4 shows a schematic diagram of the valuation network model in a game agent determination method according to this exemplary embodiment;

[0031] Figure 5 shows a schematic diagram of the decision network model in a game agent determination method according to this exemplary embodiment;

[0032] Figure 6 shows a flowchart of a game agent determination method in this exemplary embodiment;

[0033] Figure 7 illustrates the training process of an initial game agent in a game agent determination method according to this exemplary embodiment.

[0034] Figure 8 shows a flowchart of a game agent determination method in this exemplary embodiment;

[0035] Figure 9 shows a flowchart of a game agent determination method in this exemplary embodiment;

[0036] Figure 10 shows a flowchart of a game agent determination method in this exemplary embodiment;

[0037] Figure 11 shows a flowchart of a game agent determination method in this exemplary embodiment;

[0038] Figure 12 shows a schematic diagram of a game agent determination device in this exemplary embodiment;

[0039] Figure 13 shows a schematic diagram of a game running device in this exemplary embodiment;

[0040] Figure 14 shows a schematic diagram of the structure of an electronic device according to this exemplary embodiment. Detailed Implementation

[0041] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided to make this disclosure more comprehensive and complete, and to fully convey the concept of exemplary embodiments to those skilled in the art. The described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a full understanding of embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced with one or more specific details omitted, or other methods, components, apparatus, steps, etc., can be employed. In other instances, well-known technical solutions are not shown or described in detail to avoid obscuring various aspects of this disclosure.

[0042] Furthermore, the accompanying drawings are merely illustrative of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0043] The flowchart shown in the attached diagram is merely an illustrative example and does not necessarily include all steps. For example, some steps may be broken down, while others may be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0044] In related technologies, game robots have always been an important part of game design, and AI technology is being increasingly widely used in building game robots. However, a significant portion of games involve imperfect information, such as Dou Dizhu (a popular Chinese card game) and Mahjong. Despite their popularity, the imperfect information nature of these games makes it difficult to build high-level robots. Current AI agents are trained based on imperfect information, primarily using tree search or rule-based methods to mitigate the potentially huge search space. However, due to the strong strategic elements and high complexity of these games, simply using rules can only reduce the search space to a certain extent. Humanly considering all possible card situations and providing corresponding rules is extremely difficult, requiring enormous computing power to enumerate all potential card-playing possibilities, resulting in low overall training efficiency for games with imperfect information.

[0045] To address the aforementioned technical problems, this disclosure provides a method for determining game agents, thereby improving the training efficiency of games with imperfect information. The following is a brief introduction to the application environment of the game agent determination method provided by this disclosure:

[0046] The game agent determination method provided in this disclosure is applied to a terminal device, which can be a server or a user terminal, such as a laptop, mobile phone, tablet computer, or other wearable electronic devices. The following example illustrates how this game agent determination method can be applied to the aforementioned terminal device to construct a game agent that can determine game behavior based on the current game state. Referring to Figure 1, the game agent determination method provided in this disclosure includes the following steps 101-103:

[0047] Step 101: Train the valuation network model in the initial game agent based on the perfect and imperfect features in the training samples, and train the decision network model in the initial game agent based on the imperfect features.

[0048] First, it's important to clarify that "perfect information" is a game theory term referring to a situation where, in a game, each player executes actions sequentially, and each player or their corresponding game account knows all previously occurring events, including "initialization events" (such as each player's initial hand in a poker game), when making any decision. All current event information constitutes perfect information. Conversely, "imperfect information," also a game theory term, refers to a situation where, in a game, each player or their corresponding game account cannot know or can only know a portion of previously occurring events when making a decision (e.g., hiding one's current hand from an opponent in a poker game is an example of an imperfect game). The event information currently known to the player or their corresponding game account is imperfect information. Accordingly, in this embodiment, the "perfect feature" refers to the current account in the virtual game knowing all events, and the event information corresponding to all known events; the "imperfect feature" refers to the current account in the virtual game not knowing all events completely, and the event information corresponding to the events not being fully known.

[0049] The initial game agent can be either a perfect game agent or an imperfect game agent; this disclosure does not impose specific limitations. This embodiment uses an imperfect game agent as an example for illustration. It should be explained that this initial agent refers to a model that can make decisions for players or NPCs (non-player characters) in a virtual game environment according to pre-set game rules, that is, determine game strategies and execute corresponding game actions based on the current game state. The network model structure of this initial game agent includes at least an evaluation network model and a decision network model. The evaluation network model is used to determine the corresponding game actions and the action probabilities of each game action based on the input sample features. The decision network model is used to determine the target game action based on the action probabilities output by the evaluation network model. When training the evaluation network model, it is trained together with perfect and imperfect features, that is, perfect information is introduced through perfect features, thereby improving the accuracy of the action probabilities output by the evaluation network model. However, when training the decision network model, only imperfect features are used, that is, imperfect information is used for training, thereby avoiding the problem of game cheating. At the same time, the high accuracy of the action probabilities output by the evaluation network model can indirectly improve the accuracy of the decision network model.

[0050] Step 102: Update the model parameters of the initial game agent based on the model parameters of the trained valuation network model and decision network model to obtain the updated preprocessed game agent.

[0051] Model parameters are the parameters that the model needs to learn, such as weights, biases, reward coefficients, etc. This disclosure does not impose any limitations and can be configured according to the actual situation.

[0052] Step 103: Use the preprocessed game agent as the initial game agent to continue training the valuation network model and decision network model until the current preprocessed game agent reaches the preset training target, then determine the current preprocessed game agent as the target game agent.

[0053] In other words, during the training process, the model parameters of the initial game agent, or the current preprocessed game agent, are continuously updated based on each training result. Then, training continues based on the updated preprocessed game agent, and the number of training iterations and training results are recorded each time. The preset training objective in this embodiment can be a preset number of training iterations, or it can be the prediction accuracy of the target game agent or the convergence threshold of the convergence function, etc. There are no limitations here, and it can be selected or set according to the actual situation.

[0054] The training and updating process of the decision network model in this embodiment can be characterized by the following formula (1):

[0055] (1)

[0056] Formula (1) is for the decision network model The function expression for gradient calculation is obtained through the estimation network model. Introducing perfect features The valuation network model has more accurate inputs and more accurate outputs. Training results For decision network models Training and updating are performed to further guide the gradient update direction of the decision network model, resulting in a better decision network model. The output results are more accurate.

[0057] The game agent determination method provided in this disclosure trains the valuation network model in the initial game agent based on perfect features (i.e., imperfect information) and imperfect features (i.e., perfect information). By introducing perfect information, the accuracy of the action probabilities output by the valuation network model is improved. By using imperfect features to train the decision network model in the initial game agent, the problem of game cheating can be avoided. At the same time, the accuracy of the decision network model can be indirectly improved by the high accuracy of the action probabilities output by the valuation network model, so that the trained target game agent is more accurate and more reliable.

[0058] Please refer to Figure 2. In an optional embodiment of this disclosure, before training the estimated network model in the initial game agent based on perfect and imperfect features in the training samples in step 101, the agent determination method further includes the following steps 201-202:

[0059] Step 201: Determine the various game state characteristics of the current game during the game process.

[0060] Game state characteristics refer to various distinctive game states that directly affect the game outcome. For example, in the card game "Dou Dizhu", the game state characteristics are card types such as "single", "pair", "three of a kind", "bomb", "royal flush", "straight", "consecutive pairs" and "airplane".

[0061] Step 202: Encode various game state features into corresponding feature codes according to a preset format to obtain training samples containing various feature codes.

[0062] After obtaining different game state characteristics, to facilitate subsequent processing, the terminal device encodes the corresponding feature values ​​according to a preset format. For example, please refer to Figure 3. Taking the "Dou Dizhu" card game example above, all card type characteristics in the "Dou Dizhu" game are constructed into a 12×15 matrix. The first 4 rows are the number and code of each card (Card In Hand); rows 5-8 are the corresponding positions and codes of single cards (Solo), pairs (Pair), three-of-a-kind (Trio), and bombs (Bomb); row 9 is whether the current hand type contains a royal flush (Rocket); rows 10-12 are the corresponding positions and codes of the hand types chain of solo, chain of pairs, and chain of trio. Of course, the above example only shows some card type characteristics (i.e., game state characteristics) and corresponding codes (i.e., feature codes) of the "Dou Dizhu" card game in a certain game process, and does not represent all card type characteristics of the "Dou Dizhu" card game. After encoding all the game state features of the current game, the terminal device obtains the encoded feature codes. All these feature codes constitute the training samples for the initial game agent. It should be noted that these training samples include both perfect and imperfect features.

[0063] Before training the initial game agent, this embodiment first determines various game state features during the game process, and then encodes these game state features into corresponding feature codes according to a preset format to obtain training samples containing various feature codes. Training through the feature codes in this preset format is more efficient and can further improve the efficiency of the game agent determination method in this embodiment in determining the target game agent.

[0064] As in the above embodiment, the initial game agent includes an evaluation network model and a decision network model. Figure 4 is a schematic diagram of an exemplary evaluation network model, which includes an LSTM (Long Short-Term Memory) network model and an MLP (Multilayer Perceptron) network model. During training, the LSTM network model encodes various game state features (including historical game state features) separately according to perfect features and imperfect features to form training samples containing various feature encodings. The LSTM network model then concatenates the perfect features and imperfect features and feeds them into a 6-layer MLP network model. The MLP network model evaluates each game action, that is, determines the first action probability (Value) of each game action and outputs it.

[0065] Figure 5 is a schematic diagram of an exemplary decision network model, which includes an LSTM network model and an MLP network model. During training, the LSTM network model encodes various game state features, and then concatenates them with the encoded available actions features in the current game before feeding them into a 6-layer MLP network model. The MLP network model estimates each game action, that is, determines the legal action probability of each game action and outputs it.

[0066] Please refer to Figure 6. In an optional embodiment of this disclosure, taking the structure of the initial game agent described above as an example, the training process includes the following steps: Step 101, training the evaluation network model in the initial game agent based on perfect and imperfect features in the training samples, and training the decision network model in the initial game agent based on imperfect features. Steps 601-602 are as follows:

[0067] Step 601: Call the sampling module to determine the game actions corresponding to the features of each sample in the training samples.

[0068] Please refer to Figure 7. The training process includes two modules: a sampling module (Rollout Worker) and a learning module (Learner). The Rollout Worker collects information such as the features of each sample in the training samples and the environmental state in the current game environment. The Learning module is used for the actual training and interacts with the Sampling Worker during the training process. For example, it feeds back the model parameters after each training iteration to the Sampling Worker so that the Sampling Worker can continuously update the initial game agent and the current game environment.

[0069] The game actions corresponding to the sample features refer to several game actions predicted from all executable game actions in the training samples based on the current input training samples and the training result parameters and model parameters fed back by the learning module after the last training. It can also output the action probability of each game action.

[0070] Step 602: Call the learning module to train the valuation network model in the initial game agent based on perfect and imperfect features and the current game state score; train the decision network model in the initial game agent based on imperfect features, corresponding game actions, and the weight coefficients of the game actions.

[0071] The game state is used to characterize the current game progress and the influencing factors of that progress, such as the game progress itself and the card-playing information of each player in each round of the current game. The corresponding state score refers to the score given by the evaluator to the current game state during training. The evaluation system can be specifically set according to the actual situation, and this embodiment does not impose any specific limitations. The weight coefficient refers to the execution probability of each action in the decision network model, that is, a value after each training session. In other words, it can be understood as the proportion of the current game action among all game actions. For example, if three game actions are output simultaneously, the corresponding weight coefficients are 0.8, 0.1, and 0.1, respectively.

[0072] This embodiment divides the training of the initial game agent into a sampling module and a learning module. The sampling module determines the game actions corresponding to the features of each sample in the training samples, and then the learning module is used for training. Different modules perform corresponding services, which can improve the training efficiency of the agent.

[0073] Please refer to Figure 7. In one optional embodiment of this disclosure, the sampling module includes at least a game environment construction submodule (RLCard Env) and a model service submodule (Param Server). The game environment construction submodule constructs the game environment of the current game, such as the card play situation of each player and the card status of the current agent's account. After inputting training samples, the game environment construction submodule transmits them to the model service submodule. The model service submodule performs model inference, that is, determines the action probability of the training samples in the current game environment or different game environments, and feeds back the corresponding game actions and corresponding action probabilities to the game environment construction submodule.

[0074] Correspondingly, please refer to Figure 8. Step 601 above, which calls the sampling module to determine the game action corresponding to each sample feature in the training samples, includes the following steps 801-802:

[0075] Step 801: Call the game environment construction submodule in the sampling module to retrieve the features of each sample from the training samples and construct the game environment of the current game.

[0076] During the initial training, or at the start of the game, the game environment construction module determines and outputs the current game state based on the initial game state or the starting state, and feeds it back to the model service submodule. During the second training and subsequent training processes, the game environment construction submodule uses the features of each sample in the training samples as input to obtain the current game environment, and continuously updates the current game environment in combination with the game state fed back by the model service submodule.

[0077] Step 802: Call the model service submodule in the sampling module to determine the target game action corresponding to each sample feature based on the game environment, the predicted game action output by the decision network model in the last training, and the weight coefficient of each predicted game action.

[0078] It should be explained that the predicted game action is not a specific game action, but refers to all executable actions in the current game environment or game state. The weight coefficient is the reward coefficient of each executable action. For example, the three game actions in the above embodiment have corresponding weight coefficients of 0.8, 0.1 and 0.1, respectively. Among them, the action 1 corresponding to the highest weight coefficient of 0.8 is the target game action for this training.

[0079] The sampling module in this embodiment includes a game environment construction submodule and a model service submodule. During actual training, the game environment construction submodule is invoked to retrieve features from the training samples and construct the current game environment. The model service submodule is then invoked to determine the target game action corresponding to each sample feature based on the game environment, the predicted game actions output by the decision network model in the previous training, and the weight coefficients of each predicted game action. Different submodules perform different services, resulting in higher training efficiency and greater reliability, further improving the efficiency and reliability of game agent determination in this embodiment. This method also yields...

[0080] Please refer to Figure 9. In an optional embodiment of this disclosure, step 802, which involves calling the learning module to train the valuation network model in the initial game agent based on perfect and imperfect features and the current game state score, and training the decision network model in the initial game agent based on imperfect features, corresponding game actions, and the weight coefficients corresponding to the game actions, includes the following steps 901-903:

[0081] Step 901: Call the cache submodule in the learning module to store the game actions.

[0082] Step 902: Call the graphics processor in the learning module to read the current game status score from the cache submodule.

[0083] Step 903: Train the valuation network model in the initial game agent based on perfect and imperfect features and the current game state score; train the decision network model in the initial game agent based on imperfect features, corresponding game actions, and the weight coefficients of the game actions.

[0084] The learning module mainly consists of a caching submodule (such as a cache queue) and a graphics processing unit (GPU). The caching submodule stores the action data sent by the sampling module, while the GPU is responsible for training the model. Each time, the GPU retrieves data from the caching submodule, updates the model parameters of the current initial game agent or preprocessed game agent after training, and sends them to the model service submodule of the sampling module. Upon receiving the latest model parameters, the model service submodule updates its own model and uses it for subsequent inference.

[0085] The learning module in this embodiment includes a cache submodule and a graphics processor. The cache submodule stores game actions, and the graphics processor in the learning module reads the game state from the cache submodule and trains the valuation network model and the decision network model respectively. By having different modules execute different services and cooperate independently, the training efficiency is higher and more reliable, which can further improve the efficiency and reliability of determining the game agent in this embodiment.

[0086] In one optional embodiment of this disclosure, the initial game agent is a card game agent.

[0087] The corresponding imperfect and perfect features are both card pattern features and corresponding numerical features;

[0088] The card characteristics of imperfect features include: the current player's hand, all unplayed cards, the current player's played cards, the previous player's played cards, the next player's played cards, the landlord's 3 additional bottom cards, the last 15 moves of the entire game, the previous player's last move, and the next player's last move. The numerical characteristics of imperfect features mainly include: the minimum number of play-out steps of hand cards, the number of cards in the current player's hand, and the number of cards in the previous player's hand. Player's Hand, Number of Cards in Next Player's Hand, Number of Bombs, Flag of Game Control By Current Player, etc.

[0089] The perfect feature also includes two parts: card type features and numerical features. The card type features of the perfect feature include: the previous player's current hand cards, the next player's current hand cards, etc. The numerical features of the perfect feature include: the minimum play-out steps of the previous player's hand cards and the minimum play-out steps of the next player's hand cards, etc.

[0090] Board games are generally considered imperfect information games, meaning players cannot know or can only know some of the events that have occurred. In this embodiment, the initial game agent is a board game agent, meaning the board game agent is trained. This method avoids the problem of cheating caused by directly using perfect features for training. At the same time, it can indirectly improve the accuracy of the decision network model by using the high accuracy of the action probabilities output by the valuation network model, making the trained target board game agent more accurate and reliable.

[0091] Please refer to Figure 10. Another embodiment of this disclosure provides a game running method, including the following steps 1001-1003:

[0092] Step 1001: Obtain the input characteristics of the current account in the virtual game scene.

[0093] The current account can be an NPC account or a virtual character corresponding to any player in the current game. The virtual game scene refers to the game environment used by the aforementioned target game agent. This virtual game scene includes, but is not limited to, player characters, number of players, game rules, game display effects, and game action data for each player. The input feature refers to the game actions of each player and NPC in each game. These game actions include, but are not limited to, virtual skills, card types, and numerical characteristics of cards played. This input feature can specifically refer to all current game actions of a particular virtual character or NPC, or it can refer to all current game actions of all virtual characters and NPCs. This embodiment of the disclosure does not impose any limitations and can be specifically set according to the actual situation.

[0094] Step 1002: Call the decision network model in the target game agent to determine the action probability of each game action corresponding to the input features.

[0095] The target game agent is determined according to any of the above game agent determination methods. The beneficial effects of the target game agent determined by this method have been described in detail in the above embodiments and will not be repeated here.

[0096] Step 1003: Determine the game action with the highest probability among all actions as the target game action to be performed by the current account.

[0097] After each training session, each game action has a corresponding probability, and the terminal device directly identifies the game action with the highest probability as the target game action.

[0098] This embodiment indirectly introduces a decision network model trained with perfect features through an evaluation network model to determine the target game action of the current game. The resulting target game agent has a higher level of intelligence and reliability, which can greatly improve the intelligence level and reliability of the game running based on the target game agent.

[0099] In an optional embodiment of this disclosure, step 1002, which involves calling the decision network model in the target game agent to determine the action probability of each game action corresponding to the input features, includes the following step A:

[0100] Step A: Call the decision network model in the target game agent to determine the action probability corresponding to each preset executable game action based on historical game actions, input features, and the current game state in the virtual game scene.

[0101] The executable game actions can be set arbitrarily according to the actual game type or game rules, that is, all executable game actions under the current game rules and current game state. The current game state is all the executed game actions and the remaining executable game actions in the current game process. For example, for the "Dou Dizhu" game mentioned above, the current game state is the types and quantities of cards played by all players, the types and quantities of cards remaining, as well as the card-playing rules of the "Dou Dizhu" game and the types and quantities of cards played by the previous player, etc. The historical game actions can be the historical game actions of the current account, or the historical game actions performed by the previous player of the current account in the last time. This disclosed embodiment does not make specific limitations.

[0102] This embodiment calls the decision network model in the target game agent to determine the action probability corresponding to each preset executable game action based on historical game actions, input features, and the current game state in the virtual game scene. The obtained action probability of each game action is more accurate, and the final determined game result is more reliable.

[0103] Please refer to Figure 11. In an optional embodiment of this disclosure, step A above, which involves calling the decision network model in the target game agent to determine the action probabilities corresponding to each preset executable game action based on historical game actions, input features, and the current game state in the virtual game scene, includes the following steps 1101-1102:

[0104] Step 1101: Call the Long Short-Term Memory (LSTM) submodule in the decision network model to encode the historical game actions and input features to obtain preprocessed game actions.

[0105] Corresponding to step 202 in the training process, this embodiment encodes historical game actions before determining the action probability to facilitate subsequent processing by the terminal device and improve data processing efficiency. The specific encoding method can be the same as in step 202 above, and will not be elaborated here.

[0106] Step 1102: Call the perceptron network submodule in the decision network model to determine the action probabilities corresponding to each executable game action in the preprocessed game action and the current game state in the virtual game scene.

[0107] This embodiment first encodes historical game actions and input features, and then calls the perceptron network submodule in the decision network model to determine the action probabilities corresponding to each executable game action in the preprocessed game actions and the current game state in the virtual game scene. The action probability determination is more efficient and can further improve the efficiency of game operation in this embodiment.

[0108] Please refer to Figure 12. To implement the above-described game agent determination method, one embodiment of this disclosure provides a game agent determination device 1200. Figure 12 shows a schematic architecture diagram of the game agent determination device 1200, which includes: a first training module 1210, an update module 1220, and a second training module 1230, wherein:

[0109] The first training module 1210 is used to train the valuation network model in the initial game agent based on perfect features and imperfect features in the training samples, and to train the decision network model in the initial game agent based on imperfect features; wherein, perfect features refer to the event information corresponding to all known events for the current account in the virtual game; imperfect features refer to the event information corresponding to the event information corresponding to the event information corresponding to the event information corresponding to the event information for the current account in the virtual game that is not fully known for all events.

[0110] The update module 1220 is used to update the model parameters of the initial game agent based on the model parameters of the trained evaluation network model and decision network model, so as to obtain the updated preprocessed game agent.

[0111] The second training module 1230 is used to continue training the valuation network model and decision network model with the preprocessed game agent as the initial game agent until the current preprocessed game agent reaches the preset training target, and then the current preprocessed game agent is determined as the target game agent.

[0112] In an optional embodiment of this disclosure, the first training module 1210 is further configured to determine various game state features of the current game during the game process; encode each game state feature into a corresponding feature code according to a preset format, and obtain a training sample containing each feature code.

[0113] In an optional embodiment of this disclosure, the second training module 1230 is specifically used to: call the sampling module to determine the game actions corresponding to the features of each sample in the training samples; call the learning module to train the valuation network model in the initial game agent based on perfect features and imperfect features, as well as the current game state score; and train the decision network model in the initial game agent based on imperfect features and corresponding game actions, as well as the weight coefficients corresponding to the game actions.

[0114] In an optional embodiment of this disclosure, the sampling module includes at least a game environment construction submodule and a model service submodule. The second training module 1230 is specifically used to: call the game environment construction submodule in the sampling module to retrieve the features of each sample from the training samples and construct the game environment of the current game; and call the model service submodule in the sampling module to determine the target game action corresponding to each sample feature based on the game environment, the predicted game action output by the decision network model in the last training, and the weight coefficients of each predicted game action.

[0115] In an optional embodiment of this disclosure, the second training module 1230 is specifically configured to: call the cache submodule in the learning module to store game actions; call the graphics processor in the learning module to read the current game state score from the cache submodule; train the valuation network model in the initial game agent based on perfect features and imperfect features, as well as the current game state score; and train the decision network model in the initial game agent based on imperfect features, corresponding game actions, and weight coefficients corresponding to the game actions.

[0116] In one optional embodiment of this disclosure, the initial game agent is a card game agent.

[0117] Please refer to Figure 13. To implement the above-described game running method, one embodiment of this disclosure provides a game running device 1300. Figure 13 shows a schematic architecture diagram of the game running device 1300, which includes: an acquisition module 1310, a first determination module 1320, and a second determination module 1330, wherein:

[0118] The acquisition module 1310 is used to acquire the input features of the current account in the virtual game scene;

[0119] The first determining module 1320 is used to call the decision network model in the target game agent to determine the action probability of each game action corresponding to the input features; wherein, the target game agent is determined according to the game agent determination method of any of the above.

[0120] The second determining module 1330 is used to determine the game action with the highest probability among all actions as the target game action to be performed by the current account.

[0121] In an optional embodiment of this disclosure, the first determining module 1320 is specifically used to call the decision network model in the target game agent to determine the action probability corresponding to each preset executable game action based on historical game actions, input features, and the current game state in the virtual game scene.

[0122] In an optional embodiment of this disclosure, the first determining module 1320 is specifically used to: call the long short-term memory network submodule in the decision network model to encode historical game actions and input features to obtain preprocessed game actions; and call the perceptron network submodule in the decision network model to determine the preset action probabilities corresponding to each executable game action for the preprocessed game actions and the current game state in the virtual game scene.

[0123] Exemplary embodiments of this disclosure also provide a computer-readable storage medium that can be implemented as a program product including program code, which, when run on an electronic device, causes the electronic device to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure. In one embodiment, the program product can be implemented as a portable compact disc read-only memory (CD-ROM) and include program code, and can run on an electronic device, such as a personal computer. However, the program product of this disclosure is not limited thereto. In this document, the readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0124] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0125] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0126] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0127] Program code for performing the operations of this disclosure can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider). In embodiments of this disclosure, the program code stored in a computer-readable storage medium, when executed, can implement any step of the above-described game agent determination method and game execution method.

[0128] Referring to Figure 14, an exemplary embodiment of this disclosure also provides an electronic device 1400, which can be a backend server of an information platform. The electronic device 1400 will now be described with reference to Figure 14. It should be understood that the electronic device 1400 shown in Figure 14 is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this disclosure.

[0129] As shown in Figure 14, the electronic device 1400 is presented in the form of a general-purpose computing device. The components of the electronic device 1400 may include, but are not limited to: at least one processing unit 1410, at least one storage unit 1420, and a bus 1430 connecting different system components (including storage unit 1420 and processing unit 1410).

[0130] The storage unit stores program code, which can be executed by the processing unit 1410 to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of the present invention. For example, the processing unit 1410 can perform the method steps shown in FIG1, etc.

[0131] Storage unit 1420 may include volatile storage units, such as random access memory (RAM) 1421 and / or cache memory 1422, and may further include read-only memory (ROM) 1423.

[0132] Storage unit 1420 may also include a program / utility 1424 having a set (at least one) of program modules 1425, such program modules 1425 including but not limited to: an operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0133] Bus 1430 may include a data bus, an address bus, and a control bus.

[0134] Electronic device 1400 can also communicate with one or more external devices 2000 (e.g., keyboards, pointing devices, Bluetooth devices, etc.) via input / output (I / O) interface 1440. Electronic device 1400 can also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via network adapter 1450. As shown, network adapter 1450 communicates with other modules of electronic device 1400 via bus 1430. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 1400, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0135] In this embodiment of the disclosure, when the program code stored in the electronic device is executed, it can implement any step of the above-mentioned game agent determination method and game running method.

[0136] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to exemplary embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0137] Those skilled in the art will understand that various aspects of this disclosure can be implemented as systems, methods, or program products. Therefore, various aspects of this disclosure can be embodied in entirely hardware implementations, entirely software implementations (including firmware, microcode, etc.), or implementations combining hardware and software aspects, collectively referred to herein as “circuit,” “module,” or “system.” Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

[0138] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is defined only by the appended claims.

Claims

1. A method for determining a game intelligent agent, characterized in that, include: The initial game agent's valuation network model is trained based on perfect and imperfect features from the training samples, and the initial game agent's decision network model is trained based on the imperfect features. The perfect features refer to the event information corresponding to all known events for the current account in the virtual game; the imperfect features refer to the event information corresponding to events for which the current account is not fully aware. The initial game agent's model parameters are updated based on the trained valuation network model and decision network model, resulting in an updated preprocessed game agent. The preprocessed game agent is then used as the initial game agent to continue training the valuation network model and decision network model until the current preprocessed game agent reaches a preset training objective, at which point the current preprocessed game agent is identified as the target game agent. The valuation network model outputs corresponding game actions and a first action probability for each game action based on the input perfect and imperfect features. The decision network model outputs a second action probability for each game action based on the input imperfect features and the encoding of the executable game actions. The first action probabilities are used to train the decision network model.

2. The method for determining a game intelligent agent according to claim 1, characterized in that, Before training the valuation network model in the initial game agent based on the perfect and imperfect features in the training samples, the method further includes: determining various game state features of the current game during the game process; encoding each game state feature into a corresponding feature code according to a preset format to obtain the training samples containing each feature code.

3. The method for determining a game intelligent agent according to claim 1, characterized in that, The process of training the valuation network model in the initial game agent based on perfect and imperfect features in the training samples, and training the decision network model in the initial game agent based on the imperfect features, includes: calling the sampling module to determine the game actions corresponding to each sample feature in the training samples; calling the learning module to train the valuation network model in the initial game agent based on the perfect and imperfect features and the current game state score, and to train the decision network model in the initial game agent based on the imperfect features, the corresponding game actions, and the weight coefficients corresponding to the game actions.

4. The method for determining a game intelligent agent according to claim 3, characterized in that, The sampling module includes at least a game environment construction submodule and a model service submodule. Correspondingly, the step of calling the sampling module to determine the game action corresponding to each sample feature in the training samples includes: calling the game environment construction submodule in the sampling module to retrieve each sample feature from the training samples and constructing the game environment of the current game; calling the model service submodule in the sampling module to determine the target game action corresponding to each sample feature based on the game environment, the predicted game action output by the decision network model in the previous training, and the weight coefficients of each predicted game action.

5. The method for determining a game intelligent agent according to claim 3, characterized in that, The learning module is invoked to train the valuation network model in the initial game agent based on the perfect features, the imperfect features, and the current game state score. Training the decision network model in the initial game agent based on the imperfect features, the corresponding game actions, and the weight coefficients corresponding to the game actions includes: calling the cache submodule in the learning module to store the game actions; calling the graphics processor in the learning module to read the current game state score from the cache submodule; training the valuation network model in the initial game agent based on the perfect features, the imperfect features, and the current game state score; and training the decision network model in the initial game agent based on the imperfect features, the corresponding game actions, and the weight coefficients corresponding to the game actions.

6. The method for determining a game intelligent agent according to any one of claims 1-5, characterized in that, The initial game agent is a card game agent.

7. A method for running a game, characterized in that, include: Obtain the input characteristics of the current account in the virtual game scene; The decision network model in the target game agent is invoked to determine the action probability of each game action corresponding to the input feature; wherein, the target game agent is determined by the game agent determination method according to any one of claims 1-6; the game action with the maximum value among the action probabilities is determined as the target game action to be performed by the current account.

8. The game operation method according to claim 7, characterized in that, The step of calling the decision network model in the target game agent to determine the action probability of each game action corresponding to the input features includes: calling the decision network model in the target game agent to determine the action probability corresponding to each preset executable game action based on historical game actions, the input features, and the current game state in the virtual game scene.

9. The game operation method according to claim 8, characterized in that, The step of calling the decision network model in the target game agent to determine the preset action probabilities corresponding to each executable game action based on historical game actions, the input features, and the current game state in the virtual game scene includes: calling the long short-term memory network submodule in the decision network model to encode the historical game actions and the input features to obtain preprocessed game actions; and calling the perceptron network submodule in the decision network model to determine the preset action probabilities corresponding to each executable game action based on the preprocessed game actions and the current game state in the virtual game scene.

10. A device for determining a game intelligent agent, characterized in that, The device includes: a first training module, used to train an evaluation network model in an initial game agent based on perfect and imperfect features in training samples, and to train a decision network model in the initial game agent based on the imperfect features; wherein, the perfect features refer to the event information corresponding to all known events for the current account in the virtual game; the imperfect features refer to the event information corresponding to events for which the current account in the virtual game is not fully aware of all events; and an update module, used to update the model parameters of the initial game agent based on the model parameters of the trained evaluation network model and the decision network model. The system then proceeds to the updated preprocessed game agent; a second training module is used to train the valuation network model and the decision network model using the preprocessed game agent as the initial game agent, until the current preprocessed game agent reaches a preset training target, at which point the current preprocessed game agent is determined as the target game agent; wherein, the valuation network model is used to output corresponding game actions and a first action probability for each game action based on the input perfect and imperfect features, and the decision network model is used to output a second action probability for each game action based on the input imperfect features and the encoding of executable game actions, and the first action probability is used to train the decision network model.

11. A game operating device, characterized in that, include: The acquisition module is used to acquire the input features of the current account in the virtual game scene; The first determining module is used to call the decision network model in the target game agent to determine the action probability of each game action corresponding to the input features; wherein, the target game agent is determined by the game agent determining method according to any one of claims 1-6; The second determining module is used to determine the game action with the maximum value among the probabilities of each action as the target game action to be performed by the current account.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method according to any one of claims 1 to 9.

13. An electronic device, characterized in that, include: processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform the method of any one of claims 1 to 9 by executing the executable instructions.

Images