An AI object policy prediction model training method, device, and electronic equipment

Abstract

The application relates to the technical field of data processing, in particular to an AI object strategy prediction model training method, computer equipment and a storage medium. The technical scheme provided by the application can train a prediction model outputting a strategy based on real-time state information of a simulation environment and actual behavior feedback of an AI object. When the model is inferred, an explicit strategy can be given, the model is more interpretable, and users can better utilize the model. Meanwhile, the technical scheme provided by the embodiment of the application can make corresponding decisions according to real-time changes of the environment, and the flexibility of the AI can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and particularly relates to an AI object strategy prediction model training method and device and electronic equipment. BACKGROUND

[0002] With the rapid development of artificial intelligence (AI) technology, artificial intelligence technology is widely applied in various fields. For example, in the game entertainment field, through artificial intelligence technology, a game between a virtual user and a real person can be realized. Although reinforcement learning performs better than humans in some games, it lacks certain explainability. Existing reinforcement models all output a specific action according to environmental input, such as moving one step to the left. People cannot understand the strategy and intention of the AI object, so they cannot judge the rationality of these micro-operation actions, nor can they make certain reactions according to these actions, which brings great difficulties to game developers to better utilize these AI objects. SUMMARY

[0003] To solve the above technical problems, the present application provides an AI object strategy prediction model training method, computer equipment and storage medium, which solves the above problems by outputting specific strategies instead of micro-operations.

[0004] To achieve the above purpose, the technical solutions adopted by the embodiments of the present application are as follows:

[0005] In a first aspect, the embodiments of the present application provide an AI object strategy prediction model training method, comprising:

[0006] Obtaining real-time state information of an AI object in a simulation environment under a current environment;

[0007] Extracting first feature information and second feature information from the real-time state information;

[0008] Inputting the first feature information and the second feature information into a current strategy prediction model for model inference;

[0009] Obtaining a target strategy according to the output of the model inference;

[0010] Triggering a state machine to output a specific action corresponding to the AI object according to the target strategy;

[0011] Controlling the AI object to perform the specific action and interact with the current environment;

[0012] Saving the data generated in the above process as sample data for training;

[0013] The strategy prediction model is trained and optimized through multiple sample data.

[0014] In an exemplary embodiment of the present application, the strategy prediction model is a convolutional neural network model, which includes a first convolutional layer, a second convolutional layer, a first fully connected layer, a second fully connected layer, a third fully connected layer, and a fourth fully connected layer.

[0015] In an exemplary embodiment of the present application, the step of extracting first feature information and second feature information from the real-time state information specifically includes:

[0016] Extracting image feature information and vector feature information in the real-time state information.

[0017] In an exemplary embodiment of the present application, the step of inputting the first feature information and the second feature information into the current strategy prediction model for model inference specifically includes:

[0018] The image feature information is operated through the first convolutional layer and the second convolutional layer.

[0019] The vector feature information is processed through the first fully connected layer and the second fully connected layer.

[0020] The results of operation and processing are spliced and then processed through the third fully connected layer and the fourth fully connected layer to obtain a strategy probability distribution and a Q value for evaluating the strategy probability distribution as the output of model inference.

[0021] In an exemplary embodiment of the present application, the step of obtaining a target strategy according to the output of model inference specifically includes:

[0022] The target strategy is obtained by sampling the strategy probability distribution.

[0023] In an exemplary embodiment of the present application, the step of triggering a state machine to output a specific action corresponding to the AI object according to the target strategy specifically includes:

[0024] The state machine determines a corresponding execution action according to the target strategy.

[0025] The feasibility of the execution action is judged.

[0026] If feasible, the execution action is output as the specific action of the AI object, and if not feasible, an error message is returned.

[0027] In an exemplary embodiment of the present application, the sample data includes image feature information, vector feature information, generalized advantage estimation (GAE), and reward information.

[0028] In an example embodiment of the present application, the step of training and optimizing the policy prediction model by multiple sample data specifically comprises:

[0029] Obtaining multiple saved sample data;

[0030] Assembling the obtained sample data through a Batch module;

[0031] Inputting the assembled sample data into the policy prediction model to calculate a policy gradient;

[0032] According to the sample data and the policy gradient, the policy prediction model is trained through reinforcement learning.

[0033] In a second aspect, the embodiments of the present application also provide a policy prediction model training device for AI objects, comprising:

[0034] A sample production module for producing sample data;

[0035] The sample production module comprises:

[0036] A state acquisition unit for acquiring real-time state information of an AI object in a simulation environment under a current environment;

[0037] A feature extraction unit for extracting first feature information and second feature information from the real-time state information;

[0038] A model inference unit for inputting the first feature information and the second feature information into a current policy prediction model for model inference;

[0039] A policy output unit for obtaining a target policy according to the output of model inference;

[0040] An action output unit for triggering a state machine to output specific actions corresponding to the AI object according to the target policy;

[0041] An interaction control unit for controlling the AI object to perform the specific actions and interact with the current environment;

[0042] A sample saving unit for saving the data generated in the above process as sample data for training;

[0043] A model training module for training and optimizing the policy prediction model by multiple sample data.

[0044] In an example embodiment of the present application, the strategy prediction model is a convolutional neural network model, which includes a first convolutional layer, a second convolutional layer, a first full connection layer, a second full connection layer, a third full connection layer, and a fourth full connection layer.

[0045] In an example embodiment of the present application, the feature extraction unit is specifically configured to:

[0046] extract image feature information and vector feature information in the real-time state information.

[0047] In an example embodiment of the present application, the model inference unit is specifically configured to:

[0048] operate the image feature information through the first convolutional layer and the second convolutional layer;

[0049] process the vector feature information through the first full connection layer and the second full connection layer;

[0050] obtain a strategy probability distribution and a Q value for evaluating the strategy probability distribution through processing of the first full connection layer and the second full connection layer after splicing results of the operation and the processing, as an output of model inference.

[0051] In an example embodiment of the present application, the strategy output unit is specifically configured to:

[0052] sample the strategy probability distribution to obtain a target strategy.

[0053] In an example embodiment of the present application, the action output unit is specifically configured to:

[0054] determine a corresponding execution action according to the target strategy;

[0055] judge feasibility of the execution action;

[0056] if feasible, output the execution action as a specific action of the AI object, and if not feasible, return error information.

[0057] In an example embodiment of the present application, the sample data includes image feature information, vector feature information, generalized advantage estimation (GAE), and reward information.

[0058] In an example embodiment of the present application, the model training module is specifically configured to:

[0059] obtain a plurality of saved sample data;

[0060] assemble the obtained sample data through a Batch module.

[0061] inputting the assembled sample data into the strategy prediction model, and calculating a strategy gradient;

[0062] performing reinforcement learning training on the strategy prediction model according to the sample data and the strategy gradient.

[0063] In a third aspect, an electronic device is provided, and the electronic device comprises:

[0064] one or more processors; a memory; and one or more application programs stored in the memory and configured to be executed by the one or more processors, the one or more programs being configured to perform the method of the first aspect.

[0065] In a fourth aspect, a computer-readable storage medium is provided, and the computer-readable storage medium stores program code, which can be invoked by a processor to perform the method of the first aspect.

[0066] The technical solution provided in the present application can train a prediction model outputting a strategy based on the real-time state information of the simulation environment and the actual behavior feedback of the AI object. When the model is inferred, an explicit strategy can be given, which is more interpretable and enables users to better utilize the model. Meanwhile, the technical solution provided in the present application can make corresponding decisions according to real-time changes in the environment, which can improve the flexibility of AI. BRIEF DESCRIPTION OF DRAWINGS

[0067] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative effort.

[0068] The methods, systems, and / or programs in the drawings will be further described according to exemplary embodiments. These exemplary embodiments will be described in detail with reference to the drawings. These exemplary embodiments are non-limiting exemplary embodiments, in which the example numbers represent similar mechanisms in each view of the drawings.

[0069] Figure 1 is a method flowchart of the strategy prediction model training method provided in an embodiment of the present application.

[0070] Figure 2 is a method flowchart of the strategy prediction model training method provided in another embodiment of the present application.

[0071] Figure 3 is a training schematic diagram of the strategy prediction model training method provided by the embodiment of the application.

[0072] Figure 4 is a functional module block diagram of the strategy prediction model training device provided by the embodiment of the application.

[0073] Figure 5 is a structural block diagram of an electronic device for executing the strategy prediction model training method according to the embodiment of the application.

[0074] Figure 6 is a structural block diagram of a computer readable storage medium for storing or carrying program code for implementing the strategy prediction model training method according to the embodiment of the application.

[0075] Icon:

[0076] Sample production module 100; model training module 200; state acquisition unit 110; feature extraction unit 120; model inference unit 130; strategy output unit 140; action output unit 150; interaction control unit 160; sample storage unit 170; electronic device 300; processor 310; memory 320; computer readable storage medium 400; program code 410. DETAILED DESCRIPTION

[0077] Example implementations are now described with reference to the drawings; it being understood that the example implementations can be practiced with other implementations as well. The example implementations relate to systems, methods, and computer program products for training a strategy prediction model. The example implementations are described with reference to the following acts, which are not all inclusive.

[0078] In addition, the accompanying drawings are included to provide a further understanding of embodiments of the application, and are incorporated in and constitute a part of this specification. The drawings illustrate embodiments of the application and, together with the description, serve to explain the principles of the application. In the drawings:

[0079] Before the embodiments of the present application are further described, the terms and phrases involved in the embodiments of the present application are explained, and the terms and phrases involved in the embodiments of the present application are applicable to the following explanations.

[0080] (1) In response to, for indicating the condition or state on which the operation is dependent, when the dependent condition or state is met, one or more operations performed can be real-time or have a set delay; in the absence of special instructions, there is no restriction on the execution order of multiple operations performed.

[0081] (2) Based on, for indicating the condition or state on which the operation is dependent, when the dependent condition or state is met, one or more operations performed can be real-time or have a set delay; in the absence of special instructions, there is no restriction on the execution order of multiple operations performed.

[0082] (3) Model training, multi-classification learning is performed on the image data set. The model can be constructed using deep learning frameworks such as Tensor Flow, torch, etc., and multi-classification models can be composed of multiple layers of neural network layers such as CNN. The input of the model is a three-channel or original channel matrix formed by reading the image using tools such as openCV, and the output of the model is a multi-classification probability, which is finally output as a web page category through algorithms such as softmax. During training, the model approaches the correct trend through objective functions such as cross-entropy.

[0083] (4) Neural Network (Neural Network, NN): Artificial Neural Network (Artificial Neural Network, ANN), simply referred to as neural network or neural network, in the field of machine learning and cognitive science, is a mathematical model or computational model that simulates the structure and function of biological neural networks (animal central nervous system, especially brain), used to estimate or approximate functions.

[0084] (5) Game environment: a game environment displayed (or provided) by the application when running on the terminal. The game environment can be a simulation environment of the real world, a semi-simulated and semi-fictional three-dimensional environment, or a purely fictional three-dimensional environment. The game environment can be any one of a two-dimensional game environment, a 2.5-dimensional game environment, and a three-dimensional game environment. The following embodiments are exemplarily described with the game environment being a three-dimensional game environment, but are not limited thereto. Alternatively, the game environment is also used for a game environment battle between at least two virtual objects. Alternatively, the game environment is also used for a battle between at least two virtual objects through rolling of a virtual ball. Alternatively, the game environment can not be limited to a gun battle game, a parkour game, a racing game, a Multiplayer Online Battle Arena (MOBA), a Racing Game (RCG), and a sport game (SPG). The trained game model provided in the present application can be deployed in a game server corresponding to each of the above-mentioned game scenarios, used to generate real-time game strategies, execute corresponding action information, simulate virtual user operations, and complete different types of games in the game environment together with actual game participants.

[0085] (6) Action information: in a game user participating in a speed competition using a first person or a third person, taking a racing game such as a car racing game or an air racing game as an example, the action information refers to a directional key or the like for controlling a moving object as an operation instruction. For a role-playing game, the action information refers to a virtual weapon for attacking in a game environment, or a virtual bow and arrow or a virtual slingshot for shooting arrows, a virtual object in a game environment can pick up a virtual gun and attack with the picked-up virtual gun.

[0086] In the embodiments of the present application, the AI object can be a user virtual object controlled by an operation on a client, an artificial intelligence (AI) set in a game environment battle through training, or a non-user virtual object (NPC) set in a game environment interaction. The virtual object can be a virtual character participating in a competition in a game environment. The number of virtual objects participating in interaction in the game environment can be pre-set, or dynamically determined according to the number of clients joining the interaction.

[0087] For example, in a ball-elimination sports game, a user can control a virtual object to perform corresponding action information at different times in the game environment, such as moving in a specific direction, launching an attack on a target sports object, and avoiding an attack launched by the target sports object, and the virtual object can also move and contact a building configured in the simulation environment. The embodiments of the present application are not limited to the above scenario.

[0088] The method provided in the present application can be applied to virtual reality applications, three-dimensional map programs, multiplayer online battle arena games (MOBA), and the like, which will be described below through specific embodiments.

[0089] Figure 1 The method flowchart of the strategy prediction model training method for providing an AI object in an embodiment of the present application is shown in Figure 1 The strategy prediction model training method provided in an embodiment of the present application includes the following steps:

[0090] In step S101, real-time state information of an AI object in a simulation environment under a current environment is obtained.

[0091] The real-time state information obtained from the simulation environment in which the AI object is located is related to the AI object under the current environment and can affect the subsequent actions of the AI object.

[0092] In step S102, first feature information and second feature information are extracted from the real-time state information.

[0093] The first feature information and the second feature information correspond to different angles of the simulation environment in which the AI object is located, for example, the first feature information corresponds to environment background related state information, and the second feature information corresponds to AI object state related state information.

[0094] Further, as a preferred mode of the present embodiment, in order to more comprehensively describe the real-time state information under the current environment, more feature information can be extracted from more angles to provide more input information for the subsequent strategy prediction model.

[0095] In step S103, the first feature information and the second feature information are input into the current strategy prediction model for model inference.

[0096] The extracted first feature information and second feature information are taken as input, and the strategy prediction model is used to infer the strategy to be executed by the AI object. The strategy prediction model is pre-established. In the initial stage of the method, the model is an initial model. With the repeated execution of the strategy prediction model training method, the initial model is continuously trained and optimized, and the parameters of the initial model are continuously adjusted.

[0097] In step S104, the target strategy is obtained according to the output of the model inference.

[0098] One output of the strategy prediction model is the target strategy, that is, the target strategy to be executed by the AI object based on the current input first feature information and second feature information. It should be noted that in the embodiments of the present scheme, the strategy prediction model outputs the target strategy for the AI object rather than specific actions. Giving a clear strategy rather than specific actions makes the subsequent operation of the AI object more interpretable, and makes it easier for people to use the model.

[0099] In step S105, the target strategy is used to trigger the state machine to output specific actions corresponding to the AI object.

[0100] The purpose of introducing the state machine is to complete the mapping from the strategy to the action, and to execute specific actions according to the target strategy output by the strategy prediction model. The state machine is triggered according to the input target strategy, and by combining the simulation environment in which the AI object is located, the corresponding mapping rule is used to map to specific actions.

[0101] For example, if the input target strategy is to move to a place with a certain feature, then the specific action mapped by the state machine is to search for the coordinates of the place with the certain feature in the map, generate a moving path based on the coordinates of the AI object using available path algorithms, and control the AI object to move according to the moving path.

[0102] In step S106, the AI object is controlled to execute the specific action and interact with the current environment.

[0103] After the specific action of the AI object is determined by the state machine, the AI object is controlled to execute it, so as to realize the interaction between the AI object and the current environment. After the environment makes a feedback, the real-time state information changes, and then step S101 is repeated to form a closed loop.

[0104] Step S107, the data generated in the above process can be used for training as sample data. For one cycle of the above process, that is, after performing steps S101-S106 once, the data generated in the process is saved as sample data. In order to improve efficiency, the saved data is the data that can be used for training. The data generated in each cycle is a sample data. With the continuous running of the above method, sample data is continuously produced.

[0105] As a preferred embodiment of the present embodiment, the produced sample data can be saved into a Redis server. When subsequent training of the model is needed, the sample data can be pulled from the Redis server for training.

[0106] Step S108, training and optimizing the strategy prediction model through multiple sample data.

[0107] When a certain amount of sample data is accumulated, the training and optimization of the strategy prediction model can be performed through the sample data.

[0108] As a preferred embodiment of the present embodiment, when training and optimizing the strategy prediction model, the optimized strategy prediction model is saved as a historical model in a historical model library after each optimization or according to a certain period. When the above step S103 is repeatedly executed, different historical models can be selected and pulled from the historical model library for model inference, thereby generating more sample data to support further training and optimization.

[0109] As a preferred embodiment of the present embodiment, in order to improve the production efficiency of sample data, multiple AI objects can be set in the simulation environment at the same time, and the multiple AI objects perform the above steps respectively to run the sample data production work of self-play. Further, for different AI objects, when extracting the first feature information and the second feature information, the feature information of different AI objects can be extracted independently, or the feature information extracted by multiple AI objects in the same scene can be shared to speed up the training speed.

[0110] As a preferred embodiment of the present embodiment, when performing step S108 for training and optimization, multiple workers can perform the training task to speed up the training speed and convergence speed.

[0111] The strategy prediction model training method provided in the embodiment can train a prediction model outputting a strategy based on real-time state information of a simulation environment and actual behavior feedback of an AI object. When the model is inferred, an explicit strategy can be given, which is more interpretable and enables users to better utilize the model. Meanwhile, the technical solution provided in the embodiment can make corresponding decisions according to real-time changes in the environment, which can improve the flexibility of the AI.

[0112] As shown in the strategy prediction model training method of another embodiment of the application, Figure 3 the method comprises the following steps:

[0113] In step S201, real-time state information of an AI object in a simulation environment under a current environment is acquired.

[0114] In step S202, image feature information and vector feature information in the real-time state information are extracted.

[0115] In the embodiment, the first feature information and the second feature information are image feature information and vector feature information respectively. The image feature information corresponds to a map scene in the simulation environment, and the vector feature information corresponds to a related state of the AI object.

[0116] Taking a specific game scene as an example, the AI object is an NPC in the scene, and the real-time state information is related information in the game scene, including the position, state, and building of the NPC. Extracting the image feature information and the vector feature information from the real-time state information can specifically be performed as follows: the current game map is divided into 200*200 units, and a 200*200 two-dimensional array is obtained, with a default value of 0. Then, the building and player information are encoded, for example, if there is a game hall at the (10, 10) coordinate, the 【10, 10】 of the two-dimensional array is assigned a value of 1, and if it is a player, it can be assigned a value of 2, and different numbers represent different units. Then, the array of 16 units above, below, left, and right of the current player is taken as the image feature (16+1+16=37), and the vector feature is extracted from the current game state, for example, the player's empty slot numbers are 0, 1, 2, and 3, which are encoded as (0, 0, 0), (1, 0, 0), (0, 1, 0), and (0, 0, 1), and similar information such as the player's position and building state. The image feature has a size of 37*37 and a total of 25 channels, representing the position of the character, road information, and building information. The vector feature has a total of 3300 dimensions, including 500 dimensions of building position and attribute information, and 2800 dimensions of NPC skill information, position, and current attribute state.

[0117] In step S203, the image feature information is operated through the first convolutional layer and the second convolutional layer.

[0118] Step S204: The vector feature information is processed through the first fully connected layer and the second fully connected layer.

[0119] Step S205: The results of the calculation and processing are concatenated and then processed by the third fully connected layer and the fourth fully connected layer to obtain the policy probability distribution and the Q value used to evaluate the policy probability distribution, which are used as the output of the model inference.

[0120] In this embodiment, the policy prediction model is a convolutional neural network model, which includes a first convolutional layer, a second convolutional layer, a first fully connected layer, a second fully connected layer, a third fully connected layer, and a fourth fully connected layer.

[0121] like Figure 4 As shown, in the above process, the image feature information undergoes two convolution operations, the vector feature information goes through two fully connected layers, and then the two outputs are concatenated together and input into two fully connected layers to finally obtain two output policy probability distributions and Q values.

[0122] Continuing with the specific game scenario mentioned above, the concatenation of the two outputs can be represented as follows: The 7*37*3 image feature information is processed through two convolution operations in a CNN convolutional network to obtain a 128-dimensional vector. The 500-dimensional vector feature information is then processed through two fully connected neural network layers to obtain another 128-dimensional vector. These two 128-dimensional vectors are concatenated to obtain a 256-dimensional vector. This 256-dimensional vector is then processed through two fully connected neural network layers to obtain two branch outputs: one branch represents the Q-value, and the other represents the policy probability distribution. The 256-dimensional feature vector is then processed through two fully connected layers to obtain a 10-dimensional vector. These 10 vectors are used as the probabilities of 10 policies; this vector is the policy probability distribution.

[0123] The Q-value branch is used to evaluate the current state and provide the direction for model optimization; a higher Q-value means a higher reward. In a preferred embodiment, the distance between the Q-value and the true value can be calculated using the L2 loss algorithm; the closer the two values ​​are, the more accurate the Q-value.

[0124] Step S206: Sample the probability distribution of the strategy to obtain the target strategy.

[0125] In this embodiment, the branch outputs a strategy probability distribution, and a specific strategy needs to be sampled according to the probability to obtain the target strategy. The specific sampling method can be selected according to the actual use scene. As a preferred embodiment, the top1 sampling method can be used for sampling, so that the strategy with the highest probability is executed. After sampling, the target strategy is obtained. In step S207, the state machine determines the corresponding execution action according to the target strategy.

[0126] In step S208, the feasibility of the execution action is determined.

[0127] In step S209, if the execution action is feasible, the execution action is output as the specific action of the AI object, and if the execution action is not feasible, error information is returned.

[0128] The target strategy is input to the state machine, which is specifically executed by the state machine, and finally a specific action is output to the environment to complete an interaction. After the state machine determines the corresponding execution action according to the target strategy, the execution action is not directly output, but needs to be further judged whether the execution action can be executed in the current environment. For actions that cannot be executed, error information needs to be fed back.

[0129] The judgment method of the feasibility of the execution action needs to be determined in combination with the specific simulation environment. For example, in some game scenes, some actions of game characters need to meet certain preconditions to be executed, and if the preconditions are not met, the actions cannot be executed. Therefore, if the input strategy is a destination, it is first determined whether the current location is consistent with the destination. If consistent, the stop action is returned, otherwise the Astar path planning algorithm is used to plan a path, then the relative direction of the next step to the current position is calculated, and then the action of moving in the direction is output, for example, walking one step down.

[0130] In step S210, the AI object is controlled to execute the specific action and interact with the current environment.

[0131] In step S211, the data generated in the above process which can be used for training is saved as sample data.

[0132] It should be noted that the data content to be saved by the sample data is related to the training method used subsequently, and the data that can be used for training needs to be saved.

[0133] In this embodiment, the training method used is a reinforcement learning method, such as PPO or DQN. The corresponding sample data includes image feature information, vector feature information, generalized advantage estimation GAE, reward information and other information required for reinforcement learning training. When the training method used is adjusted, the information included in the sample data also needs to be adjusted according to the requirements of the training method used.

[0134] In the embodiment, the steps S201-S211 correspond to the process of sample data generation, and the following steps S212-S215 correspond to the process of model training.

[0135] In step S212, the saved multiple sample data are obtained.

[0136] In step S213, the obtained sample data are assembled by a Batch module.

[0137] In the embodiment, in order to reasonably utilize training resources, the Batch module assembles data to accelerate the training speed.

[0138] In step S214, the assembled sample data are input into the policy prediction model to calculate a policy gradient.

[0139] In step S215, the policy prediction model is trained by reinforcement learning according to the sample data and the policy gradient.

[0140] In the embodiment, as a preferred implementation, the specific action output by the state machine controls the AI object to interact with the current environment to obtain feedback information formed by the current environment, and the policy prediction model is trained according to the feedback information to update the model parameters of the policy prediction model. The specific training process is as follows:

[0141] After the AI object interacts with the current environment, real-time state information is obtained again. After the AI object performs the behavior action, the environment changes, and the changed environment state is understood by obtaining the real-time state information. Based on the newly obtained real-time state information, actual behavior value information Q(s, a) of the AI object after performing the specific action based on the target policy is obtained, that is, the Q value.

[0142] Specifically, the actual behavior mechanism information is used to indicate the state situation value of the environment after the game AI object performs the action strategy, which can be understood as that the smaller the distance between the state situation value and the real distance value, the more the change of the environment conforms to the expected situation, and the larger the actual behavior value information, the larger the reward.

[0143] In actual application, the server can calculate the actual behavior value information of the game AI object after performing the action strategy in the following manner, which is represented by the following function:

[0144] ;

[0145] Wherein, the Q is the actual behavior value information of the game AI object after performing the action strategy, the R t+1is the actual behavior value after the iteration number is increased by one, the λ is a preset coefficient, the S t+1 is the real-time state information after the iteration number is increased by one, and the a is the target strategy.

[0146] In the process of calculating the actual behavior value of the game AI object executing the action strategy, the actual behavior value can be adjusted by calculating the loss function thereof, which is represented by the following formula:

[0147]

[0148] The L is the loss in the calculation process, the E is the expectation, the r is a preset coefficient, the γ is a decay coefficient, the s' is the real-time state information at the next moment, the a' is the target strategy at the next moment, the w is the current network system, the s is the current real-time state information, and the a is the current target strategy.

[0149] It is determined whether the loss value is less than or equal to a preset loss threshold value. If the loss value is less than or equal to the preset loss threshold value, it is determined that the first actual behavior value and the second behavior value in the two processing processes converge. If the loss value is greater than the preset loss threshold value, it is determined that the actual behavior value does not converge. It should be noted that the above loss threshold value can be set based on the actual situation, and the present application does not make a specific limitation thereto.

[0150] It is determined whether the policy probability distribution converges.

[0151] The policy probability distribution in the two processes is converged by a preset loss function. For example, the loss value between the first policy probability distribution in the first processing process and the second policy probability distribution in the second processing process is calculated. It is determined whether the loss value is less than or equal to a preset loss threshold value. If the loss value is less than or equal to the preset loss threshold value, it is determined that the second policy probability distribution converges. If the loss value is greater than the preset loss threshold value, it is determined that the second policy probability distribution does not converge.

[0152] When the actual behavior value converges and the second policy probability distribution converges, the parameters of the policy prediction model can be adjusted to appropriate values to obtain a trained policy prediction model.

[0153] In order to improve the accuracy of training, the training of the model can use a proximal policy optimization algorithm (PPO), and L2 can be used as a value loss function to optimize the actual behavior value, so as to generate a more accurate target strategy.

[0154] ​In some embodiments, adjusting each parameter of the strategy prediction model to a suitable value to obtain the trained strategy prediction model can include: adjusting each parameter of the strategy prediction model to a suitable value to obtain a candidate strategy prediction model; evaluating the candidate strategy prediction model to obtain an evaluation result; when the evaluation result meets a preset condition, taking the strategy prediction model as the trained strategy prediction model; when the evaluation result does not meet the preset condition, retraining the candidate strategy prediction model until the evaluation result of the candidate strategy prediction model meets the preset condition, and obtaining the trained strategy prediction model.

[0155] To improve the reliability of the model, the candidate strategy prediction model obtained by training can be evaluated. For example, for the candidate strategy prediction model obtained by training, the candidate strategy prediction model can be played against a base model, which can be a baseline model trained based on supervised learning, a randomly initialized model, or a data model of a real player's operation. After the game is completed, the probability distribution of the game and the actual behavior value are obtained, and the evaluation result can be obtained. For example, the position of the strategy prediction model can be randomly assigned, and after 1000 games of play against two base models, the average probability distribution and win rate of the strategy prediction model are calculated to obtain the performance of the strategy prediction model. When the evaluation result meets the preset condition, the strategy prediction model is taken as the trained strategy prediction model. The preset condition can be flexibly set according to actual needs. For example, when the number of wins in 1000 evaluations is greater than 900, it is determined that the preset condition is met. When the evaluation result does not meet the preset condition, the candidate strategy prediction model is retrained until the evaluation result of the candidate strategy prediction model meets the preset condition, and the trained strategy prediction model is obtained, so that whether the strategy prediction model converges can be effectively evaluated.

[0156] In order to further illustrate the superiority of the strategy prediction model training method provided by the present application, the following will be described through an example of using the method in a specific game scene. The method is used in a simulation environment named interactive simulator, in which NPCs in the game can go to different scenes to consume and win balls, and then obtain and eliminate balls of colors corresponding to the scenes, wherein the NPC scores when three balls of the same color are obtained, and a color is randomly initialized, and the higher the score obtained within a certain time (5 minutes) is, the better. Different buildings have two different colors and are located at different positions on the map, and the two colors of the building will be alternately changed after the NPC performs an action in the building. Each NPC has 7 action spaces (moving in 4 directions, consuming balls, winning balls, and waiting in place). There are 4 NPCs, 8 buildings and 4 colors in the game, and in order to obtain a higher score within limited time and resources, each NPC needs to cooperate and plan its own action route. For example, NPC 1 has two red balls, NPC 2 has one blue ball, building A is currently red, and the other is blue, then NPC 2 should go to building A to win a ball and make building A reverse to red, and then NPC 1 goes to win a ball, NPC 1 scores three red balls and makes A reverse to blue, and NPC 2 goes to A to win a ball to score three blue balls. In the game, the player controls another interactive NPC, which can eliminate or increase the colors owned by the target NPC under certain conditions, so that the target NPC cannot score three balls of the same color. The lower the score of the target NPC is, the higher the score obtained by the player is. In the game, it is necessary to train 4 intelligent NPCs to control 4 target NPCs, so that these NPCs can obtain a higher score within a certain time. The models trained by traditional supervised learning and reinforcement learning can only issue the next specific operation, but cannot give a complete strategy. This is extremely unfavorable for users to evaluate the rationality of the model and predict the action path of the NPC. For the player, if the action planning of the NPC is known, the player can intercept or avoid in advance, make an early prediction, and the developer can also develop different difficulty and more abundant gameplay based on the strategy of the NPC.

[0157] In summary, the strategy prediction model training method provided by the present embodiment can train a model outputting a strategy, give a clear strategy instead of micro-operation, and has better interpretability, so that the model can be better utilized. Meanwhile, the method combines behavior tree and reinforcement training, solves the problem of outputting micro-operation to outputting strategy, and provides a feasible method for strategy training.

[0158] As shown in Figure 4 The strategy prediction model training device provided by the present embodiment comprises:

[0159] The sample production module 100 is configured to produce sample data.

[0160] The sample production module 100 comprises:

[0161] A state acquisition unit 110 is configured to acquire real-time state information of an AI object in a simulation environment under a current environment.

[0162] A feature extraction unit 120 is configured to extract first feature information and second feature information from the real-time state information.

[0163] A model inference unit 130 is configured to input the first feature information and the second feature information into a current policy prediction model for model inference.

[0164] A policy output unit 140 is configured to obtain a target policy according to an output of the model inference.

[0165] An action output unit 150 is configured to trigger a state machine to output a specific action corresponding to the AI object according to the target policy.

[0166] An interaction control unit 160 is configured to control the AI object to perform the specific action and interact with the current environment.

[0167] A sample saving unit 170 is configured to save data generated in the above process as sample data for training.

[0168] A model training module 200 is configured to train and optimize the policy prediction model through a plurality of sample data.

[0169] As a preferred embodiment of the present embodiment, the policy prediction model is a convolutional neural network model, which comprises a first convolutional layer, a second convolutional layer, a first fully connected layer, a second fully connected layer, a third fully connected layer, and a fourth fully connected layer.

[0170] As a preferred embodiment of the present embodiment, the feature extraction unit 120 is specifically configured to:

[0171] extract image feature information and vector feature information in the real-time state information.

[0172] As a preferred embodiment of the present embodiment, the model inference unit 130 is specifically configured to:

[0173] operate the image feature information through the first convolutional layer and the second convolutional layer;

[0174] process the vector feature information through the first fully connected layer and the second fully connected layer.

[0175] The results of the operation and processing are spliced and then processed by the third full connection layer and the fourth full connection layer to obtain a strategy probability distribution and a Q value for evaluating the strategy probability distribution as an output of model inference.

[0176] As a preferred embodiment of the present embodiment, the strategy output unit 140 is specifically configured to:

[0177] The target strategy is obtained by sampling the strategy probability distribution.

[0178] As a preferred embodiment of the present embodiment, the action output unit 150 is specifically configured to:

[0179] The state machine determines a corresponding execution action according to the target strategy.

[0180] The feasibility of the execution action is determined.

[0181] If feasible, the execution action is output as a specific action of the AI object, and if not feasible, an error message is returned.

[0182] As a preferred embodiment of the present embodiment, the sample data includes image feature information, vector feature information, generalized advantage estimation (GAE), and reward information.

[0183] As a preferred embodiment of the present embodiment, the model training module 200 is specifically configured to:

[0184] The saved multiple sample data are obtained.

[0185] The obtained sample data are assembled by a batch module.

[0186] The assembled sample data are input into the strategy prediction model to calculate a strategy gradient.

[0187] The strategy prediction model is trained by reinforcement learning according to the sample data and the strategy gradient.

[0188] The strategy prediction model training device provided in the embodiments of the present application is used to implement the strategy prediction model training method described above, and thus the specific embodiments are the same as the method described above, and will not be described here again.

[0189] As Figure 5As shown, the embodiment of the present application provides a structural block diagram of an electronic device 300. The electronic device 300 can be a smart phone, a tablet computer, an electronic book, or the like, which can run an application program. The electronic device 300 in the present application can include one or more of the following components: a processor 310, a memory 320, and one or more application programs, wherein the one or more application programs can be stored in the memory 320 and configured to be executed by the one or more processors 310, and the one or more programs are configured to perform the method as described in the foregoing method embodiments.

[0190] The processor 310 can include one or more processing cores. The processor 310 connects various parts within the entire electronic device 300 through various interfaces and lines, performs various functions of the electronic device 300 and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory 320, and calling data stored in the memory 320. Optionally, the processor 310 can be implemented in at least one of a hardware form of a digital signal processing (DSP), a field-programmable gate array (FPGA), and a programmable logic array (PLA). The processor 310 can integrate a combination of one or more of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. Among them, the CPU mainly processes the operating system, the user interface, and the application program, etc.; the GPU is responsible for rendering and drawing display content; and the modem is used for processing wireless communication. It can be understood that the above-mentioned modem can also not be integrated into the processor 310, but can be realized by a separate communication chip.

[0191] The memory 320 can include a random access memory (RAM) and can also include a read-only memory (ROM). The memory 320 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 320 can include a program storage area and a data storage area, wherein the program storage area can store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playing function, an image playing function, etc.), instructions for implementing each of the following method embodiments, etc. The data storage area can also store data created by the terminal in use (such as a phone book, audio and video data, chat record data), etc.

[0192] As Figure 6As shown, the embodiment of the present application provides a structural block diagram of a computer readable storage medium 400. The computer readable medium stores program code 410, which can be called by a processor to execute the method described in the above method embodiment.

[0193] The computer readable storage medium 400 can be an electronic memory such as a flash memory, an EEPROM (Electrically Erasable Programmable Read-Only Memory), an EPROM, a hard disk or a ROM. Alternatively, the computer readable storage medium 400 comprises a non-transitory computer readable medium. The computer readable storage medium 400 has a storage space for the program code 410 to execute any of the above methods. These program codes 410 can be read from or written to one or more computer program products. The program code 410 can be compressed in an appropriate form, for example.

[0194] To sum up, the present application provides a strategy prediction model training method and device, electronic equipment and storage medium, which can train a prediction model outputting a strategy based on the real-time state information of the simulation environment and the actual behavior feedback of the AI object. When the model is inferred, an explicit strategy can be given, which is more interpretable, so that the user can better utilize the model. At the same time, the technical solution provided by the embodiments of the present application can make corresponding decisions according to the real-time changes of the environment, which can improve the flexibility of AI.

[0195] In several embodiments disclosed in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are only schematic. For example, the flowcharts and block diagrams in the drawings show the possible implementation architectures, functions and operations of the devices, methods and computer program products according to the embodiments of the present application. In this regard, each block in the flowcharts or block diagrams can represent a module, a program segment or a part of code, which contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur in different orders from those shown in the drawings. For example, two consecutive blocks can actually be executed substantially in parallel, and sometimes they can be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and the combination of blocks in the block diagrams and / or flowcharts, can be implemented by a dedicated hardware-based system for executing the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0196] In addition, each functional module in each embodiment of the present application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0197] The functions, if implemented in the form of software functional modules and sold or used as independent products, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the parts of the prior art that make contributions or parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

Claims

1. A method for training a policy prediction model for AI objects, characterized in that, The method comprises: obtaining real-time state information of an AI object in a simulation environment under a current environment; extracting first feature information and second feature information from the real-time state information; inputting the first feature information and the second feature information into a current policy prediction model for model inference to obtain a policy probability distribution and a Q value for evaluating the policy probability distribution; sampling the policy probability distribution to obtain a target policy; triggering a state machine to output a specific action corresponding to the AI object according to the target policy; controlling the AI object to perform the specific action and interact with the current environment; saving data generated in the above process as sample data for training; training and optimizing the policy prediction model through multiple sample data; The step of extracting first feature information and second feature information from the real-time state information comprises: extracting image feature information and vector feature information from the real-time state information, wherein the image feature information corresponds to a map scene in the simulation environment, and the vector feature information corresponds to the related state of the AI object. 2.The AI object's policy prediction model training method of claim 1, wherein, The policy prediction model is a convolutional neural network model, which comprises a first convolutional layer, a second convolutional layer, a first fully connected layer, a second fully connected layer, a third fully connected layer, and a fourth fully connected layer. 3.The AI object's policy prediction model training method of claim 2, wherein, The step of inputting the first feature information and the second feature information into the current policy prediction model for model inference comprises: operating the image feature information through the first convolutional layer and the second convolutional layer; processing the vector feature information through the first fully connected layer and the second fully connected layer; concatenating the operation and processing results and processing them through the third fully connected layer and the fourth fully connected layer to obtain the policy probability distribution and the Q value for evaluating the policy probability distribution as the output of the model inference. 4.The AI object's policy prediction model training method of claim 3, wherein, The step of triggering the state machine to output the specific action corresponding to the AI object according to the target policy comprises: The state machine determines the corresponding execution action according to the target policy. determining the feasibility of the execution action; if feasible, outputting the execution action as the specific action of the AI object, and if not feasible, returning an error message. 5.The AI object's policy prediction model training method of claim 4, wherein, The sample data comprises image feature information, vector feature information, generalized advantage estimation GAE, and reward information. 6.The AI object's policy prediction model training method of claim 5, wherein, The step of training and optimizing the policy prediction model through multiple sample data comprises: obtaining multiple saved sample data; assembling the obtained sample data through a Batch module; inputting the assembled sample data into the policy prediction model to calculate a policy gradient; performing reinforcement learning training on the policy prediction model according to the sample data and the policy gradient. 7.A device for training a policy prediction model of an AI object, comprising: The device comprises: a sample production module for producing sample data; wherein the sample production module comprises: a state acquisition unit for acquiring real-time state information of an AI object in a simulation environment under a current environment; The feature extraction unit is configured to extract first feature information and second feature information from the real-time state information, the first feature information corresponding to environment background related state information, and the second feature information corresponding to AI object self state related state information. The model inference unit is configured to input the first feature information and the second feature information into a current policy prediction model for model inference, to obtain a policy probability distribution and a Q value for evaluating the policy probability distribution. The policy output unit is configured to sample the policy probability distribution to obtain a target policy. The action output unit is configured to trigger a state machine to output a specific action corresponding to the AI object according to the target policy. The interaction control unit is configured to control the AI object to perform the specific action and interact with the current environment. The sample saving unit is configured to save data generated in the above process as sample data for training. The model training module is configured to train and optimize the policy prediction model through a plurality of sample data. The feature extraction unit is further configured to extract image feature information and vector feature information in the real-time state information, the image feature information corresponding to a map scene in a simulation environment, and the vector feature information corresponding to related states of the AI object.

8. An electronic device, comprising: One or more processors; Memory; One or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to perform the method of any one of claims 1-6. ​

Images