A method and device for training a behavior strategy model of a virtual opponent

Abstract

The present invention relates to the field of deep reinforcement learning, and in particular to a method and device for training a behavior strategy model of a virtual opponent. It includes: establishing a behavior strategy model of a virtual opponent, establishing a reward model according to the reward value after the designated opponent executes each behavior; wherein, the optional behavior that is more in line with the historical behavior style of the designated opponent has a higher reward value; obtaining at least two optional behaviors of the virtual opponent in each state, comparing all optional behaviors in each state in pairs, and updating the reward model according to the comparison result; wherein, the optional behavior that is more in line with the historical behavior style of the opponent has a better comparison result; predicting the behavior of the virtual opponent in each state, using the updated reward model to obtain the reward value after each behavior is executed, and training the behavior strategy model based on the reward value to simulate the behavior of the virtual opponent through the trained behavior strategy model. The present invention can train a behavior strategy model whose prediction results are more in line with the style of the designated opponent.

Description

Technical Field

[0001] The present invention relates to the field of deep reinforcement learning, and in particular to a method and device for training a behavior strategy model of a virtual opponent. Background Art

[0002] In recent years, deep reinforcement learning technology has made breakthrough progress by combining the perception ability of deep learning with the decision-making ability of reinforcement learning. It has been widely used in the fields of chess games, game AI, autonomous driving, robot control, etc. At present, it is possible to obtain a high-level intelligent agent through artificial decision trees or reinforcement learning methods to simulate humans to make decisions in different scenarios.

[0003] In the existing decision-making process, the final victory result is usually used as the guide for strategy selection, which makes the strategy selection relatively simple. However, in competitive games, such as chess games, massively multiplayer online role-playing games (MMORPG) games, and multiplayer online tactical competitive games (MOBA) games, different opponents will have different styles of behavior characteristics, and the result-oriented behavior strategy model cannot well simulate the response of opponents of different styles to one's own strategy, nor can it well simulate the behavioral characteristics of opponents of different styles in specific environments.

[0004] In view of this, how to overcome the defects of the existing technology and solve the problem that the existing technology cannot complete the behavior strategy prediction according to the behavior style of the specified opponent is a problem to be solved in this technical field. Summary of the invention

[0005] In view of the above defects or improvement needs of the prior art, the present invention solves the problem in the prior art that the behavior strategy prediction cannot be completed according to the behavior style of a specified opponent.

[0006] The embodiment of the present invention adopts the following technical solution:

[0007] In a first aspect, the present invention provides a method for training a behavior strategy model of a virtual opponent, specifically: establishing a behavior strategy model of a virtual opponent, and establishing a reward model according to the reward value after the designated opponent executes each behavior; wherein, the optional behavior that is more in line with the historical behavior style of the designated opponent has a higher reward value; obtaining at least two optional behaviors of the virtual opponent in each state based on the behavior strategy model, comparing all optional behaviors in each state in pairs, and updating the reward model according to the comparison results; wherein, the optional behavior that is more in line with the historical behavior style of the opponent has a better comparison result; predicting the behavior of the virtual opponent in each state based on the behavior strategy model, using the updated reward model to obtain the reward value after each behavior is executed, and training the behavior strategy model based on the reward value to simulate the behavior of the virtual opponent through the trained behavior strategy model.

[0008] Preferably, the establishment of the behavior strategy model of the virtual opponent specifically includes: the behavior strategy model is expressed as: A (a|s,θ A ), the behavior strategy model includes a convolutional neural network; where θ A are the parameters of the behavior strategy model, s represents the state of the reinforcement learning environment at the current moment, and a represents the behavior performed by the virtual opponent at the current moment; the output of the behavior strategy model represents the probability that the virtual opponent chooses behavior a when the state is s.

[0009] Preferably, the reward model is established according to the reward value after the designated opponent performs each behavior, specifically including: the reward model is expressed as: θ (s t ,a t ,s t+1 ), the reward model contains a deep neural network; where θ is the parameter of the reward model, a t is the action performed by the virtual opponent at time t, s t represents the state of the reinforcement learning environment at time t, s t+1 Represents the state of the reinforcement learning environment at time t+1; the output of the reward model is the reward value r at time t t , the virtual opponent is in state s at time t t The following execution behavior is a t After that, if the state is transferred to s at time t+1 t+1 , then the reward value at time t is r t =r θ (s t ,a t ,s t+1 ).

[0010] Preferably, the loss function used in the reward model is expressed as follows:

[0011]

[0012] Where loss(·) represents the loss function, θ is the parameter of the loss function; K is the number of optional behaviors; is the expectation function, which represents the expectation of all comparison results, D is the set of comparison results; σ(·) represents the activation function; r θ (s t ,a w ,s t+1 ) represents the reward value corresponding to the better behavior in the comparison result, r θ (s t ,a l ,s t+1 ) represents the reward value corresponding to the worse behavior in the comparison result.

[0013] Preferably, the method obtains at least two optional behaviors of the virtual opponent in each state based on the behavior strategy model, compares all optional behaviors in each state in pairs, and updates the reward model according to the comparison results, specifically including: inputting the current state into the behavior strategy model, sampling at least two optional behaviors according to the output of the behavior strategy model; comparing all optional behaviors in pairs according to the historical behavior style of the designated opponent, and adding the comparison results to a comparison result set; and training the reward model using supervised learning based on the data in the comparison result set until the reward model training is completed.

[0014] Preferably, all optional behaviors are compared pairwise based on the historical behavior style of the designated opponent, and the comparison results are added to a comparison result set, specifically including: obtaining the behavior of the designated opponent in the current state based on the historical behavior style of the designated opponent, and using the obtained behavior as the baseline behavior; obtaining all pairwise combinations of all optional behaviors, comparing the two optional behaviors in each combination with the baseline behavior respectively, and using the optional behavior that is more consistent with the baseline behavior as the better optional behavior; adding the comparison result of each combination to the comparison result set; wherein the comparison result includes the relationship between the current state and the advantages and disadvantages of the two optional behaviors.

[0015] Preferably, the sampling according to the output of the behavior strategy model obtains at least two optional behaviors, specifically including: obtaining the probability of the virtual opponent executing each behavior in the current state according to the output of the behavior strategy model; sampling according to the specified probability distribution and the probability of each behavior, and taking the behavior that meets the specified probability distribution as the optional behavior.

[0016] Preferably, the method predicts the behavior of the virtual opponent in each state based on the behavior strategy model, uses the updated reward model to obtain the reward value after each behavior is executed, and trains the behavior strategy model based on the reward value, specifically including: using the behavior strategy model to obtain the predicted data and reward value in each time step, and calculating the deviation value of the behavior from the historical behavior style of the specified opponent based on the probability of each behavior in the predicted data; dividing all time steps into at least two time periods, and calculating the average reward for each time period based on the reward values ​​of all time steps in each time period; specifying the target reward to be achieved, and when the average reward of one or more time periods is less than the target reward, updating the behavior strategy model based on the deviation value and reward value of the corresponding time step until the average reward of all time periods is greater than the target reward.

[0017] In a second aspect, the present invention provides a device for training a behavior strategy model of a virtual opponent, specifically comprising: at least one processor and a memory, wherein the at least one processor and the memory are connected via a data bus, and the memory stores instructions that can be executed by the at least one processor, and after the instructions are executed by the processor, they are used to complete the behavior strategy model training method of the virtual opponent in the first aspect.

[0018] In a third aspect, the present invention further provides a non-volatile computer storage medium, wherein the computer storage medium stores computer executable instructions, and the computer executable instructions are executed by one or more processors to complete the method described in the first aspect.

[0019] In a fourth aspect, a chip is provided, comprising: a processor and an interface, for calling and running a computer program stored in the memory from a memory, and executing the method as described in the first aspect.

[0020] In a fifth aspect, a computer program product comprising instructions is provided, which, when executed on a computer or a processor, causes the computer or the processor to execute the method as described in the first aspect.

[0021] Compared with the prior art, the beneficial effect of the present invention lies in: by using the reinforcement learning method to fine-tune the strategy model through the opponent reward model on the basis of the pre-trained behavior strategy model, it can be closer to the opponent's behavior pattern, thereby greatly improving the targeting of our players during training. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0023] Figure 1 A flow chart of a method for training a behavior strategy model of a virtual opponent provided by an embodiment of the present invention;

[0024] Figure 2 A flow chart of another method for training a behavior strategy model of a virtual opponent provided by an embodiment of the present invention;

[0025] Figure 3 A flow chart of another method for training a behavior strategy model of a virtual opponent provided by an embodiment of the present invention;

[0026] Figure 4 A flow chart of another method for training a behavior strategy model of a virtual opponent provided by an embodiment of the present invention;

[0027] Figure 5 A schematic diagram of the structure of a virtual opponent behavior strategy model training device provided by an embodiment of the present invention;

[0028] The reference numerals are as follows:

[0029] 11: processor; 12: memory. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0031] The present invention is an architecture of a specific functional system, so the specific embodiments mainly illustrate the functional logical relationship between the various structural modules, and do not limit the specific software and hardware implementation methods.

[0032] In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other. The present invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0033] Embodiment 1:

[0034] In competitive sports, when conducting simulated competitive training or arranging the order of players' games, if there is an intelligent agent that matches the behavior style of the specified opponent, the pertinence of the players in training can be greatly improved. In this embodiment, in view of the problem that the existing behavior strategy prediction model cannot simulate the style of the specified opponent well, the prediction result is evaluated based on the similarity between the predicted behavior and the style of the specified opponent, rather than evaluating based on the result after the action is executed in the prior art, so that the prediction result of the behavior strategy model can be closer to the behavior pattern of the opponent.

[0035] like Figure 1 As shown, the specific steps of the behavior strategy model training method for a virtual opponent provided by an embodiment of the present invention are as follows:

[0036] Step 101: Establish a behavior strategy model of a virtual opponent, and establish a reward model according to the reward value after the designated opponent performs each behavior.

[0037] In this embodiment, the behavior strategy model needs to predict the behavior that best matches the style of the specified opponent at the next moment based on the current environmental state. The behavior strategy model is expressed as: A (a|s,θ A ), the behavior strategy model includes a convolutional neural network. Among them, θ A is the parameter of the behavior strategy model, s represents the state of the reinforcement learning environment at the current moment, and a represents the behavior performed by the virtual opponent at the current moment; the output of the behavior strategy model represents the probability that the virtual opponent chooses behavior a when the state is s. In actual implementation, a corresponding general pre-trained model can be selected as the initial behavior strategy model according to the specific competitive event. For example, in the Go event, a pre-trained model such as AlphaGO or AlphaZero can be selected, and then the training method provided in this embodiment can be used to make its prediction result conform to the style of the specified opponent.

[0038] On the other hand, in order to provide feedback on the training effect of the behavior strategy model, a reward model needs to be established to evaluate the prediction results of the behavior strategy model according to the specified opponent style. In this embodiment, the training purpose of the behavior strategy model is to predict a strategy that is more in line with the specified opponent style. Therefore, during training, the optional behavior that is more in line with the specified opponent's historical behavior style has a higher reward value. The reward model is expressed as: θ (s t ,a t ,s t+1 ), the reward model contains a deep neural network; where θ is the parameter of the reward model, a t is the action performed by the virtual opponent at time t, s t represents the state of the reinforcement learning environment at time t, s t+1 Represents the state of the reinforcement learning environment at time t+1; the output of the reward model is the reward value r at time t t , the virtual opponent is in state s at time t t The following execution behavior is a t After that, if the state is transferred to s at time t+1 t+1 , then the reward value at time t is r t =r θ (s t ,a t ,s t+1 ).

[0039] Step 102: Based on the behavior strategy model, at least two optional behaviors of the virtual opponent in each state are obtained, all optional behaviors in each state are compared pairwise, and the reward model is updated according to the comparison results.

[0040] In the method provided in this embodiment, multiple optional behaviors are obtained through the behavior strategy model, and then all optional behaviors are compared in pairs. When comparing, the optional behavior that is more in line with the opponent's historical behavior style has a better comparison result. The reward model is then trained and updated based on the comparison results. After the training and updating are completed, the reward model can give a higher reward value to the opponent's behavior that is more in line with the expectations, thereby more accurately judging the output results of the behavior strategy model.

[0041] Step 103: Predict the behavior of the virtual opponent in each state based on the behavior strategy model, use the updated reward model to obtain the reward value after each behavior is executed, and train the behavior strategy model based on the reward value to simulate the behavior of the virtual opponent through the trained behavior strategy model.

[0042] After obtaining a reward model with more accurate feedback, the updated reward model can be used to provide feedback on the virtual opponent's behavior predicted by the behavior strategy model, and the behavior strategy model can be trained through the feedback results, and finally a behavior strategy model that can accurately simulate the opponent's behavior strategy can be obtained. In specific implementation, the behavior strategy model can be trained using deep reinforcement learning, for example, the Policy Gradient algorithm (PG), the Proximity Policy Optilisation algorithm (PPO), etc.

[0043] In specific implementation, when the behavioral strategy model is trained, the opponent player can be used as the designated opponent, and the opponent player's historical behavior can be used as a training sample for iterative training. The trained intelligent agent can be used as a virtual opponent for the training of one's own players to improve the targeted training of one's own players, or predict the possible game strategy of the opponent player; one's own player can also be used as the designated opponent, and the opponent's historical behavior can be used as a training sample for iterative training. The trained intelligent agent can be used to simulate the competition of one's own player to optimize the game strategy of one's own player, or select one's own player who is most targeted against a certain opponent player; thereby different strategy simulation needs can be met through different training data.

[0044] After steps 101 to 103 provided in this embodiment, the training of the virtual opponent behavior strategy model can be completed.

[0045] like Figure 2As shown, at least two optional behaviors of the opponent in each state can be obtained based on the behavior strategy model in the following manner, all optional behaviors in each state are compared pairwise, and the reward model is updated according to the comparison results.

[0046] Step 201: Input the current state into the behavior strategy model, and sample at least two optional behaviors according to the output of the behavior strategy model.

[0047] In this embodiment, the more similar the predicted result is to the specified opponent's behavior style, the higher the reward value. Therefore, it is necessary to obtain the actions that the virtual opponent can perform in the current state through the initial behavior strategy model, and use these actions as optional behaviors so that the reward values ​​of different optional behaviors can be adjusted later according to the specified opponent's behavior style.

[0048] First, the reinforcement learning environment of the behavior strategy model is reset to obtain the initial state s0 of the reinforcement learning environment at the initial moment. Then, starting from the initial state s0, the behavior a corresponding to each state is generated through the behavior strategy model, and the state after the execution of behavior a is used as the state at the next moment. At the current moment t, the state of the reinforcement learning environment is s t . Set the current state s t Input to the behavior policy model π A (a|s,θ A ), by sampling the output results of the behavior strategy model, we can get K optional behaviors of the opponent.

[0049] In the specific implementation, optional behaviors can be sampled based on probability distribution. Specifically, according to the output of the behavior strategy model, the probability of the virtual opponent executing each behavior in the current state is obtained; sampling is performed according to the specified probability distribution and the probability of each behavior, and the behavior that meets the specified probability distribution is used as an optional behavior. In general scenarios, behavior a with a probability higher than the specified probability value can be used as an optional behavior to ensure that the optional behavior can achieve good results in confrontation while meeting the specified opponent's behavior style.

[0050] Step 202: According to the historical behavior style of the designated opponent, all optional behaviors are compared in pairs, and the comparison results are added to the comparison result set.

[0051] In this embodiment, the purpose of training the behavior strategy model is to obtain a prediction result that is more consistent with the behavior style of the specified opponent, that is, to obtain the behavior that the specified opponent is most likely to perform in the current state according to its own style. In this embodiment, all optional behaviors are compared in pairs to obtain one of any two optional behaviors that better matches the behavior style of the specified opponent, and the comparison result is added to the comparison result set for subsequent use.

[0052] Step 203: Based on the data in the comparison result set, the reward model is trained using a supervised learning method until the reward model training is completed.

[0053] After obtaining the comparison result set, the comparison results in the comparison result set can be used to train the reward model using supervised learning. The loss function used in the reward model is expressed as follows:

[0054]

[0055] Where loss(·) represents the loss function, θ is the parameter of the loss function; K is the number of optional behaviors; is the expectation function, which represents the expectation of all comparison results, D is the set of comparison results; σ(·) represents the activation function; r θ (s t ,a w ,s t+1 ) represents the reward value corresponding to the better behavior in the comparison result, r θ (s t ,a l ,s t+1 ) represents the reward value corresponding to the worse behavior in the comparison result. In the specific implementation, the activation function can use Sigmoid, Tanh, Relu, LeakyRelu, softmax, etc., preferably using the sigmoid activation function to improve the discrimination of the reward value.

[0056] Repeat steps 201 to 203 until the reward model converges to better performance or reaches the set maximum number of training times, and the training of the reward model is completed to obtain a more accurate reward model.

[0057] After steps 201 to 203 provided in this embodiment, the training of the reward model can be completed, so that the reward model can obtain the corresponding reward value according to the matching degree between the optional behavior and the behavior style of the designated opponent.

[0058] like Figure 3 As shown, all optional behaviors can be compared pairwise according to the historical behavior style of the designated opponent in the following manner, and the comparison results can be added to the comparison result set.

[0059] Step 301: Obtain the behavior of the designated opponent in the current state according to the behavior style of the designated opponent, and use the obtained behavior as the benchmark behavior.

[0060] In this embodiment, it is necessary to compare the optional behaviors according to the behavior style of the designated opponent. Therefore, it is necessary to first obtain the behavior that best matches the behavior style of the designated opponent in each state as the benchmark behavior. In actual implementation, the historical behavior of the designated opponent is the behavior that best matches its behavior style. Furthermore, in order to avoid too little data or missing historical behaviors in certain states, behaviors that match the behavior style of the designated opponent can also be selected from the prediction results through manual evaluation and other methods.

[0061] Step 302: Obtain all pairwise combinations of all optional behaviors, compare the two optional behaviors in each combination with the benchmark behavior respectively, and take the optional behavior that is more consistent with the benchmark behavior as the better optional behavior.

[0062] When there are K optional behaviors, the total number of pairwise combinations among all optional behaviors is Each combination contains an optional behavior a w With an optional behavior a l . Set the optional behavior in each combination to a w and optional behavior a l By comparing with the benchmark behavior respectively, one of the two optional behaviors can be obtained that is the same as or more similar to the benchmark behavior.

[0063] In specific implementations, the similarity between the optional behavior and the baseline behavior can be determined based on the category, purpose, and execution result of the behavior. Specifically, similar behaviors can be grouped through expert systems, clustering, knowledge graphs, and other methods. Behaviors in the same group are more consistent than behaviors in different groups. The closer the distance between behaviors in the same group, the higher the consistency. In a specific scenario, the baseline behavior is to dodge left, and the optional behavior a is to avoid the left. w is to dodge to the right, optional action a l is blocking, the baseline behavior and the optional behavior a w Both belong to the dodge category group, and blocking belongs to the defense category group, so the optional behavior is determined to be a w It is more consistent with the baseline behavior and is a better optional behavior.

[0064] Step 303: Add the comparison result of each combination to the comparison result set.

[0065] After all optional behavior combinations have been compared, The comparison results can be added to the comparison result set for subsequent use. The comparison results include the current state and the pros and cons of the two optional behaviors. t In the state, the predicted optional behavior a w Better than optional behavior a l , the comparison result of this combination is expressed as: (st ,a w >a l ).

[0066] After steps 301 to 303 provided in this embodiment, the comparison of the optional behaviors can be completed.

[0067] like Figure 4 As shown, the behavior of the virtual opponent in each state can be predicted based on the behavior strategy model, the reward value after each behavior is executed is obtained using the updated reward model, and the behavior strategy model is trained based on the reward value.

[0068] Step 401: Use the behavior strategy model to obtain the predicted data and reward value in each time step, and calculate the deviation value between the behavior and the historical behavior style of the specified opponent based on the probability of each behavior in the predicted data.

[0069] Get the current state of the environment s t , the current environmental state s t and the behavior a predicted by the current behavior strategy model t Input into the reward model to get the reward value r t .

[0070] On the other hand, in this embodiment, in order to make the output results of the behavior strategy model more consistent with the behavior style of the specified player, it is also necessary to consider the deviation between the predicted behavior and the historical behavior style of the specified opponent. t Statistics are collected for all historical behaviors performed by the opponent, the probability of each historical behavior is obtained, and the difference between the probability of the same behavior in the historical behavior and the probability in the predicted result is taken as the deviation value of the behavior.

[0071] Step 402: Divide all time steps into at least two time periods, and calculate the average reward of each time period based on the reward values ​​of all time steps in each time period.

[0072] In order to reduce the amount of data during training and balance the short-term and long-term advantages in the competition process, in this embodiment, the reward value r of each time step is not directly used. t Instead, all time steps in a game are segmented and the output results of the behavior strategy model in each time period are evaluated separately.

[0073] Step 403: Specify the target reward to be achieved. When the average reward of one or more time periods is less than the target reward, update the behavior strategy model according to the deviation value and reward value of the corresponding time step until the average reward of all time periods is greater than the target reward.

[0074] When the average reward of one or more time periods is less than the target reward, it indicates that the output value of the behavior strategy model is not optimal under certain conditions, which may cause the predicted behavior to not conform to the behavior style of the specified opponent or not be the best response behavior of the specified opponent. Therefore, the behavior strategy model still needs to be optimized. In actual implementation, the behavior strategy model π can be optimized through reinforcement learning methods. A (a|s,θ A ) to fine-tune the behavior strategy model that is more consistent with the behavior of the specified opponent.

[0075] In a specific implementation scenario, the proximal policy optimization algorithm is selected. In the proximal policy optimization algorithm, the optimization goal of the strategy is to maximize the long-term return, and the difference between the updated behavior policy model and the behavior policy model before the update is limited by the constraint term. In actual implementation, the cut function can be used to limit the difference between the new strategy and the old strategy to a given range. During the optimization process, the proximal policy optimization algorithm optimizes the strategy by alternately sampling data from the output of the behavior policy model and performing several optimization stages on the sampled data. During each optimization process, the proximal policy optimization algorithm calculates the policy gradient and uses the gradient ascent algorithm to update the parameters of the neural network in the behavior policy model.

[0076] After steps 401 to 403 provided in this embodiment, the behavior strategy model can be trained and fine-tuned according to the updated reward function, so that it can output a behavior strategy model that is more consistent with the behavior pattern of the specified opponent.

[0077] This embodiment provides a method for training a behavior strategy model of a virtual opponent. Based on a pre-trained behavior strategy model, a reinforcement learning method is used through a reward model to fine-tune the strategy model so that it can be closer to the opponent's behavior pattern.

[0078] Embodiment 2:

[0079] Based on the behavior strategy model training method for a virtual opponent provided in the above embodiment 1, the present invention also provides a behavior strategy model training device for a virtual opponent that can be used to implement the above method, such as Figure 5 , which is a schematic diagram of the device architecture of an embodiment of the present invention. The virtual opponent behavior strategy model training device of this embodiment includes one or more processors 11 and a memory 12. Figure 5 A processor 11 is taken as an example.

[0080] The processor 11 and the memory 12 may be connected via a bus or other means. Figure 5 The example of connecting through bus is taken in the following.

[0081] The memory 12 is a non-volatile computer-readable storage medium as a method for training a behavior strategy model of a virtual opponent, and can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as the method for training a behavior strategy model of a virtual opponent in Example 1. The processor 11 executes various functional applications and data processing of the device for training a behavior strategy model of a virtual opponent by running the non-volatile software programs, instructions and modules stored in the memory 12, that is, the method for training a behavior strategy model of a virtual opponent in Example 1 is implemented.

[0082] The memory 12 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage devices. In some embodiments, the memory 12 may optionally include a memory remotely arranged relative to the processor 11, and these remote memories may be connected to the processor 11 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0083] The program instructions / modules are stored in the memory 12, and when executed by one or more processors 11, the behavior strategy model training method of the virtual opponent in the above embodiment 1 is executed, for example, the above described Figures 1 to 4 The steps shown.

[0084] An embodiment of the present invention further provides a non-volatile computer storage medium, wherein the computer storage medium stores computer executable instructions, and the computer executable instructions are executed by one or more processors, such as Figure 5 A processor 11 in the embodiment may enable the one or more processors to execute the behavior strategy model training method of the virtual opponent in embodiment 1, for example, to execute the above-described Figures 1 to 4 The steps shown.

[0085] It is worth noting that the information interaction, execution process, etc. between the modules and units within the above-mentioned devices and systems are based on the same concept as the processing method embodiment of the present invention. The specific contents can be found in the description of the method embodiment of the present invention and will not be repeated here.

[0086] A person skilled in the art may understand that all or part of the steps in the various methods of the embodiments may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the storage medium may include: a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.

[0087] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for training a behavior strategy model of a virtual opponent, characterized in that: include: Establish a behavior strategy model for the virtual opponent, and establish a reward model based on the reward value after the designated opponent performs each behavior; the optional behavior that is more in line with the historical behavior style of the designated opponent has a higher reward value; Based on the behavior strategy model, at least two optional behaviors of the virtual opponent in each state are obtained, all optional behaviors in each state are compared pairwise, and the reward model is updated according to the comparison results; among them, the optional behavior that is more consistent with the opponent's historical behavior style has a better comparison result; The behavior of the virtual opponent in each state is predicted based on the behavior strategy model, and the reward value after each behavior is executed is obtained using the updated reward model. The behavior strategy model is trained based on the reward value to simulate the behavior of the virtual opponent through the trained behavior strategy model.

2. The method for training a behavior strategy model of a virtual opponent according to claim 1, characterized in that: The establishment of the behavior strategy model of the virtual opponent specifically includes: The behavior strategy model is expressed as: π A (a|s,θ A ), the behavior strategy model includes a convolutional neural network; where θ A are the parameters of the behavior strategy model, s represents the state of the reinforcement learning environment at the current moment, and a represents the behavior performed by the virtual opponent at the current moment; the output of the behavior strategy model represents the probability that the virtual opponent chooses behavior a when the state is s.

3. The method for training a behavior strategy model of a virtual opponent according to claim 1, characterized in that: The reward model is established according to the reward value after the designated opponent performs each behavior, specifically including: The reward model is expressed as: r θ (s t ,a t ,s t+1 ), the reward model contains a deep neural network; where θ is the parameter of the reward model, a t is the action performed by the virtual opponent at time t, s t represents the state of the reinforcement learning environment at time t, s t+1 Represents the state of the reinforcement learning environment at time t+1; the output of the reward model is the reward value r at time t t , the virtual opponent is in state s at time t t The following execution behavior is a t After that, if the state is transferred to s at time t+1 t+1 , then the reward value at time t is r t =r θ (s t ,a t ,s t+1 ).

4. The method for training a behavior strategy model of a virtual opponent according to claim 3, characterized in that: The loss function used in the reward model is expressed as follows: Where loss(·) represents the loss function, θ is the parameter of the loss function; K is the number of optional behaviors; is the expectation function, which represents the expectation of all comparison results, D is the set of comparison results; σ(·) represents the activation function; r θ (s t ,a w ,s t+1 ) represents the reward value corresponding to the better behavior in the comparison result, r θ (s t ,a l ,s t+1 ) represents the reward value corresponding to the worse behavior in the comparison result.

5. The method for training a behavior strategy model of a virtual opponent according to claim 1, characterized in that: The method of obtaining at least two optional behaviors of the virtual opponent in each state based on the behavior strategy model, performing pairwise comparisons on all optional behaviors in each state, and updating the reward model according to the comparison results specifically includes: Input the current state into the behavior strategy model, and sample at least two optional behaviors according to the output of the behavior strategy model; According to the historical behavior style of the designated opponent, all optional behaviors are compared pairwise, and the comparison results are added to the comparison result set; Based on the data in the comparison result set, the reward model is trained using a supervised learning method until the reward model training is completed.

6. The method for training a behavior strategy model of a virtual opponent according to claim 5, characterized in that: Based on the historical behavior style of the designated opponent, all optional behaviors are compared pairwise, and the comparison results are added to the comparison result set, including: Obtain the behavior of the specified opponent in the current state according to the historical behavior style of the specified opponent, and use the obtained behavior as the baseline behavior; Obtain all pairwise combinations of all optional behaviors, compare the two optional behaviors in each combination with the benchmark behavior respectively, and take the optional behavior that is more consistent with the benchmark behavior as the better optional behavior; The comparison result of each combination is added to the comparison result set; wherein the comparison result includes the current state and the superiority and inferiority relationship of the two optional behaviors.

7. The method for training a behavior strategy model of a virtual opponent according to claim 1, characterized in that: According to the output of the behavior strategy model, at least two optional behaviors are sampled, including: According to the output of the behavior strategy model, the probability of the virtual opponent executing each behavior in the current state is obtained; Sampling is performed according to the specified probability distribution and the probability of each behavior, and the behaviors that meet the specified probability distribution are selected as optional behaviors.

8. The method for training a behavior strategy model of a virtual opponent according to claim 1, characterized in that: The method predicts the behavior of the virtual opponent in each state based on the behavior strategy model, obtains the reward value after each behavior is executed using the updated reward model, and trains the behavior strategy model based on the reward value, specifically including: Use the behavior strategy model to obtain the predicted data and reward value in each time step, and calculate the deviation value between the behavior and the historical behavior style of the specified opponent based on the probability of each behavior in the predicted data; Divide all time steps into at least two time periods, and calculate the average reward for each time period based on the reward values ​​of all time steps in each time period; Specify the target reward to be achieved. When the average reward of one or more time periods is less than the target reward, update the behavior strategy model according to the deviation value and reward value of the corresponding time step until the average reward of all time periods is greater than the target reward.

9. A virtual opponent behavior strategy model training device, characterized by: It includes at least one processor and a memory, wherein the at least one processor and the memory are connected via a data bus, and the memory stores instructions that can be executed by the at least one processor, and after the instructions are executed by the processor, they are used to complete the behavior strategy model training method of a virtual opponent described in any one of claims 1-8.

10. A non-volatile computer storage medium, characterized in that: The computer storage medium stores computer executable instructions, which are executed by one or more processors to complete the behavior strategy model training method for a virtual opponent as described in any one of claims 1-8.

Images