A deep reinforcement training method for game AI for clearance rate prediction

Abstract

The present invention discloses a game AI deep reinforcement training method for predicting clearance rate, which relates to the field of machine learning technology, and mainly includes the steps of: obtaining the selected probability of each operation through a first deep training model; obtaining the execution operation in the current game state based on the selected probability of each feasible operation through an intelligent agent in the reinforcement training model; running the game according to the execution operation; obtaining the predicted clearance rate according to the clearance result, and adjusting the selected probability of each feasible operation based on the actual clearance rate and the predicted clearance rate of the game; training the second deep training model according to the adjusted selected probability of each feasible operation, and assigning the trained parameters to the first deep training model. The present invention also takes the player's own uncertainty operation into consideration for the impact on the clearance rate, and adjusts the selected probability of the optional operation based on the actual clearance rate and the predicted clearance rate, so that it is more adapted to the characteristic of the player's operation uncertainty, thereby improving the prediction accuracy of the game clearance rate.

Description

Technical Field

[0001] The present invention relates to the technical field of machine learning methods, and in particular to a game AI deep reinforcement training method for predicting clearance rates. Background Art

[0002] Because the game industry has a huge industrial economy and user base, a large number of companies have begun to develop games, and the competition among game companies is becoming increasingly fierce. Therefore, in order to maintain the market competitiveness of their games, game companies need to continuously launch new game levels. The clearance rate refers to the average probability of all players passing the level, which is an important indicator of new game levels. For different player groups, new levels need to be set to different clearance rates. For example, game levels designed for white-collar workers need to be mainly leisure and entertainment, and a higher clearance rate should be designed; game levels designed for college students need to have a certain degree of challenge, and a lower clearance rate should be designed. Therefore, before the new game level is launched, it is very important to predict the clearance rate of the level. At present, there are two methods for predicting the clearance rate within the enterprise: manual prediction and AI prediction. Manual prediction is achieved by opening the game beta and inviting real players to try it out. This method is time-consuming, labor-intensive and costly. AI prediction uses game AI to simulate players to play the game, and then calculates the clearance rate based on the clearance results of the game AI. This method is low-cost, but the accuracy is not high enough. Since the cost of manual prediction is very high, it is an inevitable trend for AI to replace manual prediction of game clearance rates. Therefore, improving the accuracy of AI in predicting game level completion rates is an urgent problem to be solved.

[0003] Game companies often use traditional rule-based algorithms to design game AI to predict clearance rates. However, the behavior of players is uncertain and cannot be accurately described by rules. This type of game AI cannot accurately simulate the game operations of players, so the prediction accuracy is poor. In order to improve the prediction accuracy, game AI needs to accurately simulate the game operations of players without accurately describing the laws of player behavior. Traditional rule-based algorithms cannot achieve this goal. Summary of the invention

[0004] In order to better improve the accuracy of game AI in predicting clearance rate, and fully consider the global impact of a small number of unconventional operations on game AI prediction, the present invention proposes a game AI deep reinforcement training method for clearance rate prediction, comprising the steps of:

[0005] S1: Determine whether the number of game runs reaches the preset number. If so, proceed to step S7. If not, run the game and enter step S2 after accumulating the number of runs.

[0006] S2: Obtain point pair information consisting of a feasible operation set in the current game state and feature information corresponding to each operation;

[0007] S3: Based on the game's own decision rules and point pair information, the probability of each operation being selected is obtained through the first deep training model;

[0008] S4: The agent in the reinforcement training model obtains the execution operation under the current game state based on the selection probability of each feasible operation;

[0009] S5: Run the game according to the execution operation and record the game record in the current game state;

[0010] S6: Determine whether the current game is over. If so, obtain the game clearance result and return to step S1. If not, return to step S2.

[0011] S7: Obtain a predicted clearance rate according to the clearance result, and adjust the probability of selection of each feasible operation based on the actual clearance rate of the game and the predicted clearance rate;

[0012] S8: Train the second deep training model according to the adjusted selection probability of each feasible operation, and assign the trained parameters to the first deep training model.

[0013] Furthermore, in the step S2, the point pair information includes historical execution operation selection records of feasible operations and selection records corresponding to the feasible operations, and the point pair information is processed through OneHot encoding to obtain encoding information that is independent of each other between the selection records.

[0014] Furthermore, in the step S4, the acquisition of the execution operation is obtained by the following judgment:

[0015] Accumulate the selection probabilities of the corresponding feasible operations one by one according to the ranking of the feasible operations in the feasible operation set;

[0016] Determine whether the ratio between the probability of being selected in the current accumulation state and the sum of the probabilities of being selected of all feasible operations is greater than the first random value. If so, output the currently sorted feasible operation as the execution operation. If not, continue to accumulate the feasible operations one by one according to their order in the feasible operation set.

[0017] The first random value ranges from 0 to 1.

[0018] Furthermore, in step S7, the adjustment of the probability of each feasible operation being selected is obtained by the following determination:

[0019]

[0020] In the formula, p new is the probability of being selected after the feasible operation is adjusted, p is the probability of being selected before the feasible operation is adjusted, PR t is the actual clearance rate of the game, PR eis the predicted clearance rate of the game, W is the clearance result, when W=0 it means the clearance fails, when W=1 it means the clearance succeeds.

[0021] Furthermore, in the step S8, the training of the second deep training model specifically includes the steps of:

[0022] According to the weight ratio of the game records, a preset number of game records are selected from the game record set obtained from the preset number of game runs as the training sample set;

[0023] Update parameters of the second deep training model based on the training sample set.

[0024] Furthermore, the step S5 further includes the following steps:

[0025] Set the weight of the selected operation according to the order in which the operation is recorded in the game record;

[0026] The selected weight of the execution operation increases as the number of game records increases.

[0027] Furthermore, the selection of the training sample set can be obtained by the following judgment:

[0028] Calculate the total weight of the game record according to the selected weight of each execution operation in the game record set;

[0029] According to the order of the execution operations in the game record set, the selected weights of the execution operations are accumulated one by one and the cumulative weight of the game record in the current accumulation state is obtained;

[0030] Update the selected weight of the corresponding execution operation according to the ratio of the accumulated weight of the game record in the current accumulation state to the total weight of the game record;

[0031] According to the order of the execution operations in the game record set, determine one by one whether the selected weight of the execution operation after the update is greater than the second random value. If so, add the execution operation to the training sample set. If not, determine the execution operation of the next order until the sample training set reaches the target sample size.

[0032] The second random value ranges from 0 to 1.

[0033] Furthermore, the parameter update of the second deep training model is represented by the following steps:

[0034] Determine whether the difference between the loss function value of the second deep training model in the current iteration and the previous iteration is greater than the loss change threshold. If so, proceed to the next step before the iteration reaches the maximum number of iterations.

[0035] Update the loss function value by presetting the loss function;

[0036] Update the parameters according to the updated loss function value and adjust the learning rate based on the learning decay rate.

[0037] Furthermore, the preset loss function can be expressed as the following formula:

[0038] l(f(x i ),y i )=‖f(x i )-y i ‖2

[0039]

[0040] In the formula, x i is the i-th sample input in the training sample set, y i is the point pair information corresponding to the i-th sample in the training sample set, f(x i ) is the clearance result of the i-th sample in the training sample set, L is the updated loss function value, and n is the number of samples in the training sample set;

[0041] The update of the parameters is expressed as the following formula:

[0042]

[0043] In the formula, ω is the updated parameter, ω0 is the parameter before updating, and η0 is the learning rate before adjustment;

[0044] The adjustment of the learning rate is expressed as the following formula:

[0045] η=λη0

[0046] Where η is the adjusted learning rate and λ is the learning decay rate.

[0047] Furthermore, the step S8 further includes the following steps:

[0048] S9: Determine whether the training is finished. If not, reset the number of runs and game records, and return to step S1.

[0049] In summary, compared with the prior art, the present invention has at least the following beneficial effects:

[0050] (1) The deep reinforcement training method for game AI for predicting clearance rate described in the present invention combines deep learning (DL) and reinforcement learning (RL). Compared with the existing game AI which often only considers local conditions when predicting the clearance rate of a game, the present invention also considers the influence of the player's own uncertain operation on the clearance rate, thereby obtaining a global optimal solution;

[0051] (2) OneHot encoding is used to process the point-to-point information during the game operation process, so that the feasible operations in the same game state and different game states can exist in the form of independent encoding, reducing the similarity and thus improving the accuracy of training;

[0052] (3) By analyzing all feasible operations in the current game state separately, the selection probability of the optional operation is mapped to a relative probability interval by the sum of the selection probabilities and the cumulative probability of the currently selected operation, and then the final execution operation is selected, rather than selecting the execution operation based solely on the size of the selection probability, so that the selected execution operation is more accurate;

[0053] (4) Compared with the existing game AI, which simply maps the probability of selection of optional operations to 0 or 1 to reduce the loss function value, although it can make more accurate predictions of the player's operations, it lacks global considerations. The present invention uses deep learning to adjust the probability of selection of optional operations based on the actual clearance rate and the predicted clearance rate, rather than simply setting the probability of selection to 0 or 1, so that it is more adaptable to the uncertainty of player operations, thereby improving the accuracy of predicting the game clearance rate;

[0054] (5) Training samples are selected based on the total weight of the game records and the cumulative weight of the current game records, thereby ensuring the training value of the samples while reducing the large amount of processing caused by too many samples. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 A method step diagram of a deep reinforcement training method for game AI for clearance rate prediction;

[0056] Figure 2 Simplified block diagram for deep reinforcement training. DETAILED DESCRIPTION

[0057] The following are specific embodiments of the present invention and the accompanying drawings to further describe the technical solution of the present invention, but the present invention is not limited to these embodiments.

[0058] Embodiment 1

[0059] There are two aspects to simulating games and predicting clearance rates through game AI: The first difficulty is that the training of game AI is different from general deep learning (DL). There is no unified standard for its feature information. It is necessary to extract data features according to the characteristics of the game to build a suitable sample set for training. For example, in the popular fields of deep learning: natural language, face recognition, image processing, etc., the sample sets in these fields have clear standards. The input is either a picture or a sentence, and the output is a classification. Research in these fields can be carried out according to this unified standard, and scientific researchers can learn from each other in their innovative improvements to the sample set. However, in the field of games, due to the inconsistency of various game designs, it is impossible to use a unified sample set for training. It is only possible to extract data features according to the characteristics of the game to build a suitable sample set for training. The second difficulty is that even if the model can accurately simulate the player's operation, it may not be able to accurately predict the clearance rate of the level. There are two possible reasons for this problem: The first reason is that the operations will interfere with each other and jointly determine the outcome of the game. Assuming that the game has 100 steps, the model can accurately simulate 99 kinds of processing logic of the player, that is, it has achieved 99% accuracy. However, even if there is only one step difference between the model and the real player in operation, it may lead to different wins and losses for the two at this game level; the second reason is that the player's operations are diverse and coherent. For example, 100 players may make different decisions facing the same game state, and players often consider what to do next when making these decisions. But the model does not think like the player, it will simply map out the probability of each operation according to the current state, and then choose an operation to execute based on the probability, which will undoubtedly affect the prediction accuracy of the game AI.

[0060] In response to the first difficulty, the present invention needs to analyze the characteristics of the game to construct a suitable sample set; in response to the second difficulty, the present invention believes that the game AI with a clearance rate consistent with that of real players should be trained with the help of reinforcement learning (RL), rather than simply fitting the player's behavior. However, it is difficult to use RL to train such a game AI. Because RL is mainly trained through the interaction between the agent and the environment, the agent can interact with the environment through learning strategies. During the interaction, the agent will learn the feedback given by the environment to maximize the reward or achieve a specific goal. The goal of RL training game AI is to make the AI's gaming ability stronger, and it is difficult to train a game AI with a specific clearance rate. However, the goal of the present invention is not to make the game AI always stronger, but to control its strength so that it is trained towards a specific clearance rate, and the clearance rate of the model is consistent with that of the real player when it finally converges. Based on the above background and conception, the present invention proposes a deep reinforcement training method for game AI for clearance rate prediction, and uses this method to train a deep reinforcement learning (DRL) model so that the model prediction rate converges to the target value, rather than simply strengthening the model's clearance ability, such as Figure 1As shown, it mainly includes the following steps:

[0061] S1: Determine whether the number of game runs reaches the preset number. If so, proceed to step S7. If not, run the game and enter step S2 after accumulating the number of runs.

[0062] S2: Obtain point pair information consisting of a feasible operation set in the current game state and feature information corresponding to each operation;

[0063] S3: Based on the game's own decision rules and point pair information, the probability of each operation being selected is obtained through the first deep training model;

[0064] S4: The agent in the reinforcement training model obtains the execution operation under the current game state based on the selection probability of each feasible operation;

[0065] S5: Run the game according to the execution operation and record the game record in the current game state;

[0066] S6: Determine whether the current game is over. If so, obtain the game clearance result and return to step S1. If not, return to step S2.

[0067] S7: Obtain a predicted clearance rate according to the clearance result, and adjust the probability of selection of each feasible operation based on the actual clearance rate of the game and the predicted clearance rate;

[0068] S8: training the second deep training model according to the adjusted selection probabilities of each feasible operation, and assigning the trained parameters to the first deep training model;

[0069] S9: Determine whether the training is finished. If not, reset the number of runs and game records, and return to step S1.

[0070] The point pair information includes historical execution operation selection records of feasible operations and selection records of corresponding feasible operations.

[0071] Since the state of the game is changed according to the operation selection of the player, and under each game state, the number and position of the operable points and inoperable points that can be selected by the player are different, and at the same time, the feasible operation amount at each operable point is also different. Therefore, although the two codes 234 and 235 (assuming that the first two bits are point information and the third bit is operation selection information) can be used to represent two different feasible operations at the same operable point, but in the model, they are very similar, if only the input data has only this value difference, then the obtained mapping value is not much different. But in the actual game, 234 and 235 are two feasible operations, they are completely independent, and the deep learning model should be allowed to have such a cognition, 1234 (assuming that the first three bits are point information and the fourth bit is operation selection information) and 234 are the same as 234 and 235, and they are all independent of each other. Therefore, in order to allow the model to better identify the game state, the present invention processes the selection of feasible operations under each game state by introducing OneHot coding technology, and the selection of operable points, inoperable points, feasible operations and infeasible operations is independently encoded. In other words, the feasible operation options at the corresponding positions of each historical operation point are set to 1, and the points at the positions of other non-historical operation points are set to 0.

[0072] When the game AI obtains the encoded data of the information of each point in the current game state (that is, the game information), it forms a corresponding feature information matrix. Then the game AI can analyze the game state and calculate the probability of selection of each feasible operation in the feasible operation set according to the pre-set decision rules of the game (the decision rules of each game are different, so they will not be described in detail here).

[0073] In the game industry, game AI that predicts the clearance rate often uses traditional rule-based algorithms. Since this type of game AI does not need to solve the best solution to clear the game, it is often implemented using a simple and fast greedy algorithm. When solving a problem, the greedy algorithm always starts from local interests and considers the best option at the moment. It is carried out step by step, often based on the current local state to make the best choice according to a certain measurement dimension, without considering all possibilities, reducing the large amount of time cost of exhausting all possibilities. It is a top-down algorithm that performs greedy selection step by step in an iterative method. Each time a greedy selection is made, the problem becomes a smaller sub-problem. It always considers local situations, so the final solution is often not the global optimal. Therefore, when faced with multiple feasible operations, the greedy algorithm often selects the feasible operation with the highest probability of being selected as the execution operation in the current game state, adjusts its selection probability to 1, and adjusts the selection probability of other feasible operations to 0.

[0074] However, the main problem of using greedy algorithms to predict the level clearance rate of game AI is that the prediction error is too large and the prediction result is unreliable. This is because the greedy algorithm does not accurately simulate the player's operation. This rule-based algorithm is difficult to accurately simulate the operation of real players. Real players usually have many considerations in the process of playing games. For example, players may think that this step is not optimal, but to build a bridge for the next step to make the next step take a better operation; players will weigh whether to eliminate ice cubes first or get higher points first. Therefore, it is difficult to accurately describe the thinking of real players with rules, and rule-based algorithms cannot accurately simulate the player's operation. In order to solve this problem, the present invention proposes to train game AI to predict the clearance rate by machine learning instead of rule-based algorithms. In this embodiment, deep learning (DL) in machine learning (ML) is selected, and it is trained by convolutional neural network (CNN) based on the rules of the game itself and point pair information to obtain the probability of selection of each optional operation. However, in actual application and experimental process, when selecting the execution operation by GAI-CNN (CNN-based game AI), there are still the following problems:

[0075] 1.GAI-CNN extracts the player’s operation as 1 (the player selected this operation) and 0 (the player did not select this operation) during feature processing.

[0076] 2. The game has recorded the actual clearance rate of the training levels, but this data is not used for GAI-CNN training.

[0077] 3. During the prediction experiment, the prediction error of GAI-CNN is only used for performance statistics and cannot be used for GAI-CNN training.

[0078] Based on the above three problems, the present invention does not select the execution operation through GAI-CNN, but only uses it to preliminarily obtain the selection probability of each feasible operation.

[0079] The actual screening method used in the present invention for executing operations is to use the S3 step to obtain the selection probability of each feasible operation, and then analyze all feasible operations in the current game state separately through the game AI. Through reinforcement learning (RL), the selection probability of each feasible operation can be mapped to a relative probability interval based on the sum of the selection probabilities of each feasible operation and the cumulative probability of the currently selected operation, so as to select the execution operation from all feasible operations. The specific selection process can be expressed as the following judgment process:

[0080] Accumulate the selection probabilities of the corresponding feasible operations one by one according to the ranking of the feasible operations in the feasible operation set;

[0081] Determine whether the ratio of the selected probability under the current accumulation state to the sum of the selected probabilities of all feasible operations is greater than the first random value. If so, output the currently sorted feasible operation as the execution operation. If not, continue to accumulate the feasible operations one by one according to their order in the feasible operation set.

[0082] The first random value ranges from 0 to 1.

[0083] If the above judgment process is expressed in the form of pseudo code, it can be expressed as:

[0084] Input: O, P, n;

[0085] Output:o

[0086]

[0087]

[0088] Among them, r1 is the first random value, sumP is the sum of the selection probabilities, p is the cumulative selection probability of the current feasible operation, O is the feasible operation set under the current game state, P is the selection probability corresponding to the feasible operation, n is the number of feasible operations, and o is the execution operation finally selected from the feasible operation set.

[0089] Based on the pseudocode, the execution operation is selected. For example, the feasible operation set O only includes two feasible operations, namely, the feasible operation O1 and the feasible operation O2. The selection probability of O1 is 0.4, and the selection probability of O2 is 0.1. After processing sumP and p in the pseudocode, the selection probability of O1 becomes the interval [0,0.8], and the selection probability of O2 becomes the interval (0.8,1]. Finally, a random r1 is used to determine the selected feasible operation.

[0090] Then, after running the game according to the execution operation and recording the game record in the current game state, before the current game ends, return to step S2 to obtain the point pair information. It should be noted that the point pair information in step S2 of the present invention is based on the OneHot encoding conversion performed by the intelligent agent in reinforcement learning (RL).

[0091] Among them, RL is a hot research direction of ML. The important components of RL are the agent and the environment. The agent interacts with the environment through a "trial and error" method based on the Q table, where the Q table stores the agent's cognitive information about the current environment. Cognitive information refers to the estimated value of the agent's choice of a certain behavior in the current environment. The agent interacts with the environment, updates the estimated value of the Q table based on the evaluation of the environmental feedback, and gradually adapts to the environment.

[0092] The agent will estimate the results of its own interactive behavior and choose the behavior with the best result to interact with the environment. The environment will give rewards or penalties based on the impact of the agent's behavior. In order to obtain the maximum reward, the agent will adjust its estimated results of the behavior based on the feedback given by the learning strategy learning environment. For example, in the value-based RL model, the agent will estimate the value of its own behavior and update this estimated value based on the feedback from the environment. The agent always chooses the behavior with the highest value to interact with the environment. Repeating this process continuously, the agent's estimate will gradually approach the feedback result of the environment. This cyclical process is the learning process of RL. Unlike DL, the training process of RL requires the assistance of reinforcement signals (RS). RS refers to the reward signal or punishment signal (this signal is generally a scalar) given by the environment to the quality of the behavior selected by the agent. RL will not directly tell the agent what behavior is correct, but will generate RS to tell the agent whether the behavior it chooses is good or bad. The agent can learn from its own or others' experience without providing a lot of additional information. In this way, the agent acquires knowledge in the cycle of selecting behavior-environment scoring results, and improves the scheme of selecting behavior to adapt to the environment.

[0093] However, RL has great limitations. Since the Q table needs to be stored in the form of an array, RL can only be used to solve small-scale discrete problems. It cannot solve problems with too large behavior space or state space.

[0094] To address this problem, the present invention combines deep learning and reinforcement learning, that is, adopts deep reinforcement learning (DRL) to complement the advantages of the two. DL is used to update the Q table in RL (in this article, the probability of being selected before the feasible operation is updated, that is, the initial probability of being selected mentioned above), and by setting a preset number of times, the operation of the game is continuously simulated to execute the trial and error process in RL. However, the application of DRL in the prior art still has the problem of being unable to evaluate the current model performance and only being able to improve the model performance.

[0095] In view of the problems existing in DRL and GAI-CNN, the present invention proposes, based on the method for solving problem 1 above, that after running a game, the probability of selection of each feasible operation is adjusted based on the actual clearance rate through step S7. While further solving problem 1, the game AI is adjusted by reasonably using the information of the actual clearance rate of the game, so that the model performance can be adjusted according to the actual clearance rate. Specifically, the adjustment of the probability of selection of each feasible operation is obtained by the following judgment:

[0096]

[0097] In the formula, p newis the probability of being selected after the feasible operation is adjusted, p is the probability of being selected before the feasible operation is adjusted, PR t is the actual clearance rate of the game, PR e is the predicted clearance rate of the game, W is the clearance result, when W=0 it means the clearance fails, when W=1 it means the clearance succeeds.

[0098] In order to apply the predicted error to the training of GAI-CNN and improve the applicability of DRL, in step S8, the parameter update of the second deep training model is based on the difference between the loss change threshold and the loss function value of the model before and after iteration. Specifically, it can be expressed by the following judgment:

[0099] Determine whether the difference between the loss function value of the second deep training model in the current iteration and the previous iteration is greater than the loss change threshold. If so, proceed to the next step before the iteration reaches the maximum number of iterations.

[0100] Update the loss function value by using the preset loss function;

[0101] Update the parameters according to the updated loss function value and adjust the learning rate based on the learning decay rate.

[0102] The preset loss function can be expressed as the following formula:

[0103] l(f(x i ),y i )=‖f(x i )-y i ‖2

[0104]

[0105] In the formula, x i is the i-th sample input in the training sample set, y i is the point pair information corresponding to the i-th sample in the training sample set, f(x i ) is the clearance result of the i-th sample in the training sample set, L is the updated loss function value, and n is the number of samples in the training sample set;

[0106] The parameter update is expressed as follows:

[0107]

[0108] In the formula, ω is the updated parameter, ω0 is the parameter before updating, and η0 is the learning rate before adjustment;

[0109] The adjustment of learning rate is expressed as follows:

[0110] η=λη0

[0111] Where η is the adjusted learning rate and λ is the learning decay rate.

[0112] Furthermore, as the intelligent agent in RL continues to make trial and error, a large number of training samples will be generated. In order to allow the RL model to perform targeted training, the game records in the game record set need to be screened. Considering that new samples are generated by the recent trial and error of the intelligent agent, they are more valuable for training than old samples. Therefore, when entering the game records, it is also necessary to set the corresponding weight information so that more valuable training samples can be screened out according to the weights later. Therefore, the following steps are also included in step S5:

[0113] Set the weight of the selected operation according to the order in which the operation is recorded in the game record;

[0114] The selected weight of the execution operation increases as the number of game records increases.

[0115] The specific operation of selecting the training sample set from the game record set can be obtained by the following judgment:

[0116] Calculate the total weight of the game record according to the selected weight of each execution operation in the game record set;

[0117] According to the order of the execution operations in the game record set, the selected weights of the execution operations are accumulated one by one and the cumulative weight of the game record in the current accumulation state is obtained;

[0118] Update the selected weight of the corresponding execution operation according to the ratio of the accumulated weight of the game record in the current accumulation state to the total weight of the game record;

[0119] According to the order of the execution operations in the game record set, determine one by one whether the selected weight of the execution operation after the update is greater than the second random value. If so, add the execution operation to the training sample set. If not, determine the execution operation of the next order until the sample training set reaches the target sample size.

[0120] The second random value ranges from 0 to 1.

[0121] If the above training sample screening process is expressed in the form of pseudo code, it can be expressed as:

[0122] Input:n,D,W,N

[0123] Output:D select

[0124]

[0125] Where n is the size of the training sample set to be generated, N is the size of the game record, D is the game record set, W is the selected weight of each sample, sumW is the total weight of the game record, w is the cumulative weight of the current game record, r2 is a random number, and D select is the selected training sample set.

[0126] The above content is a relatively complete description of the technical extension ideas of this application, which describes the problems existing in the prior art as a whole, and proposes improvement plans for the problems, as well as the defects improved by the improvement plans (the known principles of the prior art adopted by the improvement plans will not be described in detail in this article). In order to better understand the technical points of the present invention, this embodiment illustrates the present invention through a simplified block diagram, as shown in FIG. Figure 2 As shown in the figure, DRL performs the process of game AI training, which will continue unless it is stopped manually. The training process is divided into two independent sub-processes, involving the first deep training model A and the second deep training model B and a reinforcement training model C. The first sub-process is the trial and error process of the agent in C. The process as a whole is that the agent plays 100 rounds of simulated games, and predicts the clearance rate based on the game situation of the simulated game, and generates the training samples required for A training (corresponding to steps S1 to S6). In this process, the game program passes the game state and the optional operation set to C, and C generates multiple groups of sample inputs according to the OneHot encoding method above. Each group of samples includes the current game state (including the selection record of historical execution operations) and the feature information of a feasible operation; then C passes each group of samples to model A for calculation in turn, and after obtaining the corresponding probability of selection of each feasible operation, C selects an optional operation from multiple optional operations and feeds it back to the game program. Repeating this process, the agent will complete 100 games, record all operations and selection probabilities of each game, mark the clearance results and weight c, and then adjust the selection probabilities based on the actual clearance rate to generate training samples. The second sub-process is the training of model B, which will only start when the last training of model B is completed. First, it will copy the parameters of model B to model A, which is equivalent to updating the Q table of C. Finally, it will continue training based on D based on the current model B. After the training is completed, the predicted clearance rate is output to the developer.

[0127] In summary, the game AI deep reinforcement training method for clearance rate prediction described in the present invention, through the combination of deep learning (DL) and reinforcement learning (RL), compared with the existing game AI that often only considers local conditions when predicting the game clearance rate, the present invention also takes the player's own uncertain operations into account The impact on the clearance rate, thereby obtaining a global optimal solution.

[0128] OneHot encoding is used to process point-to-point information during game operations, so that feasible operations in the same game state and different game states can exist in the form of independent encoding, reducing similarities and thus improving training accuracy.

[0129] By analyzing all feasible operations in the current game state separately, the selection probability of the optional operation is mapped to a relative probability interval by the sum of the selection probabilities and the cumulative probability of the currently selected operation, and then the final execution operation is selected, rather than just selecting the execution operation based on the size of the selection probability, so that the selected execution operation is more accurate.

[0130] Compared with the existing game AI, which simply maps the selection probability of optional operations to 0 or 1 to reduce the loss function value, although it can make more accurate predictions on the player's operations, it lacks global considerations. The present invention uses deep learning to adjust the selection probability of optional operations based on the actual clearance rate and the predicted clearance rate, rather than simply setting the selection probability to 0 or 1, so that it is more adaptable to the uncertainty of player operations, thereby improving the accuracy of predicting the game clearance rate.

[0131] The training samples are selected based on the total weight of the game records and the cumulative weight of the current game records, which ensures the training value of the samples while reducing the large amount of processing caused by too many samples.

[0132] It should be noted that all directional indications in the embodiments of the present invention (such as up, down, left, right, front, back, etc.) are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0133] In addition, in the present invention, descriptions such as "first", "second", "one", etc. are only used for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0134] In the present invention, unless otherwise clearly specified and limited, the terms "connection", "fixation", etc. should be understood in a broad sense. For example, "fixation" can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements, unless otherwise clearly defined. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0135] In addition, the technical solutions between the various embodiments of the present invention can be combined with each other, but it must be based on the fact that ordinary technicians in the field can implement it. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

Claims

1. A game AI deep reinforcement training method for clearance rate prediction, characterized in that: Includes steps: S1: Determine whether the number of game runs reaches the preset number. If so, proceed to step S7. If not, run the game and enter step S2 after accumulating the number of runs. S2: Obtain point pair information consisting of a feasible operation set in the current game state and feature information corresponding to each operation; S3: Based on the game's own decision rules and point pair information, the probability of each operation being selected is obtained through the first deep training model; S4: The agent in the reinforcement training model obtains the execution operation under the current game state based on the selection probability of each feasible operation; S5: Run the game according to the execution operation and record the game record in the current game state; S6: Determine whether the current game is over. If so, obtain the game clearance result and return to step S1. If not, return to step S2. S7: Obtain a predicted clearance rate according to the clearance result, and adjust the probability of selection of each feasible operation based on the actual clearance rate of the game and the predicted clearance rate; S8: training the second deep training model according to the adjusted selection probabilities of each feasible operation, and assigning the trained parameters to the first deep training model; The second deep training model has the same model configuration as the first deep training model, but the parameters are independent; In step S8, the training of the second deep training model specifically includes the following steps: According to the weight ratio of the game records, a preset number of game records are selected from the game record set obtained from the preset number of game runs as the training sample set; Updating parameters of a second deep training model based on the training sample set; The parameter update of the second deep training model is represented by the following steps: Determine whether the difference between the loss function value of the second deep training model in the current iteration and the previous iteration is greater than the loss change threshold. If so, proceed to the next step before the iteration reaches the maximum number of iterations. Update the loss function value by presetting the loss function; Update the parameters according to the updated loss function value and adjust the learning rate based on the learning decay rate.

2. A method for in-depth reinforcement training of game AI for clearance rate prediction according to claim 1, characterized in that: In the step S2, the point pair information includes the historical execution operation selection records of the feasible operations and the selection records of the corresponding feasible operations, and the point pair information is processed through OneHot encoding to obtain encoding information that is independent of each other between the selection records.

3. A method for deep reinforcement training of game AI for clearance rate prediction according to claim 1, characterized in that: In the step S4, the acquisition of the execution operation is obtained by the following judgment: Accumulate the selection probabilities of the corresponding feasible operations one by one according to the ranking of the feasible operations in the feasible operation set; Determine whether the ratio between the probability of being selected in the current accumulation state and the sum of the probabilities of being selected of all feasible operations is greater than the first random value. If so, output the currently sorted feasible operation as the execution operation. If not, continue to accumulate the feasible operations one by one according to their order in the feasible operation set. The first random value ranges from 0 to 1.

4. A method for deep reinforcement training of game AI for clearance rate prediction according to claim 1, characterized in that: In step S7, the adjustment of the probability of each feasible operation being selected is obtained by the following judgment: In the formula, The probability of being selected after adjustment for feasible operations, is the probability of being selected before the feasible operation adjustment, is the actual clearance rate of the game, is the predicted clearance rate of the game, W is the clearance result, when W=0 it means the clearance fails, when W=1 it means the clearance succeeds.

5. A method for in-depth reinforcement training of game AI for clearance rate prediction according to claim 1, characterized in that: The step S5 further includes the following steps: Set the weight of the selected operation according to the order in which the operation is recorded in the game record; The selected weight of the execution operation increases as the number of game records increases.

6. A method for deep reinforcement training of game AI for clearance rate prediction according to claim 5, characterized in that: The selection of the training sample set can be obtained by the following judgment: Calculate the total weight of the game record according to the selected weight of each execution operation in the game record set; According to the order of the execution operations in the game record set, the selected weights of the execution operations are accumulated one by one and the cumulative weight of the game record in the current accumulation state is obtained; Update the selected weight of the corresponding execution operation according to the ratio of the accumulated weight of the game record in the current accumulation state to the total weight of the game record; According to the order of the execution operations in the game record set, determine one by one whether the selected weight of the execution operation after the update is greater than the second random value. If so, add the execution operation to the training sample set. If not, determine the execution operation of the next order until the sample training set reaches the target sample size. The second random value ranges from 0 to 1.

7. A method for deep reinforcement training of game AI for clearance rate prediction according to claim 1, characterized in that: The preset loss function can be expressed as the following formula: In the formula, is the i-th sample input in the training sample set, is the point pair information corresponding to the i-th sample in the training sample set, is the clearance result of the i-th sample in the training sample set, L is the updated loss function value, and n is the number of samples in the training sample set; The update of the parameters is expressed as the following formula: In the formula, is the updated parameter, is the parameter before updating, is the learning rate before adjustment; The adjustment of the learning rate is expressed as the following formula: In the formula, is the adjusted learning rate, is the learning decay rate.

8. A method for deep reinforcement training of game AI for clearance rate prediction according to claim 1, characterized in that: The step S8 further includes the following steps: S9: Determine whether the training is finished. If not, reset the number of runs and game records, and return to step S1.

Images