Game robot training method and device, storage medium and computer equipment
By filtering and executing the target actions selected by game status information and calculating the total return value, the problem of single action of game robots in the prior art is solved, and the training effect of action diversity is achieved.
Patent Information
- Application Number
- CN202311149293.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-09-06
- Publication Date
- 2025-03-07
AI Technical Summary
When the prior art trains game robots through reinforcement learning methods, the action benefits are complex and the benefits are not high in the early stage of training, resulting in a single action of the game robot.
By obtaining game status information, filtering target actions, controlling the game robot to perform actions, and calculating the total return value based on the status information after the action, we encourage the game robot to explore actions with low returns.
The movement diversity of game robots is achieved and the problem of single action is avoided.
Smart Images
Figure CN119565146A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computers, and in particular to a method and apparatus for training a gaming robot, a computer-readable storage medium, and a computer device. Background Art
[0002] In recent years, with the development and popularization of computer equipment technology, more and more game applications with three-dimensional virtual environments have emerged, such as first-person shooting game (FPS), role-playing game (RPG) and massively multiplayer online role-playing game (MMORPG).
[0003] In the prior art, taking MMORPG games as an example, there are a player versus environment (PVP) mode and a player versus environment (PVE) mode. When a user cannot match other users in the PVP mode or in the PVE mode, a game robot will be controlled to fight against the user.
[0004] In the process of researching and practicing the prior art, the inventor of the present application found that when the prior art uses the reinforcement learning method to train a game robot, when the way to obtain benefits for a certain action is relatively complicated, and the character does not gain much benefit from exploring this action in the early stage of training, it is easy to cause the problem of not exploring this action, resulting in the trained game robot having a single action. Summary of the invention
[0005] The embodiments of the present application provide a method and device for training a gaming robot, which can encourage the gaming robot to explore actions with low returns, thereby achieving action diversity for the trained gaming robot.
[0006] To solve the above technical problems, the present application provides the following technical solutions:
[0007] A method for training a game robot, applied to a computer device, comprising:
[0008] Acquire first game state information, and select a target action from a plurality of preset actions based on the first game state information;
[0009] Controlling the game robot to execute the target action, and obtaining second game state information after executing the target action;
[0010] Determine a game state reward value based on the first game state information and the second game state information;
[0011] Determine whether the game robot performs the target action within a historical time period, and obtain a determination result;
[0012] Calculating a total reward value of the target action based on the game state reward value and the determination result;
[0013] The game robot is trained based on the total reward value.
[0014] A training device for a gaming robot, comprising:
[0015] A screening module, used to obtain first game state information, and screen a target action from a plurality of preset actions based on the first game state information;
[0016] An acquisition module, used to control the game robot to perform the target action and obtain second game state information after performing the target action;
[0017] A first determination module, configured to determine a game state reward value based on the first game state information and the second game state information;
[0018] A second determination module is used to determine whether the game robot performs the target action within a historical time period to obtain a determination result;
[0019] A calculation module, used for calculating the total reward value of the target action based on the game state reward value and the determination result;
[0020] A training module is used to train the game robot based on the total reward value.
[0021] A computer-readable storage medium stores a plurality of instructions suitable for loading by a processor to execute the steps in the above-mentioned method for training a gaming robot.
[0022] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps in the above-mentioned method for training a gaming robot when executing the program.
[0023] The embodiment of the present application obtains first game state information, and based on the first game state information, selects a target action from multiple preset actions; controls the game robot to execute the target action, and obtains second game state information after executing the target action; determines the game state reward value based on the first game state information and the second game state information; determines whether the game robot executes the target action within a historical time period and obtains a determination result; calculates the total reward value of the target action based on the game state reward value and the determination result; and trains the game robot based on the total reward value. In this way, the total reward value of the target action is affected by determining the determination result of whether the game robot executes the target action within a historical time period, thereby encouraging the game robot to explore actions with low returns through the total reward value, thereby achieving action diversity of the trained game robot. BRIEF DESCRIPTION OF THE DRAWINGS
[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.
[0025] Figure 1a A system diagram of a method for training a gaming robot provided in an embodiment of the present application.
[0026] Figure 1b A flowchart of a method for training a gaming robot provided in an embodiment of the present application.
[0027] Figure 2 A schematic diagram of the structure of a training device for a gaming robot provided in an embodiment of the present application;
[0028] Figure 3 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0029] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.
[0030] Embodiments of the present application provide a method, device, and computer-readable storage medium for training a gaming robot.
[0031] See also Figure 1a , Figure 1a A system diagram of a training system for a game robot provided in an embodiment of the present application, the system may include at least one game server 1000, at least one computer device 2000, at least one database 3000, and a network 4000. A game application is installed in the game server 1000, and the application corresponds to at least one computer device 2000: the game server 1000 may be a background server corresponding to a game application installed in a client device such as a mobile phone, a computer or a personal digital assistant, and the game server 1000 may be connected to a computer device 2000 of the application, or multiple computer devices 2000 of the application through a network 4000. Among them, the network 4000 may be a wireless network or a wired network, such as a wireless network such as a wireless local area network (WLAN), a local area network (LAN), a cellular network, a 2G network, a 3G network, a 4G network, a 5G network, etc. In addition, the computer devices 2000 may also be connected to each other using their own Bluetooth network or hotspot network. In addition, the system may include multiple databases 3000, which may be databases integrated in the computer device 2000, or third-party databases that establish a communication connection with the computer device 2000 through a protocol, and may synchronize the game status information in the game server 1000 in real time and store it in the database 3000 for the computer device 2000 to obtain it.
[0032] The present application embodiment provides a method for training a game robot, which can be executed by a computer device. Figure 1a As shown, the computer device 2000 obtains first game state information, and based on the first game state information, selects a target action from multiple preset actions; controls the game robot to execute the target action, and obtains second game state information after executing the target action; determines the game state reward value based on the first game state information and the second game state information; determines whether the game robot executes the target action within a historical time period, and obtains a determination result; calculates the total reward value of the target action based on the game state reward value and the determination result; and trains the game robot based on the total reward value. In this way, the total reward value of the target action is affected by determining the determination result of whether the game robot executes the target action within a historical time period, so that the game robot is encouraged to explore actions with low returns through the total reward value, thereby achieving action diversity of the trained game robot.
[0033] It should be noted that Figure 1aThe scenario diagram of the game robot training system shown is merely an example. The game robot training system and scenario described in the embodiment of the present application are intended to more clearly illustrate the technical solution of the embodiment of the present application, and do not constitute a limitation on the technical solution provided in the embodiment of the present application. A person of ordinary skill in the art can appreciate that with the evolution of the game robot training system and the emergence of new business scenarios, the technical solution provided in the embodiment of the present application is equally applicable to similar technical problems.
[0034] In this embodiment, the training device for the gaming robot will be described from the perspective of the training device for the gaming robot. The training device for the gaming robot can be specifically integrated into a computer device having a storage unit and a microprocessor installed therein and having computing capabilities.
[0035] See also Figure 1b , Figure 1b A flowchart of a method for training a gaming robot provided in an embodiment of the present application. The method for training a gaming robot includes:
[0036] In step 101, first game state information is obtained, and a target action is selected from a plurality of preset actions based on the first game state information.
[0037] Among them, in order to train the game robot, the present application builds a virtual scene in the game server and sets up at least one game robot in the virtual scene. The developer or user controls another virtual object and the game robot, or sets up another game robot and the game robot to attack each other, thereby collecting the game status information obtained by the game server.
[0038] Specifically, the game status information may include the game robot health value of the game robot to be trained, the coordinate information of its position in the virtual scene, the profession of the game robot in the game (for example, the game includes multiple professions such as swordsman and swordsman, and the game robot belongs to the swordsman), skill release status and skill cooling status (for example, skill A has 5 seconds to be used), and other information; as well as the health value of the attack object attacked by the game robot, the coordinate information of its position in the virtual scene, the profession of the game robot in the game (for example, the game includes multiple professions such as swordsman and swordsman, and the game robot belongs to the swordsman), skill release status and skill cooling status (for example, skill A has 5 seconds to be used), and other information.
[0039] Taking the example that the game status information includes the game robot's game robot health value and the attack object's attack object health value, the game status information can be represented in the form of an array as (800,700), 800 represents the attack object health value of the attack object, and 700 represents the game robot's game robot health value; taking the example that the game status information includes the game robot's skill release status and the attack object's attack object skill release status, the game status information can be represented in the form of an array as (5,6), 5 represents that the attack object has released 5 skills, and 6 represents that the attack object has released 6 skills; taking the example that the game status information includes the game robot's skill cooling status and the attack object's attack object skill cooling status, the game status information can be represented in the form of an array as (7,25), 7 represents that the attack object's skill cooling time is 7 seconds, and 25 represents that the attack object's skill cooling status is 25 seconds.
[0040] Among them, the policy gradient series algorithm uses the first game state information obtained to decide the next action of the game robot, that is, the target action. The policy gradient series algorithm includes a "value network" and a "policy network". The "value network" is used to predict the sum of the subsequent reward values that can be obtained under the current state and strategy, and the policy network outputs the probability of taking each action in the current state. Every time the policy network makes a decision, it randomly samples the actions to be executed according to the output probability distribution and executes them in the environment.
[0041] Specifically, due to the differences in game types, the preset actions set in different types of games are also different. Taking a massively multiplayer online role-playing game (MMORPG) as an example, the virtual characters therein can perform multiple actions including moving, jumping, dodging, and using a certain skill, so the preset actions are moving, jumping, dodging, and using a certain skill.
[0042] In step 102, the game robot is controlled to execute the target action, and second game state information after executing the target action is obtained.
[0043] After the target action is predicted, the target action is executed by controlling the game robot in the virtual scene, and the second game state information after the target action is executed is obtained.
[0044] Specifically, taking the target action of using skill A as an example, when the game robot uses skill A to attack the attack target, the attack target's health value may be reduced, and the game robot's health value may be increased. Thus, the second game state information is obtained as (300, 800).
[0045] In step 103, a game state reward value is determined based on the first game state information and the second game state information.
[0046] Among them, since the present application trains the game robot using reinforcement learning, it is necessary to determine the total reward value generated by executing the target action. The total reward value includes the game state reward value generated by the change in the game state. Therefore, it is necessary to determine the game state reward value based on the first game state information and the second game state information.
[0047] In some implementations, the game state information includes the game robot life value of the game robot and the attack object life value of the attack object, and the step of determining the game state reward value based on the first game state information and the second game state information includes:
[0048] (1) obtaining the first game robot life value and the first attack target life value included in the first game state information, and obtaining the second game robot life value and the second attack target life value included in the second game state information;
[0049] (2) calculating the difference between the health value of the second game robot and the health value of the first game robot to obtain a health change value of the game robot;
[0050] (3) calculating the difference between the health value of the first attack target and the health value of the second attack target to obtain a health change value of the attack target;
[0051] (4) calculating the ratio of the health value of the first game robot to the health value of the first attack object, and determining a first weight of the game robot's health change value and a second weight of the attack object's health change value;
[0052] (5) Determine a game status return value based on the game robot life change value, the attack target life change value, the first weight, and the second weight.
[0053] Among them, the present application provides a method for determining the game state reward value through the game robot life value of the game robot included in the game state information and the attack object life value of the attack object. The specific steps include: obtaining the first game robot life value and the first attack object life value included in the first game state information, and obtaining the second game robot life value and the second attack object life value included in the second game state information; calculating the difference between the second game robot life value and the first game robot life value to obtain the game robot life change value; calculating the difference between the first attack object life value and the second attack object life value to obtain the attack object life change value; calculating the ratio of the first game robot life value to the first attack object life value, determining the first weight of the game robot life change value, and the second weight of the attack object life change value; determining the game state reward value based on the game robot life change value, the attack object life change value, the first weight and the second weight.
[0054] For example, the first game state information is (800, 700), and the second game state information is (300, 800), then the health value of the first game robot is 700, the health value of the first attack object is 800, the health value of the second game robot is 800, and the health value of the second attack object is 300. The game robot health change value is 800-700=100, the attack object health change value is 800-300=500, and the ratio of the first game robot health value to the first attack object health value is 700:800=7:8, so the first weight is 7, and the second weight is 8.
[0055] Wherein, when the game state information includes the skill release status of the game robot and the skill release status of the attack object, the step of determining the game state reward value based on the first game state information and the second game state information includes:
[0056] (1) obtaining a first release number of a skill released by a first game robot and a second release number of a skill released by a first attack target, included in the first game state information, and obtaining a third release number of a skill released by a second game robot and a fourth release number of a skill released by a second attack target, included in the second game state information;
[0057] (2) calculating the difference between the third release quantity and the first release quantity to obtain the skill release difference of the game robot;
[0058] (3) calculating the difference between the fourth release quantity and the second release quantity to obtain a skill release difference of the attack target;
[0059] (4) calculating the ratio of the first release quantity to the second release quantity, and determining a third weight of the skill release difference of the game robot and a fourth weight of the skill release difference of the attack target;
[0060] (5) Determine a game status reward value based on the skill release difference of the game robot, the skill release difference of the attack target, the third weight, and the fourth weight.
[0061] For example, if the first game state information is (5,6) and the second game state information is (8,7), the first release quantity is 5, the second release quantity is 6, the third release quantity is 8, and the fourth release quantity is 5. The difference in skill release of the game robot is 8-5=3, and the difference in skill release of the target is 7-6=1. The ratio of the first release quantity to the second release quantity is 5:6, so the third weight is 5 and the fourth weight is 6.
[0062] Wherein, when the game state information includes the skill cooling status of the game robot and the attack target skill cooling status of the attack target, the step of determining the game state reward value based on the first game state information and the second game state information includes:
[0063] (1) obtaining the first skill cooldown time of the first game robot and the second skill cooldown of the first attack target included in the first game state information, and obtaining the third skill cooldown of the second game robot and the fourth skill cooldown of the second attack target included in the second game state information;
[0064] (2) calculating the difference between the first skill cooling time and the third skill cooling time to obtain the skill cooling time difference of the game robot;
[0065] (3) calculating the difference between the second skill cooldown time and the fourth skill cooldown time to obtain the skill cooldown time difference of the target;
[0066] (4) calculating the ratio of the first skill cooling time to the second skill cooling time, determining a fifth weight of the skill cooling time difference of the game robot, and a sixth weight of the skill cooling time difference of the attack target;
[0067] (5) Determine a game status report value based on the skill cooldown time difference of the game robot, the skill cooldown time difference of the attack target, the fifth weight, and the sixth weight.
[0068] For example, if the first game state information is (7, 25) and the second game state information is (6, 15), then the first skill cooldown is 7, the second skill cooldown is 25, the third skill cooldown is 6, and the fourth skill cooldown is 15. The difference in skill cooldown of the game robot is 7-6=1, and the difference in skill cooldown of the target is 25-15=19. The ratio of the first skill cooldown to the second skill cooldown is 7:25, so the fifth weight is 7 and the sixth weight is 25.
[0069] In some implementations, the step of determining the game state reward value based on the game robot life change value, the attack object life change value, the first weight, and the second weight includes:
[0070] (1.1) Calculating the product of the life change value of the game robot and the first weight to obtain a first calculation result;
[0071] (1.2) calculating the product of the life change value of the attack target and the second weight to obtain a second calculation result;
[0072] (1.3) Calculate the sum of the first calculation result and the second calculation result to obtain a game status reward value.
[0073] Among them, after determining the life change value of the game robot, the life change value of the attack object, the first weight and the second weight, the product of the life change value of the game robot and the first weight can be calculated to obtain a first calculation result; the product of the life change value of the attack object and the second weight can be calculated to obtain a second calculation result; the sum of the first calculation result and the second calculation result is calculated to obtain a game status return value.
[0074] For example, the game robot's life change value is 800-700=100, and the attack target's life change value is 800-300=500, so the first weight is 7 and the second weight is 8. Then the first calculation result is 100*7=700, the second calculation result is 500*8=4000, and the game status return value is 4000+700=4700.
[0075] In some implementations, the step of determining the game state reward value based on the skill release difference of the game robot, the skill release difference of the attack target, the third weight, and the fourth weight includes:
[0076] (1.1) calculating the product of the skill release difference of the game robot and the third weight to obtain a third calculation result;
[0077] (1.2) calculating the product of the skill release difference of the attack target and the fourth weight to obtain a fourth calculation result;
[0078] (1.3) Calculate the sum of the third calculation result and the fourth calculation result to obtain a game status reward value.
[0079] For example, the skill release difference of the game robot is 3, the skill release difference of the attack target is 1, the first weight is 5, and the second weight is 6, then the game state return value is: 3*5+1*6=21.
[0080] In some implementations, the step of determining the game state reward value based on the skill cooling time difference of the game robot, the skill cooling time difference of the attack target, the fifth weight, and the sixth weight includes:
[0081] (1.1) Calculating the product of the skill cooldown time difference of the game robot and the fifth weight to obtain a fifth calculation result;
[0082] (1.2) calculating the product of the skill cooldown time difference of the attack target and the sixth weight to obtain a sixth calculation result;
[0083] (1.3) Calculate the sum of the fifth calculation result and the sixth calculation result to obtain the game status reward value.
[0084] For example, the skill cooling time difference of the game robot is 1, the skill cooling time difference of the attack target is 19, the fifth weight is 7, and the sixth weight is 25, then the game state return value is: 1*19+7*25=194.
[0085] In step 104, it is determined whether the game robot performs the target action within the historical time period to obtain a determination result.
[0086] Among them, the problem that the game robot trained only by the game state reward value is that the game robot will no longer try to use certain actions during the learning process. The reason for this is that when the game robot's strategy is not mature enough, using a skill that does not cause damage may not bring direct benefits, resulting in the frequency of this type of action gradually decreasing, until it is reduced to almost 0.
[0087] For example, the dodge action needs to be released precisely at the time when the opponent releases a damaging skill in order to gain benefits; however, skill-based actions can basically gain certain benefits at any time. In such an environment, the game robot will use skill-based actions more and more frequently, while the frequency of dodge will decrease. When this frequency drops to close to 0, it is considered impossible for the game robot to learn to use the dodge action again.
[0088] Based on the above problems, the present application sets an additional training encouragement method, namely, determining whether the game robot performs the target action within a historical time period to obtain a determined result.
[0089] In some embodiments, the first game state information includes execution marks of multiple preset actions, and the execution marks are used to record whether each preset action has been executed in a historical time period. The step of determining whether the game robot executes the target action in the historical time period to obtain a determination result includes:
[0090] (1) obtaining a target execution flag of the target action in the first game state information;
[0091] (2) if the execution mark type of the target execution mark is the first execution mark type, determining that the game robot executes the target action within the historical time period;
[0092] (3) If the execution mark type of the target execution mark is the second execution mark type, it is determined that the game robot has not executed the target action within the historical time period.
[0093] The method for determining whether the game robot performs the target action in the historical time period may be: the first game state information includes execution marks of multiple preset actions, for example, the first game state information is (800, 700, 0, 0, 1), which means that the health value of the first game robot is 700, the health value of the first attack object is 800, the game robot has not used movement, dodge, and has used skill 1. If the target action is dodge, the corresponding target execution mark is 0.
[0094] If the execution mark type of the target execution mark is the first execution mark type (for example, the target execution mark is 1), it is determined that the game robot has executed the target action within the historical time period; if the execution mark type of the target execution mark is the second execution mark type (for example, the target execution mark is 0), it is determined that the game robot has not executed the target action within the historical time period.
[0095] In some implementations, the step of determining whether the game robot performs the target action within a historical time period to obtain a determination result includes:
[0096] (1) querying the execution record of the target action from the stored execution records of preset actions in the historical time period;
[0097] (2) if the execution record of the target action is found, determining whether the game robot has executed the target action within a historical time period;
[0098] (3) If no execution record of the target action is found, it is determined that the game robot has not executed the target action within the historical time period.
[0099] The preset action execution record can also be obtained by recording all the execution actions of the game robot in the historical time period. The execution record of the target action is queried from the stored preset action execution record in the historical time period. If the execution record of the target action is found, it is determined that the game robot has executed the target action in the historical time period; if the execution record of the target action is not found, it is determined that the game robot has not executed the target action in the historical time period.
[0100] In step 105, based on the game state reward value and the determination result, the total reward value of the target action is calculated.
[0101] Wherein, based on the determination result of whether the game robot performs the target action within the historical time period, combined with the game state reward value, the total reward value generated by the target action is calculated.
[0102] In some implementations, the step of calculating the total reward value of the target action based on the game state reward value and the determination result includes:
[0103] (1.1) If the determination result indicates that the game robot has not executed the target action within the historical time period, the specified reward value is determined as the execution reward value;
[0104] (1.2) Based on the game state reward value and the execution reward value, calculate the total reward value of the target action.
[0105] The present application sets that no matter what kind of action is performed, as long as the game robot has not performed it in the historical time period, when the action is performed, an execution reward value of, for example, 100 (specified reward value) will be generated. Based on the game state reward value and the execution reward value, the total reward value of the target action is calculated.
[0106] In some embodiments, the method further comprises:
[0107] (1.1) If the determination result indicates that the game robot has executed the target action within the historical time period, the execution reward value is determined to be zero;
[0108] (1.2) Based on the game state reward value and the execution reward value, calculate the total reward value of the target action.
[0109] Among them, if the determination result indicates that the game robot has performed the target action in the historical time period, it means that the specified reward value has been determined as the execution reward value and the total reward value has been calculated when the target action was performed before. In order to avoid the execution reward value that increases the specified reward value every time the action is used, the game robot may generate tens of thousands of rewards for using hundreds of dodges in a game, which far exceeds the rewards brought by the real game goal. As a result, only a game robot that dodges all the time is trained, which has no application value. Therefore, it is set that if the determination result indicates that the game robot has performed the target action in the historical time period, the execution reward value is determined to be zero. And based on the game state reward value and the execution reward value, the total reward value of the target action is calculated.
[0110] In some implementations, the step of calculating the total reward value of the target action based on the game state reward value and the execution reward value includes:
[0111] The sum of the game state reward value and the execution reward value is calculated to obtain the total reward value of the target action.
[0112] Among them, based on the game state reward value and the execution reward value, the method of calculating the total reward value of the target action is: calculating the sum of the game state reward value and the execution reward value to obtain the total reward value of the target action.
[0113] In some implementations, the step of calculating the total reward value of the target action based on the game state reward value and the execution reward value includes:
[0114] The product of the game state reward value and the execution reward value is calculated to obtain the total reward value of the target action.
[0115] Among them, based on the game state reward value and the execution reward value, the method of calculating the total reward value of the target action is: calculate the product of the game state reward value and the execution reward value to obtain the total reward value of the target action.
[0116] In step 106, the game robot is trained based on the total reward value.
[0117] Among them, the total reward value is used for back propagation to update the parameters of the neural network model and enhance the strength of the strategy represented by the neural network, so that the strategy network can make the best decision on the action to control the execution of the game robot when facing similar samples.
[0118] As can be seen from the above, the embodiment of the present application obtains the first game state information, and based on the first game state information, selects the target action from multiple preset actions; controls the game robot to execute the target action, and obtains the second game state information after executing the target action; determines the game state reward value based on the first game state information and the second game state information; determines whether the game robot executes the target action within the historical time period and obtains a determination result; calculates the total reward value of the target action based on the game state reward value and the determination result; and trains the game robot based on the total reward value. In this way, the total reward value of the target action is affected by determining the determination result of whether the game robot executes the target action within the historical time period, thereby encouraging the game robot to explore actions with low benefits through the total reward value, thereby achieving action diversity of the trained game robot.
[0119] In order to better implement the game robot training method provided in the embodiment of the present application, the embodiment of the present application also provides a device based on the above-mentioned game robot training method. The meanings of the terms are the same as those in the above-mentioned game robot training method, and the specific implementation details can refer to the description in the method embodiment.
[0120] See also Figure 2 , Figure 2 A schematic diagram of the structure of a training device for a gaming robot provided in an embodiment of the present application, wherein the training device for a gaming robot may include a screening module 301, an acquisition module 302, a first determination module 303, a second determination module 304, a calculation module 305, and a training module 306, etc.
[0121] A screening module 301 is used to obtain first game state information, and screen a target action from a plurality of preset actions based on the first game state information;
[0122] An acquisition module 302 is used to control the game robot to perform the target action and obtain second game state information after performing the target action;
[0123] A first determination module 303, configured to determine a game state reward value based on the first game state information and the second game state information;
[0124] The second determination module 304 is used to determine whether the game robot performs the target action within the historical time period and obtain a determination result;
[0125] A calculation module 305, for calculating a total reward value of the target action based on the game state reward value and the determination result;
[0126] The training module 306 is used to train the game robot based on the total reward value.
[0127] In some implementations, the game state information includes the life value of the game robot and the life value of the attack object. The first determining module 303 includes:
[0128] A first acquisition submodule, used to acquire the health value of the first game robot and the health value of the first attack object included in the first game state information, and to acquire the health value of the second game robot and the health value of the second attack object included in the second game state information;
[0129] A first calculation submodule, used for calculating the difference between the second game robot's health value and the first game robot's health value, to obtain a game robot's health change value;
[0130] A second calculation submodule is used to calculate the difference between the health value of the first attack object and the health value of the second attack object to obtain a health change value of the attack object;
[0131] A third calculation submodule, used to calculate the ratio of the health value of the first game robot to the health value of the first attack object, and determine a first weight of the game robot's health change value and a second weight of the attack object's health change value;
[0132] The first determination submodule is used to determine the game state return value based on the life change value of the game robot, the life change value of the attack object, the first weight and the second weight.
[0133] In some embodiments, the determining submodule is used to:
[0134] Calculate the product of the life change value of the game robot and the first weight to obtain a first calculation result;
[0135] Calculate the product of the attack target's life change value and the second weight to obtain a second calculation result;
[0136] The sum of the first calculation result and the second calculation result is calculated to obtain a game state reward value. In some embodiments, the first game state information includes execution marks of multiple preset actions, and the execution marks are used to record whether each preset action has been executed within a historical time period. The second determination module 304 includes:
[0137] A second acquisition submodule is used to acquire a target execution mark of the target action in the first game state information;
[0138] A second determination submodule, configured to determine that the game robot executes the target action within a historical time period if the execution mark type of the target execution mark is a first execution mark type;
[0139] The third determination submodule is used to determine that the game robot has not executed the target action within a historical time period if the execution mark type of the target execution mark is a second execution mark type.
[0140] In some implementations, the second determining module 304 includes:
[0141] A query submodule, used to query the execution record of the target action from the preset action execution records stored in the historical time period;
[0142] A fourth determination submodule is used to determine whether the game robot has executed the target action within a historical time period if an execution record of the target action is found;
[0143] The fifth determination submodule is used to determine that the game robot has not executed the target action within a historical time period if no execution record of the target action is found.
[0144] In some implementations, the calculation module 305 includes:
[0145] a sixth determination submodule, configured to determine the specified reward value as the execution reward value if the determination result indicates that the game robot has not executed the target action within the historical time period;
[0146] The fourth calculation submodule is used to calculate the total reward value of the target action based on the game state reward value and the execution reward value.
[0147] In some implementations, the calculation module 305 includes:
[0148] a seventh determination submodule, configured to determine the execution reward value to be zero if the determination result indicates that the game robot has executed the target action within a historical time period;
[0149] The fourth calculation submodule calculates the total reward value of the target action based on the game state reward value and the execution reward value.
[0150] In some implementations, the fourth computing submodule is configured to:
[0151] The sum of the game state reward value and the execution reward value is calculated to obtain the total reward value of the target action.
[0152] As can be seen from the above, the embodiment of the present application obtains the first game state information through the screening module 301, and selects the target action from multiple preset actions based on the first game state information; the acquisition module 302 controls the game robot to execute the target action, and obtains the second game state information after executing the target action; the first determination module 303 determines the game state reward value based on the first game state information and the second game state information; the second determination module 304 determines whether the game robot executes the target action within the historical time period and obtains a determination result; the calculation module 305 calculates the total reward value of the target action based on the game state reward value and the determination result; the training module 306 trains the game robot based on the total reward value. In this way, the total reward value of the target action is affected by determining the determination result of whether the game robot executes the target action within the historical time period, so as to encourage the game robot to explore actions with low benefits through the total reward value, thereby achieving the action diversity of the trained game robot.
[0153] Accordingly, the embodiment of the present application also provides a computer device, which may be a terminal or a server, and the terminal may be a smart phone, a tablet computer, a laptop computer, a touch screen, a game console, a personal computer (PC), a personal digital assistant (PDA), or other terminal devices. Figure 3 As shown, Figure 3 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. The computer device 2000 includes a processor 401 having one or more processing cores, a memory 402 having one or more computer-readable storage media, and a computer program stored in the memory 402 and executable on the processor. The processor 401 is electrically connected to the memory 402. Those skilled in the art will appreciate that the computer device structure shown in the figure does not constitute a limitation on the computer device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange components differently.
[0154] The processor 401 is the control center of the computer device 2000. It uses various interfaces and lines to connect the various parts of the entire computer device 2000, executes various functions of the computer device 2000 and processes data by running or loading software programs and / or modules stored in the memory 402, and calling data stored in the memory 402, thereby monitoring the computer device 2000 as a whole.
[0155] In the embodiment of the present application, the processor 401 in the computer device 2000 will load instructions corresponding to the processes of one or more application programs into the memory 402 according to the following steps, and the processor 401 will run the application programs stored in the memory 402 to implement various functions:
[0156] Obtain first game state information, and filter out a target action from multiple preset actions based on the first game state information; control a game robot to execute the target action, and obtain second game state information after executing the target action; determine a game state reward value based on the first game state information and the second game state information; determine whether the game robot executes the target action within a historical time period and obtain a determination result; calculate a total reward value of the target action based on the game state reward value and the determination result; and train the game robot based on the total reward value.
[0157] In some implementations, the game state information includes the game robot life value of the game robot and the attack object life value of the attack object, and the step of determining the game state reward value based on the first game state information and the second game state information includes:
[0158] Acquire the first game robot life value and the first attack object life value included in the first game state information, and acquire the second game robot life value and the second attack object life value included in the second game state information;
[0159] Calculate the difference between the second game robot's health value and the first game robot's health value to obtain the game robot's health change value;
[0160] Calculate the difference between the health value of the first attack target and the health value of the second attack target to obtain a health change value of the attack target;
[0161] Calculating the ratio of the first game robot's health value to the first attack object's health value, and determining a first weight of the game robot's health change value and a second weight of the attack object's health change value;
[0162] A game state reward value is determined based on the game robot life change value, the attack object life change value, the first weight, and the second weight.
[0163] In some implementations, the step of determining the game state reward value based on the game robot life change value, the attack object life change value, the first weight, and the second weight includes:
[0164] Calculate the product of the life change value of the game robot and the first weight to obtain a first calculation result;
[0165] Calculate the product of the attack target's life change value and the second weight to obtain a second calculation result;
[0166] The sum of the first calculation result and the second calculation result is calculated to obtain a game state reward value.
[0167] In some embodiments, the first game state information includes execution marks of multiple preset actions, and the execution marks are used to record whether each preset action has been executed in a historical time period. The step of determining whether the game robot executes the target action in the historical time period to obtain a determination result includes:
[0168] Obtaining a target execution mark of the target action in the first game state information;
[0169] If the execution mark type of the target execution mark is the first execution mark type, determining that the game robot executes the target action within the historical time period;
[0170] If the execution mark type of the target execution mark is the second execution mark type, it is determined that the game robot has not executed the target action within a historical time period.
[0171] In some implementations, the step of determining whether the game robot performs the target action within a historical time period to obtain a determination result includes:
[0172] Querying the execution record of the target action from the stored preset action execution records within the historical time period;
[0173] If the execution record of the target action is found, it is determined that the game robot has executed the target action within the historical time period;
[0174] If no execution record of the target action is found, it is determined that the game robot has not executed the target action within the historical time period.
[0175] In some implementations, the step of calculating the total reward value of the target action based on the game state reward value and the determination result includes:
[0176] If the determination result indicates that the game robot has not executed the target action within the historical time period, the specified reward value is determined as the execution reward value;
[0177] Based on the game state reward value and the execution reward value, a total reward value of the target action is calculated.
[0178] In some embodiments, the method further comprises:
[0179] If the determination result indicates that the game robot has executed the target action within the historical time period, the execution reward value is determined to be zero;
[0180] Based on the game state reward value and the execution reward value, a total reward value of the target action is calculated.
[0181] In some implementations, the step of calculating the total reward value of the target action based on the game state reward value and the execution reward value includes:
[0182] The sum of the game state reward value and the execution reward value is calculated to obtain the total reward value of the target action.
[0183] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.
[0184] Optional, such as Figure 3 As shown, the computer device 2000 further includes: a touch screen 403, a radio frequency circuit 404, an audio circuit 405, an input unit 406, and a power supply 407. The processor 401 is electrically connected to the touch screen 403, the radio frequency circuit 404, the audio circuit 405, the input unit 406, and the power supply 407, respectively. Those skilled in the art can understand that Figure 3 The computer device structure shown in the figure does not constitute a limitation on the computer device, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.
[0185] The touch display screen 403 can be used to display a graphical user interface and receive operation instructions generated by the user acting on the graphical user interface. The touch display screen 403 may include a display panel and a touch panel. Among them, the display panel may be used to display information input by the user or information provided to the user and various graphical user interfaces of the computer device, and these graphical user interfaces may be composed of graphics, text, icons, videos and any combination thereof. Optionally, the display panel may be configured in the form of a liquid crystal display (LCD, Liquid Crystal Display), an organic light emitting diode (OLED, Organic Light-Emitting Diode) and the like. The touch panel may be used to collect the user's touch operation on or near it (such as the user using any suitable object or attachment such as a finger, a stylus, etc. on the touch panel or near the touch panel), and generate corresponding operation instructions, and the operation instructions execute corresponding programs. Optionally, the touch panel may include two parts, a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch orientation, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into the touch point coordinates, and then sends it to the processor 401, and can receive the command sent by the processor 401 and execute it. The touch panel can cover the display panel. When the touch panel detects a touch operation on or near it, it is transmitted to the processor 401 to determine the type of touch event, and then the processor 401 provides a corresponding visual output on the display panel according to the type of touch event. In an embodiment of the present application, the touch panel and the display panel can be integrated into the touch display screen 403 to realize the input and output functions. However, in some embodiments, the touch panel and the touch panel can be used as two independent components to realize the input and output functions. That is, the touch display screen 403 can also be used as a part of the input unit 406 to realize the input function.
[0186] In the embodiment of the present application, the processor 401 executes the game application to generate a graphical user interface on the touch screen 403. The virtual scene on the graphical user interface includes at least one skill control area, and the skill control area includes at least one skill control. The touch screen 403 is used to present the graphical user interface and receive operation instructions generated by the user acting on the graphical user interface.
[0187] The radio frequency circuit 404 may be used to send and receive radio frequency signals, so as to establish wireless communication with a network device or other computer devices through wireless communication, and to send and receive signals between the network device or other computer devices.
[0188] The audio circuit 405 can be used to provide an audio interface between the user and the computer device through a speaker and a microphone. The audio circuit 405 can transmit the electrical signal converted from the received audio data to the speaker, which is converted into a sound signal for output; on the other hand, the microphone converts the collected sound signal into an electrical signal, which is received by the audio circuit 405 and converted into audio data, and then the audio data is output to the processor 401 for processing, and then sent to another computer device through the radio frequency circuit 404, or the audio data is output to the memory 402 for further processing. The audio circuit 405 may also include an earphone jack to provide communication between an external headset and the computer device.
[0189] The input unit 406 may be used to receive input numbers, character information or user feature information (such as fingerprint, iris, facial information, etc.), and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.
[0190] The power supply 407 is used to supply power to various components of the computer device 2000. Optionally, the power supply 407 can be logically connected to the processor 401 through a power management system, so that the power management system can manage charging, discharging, and power consumption. The power supply 407 can also include one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, power status indicators, and other arbitrary components.
[0191] although Figure 3 Not shown, the computer device 2000 may also include a camera, a sensor, a wireless fidelity module, a Bluetooth module, etc., which will not be described in detail here.
[0192] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0193] As can be seen from the above, the computer device provided in this embodiment obtains the first game state information, and based on the first game state information, selects the target action from multiple preset actions; controls the game robot to execute the target action, and obtains the second game state information after executing the target action; determines the game state reward value based on the first game state information and the second game state information; determines whether the game robot executes the target action within the historical time period and obtains a determination result; calculates the total reward value of the target action based on the game state reward value and the determination result; and trains the game robot based on the total reward value. In this way, the total reward value of the target action is affected by determining the determination result of whether the game robot executes the target action within the historical time period, thereby encouraging the game robot to explore actions with low benefits through the total reward value, thereby achieving action diversity of the trained game robot.
[0194] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.
[0195] To this end, an embodiment of the present application provides a computer-readable storage medium, in which multiple computer programs are stored, and the computer program can be loaded by a processor to execute the steps in any skill control method provided in the embodiment of the present application. For example, the computer program can execute the following steps:
[0196] Obtain first game state information, and filter out a target action from multiple preset actions based on the first game state information; control a game robot to execute the target action, and obtain second game state information after executing the target action; determine a game state reward value based on the first game state information and the second game state information; determine whether the game robot executes the target action within a historical time period and obtain a determination result; calculate a total reward value of the target action based on the game state reward value and the determination result; and train the game robot based on the total reward value.
[0197] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.
[0198] The storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.
[0199] Since the computer program stored in the storage medium can execute the steps in any of the game robot training methods provided in the embodiments of the present application, the beneficial effects that can be achieved by any of the game robot training methods provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.
[0200] The above is a detailed introduction to a game robot training method, device, computer-readable storage medium and computer equipment provided in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the ideas of the present application, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the present application.
Claims
1. A method for training a game robot, characterized in that: include: Acquire first game state information, and select a target action from a plurality of preset actions based on the first game state information; Controlling the game robot to execute the target action, and obtaining second game state information after executing the target action; Determine a game state reward value based on the first game state information and the second game state information; Determine whether the game robot performs the target action within a historical time period, and obtain a determination result; Calculating a total reward value of the target action based on the game state reward value and the determination result; The game robot is trained based on the total reward value.
2. The method for training a gaming robot according to claim 1, characterized in that: The game state information includes the game robot life value of the game robot and the attack object life value of the attack object. The step of determining the game state reward value based on the first game state information and the second game state information includes: Acquire the first game robot life value and the first attack object life value included in the first game state information, and acquire the second game robot life value and the second attack object life value included in the second game state information; Calculate the difference between the second game robot's health value and the first game robot's health value to obtain the game robot's health change value; Calculate the difference between the health value of the first attack target and the health value of the second attack target to obtain a health change value of the attack target; Calculating the ratio of the first game robot's health value to the first attack object's health value, and determining a first weight of the game robot's health change value and a second weight of the attack object's health change value; A game state reward value is determined based on the game robot life change value, the attack object life change value, the first weight, and the second weight.
3. The method for training a gaming robot according to claim 2, characterized in that: The step of determining the game state reward value based on the game robot life change value, the attack object life change value, the first weight and the second weight includes: Calculate the product of the life change value of the game robot and the first weight to obtain a first calculation result; Calculate the product of the attack target's life change value and the second weight to obtain a second calculation result; The sum of the first calculation result and the second calculation result is calculated to obtain a game state reward value.
4. The method for training a gaming robot according to claim 1, characterized in that: The first game state information includes execution marks of a plurality of preset actions, wherein the execution marks are used to record whether each preset action has been executed in a historical time period. The step of determining whether the game robot executes the target action in the historical time period and obtaining a determination result includes: Obtaining a target execution mark of the target action in the first game state information; If the execution mark type of the target execution mark is the first execution mark type, determining that the game robot executes the target action within the historical time period; If the execution mark type of the target execution mark is the second execution mark type, it is determined that the game robot has not executed the target action within a historical time period.
5. The method for training a gaming robot according to claim 1, characterized in that: The step of determining whether the game robot performs the target action within the historical time period and obtaining a determination result comprises: Querying the execution record of the target action from the stored preset action execution records within the historical time period; If the execution record of the target action is found, it is determined that the game robot has executed the target action within the historical time period; If no execution record of the target action is found, it is determined that the game robot has not executed the target action within the historical time period.
6. The method for training a gaming robot according to claim 1, characterized in that: The step of calculating the total reward value of the target action based on the game state reward value and the determination result includes: If the determination result indicates that the game robot has not executed the target action within the historical time period, the specified reward value is determined as the execution reward value; Based on the game state reward value and the execution reward value, a total reward value of the target action is calculated.
7. The method for training a gaming robot according to claim 6, characterized in that: The method further comprises: If the determination result indicates that the game robot has executed the target action within the historical time period, the execution reward value is determined to be zero; Based on the game state reward value and the execution reward value, a total reward value of the target action is calculated.
8. The method for training a gaming robot according to any one of claims 6 or 7, characterized in that: The step of calculating the total reward value of the target action based on the game state reward value and the execution reward value comprises: The sum of the game state reward value and the execution reward value is calculated to obtain the total reward value of the target action.
9. A training device for a gaming robot, characterized in that: include: A screening module, used to obtain first game state information, and screen a target action from a plurality of preset actions based on the first game state information; An acquisition module, used to control the game robot to perform the target action and obtain second game state information after performing the target action; A first determination module, configured to determine a game state reward value based on the first game state information and the second game state information; A second determination module is used to determine whether the game robot performs the target action within a historical time period to obtain a determination result; A calculation module, used for calculating the total reward value of the target action based on the game state reward value and the determination result; A training module is used to train the game robot based on the total reward value.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor to execute the steps in the method for training a gaming robot according to any one of claims 1 to 8.
11. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps in the training method for a gaming robot as described in any one of claims 1 to 8 are implemented.