Game robot training method, device, equipment and storage medium

Abstract

The embodiment of the present application provides a method, device, equipment and storage medium for training a game robot, which belongs to the field of artificial intelligence technology. The method includes: obtaining a first game state of a target game and first decision data of a human player in the first game state; based on the first game state and the first decision data, training a preset reward model to obtain a trained reward model; obtaining a second game state of the target game and second decision data of the game robot in the second game state; using the trained reward model, predicting rewards based on the second game state and the second decision data to obtain a reward for the game robot; based on the second game state, the second decision data and the reward, performing reinforcement learning training on the game robot to obtain a trained game robot. The embodiment of the present application can improve training efficiency and provide a game robot with higher decision-making ability.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method, device, equipment and storage medium for training a gaming robot. Background Art

[0002] With the in-depth research on artificial intelligence, implanting game robots into games to simulate human player decision-making is one of the important research directions in the gaming field.

[0003] The game robots provided by related technologies generally use rule-based methods and traditional reinforcement learning methods to make game decisions. However, since rule-based methods rely on pre-set fixed rules, they are difficult to cope with complex and changing game scenarios, and are prone to strategy rigidity, which makes it impossible to effectively capture complex player behaviors and dynamic strategies, resulting in the lack of flexibility and adaptability of game robots' game decisions; traditional reinforcement learning methods often require a lot of trial and error and time to learn effective strategies, and have low training efficiency.

[0004] Therefore, how to provide gaming robots with higher decision-making capabilities while improving training efficiency has become a technical problem that needs to be solved urgently. Summary of the invention

[0005] The main purpose of the embodiments of the present application is to propose a method, device, equipment and storage medium for training a gaming robot, aiming to improve training efficiency and provide a gaming robot with higher decision-making ability.

[0006] To achieve the above-mentioned purpose, a first aspect of an embodiment of the present application proposes a method for training a game robot, the method comprising:

[0007] Acquire a first game state of a target game and first decision data of a human player in the first game state;

[0008] Based on the first game state and the first decision data, training a preset reward model to obtain a trained reward model;

[0009] Acquire a second game state of the target game and second decision data of the game robot in the second game state;

[0010] Using the trained reward model, a reward prediction is performed based on the second game state and the second decision data to obtain a reward for the game robot;

[0011] Based on the second game state, the second decision data and the reward, reinforcement learning training is performed on the game robot to obtain a trained game robot.

[0012] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides a training device for a gaming robot, the device comprising:

[0013] A first acquisition module, used to acquire a first game state of a target game and first decision data of a human player in the first game state;

[0014] A reward model training module, used to train a preset reward model based on the first game state and the first decision data to obtain a trained reward model;

[0015] A second acquisition module, used to acquire a second game state of the target game and second decision data of the game robot in the second game state;

[0016] A reward prediction module, used to predict a reward based on the second game state and the second decision data by using the trained reward model to obtain a reward for the game robot;

[0017] The game robot training module is used to perform reinforcement learning training on the game robot based on the second game state, the second decision data and the reward to obtain a trained game robot.

[0018] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes a computer device, which includes a memory and a processor, the memory stores a computer program, and the processor implements the method described in the first aspect when executing the computer program.

[0019] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0020] The present application proposes a method, device, equipment and storage medium for training a game robot. The method for training a game robot obtains a first game state of a target game and first decision data of a human player under the first game state; based on the first game state and the first decision data, a preset reward model is trained to obtain a trained reward model; a second game state of the target game and second decision data of the game robot under the second game state are obtained; through the trained reward model, a reward is predicted based on the second game state and the second decision data to obtain a reward for the game robot; based on the second game state, the second decision data and the reward, reinforcement learning training is performed on the game robot to obtain a trained game robot. In this way, by training the reward model through the first game state and the first decision data of the human player under the first game state, it can be ensured that the reward model is exposed to the complex and ever-changing decision-making scenarios that human players actually face during training, thereby improving the adaptability and generalization ability of the reward model, thereby improving the training effect of the reward model, and enabling the trained reward model to quickly predict more accurate rewards, so that the game robot can obtain more accurate rewards in a timely manner through the trained reward model. This not only helps the game robot reduce invalid exploration during training, thereby improving the training efficiency of the training game robot, but also can effectively guide the game robot to imitate and learn the decisions of human players, so that the decision-making ability of the trained game robot is closer to that of human players, and the intelligence level of the trained game robot is enhanced, thereby improving the decision-making ability of the trained game robot. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 is a flow chart of a method for training a gaming robot provided in an embodiment of the present application;

[0022] Figure 2 It is a schematic diagram of the training architecture provided in the embodiment of the present application;

[0023] Figure 3 yes Figure 1 A detailed flowchart of step S104;

[0024] Figure 4 is a schematic block diagram of a training device for a gaming robot provided in an embodiment of the present application;

[0025] Figure 5 It is a schematic block diagram of the structure of the computer device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0027] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0028] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0029] First, some nouns involved in this application are analyzed:

[0030] Artificial intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing and expert systems. AI can simulate the information process of human consciousness and thinking. AI is also a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0031] Target game: can be a multi-player game, where players include human players and game robots that act as AI players.

[0032] Reinforcement Learning (RL): aims to learn the best strategy through interaction with the environment. It uses a trial-and-error method to enable the agent to select actions in a specific state, and adjust its behavior based on the reward or penalty feedback obtained to maximize long-term rewards. The core elements of reinforcement learning include agent, environment, state, action, and reward. Among them, the agent refers to the learner who performs the action; the environment refers to the external system with which the agent interacts; the state refers to the specific situation of the environment, the situation in which the agent is at a certain moment. Action refers to the behavior that the agent can take in a specific state. Reward refers to the feedback obtained after the agent performs an action, which is used to evaluate the quality of the action. Through continuous cycles of exploration and utilization, reinforcement learning can effectively learn the best decision-making strategy in complex tasks. The game robot in the embodiment of the present application can be a reinforcement learning model. For the game robot, the target game scene can be understood as the environment, and the decision data of the game robot can be understood as the action taken by the game robot.

[0033] Based on this, the embodiments of the present application provide a method, device, equipment and storage medium for training a gaming robot, aiming to improve training efficiency and provide a gaming robot with higher decision-making capabilities.

[0034] The training method, device, equipment and storage medium for the gaming robot provided in the embodiments of the present application are specifically described through the following embodiments. First, the training method for the gaming robot in the embodiments of the present application is described.

[0035] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0036] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0037] The training method for a game robot provided in an embodiment of the present application relates to the field of artificial intelligence technology. The training method for a game robot provided in an embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a vehicle-mounted terminal, a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or can be configured as a server cluster or a distributed system composed of multiple physical servers, or can be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the training method for a game robot, etc., but is not limited to the above forms.

[0038] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer computer devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0039] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0040] Figure 1 is an optional flow chart of the method for training a gaming robot provided in an embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S105.

[0041] Step S101, obtaining a first game state of a target game and first decision data of a human player in the first game state.

[0042] See also Figure 2 , Figure 2 A schematic diagram of a training architecture provided for an embodiment of the present application, wherein the training architecture includes two parts: a training reward model and a training game robot.

[0043] The following combination Figure 2 The specific process of training the reward model and training the game robot is explained.

[0044] like Figure 2 As shown, the first game state of the human player playing the target game is obtained, wherein the first game state includes the position, health value, skill state of the human player's character, and the map information where the human player's character is currently located, etc.

[0045] The first decision data of the human player in the first game state is also obtained, wherein the first decision data includes actions taken by the human player in the first game state, such as moving, attacking, using props, etc.

[0046] Step S102: training a preset reward model based on the first game state and the first decision data to obtain a trained reward model.

[0047] Among them, the preset reward model can be a model based on a neural network or a model based on a decision tree.

[0048] The reward model is trained using a large amount of first game states and first decision data to iteratively update the parameters θ of the reward model, so that the reward model can learn to predict more accurate rewards from the real data of human players until the reward model converges to obtain a trained reward model, which can be represented by RM(S; θ).

[0049] In this way, it can be ensured that the reward model is exposed to the complex and ever-changing decision-making scenarios that human players actually face during training, thereby improving the overall performance and adaptability of the reward model, improving the performance of the reward model, and enabling the trained reward model to predict dynamic and refined rewards.

[0050] Step S103, obtaining a second game state of the target game and second decision data of the game robot in the second game state.

[0051] After the reward model training is completed, the game robot can be trained based on the trained reward model.

[0052] Specifically, a second game state of the game robot playing the target game is obtained, wherein the second game state includes the position, health value, skill state, and map information of the game robot's current location, etc.

[0053] Second decision data of the game robot in the second game state is also obtained, wherein the second decision data includes actions taken by the game robot in the second game state.

[0054] Step S104, using the trained reward model, a reward prediction is performed based on the second game state and the second decision data to obtain a reward for the game robot.

[0055] By using a trained reward model and predicting rewards based on the second game state and the second decision data, the accurate reward for the game robot can be quickly obtained. The accurate reward can more accurately evaluate the effectiveness of the game robot's intelligent body in executing the second decision data in the second game state.

[0056] Step S105, based on the second game state, the second decision data and the reward, reinforcement learning training is performed on the game robot to obtain a trained game robot.

[0057] Finally, based on the second game state, the second decision data and the reward predicted by the trained reward model, the game robot is trained through reinforcement learning, so that the intelligent agent of the game robot can continuously explore and adjust the decision-making strategy of the game robot to achieve the purpose of maximizing the cumulative reward, until the decision-making strategy converges, and a trained game robot with higher decision-making ability is obtained.

[0058] In this way, by combining a trained reward model with reinforcement learning, on the one hand, the intelligent agent of the game robot can obtain rewards in time during the learning process, thereby accelerating the convergence process of the decision-making strategy and greatly shortening the exploration and learning time of the intelligent agent of the game robot. On the other hand, since the rewards are more accurate, the intelligent agent of the game robot can reduce ineffective exploration, thereby improving learning efficiency. On the other hand, compared with the traditional method of obtaining rewards using manually designed reward functions, automatically obtaining rewards through a trained reward model is more flexible and provides convenience for training game robots.

[0059] The game robot training method provided in the above embodiment obtains the first game state of the target game and the first decision data of the human player in the first game state; based on the first game state and the first decision data, a preset reward model is trained to obtain a trained reward model; the second game state of the target game and the second decision data of the game robot in the second game state are obtained; through the trained reward model, a reward prediction is performed based on the second game state and the second decision data to obtain a reward for the game robot; based on the second game state, the second decision data and the reward, reinforcement learning training is performed on the game robot to obtain a trained game robot. In this way, by training the reward model through the first game state and the first decision data of the human player under the first game state, it can be ensured that the reward model is exposed to the complex and ever-changing decision-making scenarios that human players actually face during training, thereby improving the adaptability and generalization ability of the reward model, thereby improving the training effect of the reward model, and enabling the trained reward model to quickly predict more accurate rewards, so that the game robot can obtain more accurate rewards in a timely manner through the trained reward model. This not only helps the game robot reduce invalid exploration during training, thereby improving the training efficiency of the training game robot, but also can effectively guide the game robot to imitate and learn the decisions of human players, so that the decision-making ability of the trained game robot is closer to that of human players, and the intelligence level of the trained game robot is enhanced, thereby improving the decision-making ability of the trained game robot.

[0060] Application scenarios of trained gaming robots can include AI challenges. For example, a trained gaming robot with higher decision-making ability can challenge human players in a more friendly manner, increasing playability.

[0061] In step S102 of some embodiments, feature extraction processing may be performed on the first game state to obtain a first game state feature; feature extraction processing may be performed on the first decision data to obtain a first decision feature; feature fusion processing may be performed on the first game state feature and the first decision feature to obtain a first fusion feature; the first fusion feature may be input into the reward model for training to obtain a trained reward model.

[0062] In order to facilitate the understanding of the reward model, feature extraction processing is performed on the first game state. For example, the first game state is encoded using One-hot encoding to obtain the first game state feature.

[0063] The first decision feature may be obtained by performing feature extraction processing on the first decision data, for example, encoding the first decision feature using One-hot encoding.

[0064] In this way, the first game state and the first decision data are converted into digital features that are easy to understand by the reward model.

[0065] The first game state feature and the first decision feature are subjected to feature fusion processing, for example, the first game state feature and the first decision feature are concatenated to obtain a first fusion feature.

[0066] The first fused feature is input into the reward model for training to obtain a trained reward model.

[0067] In this way, by performing feature fusion processing on the first game state feature and the first decision feature, more detailed, comprehensive and richer information can be provided for training the reward model, thereby improving the training effect of the reward model.

[0068] In some embodiments, the first fusion feature is input into the reward model for training to obtain a trained reward model. The first fusion feature can be input into the reward model for training to obtain a loss function; according to the loss function, the parameters of the reward model are iteratively updated until the number of iterations is greater than a preset first number threshold, and it is determined that the reward model converges to obtain a trained reward model.

[0069] The first fusion feature is input into the reward model for supervised training to obtain the predicted reward, and the loss function is calculated based on the predicted reward. The loss function is minimized by the gradient descent method to iteratively update the parameters of the reward model until the number of iterations is greater than the preset first number threshold, and then the update is stopped to obtain a trained reward model that meets the requirements. In this way, the training stability of the training reward model can be ensured.

[0070] See also Figure 3 In some embodiments, step S104 may include but is not limited to steps S1041 to S1044.

[0071] Step S1041, performing feature extraction processing on the second game state to obtain second game state features;

[0072] Step S1042, performing feature extraction processing on the second decision data to obtain a second decision feature;

[0073] Step S1043, performing feature fusion processing on the second game state feature and the second decision feature to obtain a second fusion feature;

[0074] Step S1044, input the second fusion feature into the trained reward model to perform reward prediction to obtain a reward.

[0075] The second game state is subjected to feature extraction processing, for example, the second game state is encoded using One-hot encoding, and the second game state feature can be obtained.

[0076] The second decision data is subjected to feature extraction processing, for example, the first decision feature is encoded using One-hot encoding to obtain the second decision feature.

[0077] The second game state feature and the second decision feature are subjected to feature fusion processing, for example, the second game state feature and the second decision feature are concatenated to obtain a second fusion feature.

[0078] The second fusion feature is input into the trained reward model for reward prediction, and the reward of the game robot output by the trained reward model is quickly obtained, which can provide effective guidance for the decision-making of the game robot and enhance its intelligent performance.

[0079] In step S105 of some embodiments, the third game state of the target game can be obtained after the game robot executes the second decision data; the second game state, the second decision data, the reward and the third game state are input into the game robot for reinforcement learning training, and the state-action value function is calculated; the state-action value function is iteratively updated until the number of iterations is greater than the preset second number threshold, to obtain a trained game robot.

[0080] Obtain a third game state to which the target game transfers after the game robot executes the second decision data.

[0081] The second game state, the second decision data, the reward and the third game state are input to the game robot for reinforcement learning training, and the state-action value function (Action-Value Function, Q value function) is updated using the Bellman equation:

[0082]

[0083] Where Q(s,a) represents the Q value function;

[0084] s represents the second game state;

[0085] a represents the second decision data;

[0086] r indicates reward;

[0087] α represents the learning rate;

[0088] γ represents the discount factor;

[0089] s′ represents the third game state;

[0090] a′ represents the decision data that may be taken under the three game states.

[0091] By iteratively updating the Q-value function, the intelligent agent of the game robot gradually learns which game states and decision data under the game states can bring higher expected returns, until the number of iterations is greater than the preset second number threshold, forming a more accurate Q-value function and obtaining a trained game robot.

[0092] In this way, by inputting game status, decision data, rewards and new game status into the game robot for training, the game robot can continuously adjust and optimize its strategy, calculate the expected return of taking different decision data under each game state, so that the trained game robot can better adapt to different game scenarios and make smarter decisions.

[0093] In some embodiments, the trained reward model may also be optimized and updated. Specifically, a new first game state of a new human player playing the target game and new first decision data under the new first game state may be obtained regularly, so as to effectively optimize and update the trained reward model using the new first game state and the new first decision data, thereby ensuring that the trained reward model maintains performance and adaptability in a changing game environment.

[0094] In some embodiments, the trained game robot can also be optimized and updated. Specifically, the new second game state of the trained game robot in the target game and the new second decision data under the new second game state can be obtained regularly, so as to predict the reward based on the new second game state and the new second decision data through the trained reward model, and obtain the reward of the trained game robot. Finally, based on the new second game state, the new second decision data and the reward of the trained game robot, the trained game robot is effectively optimized and updated to improve its performance in the ever-changing game environment.

[0095] See also Figure 4 , Figure 4 It is a schematic block diagram of a game robot training device provided in an embodiment of the present application. The game robot training device can be configured in a computer device to execute the aforementioned game robot training method.

[0096] like Figure 4 As shown, the game robot training device 400 includes: a first acquisition module 401, a reward model training module 402, a second acquisition module 403, a reward prediction module 404 and a game robot training module 405.

[0097] A first acquisition module 401 is used to acquire a first game state of a target game and first decision data of a human player in the first game state;

[0098] A reward model training module 402, configured to train a preset reward model based on the first game state and the first decision data to obtain a trained reward model;

[0099] A second acquisition module 403 is used to acquire a second game state of the target game and second decision data of the game robot in the second game state;

[0100] A reward prediction module 404 is used to predict a reward based on the second game state and the second decision data by using the trained reward model to obtain a reward for the game robot;

[0101] The game robot training module 405 is used to perform reinforcement learning training on the game robot based on the second game state, the second decision data and the reward to obtain a trained game robot.

[0102] In one embodiment, the reward model training module 402 is further used to:

[0103] Performing feature extraction processing on the first game state to obtain a first game state feature;

[0104] Performing feature extraction processing on the first decision data to obtain a first decision feature;

[0105] Performing feature fusion processing on the first game state feature and the first decision feature to obtain a first fusion feature;

[0106] The first fusion feature is input into the reward model for training to obtain the trained reward model.

[0107] In one embodiment, the reward model training module 402 is further used to:

[0108] Inputting the first fusion feature into the reward model for training to obtain a loss function;

[0109] According to the loss function, the parameters of the reward model are iteratively updated until the number of iterations is greater than a preset first number threshold, thereby obtaining the trained reward model.

[0110] In one embodiment, the reward prediction module 404 is further configured to:

[0111] Performing feature extraction processing on the second game state to obtain a second game state feature;

[0112] Performing feature extraction processing on the second decision data to obtain a second decision feature;

[0113] Performing feature fusion processing on the second game state feature and the second decision feature to obtain a second fused feature;

[0114] The second fusion feature is input into the trained reward model to perform reward prediction to obtain the reward.

[0115] In one embodiment, the game robot training module 405 is further used to:

[0116] Acquire a third game state of the target game after the game robot executes the second decision data;

[0117] Inputting the second game state, the second decision data, the reward and the third game state into the game robot for reinforcement learning training, and calculating the state-action value function;

[0118] The state-action value function is iteratively updated until the number of iterations is greater than a preset second number threshold, thereby obtaining a trained game robot.

[0119] In one embodiment, the game robot training device 400 further includes an optimization update module, which is used to:

[0120] The trained reward model is optimized and updated.

[0121] In one embodiment, the optimization and updating module is further used for:

[0122] The trained game robot is optimized and updated.

[0123] Among them, each module in the above-mentioned game robot training device 400 corresponds to each step in the above-mentioned game robot training method embodiment, and its functions and implementation processes will not be repeated here one by one.

[0124] The game robot training device 400 can execute the game robot training method provided in the embodiment of the present application. Therefore, it can achieve the beneficial effects that can be achieved by the game robot training method provided in the embodiment of the present application. Please refer to the previous embodiment for details and will not be repeated here.

[0125] The method and apparatus of the present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0126] Exemplarily, the above method and apparatus may be implemented in the form of a computer program. The computer program may be implemented in the form of a computer program. Figure 5 Runs on the computer device shown.

[0127] See also Figure 5 , Figure 5 It is a schematic block diagram of the structure of a computer device provided in an embodiment of the present application.

[0128] See also Figure 5 The computer device includes a processor and a memory connected via a system bus, wherein the memory may include a non-volatile storage medium and an internal memory.

[0129] The processor is used to provide computing and control capabilities and support the operation of the entire computer equipment.

[0130] The internal memory provides an environment for the operation of the computer program in the non-volatile storage medium. When the computer program is executed by the processor, the processor can execute any training method for the game robot.

[0131] It should be understood that the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0132] In one embodiment, the processor is used to run a computer program stored in the memory to implement the following steps:

[0133] Acquire a first game state of a target game and first decision data of a human player in the first game state;

[0134] Based on the first game state and the first decision data, training a preset reward model to obtain a trained reward model;

[0135] Acquire a second game state of the target game and second decision data of the game robot in the second game state;

[0136] Using the trained reward model, a reward prediction is performed based on the second game state and the second decision data to obtain a reward for the game robot;

[0137] Based on the second game state, the second decision data and the reward, reinforcement learning training is performed on the game robot to obtain a trained game robot.

[0138] In one embodiment, when the processor implements the training of a preset reward model based on the first game state and the first decision data to obtain a trained reward model, the processor is used to implement:

[0139] Performing feature extraction processing on the first game state to obtain a first game state feature;

[0140] Performing feature extraction processing on the first decision data to obtain a first decision feature;

[0141] Performing feature fusion processing on the first game state feature and the first decision feature to obtain a first fusion feature;

[0142] The first fusion feature is input into the reward model for training to obtain the trained reward model.

[0143] In one embodiment, when the processor implements the inputting of the first fusion feature into the reward model for training to obtain the trained reward model, it is configured to implement:

[0144] Inputting the first fusion feature into the reward model for training to obtain a loss function;

[0145] According to the loss function, the parameters of the reward model are iteratively updated until the number of iterations is greater than a preset first number threshold, thereby obtaining the trained reward model.

[0146] In one embodiment, when the processor implements the reward prediction based on the second game state and the second decision data through the trained reward model to obtain the reward of the game robot, it is used to implement:

[0147] Performing feature extraction processing on the second game state to obtain a second game state feature;

[0148] Performing feature extraction processing on the second decision data to obtain a second decision feature;

[0149] Performing feature fusion processing on the second game state feature and the second decision feature to obtain a second fused feature;

[0150] The second fusion feature is input into the trained reward model to perform reward prediction to obtain the reward.

[0151] In one embodiment, when the processor performs reinforcement learning training on the game robot based on the second game state, the second decision data and the reward to obtain a trained game robot, the processor is used to implement:

[0152] Acquire a third game state of the target game after the game robot executes the second decision data;

[0153] Inputting the second game state, the second decision data, the reward and the third game state into the game robot for reinforcement learning training, and calculating the state-action value function;

[0154] The state-action value function is iteratively updated until the number of iterations is greater than a preset second number threshold, thereby obtaining a trained game robot.

[0155] In one embodiment, the processor is used to run a computer program stored in the memory, and is also used to implement:

[0156] The trained reward model is optimized and updated.

[0157] In one embodiment, the processor is used to run a computer program stored in the memory, and is also used to implement:

[0158] The trained game robot is optimized and updated.

[0159] The computer device can execute the game robot training method provided in the embodiment of the present application, and therefore can achieve the beneficial effects that can be achieved by the game robot training method provided in the embodiment of the present application. Please refer to the previous embodiment for details and will not be repeated here.

[0160] An embodiment of the present application also provides a computer-readable storage medium.

[0161] The computer-readable storage medium of the present application stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned method for training a gaming robot are implemented.

[0162] The computer-readable storage medium may be an internal storage unit of the training device or computer device of the game robot described in the above embodiments, such as a hard disk or memory of the training device or computer device of the game robot. The computer-readable storage medium may also be an external storage device of the training device or computer device of the game robot, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital card (Secure Digital Card, SD Card), a flash card (FlashCard), etc., equipped on the training device or computer device of the game robot.

[0163] Since the computer program stored in the computer-readable storage medium can execute any one of the game robot training methods provided in the embodiments of the present application, the beneficial effects that can be achieved by any one of the game robot training methods provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0164] Furthermore, the computer-readable storage medium may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function, etc.; the data storage area may store data created according to the use of the blockchain node, etc.

[0165] The blockchain referred to in this application is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.

[0166] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0167] The above description is only a specific implementation mode of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or substitutions within the technical scope disclosed in the present application, and these modifications or substitutions should be included in the protection scope of the present application.

Claims

1. A method for training a game robot, characterized in that: The method comprises: Acquire a first game state of a target game and first decision data of a human player in the first game state; Based on the first game state and the first decision data, training a preset reward model to obtain a trained reward model; Acquire a second game state of the target game and second decision data of the game robot in the second game state; Using the trained reward model, a reward prediction is performed based on the second game state and the second decision data to obtain a reward for the game robot; Based on the second game state, the second decision data and the reward, reinforcement learning training is performed on the game robot to obtain a trained game robot.

2. The method according to claim 1, characterized in that The step of training a preset reward model based on the first game state and the first decision data to obtain a trained reward model includes: Performing feature extraction processing on the first game state to obtain a first game state feature; Performing feature extraction processing on the first decision data to obtain a first decision feature; Performing feature fusion processing on the first game state feature and the first decision feature to obtain a first fusion feature; The first fusion feature is input into the reward model for training to obtain the trained reward model.

3. The method according to claim 2, characterized in that The step of inputting the first fusion feature into the reward model for training to obtain the trained reward model includes: Inputting the first fusion feature into the reward model for training to obtain a loss function; According to the loss function, the parameters of the reward model are iteratively updated until the number of iterations is greater than a preset first number threshold, thereby obtaining the trained reward model.

4. The method according to claim 1, characterized in that: The step of performing reward prediction based on the trained reward model and the second game state and the second decision data to obtain a reward for the game robot includes: Performing feature extraction processing on the second game state to obtain a second game state feature; Performing feature extraction processing on the second decision data to obtain a second decision feature; Performing feature fusion processing on the second game state feature and the second decision feature to obtain a second fused feature; The second fusion feature is input into the trained reward model to perform reward prediction to obtain the reward.

5. The method according to claim 1, characterized in that The step of performing reinforcement learning training on the game robot based on the second game state, the second decision data and the reward to obtain a trained game robot comprises: Acquire a third game state of the target game after the game robot executes the second decision data; Inputting the second game state, the second decision data, the reward and the third game state into the game robot for reinforcement learning training, and calculating the state-action value function; The state-action value function is iteratively updated until the number of iterations is greater than a preset second number threshold, thereby obtaining a trained game robot.

6. The method according to any one of claims 1 to 5, characterized in that: Also includes: The trained reward model is optimized and updated.

7. The method according to any one of claims 1 to 5, characterized in that: Also includes: The trained game robot is optimized and updated.

8. A training device for a gaming robot, characterized in that: The device comprises: A first acquisition module, used to acquire a first game state of a target game and first decision data of a human player in the first game state; A reward model training module, used to train a preset reward model based on the first game state and the first decision data to obtain a trained reward model; A second acquisition module, used to acquire a second game state of the target game and second decision data of the game robot in the second game state; A reward prediction module, used to predict a reward based on the second game state and the second decision data by using the trained reward model to obtain a reward for the game robot; The game robot training module is used to perform reinforcement learning training on the game robot based on the second game state, the second decision data and the reward to obtain a trained game robot.

9. A computer device, characterized in that: The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the game robot training method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method for training a game robot according to any one of claims 1 to 7 is implemented.

Images