Game robot training method, device, equipment and storage medium

Abstract

The embodiments of the present application provide a method, device, equipment and storage medium for training a game robot, which belongs to the field of artificial intelligence technology. The method includes: obtaining the state of the current chessboard of the elimination game where the game robot is located; performing feature extraction processing on the state of the current chessboard to obtain the image-like features of the objects to be eliminated in the current chessboard and the vector features of the levels; performing feature fusion processing on the image-like features and the vector features to obtain the fusion features of the current chessboard; based on the fusion features of the current chessboard, the game robot plays a self-game to generate training data; based on the training data, the game robot is subjected to reinforcement learning training based on PPO to obtain a trained game robot. The embodiments of the present application can reduce training costs and provide a game robot with higher decision-making ability and faster decision-making speed.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method, device, equipment and storage medium for training a gaming robot. Background Art

[0002] Elimination games are one of the most popular puzzle-type casual games. While giving players the pleasure of eliminating and clearing levels, they also test their decision-making ability. With the in-depth research of artificial intelligence, simulating human player decision-making through game robots is one of the research directions of elimination games.

[0003] The game robot provided in the related art includes a supervised learning model and a Monte Carlo Tree Search (MCTS) algorithm model trained based on the decision data of human players. However, since the performance of the supervised learning model is closely related to the decision data of human players, its decision-making ability is difficult to exceed the level of human players. Although the MCTS algorithm model can make better decisions through search, the decision-making speed is very slow.

[0004] In addition, with the development of elimination games, in order to improve the fun of elimination games, many new special game elements will be continuously added to elimination games. Every time new game elements are added, game robots need to undergo "surgical trimming", that is, the game robots need to be repeatedly adjusted and optimized, increasing training costs.

[0005] Therefore, how to provide gaming robots with higher decision-making capabilities and faster decision-making speeds while reducing training costs has become a technical problem that needs to be solved urgently. Summary of the invention

[0006] The main purpose of the embodiments of the present application is to propose a method, device, equipment and storage medium for training a gaming robot, aiming to reduce training costs and provide a gaming robot with higher decision-making ability and faster decision-making speed.

[0007] To achieve the above-mentioned purpose, a first aspect of an embodiment of the present application proposes a method for training a game robot, the method comprising:

[0008] Get the state of the current chessboard where the game robot is located;

[0009] Performing feature extraction processing on the state of the current chessboard to obtain image-like features of objects eliminated in the current chessboard and vector features of checkpoints;

[0010] Performing feature fusion processing on the image-like features and the vector features to obtain fusion features of the current chessboard;

[0011] The game robot plays against itself based on the fusion features of the current chessboard to generate training data;

[0012] Based on the training data, the game robot is trained with PPO-based reinforcement learning to obtain a trained game robot.

[0013] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides a training device for a gaming robot, the device comprising:

[0014] A state acquisition module is used to obtain the state of the current chessboard where the game robot is located in the elimination game;

[0015] A feature extraction module, used to perform feature extraction processing on the state of the current chessboard to obtain image-like features of objects to be eliminated in the current chessboard and vector features of checkpoints;

[0016] A feature fusion module, used for performing feature fusion processing on the image-like features and the vector features to obtain fusion features of the current chessboard;

[0017] A training data generating module, used for performing a self-game of the game robot based on the fusion features of the current chessboard to generate training data;

[0018] The reinforcement learning training module is used to perform PPO-based reinforcement learning training on the game robot based on the training data to obtain a trained game robot.

[0019] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes a computer device, which includes a memory and a processor, the memory stores a computer program, and the processor implements the method described in the first aspect when executing the computer program.

[0020] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0021] The present application proposes a method, device, equipment and storage medium for training a game robot. The method for training a game robot obtains the state of a current chessboard of an elimination game in which the game robot is located; performs feature extraction processing on the state of the current chessboard to obtain image-like features of objects to be eliminated in the current chessboard and vector features of levels; performs feature fusion processing on the image-like features and vector features to obtain fusion features of the current chessboard; based on the fusion features of the current chessboard, the game robot plays against itself to generate training data; based on the training data, performs PPO-based reinforcement learning training on the game robot to obtain a trained game robot. In this way, by extracting the image-like features of the objects to be eliminated and the vector features of the levels in the current chessboard from the state of the current chessboard, a more detailed feature extraction effect can be achieved. By performing feature fusion processing on the image-like features of the eliminated objects and the vector features of the levels, the fused features of the current chessboard are obtained, which can provide more useful, comprehensive and richer information for training game robots. Based on the fused features of the current chessboard, the game robot can be trained by generating training data with a higher quality than the decision-making data of human players through self-playing games. This not only reduces the training cost, but also improves the training effect of the game robot, thereby improving the performance of the trained game robot, and then improving the decision-making speed of the trained game robot, so that the decision-making ability of the trained game robot is higher than that of human players. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 is a flow chart of a method for training a gaming robot provided in an embodiment of the present application;

[0023] Figure 2 is an example diagram of a chessboard for an elimination game provided in an embodiment of the present application;

[0024] Figure 3 is a schematic diagram of the architecture of a gaming robot provided in an embodiment of the present application;

[0025] Figure 4 Schematic diagram of a training architecture for a game robot provided in an embodiment of the present application;

[0026] Figure 5 is a schematic block diagram of a training device for a gaming robot provided in an embodiment of the present application;

[0027] Figure 6 It is a schematic block diagram of the structure of the computer device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0029] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0030] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0031] First, some nouns involved in this application are analyzed:

[0032] Artificial intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing and expert systems. AI can simulate the information process of human consciousness and thinking. AI is also a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0033] Reinforcement Learning (RL): aims to learn the best strategy through interaction with the environment. It uses a trial-and-error method to enable the agent to choose actions in a specific state, and adjust its behavior based on the reward or penalty feedback obtained to maximize long-term rewards. The core elements of reinforcement learning include agent, environment, state, action, and reward. Among them, agent refers to the learner who performs actions; environment refers to the external system with which the agent interacts; state refers to the specific situation of the environment, the situation in which the agent is at a certain moment. Action refers to the behavior that the agent can take in a specific state. Reward refers to the feedback obtained after the agent performs an action, which is used to evaluate the quality of the action. Through continuous cycles of exploration and utilization, reinforcement learning can effectively learn the best decision-making strategies in complex tasks. For the game robot in the embodiment of the present application, the elimination game scene can be understood as the environment.

[0034] PPO (Proximal Policy Optimization): is a reinforcement learning algorithm that aims to improve the stability and efficiency of the policy optimization process. It limits the magnitude of policy updates to avoid performance instability caused by excessive updates. The core idea of ​​PPO is to ensure that there is not much change between the new and old policies through constrained updates to maintain the stability of the learning process. This method can effectively avoid the problem of policy collapse. By limiting updates, PPO significantly improves the stability of training. Compared with other reinforcement learning algorithms, PPO is simpler to implement and has high computational efficiency.

[0035] Based on this, the embodiments of the present application provide a method, device, equipment and storage medium for training a gaming robot, aiming to reduce training costs and provide a gaming robot with higher decision-making ability and faster decision-making speed.

[0036] The training method, device, equipment and storage medium for the gaming robot provided in the embodiments of the present application are specifically described through the following embodiments. First, the training method for the gaming robot in the embodiments of the present application is described.

[0037] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0038] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0039] The training method for a game robot provided in an embodiment of the present application relates to the field of artificial intelligence technology. The training method for a game robot provided in an embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a vehicle-mounted terminal, a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or can be configured as a server cluster or a distributed system composed of multiple physical servers, or can be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the training method for a game robot, etc., but is not limited to the above forms.

[0040] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer computer devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0041] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0042] Figure 1 is an optional flow chart of the method for training a gaming robot provided in an embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S105.

[0043] Step S101, obtaining the state of the current chessboard of the elimination game where the game robot is located.

[0044] The elimination game is played on a chessboard. Figure 2 , Figure 2 This is an example of a chessboard for an elimination game. The size of the chessboard is H*W, which means there are H columns and W rows of squares on the chessboard. Each square has an elimination object (elimination object can also be called an element), which means there are H*W elimination objects on the chessboard. Different colors and patterns (such as fruits, gems, stars, etc.) are used to represent different elimination objects. Players can eliminate three or more identical elimination objects by swapping left and right, right and left, up and down, down and up, clicking, and other elimination actions to form a line to eliminate them and complete the goal of a level. The goal of a level is, for example, to eliminate a certain number of elimination objects, or to eliminate a certain color of elimination objects, or to use a certain number of props, or to pass the level within a limited number of steps, or to pass the level within a limited time.

[0045] See also Figure 3 , Figure 3 The schematic diagram of the game robot architecture includes a residual network (Resnet) layer, a fully connected layer (FC), a value network, and a policy network. The value network and the policy network share two FCs.

[0046] See also Figure 4 , Figure 4 This is a diagram of the training architecture for training game robots. The training architecture includes three parts: Actor, Redis Server, and Learner.

[0047] Among them, Actor is responsible for interacting with the self-play simulation program, and it is used to collect the training data generated by the self-play simulation program. It should be noted that the self-play simulation program is used to create a simulated elimination game in which the game robot plays multiple self-play games (that is, multiple games with itself), generating a large amount of training data.

[0048] Redis Server is used to store the training data collected by the Actor, so that subsequent reinforcement learning can be performed based on the training data, supporting more efficient training data utilization and a more stable learning process.

[0049] Learner is responsible for learning and updating the value network and policy network. It uses the training data in the Redis Server to update the parameters of the value network and policy network to improve the decision-making ability of the policy network. Learner can regularly obtain training data from the Redis Server to perform reinforcement learning training on the game robot.

[0050] Such a training architecture can improve the efficiency and stability of training, making the PPO-based reinforcement learning process more effective.

[0051] The following combination Figure 2 , Figure 3 and Figure 4 , the technical solution provided in the embodiments of the present application is described in detail.

[0052] The game robot plays against itself in the elimination game created by the self-playing simulation program. After completing an elimination action, the self-playing simulation program will provide real-time feedback on the state of a chessboard frame of the elimination game, that is, the state of the current chessboard of the elimination game where the game robot is located.

[0053] Get the current state of the chessboard from the self-playing simulation program through Actor.

[0054] Step S102, performing feature extraction processing on the state of the current chessboard to obtain image-like features of the objects to be eliminated in the current chessboard and vector features of the checkpoints.

[0055] In order to facilitate the understanding of the game robot, the feature extraction process is performed on the current state of the chessboard to obtain the image-like features of the objects eliminated in the current chessboard and the vector features of the levels, thereby achieving a more detailed feature extraction effect.

[0056] In step S102 of some embodiments, the current state of the chessboard includes the attributes of the eliminated object and the attributes of the target of the level. The attributes of the eliminated object can be encoded to obtain image-like features; the attributes of the target can be encoded to obtain vector features.

[0057] Since the current chessboard has H*W elimination objects distributed thereon and a goal of a level (for example, eliminating p elimination objects of a specified color), the elimination objects have attributes and the goals also have attributes. Therefore, the state of the current chessboard includes the attributes of the elimination objects in the current chessboard and the attributes of the goals of the level.

[0058] Exemplarily, the properties of the elimination object include gravity properties, such as the four gravity directions of up, down, left, and right; the properties of the target include elimination method properties, such as the four elimination methods of ineliminable, ordinary elimination, prop elimination, and adjacent elimination.

[0059] For example, the attributes of the eliminated objects can be encoded using a one-hot encoding method, so that all eliminated objects are mapped to an H*W*N image-like feature at the H*W grid position, where N represents the number of channel layers of the image-like feature. For example, if the gravity attribute of an eliminated object is the gravity direction below, a four-layer image-like feature can be obtained:

[0060] Layer 1(Up): 0

[0061] Layer 2(Down): 1

[0062] Layer 3(Left): 0

[0063] Layer 4 (Right): 0

[0064] In addition, if there are multiple elements in the same grid position, their attributes will be superimposed. For example, if the attribute of the elimination object is health (health means that the elimination object needs to be eliminated multiple times), if there are two elimination objects with 4 health points in a grid position, then the attribute of the elimination object in this grid position will eventually be 8 health points.

[0065] Similarly, the one-hot encoding method can be used to encode the attributes of the target to obtain vector features. For example, if the attribute of the target is not eliminable, after one-hot encoding, a four-bit vector feature of -1000 can be obtained; if the attribute of the target is ordinary elimination, after one-hot encoding, a four-bit vector feature of -0100 can be obtained; if the attribute of the target is prop elimination, after one-hot encoding, a four-bit vector feature of -0010 can be obtained; if the attribute of the target is adjacent elimination, after one-hot encoding, a four-bit vector feature of -0001 can be obtained.

[0066] Through the above method, from the perspective of the attributes of the elimination objects and the attributes of the level targets, the state of the chessboard can be converted into more detailed and comprehensive digital features, which are easy for the game robot to understand. Even when faced with new elimination objects and new level targets that the game robot has never seen before, feature extraction processing can be performed well from the attribute perspective, which can improve the generalization ability of the game robot.

[0067] Step S103, performing feature fusion processing on the image-like features and the vector features to obtain fusion features of the current chessboard.

[0068] The image-like features of the eliminated objects and the vector features of the target are subjected to feature fusion processing to obtain the fusion features of the current chessboard.

[0069] like Figure 3 As shown, specifically, the image-like features of the eliminated object are input into the Resnet layer for processing, the vector features of the target are input into the three FCs for processing, and then the output of the Resnet layer is concatenated with the final output of the three FCs to obtain the fusion features of the current chessboard.

[0070] In this way, by performing feature fusion processing on the image-like features of the eliminated objects and the vector features of the targets of the levels, more useful, comprehensive and richer information can be provided for training game robots, thereby improving the training effect of game robots.

[0071] Step S104, the game robot plays against itself based on the fusion features of the current chessboard to generate training data.

[0072] Based on the fusion features of the current chessboard, the game robot is allowed to play against itself to generate training data.

[0073] In step S104 of some embodiments, the game robot can play against itself based on the fusion features to obtain the elimination action labels of the game robot; obtain the rewards, values ​​and the state of the next chessboard generated by the game robot executing the elimination action labels; and obtain the fusion features of the next chessboard based on the state of the next chessboard, wherein the fusion features of the current chessboard, the elimination action labels, rewards, values ​​and the fusion features of the next chessboard constitute the training data.

[0074] When playing against itself, the game robot can use the value network and the strategy network to predict the elimination action according to the fusion characteristics of the current chessboard, obtain the elimination action label of the game robot in the current chessboard state, and then the game robot executes the elimination action label to obtain the reward generated after the game robot executes the elimination action label, the value of the current chessboard state and the state of the next chessboard.

[0075] like Figure 3 As shown, specifically, the fusion features of the current chessboard are input into the value network and the strategy network for processing. The value network calculates and outputs the value of the current chessboard state, and the strategy network outputs the probability of the elimination action label predicted under the current chessboard state. When the probability of the elimination action label is greater than or equal to the preset probability threshold, the elimination action label is used as the elimination action label of the game robot under the current chessboard state. In order to facilitate the analysis of the game robot, the executable elimination actions in the elimination game are abstracted into multiple elimination action labels.

[0076] The calculation of value is related to the goal of the level, which aims to make the game robot more anthropomorphic, thereby making the game robot behave more naturally and intelligently.

[0077] For example, suppose there are two goals in a level, and the goal is to pass the level within a limited number of steps (which can be referred to as passing the level). The value V(s t ) must take into account the remaining steps in the case of not clearing the level and the remaining steps in the case of clearing the level. Specifically, the value V(s) of the current state of the chessboard can be calculated in the following two ways: t ).

[0078] Method 1 is as follows:

[0079] V(s t )=r t +(S l / S a ), r t =1

[0080] V(s t )=r t -(T l,1 / T a,1+T l,2 / T a,2 ) / 2, r t =-1

[0081] Method 2 is as follows:

[0082] V(s t )=r t +S l *0.1, r t =1

[0083] V(s t )=r t -(T l,1 / T a,1 +T l,2 / T a,2 ) / 2, r t =-1

[0084] Among them, r t Indicates the reward, the value is 1 if the level is cleared, and -1 if the level is not cleared;

[0085] S l Indicates the number of remaining steps in the case of clearing the level;

[0086] S a Indicates the total number of steps in the level;

[0087] T l,1 Indicates the number of steps remaining to the first goal of the level if the level has not been cleared;

[0088] T a,1 Indicates the total number of steps for the first goal;

[0089] T l,2 Indicates the number of steps remaining to the second goal of the level if the level has not been cleared;

[0090] T a,2 Indicates the total number of steps to the second goal.

[0091] That is, in the case of not clearing the level, the V(s) calculated by the above two methods are t ) is the same as above, except that the game robot is given a basic -1 penalty. If the number of targets eliminated is greater, V(s t ) is larger, if the number of eliminated targets is smaller, V(s t ) is smaller.

[0092] In the case of clearing a level, the more steps are left, the higher V(s t). The difference is that in method 1, the additional reward is normalized within the range of 0 to 1, which allows the game robot to pay more attention to successfully passing the level and has a relatively weak perception of the remaining steps; in method 2, no normalization is performed, which allows the game robot to better weigh the relationship between passing the level and the remaining steps, so that it can be more flexible in strategy selection.

[0093] In other words, by calculating V(s t ), which can make the game robot's strategy selection decision give priority to clearing the level; calculate V(s) by method 2 t ), which enables the game robot’s strategy selection decision to consider both the level-clearing and the remaining steps, prompting the game robot to seek fewer steps while clearing the level.

[0094] The state of the next chessboard is also subjected to feature extraction processing to obtain the image-like features of the objects eliminated in the next chessboard and the vector features of the targets of the checkpoints, and then the image-like features of the objects eliminated in the next chessboard and the vector features of the targets of the checkpoints are subjected to feature fusion processing to obtain the fusion features of the next chessboard.

[0095] In this way, the Actor can collect the fusion features of the current board, eliminate actions, rewards, values, and the fusion features of the next board, which can constitute training data. This cycle can continuously generate a large amount of training data with higher quality than the decision-making data of human players.

[0096] In some embodiments, Figure 4 As shown, a level selection mechanism can also be set for the self-playing simulation program, the purpose of which is to encourage the game robot to play self-play games more often at more difficult levels to generate training data.

[0097] Since the clearance rate of a level can be used as an indicator to measure the difficulty of the level, the lower the clearance rate, the higher the level difficulty is usually, so the level selection mechanism is designed to select the level according to the clearance rate of the level. For example, assuming that the elimination game has n levels, the probability of selecting the i-th level is:

[0098]

[0099] Among them, P(i) represents the selection probability of the i-th level; W(i) represents the clearance rate of the i-th level.

[0100] In this way, it can ensure that the game robot plays against itself more often at more difficult levels to generate training data, so that the training data of more difficult levels can be used to train the game robot, ensuring that the game robot can be exposed to more challenging levels during training, thereby improving the overall performance and adaptability of the game robot and improving the performance of the game robot.

[0101] In some embodiments, after step S104, the training data may also be stored in a preset cache area; when a training request for performing reinforcement learning training on the game robot is received, the training data is read from the preset cache area.

[0102] Through Redis Server, the training data collected by the Actor is stored in the preset cache area - Redis, realizing the reasonable management of the training data.

[0103] Learner can send a training request to Redis Server to perform reinforcement learning training on the game robot. Redis Server responds to the training request, pulls the training data from Redis and sends it to Learner, thereby improving the training stability.

[0104] Step S105, based on the training data, the game robot is subjected to PPO-based reinforcement learning training to obtain a trained game robot.

[0105] Through Learner, according to the training data, the value network and strategy network of the game robot are trained with PPO-based reinforcement learning until the value network and strategy network converge to obtain a trained game robot.

[0106] In step S105 of some embodiments, training data can be input into the game robot; cumulative rewards and advantage functions are obtained through the value network; the agent loss of the policy network is obtained based on the advantage function; the entropy loss of the policy network is obtained; based on the cumulative rewards, the value network is updated; based on the agent loss and the entropy loss, the policy network is updated.

[0107] The training data is input into the game robot, and the cumulative reward and advantage function (Generalized Advantage Estimation, GAE) are obtained through the value network. The advantage function represents the relative value of eliminating the action label, which is used to help the policy network better predict the probability of eliminating the action label.

[0108] Specifically, the training data includes the fusion feature s of the current chessboard t 、Game robot elimination action tag a t , Reward t , the value of the current chessboard state V(s t ) and the fusion feature s of the next chessboard t+1 The training data is input to the game robot, and the value V(s) of the next chessboard state is obtained through the value network. t+1 ) and cumulative reward R t .

[0109] In this way, the value network can be updated based on the accumulated rewards.

[0110] For the update of the policy network, first according to V(s t ) and V(s t+1 ), the advantage function is calculated and the advantage function is as follows:

[0111] A t =r t +γV(s t+1 )-V(s t )

[0112] Here, γ represents the discount factor and t represents the time step.

[0113] Based on the advantage function, the surrogate loss of the policy network is obtained. The surrogate loss is used to guide the policy optimization of the policy network, which can better utilize the advantage function and avoid the policy network's policy being updated too much. The surrogate loss is as follows:

[0114] L CLIP (θ) = E t [min(r t (θ)A t ,clip(r t (θ),1-ε,1+ε)A t ]

[0115] Among them, E t Indicates the current state of the chessboard s t Next, take the elimination action label a t Expected reward;

[0116] represents the probability ratio of the new and old strategies;

[0117] clip(r t (θ),1-ε,1+ε) is used to limit r t The value of (θ) to prevent the policy update from being too large;

[0118] π represents the current state of the chessboard s t The strategy selection of the next policy network eliminates the action label a t probability;

[0119] θ represents the parameters of the policy network;

[0120] ε represents the clipping threshold.

[0121] It should be noted that by adjusting θ, PPO can gradually improve the strategy of the policy network to choose to eliminate action labels, thereby controlling the amplitude of the update and ensuring the stability of the training.

[0122] The entropy loss of the policy network is also obtained. The entropy loss is used to enhance the policy exploratory nature by encouraging the policy to predict more random elimination of action labels to improve exploratory nature and avoid premature convergence to suboptimal policies. The entropy loss is shown below:

[0123]

[0124] In this way, the policy network can be updated based on the proxy loss and entropy loss.

[0125] In some embodiments, updating the value network based on the accumulated rewards may include calculating a first loss function of the value network according to the accumulated rewards, and updating the value network according to the first loss function.

[0126] In some embodiments, updating the policy network based on the proxy loss and the entropy loss may be calculating a second loss function of the policy network according to the proxy loss and the entropy loss; and updating the policy network according to the second loss function.

[0127] The L2 loss function can be used as the first loss function of the value network to make the value network predict more accurately, thereby better guiding the policy network to optimize its strategy. The first loss function is as follows:

[0128]

[0129] Here, n represents the number of training data.

[0130] The proxy loss and entropy loss can be combined to obtain the second loss function of the policy network. The second loss function is as follows:

[0131] L(θ)=L CLIP (θ)-βH(π)

[0132] Among them, β represents the hyperparameter of the policy network, which is used to control the impact of entropy loss.

[0133] In this way, the value network is updated according to the first loss function, and the strategy network is updated according to the second loss function, until the number of updates reaches a preset update number threshold, it is determined that the value network and the strategy network converge, and a trained game robot is obtained.

[0134] Since the training data covers more difficult levels and decision-making scenarios, the quality of the training data is higher than the decision-making data of human players. Therefore, the decision-making ability of trained game robots exceeds the human level.

[0135] In some embodiments, after the Learner obtains the trained game robot, it can also evaluate the trained game robot. Exemplary, for example, the trained game robot is used to pass the level multiple times (such as 10,000 times), and the average clearance rate of the game robot is calculated. When the average clearance rate of the trained game robot is higher than the average clearance rate of human players, the trained game robot can be deployed. The application scenarios of the trained game robot include level-breaking teaching (such as instructing players how to pass difficult levels / normal levels), AI challenge (such as for high-level players, using trained game robots to challenge players and increase activity), level anomaly detection (such as when developing levels of elimination games, using trained game robots to pass levels to observe whether there are levels that cannot be passed), etc.

[0136] In some embodiments, the trained game robot can also be used to play games against itself regularly to generate new training data. Since the performance of the trained game robot is better, the quality of the new training data generated is also higher. In this way, the trained game robot can be further optimized with the new training data of higher quality. In this way, the training data can be continuously updated, so that the game robot can be continuously optimized and improved, so that the decision-making ability can be continuously improved.

[0137] The training method for a game robot provided in the above embodiment obtains the state of the current chessboard of the elimination game in which the game robot is located; performs feature extraction processing on the state of the current chessboard to obtain image-like features of the objects to be eliminated in the current chessboard and vector features of the levels; performs feature fusion processing on the image-like features and the vector features to obtain fusion features of the current chessboard; based on the fusion features of the current chessboard, the game robot plays against itself to generate training data; based on the training data, the game robot performs PPO-based reinforcement learning training to obtain a trained game robot. In this way, by extracting the image-like features of the objects to be eliminated and the vector features of the levels in the current chessboard from the state of the current chessboard, a more detailed feature extraction effect can be achieved. By performing feature fusion processing on the image-like features of the eliminated objects and the vector features of the levels, the fused features of the current chessboard are obtained, which can provide more useful, comprehensive and richer information for training game robots. Based on the fused features of the current chessboard, the game robot can be trained by generating training data with a higher quality than the decision-making data of human players through self-playing games. This not only reduces the training cost, but also improves the training effect of the game robot, thereby improving the performance of the trained game robot, and then improving the decision-making speed of the trained game robot, so that the decision-making ability of the trained game robot is higher than that of human players.

[0138] See also Figure 5 , Figure 5It is a schematic block diagram of a game robot training device provided in an embodiment of the present application. The game robot training device can be configured in a computer device to execute the aforementioned game robot training method.

[0139] like Figure 5 As shown, the game robot training device 500 includes: a state acquisition module 501, a feature extraction module 502, a feature fusion module 503, a training data generation module 504 and a reinforcement learning training module 505.

[0140] The state acquisition module 501 is used to acquire the state of the current chessboard of the elimination game where the game robot is located;

[0141] A feature extraction module 502 is used to perform feature extraction processing on the state of the current chessboard to obtain image-like features of objects to be eliminated in the current chessboard and vector features of checkpoints;

[0142] A feature fusion module 503 is used to perform feature fusion processing on the image-like features and the vector features to obtain fusion features of the current chessboard;

[0143] A training data generating module 504 is used to generate training data by having the game robot play against itself based on the fusion features of the current chessboard;

[0144] The reinforcement learning training module 505 is used to perform PPO-based reinforcement learning training on the game robot based on the training data to obtain a trained game robot.

[0145] In one embodiment, the current state of the chessboard includes the attributes of the eliminated object and the attributes of the target of the level, and the feature extraction module 502 is further used to:

[0146] Encoding the attributes of the elimination object to obtain the class image features;

[0147] The attributes of the target are encoded to obtain the vector features.

[0148] In one embodiment, the game robot includes a value network and a strategy network, and the reinforcement learning training module 505 is further used to:

[0149] Inputting the training data into the game robot;

[0150] Obtaining cumulative rewards and advantage functions through the value network;

[0151] Obtaining a proxy loss of the policy network based on the advantage function;

[0152] Obtaining the entropy loss of the policy network;

[0153] Based on the accumulated rewards, updating the value network;

[0154] The policy network is updated based on the proxy loss and the entropy loss.

[0155] In one embodiment, the reinforcement learning training module 505 is further used to:

[0156] Calculating a first loss function of the value network according to the accumulated reward;

[0157] The value network is updated according to the first loss function.

[0158] In one embodiment, the reinforcement learning training module 505 is further used to:

[0159] Calculating a second loss function of the policy network according to the proxy loss and the entropy loss;

[0160] According to the second loss function, the policy network is updated.

[0161] In one embodiment, the training data generation module 504 is further used to:

[0162] The game robot plays against itself based on the fusion feature to obtain an elimination action label of the game robot;

[0163] Obtaining the reward, value and next chessboard state generated by the game robot executing the elimination action tag;

[0164] Based on the state of the next chessboard, a fusion feature of the next chessboard is obtained, wherein the fusion feature of the current chessboard, the elimination action label, the reward, the value and the fusion feature of the next chessboard constitute the training data.

[0165] In one embodiment, the game robot training device 500 further includes a training data storage module, and the training data storage module is used to:

[0166] Storing the training data in a preset buffer area;

[0167] When a training request for performing reinforcement learning training on the game robot is received, the training data is read from the preset buffer area.

[0168] Among them, each module in the above-mentioned game robot training device 500 corresponds to each step in the above-mentioned game robot training method embodiment, and its functions and implementation processes are no longer repeated here one by one.

[0169] The game robot training device 500 can execute the game robot training method provided in the embodiment of the present application. Therefore, it can achieve the beneficial effects that can be achieved by the game robot training method provided in the embodiment of the present application. Please refer to the previous embodiment for details and will not be repeated here.

[0170] The method and apparatus of the present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0171] Exemplarily, the above method and apparatus may be implemented in the form of a computer program. The computer program may be implemented in the form of a computer program. Figure 6 Runs on the computer device shown.

[0172] See also Figure 6 , Figure 6 It is a schematic block diagram of the structure of a computer device provided in an embodiment of the present application.

[0173] See also Figure 6 The computer device includes a processor and a memory connected via a system bus, wherein the memory may include a non-volatile storage medium and an internal memory.

[0174] The processor is used to provide computing and control capabilities and support the operation of the entire computer equipment.

[0175] The internal memory provides an environment for the operation of the computer program in the non-volatile storage medium. When the computer program is executed by the processor, the processor can execute any training method for the game robot.

[0176] It should be understood that the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0177] In one embodiment, the processor is used to run a computer program stored in the memory to implement the following steps:

[0178] Get the state of the current chessboard where the game robot is located;

[0179] Performing feature extraction processing on the state of the current chessboard to obtain image-like features of objects eliminated in the current chessboard and vector features of checkpoints;

[0180] Performing feature fusion processing on the image-like features and the vector features to obtain fusion features of the current chessboard;

[0181] The game robot plays against itself based on the fusion features of the current chessboard to generate training data;

[0182] Based on the training data, the game robot is trained with PPO-based reinforcement learning to obtain a trained game robot.

[0183] In one embodiment, the state of the current chessboard includes the attributes of the eliminated object and the attributes of the target of the checkpoint. When the processor performs feature extraction processing on the state of the current chessboard to obtain the image-like features of the eliminated object and the vector features of the checkpoint in the current chessboard, it is configured to implement:

[0184] Encoding the attributes of the elimination object to obtain the class image features;

[0185] The attributes of the target are encoded to obtain the vector features.

[0186] In one embodiment, the game robot includes a value network and a strategy network, and the processor, when implementing the PPO-based reinforcement learning training of the game robot based on the training data, is used to implement:

[0187] Inputting the training data into the game robot;

[0188] Obtaining cumulative rewards and advantage functions through the value network;

[0189] Obtaining a proxy loss of the policy network based on the advantage function;

[0190] Obtaining the entropy loss of the policy network;

[0191] Based on the accumulated rewards, updating the value network;

[0192] The policy network is updated based on the proxy loss and the entropy loss.

[0193] In one embodiment, when the processor implements the updating of the value network based on the accumulated reward, the processor is used to implement:

[0194] Calculating a first loss function of the value network according to the accumulated reward;

[0195] The value network is updated according to the first loss function.

[0196] In one embodiment, the processor, when implementing the updating of the policy network based on the proxy loss and the entropy loss, is used to implement:

[0197] Calculating a second loss function of the policy network according to the proxy loss and the entropy loss;

[0198] According to the second loss function, the policy network is updated.

[0199] In one embodiment, when the processor performs the self-playing of the game robot based on the fusion features of the current chessboard to generate training data, it is used to implement:

[0200] The game robot plays against itself based on the fusion feature to obtain an elimination action label of the game robot;

[0201] Obtaining the reward, value and next chessboard state generated by the game robot executing the elimination action tag;

[0202] Based on the state of the next chessboard, a fusion feature of the next chessboard is obtained, wherein the fusion feature of the current chessboard, the elimination action label, the reward, the value and the fusion feature of the next chessboard constitute the training data.

[0203] In one embodiment, after implementing the self-playing of the game robot based on the fusion features of the current chessboard to generate training data, the processor is further configured to implement:

[0204] Storing the training data in a preset buffer area;

[0205] When a training request for performing reinforcement learning training on the game robot is received, the training data is read from the preset buffer area.

[0206] The computer device can execute the game robot training method provided in the embodiment of the present application, and therefore can achieve the beneficial effects that can be achieved by the game robot training method provided in the embodiment of the present application. Please refer to the previous embodiment for details and will not be repeated here.

[0207] An embodiment of the present application also provides a computer-readable storage medium.

[0208] The computer-readable storage medium of the present application stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned method for training a gaming robot are implemented.

[0209] The computer-readable storage medium may be an internal storage unit of the training device or computer device of the game robot described in the above embodiments, such as a hard disk or memory of the training device or computer device of the game robot. The computer-readable storage medium may also be an external storage device of the training device or computer device of the game robot, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital card (Secure Digital Card, SD Card), a flash card (FlashCard), etc., equipped on the training device or computer device of the game robot.

[0210] Since the computer program stored in the computer-readable storage medium can execute any one of the game robot training methods provided in the embodiments of the present application, the beneficial effects that can be achieved by any one of the game robot training methods provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0211] Furthermore, the computer-readable storage medium may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function, etc.; the data storage area may store data created according to the use of the blockchain node, etc.

[0212] The blockchain referred to in this application is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.

[0213] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0214] The above description is only a specific implementation mode of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or substitutions within the technical scope disclosed in the present application, and these modifications or substitutions should be included in the protection scope of the present application.

Claims

1. A method for training a game robot, characterized in that: The method comprises: Get the state of the current chessboard where the game robot is located; Performing feature extraction processing on the state of the current chessboard to obtain image-like features of objects eliminated in the current chessboard and vector features of checkpoints; Performing feature fusion processing on the image-like features and the vector features to obtain fusion features of the current chessboard; The game robot plays against itself based on the fusion features of the current chessboard to generate training data; Based on the training data, the game robot is trained with PPO-based reinforcement learning to obtain a trained game robot.

2. The method according to claim 1, characterized in that: The state of the current chessboard includes the attributes of the eliminated object and the attributes of the target of the checkpoint, and the feature extraction process is performed on the state of the current chessboard to obtain the class image features of the eliminated object and the vector features of the checkpoint in the current chessboard, including: Encoding the attributes of the elimination object to obtain the class image features; The attributes of the target are encoded to obtain the vector features.

3. The method according to claim 1, characterized in that The game robot includes a value network and a strategy network. Based on the training data, the game robot is subjected to reinforcement learning training based on PPO, including: Inputting the training data into the game robot; Obtaining cumulative rewards and advantage functions through the value network; Obtaining a proxy loss of the policy network based on the advantage function; Obtaining the entropy loss of the policy network; Based on the accumulated rewards, updating the value network; The policy network is updated based on the proxy loss and the entropy loss.

4. The method according to claim 3, characterized in that: The updating of the value network based on the accumulated reward comprises: Calculating a first loss function of the value network according to the accumulated reward; The value network is updated according to the first loss function.

5. The method according to claim 3, characterized in that: The updating of the policy network based on the proxy loss and the entropy loss comprises: Calculating a second loss function of the policy network according to the proxy loss and the entropy loss; According to the second loss function, the policy network is updated.

6. The method according to claim 1, characterized in that The step of performing a self-game by the game robot based on the fusion features of the current chessboard to generate training data includes: The game robot plays against itself based on the fusion feature to obtain an elimination action label of the game robot; Obtaining the reward, value and next chessboard state generated by the game robot executing the elimination action tag; Based on the state of the next chessboard, a fusion feature of the next chessboard is obtained, wherein the fusion feature of the current chessboard, the elimination action label, the reward, the value and the fusion feature of the next chessboard constitute the training data.

7. The method according to claim 1, characterized in that After the game robot plays a game against itself based on the fusion features of the current chessboard to generate training data, the method further includes: Storing the training data in a preset buffer area; When a training request for performing reinforcement learning training on the game robot is received, the training data is read from the preset buffer area.

8. A training device for a gaming robot, characterized in that: The device comprises: A state acquisition module is used to obtain the state of the current chessboard where the game robot is located in the elimination game; A feature extraction module, used to perform feature extraction processing on the state of the current chessboard to obtain image-like features of objects to be eliminated in the current chessboard and vector features of checkpoints; A feature fusion module, used for performing feature fusion processing on the image-like features and the vector features to obtain fusion features of the current chessboard; A training data generating module, used for performing a self-game of the game robot based on the fusion features of the current chessboard to generate training data; The reinforcement learning training module is used to perform PPO-based reinforcement learning training on the game robot based on the training data to obtain a trained game robot.

9. A computer device, characterized in that: The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the game robot training method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method for training a game robot according to any one of claims 1 to 7 is implemented.

Images