A game robot training method and device

Abstract

The application discloses a game robot training method and device. The game robot training method comprises the following steps: acquiring a plurality of groups of game state information of a current game situation of a game robot to be trained; performing feature extraction processing on each group of game state information to obtain an embedding vector corresponding to each group of game state information; performing feature fusion processing on the embedding vectors corresponding to the plurality of groups of game state information respectively to obtain a fused embedding vector; performing reinforcement learning based on the fused embedding vector to obtain a behavior strategy of the game robot; acquiring a reward signal of the game robot based on an execution result of the game robot after the behavior strategy is executed; and training the game robot based on the reward signal and the game state information. The method solves the problem of low decision-making capability of a game robot in a computer game.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, specifically to a game robot training method, device, electronic equipment, and storage medium. Background Technology

[0002] With the development of computer game technology, game robots with human-like intelligence can be embedded in games to simulate human players' decision-making. This can allow them to fight and interact with real players as virtual players when the number of players is limited, or to provide real players with suggestions on game behavior strategies.

[0003] Existing game robots typically employ rule-based methods for game decision-making, which has the following drawbacks: First, rule-based decisions are often fixed and predictable, lacking the flexibility and higher-order rationality of human intelligence, resulting in a lack of immersion and realism. Second, the intelligence of rule-based game decisions depends on the complexity of the rules. Achieving more intelligent game decisions requires adding more rules, which complicates game maintenance and optimization.

[0004] Therefore, how to provide game robots with high decision-making capabilities is a problem that needs to be solved. Summary of the Invention

[0005] The game robot training method provided in this application solves the problem of low decision-making ability of game robots in computer games.

[0006] This application provides a method for training a game robot, comprising: acquiring multiple sets of game state information of the current game situation in which the game robot to be trained is located; performing feature extraction processing on each set of game state information to obtain an embedding vector corresponding to each set of game state information; performing feature fusion processing on the embedding vectors corresponding to the multiple sets of game state information to obtain a fused embedding vector; performing reinforcement learning based on the fused embedding vector to obtain a behavior strategy of the game robot; acquiring a reward signal of the game robot based on the execution result of the game robot executing the behavior strategy; and training the game robot based on the reward signal and the game state information.

[0007] Optionally, the step of performing feature extraction processing on each group of game state information to obtain the embedding vector corresponding to each group of game state information includes: obtaining the embedding vector corresponding to each group of game state information through a first multilayer perceptron corresponding to each group of game state information.

[0008] Optionally, obtaining multiple sets of game state information in the game situation where the game robot to be trained is located includes: obtaining multiple sets of game state information in the game situation where the game robot to be trained is located; and grouping the multiple sets of game state information based on the semantic correlation between the multiple sets of game state information to obtain multiple sets of game state information.

[0009] Optionally, the game state information includes absolute numerical information and relative numerical information.

[0010] Optionally, the game state information includes the game mode and / or the game robot's historical action sequence.

[0011] Optionally, the method further includes: adjusting the number of input layer neurons of the first multilayer perceptron corresponding to each group of game state information according to the dimensional change of each group of game state information, wherein the input layer neurons are used to receive game state information input to the first multilayer perceptron.

[0012] Optionally, adjusting the number of neurons in the input layer of the first multilayer perceptron corresponding to each set of game state information based on the dimensional change of each set of game state information includes: if the dimensionality of the game state information increases, increasing the number of neurons in the input layer of the first multilayer perceptron corresponding to the game state information, establishing connections between the increased neurons and the hidden layer neurons of the first multilayer perceptron, setting the initial weight of the added parameters, and training the game robot based on the existing parameters of the trained game robot.

[0013] Optionally, the execution result includes the position of the game robot in the game situation.

[0014] Optionally, obtaining the reward signal of the game robot based on the execution result of the game robot executing the behavior strategy includes: obtaining preset key position information and reward triggering conditions in the game situation; if it is determined that the position of the game robot in the game situation matches the key position information, and the current game state of the game situation satisfies the reward triggering conditions corresponding to the key position information, then the reward signal of the game robot is generated.

[0015] Optionally, based on the execution result of the game robot executing the behavior strategy, a reward signal for the game robot is obtained, including: detecting that the game robot controls a target game element, or detecting that other game robots with the same group attribute as the game robot control the target game element; determining a polygon formed by the game robot and other game robots with the same group attribute as it as a first region; obtaining a first key region corresponding to preset key position information; if the first region moves towards the target position to at least partially overlap with the first key region, a reward signal for representing a positive reward is generated; if the overlap between the first region and the first key region decreases, a reward signal for representing a negative reward is generated.

[0016] Optionally, the step of performing reinforcement learning based on the fused embedding vector to obtain the behavior policy of the game robot includes: inputting the fused embedding vector into the second multilayer perceptron of the game robot to generate the behavior policy of the game robot.

[0017] This application embodiment also provides a game robot training device, comprising: a state information acquisition unit, used to acquire multiple sets of game state information of the current game situation of the game robot to be trained; a feature extraction unit, used to perform feature extraction processing on each set of game state information to obtain an embedding vector corresponding to each set of game state information; a feature fusion unit, used to perform feature fusion processing on the embedding vectors corresponding to the multiple sets of game state information to obtain a fused embedding vector; a decision unit, used to perform reinforcement learning based on the fused embedding vector to obtain the behavior strategy of the game robot; a reward signal generation unit, used to obtain a reward signal of the game robot based on the execution result of the game robot after executing the behavior strategy; and a training unit, used to train the game robot based on the reward signal and the game state information.

[0018] This application also provides an electronic device, including: a memory and a processor; the memory is used to store a computer program, which, when run by the processor, executes the method provided in this application.

[0019] This application also provides a computer storage medium storing computer execution instructions, which, when executed by a processor, are used to implement the method provided in this application.

[0020] Compared with the prior art, this application has the following advantages:

[0021] This application provides a game robot training method, apparatus, electronic device, and storage medium. The method involves acquiring multiple sets of game state information of the current game situation of the game robot to be trained; performing feature extraction processing on each set of game state information to obtain an embedding vector corresponding to each set of game state information; performing feature fusion processing on the embedding vectors corresponding to the multiple sets of game state information to obtain a fused embedding vector; performing reinforcement learning based on the fused embedding vector to obtain the game robot's behavior strategy; obtaining the game robot's reward signal based on the execution result of the game robot executing the behavior strategy; and training the game robot based on the reward signal and the game state information. Training the game robot based on the game state information and reward signal enables the game robot to acquire a behavior strategy with a high level of intelligence, thereby solving the problems of fixed patterns and low intelligence levels in rule-based game robots. Furthermore, by forming an embedding vector corresponding to each set of game state information, better feature extraction results can be achieved, reducing the learning difficulty of the game robot at the feature level. Further, the game robot includes a first multilayer perceptron and a second multilayer perceptron, possessing good expressive capabilities and supporting parallel training under limited computing resources. Attached Figure Description

[0022] Figure 1 This is a flowchart of a game robot training method provided in one embodiment of this application;

[0023] Figure 2 This is a schematic diagram of a game robot structure provided in one embodiment of this application;

[0024] Figure 3 This is a schematic diagram of another game robot structure provided in one embodiment of this application;

[0025] Figure 4 This is a schematic diagram of the dynamic expansion of a game robot model provided in one embodiment of this application;

[0026] Figure 5 This is a flowchart of a reward signal generation method provided in one embodiment of this application;

[0027] Figure 6 This is a schematic diagram of a location-based dense reward implemented based on the reward signal generation method provided in one embodiment of this application;

[0028] Figure 7 This is a schematic diagram of a game robot training device provided in one embodiment of this application;

[0029] Figure 8 This is a schematic diagram of the electronic device provided in this application. Detailed Implementation

[0030] Many specific details are set forth in the following description to provide a full understanding of this application. However, this application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this application; therefore, this application is not limited to the specific embodiments disclosed below.

[0031] This application provides a method, apparatus, electronic device, and storage medium for training game robots. These will be described in detail in the following embodiments.

[0032] One embodiment of this application provides a game robot training method. The game robot can manifest as one or more virtual characters controlled by an algorithm model in a game scene. Its application scenarios include, but are not limited to, simulating real players participating in games and replacing real players who have been assigned game management roles. The game robot includes multiple stacked multilayer perceptrons, which can simulate human players making game decisions, acquire game state information in the game situation as input, and generate behavioral strategies to control the actions to be taken by the game robot. A game robot can be understood as a computer program in a game scene that has perception capabilities, reinforcement learning capabilities, and can automatically complete a certain task. The game robot training method performs reinforcement learning training based on the game state information of the game robot and the corresponding reward signals. Through continuous training, the game robot gradually generates better game decisions or behavioral strategies for the game environment. This solves the problems of fixed patterns and low intelligence levels in rule-based game robots.

[0033] Reinforcement Learning (RL) is a machine learning method for solving sequential decision-making problems. Its main process involves an agent perceiving the current state in the environment, taking actions to interact with the environment, receiving feedback signals, and adjusting its policy based on these signals. This process of "perception, action, feedback, and optimization" is repeated continuously to maximize the accumulated reward signal. Through continuous training, the agent gradually develops an optimal policy. In the context of the game robot in this embodiment, the game scene (or game situation) can be understood as the environment; the current game state can be understood as the current condition, specifically represented by game state information; the game robot's policy can be understood as actions, which can be a single action or a sequence of actions; and the reward signal can be understood as the feedback signal from the environment.

[0034] The following combination Figures 1 to 6 A detailed description of a game robot training method provided in one embodiment of this application will be given. Figure 1 The game robot training method shown includes the following steps:

[0035] Step S101: Obtain multiple sets of game state information of the current game situation of the game robot to be trained;

[0036] Step S102: Perform feature extraction processing on each group of game state information to obtain the embedding vector corresponding to each group of game state information;

[0037] Step S103: Perform feature fusion processing on the embedding vectors corresponding to the multiple sets of game state information to obtain fused embedding vectors;

[0038] Step S104: Perform reinforcement learning based on the fused embedding vector to obtain the behavior strategy of the game robot;

[0039] Step S105: Based on the execution result of the game robot after executing the behavior strategy, obtain the reward signal of the game robot;

[0040] Step S106: Train the game robot based on the reward signal and the game state information.

[0041] In this embodiment, the game robot is composed of multiple stacked multilayer perceptrons, including multiple first multilayer perceptrons for processing game state information and a second multilayer perceptron for processing fused embedding vectors. Each of the multiple first multilayer perceptrons corresponds one-to-one with each group of game state information; each first multilayer perceptron processes a group of game state information and outputs an embedding vector corresponding to that group. The structures of the first multilayer perceptrons can be different or the same, without limitation. In implementation, the structure of the corresponding first multilayer perceptron is determined based on the dimension of each group of game state information. The first and second multilayer perceptrons are constructed from different types of neurons.

[0042] As described in step S101, multiple sets of game state information of the current game situation of the game robot to be trained are obtained.

[0043] This step involves acquiring multiple sets of game state information for training the game robot. Game state information refers to the feature descriptions of the game states, which form the input samples for the game robot. Different games require different game state information, and feature extraction needs to be performed based on the type of game application and the information it can provide. For competitive sports games, the key game elements involve at least two opposing sides; therefore, information such as relative positions and movement directions is generally required.

[0044] Preferably, the game state information includes absolute numerical information and relative numerical information. The absolute numerical information can be understood as information representing the game state of the game situation in which the game robot to be trained is located, in numerical form, based on a unified coordinate system. An example of a unified coordinate system is the game world coordinate system. For example, the coordinate position and direction of movement of each game robot or other game element in the game situation. The absolute numerical information reflects the game situation on the field. The relative numerical information can be understood as information representing the game state of the game situation in which the game robot to be trained is located, in numerical form, with the game robot to be trained as a reference. For example, the relative position and direction of movement of other game robots or game elements in the game situation in which the game robot to be trained is located relative to the game robot to be trained. The relative numerical information enables the game robot to have strong generalization performance and the ability to reuse behavioral strategies in similar scenarios. For example, using relative numerical information to represent the game state information of triangular passing combinations in different positions in a football match, the behavioral strategies learned from this game state information can be reused in different game situations.

[0045] Preferably, the game state information includes the game mode and / or the historical action sequence of the game robot. The game mode is a mode set based on game domain knowledge, such as corner kick mode, penalty kick mode, and duel mode in a football game.

[0046] In specific implementation, obtaining the current game state information of the game robot includes the following processing: obtaining at least one of the following information about the key elements of the game application in the game situation: absolute numerical information, relative numerical information, historical action sequences, and the game model corresponding to the current game situation, such as the match mode; forming the game state information from the above at least one piece of information. The game state information can also be data represented by vectors. Vector-based information does not involve complex calculation operations, which can avoid huge computational overhead.

[0047] In this embodiment, the game robot can be trained using either offline data or reinforcement learning in an online environment. If offline data is used, game state information can be extracted from the game logs. If online data is used for reinforcement training in an online environment, basic data can be obtained by calling the game application's interface. Game state information can then be identified from this basic data based on the application's domain knowledge, and / or other game state information can be calculated using the basic data. Taking a football game as an example, game state information is mainly calculated based on basic data such as player position and movement direction. For instance, by obtaining each player's position through the interface, the distance between two players can be calculated. Furthermore, the football game application itself can provide information such as player team number, whether a player has received a red or yellow card, and the match mode.

[0048] The example of a football game is used to further illustrate the game state information. Game elements in a football game include, but are not limited to, the ball, the player carrying the ball, the controller (i.e., the game robot acting as the controlled player), teammates (i.e., our players), and opponents (i.e., the opposing players). Information about these game elements in the game application can be used as the game state information extracted in this step. The information about these key elements may include, but is not limited to, absolute numerical information (e.g., coordinate position, direction of movement) and relative numerical information (e.g., relative position), game mode, and historical action sequences. Basketball games are similar to football games. In this embodiment, the game state information can comprehensively summarize the game state of competitive sports games such as football and basketball. The absolute numerical information reflects the game situation on the field; the relative numerical information enables the game robot to have strong generalization performance and the ability to reuse strategies in similar scenarios (e.g., triangular passing combinations in different positions during a football match); and the historical action sequences enable the game robot to have a certain memory ability. Table 1 shows a specific example of the game state information of the game robot.

[0049] Table 1. Examples of status information in sports games, specifically in a football game.

[0050]

[0051]

[0052] This embodiment includes grouping game state information using prior domain knowledge of the game application. Specifically, obtaining multiple sets of game state information in the game situation where the game robot to be trained is located includes: obtaining multiple sets of game state information in the game situation where the game robot to be trained is located; and grouping the multiple sets of game state information based on the semantic relevance between the multiple sets of game state information to obtain multiple sets of game state information. The semantic relevance between the game state information in each set of game state information meets a preset relevance condition. That is, game state information that is semantically close (i.e., semantically related) is grouped together, while game state information that is semantically far apart (i.e., semantically unrelated) is grouped into different groups. In practice, the grouping of game state information can be based on experience or semantic grouping of game state information can be performed using a neural network. By grouping the input, the game robot can have a smaller number of parameters, reducing its learning difficulty at the feature level.

[0053] In step S102, feature extraction processing is performed on each group of game state information to obtain an embedding vector corresponding to each group of game state information. Specifically, the feature extraction processing for each group of game state information to obtain an embedding vector corresponding to each group of game state information includes: inputting each group of game state information into a first multilayer perceptron corresponding to that group to obtain an embedding vector corresponding to that group.

[0054] In step S103, the embedding vectors corresponding to the multiple sets of game state information are subjected to feature fusion processing to obtain a fused embedding vector. Specifically, each embedding vector can be concatenated or accumulated to form the fused embedding vector, which can characterize the game state fusion features corresponding to the game robot.

[0055] In step S104, reinforcement learning is performed based on the fused embedding vector to obtain the behavior policy of the game robot. Specifically, the fused embedding vector is input into the second multilayer perceptron of the game robot to generate the behavior policy of the game robot. In this embodiment, the behavior policy can be calculated for each game robot controlled by the algorithm model in turn, or the behavior policy can be calculated for some game robots. Specifically, this includes: for one or more game robots, obtaining the game state information of each game robot in the game situation one by one, and generating a behavior policy for each game robot one by one; or, for one or more game robots, obtaining the game state information of at least some game robots in the game situation, and generating a behavior policy, wherein the behavior policy is used to control the game behavior of the at least some game robots.

[0056] Please refer to Figure 2The figure illustrates a game robot structure, including: multiple first multilayer perceptrons 201, second multilayer perceptrons 202, an embedding vector fusion unit 203, and a game decision output unit 204. The multiple first multilayer perceptrons correspond one-to-one with state group 1, state group 2, ..., state group N, respectively. Each first multilayer perceptron inputs a set of game state information and outputs an embedding vector. Each state group refers to a group of game state information of the game robot. The embedding vector fusion unit concatenates each embedding vector into a fused embedding vector representing the fused game state features of the game robot. This fused embedding vector is input into the second multilayer perceptron to form a behavioral strategy, which is then output by the game decision output unit. It should be understood that the numbers in the figure are exemplary and not a limitation on the number of perceptrons.

[0057] Please refer to this again. Figure 3 The gaming robot shown in the picture is particularly suitable for competitive sports games, such as basketball and football, and its structure is similar to... Figure 2Similarly, it includes: multiple first multilayer perceptrons 301, second multilayer perceptrons 302, embedded vector fusion units 303, and game decision output units 304. For example, each first multilayer perceptron processes the following grouped game state information: ball, ball holder, controller (i.e., game robot), relative features between controller and ball, relative features between controller and ball holder, relative features between teammates, relative features between opponents, absolute features between teammates, absolute features between opponents, features of surrounding players, game state patterns, and historical action sequences. Here, the ball holder can be understood as the game robot possessing the ball in each state of the game scene; it can be a game robot acting as a player holding the ball on our side, or it can be an opponent holding the ball who cannot be controlled. The controller can be understood as a training game robot that needs to calculate behavioral strategies. Teammates include all game robots on our side except the controller, and opponents refer to all game robots on the opposing side. Surrounding players refer to one or more game robots around a controller whose distance from the controller meets a preset distance condition; these can be teammates or opponents. The game state information input from multiple first multilayer perceptrons can be understood as the operator's perception of the game situation; specifically, in sports games, it's the perception of the court situation. This perception includes the game robot's observation of other game robots and its own related information. In implementation, the structure of the corresponding first multilayer perceptron is determined based on the dimension of each set of game state information. Each set of game state information is processed by the corresponding first multilayer perceptron to calculate its corresponding embedding vector. The dimension of each embedding vector is not limited in this method and is set according to actual needs. Each embedding vector is concatenated or accumulated by the embedding vector fusion unit to form a fused embedding vector for each game state information, representing the game robot's game state fusion characteristics. The fused embedding vector is input to the second multilayer perceptron for calculation to obtain a behavioral strategy, which is output by the game decision output unit. In implementation, the game decision output unit can be displayed in the game scene in the visual form of the game robot. The behavioral strategy is the action or action sequence to be performed by the game robot. It is understood that the structures of the first and second multilayer perceptrons in the figure are illustrative and do not limit the specific structure of each multilayer perceptron. The specific game state information and numbers shown in the figure are illustrative and do not limit the game state information that the method can process or the number of features.

[0058] In this embodiment, the game robot can also be dynamically expanded and transferred for training. This means the model structure of the game robot can be changed within a certain range while retaining existing training results. The game robot can be understood as a neural network model. As the game state information it uses changes, the model structure of the game robot can change accordingly. Its input layer, intermediate layer, and output layer can adjust the number of neurons and the connections between neurons based on changes in the game state information of the corresponding groups. Each time the game robot model structure changes, training can continue based on existing results, rather than training a completely new model from scratch. When dynamically expanding neuron nodes, the original parameter weights remain unchanged, and the values ​​of the newly added parameter weights can be randomly initialized (e.g., small random values) or zeroed out according to the actual situation. This ensures that the results of each layer in the forward computation of the game robot model remain essentially unchanged, thereby maximizing the preservation of the original input-output mapping relationship. Therefore, the expanded game robot can still remember the previously learned strategy, and the new round of training will unfold based on the obtained results, rather than starting from scratch.

[0059] Preferably, the dynamic expansion and transfer training of the game robot includes: adjusting the number of input layer neurons of the first multilayer perceptron corresponding to each set of game state information according to the dimensional changes of each set of game state information. The input layer neurons are used to receive game state information input to the first multilayer perceptron. Specifically, adjusting the number of input layer neurons of the first multilayer perceptron corresponding to each set of game state information according to the dimensional changes of each set of game state information includes: if the dimensionality of the game state information increases, increasing the number of input layer neurons of the first multilayer perceptron corresponding to the game state information, establishing connections between the increased neurons and the hidden layer neurons of the first multilayer perceptron, setting initial weights for the newly added parameters, and training the game robot based on the existing parameters of the trained game robot. The dimensionality of the game state information refers to the number of game states. For example, position coordinates and distance information obtained from the orientation dimension, and movement direction and passing actions obtained from the action dimension. Please refer to [reference needed]. Figure 4The figure illustrates a dynamic expansion example of a game robot model, showing how the input layer of a single-layer neural network (single-layer means with only one hidden layer) is expanded from a 4-dimensional vector to a 6-dimensional vector. Besides the input layer, the neural network contains an 8-unit hidden layer and a 4-unit output layer. As shown, the two newly added units in the input layer are connected to the 8 units in the hidden layer. Correspondingly, the original input layer parameter weight matrix changes from a 4x8 matrix to a 6x8 matrix. When dynamically expanding neuron nodes, the original parameter weights remain unchanged, while the values ​​of the newly added parameter weights (marked in gray in the figure) can be randomly initialized (e.g., with small random values) or zeroed out based on the actual situation. The expanded input layer retains the original input-output mapping relationship to the greatest extent possible, still remembering the previously learned strategy. The new round of training will proceed based on the obtained results, rather than starting from scratch. The principle of dynamically expanding intermediate and output layers is similar to that of the input layer.

[0060] In step S105, a reward signal for the game robot is obtained based on the execution result of the game robot executing the behavior strategy. The execution result includes the position of the game robot in the game situation. In one embodiment, obtaining the reward signal based on the execution result of the game robot executing the behavior strategy includes: obtaining preset key position information and reward triggering conditions in the game situation; if it is determined that the position of the game robot in the game situation matches the key position information, and the current game state satisfies the reward triggering condition corresponding to the key position information, then a reward signal for the game robot is generated. In one implementation, based on the execution result of the game robot executing the behavioral strategy, a reward signal for the game robot is obtained, including: detecting that the game robot controls a target game element, or detecting that other game robots with a common group attribute control the target game element; determining a polygon formed by the game robot and other game robots with a common group attribute as a first region; obtaining a first key region corresponding to preset key position information; if the first region moves towards the target position to at least partially overlap with the first key region, a reward signal for representing a positive reward is generated; if the overlap between the first region and the first key region decreases, a reward signal for representing a negative reward is generated.

[0061] In step S106, the game robot is trained based on the reward signal and the game state information. In this embodiment, the game robot executes actions according to the behavior strategy to obtain a reward signal (i.e., environmental feedback signal). It can further adjust its behavior strategy based on the reward signal, continuously reinforcing learning and training based on game state information and reward signals to obtain a game robot with a certain level of intelligence. Thus, the game robot can act as a virtual player to fight and interact with real players.

[0062] It is understood that while the methods provided in this embodiment and other embodiments of this application are more applicable to competitive sports games, they are not limited to competitive sports games. They can also be applied to any environment where the state information of an intelligent agent, similar to a robot, is used to generate behavioral strategies for the agent. The robot can be composed of multiple first multilayer perceptrons and one second multilayer perceptron. The state information of the intelligent agent is grouped, with each of the multiple first multilayer perceptrons corresponding one-to-one with each group of state information of the intelligent agent. A group of state information is input into the input layer of a first multilayer perceptron, and the output layer of the first multilayer perceptron outputs the corresponding embedding vector. The embedding vectors corresponding to each group of state information are concatenated or accumulated to form a fused embedding vector, which is input into the input layer of the second multilayer perceptron. The second multilayer perceptron outputs a behavioral strategy generated based on the state information of the intelligent agent. The intelligent agent executes actions according to the behavioral strategy and obtains feedback signals (i.e., reward signals). It can further adjust the behavioral strategy according to the feedback signals, continuously reinforcing learning and training based on state information and feedback signals to obtain a robot with a certain level of intelligence.

[0063] This embodiment further provides a method for generating reward signals. The following is in conjunction with... Figure 5 and Figure 6 The method will be described below. For relevant parts, please refer to the corresponding descriptions in the above embodiments. Figure 5 The reward signal generation method shown includes steps S501 to S503.

[0064] Step S501: It is detected that the game robot in the game scene is performing game behavior according to the target execution strategy; the game robot is trained using the game robot training method provided in the embodiments of this application.

[0065] Step S502: Obtain the location information of game elements related to the game robot in the game scene;

[0066] Step S503: Determine whether the location information matches the preset key location information. If so, generate a reward signal for the game robot.

[0067] The game elements are determined based on the specific game application. For example, in competitive sports games, the ball, ball carrier, controller (i.e., the game bot), teammates, and opponents can be key elements. The ball carrier can be understood as the game bot possessing the ball in each state of the game scenario; it could be a player on your team with the ball or an opponent with the ball who cannot be controlled. Teammates include all game bots on your team except the controller, and opponents refer to all game bots on the opposing team. The positional information of the game elements related to the game bot can refer to the positional information of the game bot itself or the positional information of teammates. Subsequent steps can generate reward signals based on the overall progress of the team to which the game bot belongs.

[0068] In step S503, a reward signal is generated based on the preset key location information and the corresponding preset triggering conditions. Specifically, the current game state related to the game robot is obtained; it is determined whether the location information matches the preset key location information and whether the current game state satisfies the preset triggering conditions corresponding to the preset key location information. If so, a reward signal is generated for the game agent. This specifically includes the following processing: if it is detected that the game robot controls a target game element, or if it is detected that a game robot or other character with a common group attribute with the game robot controls a target game element; and if it is detected that the distance between the game robot's position and the target position has decreased to less than a first preset distance threshold, or the distance is less than the first preset distance threshold and has decreased to less than a second preset distance threshold; then it is determined that the location information matches the preset key location information, and the current game state satisfies the preset reward triggering conditions; the reward signal generated for the game robot is a positive reward signal representing a positive reward. For example, the target game element is a ball in a sports game, and the target position is the opponent's goal or the center of the opponent's goal. When the game robot is the ball carrier, and it dribbles the ball forward into a key area formed by a set of key positions between a first preset distance threshold and the target position, a positive reward signal is generated to represent a positive reward. Conversely, if the ball carrier leaves the key area after dribbling the ball, a negative reward signal is generated to represent a negative reward. This incentivizes the robot to advance towards the front. The specific reward signal generation method can be set according to the prior knowledge of the game application's domain. For sports games such as basketball and football, existing reward signals are generally generated after the game robot performs an action and scores a goal, resulting in a lack of reward signals and a sparse reward problem. In this embodiment, based on the spatial characteristics of this type of game, which involves two-sided confrontation and overall advancement from the backcourt to the frontcourt, reward signals are generated based on preset key position information to achieve dense rewards. Specifically, this includes: presetting one or more key positions and corresponding reward triggering conditions in the game application's field; whenever the preset reward triggering condition for a key position is met, a corresponding reward signal is provided to the game robot currently performing reinforcement learning. Specifically, based on the position information of the game robot and / or its teammates, it can be determined whether the position information matches preset key position information and whether it meets the preset reward triggering conditions corresponding to the preset key position information. If so, a reward signal is generated, thereby achieving dense rewards, allowing the game robot and / or its teammates to receive rich feedback signals and converge quickly during training. The teammates refer to game robots or other characters that share group attributes with the game robot.One implementation includes: if it is detected that the game robot controls the target game element, or if it is detected that a game robot or other character with the same group attribute as the game robot controls the target game element; and if it is detected that the distance between the position of the game robot and the target position increases to not less than a first preset distance threshold, or the distance is less than a second preset distance threshold and increases to not less than the second preset distance threshold; then it is determined that the position information matches preset key position information, and the current game state satisfies the preset reward triggering condition; the reward signal generated for the game robot is a negative reward signal representing a negative reward.

[0069] Please refer to Figure 6 The diagram illustrates a location-based dense reward method, including setting preset key locations and corresponding preset reward trigger conditions. Taking an attacking team moving from left to right as an example, each gray dashed line is an arc with a radius equal to a specific distance from the opponent's goal, representing a set of locations at the same distance. When the game robot dribbles the ball from left to right across a certain red line, a positive numerical reward is given (a negative numerical reward or no reward is generated if the ball is dribbled in the opposite direction), encouraging the agent to push the attack from the backfield to the frontfield. In game environments with sparse reward characteristics (i.e., goal rewards) such as football and basketball, the key location-based reward generation method can greatly enrich the reward signals of the environment, providing more guidance for reinforcement learning game robots.

[0070] The reward signal generation method generates reward signals based on key location information. For game applications with spatial characteristics of forward propagation, it is convenient to provide rich environmental feedback, avoid the problem of sparse rewards, and enable the game robot to receive rich feedback signals during training, thereby achieving rapid convergence.

[0071] It should be noted that, unless otherwise specified, the features given in this embodiment and other embodiments of this application can be combined with each other, and steps S101 and S102 or similar terms do not limit the steps to be performed in a specific order.

[0072] This concludes the description of the method provided in this embodiment. The method trains a game robot based on game state information and reward signals. The trained game robot acquires behavioral strategies with a high level of intelligence, thus solving the problems of fixed patterns and low intelligence levels inherent in rule-based game robots. Furthermore, by forming an embedding vector corresponding to each set of game state information, better feature extraction results can be achieved, reducing the learning difficulty for the game robot at the feature level. Further, the game robot includes a first multilayer perceptron and a second multilayer perceptron, possessing good expressive power and supporting parallel training under limited computing resources.

[0073] Corresponding to the embodiment of the game robot training method described above, one embodiment of this application provides a game robot training device. Please refer to... Figure 7 The game robot training device shown in the figure includes:

[0074] The state information acquisition unit 701 is used to acquire multiple sets of game state information of the current game situation of the game robot to be trained.

[0075] The feature extraction unit 702 is used to perform feature extraction processing on each group of game state information to obtain the embedding vector corresponding to each group of game state information.

[0076] The feature fusion unit 703 is used to perform feature fusion processing on the embedding vectors corresponding to the multiple sets of game state information to obtain a fused embedding vector.

[0077] Decision unit 704 is used to perform reinforcement learning based on the fused embedding vector to obtain the behavior strategy of the game robot;

[0078] The reward signal generation unit 705 is used to obtain the reward signal of the game robot based on the execution result of the game robot executing the behavior strategy;

[0079] Training unit 706 is used to train the game robot based on the reward signal and the game state information.

[0080] Optionally, the feature extraction unit 702 is specifically used to: obtain the embedding vector corresponding to each group of game state information through the first multilayer perceptron corresponding to each group of game state information.

[0081] Optionally, the state information acquisition unit 701 is specifically used to: acquire multiple game state information in the game situation where the game robot to be trained is located; and group the multiple game state information based on the semantic correlation between the multiple game state information to obtain multiple groups of game state information.

[0082] Optionally, the game state information includes absolute numerical information and relative numerical information.

[0083] Optionally, the game state information includes the game mode and / or the game robot's historical action sequence.

[0084] Optionally, the device further includes a dynamic expansion unit, which is used to: adjust the number of input layer neurons of the first multilayer perceptron corresponding to each group of game state information according to the dimensional changes of each group of game state information, wherein the input layer neurons are used to receive game state information input to the first multilayer perceptron.

[0085] Optionally, the dynamic expansion unit is specifically used to: if the dimension of the game state information increases, increase the number of neurons in the input layer of the first multilayer perceptron corresponding to the game state information, establish connections between the increased neurons and the hidden layer neurons of the first multilayer perceptron, set the initial weights of the added parameters, and train the game robot based on the existing parameters of the trained game robot.

[0086] Optionally, the execution result includes the position of the game robot in the game situation.

[0087] Optionally, the reward signal generation unit 705 is specifically used to: acquire preset key position information and reward triggering conditions in the game situation; if it is determined that the position of the game robot in the game situation matches the key position information, and the current game state of the game situation satisfies the reward triggering conditions corresponding to the key position information, then generate a reward signal for the game robot.

[0088] Optionally, the reward signal generation unit 705 is specifically configured to: detect that the game robot controls a target game element, or detect that other game robots with the same group attribute as the game robot control a target game element; determine the polygon formed by the game robot and other game robots with the same group attribute as it as a first region; acquire a first key region corresponding to preset key position information; if the first region moves toward the target position to at least partially overlap with the first key region, generate a reward signal to represent a positive reward; if the overlap between the first region and the first key region decreases, generate a reward signal to represent a negative reward.

[0089] Optionally, the decision unit 704 is specifically used to: input the fused embedding vector into the second multilayer perceptron of the game robot to generate the behavior strategy of the game robot.

[0090] Based on the above embodiments, one embodiment of this application provides an electronic device; for relevant parts, please refer to the corresponding descriptions of the above embodiments. Figure 8 The electronic device shown in the figure includes a memory 801 and a processor 802; the memory is used to store a computer program, which, after being run by the processor, executes the method provided in the embodiments of this application.

[0091] Based on the above embodiments, one embodiment of this application provides a storage medium; relevant parts can be found in the corresponding descriptions of the above embodiments. The schematic diagram of the storage medium is similar. Figure 8 The storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method provided in the embodiments of this application.

[0092] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0093] Although this application discloses preferred embodiments as described above, it is not intended to limit this application. Any person skilled in the art can make possible changes and modifications without departing from the spirit and scope of this application. Therefore, the scope of protection of this application should be determined by the scope defined in the claims of this application.

Claims

1. A method for training a game robot, characterized in that, include: Obtain multiple sets of game state information of the current game situation of the game robot to be trained; Feature extraction processing is performed on each group of game state information to obtain an embedding vector corresponding to each group of game state information; wherein, the embedding vector corresponding to each group of game state information is obtained through a first multilayer perceptron corresponding to each group of game state information. The embedding vectors corresponding to the multiple sets of game state information are subjected to feature fusion processing to obtain a fused embedding vector. Reinforcement learning is performed based on the fused embedding vectors to obtain the behavioral strategy of the game robot; Based on the execution result of the game robot after executing the behavioral strategy, obtain the reward signal of the game robot; The game robot is trained based on the reward signal and the game state information; The method further includes: if the dimension of the game state information increases, increasing the number of input layer neurons of the first multilayer perceptron corresponding to the game state information, establishing connections between the increased neurons and the hidden layer neurons of the first multilayer perceptron, setting initial weights for the newly added parameters, and training the game robot based on the existing parameters of the trained game robot; the input layer neurons are used to receive the game state information input to the first multilayer perceptron.

2. The method according to claim 1, characterized in that, The acquisition of multiple sets of game state information in the game situation where the game robot to be trained is located includes: Obtain multiple game state information in the game situation where the game robot to be trained is located; Based on the semantic correlation between the multiple game state information, the multiple game state information is grouped to obtain multiple groups of game state information.

3. The method according to claim 1, characterized in that, The game status information includes absolute numerical information and relative numerical information.

4. The method according to claim 1, characterized in that, The game status information includes the game mode and / or the historical action sequence of the game robot.

5. The method according to claim 1, characterized in that, The execution result includes the position of the game robot in the game situation.

6. The method according to claim 5, characterized in that, The step of obtaining the reward signal of the game robot based on the execution result of the behavior strategy executed by the game robot includes: Obtain the preset key location information and reward triggering conditions in the game situation; If it is determined that the position of the game robot in the game situation matches the key position information, and the current game state of the game situation satisfies the reward triggering condition corresponding to the key position information, then a reward signal for the game robot is generated.

7. The method according to claim 5, characterized in that, Based on the execution result of the game robot after executing the behavioral strategy, the reward signal of the game robot is obtained, including: The game robot is detected controlling the target game element, or other game robots with the same group attributes as the game robot are detected controlling the target game element; The polygon formed by the game robot and other game robots that share the same group attributes is defined as the first region. Obtain the first key area corresponding to the preset key location information; If the first region moves toward the target location to at least partially overlap with the first key region, a reward signal is generated to characterize the positive reward. If the overlap between the first region and the first key region decreases, a reward signal is generated to characterize the negative reward.

8. The method according to claim 1, characterized in that, The step of performing reinforcement learning based on the fused embedding vector to obtain the behavioral strategy of the game robot includes: The fused embedding vector is input into the second multilayer perceptron of the game robot to generate the game robot's behavior strategy.

9. A game robot training device, characterized in that, include: The state information acquisition unit is used to acquire multiple sets of game state information of the current game situation of the game robot to be trained. The feature extraction unit is used to perform feature extraction processing on each group of game state information to obtain the embedding vector corresponding to each group of game state information; wherein, the embedding vector corresponding to each group of game state information is obtained through the first multilayer perceptron corresponding to each group of game state information. The feature fusion unit is used to perform feature fusion processing on the embedding vectors corresponding to the multiple sets of game state information to obtain a fused embedding vector. A decision-making unit is used to perform reinforcement learning based on the fused embedding vector to obtain the behavioral strategy of the game robot. A reward signal generation unit is used to obtain a reward signal of the game robot based on the execution result of the game robot executing the behavior strategy; A training unit is used to train the game robot based on the reward signal and the game state information; This also includes: if the dimension of the game state information increases, increasing the number of input layer neurons of the first multilayer perceptron corresponding to the game state information, establishing connections between the increased neurons and the hidden layer neurons of the first multilayer perceptron, setting initial weights for the new parameters, and training the game robot based on the existing parameters of the trained game robot; the input layer neurons are used to receive the game state information input to the first multilayer perceptron.

10. An electronic device, characterized in that, include: A memory and a processor; the memory is used to store a computer program, which, when executed by the processor, performs the method according to any one of claims 1-8.

11. A computer storage medium, characterized in that, The device stores computer execution instructions, which, when executed by a processor, are used to implement the method described in any one of claims 1-8.

Images