A training method, system and electronic device for a robot in a game

Abstract

The application provides a training method of a robot in a game, comprising: obtaining state information of a robot to be trained and player information (including settlement information, state information and action information) of a player in a preset range of the robot; processing the settlement information into a first data table in the form of metadata, processing the state information and the action information into a second data table in the form of data pairs, and splicing the first data table and the second data table into a result data table; and based on the state information of the robot and the result data table, establishing a state-action model for outputting an action to be executed by the robot through imitative learning. The method processes the obtained player settlement information, state information and action information into training data for imitative learning of the robot, so that the trained robot is closer to a real player, and when the robot replaces an absent player to participate in the game, the game experience of the player can be significantly improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a method and system for training a robot in a game, an electronic device and a computer readable storage medium. BACKGROUND

[0002] With the development of computer game technology, more and more players enjoy multiplayer online battle games. However, multiplayer online battle games can only start when the number of participants reaches a certain requirement (for example, 40 players). In order to allow players to play the game without being limited by the number of players, an artificial intelligence robot needs to be provided to replace the players and make up for the vacancy of the players.

[0003] The robot provided by the prior art has the technical problem that the behavior style is quite different from that of the players, and the player experience is not high. SUMMARY

[0004] The present application provides a method and system for training a robot in a game and an electronic device to solve the technical problem that the prior art cannot provide a robot with a human-like style in a large-scale multiplayer battle game.

[0005] The present application provides a method for training a robot in a game, comprising:

[0006] Obtaining state information of a robot to be trained and player information of a player within a preset range of the robot; the player information at least includes settlement information of a game battle corresponding to the player, state information of the player and action information of the player; the state information of the player includes self-state information of the player and field information corresponding to the player;

[0007] Processing the settlement information into a first data table in the form of metadata, processing the state information of the player and the action information of the player into a second data table in the form of data pairs, and splicing the first data table and the second data table into a result data table according to the field information corresponding to the player; each row in the result data table contains settlement metadata information, state information and action information of the player in a game battle;

[0008] Based on the state information of the robot and the result data table, a state-action model for outputting an action to be executed by the robot is established through imitative learning.

[0009] Optionally, the state information of the robot includes self-state information of the robot and field information corresponding to the robot.

[0010] Optionally, the action information includes at least one of the following: a moving action, a skill action and a picking action.

[0011] Optionally, the processing of the settlement information into the first data table in the form of metadata, the processing of the state information of the player and the action information of the player into the second data table in the form of data pairs, and the splicing of the first data table and the second data table into the result data table according to the field information corresponding to the player comprise: using a Spark operator to process the settlement information into the first data table in the form of metadata, to process the state information of the player and the action information of the player into the second data table in the form of data pairs, and to splice the first data table and the second data table into the result data table according to the field information corresponding to the player.

[0012] Optionally, the processing of the state information of the player and the action information of the player into the second data table in the form of data pairs comprises:

[0013] determining whether the first N action frames of the target skill frame in the action information are all movement frames, and if so, modifying the first N action frames of the target skill frame in the action information into skill frames;

[0014] and / or,

[0015] determining whether the last N action frames of the target skill frame are all movement frames, and if so, modifying the last N action frames of the target skill frame in the action information into skill frames;

[0016] processing the state information of the player and the modified action information of the player into the second data table in the form of data pairs, wherein the N is a positive integer.

[0017] Optionally, the method further comprises:

[0018] adjusting the skill-related information corresponding in the state information of the player according to the modification of the first N action frames of the target skill frame and / or the modification of the last N action frames of the target skill frame.

[0019] Optionally, the processing of the state information of the player and the action information of the player into the second data table in the form of data pairs comprises:

[0020] detecting whether there are M continuous movement frames in the action information of the player, wherein M is a positive integer;

[0021] if so, deleting part of the movement frames in the M movement frames to obtain modified action information;

[0022] processing the state information of the player and the modified action information into the second data table in the form of data pairs.

[0023] Optionally, the processing the state information of the player and the action information of the player into a second data table in the form of a data pair comprises:

[0024] According to the state information of the player and the action information of the player, target state information and target action information meeting a preset condition are screened, the state information comprises the target state information, and the action information comprises the target action information;

[0025] The target state information and the target action information are processed into a second data table in the form of a data pair.

[0026] Optionally, the state-action model for outputting the action to be performed by the robot is established through imitative learning based on the state information of the robot and the result data table, and the method comprises:

[0027] Based on the state information of the robot and the result data table, a predicted action of the robot is obtained through an initial state-action model.

[0028] According to the predicted action, the state information and the result data table, a loss function value is calculated and obtained.

[0029] When the loss function value reaches a preset threshold value, a state-action model for outputting the action to be performed by the robot is obtained.

[0030] Optionally, the method further comprises:

[0031] In response to a game start instruction, the state-action model and the robot are loaded.

[0032] The state information of the loaded robot is acquired.

[0033] The acquired state information is input into the state-action model, and a target action to be performed by the robot is obtained.

[0034] The robot is controlled to perform the target action.

[0035] Optionally, the method further comprises:

[0036] Action data of the robot is acquired.

[0037] According to the action data, an action distribution of the robot is obtained.

[0038] According to the action information of the player, an action distribution of the player is obtained.

[0039] According to the action distribution of the robot and the action distribution of the player, a KL divergence value is obtained.

[0040] determine whether the state-action model needs to be trained according to the KL divergence value.

[0041] Optionally, the method further comprises:

[0042] obtaining action data of the robot;

[0043] obtaining an action distribution of the robot according to the action data;

[0044] determining a proportion of skill frames according to the action distribution of the robot;

[0045] determining whether the state-action model needs to be trained according to the proportion of the skill frames.

[0046] Embodiments of the present application also provide a robot training system in a game, comprising an information acquisition module, an information processing module and a model construction module.

[0047] The information acquisition module is configured to acquire state information of a robot to be trained and player information of a player within a preset range of the robot, wherein the player information at least comprises settlement information of a game battle corresponding to the player, state information of the player and action information of the player, and the state information of the player comprises self state information of the player and field information corresponding to the player.

[0048] The information processing module is configured to process the settlement information into a first data table in the form of metadata, process the state information of the player and the action information of the player into a second data table in the form of data pairs, and splice the first data table and the second data table into a result data table according to the field information corresponding to the player, wherein each row in the result data table contains settlement metadata information, state information and action information of the player in a game battle.

[0049] The model construction module is configured to establish a state-action model for outputting an action to be executed by the robot by imitative learning based on the state information of the robot and the result data table.

[0050] Embodiments of the present application also provide an electronic device, comprising a processor and a memory.

[0051] The memory is configured to store one or more computer instructions.

[0052] The processor is configured to execute the one or more computer instructions to implement the robot training method in a game as described above.

[0053] The embodiment of the present application further provides a computer readable storage medium, which stores one or more computer instructions, and the instructions are executed by a processor to implement the robot training method in a game as described above.

[0054] Compared with the prior art, the robot training method in a game provided by the present application comprises the following steps: obtaining state information of a robot to be trained and player information of a player in a preset range of the robot; the player information at least comprises settlement information of a game battle corresponding to the player, state information of the player and action information of the player; the state information of the player comprises self state information of the player and field information corresponding to the player; the settlement information is processed into a first data table in the form of metadata, the state information of the player and the action information of the player are processed into a second data table in the form of data pairs, and the first data table and the second data table are spliced into a result data table according to the field information corresponding to the player, wherein each row in the result data table contains settlement metadata information, state information and action information of the player in a game battle; and a state-action model for outputting an action to be executed by the robot is established through imitative learning based on the state information of the robot and the result data table. The method can obtain a state-action model corresponding to the robot by processing the obtained settlement information, state information and action information of the player into training data for imitative learning of the robot, and output an action to be executed by the robot under the state information of the robot. The robot training method in a game provided by the present application can make the trained robot more close to a real player, and can significantly improve the game experience of the player when the robot replaces an absent player to participate in the game. BRIEF DESCRIPTION OF DRAWINGS

[0055] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the description of the embodiments of the present application. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without any creative labor under the premise of the drawings.

[0056] Figure 1 is a composition architecture schematic diagram of a scene to which the robot training method provided by the embodiments of the present application is applied;

[0057] Figure 2 is a flowchart of the robot training method in a game provided by an embodiment of the present application;

[0058] Figure 3 is still another flowchart of the robot training method in a game provided by another embodiment of the present application;

[0059] Figure 4is a flowchart of processing player information by using a Spark operator provided by another embodiment of the present application;

[0060] Figure 5 is a flowchart of constructing a robot state action model provided by an embodiment of the present application;

[0061] Figure 6 is a flowchart of evaluating whether a robot is human-like as expected provided by an embodiment of the present application;

[0062] Figure 7 is a structural schematic diagram of a training system of a robot in a game provided by an embodiment of the present application;

[0063] Figure 8 is a structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0064] In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present application. However, it will be apparent to one skilled in the art that the present application can be practiced without the specific details set forth in this description. In other instances, well-known methods, procedures, components, and circuits have not been described in detail as not to unnecessarily obscure aspects of the present application.

[0065] The training method of a robot in a game provided by the present application is a robot training based on cloud technology.

[0066] Cloud computing is an Internet-based computing method, which can provide shared software and hardware resources and information to computers and other devices according to demand through a network. The network providing resources is called "cloud", and the resources in the "cloud" can be obtained at any time through the network.

[0067] Cloud technology is a related technology for realizing cloud computing, which is a general term for network technology, information technology, integration technology and the like based on cloud computing. It can integrate software and hardware resources in a network environment, such as a wide area network or a local area network, to realize data calculation, storage, processing and sharing.

[0068] Artificial intelligence (AI) is a branch of computer science that can generate intelligent machines close to human intelligence. The research in this field mainly involves technologies such as robots, language recognition, image recognition, etc. Artificial intelligence is to analyze and simulate the process of human consciousness and thinking to design artificial intelligent robots that can behave like humans. In terms of single intelligence, such as calculation, it can even exceed human intelligence.

[0069] Mathematical Modeling is a process of establishing a mathematical model, which is an important means of solving practical problems by using mathematical tools, and is a process of establishing a clear mathematical relationship between variables and parameters according to certain laws or rules.

[0070] The KL divergence value is also called relative entropy or information divergence. It is used to describe the degree of similarity between two probability distributions. For example, there are two probability distributions P and Q, and the KL divergence value represents the closeness of P and Q, that is, the degree of similarity.

[0071] Spark operators can be understood as the computing methods of Spark RDD (Resilient Distributed Dataset), which can act on each partition of the RDD. For an RDD, each partition will be processed by a computing task, and each partition can be processed in parallel. The RDD in Spark is a resilient distributed dataset. Since each transformation of the RDD generates a new RDD, there will be a dependency relationship between the RDDs. Therefore, when a partition has data errors or data loss, Spark can calculate the data of the partition according to the dependency relationship, without the need to calculate and repair all RDDs. The RDD of Spark belongs to a lazy computation, which must rely on a specific trigger to start the calculation, otherwise no result will be generated. Spark operators can be divided into Transformation (transformation / conversion operator) operators and Action (action) operators from the big category. Among them, Transformation operators such as map, flatMap, glom, etc. do not trigger the submission of jobs, but only complete the processing in the process of the job. Action operators such as foreach, collect, etc. will trigger execution and submit jobs.

[0072] The settlement information refers to the summary data of the score, win or lose, and the score, skill usage times, and remaining blood of each player in each game, and the settlement mode can depend on the application environment and design intention of the specific game.

[0073] The state information includes the state information of the robot and the state information of the player, which refers to the self-state information of the player or the robot in the game scene and the game site information, including but not limited to the position of the player or the robot in the game scene, the remaining blood, the position of the pickupable items in the game site, the remaining game time, etc. The definition of the state information can depend on the application environment and design needs of the specific game.

[0074] The action information refers to information generated when the player moves or performs a skill, including but not limited to actions performed by the player, positions at which the player performs a skill, and the like. The definition of the state information can depend on the application environment and design needs of a specific game.

[0075] To facilitate the understanding of the scheme of the present application, the following embodiments are provided for illustration. Figure 1 A composition architecture schematic diagram of a scenario to which the scheme of the present application is applicable is provided.

[0076] The robot training method of the present application is implemented by deploying a robot training system on a cloud platform. The number of robots for training can be dynamically expanded by using distributed cloud computing technology. The robot simulates the game actions and the like of a game player, and replaces the missing player to perform actions or tasks in a multiplayer online battle game.

[0077] As shown in the Figure 1 The scenario includes a cloud platform 10. The cloud platform includes a plurality of cloud servers, and the plurality of cloud servers can be communicatively connected.

[0078] The cloud platform 10 is deployed with a training-related module of a game robot. The training-related module includes a module for obtaining player data (hereinafter referred to as an obtaining module). The obtaining module can be deployed on any cloud server. In the present embodiment, the cloud server on which the obtaining module is deployed is referred to as an obtaining server 101.

[0079] In addition to the obtaining server 101, the cloud platform also includes a data processing server 102 for processing data, a model construction server 103 for training an intelligent model, a test server 104 for detection, and an implantation server 105 for implanting a robot. According to actual needs, one or more related modules can be started on each cloud server. The plurality of servers on the cloud platform are communicatively connected. For example, after the obtaining module on the obtaining server is started, other training-related modules can be simultaneously started through the communication connection.

[0080] The robot training-related module is used to train at least one robot. The trained robot can be used to simulate game actions performed by a game client. The robot establishes a communication connection between the simulation game client and the game server, and performs a game on the simulation game client, so as to simulate the game actions and the like of a game player, and replace the absent player to perform actions in the game.

[0081] In the present embodiment, the training-related module running on the cloud platform can train one or more robot models, and deploy one or more robots on the game server 20 to replace one or more absent players to perform certain or all game actions or game tasks.

[0082] As Figure 1 shown in the scenario, the robot trained in the cloud platform 10 can establish a communication connection with the game server 20 to be started, and through the robot, at least one game client terminal 30 can be simulated to log in the game server 20 and perform different game actions or tasks.

[0083] An embodiment of the present application provides a training method of a robot in a game. Figure 2 is a flowchart of the training method of the robot in the game provided by the embodiment.

[0084] As Figure 2 shown, the robot training method provided by the embodiment includes the following steps:

[0085] Step S201, obtaining state information of a robot to be trained and player information of a player in a preset range of the robot; the player information at least includes settlement information of a game battle corresponding to the player, state information of the player and action information of the player; the state information of the player includes self state information of the player and field information corresponding to the player.

[0086] This step is used to obtain state information of a robot to be trained and player information of a player required for training the robot, and to provide training data for training of the robot.

[0087] Step S202, processing the settlement information into a first data table in the form of metadata, processing the state information of the player and the action information of the player into a second data table in the form of data pairs, and splicing the first data table and the second data table into a result data table according to the field information corresponding to the player, each row in the result data table containing settlement metadata information, state information and action information of the player in a game battle.

[0088] This step is used to process the obtained training data, so that the obtained training data can meet the training requirements of the robot.

[0089] Step S203, based on the state information of the robot and the result data table, a state-action model for outputting an action to be performed by the robot is established through imitative learning.

[0090] This step is used to train a state-action model capable of outputting an action to be performed by the robot through imitative learning.

[0091] In order to more clearly illustrate the training method of the robot in the game provided by the present application, the present application further provides an embodiment to introduce the training method of the robot in the game in detail.

[0092] Figure 3 is another flowchart of the method for training a robot in a game according to an embodiment. As shown in Figure 3 the method for training a robot according to an embodiment specifically includes the following steps:

[0093] In step S301, state information of a robot to be trained and player information of players within a preset range of the robot are acquired.

[0094] The preset range refers to a distance range between a player and a robot set by a game system, which is usually a circular area with the robot as the center and a preset distance as the radius. The players within the preset range are usually players related to the state and actions of the robot, and thus the player information of the players within the preset range is acquired as the training data of the robot, so that more real robot state and actions can be obtained. The preset range depends on the application environment and design needs of a specific game.

[0095] The player information at least includes settlement information of a game battle corresponding to the player, state information of the player, and action information of the player.

[0096] The state information of the player includes self-state information of the player and field information corresponding to the player. The self-state information of the player at least includes a position of the player in a game field, a remaining blood volume of the player, a skill value of the player, etc. The field information corresponding to the player at least includes positions of obstacles in the game, positions of reward items, a remaining game time, a position of a referee, etc.

[0097] The action information of the player includes action information that can be used by the player and action information that has been used by the player, etc., and specifically includes at least one of the following: a moving action, a skill action, and a pickup action.

[0098] The state information of the robot includes self-state information of the robot and field information corresponding to the robot.

[0099] The acquired player information is related to the robot to be trained. The relevance includes: first, the acquired player information is player information of players within a preset range of the robot to be trained; second, the acquired player information is related to a game type corresponding to the robot to be trained, for example, for a battle game, the player information to be acquired can include a skill value of a player, a remaining blood volume of the player, a degree of movement of the player in the game, a time of using a skill by the player in the game, a number of opponents killed by the player, etc. For example, for a speed competition game, the player information to be acquired can include a racing car type selected by the player, a scene selected by the player, a speed distribution of the player in the game, etc.

[0100] The state information of the robot to be trained and the player information of the player can be obtained by deploying a function module for obtaining information on the obtaining server 101. For example, a function module for obtaining data of a battle game or a function module for obtaining data of a speed competition game is deployed on the obtaining server 101.

[0101] When the game server 20 is started, the obtaining server 101 starts to obtain the state information of the robot to be trained and the player information of the player in the preset range of the robot through the communication connection with the game server 20. The obtained robot state information and player information are stored in the data processing server 102. Of course, the above-mentioned function of the obtaining server 101 can also be integrated into the game server 20.

[0102] In step S302, the settlement information is processed into a first data table in the form of metadata, the state information of the player and the action information of the player are processed into a second data table in the form of data pairs, and the first data table and the second data table are spliced into a result data table according to the corresponding field information of the player.

[0103] After obtaining the player information of the player, the player information needs to be processed to become data that can be used for robot training.

[0104] In this step, the Spark operator is mainly used to process the settlement information into a first data table in the form of metadata, process the state information of the player and the action information of the player into a second data table in the form of data pairs, and splice the first data table and the second data table into a result data table according to the corresponding field information of the player.

[0105] The Spark operator can be mapped on the data processing server 102, and the obtained player information can be processed into a data form that can be used for robot training through the Spark operator. The Spark operator can include a plurality of data parsing functions, such as a map operator and a reduce operator. The type of the Spark operator mapped on the data processing server is determined according to the type of the player data to be processed. Different games can require different Spark operators.

[0106] The Spark operator can also simultaneously map the data parsing function to a plurality of data blocks in the data processing server 102 to perform parallel parsing on the player information. The parsing time can also be set, and the Spark operator will automatically process the player information periodically at the set time.

[0107] Figure 4 is a flowchart of processing player information by using a Spark operator provided in this embodiment.

[0108] As Figure 4 shown, the step of processing the player information by using the Spark operator includes:

[0109] Step S302-1, processing the settlement information into a first data table in the form of metadata.

[0110] Specifically, the settlement data at the end of each round is obtained and processed into a first data table in the form of metadata. The first data table describes the settlement information of all players at the end of a game, including the win or loss and scores, the number of skill uses, the remaining blood volume, etc.

[0111] According to the first data table, information about the familiarity, activity, investment of the players to the game, and the player loss can be obtained.

[0112] For example, in the first week, there are a total of 56577 players, including 36577 non-loss players and 20000 loss players. After excluding players with less than 20 actions, there are 30000 non-loss players and 10000 loss players, with a ratio of about 3:1.

[0113] The average value of the time interval of the player actions, the battle power value of the player, and the average growth rate of the level can reflect the familiarity of the player to the game.

[0114] The battle power value of several players and the average growth rate of the level can reflect the activity of the players in the game.

[0115] The type of player actions and the duration of the players in the game can reflect the investment of the players to the game itself.

[0116] According to the first data table, players meeting the preset conditions can also be screened.

[0117] For example, a multi-player online battle game R is settled, and the settlement information is processed into a first data table in the form of metadata. This embodiment only gives the related data of 10 people. The first data table at least includes the player code, the remaining blood volume, the score of this round, the number of enemy soldiers killed, the moving time in the game, the number of skill uses, the type of skills used, etc. The contents in the first data table can be used as parameters for screening players meeting the preset conditions. This embodiment provides the following first data table:

[0118]

[0119]

[0120] Of course, in the actual processing process, the expression form of this data table can be a digital code or the like.

[0121] The embodiment also provides a method for screening players meeting preset conditions according to the first data table. The preset conditions are set as medium-high levels, that is, players of medium-high quality are screened according to the first data table. The method comprises the following steps:

[0122] Firstly, a parameter standard value is determined, that is, a parameter value capable of reaching a medium-high level is determined according to big data analysis of player levels in a previous period of time, and the parameter value is taken as the parameter standard value. For example, big data analysis is performed on all sessions of the multi-player online battle game R performed last week, and the residual blood volume, session score, number of enemy soldiers killed, moving time in the game, skill use frequency and skill type are taken as parameters for ranking players. There are totally 50,000 players in the game last week, and the players are ranked from high to low quality. The top 10% of the players, that is, players ranked 1-5000, are defined as high-quality players, and the top 20% of the players, that is, players ranked 5001-10000, are defined as medium-quality players. Then, the parameter value of the player ranked 5000 is taken as the parameter standard value of the high-quality player, and the parameter value of the player ranked 10000 is taken as the parameter standard value of the medium-quality player.

[0123] The embodiment is further described by taking screening of medium-high-quality players as an example. The residual blood volume of the player ranked 10000 is 2000 points, the session score is 350 points, the number of enemy soldiers killed is 5, the moving time is 15 minutes, the skill use frequency is 20 times, and the skill type is 5. Then, in this week, the residual blood volume of 2000 points, the session score of 350 points, the number of enemy soldiers killed of 5, the moving time of 15 minutes, the skill use frequency of 20 times and the skill type of 5 will become the parameter standard value.

[0124] Secondly, medium-high-quality players are screened according to the parameter standard value. The players of the multi-player online battle game R performed this week are screened according to the determined parameter standard value. Only the players whose corresponding parameters are all greater than the parameter standard value can be determined as medium-high-quality players. The 10 players shown in the first data table are taken as players to be screened this week. Through comparison, it is found that there are two players whose corresponding parameters can all meet the conditions of the residual blood volume being greater than 2000 points, the session score being greater than 350 points, the number of enemy soldiers killed being greater than 5, the moving time being greater than 15 minutes, the skill use frequency being greater than 20 times and the skill type being greater than 5, and the player codes are player 1 and player 4. Only 10 players are taken as examples for description in the embodiment, and therefore, the medium-high-quality players in the multi-player online battle game R this week at least include player 1 and player 4.

[0125] Similarly, the parameter standard value of the high-quality player in the next week screening is determined according to the parameter analysis result of all players in the multi-player online game R of all matches in this week. Of course, the time period can be one week, two weeks, or 3 days, which is determined according to the popularity of the game.

[0126] In step S302-2, the state information and the action information of the player are processed into a second data table in the form of data pairs.

[0127] Specifically, the state information and the action information of the player in the game are obtained and processed into a second data table in the form of data pairs. The second data table describes the state information and the action information of all players in the game process, and the state information and the action information are one-to-one corresponding.

[0128] The embodiment is also described with the multi-player online game R as an example.

[0129] The state information and the action information of multiple players obtained in a multi-player online game R are processed into a second data table in the form of state-action data pairs.

[0130] The embodiment only gives the second data table corresponding to the 10 players in the first data table. The second data table at least includes the player code, the player's remaining blood volume, the game remaining time, the moving step number, the fire attack skill use number, the ice freezing skill use number, the machine gun skill use number, and the pickup item number. The second data table is a data table containing real-time state information and real-time action information of the player, so in a game, a player can have multiple state-action data pairs, and a game will include multiple second data tables. The embodiment only shows the second data table at a certain moment of the game, and the second data table provided by the embodiment is as follows:

[0131]

[0132]

[0133] Taking player 1 as an example, the second data table includes the state information and the action information of player 1, wherein the state information of player 1 includes: 4000 points of remaining blood volume, 10 minutes of remaining game time, and the action information of player 1 includes: 10564 moving step number, 6 times of fire attack skill use number, 7 times of ice freezing skill use number, 9 times of machine gun skill use number, and 8 pickup item numbers. These information are all corresponding to the player information at 10 minutes of the game, that is, for player 1, 30, or 60, or more state-action data pairs will be generated in a 30-minute multi-player online game R, and relatively, 30, or 60, or more second data tables will be generated in a 30-minute multi-player online game R.

[0134] The number of second data tables is determined according to specific game scenes. For example, in a 3-minute speed racing game, the number of second data tables required for a simple level game scene is different from that for a difficult level game scene. For the simple level game scene, because the scene changes little and the player moves slowly, the player's state information, scene information, and player's action information may not change much in 3 seconds, so 3 seconds can be used as the frequency of obtaining state action data tables, and thus 60 second data tables can be obtained for the game. For the difficult level game scene, because the scene changes greatly and the player moves quickly, the player's state information, scene information, and player's action information may change greatly in 2 seconds, so a time less than 2 seconds, such as 1 second, can be used as the frequency of obtaining second data tables, and thus 180 second data tables can be obtained.

[0135] The second data table in the form of a data pair obtained by processing the player's state information and the player's action information further includes: screening target state information and target action information that meet a preset condition according to the player's state information and the player's action information; and processing the target state information and the target action information into a second data table in the form of a data pair.

[0136] The first data table can be used to screen players that meet a preset condition, and the state information corresponding to the players that meet the preset condition is target state information that meets the preset condition, and the action information corresponding to the players that meet the preset condition is target action information that meets the preset condition. Therefore, the state information includes the target state information, and the action information includes the target action information.

[0137] Taking the above multi-player online game R as an example, and taking a medium-high level as a preset condition, the medium-high quality players screened by the first data table include player 1 and player 4. The state information corresponding to player 1 and player 4 is target state information, and the action information corresponding to player 1 and player 4 is target action information.

[0138] The second data table in the form of a data pair obtained by processing the target state information and the target action information is a second data table that meets the preset condition.

[0139] That is, the second data table includes state information and action information of all players, and the second data table that meets the preset condition includes only target state information and target action information of players that meet the preset condition.

[0140] The second data table meeting the preset condition is used as robot training data, and a robot meeting the preset condition can be trained.

[0141] In the large-scale multi-player battle game R provided in the embodiment, a player needs to perform a moving action to find a path, find an enemy, avoid an enemy, find a pickup, and arrange a tactic with a teammate. It can be said that the player spends most of the time in the game in moving. Therefore, the moving data will be much more than the skill data in the obtained player action information, that is, the action tags that are the data of moving (referred to as moving frames) will account for a relatively large quantity, and the action tags that are the data of skills (referred to as skill frames) will account for a relatively small quantity. If the data with an unbalanced ratio of moving frames and skill frames is directly connected to the training model for robot training, the robot trained may only move and will not release a skill after reaching a destination, for example, will not perform an item pickup action after moving to the address of a pickup, or will not shoot an enemy after moving to the vicinity of the enemy. The robot

[0142] In a speed competition game, the moving action accounts for an absolutely large proportion. The robot trained with such data may only move in the game and will not avoid a trap or accelerate. Such a robot does not meet the requirements and will drag a player in the game and reduce the experience of the player.

[0143] Therefore, it is necessary to optimize the quantity of skill frames and moving frames in the action information.

[0144] The optimization content includes: determining whether the first N action frames before a target skill frame in the action information are moving frames, and if so, modifying the first N action frames before the target skill frame in the action information to skill frames; and / or determining whether the last N action frames after the target skill frame are moving frames, and if so, modifying the last N action frames after the target skill frame in the action information to skill frames; and processing the state information of the player and the modified action information of the player into a second data table in the form of a data pair, wherein the N is a positive integer.

[0145] Based on the analysis of the action information, it is found that the N action frames (the specific value of N can be adjusted) before and after the target skill frame are all moving frames. It can be considered that the state of these moving frames is close to the target skill frame, and these part of action frames can be modified as skill frames. Specifically, the action label of the moving frame is modified as the action label corresponding to the skill frame. For example, the action data of a player in a large multi-player battle game R is analyzed, and it is found that the 100 frames before the second skill frame are all moving frames, the 101st frame is the first skill frame, the 130 frames after the second skill frame are all moving frames, and the 131st frame is the third skill frame. The smaller value of the number of moving frames before and after the second skill frame is taken as the value of N, and it can be considered that the state of the 100 moving frames before and after the second skill frame is close to the second skill frame. Further, the action label of the 100 frames before and after the second skill frame is modified from the moving frame to the skill frame, thereby increasing the number of skill frames.

[0146] At the same time of modification, the corresponding skill-related information in the state information of the player needs to be adjusted according to the modification of the N action frames before the target skill frame and / or the modification of the N action frames after the target skill frame. Specifically, the skill-related information in the state information is modified to a legal value.

[0147] The optimization content further includes: detecting whether there are continuous M moving frames in the action information of the player, wherein M is a positive integer; if so, deleting part of the M moving frames to obtain modified action information; and processing the state information of the player and the modified action information into a second data table in the form of a data pair.

[0148] Based on the analysis of the action information, it is found that the M action frames (the specific value of M can be adjusted) in a certain time period are all the same moving actions. It can be considered that only one frame needs to be retained, and the other frames are deleted. For example, the action information of a player in a large multi-player battle game R is analyzed, and it is found that the 150 frames of moving frames in the 3rd minute are all moving actions of slow walking forward. The 150 frames can be taken as the value of M. Further, one frame is randomly selected from the 150 frames to be retained, and the other 149 frames are deleted from the action information, thereby reducing the number of moving frames.

[0149] The embodiment provides an optional way to increase the number of skill frames and reduce the number of moving frames. Of course, data augmentation and data balancing can also be achieved through more ways, which will not be listed one by one here.

[0150] In step S302-3, the first data table and the second data table are spliced into a result data table according to the corresponding field information of the player.

[0151] The first data table obtained through step S302-1 is spliced with the second data table obtained through step S302-2, and a result data table is obtained.

[0152] The result data table contains player information of multiple players. Each row in the result data table contains settlement metadata information, state information, and action information of a player in a game battle. Since multiple state-action data pairs of a player are generated in a game, the result data table is actually a data set containing player information of all players. Specifically, the content of a row in the result data table includes settlement metadata information, a set of state information, and a set of action information of a player.

[0153] This embodiment also takes the above multi-player online battle game R as an example for specific description. This embodiment only shows a result data table of 10 players, which corresponds to the players in the first data table and the second data table. The result data table shown in this embodiment is only a result data table corresponding to a moment of the game and only contains a part of data. The result data table provided in this embodiment is as follows:

[0154]

[0155]

[0156] Taking the row with the serial number 1 as an example, this row includes settlement metadata information, state information, and action information of player 1.

[0157] In step S303, a state-action model for outputting an action to be performed by the robot is established through imitation learning based on the state information of the robot and the result data table.

[0158] The processed result data table is used as training data, and a state-action model of the robot is constructed through iterative training. The model can output an action to be performed by the robot based on state information of the robot. This step can be implemented in the model construction server 103.

[0159] Figure 5 This embodiment provides a flowchart for constructing a state-action model of a robot. As shown in Figure 5 The step of establishing a state-action model for outputting an action to be performed by the robot through imitation learning includes:

[0160] In step S303-1, a predicted action of the robot is obtained through an initial state-action model based on the state information of the robot and the result data table.

[0161] One optional implementation of this step includes the following content:

[0162] First, the players meeting the preset condition are filtered according to the settlement metadata information in the result data table, and the corresponding target state information and target action information are obtained.

[0163] Of course, the target state information and target action information meeting the preset condition can also be filtered in the data processing stage.

[0164] The specific filtering method of this step is consistent with the filtering method in the data processing stage, and will not be described here.

[0165] Second, the result data table is used as training data to perform iterative training of the robot. Of course, using the result data table that has been filtered and optimized as training data to perform iterative training of the robot is a preferred implementation manner, and a robot meeting the preset condition can be obtained.

[0166] The training data can be continuously expanded. For example, the state information and action information of the medium and high quality players that have been processed and filtered can be added to the result data table on a regular basis. The current result data table includes the state information and action information of 10,000 medium and high quality players. On the first day, the player information of 10,000 players is obtained, and 2,000 medium and high quality players are filtered according to the settlement metadata information. The state information and action information corresponding to the 2,000 medium and high quality players can be connected to the current result data table. On the second day, the player information of 15,000 players is obtained, and 2,200 medium and high quality players are filtered according to the settlement metadata information. The state information and action information corresponding to the 2,200 medium and high quality players can be connected to the current result data table. On the third day, the data processing, filtering and connection can be continued. Of course, the data connection period and the connection quantity are determined according to the specific game situation.

[0167] The result data table is used as training data to perform 1,000 or 2,000 or more iterations of training of the robot. The iterations, the specific number of iterations is determined according to the quantity and difficulty of the content that the robot needs to learn. For example, for a slightly simple game, the learning content required by the robot will be less. For such a robot, the target accuracy rate can appear after 1,000 iterations. For a more difficult game, the content that the robot needs to learn will be more, and the learning content will be more difficult. For such a robot, the target accuracy rate can appear after 2,000 or 3,000 or more iterations.

[0168] Third, the predicted action of the robot is obtained through the initial state-action model.

[0169] After iterative training, the robot can obtain the predicted action of the robot according to the state information of the robot.

[0170] Since the state information and action information of the player in the training data are one-to-one corresponding data pairs, the state-action model obtained by the robot imitation learning also has a one-to-one corresponding relationship between the state information and the action information, and therefore the predicted action of the robot can be obtained according to the state information of the robot.

[0171] In step S303-2, a loss function value is calculated according to the predicted action, the state information and the result data table.

[0172] In the iterative training of the robot, all algorithms depend on minimizing or maximizing a certain function, which is referred to as a “target function”. The minimized target function is a loss function, which can also be called a cost function. The loss function can be used to evaluate the degree of inconsistency between the predicted value of the model and the true value, and is an index for measuring the performance of the prediction model in predicting the expected result, and is also the optimization target function in the neural network. The neural network training process is actually a process of minimizing the loss function. If the obtained loss function is smaller, it means that the predicted value of the model is closer to the true value, and the accuracy and robustness of the model are better.

[0173] For example, the loss function between the input and the target is calculated. The loss function can be used to measure the difference between different continuous distributions and is very effective when directly regressing in the continuous output distribution space.

[0174] One way to evaluate the difference between two groups of data is to calculate the loss function between the actual output value and the target output value. In this embodiment, the predicted action is the actual output value, and the action information corresponding to the state information of the robot in the result data table is the target output value.

[0175] The calculation method of the loss function is as follows:

[0176]

[0177]

[0178] The loss function is

[0179] Where y represents the actual output value, y* represents the target output value, and n represents n pieces of data.

[0180] The calculation process is as follows:

[0181]

[0182] Step S303-3, when the loss function value reaches a preset threshold, obtaining a state-action model for outputting the action to be performed by the robot.

[0183] The smaller the loss function, the closer the predicted value of the model to the true value, and the better the accuracy and robustness of the model. That is, the smaller the loss function, the closer the action to be performed by the robot output by the state-action model to the action information of the player. In actual operation, a preset threshold (such as 0.1) is often set, and when the loss function is less than the preset threshold, it can be judged that the model has been trained.

[0184] Step S304, loading the state-action model and the robot in the real battle scene, and judging whether the robot meets the expectation.

[0185] The real battle scene in this step refers to the same game scene as online games, but no real players are added, and the participants are robots. This step can be performed on the test server 104. Specifically, according to the game requirements, a certain number of robots are loaded in the real battle scene, and the game is started. For example, for a 40-player online battle game, 40 robots need to be loaded in a game for group battle.

[0186] Since the real battle scene where the robot is located does not add real players, it will not affect the online game. Only the robots that meet the expectations can be implanted into the online game to replace the absent players and play with real players.

[0187] The robot performs tasks according to the game progress in the real battle scene, such as automatically finding a path, reaching a city, and starting an attack. Or, cutting trees with stones, hunting, digging minerals… All actions and skills contained in each game can be implemented. From collecting raw materials to building, hunting, farming, animal husbandry, attacking opponents, etc., perform the corresponding actions according to the current state information of the robot. No human intervention or help or stopping the game is needed during the process.

[0188] Judging whether the robot meets the expectation mainly includes the degree of human-likeness and intensity. The degree of human-likeness is taken as an example for detailed description.

[0189] Figure 6 is a flowchart provided by the embodiment for evaluating whether the degree of human-likeness of the robot meets the expectation. As shown in Figure 6 the method includes the following steps:

[0190] Step S304-1, obtaining the action data of the robot, and obtaining the action distribution of the robot according to the action data.

[0191] The action information of the robot is collected during the game, and the action data table corresponding to the robot is made, and the action distribution of the robot is calculated according to the action data table. The robot action data table at least includes the moving step number, the trap avoidance number, the fire attack number, the shooting number, and the total skill use number of the robot.

[0192] Specifically, the average value of each action data of all robots implanted in a game is calculated to obtain the action distribution of the robot. The action distribution at least includes the moving step number, the trap avoidance number, the fire attack number, the shooting number, and the total skill use number of the robot in the game.

[0193] This embodiment gives a case of the robot action data table, and only the action data of 5 robots is randomly selected, and the action data table is as follows:

[0194]

[0195] Further, the action distribution of the robot is calculated according to the action data provided in the above embodiment. Specifically, the average value of the moving step number, the trap avoidance number, the fire attack number, the shooting number, and the total skill use number is calculated, and the calculation result is as follows:

[0196]

[0197]

[0198] That is, the action distribution of the robot provided in this embodiment is that the moving step number is 9894.6 steps, the trap avoidance number is 4.6 times, the fire attack number is 5.6 times, the shooting number is 8.6 times, and the total skill use number is 23.4 times.

[0199] Of course, multiple robots can be deployed at the same time in a real battle scene. For example, according to the number of players required to start the game, the same number of robots is deployed, the robots are grouped, and all the robots replace the real players to play the game. In this way, the comprehensive ability of the robot can be evaluated. If too few robots are deployed, some skills may not be used or used too little in the evaluation, so that the overall intelligence of the robot cannot be evaluated.

[0200] Step S304-2, obtaining the action distribution of the player according to the action information of the player.

[0201] The action information of the player can be obtained from the result data table, and the calculation method of the action distribution of the player is consistent with the calculation method of the action distribution of the robot, which will not be described here.

[0202] Step S304-3: Obtain the KL divergence value based on the robot's action distribution and the player's action distribution.

[0203] Human-likeness assessment compares the robot's motion distribution with the player's motion distribution, calculating the Karl von Leyen divergence (KL divergence) between the two sets of motion distributions. This KL divergence serves as the evaluation value for human-likeness. A smaller KL divergence indicates a higher degree of human-likeness, while a larger KL divergence indicates a lower degree of human-likeness. If the KL divergence is less than a set threshold, the robot is considered to meet expectations in terms of human-likeness.

[0204] The following uses the KL divergence between the rule-based robot's and the player's action distributions as a threshold to evaluate the human-likeness of the trained robot. The specific steps include:

[0205] First, the frequency of each action in the player action data collected before training is statistically analyzed to obtain the player action distribution (A); second, the frequency of each action of the robot during testing is statistically analyzed to obtain the robot action distribution (B); third, the frequency of each action of the rule-based robot during testing is statistically analyzed to obtain the rule-based robot action distribution (C); fourth, the KL divergence (D) between the robot and player action distributions is calculated. KL (A||B)); Fifth, calculate the KL divergence (D) between the action distributions of the rule-based robot and the player. KL (A||C)); Sixth, using the KL divergence between the robot's and the player's action distributions as a threshold, we determine whether the robot meets expectations. Specifically, if the KL divergence between the robot's and the player's action distributions is less than the threshold, then the robot's action distribution is closer to the player's, and we can judge that the robot is more human-like and meets expectations. Otherwise, the human-likeness is low and does not meet expectations.

[0206] This embodiment uses four actions in the action distribution as an example for illustration. The specific calculation method is as follows:

[0207] The player's action distribution (A) is [0.1, 0.2, 0.4, 0.3].

[0208] The robot's motion distribution (B) is [0.08, 0.22, 0.37, 0.33].

[0209] The distribution of robot actions (C) according to the rules is [0.25, 0.25, 0.25, 0.25].

[0210] Calculate the KL divergence between the robot and player action distributions:

[0211]

[0212] Calculate the KL divergence between the action distributions of the rule-based bot and the player:

[0213]

[0214] From the above calculation results, it can be seen that the KL divergence of the robot and the player action distribution is less than the KL divergence of the rule robot and the player action distribution, which indicates that the action distribution of the trained robot is closer to the player, the robot is more human-like, and meets the expectation.

[0215] Of course, the threshold value can also have various setting methods, such as the relative entropy between the action distribution of the player in a game and the average value of the action distribution of the player in multiple games. For example, the relative entropy between the action distribution of the previously trained robot in the current game and the action distribution of the player.

[0216] Step S304-4, determining whether the state-action model needs to be trained according to the divergence value.

[0217] As described above, if the KL divergence is less than the set threshold value, it can be judged that the robot meets the expectation in terms of human-like degree. If the robot meets the expectation in terms of human-like degree, the training of the state-action model can be stopped, otherwise, the state-action model needs to be trained again.

[0218] The strength evaluation is an evaluation of the game quality of the robot, and the evaluation content at least includes the score and the skill proportion.

[0219] The score is an important indicator for evaluating the strength of the robot, and the embodiment provides a case of evaluating the strength of the robot through the score. Specifically, in the same game scene, if the average score of the robot is greater than the average score of the player, it can be judged that the strength of the robot is high and meets the expectation. Otherwise, the strength is low and does not meet the expectation.

[0220] The method for evaluating the strength of the robot through the score is not only the above-mentioned case, but also can be to compare the average score of the robot with the average score of the rule robot in the same game scene. If the average score of the robot is greater than the average score of the rule robot, it can be judged that the strength of the robot is high and meets the expectation. Otherwise, the strength is low and does not meet the expectation.

[0221] The method for evaluating the strength of the robot through the score can also be to compare the average score of the robot obtained by the current training with the average score of the robot obtained by the previous training in the same game scene. If the average score of the robot obtained by the current training is greater than the average score of the robot obtained by the previous training, it can be judged that the strength of the robot is high and meets the expectation. Otherwise, the strength is low and does not meet the expectation.

[0222] Of course, the method for evaluating the strength of the robot through the score includes but is not limited to the above-mentioned method.

[0223] The skill proportion is another important indicator for evaluating the strength of the robot. The skill proportion is determined by calculating the proportion of the skill frame of the robot in the action data (hereinafter referred to as the skill frame proportion value) and comparing it with the skill frame proportion value of the real player to determine whether the skill level of the robot reaches the expectation.

[0224] The specific method includes: obtaining the action data of the robot; obtaining the action distribution of the robot according to the action data; determining the proportion of the skill frame according to the action distribution of the robot; and determining whether the skill level of the robot reaches the expectation and whether the state-action model needs to be trained according to the proportion of the skill frame.

[0225] As an optional implementation for evaluating the strength of the robot by the proportion of the skill frame, the average skill frame number of the robot is calculated according to the action distribution of the robot, and the skill frame proportion value is calculated. The skill frame proportion value of the robot is compared with the skill frame proportion value of the player. If the skill frame proportion value of the robot is within the range of 0-0.5% of the skill frame proportion value of the player, it is determined that the skill of the robot reaches the expectation. If the skill frame proportion value of the robot exceeds the range of ±5% of the skill frame proportion value of the player, it is determined that the skill of the robot does not reach the expectation.

[0226] The embodiment provides an implementation of skill proportion evaluation as shown in the following table.

[0227]

[0228] As shown in the above table, the skill frame proportion value of the robot is 19.68%, and the skill frame proportion value of the player is 19.27%. By calculating 19.68%-19.27%=0.41%, 0.41% is greater than 0 and less than 0.5%, which indicates that the skill frame proportion value of the robot is within the range of 0-0.5% of the skill frame proportion value of the player. The skill proportion of the robot reaches the expectation.

[0229] In this step, the data of the corresponding player is selected for comparison according to the expectation of the robot. For example, if the expected robot needs to reach a high quality level, the data of the high level player is selected for comparison.

[0230] Of course, the skill proportion can also be determined by comparing the skill frame proportion value of the robot with the skill frame proportion value of the rule robot to determine whether the robot reaches the expectation.

[0231] The above performance evaluation of the robot according to the action distribution also includes: if the human-like degree and the strength of the robot meet the expectation, the training of the state-action model is ended. If the expectation is not met, the training is continued in step S303.

[0232] Step S305: Load the state action model and the robot into the game, and control the robot to perform the target action.

[0233] The robot and its state action model that meet the requirements are stored on the implantation server 105, and the robot is implanted according to the game requirements. Specifically, the number of robots to be implanted is determined based on the number of players in the game.

[0234] The specific method is as follows: in response to the game start command, load the state action model and the robot; obtain the state information of the loaded robot; input the obtained state information into the state action model to obtain the target action to be performed by the robot; control the robot to perform the target action.

[0235] The state-action model includes the robot's state information and the corresponding action information obtained through imitation learning. When the game starts, based on the robot's state information, the target action to be performed by the robot can be obtained through the trained state-action model. For example, if the robot's current state is next to the object to be picked up but the object is not in the robot's backpack, then the current target action to be performed by the robot can be determined through the state-action model as picking up the object.

[0236] Once the target action to be performed by the robot is obtained, the robot can be controlled to perform the target action, such as picking up the object to be picked up.

[0237] One embodiment of this application provides a training system for robots in games. Figure 7 This is a schematic diagram of the structure of the training system for the robot in the game provided in this embodiment.

[0238] like Figure 7 As shown, the training system for a robot in a game provided in this embodiment includes: an information acquisition module 701, an information processing module 702, a model building module 703, a first selection module 704, and a second selection module 705.

[0239] The information acquisition module 701 is used to acquire the status information of the robot to be trained, and the player information of the players within the preset range of the robot; the player information includes at least the game battle settlement information corresponding to the player, the player's status information, and the player's action information; the player's status information includes the player's own status information and the field information corresponding to the player.

[0240] Optionally, the robot's state information includes the robot's own state information and the site information corresponding to the robot.

[0241] Optionally, the action information comprises at least one of the following: a moving action, a skill action, and a picking action.

[0242] The information processing module 702 is configured to process the settlement information into a first data table in the form of metadata, process the state information of the player and the action information of the player into a second data table in the form of data pairs, and splice the first data table and the second data table into a result data table according to the field information corresponding to the player, wherein each row in the result data table contains settlement metadata information, state information, and action information of the player in a game match.

[0243] Optionally, the processing of the settlement information into the first data table in the form of metadata, the processing of the state information of the player and the action information of the player into the second data table in the form of data pairs, and the splicing of the first data table and the second data table into the result data table according to the field information corresponding to the player comprise: using a Spark operator to process the settlement information into the first data table in the form of metadata, process the state information of the player and the action information of the player into the second data table in the form of data pairs, and splice the first data table and the second data table into the result data table according to the field information corresponding to the player.

[0244] Optionally, the processing of the state information of the player and the action information of the player into the second data table in the form of data pairs comprises:

[0245] determining whether the first N action frames of the target skill frame in the action information are all moving frames, and if so, modifying the first N action frames of the target skill frame in the action information to skill frames;

[0246] and / or,

[0247] determining whether the last N action frames of the target skill frame are all moving frames, and if so, modifying the last N action frames of the target skill frame in the action information to skill frames;

[0248] processing the state information of the player and the modified action information of the player into the second data table in the form of data pairs, wherein the N is a positive integer.

[0249] Optionally, the method further comprises:

[0250] adjusting the skill-related information corresponding to the state information of the player according to the modification of the first N action frames of the target skill frame and / or the modification of the last N action frames of the target skill frame.

[0251] Optionally, the processing of the state information of the player and the action information of the player into the second data table in the form of data pairs comprises:

[0252] detecting whether there are continuous M movement frames in the action information of the player, wherein M is a positive integer;

[0253] if so, deleting part of the M movement frames to obtain modified action information;

[0254] processing the state information of the player and the modified action information into the second data table in the form of data pairs.

[0255] Optionally, the processing of the state information of the player and the action information of the player into the second data table in the form of data pairs comprises:

[0256] screening target state information and target action information meeting a preset condition according to the state information of the player and the action information of the player, wherein the state information comprises the target state information and the action information comprises the target action information;

[0257] processing the target state information and the target action information into the second data table in the form of data pairs.

[0258] The model construction module 703 is configured to establish, based on the state information of the robot and the result data table, a state-action model for outputting an action to be performed by the robot through imitative learning.

[0259] Optionally, the establishment, based on the state information of the robot and the result data table, of the state-action model for outputting an action to be performed by the robot through imitative learning comprises:

[0260] obtaining, based on the state information of the robot and the result data table, a predicted action of the robot through an initial state-action model;

[0261] calculating a loss function value according to the predicted action, the state information and the result data table;

[0262] when the loss function value reaches a preset threshold, obtaining the state-action model for outputting an action to be performed by the robot.

[0263] The first selection module 704 is configured to, in response to a game start instruction, load the state-action model and the robot, acquire state information of the loaded robot, input the acquired state information into the state-action model to obtain a target action to be performed by the robot, and control the robot to perform the target action.

[0264] The second selection module 705 is used to acquire the robot's motion data; obtain the robot's motion distribution based on the motion data; obtain the player's motion distribution based on the player's motion information; obtain the KL divergence value based on the robot's motion distribution and the player's motion distribution; and determine whether the state-motion model needs to be trained based on the divergence value.

[0265] The second selection module 705 is further configured to acquire the robot's motion data; obtain the robot's motion distribution based on the motion data; determine the proportion of skill frames based on the robot's motion distribution; and determine whether the state-motion model needs to be trained based on the proportion of skill frames.

[0266] One embodiment of this application provides an electronic device. Figure 8 This is a schematic diagram of the structure of the electronic device provided in this embodiment.

[0267] like Figure 8 As shown, the electronic device provided in this embodiment includes a memory 801 and a processor 802.

[0268] The memory 801 is used to store computer instructions for executing training methods for robots in the game.

[0269] The processor 802 is used to execute computer instructions stored in the memory 801, and performs the following steps:

[0270] The system acquires the status information of the robot to be trained, as well as the player information of players within a preset range of the robot; the player information includes at least the game battle settlement information corresponding to the player, the player's status information, and the player's action information; the player's status information includes the player's own status information and the field information corresponding to the player.

[0271] The settlement information is processed into a first data table in the form of metadata, and the player's status information and the player's action information are processed into a second data table in the form of data pairs. The first data table and the second data table are concatenated into a result data table according to the player's corresponding venue information. Each row in the result data table contains the player's settlement metadata information, status information and action information in a game battle.

[0272] Based on the robot's state information and the result data table, a state-action model for outputting the robot's actions to be performed is established through imitation learning.

[0273] Optionally, the robot's state information includes the robot's own state information and the site information corresponding to the robot.

[0274] Optionally, the action information comprises at least one of the following: a moving action, a skill action, and a picking action.

[0275] Optionally, the processing of the settlement information into the first data table in the form of metadata, the processing of the state information of the player and the action information of the player into the second data table in the form of data pairs, and the splicing of the first data table and the second data table into the result data table according to the field information corresponding to the player comprise: using a Spark operator to process the settlement information into the first data table in the form of metadata, to process the state information of the player and the action information of the player into the second data table in the form of data pairs, and to splice the first data table and the second data table into the result data table according to the field information corresponding to the player.

[0276] Optionally, the processing of the state information of the player and the action information of the player into the second data table in the form of data pairs comprises:

[0277] determining whether the first N action frames of the target skill frame in the action information are all moving frames, and if so, modifying the first N action frames of the target skill frame in the action information into skill frames;

[0278] and / or,

[0279] determining whether the last N action frames of the target skill frame are all moving frames, and if so, modifying the last N action frames of the target skill frame in the action information into skill frames;

[0280] processing the state information of the player and the modified action information of the player into the second data table in the form of data pairs, wherein the N is a positive integer.

[0281] Optionally, the method further comprises:

[0282] adjusting the skill-related information in the state information of the player according to the modification of the first N action frames of the target skill frame and / or the modification of the last N action frames of the target skill frame.

[0283] Optionally, the processing of the state information of the player and the action information of the player into the second data table in the form of data pairs comprises:

[0284] detecting whether there are M continuous moving frames in the action information of the player, wherein M is a positive integer;

[0285] if so, deleting part of the moving frames in the M moving frames to obtain modified action information;

[0286] The state information of the player and the modified action information are processed into a second data table in the form of data pairs.

[0287] Optionally, the processing of the state information of the player and the action information of the player into the second data table in the form of data pairs comprises:

[0288] According to the state information of the player and the action information of the player, target state information and target action information meeting preset conditions are screened, the state information comprises the target state information, and the action information comprises the target action information.

[0289] The target state information and the target action information are processed into a second data table in the form of data pairs.

[0290] Optionally, the state-action model for outputting the action to be performed by the robot is established through imitative learning based on the state information of the robot and the result data table, and the state-action model comprises:

[0291] Based on the state information of the robot and the result data table, a predicted action of the robot is obtained through an initial state-action model.

[0292] According to the predicted action, the state information and the result data table, a loss function value is calculated and obtained.

[0293] When the loss function value reaches a preset threshold value, a state-action model for outputting the action to be performed by the robot is obtained.

[0294] Optionally, the method further comprises:

[0295] In response to a game start instruction, the state-action model and the robot are loaded.

[0296] State information of the loaded robot is acquired.

[0297] The acquired state information is input into the state-action model, and a target action to be performed by the robot is obtained.

[0298] The robot is controlled to perform the target action.

[0299] Optionally, the method further comprises:

[0300] Action data of the robot is acquired.

[0301] According to the action data, an action distribution of the robot is obtained.

[0302] According to the action information of the player, an action distribution of the player is obtained.

[0303] According to the action distribution of the robot and the action distribution of the player, a KL divergence value is obtained;

[0304] According to the divergence value, it is determined whether the state-action model needs to be trained.

[0305] Optionally, the method further comprises:

[0306] Obtaining action data of the robot;

[0307] According to the action data, an action distribution of the robot is obtained;

[0308] According to the action distribution of the robot, a proportion of skill frames is determined;

[0309] According to the proportion of the skill frames, it is determined whether the state-action model needs to be trained.

[0310] An embodiment of the present application provides a computer readable storage medium, which stores one or more computer instructions, and the instructions are executed by a processor to implement the method described in the above embodiments of the present application.

[0311] It should be noted that the terms such as "first", "second" and the like in the present document are only used to distinguish one entity or operation from another entity or operation, and do not require or imply any actual relationship or sequence between the entities or operations. In addition, the terms "include", "have", "contain" and "comprise" and other similar forms are the same in meaning, and the end of any one or more items after any one of the above terms is open, and any one of the above nouns does not mean that the one or more items have been listed exhaustively or are limited to the one or more items listed.

[0312] As used herein, unless otherwise expressly specified, the term "or" includes all possible combinations, except that which is not feasible. For example, if it is expressed that a database can include A or B, unless otherwise specified or not feasible, the database can include A, or B, or A and B. The second example, if it is expressed that a certain database can include A, B or C, unless otherwise specified or not feasible, the database can include database A, or B, or C, or A and B, or A and C, or B and C, or A and B and C.

[0313] It is to be noted that the above-mentioned embodiments can be implemented in software or hardware, or a combination of software and hardware. If implemented in software, the functions can be stored on or transmitted over as one or more instructions or code on a computer-readable medium. Computer-readable media include both computer storage media and communication media including any medium that facilitates transfer of a computer program from one place to another. A storage media can be any available media that can be accessed by a computer. By way of example, and not limitation, such computer-readable media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer. Also, any connection is properly termed a computer-readable medium. For example, if the software is transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or any other medium that is suitable for transmitting software, then the coaxial cable, fiber optic cable, twisted pair, DSL, or any other medium is included in the definition of medium. Disk and disc, as used herein, includes compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), and Blu-Ray® disc where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0314] In the detailed description above, embodiments have been described with reference to numerous specific details that can vary from implementation to implementation. Certain adaptations and modifications of the described embodiments can occur to those skilled in the art. Other implementations can be apparent to those of ordinary skill in the art upon reading this disclosure. The true scope of the application is defined solely by the appended claims and their equivalents. The steps shown in the figures are only for the purpose of explanation and are not meant to limit the scope of the application to any particular sequence of steps. Therefore, those skilled in the art will recognize the embodiments are not limited to the specific sequences of steps described, but can be practiced with variation of the steps in different orders, as well as with other steps not expressly given in the written description.

[0315] In the drawings and specification, there have been disclosed exemplary embodiments. However, many variations and modifications can be made to these embodiments. Accordingly, although specific terms are employed, they are used in a generic and descriptive sense only and not for purposes of limitation.

Claims

1. A training method for a robot in a game, characterized by, The method comprises: obtaining state information of a robot to be trained and player information of a player within a preset range of the robot; the player information at least includes settlement information of a game battle corresponding to the player, state information of the player and action information of the player; the state information of the player includes self-state information of the player and field information corresponding to the player; using a Spark operator, processing the settlement information into a first data table in the form of metadata, processing the state information of the player and the action information of the player into a second data table in the form of data pairs, and splicing the first data table and the second data table into a result data table according to the field information corresponding to the player, each row in the result data table containing settlement metadata information, state information and action information of the player in a game battle; based on the state information of the robot and the result data table, establishing a state-action model for outputting an action to be executed by the robot through imitative learning.

2. The method of claim 1, wherein, The state information of the robot includes self-state information of the robot and field information corresponding to the robot.

3. The method of claim 1, wherein, The action information includes at least one of the following: a movement action, a skill action and a pickup action.

4. The method of claim 1, wherein, The processing of the state information of the player and the action information of the player into the second data table in the form of data pairs comprises: determining whether the first N action frames of a target skill frame in the action information are all movement frames, and if so, modifying the first N action frames of the target skill frame in the action information into skill frames; and / or, determining whether the last N action frames of the target skill frame are all movement frames, and if so, modifying the last N action frames of the target skill frame in the action information into skill frames; processing the state information of the player and the modified action information of the player into the second data table in the form of data pairs, wherein N is a positive integer.

5. The method of claim 4, wherein, The method further comprises: adjusting corresponding skill-related information in the state information of the player according to the modification of the first N action frames of the target skill frame and / or the modification of the last N action frames of the target skill frame.

6. The method of claim 1, wherein, The processing of the state information of the player and the action information of the player into the second data table in the form of data pairs comprises: detecting whether there are M consecutive movement frames in the action information of the player, wherein M is a positive integer; if so, deleting part of the movement frames in the M movement frames to obtain modified action information; processing the state information of the player and the modified action information into the second data table in the form of data pairs.

7. The method of claim 1, wherein, The processing of the state information of the player and the action information of the player into the second data table in the form of data pairs comprises: screening target state information and target action information meeting a preset condition according to the state information of the player and the action information of the player, the state information including the target state information and the action information including the target action information; processing the target state information and the target action information into the second data table in the form of data pairs.

8. The method of claim 1, wherein, The state-action model for outputting the action to be performed by the robot is established through imitation learning based on the state information of the robot and the result data table, including: Based on the state information of the robot and the result data table, the predicted action of the robot is obtained through the initial state-action model; According to the predicted action, the state information and the result data table, the loss function value is calculated and obtained; When the loss function value reaches a preset threshold, the state-action model for outputting the action to be performed by the robot is obtained.

9. The method of claim 1, wherein, The method further comprises: In response to a game start instruction, the state-action model and the robot are loaded; The state information of the loaded robot is obtained; The obtained state information is input into the state-action model to obtain the target action to be performed by the robot; The robot is controlled to perform the target action.

10. The method of claim 9, wherein, The method further comprises: Action data of the robot is obtained; According to the action data, the action distribution of the robot is obtained; According to the action information of the player, the action distribution of the player is obtained; According to the action distribution of the robot and the action distribution of the player, the KL divergence value is obtained; According to the KL divergence value, it is determined whether the state-action model needs to be trained.

11. The method of claim 9, wherein, The method further comprises: Action data of the robot is obtained; According to the action data, the action distribution of the robot is obtained; According to the action distribution of the robot, the proportion of skill frames is determined; According to the proportion of skill frames, it is determined whether the state-action model needs to be trained.

12. A training system for a robot in a game, characterized by, It comprises: Information acquisition module, information processing module, model construction module; The information acquisition module is used to acquire the state information of the robot to be trained and the player information of the players within the preset range of the robot; the player information at least includes the settlement information of the game battle corresponding to the player, the state information of the player and the action information of the player; the state information of the player includes the self-state information of the player and the field information corresponding to the player; The information processing module is used to process the settlement information into a first data table in the form of metadata by using a Spark operator, process the state information and the action information of the player into a second data table in the form of data pair, and splice the first data table and the second data table into a result data table according to the field information corresponding to the player; each row in the result data table contains the settlement metadata information, the state information and the action information of the player in a game battle; The model construction module is used to establish a state-action model for outputting the action to be performed by the robot through imitation learning based on the state information of the robot and the result data table.

13. An electronic device, comprising: It comprises: A memory and a processor; The memory is used to store one or more computer instructions; The processor is used to execute the one or more computer instructions to realize the method of any one of claims 1-11.

14. A computer-readable storage medium having stored thereon one or more computer instructions, wherein, The instructions are executed by the processor to realize the method of any one of claims 1-11.

Images