Model training method and device, electronic equipment and computer readable storage medium

Abstract

The application discloses a model training method and device, electronic equipment and a computer readable storage medium. The method comprises: obtaining a sample data set, wherein the sample data set comprises a plurality of sample data, each sample data comprises first state information and second state information of a target virtual role in a first game application and first action information of a first virtual role; inputting the first state information and the first action information in the sample data into a preset model to obtain predicted state information output by the preset model; training the preset model to minimize a preset loss function, adjusting parameters of the preset model, and taking the trained preset model as a twin model corresponding to the first game application; and the twin model is used for predicting next state information of the target virtual role according to current state information of the target virtual role and action information executed by the first virtual role. The twin model obtained based on the application can greatly improve the production rate of sample data.

Description

Technical Field

[0001] This application relates to the field of computer technology, specifically to a model training method, apparatus, electronic device, and computer-readable storage medium. Background Technology

[0002] With the rise of the artificial intelligence industry, more and more game companies are using reinforcement learning technology to develop game bots, or non-player characters (NPCs). Developing game bots based on reinforcement learning requires a large amount of game behavior data, i.e., sample data. Currently, the slow production speed of sample data is a major bottleneck limiting the efficiency of game bot development.

[0003] In related technologies, by increasing the number of machines running game environments and opening multiple game environments to run game robots, sample data generated in each game environment can be collected in parallel, thereby increasing the amount of sample data produced per unit time, i.e., the sample data production speed.

[0004] However, since most game environments are currently large in scale, running games requires a lot of system resources. A single device can often only run a limited number of game environments. Related technologies require more machines to effectively improve the efficiency of sample data production, and therefore require huge costs. Summary of the Invention

[0005] This application provides a model training method, apparatus, electronic device, and computer-readable storage medium to improve the production efficiency of sample data at low cost.

[0006] In a first aspect, embodiments of this application provide a model training method, the method comprising:

[0007] A sample dataset is obtained, which includes multiple sample data generated during the operation of the first game application. Each sample data includes first state information and second state information of a target virtual character in the first game application, as well as first action information of the first virtual character. The target virtual character includes the first virtual character and the second virtual character. The second virtual character refers to the virtual character located within the field of view of the first virtual character in the game scene of the first game application. The second state information is the state information of the target virtual character after the first virtual character performs the action corresponding to the first action information.

[0008] The first state information and the first action information from the sample data are input into a preset model to obtain the predicted state information output by the preset model. The predicted state information is the state information of the target virtual character predicted by the preset model based on the action corresponding to the first action information performed by the first virtual character. The preset model is trained with the goal of minimizing a preset loss function, and the parameters of the preset model are adjusted. The trained preset model is used as the twin model corresponding to the first game application. The preset loss function is used to calculate the difference between the predicted state information and the second state information. The twin model is used to predict the next state information of the target virtual character in the first game application based on the current state information of the target virtual character and the first action information performed by the first virtual character.

[0009] Secondly, embodiments of this application provide a model training apparatus, the apparatus comprising:

[0010] The acquisition module is used to acquire a sample dataset, which includes multiple sample data generated during the operation of the first game application. Each sample data includes first state information and second state information of a target virtual character in the first game application, as well as first action information of the first virtual character. The target virtual character includes the first virtual character and the second virtual character. The second virtual character refers to the virtual character located within the field of view of the first virtual character in the game scene of the first game application. The second state information is the state information of the target virtual character after the first virtual character performs the action corresponding to the first action information.

[0011] The processing module is used to input the first state information and the first action information in the sample data into a preset model to obtain the predicted state information output by the preset model. The predicted state information is the state information of the target virtual character predicted by the preset model based on the first virtual character performing the action corresponding to the first action information.

[0012] The control module is used to train the preset model with the goal of minimizing the preset loss function, adjust the parameters of the preset model, and use the trained preset model as the twin model corresponding to the first game application; wherein, the preset loss function is used to calculate the difference between the predicted state information and the second state information, and the twin model is used to predict the next state information of the target virtual character in the first game application based on the current state information of the target virtual character and the first action information performed by the first virtual character.

[0013] Thirdly, embodiments of this application provide an electronic device, the electronic device comprising:

[0014] The memory and the processor are coupled;

[0015] The memory is used to store one or more computer instructions;

[0016] The processor is used to execute one or more computer instructions to implement the model training method described in any of the first aspects above.

[0017] Fourthly, embodiments of this application provide a computer-readable storage medium storing one or more computer instructions, characterized in that the instructions are executed by a processor to implement the model training method described in any of the first aspects above.

[0018] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the model training method described in any of the first aspects above.

[0019] Compared with the prior art, this application has the following advantages:

[0020] The model training method provided in this application obtains a sample dataset, which includes multiple sample data generated during the operation of a first game application. Each sample data includes first and second state information of a target virtual character in the first game application, as well as first action information of the first virtual character. The target virtual character includes both a first virtual character and a second virtual character. The second virtual character refers to a virtual character located within the field of view of the first virtual character in the game scene of the first game application. The first and second state information represent the state information of the target virtual character before and after the first virtual character performs the action corresponding to the first action information, respectively. The first state information and first action information from the sample data are input into a preset model to obtain the predicted state information output by the preset model. The predicted state information is the state information of the target virtual character predicted by the preset model based on the action corresponding to the first action information performed by the first virtual character. The preset model is trained with the objective of minimizing a preset loss function, and the parameters of the preset model are adjusted. The trained preset model is then used as a twin model corresponding to the first game application. The preset loss function is used to calculate the difference between the predicted state information and the second state information, and the twin model is used to predict the next state information of the target virtual character in the first game application based on the current state information of the target virtual character and the first action information performed by the first virtual character.

[0021] Compared to existing technologies, this application's embodiments train a preset model based on sample data obtained from running a first game application. The trained preset model can be considered a twin model corresponding to the first game application. This twin model can approximate the real game environment of the first game application as closely as possible, allowing it to simulate the state output of the real game environment. For example, the current state information of the target virtual character in the real game environment of the first game application is the first state information s1. After the first virtual character performs action a1 in the first game application, the state information of the target virtual character in the real game environment is converted into the second state information s1'. Correspondingly, the first state information s1 and the first action information a1 are input into the twin model corresponding to the first game application. The predicted state information output by the twin model is infinitely close to the second state information s1' of the target virtual character in the real game environment of the first game application. Since the twin model's inference and prediction of the second state information of the target virtual character does not require image rendering like in the real game environment, the twin model trained based on this application's model can greatly improve the sample data production rate. The amount of sample data produced per unit time far exceeds the amount of sample data produced in the real game environment, thereby reducing the sample data collection cost. Furthermore, the twin model corresponding to the first game application is used as the game environment for reinforcement learning. This helps to greatly shorten the development cycle of reinforcement learning-based game robots and solves the problem that the game environment cannot be accelerated during reinforcement learning training. Attached Figure Description

[0022] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0023] Figure 1 A schematic flowchart illustrating a model training method provided in one embodiment of this application;

[0024] Figure 2 A schematic flowchart illustrating a sample data collection method provided in one embodiment of this application;

[0025] Figure 3 This is a schematic diagram of a process for training a target behavior policy network through reinforcement learning, provided in one embodiment of this application.

[0026] Figure 4 This is a schematic diagram of the structure of a model training device provided in one embodiment of this application;

[0027] Figure 5 This is a schematic diagram of the hardware structure of an electronic device provided in one embodiment of this application.

[0028] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0029] To make the objectives, advantages, and features of this application clearer, the application will be described clearly and completely below in conjunction with the accompanying drawings and specific embodiments. In the following description, many specific details are set forth to provide a full understanding of this application. However, the described embodiments are only some, not all, of the embodiments of this application, and all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of this application.

[0030] It should be noted that in the description of this application, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance, or a specific order or sequence. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances. Furthermore, in the description of this application, unless otherwise stated, the term "multiple" refers to two or more. The term "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. The terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or devices.

[0031] To facilitate understanding of the technical solution of this application, the relevant concepts involved in this application will be introduced first.

[0032] The basic idea of ​​reinforcement learning is to learn through trial and error. Trial and error learning consists of two steps: First, in the current state of a specific environment, the agent interacts with that environment and observes the results of the interaction. For example, after the interaction, the state of the environment changes from the current state to the next state, and the reward value is observed. Then, based on the reward value obtained after the interaction, the agent's corresponding behavioral policy is optimized. The goal of optimizing the behavioral policy is to maximize the reward, that is, to obtain as much reward as possible. Therefore, reinforcement learning is a goal-oriented learning method that learns through interaction with the environment.

[0033] An intelligent agent is a computer-controlled artificial intelligence that interacts within a specific environment (such as a game application). Artificial intelligence utilizes digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceiving the environment, acquiring knowledge, and using that knowledge to achieve optimal results. In other words, the non-player-controlled virtual character in the first game application of this application can act as an intelligent agent, simulating a real player controlling a virtual character to play the game, and cooperating or competing with the real player-controlled virtual character.

[0034] The status information of a virtual character describes the current state of a virtual character in a first game application. For example, the status information of a virtual character may include, but is not limited to, information such as the virtual character's health, equipment, location, and party affiliation. This is merely an example, and this application does not specifically limit the content included in the status information.

[0035] Actions refer to the actions performed by an intelligent agent during its interaction with a specific environment. For example, in this embodiment, a first virtual character can be considered as an intelligent agent, and the first virtual character performs corresponding actions within the game scene of a first game application.

[0036] Rewards are numerical values ​​fed back to the agent from the environment. The magnitude of the reward value reflects the quality of the agent's actions.

[0037] The prior art involved in this application and the problems existing in the prior art will be described below.

[0038] Game bots (NPCs, non-player-controlled virtual characters) play a crucial role in a game. They can act as teammates to accompany players in completing game tasks or as opponents to combat players, enhancing the gaming experience. Early game bots involved game developers setting rules and executing actions, a time-consuming and labor-intensive process. With the rise of artificial intelligence, to reduce the human resource investment in game bot development, more and more game companies are utilizing reinforcement learning techniques. This involves setting reward values ​​for different outcomes for the bot, continuously inputting the current environmental state, and then the bot performs corresponding actions. Based on the changes in the environmental state caused by the actions, a corresponding reward value is given to the bot, allowing it to determine the quality of its actions based on the reward value. Through this continuous trial and error process, the game bot constantly explores the optimal action for the current state.

[0039] Reinforcement learning is an online learning method that learns online during the interaction between an agent and its environment. Therefore, reinforcement learning requires optimizing the agent's behavioral strategies based on sample data generated during the operation of a game application. Currently, most game processes cannot be accelerated; a single game takes anywhere from 5 minutes to an hour to run. The training process for reinforcement learning requires a large amount of game sample data. Because the generation rate of game sample data is too slow, training game robots based on reinforcement learning can often take several weeks or even months. If there are any errors in the robot's training code or unexpected behaviors during training, it may be necessary to modify the training logic and restart training, which will affect the game development schedule to some extent. Therefore, improving the training speed of game robots is a major bottleneck in the current training and development of game robots using reinforcement learning technology.

[0040] To address the issue of game environments not being able to accelerate during reinforcement learning training, existing technologies offer various solutions. The following is a brief explanation of each solution and its respective problems.

[0041] Option 1: Increase the number of machines running the game: Specifically, multiple machines can be used to run multiple game environments simultaneously, collecting sample data generated in each environment in parallel. This increases the rate of sample data generation per unit time, thereby increasing the amount of training sample data and improving the learning efficiency of reinforcement learning algorithms to some extent. While this option can improve the speed of collecting reinforcement learning sample data, most games currently consume significant resources, and a single device can often only run one game environment. Therefore, this option requires purchasing more machines, undoubtedly increasing the training cost of the game robot.

[0042] Option 2: Reusing data multiple times: Specifically, a data cache pool is established to record previously encountered sample data. This way, whenever the environment generation rate is too slow to meet training needs, sample data can be read from the cache pool to train the game robot. In other words, each training process no longer relies solely on the sample data obtained from the current interaction with the environment, but also draws on some historical sample data to optimize the reinforcement learning algorithm. While this option can increase the amount of sample data during game robot training and improve the learning efficiency of the reinforcement learning algorithm to some extent, the improvement in learning efficiency is very limited because the samples in the cache pool are all already learned samples and are not generated by the latest reinforcement strategy.

[0043] Option 3 uses a previously learned model as a base model, assuming that this base model already possesses certain experience. Learning new tasks / game robots based on this base model can effectively improve training efficiency; this method is called transfer learning. While this approach can utilize existing knowledge to guide the training direction of reinforcement strategies and improve training efficiency, it relies on the existing knowledge having some relevance to the current training objective (i.e., the downstream task objective). Otherwise, it may have a counterproductive effect. Games, in order to enhance playability and attract more players, set a variety of different game objectives and characters, making it difficult to find commonalities between different objectives. Therefore, the effectiveness of transfer learning in game robot training is very limited.

[0044] To address at least one of the aforementioned problems, this application provides a model training method, a corresponding model training apparatus, an electronic device capable of implementing the model training method, and a computer-readable storage medium. The following embodiments provide a detailed description of the method, apparatus, electronic device, and computer-readable storage medium.

[0045] To make the objectives and technical solutions of this application clearer and more intuitive, the methods provided by the embodiments of this application will be described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely for explaining this application and are not intended to limit this application. It is understood that the following embodiments may exist independently, and the embodiments and features described below may be combined with each other where there is no conflict between the various embodiments provided in this application. For the same or similar content, it will not be repeated in different embodiments. Furthermore, the timing of the steps in the following method embodiments is merely an example and not a strict limitation. In some cases, the steps shown or described may be performed in a different order.

[0046] This application provides a model training method, apparatus, electronic device, and computer-readable storage medium. Specifically, the model training method of one embodiment of this application can be executed by a computer device, which can be a terminal or a server. The terminal can be a smartphone, tablet computer, laptop computer, touch screen, game console, or other terminal device, and may also include a client. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, and big data and artificial intelligence platforms.

[0047] Below, in conjunction with Figure 1The present application will describe a model training method provided in one embodiment. Figure 1 This is a flowchart illustrating a model training method provided in one embodiment of this application.

[0048] like Figure 1 As shown, the model training method includes steps S10-S30:

[0049] S10. Obtain a sample dataset. The sample dataset includes multiple sample data generated during the operation of the first game application. Each sample data includes the first state information and second state information of the target virtual character in the first game application, as well as the first action information of the first virtual character. The target virtual character includes both the first virtual character and the second virtual character. The second virtual character refers to the virtual character located within the field of view of the first virtual character in the game scene of the first game application. The second state information is the state information of the target virtual character after the first virtual character performs the action corresponding to the first action information.

[0050] The aforementioned first game application includes, but is not limited to, any one of the following: first-person game applications, third-person game applications, single-player game applications, and multiplayer online battle arena (MOBA) game applications. The types of games mentioned above may include, but are not limited to, at least one of the following: two-dimensional (2D) game applications, three-dimensional (3D) game applications, virtual reality (VR) game applications, augmented reality (AR) game applications, and mixed reality (MR) game applications. The above is merely an example, and this application embodiment does not impose any limitations on it.

[0051] As mentioned above, the first virtual character is a non-player-controlled virtual character (i.e., an NPC), but it can also be a player-controlled virtual character. This application embodiment does not limit this in any way.

[0052] The sample data described above was generated during the operation of the first game application in a real game scenario. Specifically, it was obtained by controlling a first virtual character to interact with the game environment while running the first game application on a computer device. Each sample data entry includes the first and second state information of the target virtual character within the first game application, as well as the first action information of the first virtual character. The first state information represents the state of the target virtual character before the first virtual character performs the action corresponding to the first action information. The second state information represents the state of the target virtual character after the first virtual character performs the action corresponding to the first action information. The target virtual character includes both the first and second virtual characters; the second virtual character refers to the virtual character located within the first virtual character's field of vision within the game scenario of the first game application. It is understandable that, typically, real players make behavioral decisions based on the state of their controlled virtual character and the state information of other virtual characters they can observe on the game interface. These other virtual characters observed by the player include teammates, enemy virtual characters, and other non-player-controlled virtual characters. Therefore, setting the target virtual character as a first virtual character and a second virtual character within the first virtual character's field of vision is to allow the first virtual character (i.e., a virtual object not controlled by the player) to simulate a real player, so as to allow the first virtual character to think like a real player as much as possible. That is, to allow the first virtual character to make behavioral decisions in the game scene based on its own state information and the state information of the virtual characters within its field of vision.

[0053] It should be noted that the first state information of the target virtual character in the sample data is the state information of the target virtual character before the first virtual character performs the action corresponding to the first action information in the game environment. After the first virtual character performs the action corresponding to the first action information, the state information of the target virtual character changes, resulting in the next state information of the target virtual character, i.e., the second state information.

[0054] For example, the game scene of the first game application includes virtual characters 1, 2, 3, 4, 5, and 6. Virtual characters 1-3 belong to the first faction, and virtual characters 4-6 belong to the second faction. The first and second factions are hostile to each other. In this first game application, it is assumed that virtual character 1 is a non-player-controlled virtual character, and virtual characters 4 and 6 are observable by virtual character 1 in the game scene. In this case, the first virtual character is virtual character 1, and the target virtual characters include virtual characters 1, 4, and 6. This is merely an example and does not constitute any limitation in this embodiment.

[0055] The first state information of the aforementioned target virtual characters, taking virtual characters 1, 4, and 6 as examples, provides an illustrative explanation of the content included in the first state information. The first state information of the target virtual characters includes, but is not limited to, the first state information of each of virtual characters 1, 4, and 6. The state information of each virtual character includes, but is not limited to, health status, location information, and detailed skill information (such as cooldown status, damage value, etc.). This is merely an example; the embodiments of this application do not impose any limitations on the content included in the state information, and it can be set according to the actual situation.

[0056] S20. Input the first state information and the first action information in the sample data into the preset model to obtain the predicted state information output by the preset model. The predicted state information is the state information of the target virtual character predicted by the preset model based on the action corresponding to the first action information of the first virtual character.

[0057] In one optional implementation, the network structure of the preset model includes, but is not limited to, any one of the following: feedforward neural network, convolutional neural network, recurrent neural network, and Transform network.

[0058] The aforementioned preset model is used to predict the state information of the target virtual character after it performs the action corresponding to the first action information, based on the first state information and the first action information of the target virtual character in the sample data. In other words, taking any sample data as an example, the sample data includes the first and second state information of the target virtual character in the first game application, as well as the first action information of the first virtual character. The first state information and the first action information of the target virtual character are input into the preset model. Based on the first state information and the first action information performed by the first virtual character, and assuming the current state information of the target virtual character is the first state information, the prediction model predicts the state information of the target virtual character after it completes the action corresponding to the first action information.

[0059] Then, the first state information and the first action information of the target virtual character in the sample data are input into the preset model to obtain the state information of the target virtual character after the first virtual character performs the action corresponding to the first action information.

[0060] For example, the sample dataset includes n sample data points, and the sample dataset is {(s1,s1',a1),(s2=s1',s2',a2),(s3=s2',s3',a3),...,(s n =s n-1 ',sn ',a n The following example illustrates the elements in the sample data, using (s1, s1', a1) as an example. In (s1, s1', a1), s1 represents the state information of the target virtual character in the first game application at time t1, a1 represents the action performed by the first virtual character in the first game application at time t1, and s1' represents the second state information of the target virtual character at time t2 after the first virtual character performs action a1 in the first game application. During training, the preset model is trained sequentially based on each sample data. Specifically, using (s1, s1', a1) in the sample dataset as an example, the preset model is trained by inputting s1 and a1 into the preset model. The preset model outputs its predicted state information of the target virtual character at time t2, thus obtaining the predicted state information. In other words, the prediction model actually predicts the second state information of the target virtual character based on the first state information and the first action performed by the first virtual character in the sample data.

[0061] S30. Train the preset model with the goal of minimizing the preset loss function, adjust the parameters of the preset model, and use the trained preset model as the twin model corresponding to the first game application. The preset loss function is used to calculate the difference between the predicted state information and the second state information. The twin model is used to predict the next state information of the target virtual character in the first game application based on the first state information of the target virtual character and the first action information performed by the first virtual character.

[0062] As described above, the preset loss function is used to calculate the difference between the second state information of the target virtual character in the first game application predicted by the preset model and the second state information in the sample data. The second state information is the state information of the target virtual character after the first virtual character performs the action corresponding to the first action information, as predicted by the preset model.

[0063] In this embodiment, the parameters of the preset model are adjusted based on the difference (i.e., the loss value) calculated by the preset loss function, so that the loss value calculated by the trained preset model is infinitely close to zero. That is, the preset model can make increasingly accurate predictions about the state information of the target virtual character after the first virtual character performs the action corresponding to the first action information, based on the first state information and the first action information in the sample data. After training, the trained preset model is the twin model corresponding to the first game application. Taking any sample data as an example, it can simulate the first game application based on the first state information and the first action information in the same sample data, and output the same or similar predicted state information as the second state information of the target virtual character in the sample data.

[0064] For example, when training a preset model using any sample data (s1, s1', a1) from the sample dataset, s1 and a1 are input into the preset model. The preset model outputs its predicted state information w(s1, a1) of the target virtual character in the first game application at time t2, which is the second state information. The loss value loss = |w(s1, a1) - s1'| is calculated according to the preset loss function. The parameters of the preset model are then adjusted based on the loss value loss.

[0065] In one alternative implementation, the preset model is obtained after training, provided that all sample data in the sample dataset has been used to train the preset model.

[0066] In another optional implementation, if the number of sample data trained on the above-mentioned preset model exceeds a preset sample number threshold, a preset model after training is obtained.

[0067] In another optional implementation, if the number of training iterations in which the loss value is less than a preset loss threshold exceeds a preset number threshold during the training process of the above-mentioned preset model, the preset model after training is obtained.

[0068] In the model training method provided in this application embodiment, a sample dataset is obtained. The sample dataset includes multiple sample data generated during the running of a first game application. Each sample data includes first state information and second state information of a target virtual character in the first game application, as well as first action information of the first virtual character. The target virtual character includes a first virtual character and a second virtual character. The second virtual character refers to a virtual character located within the field of view of the first virtual character in the game scene of the first game application. The first state information and second state information are the state information of the target virtual character before and after the first virtual character performs the action corresponding to the first action information, respectively. The first state information and first action information from the sample data are input into a preset model to obtain the predicted state information output by the preset model. The predicted state information is the state information of the target virtual character predicted by the preset model based on the action corresponding to the first action information performed by the first virtual character. The preset model is trained with the objective of minimizing a preset loss function, and the parameters of the preset model are adjusted. The trained preset model is used as a twin model corresponding to the first game application. The preset loss function is used to calculate the difference between the predicted state information and the second state information. The twin model is used to predict the next state information of the target virtual character in the first game application based on the current state information of the target virtual character and the first action information performed by the first virtual character.

[0069] Compared to existing technologies, this application's embodiments train a preset model based on sample data obtained from running a first game application. The trained preset model can be considered a twin model corresponding to the first game application. This twin model can approximate the real game environment of the first game application as closely as possible, allowing it to simulate the state output of the real game environment. For example, the current state information of the target virtual character in the real game environment of the first game application is the first state information s1. After the first virtual character performs action a1 in the first game application, the state information of the target virtual character in the real game environment is converted into the second state information s1'. Correspondingly, the first state information s1 and the first action information a1 are input into the twin model corresponding to the first game application. The predicted state information output by the twin model is infinitely close to the second state information s1' of the target virtual character in the real game environment of the first game application. Since the twin model's inference and prediction of the second state information of the target virtual character does not require image rendering like in the real game environment, the twin model trained based on this application's model can greatly improve the sample data production rate. The amount of sample data produced per unit time far exceeds the amount of sample data produced in the real game environment, thereby reducing the sample data collection cost. Furthermore, the twin model corresponding to the first game application is used as the game environment for reinforcement learning. This helps to greatly shorten the development cycle of reinforcement learning-based game robots and solves the problem that the game environment cannot be accelerated during reinforcement learning training.

[0070] Based on the above embodiments, the model training method provided in this application will be further described below.

[0071] In an optional implementation, step S10 can also be implemented as step S101:

[0072] S101. Repeat the following first step until the number of sample data in the sample dataset reaches the preset threshold.

[0073] The first step includes steps S1011-S1014:

[0074] S1011. Input the current state information and historical state information of the target virtual character in the first game application into the preset behavior policy network to obtain the second action information output by the preset behavior policy network. The preset behavior policy network is used to predict the next action information to be performed by the first virtual character based on the current state information and historical state information of the target virtual character.

[0075] S1012. In the first game application, control the first virtual character to perform the action corresponding to the second action information.

[0076] S1013. After the first virtual character completes the action corresponding to the second action information, obtain the third state information of the target virtual character in the first game application.

[0077] S1014. Using the current state information of the target virtual character as the first state information, the second action information as the first action information, and the third state information as the second state information, construct a sample data and save the sample data to the sample dataset.

[0078] As described above, a pre-defined action policy network (PPR) is a network model in reinforcement learning used to learn and determine the next action an agent should take in a specific environment. The PPR can be viewed as a mapping function that maps the current state information of a specific environment to the corresponding action probability distribution. Inputting the current state information and / or historical state information into the PPR causes it to output a specific action, or output the probability value of each possible action, and determine the action with the highest probability value as the next action to be executed.

[0079] In this embodiment, when the first virtual character is a non-player-controlled virtual character, the first game application controls the first virtual character to perform interactive operations in the real game environment of the first game application. Since the first virtual character (i.e., the game robot) is non-player-controlled and lacks subjective consciousness, it cannot autonomously make behavioral decisions based on the current and historical state information of the target virtual character in the first game application. Therefore, in this embodiment, a preset behavior strategy network is introduced to complete the behavior decision for the first virtual character. This preset behavior strategy network is used to output the action information corresponding to the next action to be performed by the first virtual character, i.e., the second action information, based on the current and historical state information of the target virtual character in the first game application. Subsequently, in the first game application, the game system controls the first virtual character to perform the action corresponding to the second action information, and the state information of the target virtual character changes accordingly, resulting in the state information of the target virtual character after the first virtual character has completed the action corresponding to the second action information, i.e., the third state information of the target virtual character.

[0080] In one optional implementation, the preset behavior policy network can be a neural network model, which can be any of the following: feedforward neural network, convolutional neural network, recurrent neural network, or Transform network.

[0081] For example, during the operation of the first game application, the current state information s1 and the historical state information of the target virtual character are input into the preset behavior strategy network. The preset behavior strategy network outputs multiple different actions and the probability of each action. The output information of the preset behavior strategy network can be referred to Table 1.

[0082] Table 1

[0083] action probability value Action 1 20％ Action 2 25％ Action 3 45％ Action 4 10％

[0084] As shown in Table 1, the preset behavior policy network outputs four actions and their respective probability values. For example, the action with the highest probability value is determined as action a1 corresponding to the second action information. As shown in Table 1, action 3 has the highest probability value, so action 3 is determined as action a1. Then, controlling the first virtual character to execute a1 = action 3 in the first game application (i.e., the real game environment) changes the target virtual character's state information, subsequently yielding the target virtual character's next state information s1', i.e., the virtual character's third state information. Based on the second action information a1 = action 3, the target virtual character's current state information s1, and the next state information s1', the constructed sample data is (s1, s1', a1). That is, by using the target virtual character's current state information as the first state data s1 in the sample data, the second action information a1 as the first action information in the sample data, and the target virtual character's third state information s1' as the second state information in the sample data, a sample data (s1, s1', a1) is constructed.

[0085] It is understandable that after the first virtual character completes the action corresponding to the second action information, the current state information of the target virtual character is s2 = s1'. The preset behavior policy network outputs the second action information a2. After the first virtual character completes a2, the state information of the target virtual character changes accordingly. Subsequently, the next state information s2' of the target virtual character in the first game application can be obtained, which is the third state information of the virtual character. Based on the current state information s2 = s1' of the target virtual character before the first virtual character executes the action corresponding to the second action information a2, the second action information a2, and the third state information s2' of the target virtual character, the sample data constructed is (s2 = s1', s2', a2). Similarly, by repeating the above steps S1011-S1014, other sample data such as (s3 = s2', s3', a3), (s4 = s3', s4', a4), ..., (s n =s n-1 ',s n ',a n ).

[0086] In this embodiment, during the process of acquiring sample data based on the real game environment of the first game application, a preset behavior strategy network is used to help the first virtual character make intelligent decisions on the next action to be performed based on the current state information and historical state information of the target virtual character. This enables the first virtual character to simulate a real player in the game, which improves the realism of the game behavior of the game robot (i.e., the first virtual character). As a result, the quality of the sample data obtained is higher, and the twin model trained based on the high-quality sample data can more realistically simulate the state output of the first game application.

[0087] In an optional implementation, the first step further includes the following steps S1015-S1017:

[0088] S1015. Input the third state information into the first network and the second network to obtain the first predicted value output by the first network and the second predicted value output by the second network. The network structure of the first network and the network structure of the second network are the same.

[0089] S1016. Adjust the parameters of the second network based on the difference between the first predicted value and the second predicted value, and determine the difference as the reward information corresponding to the second action information.

[0090] S1017. Construct a training dataset based on the current state information, third state information, second action information, and reward information of the target virtual character, and perform reinforcement learning training on the preset behavior strategy network based on the training dataset.

[0091] As mentioned above, the first network and the second network have the same network structure, but the initial parameter values ​​of the first network and the second network are different.

[0092] In one alternative implementation, the network structures of the first network and the second network include any of the following: feedforward neural network, convolutional neural network, recurrent neural network, support vector machine, decision tree, random forest, etc. These are merely examples and the embodiments of this application do not impose any limitations on them.

[0093] In this embodiment, the parameters of the first network are frozen to keep them fixed. The first network is essentially a random network whose output has no fixed pattern or rule. The third state information of the target virtual character is input into the first network and the second network to obtain the first predicted value output by the first network and the second predicted value output by the second network. The explanation of the third state information of the target virtual character can be found in the explanation of the third state information in step S1013 above, and the meaning is the same, so it will not be repeated here.

[0094] For example, taking sample data (s1, s1', a1) as an example, in the sample data, the state information of the target virtual character before and after the first virtual character performs action a1 is s1 and the third state information is s1', respectively. When the first network is network f and the second network is network g, the third state information s1' of the target virtual character is input into network f, and the third state information and s1' of the target virtual character are input into network g. Network f outputs the first predicted value f(s1'), and network g outputs the first predicted value g(s1').

[0095] In this embodiment, the parameters of the second network are adjusted based on the difference between the first predicted value f(s1') output by the first network and the second predicted value g(s1') output by the second network (e.g., |f(s1')-g(s1')|). That is, the difference between the first and second predicted values ​​is used as the loss value to adjust the parameters of the second network, enabling the second network to learn the output of the first network. Therefore, whenever the third state information (e.g., s1') of the target virtual character is input into the first and second networks after the first virtual character performs a corresponding action, it indicates that both the first and second networks have seen the third state information s1'. At this point, the second network has completed parameter adjustment based on the difference between the first and second predicted values ​​output by the first and second networks, resulting in a parameter-adjusted second network. When the third state information s1' is input into the parameter-adjusted second network, the difference between the second predicted value output by the second network and the first predicted value output by the first network is very small (or close to zero).

[0096] Understandably, the difference between the first and second predicted values ​​(e.g., |f(s1')-g(s1')|) reflects whether the third state information s1' has been explored by the first virtual character (or, in other words, the novelty of the third state information s1' to the first virtual character). That is, the larger the difference, the greater the probability that the state information s1' has not been explored by the first virtual character (i.e., the higher the novelty of the third state information s1' to the first virtual character). Conversely, the larger the difference, the greater the probability that the state information s1' has been explored by the first virtual character (i.e., the lower the novelty of the state information s1' to the first virtual character). The difference between the first predicted value output by the first network and the second predicted value output by the second network is determined as the reward information r (i.e., reward) corresponding to the second action information.

[0097] In this embodiment, the current state information, third state information, second action information, and reward information of the target virtual character are then used to construct training data. Reinforcement learning is then performed on the preset behavior policy network based on this training data. During reinforcement learning, the quality of the action corresponding to the second action information is judged based on the magnitude of the reward information, thereby optimizing the preset behavior policy network. This ensures that the first virtual character continuously explores previously unseen states (i.e., exhibits strong curiosity) until it covers as much of the state space as possible, resulting in a more diverse sample dataset with varied state information.

[0098] It's important to note that during reinforcement learning training, the pre-defined behavior policy network continuously learns and optimizes by interacting with the real-world game environment of the first game application. For example, using Monte Carlo sampling or temporal difference learning methods, the parameters of the pre-defined behavior policy network are updated by collecting a series of state information, action information, and corresponding reward information, enabling the first virtual character to better adapt to the environment and produce superior actions. In short, optimizing the pre-defined behavior policy network using training data allows the first virtual character to continuously explore previously unseen state information, thus enriching the sample dataset with more diverse state information.

[0099] Below, in conjunction with Figure 2 and one A specific example will be provided to further illustrate steps S1011-S1017 above. Figure 2 This is a flowchart illustrating a sample data collection method provided in one embodiment of this application.

[0100] like Figure 2 As shown, steps S201-S209 are included:

[0101] S201. Initialize the first network and the second network with different parameters, wherein the network structures of the first network and the second network are the same.

[0102] S202. Obtain a preset behavior policy network, which is used to output actions to explore more states.

[0103] S203. Input the current state information of the target virtual character into the preset behavior strategy network. The preset behavior strategy network outputs the second action information and controls the first virtual character to perform the action corresponding to the second action information. After the first virtual character finishes performing the action corresponding to the second action information, obtain the third state information of the target virtual character.

[0104] The target virtual characters include the first virtual character and the virtual characters within the field of view of the first virtual character in the first game application.

[0105] S204. Input the third state information of the target virtual character into the first network and the second network. Use the difference between the output values ​​of the first network and the output values ​​of the second network as the loss value to optimize the parameters of the second network. At the same time, the loss value will be used as the reward information for reinforcement learning.

[0106] S205. Based on the current state information, third state information, second action information, and reward information of the target virtual character in steps S203-S204 above, construct a training dataset.

[0107] S206. Based on the above training data, perform reinforcement learning training on the preset behavior policy network.

[0108] S207. Using the current state information of the target virtual character as the first state information, the second action information as the first action information, and the third state information as the second state information, construct a sample data and save the sample data to the sample dataset.

[0109] S208. Determine whether the number of sample data in the sample dataset has reached the preset threshold. If not, proceed to steps S203-S208; if yes, proceed to step S209.

[0110] S209, Output sample dataset.

[0111] The following section details the implementation method for obtaining sample data for training the twin model corresponding to the first game application by running the first game application.

[0112] In an optional implementation, step S10 can also be implemented as step S102:

[0113] S102. Repeat the second step below until the current round of the first game application ends, and obtain a sample dataset including multiple sample data.

[0114] The second step includes S1021-S1025:

[0115] S1021. Obtain the current status information of the target virtual character in the first game application.

[0116] S1022. Input the current state information of the target virtual character into the target behavior policy network to obtain the third action information output by the target behavior policy network.

[0117] Among them, the target behavior strategy network is a network designed to make action decisions in order to achieve a preset game objective.

[0118] S1023. In the first game application, control the first virtual character to perform the action corresponding to the third action information.

[0119] S1024. After the first virtual character completes the action corresponding to the third action information, obtain the fourth state information of the target virtual character in the first game application.

[0120] S1025. Take the current state information of the target virtual character as the first state information, the third action information as the first action information, and the fourth state information as the second state information, construct a sample data and save the sample data to the sample dataset.

[0121] The aforementioned target behavior policy network is a network model guided by a preset game objective, used to learn and decide which actions a first virtual character should perform in the real game environment corresponding to the first game application. It is important to emphasize that the objectives of the preset behavior policy network and the target behavior policy network are different. The preset behavior policy network aims to help the first virtual character explore more diverse state information of the target virtual character, thus making action decisions for the first virtual character. The target behavior policy network, on the other hand, aims to help the first virtual character achieve a certain preset game objective (such as winning the current game round). This can be understood as follows: in practical applications, reinforcement learning is used to train and optimize the behavior policy network so that the game robot can complete training tasks or downstream tasks; the trained behavior policy network is the aforementioned target behavior policy network.

[0122] In this embodiment, at the start of the first game application, the current state information of the target virtual character is obtained and input into the target behavior policy network to obtain the third action information output by the target behavior policy network. In the game scene of the first game application, the first virtual character is controlled to perform the action corresponding to the third action information, and the state information of the target virtual character changes accordingly, resulting in the changed state information of the target virtual character, i.e., the fourth state information of the target virtual character. Using the current state information of the target virtual character as the first state information, the third action information as the first action information, and the fourth state information as the second state information, a sample data point is constructed and saved to a sample dataset. The above steps are repeated until the current round of the first game application ends, resulting in a sample dataset containing multiple sample data points. In this way, the sample data obtained during the training of the target behavior policy network based on reinforcement learning in the real game environment of the first game application with the training task or downstream task as the objective can be used to train the corresponding Siamese model (or preset model) of the first game application. This can further improve the similarity between the Siamese model and the first game application, so that the Siamese model can more accurately simulate the real game environment of the first game application and output its state.

[0123] In an optional implementation, the model training method provided in this application embodiment may further include the following step S40:

[0124] S40. Repeat the following third step until the current round of the first game application ends, and obtain multiple target sample data. The target sample data is used to perform reinforcement learning training on the target behavior policy network. The target behavior policy network is a network designed to achieve the preset game goal and make action decisions.

[0125] The third step includes S4011-S4015:

[0126] S4011. Obtain the current state information of the target virtual character in the twin model.

[0127] S4012. Input the current state information into the target behavior policy network to obtain the fourth action information output by the target behavior policy network.

[0128] S4013. Input the current state information and the fourth action information into the twin model to obtain the fifth state information output by the twin model. The fifth state information is the state information of the target virtual character predicted by the twin model based on the action corresponding to the fourth action information performed by the first virtual character.

[0129] S4014. Determine the target reward value based on the preset reward function, the current state information of the target virtual character, and the fifth state information.

[0130] S4015. Based on the current state information, fifth state information, fourth action information of the target virtual character, and the target reward value, construct a target sample data.

[0131] The target behavior policy network in step S4015 has the same meaning as the target behavior policy network in step S1022. Please refer to the above explanation of the target behavior policy network. It will not be repeated here.

[0132] In this embodiment of the application, step S40 is used to generate sample data for subsequent training and optimization of the target behavior policy network based on the twin model after obtaining the twin model corresponding to the first game application.

[0133] In this embodiment, the current state information of the target virtual character is obtained from the twin model, and this information is input into the target behavior policy network. The target behavior policy network then outputs information for the fourth action. The current state information and the fourth action information are input together into the twin model corresponding to the first game application to obtain the fifth state information output by the twin model. The fifth state information is the state information of the target virtual character predicted by the twin model after the first virtual character performs the action corresponding to the fourth action information. A target reward value is determined based on a preset reward function, the current state information of the target virtual character, and the fifth state information. A target sample data set is constructed based on the current state information, the fifth state information, the fourth action information, and the target reward value.

[0134] For example, given that the current state information of the target virtual character is s5, the fifth state information is s5', and the fourth action information is a5, the target reward value r5 = R(s5, s5') is obtained based on the preset reward function R. Based on the current state information s5, the first state information s5', the fourth action information a5, and the target reward value r5 = R(s5, s5'), a target sample data is constructed as (s5, s5', a5, r5) = (s5, s5', a5, R(s5, s5')).

[0135] In this embodiment, generating target sample data based on the trained twin model can significantly improve the production rate of target sample data. The amount of target sample data produced per unit time far exceeds the production rate of target sample data in a real game environment, thereby reducing the acquisition cost of target sample data. Furthermore, using the twin model corresponding to the first game application as the game environment for reinforcement learning helps to greatly shorten the development cycle of reinforcement learning-based game robots and solves the problem that the game environment cannot be accelerated during reinforcement learning training.

[0136] In one optional implementation, a preset reward function is used to calculate the difference between the current state information and the fifth state information of the target virtual character, and the difference is determined as the target reward value.

[0137] For example, assume that the status information includes the virtual character's health and location information, and the target virtual characters include virtual character 1 (i.e., the first virtual character), virtual character 3, and virtual character 5. Then, assume that the current status information and fifth status information of the target virtual characters are as shown in Table 2.

[0138] Table 2

[0139]

[0140] As shown in Table 2, the current status information of the target virtual characters is as follows: Virtual character 1 has 50 HP and is located at (x11, y11); Virtual character 3 has 60 HP and is located at (x13, y13); Virtual character 5 has 40 HP and is located at (x15, y15). The fifth status information of the target virtual characters is as follows: Virtual character 1 has 30 HP and is located at (x21, y21); Virtual character 3 has 20 HP and is located at (x23, y23); Virtual character 5 has 33 HP and is located at (x25, y25).

[0141] Therefore, the difference between the current state information and the fifth state information of the target virtual character, calculated by combining the current state information and the fifth state information of the target virtual character shown in Table 2, can be referred to Table 3.

[0142] Table 3

[0143]

[0144] It should be noted that when the status information includes location information, the difference between two location information can be the distance between the two locations, such as Euclidean distance, Manhattan distance, etc. This is just an example, and the embodiments of this application do not impose any restrictions on it.

[0145] Below, in conjunction with Figure 3 The process of obtaining a target behavior policy network by performing reinforcement learning on the twin model corresponding to the first game application obtained according to the model training method provided in the embodiments of this application is described. Figure 3 This is a schematic diagram illustrating the process of reinforcement learning training for a target behavior policy network, as provided in one embodiment of this application.

[0146] First, the process of reinforcement learning of the target behavior policy network based on the real game environment, i.e., in the first game application, is explained.

[0147] S301. Obtain the initial target behavior policy network P.

[0148] S302, Obtain the current state information s1 of the target virtual character in the first game application.

[0149] S303. Input the current state information s1 of the target virtual character into the target behavior policy network P, and the target behavior policy network P outputs action information a1 = P(s1).

[0150] S304. Control the first virtual character to perform the action corresponding to the action information a1 in the first game application, and obtain the next state information s1' and reward value r1 of the target virtual character in the first game application.

[0151] S305. Optimize the target behavior policy network P using (s1,a1,s1',r1).

[0152] S306. Let s = s1'. Determine whether the target behavior policy network P converges. If yes, proceed to step S307; otherwise, proceed to steps S302-S306.

[0153] S307. Obtain the target behavior policy network P after training.

[0154] The following section explains the process of reinforcement learning of the target behavior policy network based on the twin environment corresponding to the first game application, i.e., the twin model corresponding to the first game application.

[0155] S301. Obtain the initial target behavior policy network P.

[0156] S308. Obtain the current state information s2 of the target virtual character in the first game application.

[0157] S309. Input the current state information s2 of the target virtual character into the target behavior policy network P, and the target behavior policy network P outputs action information a2 = P(s2).

[0158] S310. Input the current state information s2 and the first action information a2 of the target virtual character into the twin model W corresponding to the first game application to obtain the next state information s2' = W(s2,a2) of the target virtual character and the reward value r2 output by the twin model.

[0159] S311. Optimize the target behavior policy network P using (s2,a2,s2',r2).

[0160] S312. Let s = s2'. Determine whether the target behavior policy network P converges. If yes, proceed to step S307; otherwise, proceed to steps S308-S312.

[0161] S307. Obtain the target behavior policy network P after training.

[0162] The model training apparatus provided in this application will be described below. The model training apparatus described below can be referred to in correspondence with the model training method described above.

[0163] Figure 4 This is a schematic diagram of the structure of a model training device provided in one embodiment of this application. Figure 4 As shown, the model training device includes: an acquisition module 401, a processing module 402, and a control module 403.

[0164] The acquisition module is used to acquire a sample dataset, which includes multiple sample data generated during the operation of the first game application. Each sample data includes first state information and second state information of a target virtual character in the first game application, as well as first action information of the first virtual character. The target virtual character includes the first virtual character and the second virtual character. The second virtual character refers to the virtual character located within the field of view of the first virtual character in the game scene of the first game application. The second state information is the state information of the target virtual character after the first virtual character performs the action corresponding to the first action information.

[0165] The processing module is used to input the first state information and the first action information in the sample data into a preset model to obtain the predicted state information output by the preset model. The predicted state information is the state information of the target virtual character predicted by the preset model based on the first virtual character performing the action corresponding to the first action information.

[0166] The control module is used to train the preset model with the goal of minimizing the preset loss function, adjust the parameters of the preset model, and use the trained preset model as a twin model corresponding to the first game application; wherein, the preset loss function is used to calculate the difference between the predicted state information and the second state information; the twin model is used to predict the next state information of the target virtual character in the first game application based on the current state information of the target virtual character and the first action information performed by the first virtual character.

[0167] Optionally, the acquisition module is specifically used for:

[0168] Repeat the first step below until the number of sample data in the sample dataset reaches the preset threshold.

[0169] The first step includes:

[0170] The current state information and historical state information of the target virtual character in the first game application are input into a preset behavior strategy network to obtain the second action information output by the preset behavior strategy network. The preset behavior strategy network is used to predict the next action information to be performed by the first virtual character based on the current state information and historical state information of the target virtual character.

[0171] In the first game application, the first virtual character is controlled to perform the action corresponding to the second action information;

[0172] After the first virtual character completes the action corresponding to the second action information, the third state information of the target virtual character in the first game application is obtained;

[0173] Using the current state information of the target virtual character as the first state information, the second action information as the first action information, and the third state information as the second state information, a sample data is constructed and the sample data is saved to the sample dataset.

[0174] Optionally, the first step further includes:

[0175] The third state information is input into the first network and the second network to obtain the first predicted value output by the first network and the second predicted value output by the second network. The network structure of the first network and the network structure of the second network are the same.

[0176] Based on the difference between the first predicted value and the second predicted value, the parameters of the second network are adjusted, and the difference is determined as the reward information corresponding to the second action information;

[0177] Training data is constructed based on the current state information of the target virtual character, the third state information, the second action information, and the reward information, and the preset behavior strategy network is trained by reinforcement learning based on the training data.

[0178] Optionally, the acquisition module is further configured to:

[0179] The second step is executed repeatedly until the current round of the first game application ends, resulting in a sample dataset containing multiple sample data.

[0180] The second step includes:

[0181] Obtain the current state information of the target virtual character in the first game application;

[0182] The current state information of the target virtual character is input into the target behavior strategy network to obtain the third action information output by the target behavior strategy network. The target behavior strategy network is a network designed to make action decisions to achieve a preset game goal.

[0183] In the first game application, the first virtual character is controlled to perform the action corresponding to the third action information;

[0184] After the first virtual character completes the action corresponding to the third action information, the fourth state information of the target virtual character in the first game application is obtained;

[0185] The current state information of the target virtual character is used as the first state information, the third action information is used as the first action information, and the fourth state information is used as the second state information to construct a sample data and save the sample data to the sample dataset.

[0186] Optionally, the device further includes a training module, which is specifically used for:

[0187] The third step is executed repeatedly until the current round of the first game application ends, resulting in multiple target sample data. The target sample data is used to perform reinforcement learning training on the target behavior policy network, which is a network designed to achieve a preset game goal and make action decisions.

[0188] The third step includes:

[0189] Obtain the current state information of the target virtual character in the twin model;

[0190] The current state information is input into the target behavior policy network to obtain the fourth action information output by the target behavior policy network;

[0191] The current state information and the fourth action information are input into the twin model to obtain the fifth state information output by the twin model. The fifth state information is the state information of the target virtual character predicted by the twin model based on the first virtual character performing the action corresponding to the fourth action information.

[0192] The target reward value is determined based on the preset reward function, the current state information of the target virtual character, and the fifth state information.

[0193] Based on the current state information, the fifth state information, the fourth action information, and the target reward value of the target virtual character, a target sample data is constructed.

[0194] Optionally, the preset reward function is used to calculate the difference between the current state information of the target virtual character and the fifth state information, and to determine the difference as the target reward value.

[0195] Optionally, the network structure of the preset model includes any one of the following: feedforward neural network, convolutional neural network, recurrent neural network, and Transform network.

[0196] The model training device provided in this embodiment can be used to execute the technical solution of the above-described model training method embodiment. Its implementation principle and technical effect are similar, and will not be described again here.

[0197] Figure 5 This is a schematic diagram of the hardware structure of an electronic device provided in one embodiment of this application, such as... Figure 5 As shown, the electronic device 500 of this embodiment includes: a processor 501 and a memory 502; wherein

[0198] Memory 502 is used to store instructions executed by the computer;

[0199] The processor 501 is used to execute computer execution instructions stored in the memory to implement the various steps of the model training method in the above embodiments. For details, please refer to the relevant descriptions in the foregoing method embodiments.

[0200] Alternatively, the memory 502 can be either standalone or integrated with the processor 501.

[0201] When the memory 502 is set up independently, the electronic device also includes a bus 503 for connecting the memory 502 and the processor 501.

[0202] One embodiment of this application also provides a computer-readable storage medium storing computer-executable instructions. When a processor executes the computer-executable instructions, it implements the technical solution corresponding to the model training method in any of the above embodiments executed by the above-mentioned electronic device.

[0203] One embodiment of this application also provides a computer program product, the program product including: a computer program, the computer program being stored in a readable storage medium, at least one processor of an electronic device being able to read the computer program from the readable storage medium, and at least one processor executing the computer program causing the electronic device to execute the technical solution corresponding to the model training method in any of the above embodiments.

[0204] Although this application discloses preferred embodiments as described above, it is not intended to limit this application. Any person skilled in the art can make possible changes and modifications without departing from the spirit and scope of this application. Therefore, the scope of protection of this application should be determined by the scope defined in the claims of this application.

[0205] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or modules, and may be electrical, mechanical, or other forms.

[0206] The integrated modules implemented as software functional modules described above can be stored in a computer-readable storage medium. These software functional modules, stored in a storage medium, include several instructions to cause an electronic device (which may be a personal computer, server, or network device, etc.) or a processor to execute some steps of the methods described in the various embodiments of this application.

[0207] It should be understood that the aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.

[0208] The memory may include high-speed RAM, and may also include non-volatile storage (NVM), such as at least one disk storage device, and may also be a USB flash drive, external hard drive, read-only memory, disk or optical disc, etc.

[0209] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0210] The aforementioned storage medium can be implemented from any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The storage medium can be any available medium accessible to general-purpose or special-purpose computers.

[0211] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0212] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A model training method, characterized in that, The method includes: A sample dataset is obtained, which includes multiple sample data generated during the operation of the first game application. Each sample data includes first state information and second state information of a target virtual character in the first game application, as well as first action information of the first virtual character. The target virtual character includes the first virtual character and the second virtual character. The second virtual character refers to the virtual character located within the field of view of the first virtual character in the game scene of the first game application. The second state information is the state information of the target virtual character after the first virtual character performs the action corresponding to the first action information. The first state information and the first action information in the sample data are input into a preset model to obtain the predicted state information output by the preset model. The predicted state information is the state information of the target virtual character predicted by the preset model based on the first virtual character performing the action corresponding to the first action information. The preset model is trained with the goal of minimizing the preset loss function, and the parameters of the preset model are adjusted. The trained preset model is then used as the twin model corresponding to the first game application. The preset loss function is used to calculate the difference between the predicted state information and the second state information. The twin model is used to predict the next state information of the target virtual character in the first game application based on the current state information of the target virtual character and the first action information performed by the first virtual character. The acquisition of the sample dataset includes: Repeat the first step below until the number of sample data in the sample dataset reaches the preset threshold. The first step includes: The current state information and historical state information of the target virtual character in the first game application are input into a preset behavior strategy network to obtain the second action information output by the preset behavior strategy network. The preset behavior strategy network is used to predict the next action information to be performed by the first virtual character based on the current state information and historical state information of the target virtual character. In the first game application, the first virtual character is controlled to perform the action corresponding to the second action information; After the first virtual character completes the action corresponding to the second action information, the third state information of the target virtual character in the first game application is obtained; Using the current state information of the target virtual character as the first state information, the second action information as the first action information, and the third state information as the second state information, a sample data is constructed and the sample data is saved to the sample dataset.

2. The method according to claim 1, characterized in that, The first step also includes: The third state information is input into the first network and the second network to obtain the first predicted value output by the first network and the second predicted value output by the second network. The network structure of the first network and the network structure of the second network are the same. Based on the difference between the first predicted value and the second predicted value, the parameters of the second network are adjusted, and the difference is determined as the reward information corresponding to the second action information; Training data is constructed based on the current state information of the target virtual character, the third state information, the second action information, and the reward information, and the preset behavior strategy network is trained by reinforcement learning based on the training data.

3. The method according to claim 1, characterized in that, The acquisition of the sample dataset includes: The second step is executed repeatedly until the current round of the first game application ends, resulting in a sample dataset containing multiple sample data. The second step includes: Obtain the current state information of the target virtual character in the first game application; The current state information of the target virtual character is input into the target behavior strategy network to obtain the third action information output by the target behavior strategy network. The target behavior strategy network is a network designed to make action decisions to achieve a preset game goal. In the first game application, the first virtual character is controlled to perform the action corresponding to the third action information; After the first virtual character completes the action corresponding to the third action information, the fourth state information of the target virtual character in the first game application is obtained; The current state information of the target virtual character is used as the first state information, the third action information is used as the first action information, and the fourth state information is used as the second state information to construct a sample data and save the sample data to the sample dataset.

4. The method according to claim 1, characterized in that, The method further includes: The third step is executed repeatedly until the current round of the first game application ends, resulting in multiple target sample data. The target sample data is used to perform reinforcement learning training on the target behavior policy network, which is a network designed to achieve a preset game goal and make action decisions. The third step includes: Obtain the current state information of the target virtual character in the twin model; The current state information is input into the target behavior policy network to obtain the fourth action information output by the target behavior policy network; The current state information and the fourth action information are input into the twin model to obtain the fifth state information output by the twin model. The fifth state information is the state information of the target virtual character predicted by the twin model based on the first virtual character performing the action corresponding to the fourth action information. The target reward value is determined based on the preset reward function, the current state information of the target virtual character, and the fifth state information. Based on the current state information, the fifth state information, the fourth action information, and the target reward value of the target virtual character, a target sample data is constructed.

5. The method according to claim 4, characterized in that, The preset reward function is used to calculate the difference between the current state information and the fifth state information of the target virtual character, and to determine the difference as the target reward value.

6. The method according to claim 1, characterized in that, The network structure of the preset model includes any one of the following: feedforward neural network, convolutional neural network, recurrent neural network, and Transform network.

7. A model training device, characterized in that, The device includes: The acquisition module is used to acquire a sample dataset, which includes multiple sample data generated during the operation of the first game application. Each sample data includes first state information and second state information of a target virtual character in the first game application, as well as first action information of the first virtual character. The target virtual character includes the first virtual character and the second virtual character. The second virtual character refers to the virtual character located within the field of view of the first virtual character in the game scene of the first game application. The second state information is the state information of the target virtual character after the first virtual character performs the action corresponding to the first action information. The processing module is used to input the first state information and the first action information in the sample data into a preset model to obtain the predicted state information output by the preset model. The predicted state information is the state information of the target virtual character predicted by the preset model based on the first virtual character performing the action corresponding to the first action information. The control module is used to train the preset model with the goal of minimizing the preset loss function, adjust the parameters of the preset model, and use the trained preset model as the twin model corresponding to the first game application; wherein, the preset loss function is used to calculate the difference between the predicted state information and the second state information, and the twin model is used to predict the next state information of the target virtual character in the first game application based on the current state information of the target virtual character and the first action information performed by the first virtual character; The acquisition of the sample dataset includes: Repeat the first step below until the number of sample data in the sample dataset reaches the preset threshold. The first step includes: The current state information and historical state information of the target virtual character in the first game application are input into a preset behavior strategy network to obtain the second action information output by the preset behavior strategy network. The preset behavior strategy network is used to predict the next action information to be performed by the first virtual character based on the current state information and historical state information of the target virtual character. In the first game application, the first virtual character is controlled to perform the action corresponding to the second action information; After the first virtual character completes the action corresponding to the second action information, the third state information of the target virtual character in the first game application is obtained; Using the current state information of the target virtual character as the first state information, the second action information as the first action information, and the third state information as the second state information, a sample data is constructed and the sample data is saved to the sample dataset.

8. An electronic device, characterized in that, The electronic device includes: Processor; and A memory for storing a data processing program, which, when powered on and run by the processor, executes the model training method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The system contains a data processing program that is executed by a processor to perform the model training method as described in any one of claims 1-6.

Images