Training method and device for intelligent agent in game, computer equipment and storage medium

Abstract

The invention provides a training method and device for an agent in a game, computer equipment and a storage medium, relates to the technical field of games, and solves the technical problem that the reinforcement learning training effect of the agent is poor. The method comprises the following steps: in response to a trigger event that the intelligent agent participates in the current game, determining the step number of corresponding initial training steps when the current game starts to run; determining a corresponding initial exploration probability when the current game starts to run; determining a first exploration probability corresponding to the step number of a first training step in the current game operation process according to the exploration strategy adjustment rule, and executing the first training step based on the first exploration probability; and running the current game to the cut-off duration corresponding to the current game, and repeatedly executing the steps in the running process of the multiple games until the training is finished, thereby obtaining the trained intelligent agent.

Description

Technical Field

[0001] The present disclosure relates to the field of game technology, and in particular to a method, device, computer equipment and storage medium for training an intelligent agent in a game. Background Art

[0002] Reinforcement learning is widely used in the field of artificial intelligence. Reinforcement learning trains intelligent agents to explore and exploit the environment, so that they can find the optimal solution through continuous attempts. Among them, exploration and exploitation are two basic concepts in reinforcement learning. Exploration refers to the intelligent agent trying new behaviors in an unknown environment to obtain more environmental information; exploitation refers to the intelligent agent choosing the current optimal solution to act based on the information it has mastered. In the process of reinforcement learning training, in order to improve the learning effect of the intelligent agent, it is necessary to reasonably set the ratio of exploration and exploitation.

[0003] At present, the common exploration strategy is the ε-greedy strategy, in which the agent randomly selects actions with a certain probability ε, that is, the exploration probability is set to ε, and then selects the current optimal action with a probability of 1-ε, that is, the utilization rate is set to 1-ε. This method is simple and easy to implement. However, this exploration strategy cannot balance the relationship between exploration and utilization, resulting in poor reinforcement learning training effects of the agent. Summary of the invention

[0004] The purpose of the present disclosure is to provide a method, device, computer equipment and storage medium for training an intelligent agent in a game, so as to alleviate the technical problem that the reinforcement learning training effect of the intelligent agent is poor.

[0005] In a first aspect, an embodiment of the present disclosure provides a method for training an intelligent agent in a game, wherein the intelligent agent participates in a cyclic operation of multiple games and performs reinforcement learning training during the operation; wherein the game corresponds to a preset exploration strategy adjustment rule, and the exploration strategy adjustment rule includes: when each game of the game starts running, there is an initial exploration probability corresponding to the initial training step number corresponding to the initial operation of each game, and the initial exploration probability is smaller when the initial training step number is larger, and the exploration probability is used to characterize the probability of the algorithm choosing to explore a new solution in the reinforcement learning algorithm; the method includes:

[0006] In response to a triggering event in which the agent participates in a current round of the game, determining the number of initial training steps corresponding to when the current round of the game starts running;

[0007] Determine the initial exploration probability corresponding to the start of the current round of the game;

[0008] Determining a first exploration probability corresponding to the number of steps in a first training step during the current round of the game according to the exploration strategy adjustment rule, and executing the first training step based on the first exploration probability;

[0009] Run the current round of the game to the deadline corresponding to the current round of the game, and repeat the above steps during the running of multiple games until the training is completed to obtain a trained intelligent agent.

[0010] In a second aspect, a training device for an intelligent agent in a game is provided, wherein the intelligent agent participates in a cyclic operation of multiple rounds of the game and performs reinforcement learning training during the operation; wherein the game corresponds to a preset exploration strategy adjustment rule, and the exploration strategy adjustment rule includes: an initial exploration probability corresponds to the start of each round of the game, and the larger the number of initial training steps corresponding to the start of each round of the game, the smaller the initial exploration probability, and the exploration probability is used to characterize the probability of the algorithm choosing to explore a new solution in the reinforcement learning algorithm; the device includes:

[0011] A first determination module is used to determine the number of initial training steps corresponding to the start of the current round of the game in response to a triggering event in which the agent participates in the current round of the game; and to determine the initial exploration probability corresponding to the start of the current round of the game;

[0012] A second determination module is used to determine a first exploration probability corresponding to the number of steps of a first training step in the current game according to the exploration strategy adjustment rule, and execute the first training step based on the first exploration probability;

[0013] The running module is used to run the current round of the game to the deadline corresponding to the current round of the game, and repeatedly execute the above steps during the running of multiple games until the training is completed to obtain a trained intelligent agent.

[0014] In a third aspect, an embodiment of the present disclosure further provides a computer device, including a memory and a processor, wherein the memory stores a computer program executable on the processor, and when the processor executes the computer program, the method described in the first aspect is implemented.

[0015] In a fourth aspect, an embodiment of the present disclosure further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions cause the processor to execute the method described in the first aspect.

[0016] The embodiments of the present disclosure bring the following beneficial effects:

[0017] The embodiments of the present disclosure provide a method, device, computer equipment and storage medium for training an intelligent agent in a game. The intelligent agent participates in a cyclic operation of multiple games and performs reinforcement learning training during the operation. The games correspond to preset exploration strategy adjustment rules. The exploration strategy adjustment rules include an initial exploration probability corresponding to the start of each game, and the larger the number of steps of the initial training step corresponding to the start of each game, the smaller the initial exploration probability. The exploration probability is used to characterize the probability of the algorithm choosing to explore new solutions in the reinforcement learning algorithm. The method can respond to a trigger event of the intelligent agent participating in the current game, determine the number of steps of the initial training step corresponding to the start of the current game, determine the initial exploration probability corresponding to the start of the current game, determine the first exploration probability corresponding to the number of steps of the first training step during the operation of the current game according to the exploration strategy adjustment rules, execute the first training step based on the first exploration probability, and run the current game to the deadline corresponding to the current game, and repeat the above steps during the operation of multiple games until the training is completed to obtain a trained intelligent agent. In this scheme, by adopting the exploration strategy adjustment rule that the larger the number of training steps, the smaller the initial exploration probability at the beginning of each game during the reinforcement learning training of the intelligent agent, not only does it not affect the exploration strategy during the operation of each game, but more importantly, it makes the initial exploration probability at the beginning of each game in the early stage of training larger, which can promote the intelligent agent to explore different game behaviors and avoid falling into local optimal solutions too early, and also makes the training volume in the early stage of training sufficient. By gradually reducing the initial exploration probability at the beginning of each game as the training progresses, the intelligent agent can retain the behaviors that have been learned in the early stage of training. In the late stage of training, the initial exploration probability at the beginning of each game gradually decreases to a smaller value, which can enable the intelligent agent to use the behaviors in the late stage of training more attentively for optimization after completing the learning of the early behaviors. Through this exploration strategy with dynamic adjustment of the exploration probability, the situation of insufficient exploration in the early stage of training and excessive exploration in the late stage of training is avoided, and a dynamic balance between exploration and utilization is achieved in the entire training process, which improves the reinforcement learning training effect of the intelligent agent and alleviates the technical problem of poor reinforcement learning training effect of the intelligent agent.

[0018] In order to make the above-mentioned objectives, features and advantages of the present disclosure more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the specific embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the specific embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0020] Figure 1 A schematic diagram of an application scenario provided by an embodiment of the present disclosure is shown;

[0021] Figure 2 A schematic diagram of the structure of a computer device provided by an embodiment of the present disclosure is shown;

[0022] Figure 3 A flowchart of a method for training an intelligent agent in a game provided by an embodiment of the present disclosure;

[0023] Figure 4 An example diagram of the relationship between the number of training steps and the exploration probability in the method for training an intelligent agent in a game provided by an embodiment of the present disclosure;

[0024] Figure 5 Another example diagram of the relationship between the number of training steps and the exploration probability in the method for training an intelligent agent in a game provided by an embodiment of the present disclosure;

[0025] Figure 6 A schematic diagram of the structure of a training device for an intelligent agent in a game provided by an embodiment of the present disclosure;

[0026] Figure 7 A schematic structural diagram of an electronic device provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present disclosure, rather than all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present disclosure.

[0028] The terms "including" and "having" and any variations thereof mentioned in the embodiments of the present disclosure are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device including a series of steps or units is not limited to the listed steps or units, but may optionally include other steps or units that are not listed, or may optionally include other steps or units that are inherent to these processes, methods, products or devices.

[0029] At present, there are many existing exploration strategies. For example, the ε-greedy exploration strategy is that the agent randomly selects an action with a certain probability ε, and then selects the current optimal action with a probability of 1-ε. The exploration probability in this exploration strategy remains unchanged throughout the training process, which may lead to insufficient exploration in the early stage of training and excessive exploration in the later stage. For another example, the Bayesian exploration strategy is based on a probability model for exploration. The agent updates the probability model based on prior knowledge and environmental feedback, so that it can adaptively adjust the exploration probability. However, this exploration strategy has high computational complexity and is not suitable for high-dimensional action spaces. For another example, the Upper Confidence Bound (UCB) algorithm calculates the confidence interval and selects the action with the maximum confidence upper limit for exploration, so that it can balance exploration and utilization to a certain extent, but this exploration strategy is difficult to apply to continuous action spaces.

[0030] It can be seen that the existing exploration strategy is difficult to achieve a dynamic balance between exploration and utilization during the entire training process. Furthermore, the existing fixed exploration probability and complex exploration strategy lead to a long training process and slow convergence, which leads to low training efficiency. Therefore, the reinforcement learning training effect of the intelligent agent trained using the existing exploration strategy is poor.

[0031] Based on this, the embodiments of the present disclosure provide a method, apparatus, computer device and storage medium for training an intelligent agent in a game, by which the technical problem of poor reinforcement learning training effect of the intelligent agent can be alleviated.

[0032] In one embodiment of the present disclosure, the method for training an agent in a game can be run on a local terminal device or a server. When the method for training an agent in a game is run on a server, the method can be implemented and executed based on a cloud interaction system, wherein the cloud interaction system includes a server and a client device.

[0033] In an optional implementation, various cloud applications can be run under the cloud interaction system, such as cloud games. Taking cloud games as an example, cloud games refer to a game mode based on cloud computing. In the operation mode of cloud games, the operating body of the game program and the main body of the game screen presentation are separated. The storage and operation of the training method of the intelligent body in the game are completed on the cloud game server. The role of the client device is used for receiving and sending data and presenting the game screen. For example, the client device can be a display device with data transmission function close to the user side, such as a mobile terminal, a TV, a computer, a handheld computer, etc.; but the cloud game server in the cloud is used for information processing. When playing the game, the player operates the client device to send an operation instruction to the cloud game server. The cloud game server runs the game according to the operation instruction, encodes and compresses the game screen and other data, and returns it to the client device through the network. Finally, the client device decodes and outputs the game screen.

[0034] In an optional embodiment, taking a game as an example, a local terminal device stores a game program and is used to present a game screen. The local terminal device is used to interact with the player through a graphical user interface, that is, the game program is downloaded and installed by an electronic device and run conventionally. The local terminal device may provide the graphical user interface to the player in a variety of ways, for example, it may be rendered and displayed on a display screen of the terminal, or provided to the player through a holographic projection. For example, the local terminal device may include a display screen and a processor, the display screen is used to present a graphical user interface, the graphical user interface includes a game screen, and the processor is used to run the game, generate a graphical user interface, and control the display of the graphical user interface on the display screen.

[0035] In one possible implementation, the present disclosure provides a method for training an intelligent agent in a game, providing a graphical user interface through a terminal device, wherein the terminal device may be the local terminal device mentioned above, or may be a client device in the cloud interaction system mentioned above.

[0036] For example, Figure 1 As shown, Figure 1 A schematic diagram of an application scenario provided by an embodiment of the present disclosure. The application scenario may include a terminal device (such as a computer device 102) and a server 101, and the terminal device may communicate with the server 101 via a wired network or a wireless network. The terminal device is used to run a virtual desktop, and through the virtual desktop, it can interact with the server 101 to control the content in the server 101.

[0037] The terminal device of this embodiment is described by taking a computer device 102 as an example. The computer device 102 includes components such as a radio frequency (RF) circuit 110, a memory 120, a touch screen 130, and a processor 140. Those skilled in the art can understand that Figure 2 The computer device structure shown in the figure does not constitute a limitation on the computer device, and may include more or fewer components than shown in the figure, or combine some components, or separate some components, or arrange the components differently. Those skilled in the art can understand that the touch screen 130 belongs to the user interface (UI), and the computer device 102 may include more or fewer user interfaces than shown in the figure.

[0038] The RF circuit 110 can also communicate with the network and other devices through wireless communication. The wireless communication can use any communication standard or protocol, including but not limited to Global System of Mobilecommunication (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, Short Messaging Service (SMS), etc.

[0039] The memory 120 can be used to store software programs and modules. The processor 140 executes various functional applications and data processing of the computer device 102 by running the software programs and modules stored in the memory 120. The memory 120 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function, etc.; the data storage area can store data created according to the use of the computer device 102, etc. In addition, the memory 120 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0040] The processor 140 is the control center of the computer device 102. It uses various interfaces and lines to connect the various parts of the entire computer device. It executes various functions of the computer device 102 and processes data by running or executing software programs and / or modules stored in the memory 120 and calling data stored in the memory 120, thereby monitoring the computer device as a whole.

[0041] The embodiments of the present disclosure are further described below in conjunction with the accompanying drawings.

[0042] Figure 3 A flowchart of a method for training an intelligent agent in a game provided by an embodiment of the present disclosure. The method can be applied to a computer device capable of running games, wherein the intelligent agent participates in a cycle of multiple games and performs reinforcement learning training during the running process, wherein the game corresponds to a preset exploration strategy adjustment rule, and the exploration strategy adjustment rule includes: an initial exploration probability corresponds to the start of each game, and the larger the number of initial training steps corresponding to the start of each game, the smaller the initial exploration probability, and the exploration probability is used to characterize the probability of the algorithm choosing to explore new solutions in the reinforcement learning algorithm. Figure 3 As shown, the method includes:

[0043] Step S310, in response to a triggering event in which the agent participates in the current round of the game, determines the number of initial training steps corresponding to the start of the current round of the game.

[0044] It should be noted that reinforcement learning (RL) is a machine learning paradigm that learns the optimal behavior strategy through the interaction between the agent and the environment and based on reward signals. Among them, machine learning (ML) is a technology that enables computers to learn from data and make predictions or decisions. It uses algorithms to allow computers to simulate human learning behavior, thereby realizing intelligent data processing and analysis.

[0045] In practical applications, the algorithm in the reinforcement learning algorithm can choose to explore new solutions, and the algorithm in the reinforcement learning algorithm can also choose to use the information already mastered to select the optimal solution. The number of steps in the embodiment of the present disclosure can be the number of training steps to complete an exploration in the reinforcement learning algorithm, or the number of training steps to complete an utilization.

[0046] In the disclosed embodiment, in the early stage (hereinafter referred to as the early training stage) of the entire training process (including multiple games) of reinforcement learning, the initial exploration probability at the beginning of each game is a relatively large preset value, and in the later stage (hereinafter referred to as the later training stage) of the entire training process (including multiple games), the initial exploration probability at the beginning of each game gradually decreases to a preset minimum value.

[0047] Exemplarily, as Figure 4 shown, for training steps other than the training step with the training step number N = 0, the initial exploration random probability value at the 0 moment of the game time (i.e., the initial exploration probability when each game starts to run) starts from a relatively low value, and this relatively low value at the beginning of each game gradually decreases as the entire training process (including multiple games) progresses. For example, as Figure 5 shown, as the training step number increases from N = 0k to N = 50k, N = 50k to N = 100k, N = 100k to N = 150k, N = 150k to N = 200k, N = 200k to N = 250k, N = 250k to N = 300k, N = 300k to N = 350k, N = 350k to N = 500k, N = 400k to N = 450k, this relatively low value at the beginning of each game (the initial exploration probability when each game starts to run) decreases from 1 to 0.8, from 0.8 to 0.6, from 0.6 to 0.4, and from 0.4 to 0.2.

[0048] Step S320, determine the initial exploration probability corresponding to the start of the current game.

[0049] In a possible implementation manner, the initial exploration probability corresponding to the start of the current game can be determined according to the total training step number corresponding to the end of the previous game of the current game, the preset minimum running game duration, the preset maximum probability, and the preset minimum probability.

[0050] Exemplarily, according to the total training step number corresponding to the end of the previous game of the current game, the preset minimum running game duration, the preset maximum probability, and the preset minimum probability, the initial exploration probability corresponding to the start of the current game is determined through the following formula: p0 = max(P min , P max - ΔP) (Formula 1), where P max is the preset maximum probability with a value between 0 and 1 (such as Figure 4 the P max in it); P min is the preset minimum probability with a value between 0 and P max (such as Figure 4 the P min in it); α, β, and γ are all preset constant coefficients with values greater than 0; N is the total training step number corresponding to the end of the previous game of the current game; T0 is the preset minimum running game duration; p0 is the initial exploration probability corresponding to the start of each game.

[0051] Such as Figure 4 and Figure 5 As shown, in the early stage of training, the initial exploration probability value at the beginning of each game is large, for example Figure 4 P max ,As the number of steps increases during the entire training process (including multiple games), in the later stages of training, the initial exploration probability value at the beginning of each game gradually decreases to a preset minimum value, e.g. Figure 4 P min , it can be seen that the initial exploration random probability value p0 when the game time is 0 can be expressed by the above formula 1.

[0052] The above formulas 1 and 2 make the exploration probability value more accurate as the number of training steps is calculated. Using a larger initial exploration probability value in the early stages of training can encourage the agent to explore different game strategies and avoid falling into the local optimal solution too early. As the training progresses, the amount of training in the early stages of training is sufficient so that there is no need for excessive exploration. By gradually reducing the initial exploration probability at the beginning of each game, the agent can retain the learned early strategies. For the later stages of training, the initial exploration probability at the beginning of each game is reduced to a smaller value, which allows the agent to focus more on optimizing the behavior in the later stages of training after completing the early learning behavior, thereby balancing the relationship between exploration and utilization in the reinforcement learning process.

[0053] For the above-mentioned process of the agent participating in the multi-game loop operation, for example, the training process of the first game can be understood as the number of training steps from 0 (for example Figure 5 N = 0k training steps) to n1 training steps (e.g. Figure 5 N = 50k training steps), the training process of the second game can be understood as starting from n1 training steps (e.g. Figure 5 N = 50k training steps) to n2 training steps (e.g. Figure 5 The training process of the third game can be understood as starting from n2 training steps (e.g. Figure 5 N = 100k training steps) to n3 training steps (e.g. Figure 5 N = 150k training steps) ... and so on. It can also be understood that the number of training steps gradually increases. After executing a certain number of training steps, the current game ends and a new game starts, that is, the game restarts (the same game can be restarted multiple times, that is, the agent can participate in the cyclic operation of multiple games).

[0054] Step S330, determining a first exploration probability corresponding to the number of steps of a first training step during the current game session according to the exploration strategy adjustment rule, and executing the first training step based on the first exploration probability.

[0055] It should be noted that the first training step refers to the training step corresponding to any other time during the current game except the initial training step. For example, during the training of the first game, the number of steps of the initial training step is the first step, and the number of steps of the first training step can be the second step, the third step or the fourth step, etc., any other number of steps during the training of the first game except the first step.

[0056] During the reinforcement learning training process of the intelligent agent, the exploration strategy adjustment rules can be used to control the exploration probability of each training step and gradually adjust it as the training process progresses, so that the intelligent agent can adaptively balance exploration and utilization during the training process and achieve a dynamic balance between exploration and utilization.

[0057] Step S340, run the current round of the game to the deadline corresponding to the current round of the game, and repeat the above steps during the running of multiple rounds of the game until the training is completed to obtain a trained intelligent agent.

[0058] In practical applications, the intelligent characters trained in the embodiments of the present disclosure can be used to participate in human-computer games, for example, robots playing chess games with users, enemy virtual characters and friendly virtual characters in game scenes, etc. The intelligent characters can play the roles controlled by other users in the game.

[0059] By adopting the exploration strategy adjustment rule that the larger the number of training steps, the smaller the initial exploration probability at the beginning of each game during the reinforcement learning training of the agent, not only does it not affect the exploration strategy during the operation of each game, but more importantly, it makes the initial exploration probability at the beginning of each game in the early stage of training larger, which can promote the agent to explore different game behaviors and avoid falling into the local optimal solution too early, and also makes the training volume in the early stage of training sufficient. By gradually reducing the initial exploration probability at the beginning of each game as the training progresses, the agent can retain the learned behavior in the early stage of training. In the late stage of training, the initial exploration probability at the beginning of each game gradually decreases to a smaller value, which can enable the agent to use the behavior in the late stage of training more attentively for optimization after completing the learning of the early behavior. Through this exploration strategy with dynamic adjustment of the exploration probability, the situation of insufficient exploration in the early stage of training and excessive exploration in the late stage of training is avoided, and a dynamic balance of exploration and utilization is achieved in the whole training process, which improves the reinforcement learning training effect of the agent, alleviates the technical problem of poor reinforcement learning training effect of the agent, and realizes the agent reinforcement learning exploration strategy method that adapts to the evolution of the learning stage.

[0060] The above steps are described in detail below.

[0061] In some embodiments, the current round deadline of the game can be controlled to gradually increase as the training process proceeds, so as to speed up the learning speed of the agent, thereby shortening the training time and improving the training efficiency. As an example, the exploration strategy adjustment rule includes: the larger the total number of training steps corresponding to the end of the previous round of the current round of the game, the larger the current round deadline of the corresponding game; the method may also include the following steps: in response to the current running time of the current round of the game reaching the target current round deadline corresponding to the current round of the game, controlling the end of the current round of the game.

[0062] Exemplarily, in the early stages of training, the current round deadline of the game is set to a smaller value. As the number of training steps increases during the entire training process (including multiple games), the current round deadline of the game continues to increase, but does not exceed the time when the game ends naturally. For example, for a Multiplayer Online Battle Arena (MOBA) game, the game ends naturally when one side wins.

[0063] For example, when the agent participates in a multi-game loop, the first batch of training processes is executed in the first game, and the current game deadline of the first game is T0, that is, the game time of the first game is from 0 to T0, as the number of training steps increases (for example, the training process of the first game is from 0 training steps to n1 training steps), when the current running time of the first game reaches T0, the first game ends, and the time of the second game is calculated from 0, and the second batch of training processes is executed in the second game, and the current game deadline of the second game is T2 (T2 is longer than the above T1, because as the training process proceeds, that is, the number of training steps increases, the game deadline of each game continues to increase), that is, the game time of the second game is from 0 to T2, as the number of training steps increases (for example, the training process of the second game is from At the end of the first batch of training processes, the training steps of n1 continue to the training steps of n2). When the current running time of the second game reaches T2, the second game ends, and the time of the third game starts from 0. The third batch of training processes is executed in the third game, and the current game end time of the third game is T3 (T3 is longer than the above T2, because as the training process progresses, that is, the training steps increase, the game end time of each game continues to increase), that is, the game time of the third game is from 0 to T3, as the number of training steps increases (for example, the training process of the third game is from the training steps of n2 at the end of the second batch of training processes to the training steps of n3), when the current running time of the third game reaches T3, the third game ends, and the time of the fourth game starts from 0..., and so on. It can be seen that each time a new game is started, that is, the game is restarted (the same game can be restarted multiple times, that is, the agent can participate in the cyclic operation of multiple games), the game time is calculated from 0.

[0064] In the disclosed embodiment, during the reinforcement learning training of the intelligent agent, the current round deadline of the game is controlled to gradually increase as the training process proceeds. Since the intelligent agent needs to quickly learn basic strategies and rules in the early stages of training without having to deal with all the complexities of the complete game, the learning process and learning speed of the intelligent agent can be accelerated, the training time can be shortened, and the training efficiency can be improved.

[0065] In some embodiments, a shorter game current round cut-off time can be set in the early stage of training to speed up the learning process and learning speed of the agent, thereby shortening the training time. As an example, the minimum current round cut-off time in the exploration strategy adjustment rule is the preset minimum running game time.

[0066] As an optional implementation, in the early stages of training, the game's current round cutoff time is set to a smaller value, and as the number of training steps increases, the game's current round cutoff time is continuously increased. Since the agent needs to quickly learn basic strategies and rules in the early stages of training, it does not have to deal with all the complexities of the complete game. By setting a shorter game current round cutoff time in the early stages of training, the learning process and learning speed of the agent can be accelerated, the training time can be shortened, and the training efficiency can be further improved.

[0067] In some embodiments, the current round cut-off time of the game can be determined based on the time corresponding to the natural end of the game, the preset minimum running game time, and the total number of training steps corresponding to the start of the current round of the game, so that the data calculation of the current round cut-off time of the game is more accurate. As an example, the current round cut-off time of the game is determined based on the time corresponding to the natural end of the game, the preset minimum running game time, and the total number of training steps corresponding to the end of the previous round of the current round of the game.

[0068] Exemplarily, the current round deadline of the game can be expressed by the following formula: N =min(T e ,T0+αN)(Formula 3); where T e is the duration of the natural end of the game; T0 is the preset minimum duration; α is a preset constant coefficient with a value greater than 0; N is the total number of training steps corresponding to the end of the previous game of the current game; t N The current game end time.

[0069] It should be noted that the current round deadline of the first game is a fixed value of the preset minimum game running time, and the total number of training steps executed during the running process of the first game is determined according to the preset minimum game running time and the number of games running in parallel.

[0070] Then, the current game deadline of each game after the first game and the total number of training steps corresponding to the end of the previous game before the current game are in an iterative relationship. For example, at the beginning of the first game, the total number of training steps N=0, corresponding to the deadline t0=T0 of the current game (first game); at the beginning of the second game, the corresponding total number of training steps N1=T0×specified sample collection frequency (if the specified sample collection frequency is 0.2, it means 5 samples per second)×the number of games running in parallel at the same time, and the corresponding deadline t1=T0+αN1 of the current game (second game); at the beginning of the third game, the corresponding total number of training steps N2=t1×specified sample collection frequency×the number of games running in parallel at the same time, and the corresponding deadline t2=T0+αN2 of the current game (third game); at the beginning of the fourth game, the corresponding total number of training steps N3=t2×specified sample collection frequency×the number of games running in parallel at the end of the previous game, and the corresponding deadline t3=T0+αN3 of the current game (fourth game).

[0071] In the early stage of training, the game deadline is a smaller value. At the beginning of training, the current game deadline is T=T0. As the number of training steps increases, the game deadline when the number of training steps is N is adjusted to the above formula 3. This formula can make the value of the current game deadline gradually increase with the increase of the number of training steps more accurate.

[0072] In some embodiments, the number of training steps executed in each game can be determined in combination with the game duration and the number of games running in parallel, so that the determination of the number of training steps is more accurate, so as to more accurately determine the corresponding exploration probability based on the accurate number of training steps. As an example, the total number of training steps executed during the running of each game is determined based on the current game deadline of the game and the number of games running in parallel at the same time.

[0073] Exemplarily, in the first game corresponding to the first batch of training stages (pre-training stage), the current game deadline of the first game is T0, that is, the game time of the first game is from 0 to T0, then the total number of training steps executed during the running of the first game is 0 to T0*M, where M is the number of parallel game environments, that is, the number of games running in parallel at the same time. In the second game corresponding to the second batch of training stages (such as the middle training stage), the game time is calculated from 0, and the current game deadline of the second game is T1, that is, the game time of the second game is from 0 to T1 (longer than T0, that is, the game deadline increases with the increase of the number of training steps), then the total number of training steps executed during the running of the second game is 0 to T0*M, which is from the above T0*M to (T0+T1)*M, that is, from the step T0*M executed corresponding to the end time T0 of the previous batch of training processes to the step (T0+T1)*M at the time T1+T2. By analogy, the process of the subsequent nth batch of training (such as the later stage of training) is the same. The current game deadline of the nth game is Tn (longer than Tn-1, that is, the game deadline continues to increase with the increase of training steps), the game time is calculated from 0, and the total number of training steps executed during the running of the nth game is 0 to T0*M, which is the steps (T1+…Tn-1))*M executed from the end time Tn-1 of the previous batch of training processes to the steps (T1+…+Tn-1+Tn)*M at the time Tn.

[0074] In the disclosed embodiment, the number of training steps executed in each game is determined in combination with the game duration and the number of games running in parallel, so that the determination of the number of training steps is more precise, thereby being able to more accurately determine the corresponding exploration probability and the deadline of the current game according to the precise number of training steps.

[0075] In some embodiments, the exploration probability can be gradually increased during the running of each game, so that the balance between exploration and utilization can be maintained during the running of each game. As an example, the above step S330 can specifically include the following steps:

[0076] In response to a triggering event in which an intelligent agent participates in a current round of a game, a target current running time of the current round of the game is determined; the number of steps of a first training step in the running process of the current round of the game is determined according to the target current running time; a first exploration probability corresponding to the number of steps of the first training step is determined according to an exploration strategy adjustment rule, and the first training step is executed based on the first exploration probability and the number of steps of the first training step corresponding to the target current running time.

[0077] For example, Figure 4 and Figure 5As shown in the figure, in the same period of training, for example, in the late stage of training, the randomness of sample exploration, that is, the exploration probability, gradually increases with the continuous increase of game time. In this way, the exploration probability gradually increases during the running of each game, so that the balance between exploration and exploitation can be maintained during the running of each game.

[0078] In some embodiments, in the early stages of training, the exploration probabilities during the same game run are kept at a maximum probability, so that the agent can be encouraged to explore different game behaviors in the early stages of training, so that the training volume in the early stages of training is sufficient and it avoids falling into a local optimal solution too early.

[0079] As an example, the exploration strategy adjustment rules also include: for the training steps whose number of initial training steps corresponding to the start of the current game round is greater than the preset number of steps, during the running of the same game round, the longer the current running time of the game is, the greater the corresponding first exploration probability is; for the training steps whose number of initial training steps corresponding to the start of the current game round is less than or equal to the preset number of steps, the first exploration probability during the running of the same game round is the preset maximum probability.

[0080] For example, Figure 4 As shown in the figure, when the training step number N = 0, the exploration probability (random probability) during the current game is maintained at the maximum exploration probability P as the game time progresses. max unchanged; when the training step number N = n1, the exploration probability (random probability) during the current game operation gradually increases to P as the game time progresses max ; When the number of training steps N = n2, the exploration probability (random probability) during the current game is P as the game time progresses min And gradually increase to P max .

[0081] For example, Figure 5 As shown, when the training step is the N=0kth step, the exploration probability value maintains the maximum exploration probability and does not change with the gradual increase of the first game time, that is, the maximum exploration probability is always 1.0; when the training step just starts to enter the N=50kth step and enters the second game, the exploration probability value is reduced to 0.8, but as the second game time gradually increases, the exploration probability value gradually increases to the maximum exploration probability of 1.0, and the exploration probability value does not change after it increases to 1.0; when the training step just starts to enter the N=100kth step and enters the third game, the exploration probability value is reduced to 0.6, but as the third game time gradually increases, the exploration probability value gradually increases to the maximum exploration probability of 1.0, and the exploration probability value does not change after it increases to 1.0; ... and so on.

[0082] In the disclosed embodiment, in the early stage of training, the first exploration probability during the same game run remains unchanged at the maximum probability, so that the intelligent agent can be encouraged to explore different game behaviors in the early stage of training, so that the training volume in the early stage of training is sufficient and it avoids falling into the local optimal solution too early.

[0083] In some embodiments, the values ​​of the minimum exploration probability and the maximum exploration probability can be preset to avoid the exploration probability being too large or too small during the training process, thereby affecting the balance between exploration and utilization. As an example, the minimum exploration probability in the exploration strategy adjustment rule is the preset minimum probability; the maximum exploration probability in the exploration strategy adjustment rule is the preset maximum probability.

[0084] For example, Figure 4 As shown, the minimum exploration probability is P min , the maximum exploration probability is P max . For example, Figure 5 As shown in the figure, the minimum exploration probability is 0.2 and the maximum exploration probability is 1.0. By presetting the values ​​of the minimum exploration probability and the maximum exploration probability, it is possible to avoid the balance between exploration and utilization caused by the exploration probability being too large or too small during the reinforcement learning training process.

[0085] In some embodiments, during the running of each game, the corresponding exploration probability of the game at a certain running time can be calculated by a formula to make the calculation of the exploration probability more accurate and flexible. As an example, the process of determining the first exploration probability corresponding to the number of steps in the first training step according to the exploration strategy adjustment rule can specifically include the following steps:

[0086] The first exploration probability corresponding to the number of steps in the first training step is determined based on the current target running time, the total number of training steps corresponding to the end of the previous game of the current game, the initial exploration probability corresponding to the start of the current game, and the preset minimum running game time.

[0087] Exemplarily, for a training step with a number of steps greater than a preset number of steps, during the running of the same game, according to the current running time of the target, the total number of training steps corresponding to the end of the previous game of the current game, the initial exploration probability corresponding to the start of the current game, and the preset minimum running game time, the first exploration probability corresponding to the number of steps in the first training step is determined by the following formula: p t =p0+γΔP (Formula 4), where, p0=max(P min ,P max -ΔP); P max is the preset maximum probability between 0 and 1; P min It is a value between 0 and P maxThe preset minimum probability between α, β and γ are all preset constant coefficients with values ​​greater than 0; N is the total number of training steps corresponding to the end of the previous game of the current game; T0 is the preset minimum running game duration; p0 is the initial exploration probability corresponding to the start of each game; p t It is the exploration probability corresponding to the time when the game runs to time t.

[0088] For example, Figure 4 As shown, when the training step number N = n2, during the current game, for the moment of game time t, the exploration random probability value (exploration probability) p t The calculation method satisfies the above formula 4. It can be seen that during the operation of each game, the corresponding exploration probability of the game at a certain running time can be calculated by the above formula 4, so that the calculation of the exploration probability is more accurate and flexible.

[0089] Figure 6 A schematic diagram of the structure of a training device for an intelligent agent in a game is provided. The intelligent agent participates in a cyclic operation of multiple rounds of the game and performs reinforcement learning training during the operation; wherein, the game corresponds to a preset exploration strategy adjustment rule, and the exploration strategy adjustment rule includes: when each round of the game starts running, there is an initial exploration probability corresponding to the initial training step number corresponding to the start of each round of the game, and the initial exploration probability is smaller when the number of steps corresponding to the initial training step number is larger, and the exploration probability is used to characterize the probability of the algorithm choosing to explore new solutions in the reinforcement learning algorithm. Figure 6 As shown, the training device 600 of the intelligent agent in the game includes:

[0090] The first determination module 601 is used to determine the number of initial training steps corresponding to the start of the current round of the game in response to the triggering event of the agent participating in the current round of the game; and determine the initial exploration probability corresponding to the start of the current round of the game;

[0091] A second determination module 602 is used to determine a first exploration probability corresponding to the number of steps in the first training step during the current round of the game according to the exploration strategy adjustment rule, and execute the first training step based on the first exploration probability;

[0092] The running module 603 is used to run the current round of the game to the deadline corresponding to the current round of the game, and repeatedly execute the above steps during the running of multiple games until the training is completed to obtain a trained intelligent agent.

[0093] In the above manner, by adopting the exploration strategy adjustment rule that the larger the number of training steps, the smaller the corresponding initial exploration probability at the beginning of each game during the reinforcement learning training of the intelligent agent, not only does it not affect the exploration strategy during the operation of each game, but more importantly, it makes the initial exploration probability at the beginning of each game in the early stage of training larger, which can promote the intelligent agent to explore different game behaviors and avoid falling into local optimal solutions too early, and also makes the training volume in the early stage of training sufficient. By gradually reducing the initial exploration probability at the beginning of each game as the training progresses, the intelligent agent can retain the learned behavior in the early stage of training. In the late stage of training, the initial exploration probability at the beginning of each game gradually decreases to a smaller value, which can enable the intelligent agent to use the behavior in the late stage of training more attentively for optimization after completing the learning of the early behavior. Through this exploration strategy with dynamic adjustment of the exploration probability, the situation of insufficient exploration in the early stage of training and excessive exploration in the late stage of training is avoided, and a dynamic balance between exploration and utilization in the entire training process is achieved, which improves the reinforcement learning training effect of the intelligent agent and alleviates the technical problem of poor reinforcement learning training effect of the intelligent agent.

[0094] In a feasible implementation scheme, the exploration strategy adjustment rules include: the greater the total number of training steps corresponding to the end of the previous game of the current game, the greater the corresponding end time of the current game; the device also includes: a control module, which is used to control the end of the current game in response to the current running time of the current game reaching the target current game end time corresponding to the current game.

[0095] In a feasible implementation scheme, the end time of the current round of the game is determined based on the time corresponding to the natural end of the game, the preset minimum running game time, and the total number of training steps corresponding to the end of the previous round of the current round of the game.

[0096] In a feasible implementation manner, the total number of training steps executed during the running of each round of the game is determined according to the current round deadline of the game and the number of games running in parallel at the same time.

[0097] In a feasible implementation scheme, the second determination module is specifically used to: determine the target current running time of the current round of the game in response to a triggering event in which the agent participates in the current round of the game; determine the number of steps of the first training step during the running of the current round of the game according to the target current running time; determine the first exploration probability corresponding to the number of steps of the first training step according to the exploration strategy adjustment rule, and execute the first training step based on the first exploration probability and the number of steps of the first training step corresponding to the target current running time.

[0098] In a feasible implementation scheme, the exploration strategy adjustment rules also include: for the training steps whose number of steps corresponding to the initial training steps when the current round of the game starts is greater than the preset number of steps, during the running of the same round of the game, the longer the current running time of the game is, the greater the corresponding first exploration probability is; for the training steps whose number of steps corresponding to the initial training steps when the current round of the game starts is less than or equal to the preset number of steps, the first exploration probability during the running of the same round of the game is the preset maximum probability.

[0099] In a feasible implementation scheme, the first determination module is specifically used to determine the initial exploration probability corresponding to the start of the current game according to the total number of training steps corresponding to the end of the previous game of the current game, the preset minimum game running time, the preset maximum probability, and the preset minimum probability.

[0100] In a feasible implementation scheme, the second determination module is also used to determine the first exploration probability corresponding to the number of steps of the first training step based on the target current running time, the total number of training steps corresponding to the end of the previous game of the current game, the initial exploration probability corresponding to the start of the current game, and the preset minimum running game time.

[0101] The training device for an intelligent agent in a game provided by the embodiment of the present disclosure has the same technical features as the training method for an intelligent agent in a game provided by the above-mentioned embodiment, and therefore can solve the same technical problems and achieve the same technical effects.

[0102] Figure 7 The structure diagram of an electronic device provided by an embodiment of the present disclosure is shown, including: a memory 701, a processor 702 and a bus 703, wherein the memory 701 stores machine-readable instructions executable by the processor 702. When the electronic device runs a method for training an agent in a game in the embodiment, the processor 702 communicates with the memory 701 through the bus 703, and the processor 702 executes the machine-readable instructions, and the processor 702 executes the preamble of the method item to perform the following steps:

[0103] In response to a triggering event in which the agent participates in a current round of the game, determining the number of initial training steps corresponding to when the current round of the game starts running;

[0104] Determine the initial exploration probability corresponding to the start of the current round of the game;

[0105] Determining a first exploration probability corresponding to the number of steps in a first training step during the current round of the game according to the exploration strategy adjustment rule, and executing the first training step based on the first exploration probability;

[0106] Run the current round of the game to the deadline corresponding to the current round of the game, and repeat the above steps during the running of multiple games until the training is completed to obtain a trained intelligent agent.

[0107] In a feasible implementation scheme, the exploration strategy adjustment rules include: the greater the total number of training steps corresponding to the end of the previous game of the current game, the greater the corresponding deadline of the current game; the processor is also used to: in response to the current running time of the current game reaching the target current game deadline corresponding to the current game, control the end of the current game.

[0108] In a feasible implementation scheme, the end time of the current round of the game is determined based on the time corresponding to the natural end of the game, the preset minimum running game time, and the total number of training steps corresponding to the end of the previous round of the current round of the game.

[0109] In a feasible implementation manner, the total number of training steps executed during the running of each round of the game is determined according to the current round deadline of the game and the number of games running in parallel at the same time.

[0110] In a feasible implementation scheme, when the processor determines the first exploration probability corresponding to the number of steps in the first training step during the current game according to the exploration strategy adjustment rule, and executes the first training step based on the first exploration probability, it is specifically used to:

[0111] In response to a triggering event in which the agent participates in a current round of the game, determining a target current running time of the current round of the game;

[0112] Determine the number of steps of the first training step in the current game running process according to the target current running time;

[0113] A first exploration probability corresponding to the number of steps in the first training step is determined according to the exploration strategy adjustment rule, and the first training step is executed based on the first exploration probability and the number of steps in the first training step corresponding to the current running time of the target.

[0114] In a feasible implementation scheme, the exploration strategy adjustment rules also include: for the training steps whose number of steps corresponding to the initial training steps when the current round of the game starts is greater than the preset number of steps, during the running of the same round of the game, the longer the current running time of the game is, the greater the corresponding first exploration probability is; for the training steps whose number of steps corresponding to the initial training steps when the current round of the game starts is less than or equal to the preset number of steps, the first exploration probability during the running of the same round of the game is the preset maximum probability.

[0115] In a feasible implementation scheme, when the processor is executing to determine the initial exploration probability corresponding to the start of the current round of the game, it is specifically used to: determine the initial exploration probability corresponding to the start of the current round of the game based on the total number of training steps corresponding to the end of the previous round of the current round of the game, the preset minimum game running time, the preset maximum probability, and the preset minimum probability.

[0116] In a feasible implementation scheme, when the processor executes the first exploration probability corresponding to the number of steps of the first training step according to the exploration strategy adjustment rule, it is specifically used to: determine the first exploration probability corresponding to the number of steps of the first training step according to the current running time of the target, the total number of training steps corresponding to the end of the previous game of the current game, the initial exploration probability corresponding to the start of the current game, and the preset minimum running game time.

[0117] In the above manner, by adopting the exploration strategy adjustment rule that the larger the number of training steps, the smaller the corresponding initial exploration probability at the beginning of each game during the reinforcement learning training of the intelligent agent, not only does it not affect the exploration strategy during the operation of each game, but more importantly, it makes the initial exploration probability at the beginning of each game in the early stage of training larger, which can promote the intelligent agent to explore different game behaviors and avoid falling into local optimal solutions too early, and also makes the training volume in the early stage of training sufficient. By gradually reducing the initial exploration probability at the beginning of each game as the training progresses, the intelligent agent can retain the learned behavior in the early stage of training. In the late stage of training, the initial exploration probability at the beginning of each game gradually decreases to a smaller value, which can enable the intelligent agent to use the behavior in the late stage of training more attentively for optimization after completing the learning of the early behavior. Through this exploration strategy with dynamic adjustment of the exploration probability, the situation of insufficient exploration in the early stage of training and excessive exploration in the late stage of training is avoided, and a dynamic balance between exploration and utilization in the entire training process is achieved, which improves the reinforcement learning training effect of the intelligent agent and alleviates the technical problem of poor reinforcement learning training effect of the intelligent agent.

[0118] In practical applications, the memory 701 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk storage. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 704 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used.

[0119] The bus 703 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 7 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0120] Among them, the memory 701 is used to store programs, and the processor 702 executes the program after receiving the execution instruction. The method executed by the device defined by the process disclosed in any embodiment of the present disclosure can be applied to the processor 702 or implemented by the processor 702.

[0121] The processor 702 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 702. The above processor 702 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present disclosure can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiments of the present disclosure can be directly embodied as a hardware decoding processor for execution, or a combination of hardware and software modules in the decoding processor for execution. The software module can be located in a storage medium mature in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 701, and the processor 702 reads the information in the memory 701 and completes the steps of the above method in combination with its hardware.

[0122] The present disclosure also provides a computer-readable storage medium, on which a computer program is stored. The computer program is executed when a processor is running, and the processor performs the following steps:

[0123] In response to a triggering event in which the agent participates in a current round of the game, determining the number of initial training steps corresponding to when the current round of the game starts running;

[0124] Determine the initial exploration probability corresponding to the start of the current round of the game;

[0125] Determining a first exploration probability corresponding to the number of steps in a first training step during the current round of the game according to the exploration strategy adjustment rule, and executing the first training step based on the first exploration probability;

[0126] Run the current round of the game to the deadline corresponding to the current round of the game, and repeat the above steps during the running of multiple games until the training is completed to obtain a trained intelligent agent.

[0127] In a feasible implementation scheme, the exploration strategy adjustment rules include: the greater the total number of training steps corresponding to the end of the previous game of the current game, the greater the corresponding deadline of the current game; the processor is also used to: in response to the current running time of the current game reaching the target current game deadline corresponding to the current game, control the end of the current game.

[0128] In a feasible implementation scheme, the end time of the current round of the game is determined based on the time corresponding to the natural end of the game, the preset minimum running game time, and the total number of training steps corresponding to the end of the previous round of the current round of the game.

[0129] In a feasible implementation manner, the total number of training steps executed during the running of each round of the game is determined according to the current round deadline of the game and the number of games running in parallel at the same time.

[0130] In a feasible implementation scheme, when the processor determines the first exploration probability corresponding to the number of steps in the first training step during the current game according to the exploration strategy adjustment rule, and executes the first training step based on the first exploration probability, it is specifically used to:

[0131] In response to a triggering event in which the agent participates in a current round of the game, determining a target current running time of the current round of the game;

[0132] Determine the number of steps of the first training step in the current game running process according to the target current running time;

[0133] A first exploration probability corresponding to the number of steps in the first training step is determined according to the exploration strategy adjustment rule, and the first training step is executed based on the first exploration probability and the number of steps in the first training step corresponding to the current running time of the target.

[0134] In a feasible implementation scheme, the exploration strategy adjustment rules also include: for the training steps whose number of steps corresponding to the initial training steps when the current round of the game starts is greater than the preset number of steps, during the running of the same round of the game, the longer the current running time of the game is, the greater the corresponding first exploration probability is; for the training steps whose number of steps corresponding to the initial training steps when the current round of the game starts is less than or equal to the preset number of steps, the first exploration probability during the running of the same round of the game is the preset maximum probability.

[0135] In a feasible implementation scheme, when the processor is executing to determine the initial exploration probability corresponding to the start of the current round of the game, it is specifically used to: determine the initial exploration probability corresponding to the start of the current round of the game based on the total number of training steps corresponding to the end of the previous round of the current round of the game, the preset minimum game running time, the preset maximum probability, and the preset minimum probability.

[0136] In a feasible implementation scheme, when the processor executes the first exploration probability corresponding to the number of steps of the first training step according to the exploration strategy adjustment rule, it is specifically used to: determine the first exploration probability corresponding to the number of steps of the first training step according to the current running time of the target, the total number of training steps corresponding to the end of the previous game of the current game, the initial exploration probability corresponding to the start of the current game, and the preset minimum running game time.

[0137] In the above manner, by adopting the exploration strategy adjustment rule that the larger the number of training steps, the smaller the corresponding initial exploration probability at the beginning of each game during the reinforcement learning training of the intelligent agent, not only does it not affect the exploration strategy during the operation of each game, but more importantly, it makes the initial exploration probability at the beginning of each game in the early stage of training larger, which can promote the intelligent agent to explore different game behaviors and avoid falling into local optimal solutions too early, and also makes the training volume in the early stage of training sufficient. By gradually reducing the initial exploration probability at the beginning of each game as the training progresses, the intelligent agent can retain the learned behavior in the early stage of training. In the late stage of training, the initial exploration probability at the beginning of each game gradually decreases to a smaller value, which can enable the intelligent agent to use the behavior in the late stage of training more attentively for optimization after completing the learning of the early behavior. Through this exploration strategy with dynamic adjustment of the exploration probability, the situation of insufficient exploration in the early stage of training and excessive exploration in the late stage of training is avoided, and a dynamic balance between exploration and utilization in the entire training process is achieved, which improves the reinforcement learning training effect of the intelligent agent and alleviates the technical problem of poor reinforcement learning training effect of the intelligent agent.

[0138] In the embodiments of the present disclosure, the computer program can also execute other machine-readable instructions when run by the processor to execute other methods described in the embodiments. For the specific execution method steps and principles, please refer to the description of the embodiments, which will not be repeated here.

[0139] In the embodiments provided in the present disclosure, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0140] For another example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0141] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0142] In addition, each functional unit in the embodiments provided in the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0143] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present disclosure is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the training method of the intelligent agent in the game described in each embodiment of the present disclosure. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store program codes.

[0144] It should be noted that similar numbers and letters represent similar items in the following figures. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are only used to distinguish the description and are not to be understood as indicating or implying relative importance.

[0145] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present disclosure, which are used to illustrate the technical solutions of the present disclosure, rather than to limit them. The protection scope of the present disclosure is not limited thereto. Although the present disclosure is described in detail with reference to the aforementioned embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the aforementioned embodiments within the technical scope disclosed in the present disclosure, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present disclosure. They should all be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure shall be based on the protection scope of the claims.

Claims

1. A method for training an intelligent agent in a game, characterized in that: The agent participates in a cyclic operation of multiple games and performs reinforcement learning training during the operation; wherein the game corresponds to a preset exploration strategy adjustment rule, and the exploration strategy adjustment rule includes: when each game of the game starts running, there is an initial exploration probability corresponding to the initial training step number corresponding to the start of each game of the game, and the initial exploration probability is smaller when the number of steps corresponding to the initial training step is larger, and the exploration probability is used to characterize the probability of the algorithm choosing to explore a new solution in the reinforcement learning algorithm; the method includes: In response to a triggering event in which the agent participates in a current round of the game, determining the number of initial training steps corresponding to when the current round of the game starts running; Determine the initial exploration probability corresponding to the start of the current round of the game; Determining a first exploration probability corresponding to the number of steps in a first training step during the current round of the game according to the exploration strategy adjustment rule, and executing the first training step based on the first exploration probability; Run the current round of the game to the deadline corresponding to the current round of the game, and repeat the above steps during the running of multiple games until the training is completed to obtain a trained intelligent agent.

2. The method according to claim 1, characterized in that The exploration strategy adjustment rule includes: the greater the total number of training steps corresponding to the end of the previous round of the current round of the game, the greater the deadline of the current round of the corresponding game; the method also includes: In response to the current running time of the current round of the game reaching the target current round end time corresponding to the current round of the game, the running of the current round of the game is controlled to end.

3. The method according to claim 2, characterized in that The end time of the current round of the game is determined based on the time corresponding to the natural end of the game, the preset minimum running game time, and the total number of training steps corresponding to the end of the previous round of the current round of the game.

4. The method according to claim 2, characterized in that: The total number of training steps executed during the running of each round of the game is determined according to the current round deadline of the game and the number of games running in parallel at the same time.

5. The method according to claim 2, characterized in that: The step of determining, according to the exploration strategy adjustment rule, a first exploration probability corresponding to the number of steps in the first training step during the current game, and executing the first training step based on the first exploration probability includes: In response to a triggering event in which the agent participates in a current round of the game, determining a target current running time of the current round of the game; Determine the number of steps of the first training step in the current game running process according to the target current running time; A first exploration probability corresponding to the number of steps in the first training step is determined according to the exploration strategy adjustment rule, and the first training step is executed based on the first exploration probability and the number of steps in the first training step corresponding to the current running time of the target.

6. The method according to claim 5, characterized in that The exploration strategy adjustment rules also include: For a training step in which the number of steps of the initial training step corresponding to the start of the current round of the game is greater than the preset number of steps, during the running of the same round of the game, the longer the current running time of the game is, the greater the corresponding first exploration probability is; For the training steps in which the number of steps of the initial training steps corresponding to the start of the current round of the game is less than or equal to the preset number of steps, the first exploration probability during the running process of the same round of the game is the preset maximum probability.

7. The method according to claim 1, characterized in that Determining the initial exploration probability corresponding to the start of the current round of the game includes: The initial exploration probability corresponding to the start of the current game is determined based on the total number of training steps corresponding to the end of the previous game of the current game, the preset minimum game running time, the preset maximum probability, and the preset minimum probability.

8. The method according to claim 5, characterized in that The first exploration probability corresponding to the number of steps in the first training step is determined according to the exploration strategy adjustment rule, including: determining the first exploration probability corresponding to the number of steps in the first training step according to the current running time of the target, the total number of training steps corresponding to the end of the previous game of the current game, the initial exploration probability corresponding to the start of the current game, and a preset minimum running game time.

9. A training device for an intelligent agent in a game, characterized in that: The intelligent agent participates in a cyclic operation of multiple rounds of the game and performs reinforcement learning training during the operation; wherein the game corresponds to a preset exploration strategy adjustment rule, and the exploration strategy adjustment rule includes: when each round of the game starts running, there is an initial exploration probability corresponding to the initial training step at the beginning of each round of the game, and the larger the number of steps corresponding to the initial training step at the beginning of each round of the game, the smaller the initial exploration probability, and the exploration probability is used to characterize the probability of the algorithm choosing to explore a new solution in the reinforcement learning algorithm; the device includes: A first determination module is used to determine the number of initial training steps corresponding to the start of the current round of the game in response to a triggering event in which the agent participates in the current round of the game; and to determine the initial exploration probability corresponding to the start of the current round of the game; A second determination module is used to determine a first exploration probability corresponding to the number of steps of a first training step in the current game according to the exploration strategy adjustment rule, and execute the first training step based on the first exploration probability; The running module is used to run the current round of the game to the deadline corresponding to the current round of the game, and repeatedly execute the above steps during the running of multiple games until the training is completed to obtain a trained intelligent agent.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to execute the method according to any one of claims 1 to 8.

Images