Training method, device and electronic device for generator network and strategy generation network

Abstract

Disclosed are a training method for a generator network, a training method for a policy generation network for reinforcement learning, a device, and an electronic device. The training method for the generator network generates a layer of a prediction network through each network unit of the generator network, and updates the parameters of the generator network based on the KL divergence value between the probability distribution of the state vector predicted by the prediction network and the true state vector. And the training method for the policy generation network for reinforcement learning calculates the intrinsic reward function value of the policy generation network for reinforcement learning based on the difference between multiple state vectors predicted by multiple prediction networks generated by the generator network, so as to train the policy generation network. In this way, the performance of the generator network and the exploration efficiency of the policy generation network are improved.

Description

Technical Field

[0001] The present disclosure relates to the technical field of reinforcement learning, and more specifically, to a training method of a generator network, a training method, a device and an electronic device for a strategy generation network for reinforcement learning. Background Art

[0002] Recently, reinforcement learning (RL) has achieved success in many application areas, including super-human performance in various games, as well as excellent performance in robotic control tasks and image-based control tasks.

[0003] However, despite these achievements, current reinforcement learning techniques suffer from poor sampling efficiency, often requiring millions or even billions of simulation steps of training before achieving reasonable performance for real-world objects performing the task. As a result, this lack of statistical efficiency makes reinforcement learning difficult to apply to real-world tasks, where the cost of the actions of the objects performing the task in the real world is much higher than the cost of the simulated actions of the objects performing the task in the simulator.

[0004] That is, in current practical reinforcement learning tasks, the training scheme of the policy generation network used to generate the actions of the object performing the task needs to be improved to improve the effectiveness of the actions of the object performing the task. Summary of the invention

[0005] In order to solve the above technical problems, the present disclosure is proposed. The embodiments of the present disclosure provide a training method for a generator network, a training method for a policy generation network for reinforcement learning, a device and an electronic device, which updates the parameters of the generator network for generating a prediction network through the KL divergence value between the probability distribution of the predicted state vector and the probability distribution of the real state vector, and calculates the intrinsic reward function value of the policy generation network for reinforcement learning based on the difference between the multiple state vectors predicted by the multiple prediction networks generated by the generator network. Since the intrinsic reward function value can reflect the uncertainty of the cognition of the policy generation network relative to the dynamic environment state, the policy generation network can be trained by it, which can promote the exploration of the environment by the policy generation network, thereby improving the exploration efficiency of the policy generation network.

[0006] According to one aspect of the present disclosure, a training method for a generator network is provided, comprising: obtaining a training current state vector, a training action vector, and a training next state vector corresponding to the training current state vector and the training action vector for a reinforcement learning task, and a true posterior probability distribution determined thereby; inputting a set of random noise vectors of known probability distribution into the generator network to obtain a set of prediction networks, wherein the generator network comprises a plurality of network units, each of which is used to generate a layer of the prediction network; inputting the training current state vector and the training action vector into the set of prediction networks to obtain a predictive probability distribution; determining a KL divergence value between the predictive probability distribution and the true posterior probability distribution; and updating the parameters of the generator network based on the KL divergence value.

[0007] According to another aspect of the present disclosure, a training method for a policy generation network for reinforcement learning is provided, comprising: obtaining a generator network trained by the training method for the generator network as described above; generating N prediction networks by the generator network; obtaining a current state vector and an action vector generated by the policy generation network; inputting the current state vector and the action vector into the N prediction networks to obtain N next state vectors; calculating an intrinsic reward function value for reinforcement learning based on the difference between the N next state vectors; and updating parameters of the policy generation network based on the intrinsic reward function value.

[0008] According to another aspect of the present disclosure, a training device for a generator network is provided, comprising: a training vector acquisition unit, for acquiring a training current state vector, a training action vector, and a training next state vector corresponding to the training current state vector and the training action vector for a reinforcement learning task, and a true posterior probability distribution determined thereby; a prediction network generation unit, for inputting a group of random noise vectors of known probability distribution into the generator network to obtain a group of prediction networks, wherein the generator network comprises a plurality of network units, each of which is used to generate a layer of the prediction network; a vector prediction unit, for inputting the training current state vector and the training action vector acquired by the training vector acquisition unit into the group of prediction networks generated by the prediction network generation unit to obtain a predictive probability distribution; a divergence value determination unit, for determining a KL divergence value between the predictive probability distribution acquired by the vector prediction unit and the true posterior probability distribution acquired by the training vector acquisition unit; and a generator update unit, for updating the parameters of the generator network based on the KL divergence value determined by the divergence value determination unit.

[0009] According to another aspect of the present disclosure, a training device for a policy generation network for reinforcement learning is provided, comprising: a network acquisition unit, for acquiring a generator network trained by the training device for the generator network as described above; a network generation unit, for generating N prediction networks from the generator network acquired by the network acquisition unit; a vector acquisition unit, for acquiring a current state vector and an action vector generated by the policy generation network; a prediction vector acquisition unit, for inputting the current state vector and the action vector acquired by the vector acquisition unit into the N prediction networks generated by the network generation unit to obtain N next state vectors; a reward function calculation unit, for calculating an intrinsic reward function value for reinforcement learning based on the difference between the N next state vectors obtained by the prediction vector acquisition unit; and a network update unit, for updating the parameters of the policy generation network based on the intrinsic reward function value calculated by the reward function calculation unit.

[0010] According to another aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory, in which computer program instructions are stored, and when the computer program instructions are executed by the processor, the processor executes the training method of the generator network as described above or the training method of the policy generation network for reinforcement learning as described above.

[0011] According to another aspect of the present disclosure, a computer-readable medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the processor executes the training method of the generator network as described above or the training method of the policy generation network for reinforcement learning as described above.

[0012] The training method, device and electronic device of the generator network provided by the embodiments of the present disclosure generate a layer of the prediction network through each network unit of the generator network, and update the parameters of the generator network based on the KL divergence value between the probability distribution of the state vector predicted by the prediction network and the true posterior probability distribution. A prediction network of any number of layers can be generated by changing the number of network units used, thereby enhancing the flexibility of the generator network. Moreover, based on the KL divergence value, the parameters of the generator network can be directly updated without updating the parameters of the prediction network, thereby simplifying the training process of the generator network.

[0013] In addition, the training method, device and electronic device of the policy generation network for reinforcement learning provided by the embodiments of the present disclosure calculate the intrinsic reward function value of the policy generation network for reinforcement learning based on the difference between the state vectors predicted by multiple prediction networks generated by the generator network, that is, based on the Bayesian uncertainty between the multiple prediction networks, to train the policy generation network. Since the intrinsic reward function value can reflect the uncertainty of the cognition of the policy generation network relative to the dynamic environment state, the policy generation network can be trained through its training, which can promote the exploration of the environment by the policy generation network in the process of reinforcement learning, so that the policy generation network can explore more effectively in different environments, thereby improving the exploration efficiency of the trained policy generation network. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The above and other purposes, features and advantages of the present disclosure will become more apparent by describing the embodiments of the present disclosure in more detail in conjunction with the accompanying drawings. The accompanying drawings are used to provide a further understanding of the embodiments of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the present disclosure and do not constitute a limitation of the present disclosure. In the drawings, the same reference numerals generally represent the same components or steps.

[0015] Figure 1 A schematic diagram of a standard reinforcement learning model is shown.

[0016] Figure 2 A flowchart of a method for training a generator network according to an embodiment of the present disclosure is illustrated.

[0017] Figure 3 A schematic diagram illustrating the architecture of a generator network according to an embodiment of the present disclosure is illustrated.

[0018] Figure 4 The flowchart of the training method of the policy generation network for reinforcement learning according to the embodiment of the present disclosure is illustrated.

[0019] Figure 5 A block diagram of a training apparatus for a generator network according to an embodiment of the present disclosure is illustrated.

[0020] Figure 6 A block diagram of a training device for a policy generation network for reinforcement learning according to an embodiment of the present disclosure is illustrated.

[0021] Figure 7 A block diagram of an electronic device according to an embodiment of the present disclosure is illustrated. DETAILED DESCRIPTION

[0022] Below, the exemplary embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all the embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described here.

[0023] Application Overview

[0024] Figure 1 The figure shows a schematic diagram of a standard reinforcement learning model. Figure 1 As shown, the policy generation network N generates action A, the current state S0 of the environment migrates to the next state S1 of the environment based on action A, and p is used to represent the migration probability from the current state to the next state. In addition, the reward function r is input to the policy generation network N, which is used by the policy generation network N to update its policy for generating action A. For example, the general purpose is to maximize the cumulative value of the reward function.

[0025] In reinforcement learning, in order to build a strategy that can effectively generate actions, one aspect is to effectively use the collected data, including the state data and action data of the environment. This usually includes three key aspects: generalization, exploration, and long-term consequence awareness.

[0026] Here, the object (agent) performing the task performs actions through the strategy generated by the strategy generation network. Therefore, the exploration of the environment can be called the exploration of the environment by the object performing the task, the exploration of the environment by the strategy, or the exploration of the environment by the strategy generation network. In addition, since the exploration of the environment by the object performing the task is essentially carried out by executing the strategy generated by the strategy generation network, the ultimate goal is to improve the exploration performance of the strategy generation network.

[0027] In the field of reinforcement learning, exploration refers to obtaining more information about the environment. In the standard reinforcement learning model described above, the policy generation network receives a reward function r for each migration step of the environment state. In many cases, the reward function r is an external reward function. However, in many task situations, there are environments where external rewards are sparse or almost negligible. In this case, it is necessary to drive the exploration of the policy generation network with some type of intrinsic reward, which can be considered a pure exploration problem that effectively accumulates information about the environment.

[0028] Therefore, the basic concept of the present disclosure is to promote the effective exploration of the policy generation network of the reinforcement learning model through the estimation of Bayesian uncertainty, where the estimated value of Bayesian uncertainty can characterize the uncertainty of the cognition of the policy generation network relative to the dynamic environment state in a non-parametric manner. Specifically, Bayesian uncertainty can be expressed as the uncertainty of the next state predicted by different prediction networks from the current state and the current action. In this way, by introducing the estimated value of Bayesian uncertainty for the intrinsic reward value in reinforcement learning, the exploration of the policy generation network based on the intrinsic reward value in reinforcement learning can be promoted, and the exploration performance of the policy generation network can be improved.

[0029] Furthermore, in the present disclosure, in order to estimate Bayesian uncertainty, it is necessary to generate multiple prediction models that predict the next state from the current state and action. Therefore, in the training method of the policy generation network for reinforcement learning of the present disclosure, a generator network is used to obtain N prediction networks, and the N next states are predicted from the current state and the current action through the N prediction networks to calculate the differences between the N predicted next states, that is, the estimated value of Bayesian uncertainty.

[0030] Specifically, the training method, device and electronic device of the generator network provided by the present disclosure first obtain a training current state vector, a training action vector and a training next state vector corresponding to the training current state vector and the training action vector for a reinforcement learning task, as well as a true posterior probability distribution determined thereby; then a set of random noise vectors of known probability distribution are input into the generator network to obtain a set of prediction networks, wherein the generator network includes a plurality of network units, each of which is used to generate a layer of the prediction network; then the training current state vector and the training action vector are input into the set of prediction networks to obtain a predictive probability distribution; then the KL divergence value between the predictive probability distribution and the true posterior probability distribution is determined; and finally the parameters of the generator network are updated based on the KL divergence value.

[0031] In this way, the training method of the generator network provided by the present disclosure can make the generator network more flexible, that is, by generating one layer of the prediction network through the network unit, a prediction network of any number of layers can be generated. In addition, when updating the generator network, only the parameters of the generator network need to be updated, and the parameters of the prediction network generated by the generator network do not need to be updated, which can simplify the training process of the generator network.

[0032] On the other hand, the training method, device and electronic device of the policy generation network for reinforcement learning provided by the present disclosure first obtain the generator network trained by the training method of the generator network as described above; generate N prediction networks by the generator network, and then obtain the current state vector and the action vector generated by the policy generation network, and then input the current state vector and the action vector into the N prediction networks to obtain N next state vectors, and then calculate the intrinsic reward function value for reinforcement learning based on the difference between the N next state vectors, and finally update the parameters of the policy generation network based on the intrinsic reward function value.

[0033] In this way, the training method, device and electronic device of the policy generation network for reinforcement learning provided by the present disclosure calculate the intrinsic reward function value of the policy generation network for reinforcement learning based on the difference between the state vectors predicted by multiple prediction networks generated by the generator network, that is, based on the Bayesian uncertainty between the multiple prediction networks, to train the policy generation network. Since the intrinsic reward function value can reflect the uncertainty of the cognition of the policy generation network relative to the dynamic environment state, the policy generation network can be trained through its training, which can promote the exploration of the environment by the policy generation network in the process of reinforcement learning, so that the policy generation network can explore more effectively in different environments, thereby improving the exploration efficiency of the trained policy generation network.

[0034] After introducing the basic principles of the present disclosure, various non-limiting embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0035] Exemplary Methods

[0036] Figure 2 A flowchart of a method for training a generator network according to an embodiment of the present disclosure is illustrated.

[0037] like Figure 2 As shown, the training method of the generator network according to an embodiment of the present disclosure includes the following steps.

[0038] Step S110, obtaining a training current state vector, a training action vector, and a training next state vector corresponding to the training current state vector and the training action vector for a reinforcement learning task, as well as a true posterior probability distribution determined thereby.

[0039] As mentioned above, for reinforcement learning tasks, the current state of the environment is transferred to the next state through the action of the object that performs the action (usually called Agent in reinforcement learning tasks). For example, in maze exploration tasks, the action refers to the moving direction of the object exploring the maze, and the state can be expressed as the percentage of the explored part of the maze to the entire part. In addition, in Go tasks, the action refers to the position of the chess piece, that is, the position coordinates in the 81×81 grid coordinates of the chessboard, and the state can be the statistical probability of winning under the current state of the chessboard.

[0040] In addition, for the state and action, they are converted into a state vector and an action vector respectively. In the embodiment of the present disclosure, the current state, action and next state are obtained and converted into vectors for training the generator network by actually acting on the environment.

[0041] Here, the true posterior probability distribution is equal to the prior probability distribution multiplied by the likelihood function. In the disclosed embodiment, because the prior probability distribution is assumed to be uniform, it can be ignored, and based on the real training data, the logarithm of the likelihood function can be designed as the loss function of the prediction result. Therefore, the true posterior probability distribution can be obtained based on the real training data and the designed loss function through a general method. In the disclosed embodiment, the true posterior probability distribution can be denoted as p(f|D), and can be simply denoted as p, where D represents the given current state vector and action vector.

[0042] Step S120, inputting a set of random noise vectors of known probability distribution into a generator network to obtain a set of prediction networks, wherein the generator network includes a plurality of network units, each of which is used to generate a layer of the prediction network.

[0043] Figure 3 A schematic diagram of the architecture of a generator network according to an embodiment of the present disclosure is illustrated. Figure 3 As shown, the generator network according to the embodiment of the present disclosure includes multiple network units, such as Figure 3 G1, G2, ..., Gn shown. Independent noise samples of a random noise vector, such as Figure 3 Noise samples Z1, Z2, ..., Zn of standard Gaussian noise Z with diagonal covariance as shown are input into each network unit to generate parameter vectors of each layer of the prediction network, such as Figure 3 θ1, θ2, …, θn are shown.

[0044] For example, in the embodiment of the present disclosure, the generator network may include 4 network units, thereby generating a prediction network including a 4-layer network. Moreover, each network unit of the generator network may be a fully connected neural network.

[0045] Thus, as mentioned above, Figure 3 The advantages of the architecture of the generator network shown are its flexibility and high efficiency, because it only needs to maintain the parameters of multiple network units, and can generate prediction networks of any number of layers by increasing or decreasing the number of network units used. In addition, by inputting a random noise vector, any number of prediction networks can be generated for the training of the policy generation network to be described below.

[0046] Furthermore, in the disclosed embodiment, after a known probability distribution, such as a set of random noise vectors of a standard normal distribution, is input into the generator network, the function corresponding to the generated prediction network will present a probability distribution transformed by the function corresponding to the generator network.

[0047] It is worth noting that in the embodiment of the present disclosure, the set of random noise vectors are independently generated from a probability distribution of a predetermined dimension, such as a d-dimensional standard normal distribution, rather than being jointly generated.

[0048] This is because if multiple random noise vectors are jointly generated, the correlation between the random noise vectors needs to be considered. In the generator network according to the embodiment of the present disclosure, it is hoped that the initial input is as simple as possible, and the possible correlation is left to the generator network to learn, which can improve the performance of the generator network.

[0049] In the embodiment of the present disclosure, the generator network can be represented as, and its parameter can be represented as η. And the prediction network can be represented as f θ , and its parameter can be expressed as θ.

[0050] Step S130, input the training current state vector and the training action vector into the set of prediction networks to obtain a predictive probability distribution. That is, each prediction network in the set of prediction networks predicts the next state from the current state and action, that is, obtains a predictive next state vector from the training current state vector and the training action vector. Therefore, by inputting the training current state vector and the training action vector, the probability distribution of a set of predicted vectors can be obtained by a function corresponding to a set of prediction networks that satisfies the transformed probability distribution, that is, a predictive probability distribution. In the embodiment of the present disclosure, the prediction network is represented as θ In the case of θ ), and can be simply recorded as q.

[0051] Step S140, determining the KL divergence value between the predictive probability distribution and the true posterior probability distribution. Here, in order to make the predictive probability distribution as close as possible to the true posterior probability distribution, this proximity is measured by using the KL divergence value. Therefore, in the disclosed embodiment, the KL divergence value between the predictive probability distribution and the true posterior probability distribution is determined.

[0052] Step S150, updating the parameters of the generator network based on the KL divergence value. Specifically, in order to make the predictive probability distribution obtained by the prediction network generated by the generator network as close as possible to the true posterior probability distribution, the parameters of the generator network are updated by minimizing the KL divergence value. For example, a commonly used variational inference method can be used to minimize the KL divergence value between the predictive probability distribution and the true posterior probability distribution.

[0053] In this way, the training method of the generator network according to the embodiment of the present disclosure can only update the parameters of the generator network during the training of the generator network, without updating the parameters of the prediction network generated by the generator network, thereby simplifying the training process of the generator network.

[0054] In addition, in one example, in the presently disclosed embodiment, the Stein Variational Gradient Descent method can be used. This method is a non-parametric variational inference method that represents the network to be trained as a set of particles without making any parametric assumptions, thereby obtaining the function gradient descent value through iterative particle evolution.

[0055] Therefore, in the training method of the generator network according to an embodiment of the present disclosure, updating the parameters of the generator network based on the KL divergence value includes: using the Stern variational gradient descent method to calculate the function gradient descent value relative to the KL divergence value; and, updating the parameters of the generator network based on the function gradient descent value.

[0056] In this way, through the Stern variational gradient descent method, the function gradient descent value relative to the KL divergence value can be calculated in a non-parametric manner to update the parameters of the generator network, making the calculation simple.

[0057] Specifically, using the Stern variational gradient descent method, the function gradient needs to be projected into the parameter space of the generator network through back propagation through the generator network. For a set of dynamic functions of the prediction network generated by the generator network G, Stern variational gradient descent updates the prediction network using the following formula:

[0058]

[0059] where ∈ is the step size, and φ * is the reproducing kernel Hilbert space (RKHS) The function of the unit ball, which minimizes the prediction network The KL divergence value between the predictive probability distribution q and the true posterior probability distribution p, that is, D KL (q||p). Specifically, the function is expressed by the following formula:

[0060]

[0061] This optimization problem has a solution of the approximate form:

[0062]

[0063] in, represents the expectation about f, and f obeys the probability distribution q. Moreover, the function k(·,·) is a positive finite kernel associated with RKHS. In the embodiment of the present disclosure, a Gaussian kernel can be used, that is, the prediction network generated by calculating two adjacent times The distance between them. log p(f) corresponds to the negative of the recursive loss function for the future state prediction of all state transitions in space D, that is,

[0064]

[0065] Among them, s represents the current state, a represents the action, and s′ represents the next state.

[0066] In this way, since the prediction network is determined by the parameters of the generator network, if the prediction network The parameter is defined as θ i , then:

[0067] θ i ←θ i +∈φ * (θ i )

[0068] in,

[0069]

[0070] in, Represents the expectation about θ, and θ obeys the probability distribution G. And, the parameter θ of the prediction network i Generated by the generator network G, so the update rule of the parameter η of the generator network can also be obtained by the chain rule:

[0071]

[0072] Among them, φ * (θ i ) is calculated by using the empirical expectation of the sample:

[0073]

[0074] That is, in the training method of the generator network according to the embodiment of the present disclosure, the function gradient descent value of calculating the KL divergence value using the Stern variational gradient descent method includes: calculating a predetermined function of each prediction network generated by the generator network to reproduce the unit ball of the kernel Hilbert space; calculating the gradient of the product of the generator network and the predetermined function with respect to the parameters of the generator network; and, summing the gradient for the set of prediction networks to obtain the gradient sum; and, updating the parameters of the generator network based on the function gradient descent value includes: obtaining the parameters of the updated generator network based on the parameters of the current generator network, the gradient sum and the first coefficient.

[0075] Furthermore, in the training method of the generator network according to the embodiment of the present disclosure, calculating the predetermined function of each prediction network to reproduce the unit ball of the kernel Hilbert space includes: calculating the gradient of the difference function value between the next state predicted by the prediction network from the current state and action and the actual next state with respect to the prediction network; summing the gradient with respect to all states and actions in the state space and action space and multiplying it by a kernel function to obtain a kernel function product, wherein the kernel function is used to calculate the distance between two adjacent generated prediction networks; and subtracting the kernel function product from the gradient of the kernel function with respect to the prediction network and summing the result with respect to the group of prediction networks to obtain the predetermined function.

[0076] In this way, after the generator network is fully trained, the policy generation network for reinforcement learning can be trained by using the prediction network generated by the generator network. That is, multiple prediction networks are generated by the generator network, and the differences between the state vectors predicted by the multiple prediction networks, that is, the intrinsic reward function value of the policy generation network for reinforcement learning is calculated based on the Bayesian uncertainty between the multiple prediction networks to train the policy generation network. Since the intrinsic reward function value can reflect the uncertainty of the cognition of the policy generation network relative to the dynamic environment state, the policy generation network can be trained through its training. In the process of reinforcement learning, the exploration of the environment by the policy generation network can be promoted, so that the policy generation network can explore more effectively in different environments, thereby improving the exploration efficiency of the trained policy generation network.

[0077] Figure 4 The flowchart of the training method of the policy generation network for reinforcement learning according to the embodiment of the present disclosure is illustrated.

[0078] like Figure 4 As shown, the training method of the strategy generation network for reinforcement learning according to the embodiment of the present disclosure includes the following steps.

[0079] S210, obtaining a generator network trained by the generator network training method as described above, that is, obtaining a trained generator network.

[0080] S220, the generator network generates N prediction networks. In one example, the generator network can be used to generate 32 prediction networks. Here, due to the random noise vector input by the generator network, the parameters of the generated 32 prediction networks are also different, so the generated N prediction networks can be called dynamic prediction networks.

[0081] S230, obtaining the current state vector and the action vector generated by the strategy generation network. Figure 1 As shown, the vector of the current state S0 and the vector of the action A generated by the strategy generation network N are obtained.

[0082] S240, input the current state vector and the action vector into the N prediction networks to obtain N next state vectors. That is, when the current state vector and the action vector are input into N prediction networks with different parameters, the obtained N next state vectors are also different, which can be called the Bayesian uncertainty of the dynamic prediction network.

[0083] S250, calculating an intrinsic reward function value for reinforcement learning based on the differences between the N next state vectors. That is, using the Bayesian uncertainty of the dynamic prediction network to calculate the intrinsic reward function value for the reinforcement learning task.

[0084] S260, updating the parameters of the policy generation network based on the intrinsic reward function value. In the disclosed embodiment, a reinforcement learning algorithm not constrained by a model may be used to update the parameters of the policy generation network, for example, a soft actor critic algorithm may be used.

[0085] In this way, the intrinsic reward function value of the policy generation network used for reinforcement learning is calculated based on the difference between the state vectors predicted by the multiple prediction networks generated by the generator network, that is, based on the Bayesian uncertainty between the multiple prediction networks, to train the policy generation network. In the process of reinforcement learning, it can be promoted that the object performing the task explores the environment based on the intrinsic reward by using the strategy generated by the policy generation network, so that it can explore more effectively in different environments, thereby improving the performance of the trained policy generation network.

[0086] In one example, the intrinsic reward function value can be calculated by the following formula

[0087]

[0088] Among them, s t represents the current state vector, a t represents the action vector, and It represents the next vector predicted by the prediction network from the current state vector and action vector, and m is the number of prediction networks.

[0089] Therefore, in the training method of the policy generation network for reinforcement learning according to the embodiment of the present disclosure, calculating the intrinsic reward function value for reinforcement learning based on the N next state vectors includes: calculating the mean vector of the N next state vectors; calculating the L2 distance value between each next state vector of the N next state vectors and the mean vector to obtain N L2 distance values; and, calculating the mean of the N L2 distance values ​​to obtain the reward function value for reinforcement learning.

[0090] In addition, in the embodiments of the present disclosure, in addition to the next state vector predicted by the prediction network from the current state vector and the action vector, the next state vector may also include a real next state vector obtained by the real object performing the task in the reinforcement learning task based on the current state vector and the action vector.

[0091] That is, the state vector and action vector can be collected from the actual environment or simulated by the dynamic prediction network generated by the generator network, because in the embodiment of the present disclosure, the Bayesian uncertainty of the dynamic prediction network is used to calculate the intrinsic reward function value for the reinforcement learning task to reflect the uncertainty of the policy generation network's cognition relative to the dynamic environment state. In this way, training the policy generation network through the intrinsic reward function value can promote the exploration of the environment by the policy generation network during the reinforcement learning process, so that the policy generation network can explore more effectively in different environments, thereby improving the exploration efficiency of the trained policy generation network.

[0092] Exemplary Devices

[0093] Figure 5 A block diagram of a training apparatus for a generator network according to an embodiment of the present disclosure is illustrated.

[0094] like Figure 5As shown, the training device 300 of the generator network according to the embodiment of the present disclosure includes: a training vector acquisition unit 310, which is used to acquire a training current state vector, a training action vector, and a training next state vector corresponding to the training current state vector and the training action vector for a reinforcement learning task, and a true posterior probability distribution determined thereby; a prediction network generation unit 320, which is used to input a group of random noise vectors of known probability distribution into the generator network to obtain a group of prediction networks, wherein the generator network includes a plurality of network units, each of which is used to generate a layer of the prediction network; a vector prediction unit 330 , used to input the training current state vector and the training action vector obtained by the training vector acquisition unit 310 into the set of prediction networks generated by the prediction network generation unit 320 to obtain a predictive probability distribution; a divergence value determination unit 340, used to determine the KL divergence value between the predictive probability distribution obtained by the vector prediction unit 330 and the true posterior probability distribution obtained by the training vector acquisition unit 310; and a generator update unit 350, used to update the parameters of the generator network based on the KL divergence value determined by the divergence value determination unit 340.

[0095] In one example, in the training device 300 of the above-mentioned generator network, the generator update unit 350 includes: a gradient calculation subunit, used to calculate the function gradient descent value of the KL divergence value determined by the divergence value determination unit 340 using the Stern variational gradient descent method; and a parameter updating subunit, used to update the parameters of the generator network based on the function gradient descent value calculated by the gradient calculation subunit.

[0096] In one example, in the training device 300 of the above-mentioned generator network, the gradient calculation subunit is used to: calculate a predetermined function of each prediction network generated by the generator network to reproduce the unit ball of the kernel Hilbert space; calculate the gradient of the product of the generator network and the predetermined function with respect to the parameters of the generator network; and, sum the gradient for the set of prediction networks to obtain the gradient sum; and, the parameter updating subunit is used to: obtain the parameters of the updated generator network based on the parameters of the current generator network, the gradient sum and the first coefficient.

[0097] In one example, in the training device 300 of the generator network mentioned above, the gradient calculation subunit calculates the predetermined function of each prediction network to reproduce the unit ball of the kernel Hilbert space, including: calculating the gradient of the difference function value between the next state predicted by the prediction network from the current state and action and the actual next state with respect to the prediction network; summing the gradient with respect to all states and actions in the state space and action space and multiplying it by the kernel function to obtain the kernel function product, wherein the kernel function is used to calculate the distance between two adjacent generated prediction networks; and subtracting the kernel function product from the gradient of the kernel function with respect to the prediction network and summing it with respect to the group of prediction networks to obtain the predetermined function.

[0098] In one example, in the above-mentioned generator network training device 300, the prediction network generation unit 320 is used to: for each prediction network, input an independent noise sample obtained from standard Gaussian noise with diagonal covariance into each network unit to generate a layer of the prediction network.

[0099] Figure 6 A block diagram of a training device for a policy generation network for reinforcement learning according to an embodiment of the present disclosure is illustrated.

[0100] like Figure 6 As shown, according to an embodiment of the present disclosure, a training device 400 for a policy generation network for reinforcement learning includes: a network acquisition unit 410, used to acquire a generator network trained by the training device 300 for the generator network as described above; a network generation unit 420, used to generate N prediction networks from the generator network acquired by the network acquisition unit 410; a vector acquisition unit 430, used to acquire a current state vector and an action vector generated by the policy generation network; a prediction vector acquisition unit 440, used to input the current state vector and the action vector acquired by the vector acquisition unit 430 into the N prediction networks generated by the network generation unit 420 to obtain N next state vectors; a reward function calculation unit 450, used to calculate an intrinsic reward function value for reinforcement learning based on the difference between the N next state vectors obtained by the prediction vector acquisition unit 440; and a network update unit 460, used to update the parameters of the policy generation network based on the intrinsic reward function value calculated by the reward function calculation unit 450.

[0101] In one example, in the above-mentioned training device 400 for the policy generation network for reinforcement learning, the reward function calculation unit 450 is used to: calculate the mean vector of the N next state vectors; calculate the L2 distance value between each of the N next state vectors and the mean vector to obtain N L2 distance values; and, calculate the mean of the N L2 distance values ​​to obtain the reward function value for reinforcement learning.

[0102] In one example, in the training device 400 for the policy generation network for reinforcement learning, the next state vector includes a real next state vector obtained by a real object performing the task in the reinforcement learning task based on the current state vector and the action vector.

[0103] Here, those skilled in the art will appreciate that the specific functions and operations of the various units and modules in the above-mentioned generator network training device 300 and the strategy generation network training device 400 for reinforcement learning have been described in the above reference. Figures 1 to 4 The training method of the generator network and the training method of the policy generation network for reinforcement learning are described in detail, and therefore, their repeated description will be omitted.

[0104] As described above, the training device 300 of the generator network and the training device 400 of the policy generation network for reinforcement learning according to the embodiment of the present disclosure can be implemented in various terminal devices, such as a server for reinforcement learning tasks, etc. In one example, the training device 300 of the generator network and the training device 400 of the policy generation network for reinforcement learning according to the embodiment of the present disclosure can be integrated into the terminal device as a software module and / or a hardware module. For example, it can be a software module in the operating system of the terminal device, or it can be an application developed for the terminal device; of course, the training device 300 of the generator network and the training device 400 of the policy generation network for reinforcement learning can also be one of the many hardware modules of the terminal device.

[0105] Alternatively, in another example, the training device 300 of the generator network and the training device 400 of the policy generation network for reinforcement learning may also be separate devices from the terminal device, and they may be connected to the terminal device via a wired and / or wireless network and transmit interactive information in accordance with an agreed data format.

[0106] Exemplary Electronic Devices

[0107] Below, reference Figure 7 An electronic device according to an embodiment of the present disclosure is described.

[0108] Figure 7A block diagram of an electronic device according to an embodiment of the present disclosure is illustrated.

[0109] like Figure 7 As shown, the electronic device 10 includes one or more processors 11 and a memory 12 .

[0110] The processor 11 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 10 to perform desired functions.

[0111] The memory 12 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. The non-volatile memory may include, for example, a read-only memory (ROM), a hard disk, a flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 11 may run the program instructions to implement the training method of the generator network of each embodiment of the present disclosure described above and the training method of the strategy generation network for enhanced learning and / or other desired functions. Various contents such as state vectors, action vectors, parameters of the prediction network, etc. may also be stored in the computer-readable storage medium.

[0112] In one example, the electronic device 10 may further include: an input device 13 and an output device 14, and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0113] The input device 13 may include, for example, a keyboard, a mouse, etc.

[0114] The output device 14 can output various information to the outside, including parameters of the trained generator network and the strategy generation network, etc. The output device 14 can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, etc.

[0115] Of course, to simplify, Figure 7 Only some of the components related to the present disclosure in the electronic device 10 are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application situations, the electronic device 10 may also include any other appropriate components.

[0116] Exemplary computer program products and computer-readable storage media

[0117] In addition to the above-mentioned methods and devices, an embodiment of the present disclosure may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the training method of the generator network according to various embodiments of the present disclosure and the training method of the policy generation network for reinforcement learning described in the above-mentioned "Exemplary Method" section of this specification.

[0118] The computer program product may be written in any combination of one or more programming languages ​​to write program code for performing the operations of the disclosed embodiments, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0119] In addition, an embodiment of the present disclosure may also be a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, cause the processor to execute the steps of the training method of the generator network according to various embodiments of the present disclosure and the training method of the policy generation network for reinforcement learning described in the above “Exemplary Method” section of this specification.

[0120] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0121] The basic principles of the present disclosure are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present disclosure are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. are required by each embodiment of the present disclosure. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, and are not limitations. The above details do not limit the present disclosure to the necessity of adopting the above specific details to be implemented.

[0122] The block diagrams of the devices, apparatuses, equipment, and systems involved in this disclosure are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including," "comprising," "having," and the like are open words, referring to "including but not limited to," and can be used interchangeably therewith. The words "or" and "and" used herein refer to the words "and / or," and can be used interchangeably therewith, unless the context clearly indicates otherwise. The word "such as" used herein refers to the phrase "such as but not limited to," and can be used interchangeably therewith.

[0123] It should also be noted that in the apparatus, device and method of the present disclosure, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present disclosure.

[0124] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

[0125] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present disclosure to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. A training method for a generator network, comprising: Obtaining a training current state vector, a training action vector, and a training next state vector corresponding to the training current state vector and the training action vector for a reinforcement learning task, as well as a true posterior probability distribution determined thereby; wherein, when the task is a Go task, the action vector refers to the position where a chess piece is placed, and the state vector is the statistical probability of winning under the current state of the chessboard; inputting a set of random noise vectors of known probability distribution into a generator network to obtain a set of prediction networks, wherein the generator network includes a plurality of network units, each of which is used to generate a layer of the prediction network; Inputting the training current state vector and the training action vector into the set of prediction networks to obtain a predictive probability distribution; determining a KL divergence value between the predictive probability distribution and the true posterior probability distribution; and Updating parameters of the generator network based on the KL divergence value; Wherein, updating the parameters of the generator network based on the KL divergence value includes: Calculate the function gradient descent value of the KL divergence value using Stern's variational gradient descent method; Update the parameters of the generator network based on the function gradient descent value; The function gradient descent value of calculating the KL divergence value using the Stern variational gradient descent method includes: Calculating a predetermined function of each prediction network generated by the generator network to reproduce a unit ball of the kernel Hilbert space; calculating the gradient of the product of the generator network and the predetermined function with respect to parameters of the generator network; and, summing the gradients for the set of prediction networks to obtain a gradient sum; and Updating the parameters of the generator network based on the function gradient descent value includes: Based on the parameters of the current generator network, the gradient and the parameters of the updated generator network are obtained.

2. The training method of the generator network as claimed in claim 1, wherein: The predetermined function for calculating each prediction network to reproduce the unit ball of the kernel Hilbert space includes: Calculate the gradient of the difference function value between the next state predicted by the prediction network from the current state and action and the actual next state with respect to the prediction network; Sum the gradients with respect to all states and actions in the state space and the action space and multiply them by a kernel function to obtain a kernel function product, wherein the kernel function is used to calculate the distance between two adjacent generated prediction networks; and The kernel function product is subtracted from the kernel function gradient with respect to the prediction network and summed with respect to the set of prediction networks to obtain the predetermined function.

3. The training method of the generator network as claimed in claim 1, wherein: A set of random noise vectors with known probability distribution is input into the generator network to obtain a set of prediction networks including: For each prediction network, independent noise samples obtained from standard Gaussian noise with diagonal covariance are input to each network unit to generate a layer of the prediction network.

4. A training method for a strategy generation network for reinforcement learning, comprising: Obtain a generator network trained by the generator network training method according to any one of claims 1 to 3; Generating N prediction networks from the generator network; Get the current state vector and the action vector generated by the policy generation network; Input the current state vector and the action vector into the N prediction networks to obtain N next state vectors; Calculating an intrinsic reward function value for reinforcement learning based on the differences between the N next state vectors; as well as Parameters of the policy generation network are updated based on the intrinsic reward function value.

5. The training method for a strategy generation network for reinforcement learning as claimed in claim 4, wherein: Calculating the intrinsic reward function value for reinforcement learning based on the differences between the N next state vectors includes: Calculating a mean vector of the N next state vectors; Calculating an L2 distance value between each of the N next state vectors and the mean vector to obtain N L2 distance values; and An average of the N L2 distance values ​​is calculated to obtain the reward function value for reinforcement learning.

6. The training method for a strategy generation network for reinforcement learning according to claim 5, wherein: The next state vector includes a real next state vector obtained by a real object performing the task in the reinforcement learning task based on the current state vector and the action vector.

7. A training device for a generator network, comprising: A training vector acquisition unit, used to acquire a training current state vector, a training action vector, and a training next state vector corresponding to the training current state vector and the training action vector for a reinforcement learning task, and a true posterior probability distribution determined thereby; wherein, when the task is a Go task, the action vector refers to the position where the chess piece falls, and the state vector is the statistical probability of winning under the current state of the chessboard; A prediction network generation unit, used for inputting a set of random noise vectors of known probability distribution into a generator network to obtain a set of prediction networks, wherein the generator network includes a plurality of network units, each of which is used for generating a layer of the prediction network; a vector prediction unit, configured to input the training current state vector and the training action vector acquired by the training vector acquisition unit into the set of prediction networks generated by the prediction network generation unit to obtain a predictive probability distribution; a divergence value determining unit, configured to determine a KL divergence value between the predictive probability distribution obtained by the vector prediction unit and the true posterior probability distribution obtained by the training vector obtaining unit; and A generator updating unit, configured to update parameters of the generator network based on the KL divergence value determined by the divergence value determining unit; Wherein, updating the parameters of the generator network based on the KL divergence value determined by the divergence value determining unit includes: Calculate the function gradient descent value of the KL divergence value using Stern's variational gradient descent method; Update the parameters of the generator network based on the function gradient descent value; The function gradient descent value of calculating the KL divergence value using the Stern variational gradient descent method includes: Calculating a predetermined function of each prediction network generated by the generator network to reproduce a unit ball of the kernel Hilbert space; calculating the gradient of the product of the generator network and the predetermined function with respect to parameters of the generator network; and, summing the gradients for the set of prediction networks to obtain a gradient sum; and Updating the parameters of the generator network based on the function gradient descent value includes: Based on the parameters of the current generator network, the gradient and the parameters of the updated generator network are obtained.

8. A training device for a strategy generation network for reinforcement learning, comprising: A network acquisition unit, used to acquire the generator network trained by the generator network training device according to claim 7; A network generating unit, configured to generate N prediction networks from the generator network acquired by the network acquiring unit; A vector acquisition unit, used to acquire the current state vector and the action vector generated by the strategy generation network; A prediction vector obtaining unit, configured to input the current state vector and the action vector obtained by the vector obtaining unit into the N prediction networks generated by the network generating unit to obtain N next state vectors; a reward function calculation unit, configured to calculate an intrinsic reward function value for reinforcement learning based on the difference between the N next state vectors obtained by the prediction vector obtaining unit; as well as A network updating unit is used to update the parameters of the policy generation network based on the intrinsic reward function value calculated by the reward function calculation unit.

9. An electronic device, comprising: processor; as well as A memory, in which computer program instructions are stored, and when the computer program instructions are executed by the processor, the processor executes the training method of the generator network as described in any one of claims 1 to 3 or the training method of the policy generation network for reinforcement learning as described in any one of claims 4 to 6.

Images