A deep reinforcement learning method based on marginal normalization flow strategy and a storage medium

Abstract

The application discloses a deep reinforcement learning method based on a marginal normalization flow strategy and a storage medium, wherein the method comprises the following steps: constructing a strategy network and a decision network, the strategy network comprises a plurality of simulation coupling layers, the simulation coupling layers comprise and all adopt a full connection neural network structure, the strategy network is used for receiving environment parameter input and generating control output, the decision network is a full connection neural network structure, and the decision network is used for predicting a maximum value of the control output of the strategy network. Through the design of the decision network, the output function of the strategy network can be predicted, a strategy beyond a local optimal value is obtained, and it is determined that which action parameter can be applied to an agent, so that a technical effect of better controlling an action strategy of the agent is achieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of automation control, in particular to a method capable of improving the efficiency of marginal normalization flow model reinforcement learning. BACKGROUND

[0002] In recent years, deep reinforcement learning has been widely used in robot continuous control and game-related fields due to its excellent performance in controlling complex uncertain scenarios. Deep reinforcement learning requires the controlled agent to interact with the complex and uncertain environment, such as Figure 1 As shown in the figure, the agent obtains a state in the environment, the agent outputs an action using the state, and the action decision is executed in the environment. The environment outputs the next state and the reward obtained by the current decision, and the agent aims to maximize the reward it can obtain.

[0003] Among them, random policy sampling and action probability density calculation are inevitable steps in the deep reinforcement learning framework. For random policy methods such as PPO algorithm and SAC algorithm, the random policy distribution usually chooses a Gaussian distribution, and the action neural network directly outputs the mean and variance. For deterministic policy methods such as DDPG algorithm and TD3 algorithm, the action neural network directly outputs a deterministic action value, and Gaussian noise is usually added to improve exploration ability and algorithm robustness. However, in the real world, effective policy distribution is often not Gaussian distribution, and Gaussian distribution policy is not suitable for exploration in complex high-dimensional control tasks, and may even fall into local optimum.

[0004] In addition to Gaussian models, scholars have proposed many other probability distribution models to describe complex distributions, including mixture Gaussian distribution, normalized flow probability model, etc. As a probability model that can realize sampling, inference and accurate probability density calculation, the normalized flow probability model has been widely used in data generation, data repair and other fields. The classic normalized flow model includes Real NVP, etc. SUMMARY

[0005] Therefore, it is necessary to provide a new deep reinforcement learning method that can achieve the technical effect of diverse policy distribution sampling and improve the exploration ability of reinforcement learning algorithm in complex high-dimensional tasks.

[0006] To achieve the above purpose, the inventors provide a deep reinforcement learning method based on marginal normalized flow policy, including the following steps,

[0007] A strategy network and a decision network are constructed, the strategy network includes a plurality of simulation coupling layers, the simulation coupling layers include s(m) and t(m), both of which adopt a full connection neural network structure, the strategy network is used to receive an environment parameter input and generate a control output, the decision network is a full connection neural network structure, and the decision network is used to predict a maximum value of the control output of the strategy network.

[0008] In an embodiment of the present application, the following steps are included: randomly initializing a Q value network parameters of the strategy network and the decision network , i = 1, 2; the Q value network adopts a double Q structure and adopts a full connection neural network structure, the strategy network adopts the proposed marginal normalization flow strategy network, s(m) and t(m) each adopt a full connection neural network structure; the decision network is used to predict a maximum value of a strategy distribution and also adopts a full connection neural network structure;

[0009] initializing a target value Q network and a target decision network parameters of the Q value network strategy network and the decision network are the same as the initialized parameters. The algorithm is trained by using the target value network and the target decision network method, and the robustness of the algorithm is improved;

[0010] initializing a replay buffer pool

[0011] The strategy network guides an agent to continuously interact with an environment, and an action value a is generated by sampling a hidden variable t = f φ (m t , z t ; s t ), the environment returns a reward value r(s t , a t ) and a next state s t+1 , and such a tuple is stored in the replay buffer pool , that is

[0012] The Q value network parameters are updated by minimizing a Bellman residual error loss function ,

[0013] The strategy network π parameters are updated by minimizing a KL-divergence expected loss function φ ,

[0014] The action value a t is generated by the strategy network πφ is generated as follows:

[0015] a t = tanh(f φ -1 (m t , z t ; s t )),

[0016] The parameters of the decision network are updated by minimizing a network loss function .

[0017] The decision network is used to predict the latent variable values [m t max , z t max ] corresponding to the action distribution maximum a t max .

[0018] In an embodiment of the present application, the loss function for minimizing the Bellman residual error loss function is:

[0019]

[0020] The loss function can be set to include two versions of considering cross-entropy and not considering cross-entropy as needed. When not considering cross-entropy,

[0021]

[0022] When considering cross-entropy,

[0023]

[0024] In some specific embodiments of the present application, the KL-divergence expected loss function is as follows:

[0025]

[0026] The tanh activation function is used to limit the action boundary, so further obtained:

[0027]

[0028] Where Z(s t ) represents the partition function, which is used to normalize the distribution of the Q value function.

[0029] In some specific embodiments of the present application, the decision network loss function is as follows:

[0030]

[0031] Where zt μ Indicate z t The mean of a Gaussian distribution.

[0032] A deep reinforcement learning storage medium based on a marginal normalized flow strategy stores a computer program, which, when executed, includes the following steps.

[0033] A policy network and a decision network are constructed. The policy network includes several simulation coupling layers. The simulation coupling layers include s(m) and t(m), which are both fully connected neural network structures. The policy network is used to receive environmental parameter inputs and generate control outputs. The decision network is a fully connected neural network structure and is used to predict the maximum value of the control output of the policy network.

[0034] In one embodiment of this application, the following steps are included: randomly initializing the Q-value network. Policy Network π φ and determine the network The parameters are i = 1, 2; the Q-value network adopts a double Q structure and a fully connected neural network structure; the policy network adopts the proposed marginal normalized flow policy network; s(m) and t(m) adopt fully connected neural network structures respectively; the determination network used to predict the maximum value of the policy distribution also adopts a fully connected neural network structure.

[0035] Initialize the target value Q network and target decision network Parameters and Q-value network Policy Network π φ and determine the network The initialization parameters are the same. The algorithm is trained using a target value network and a target decision network method to improve its robustness;

[0036] Initialize the replay cache pool

[0037] Through policy network π φ The agent is guided to continuously interact with the environment by sampling latent variables. Generate action value a t =f φ (m t , z t ;s t The environment returns a reward value r(s). t a t ) and the next state s t+1 Store such a tuple in the replay cache pool. Li, that is

[0038] updating the Q-value network by minimizing a Bellman residual error loss function parameters,

[0039] updating the policy network p by minimizing a KL-divergence expected loss function φ parameters,

[0040] the action value a t generated by the policy network p φ as follows:

[0041] a t = tanh(f φ -1 (m t , z t ; s t ))

[0042] updating the decision network by minimizing a decision network loss function parameters,

[0043] using the decision network to predict the latent variable values [m t max , z t max ] corresponding to the action distribution maximum a t max .

[0044] In an embodiment of the present application, the loss function of the Bellman residual error loss function is:

[0045]

[0046] Without considering the cross-entropy,

[0047]

[0048] With considering the cross-entropy,

[0049]

[0050] In an embodiment of the present application, the KL-divergence expected loss function is as follows:

[0051] The loss function thereof is as follows:

[0052]

[0053] The tanh activation function is used to limit the action boundary, so further obtained:

[0054]

[0055] where Z(st ) represents a partition function used to normalize the distribution of the Q value function.

[0056] In an embodiment of the present application, the decision network loss function is as follows:

[0057]

[0058] where z t μ represents z t the mean of the Gaussian distribution.

[0059] Unlike the prior art, the above technical solution has strong complex distribution fitting capability by designing marginal normalization flow strategy, so that the strategy network has stronger exploration ability; the proposed marginal normalization flow has strong expression ability while greatly reducing the required parameter amount compared to the Real NVP network, and is more suitable for the training environment of reinforcement learning. At the same time, the proposed decision network can predict the maximum value of the marginal normalization flow strategy distribution, solving the problem that the probability density function maximum value of the normalization flow type strategy cannot be directly obtained. BRIEF DESCRIPTION OF DRAWINGS

[0060] Figure 1 The deep reinforcement learning method based on marginal normalization flow strategy described in the specific embodiment of the present application is a flowchart;

[0061] Figure 2 The schematic diagram of the strategy network structure described in another embodiment of the present application is shown in the figure;

[0062] Figure 3 The affine coupling layer grid structure schematic diagram described in the specific embodiment of the present application is shown in the figure;

[0063] Figure 4 The deep reinforcement learning method flowchart described in the specific embodiment of the present application is shown in the figure;

[0064] Figure 5 The x two-dimensional probability density graph of an experimental example described in the specific embodiment of the present application is shown in the figure;

[0065] Figure 6 The Mujuco partially continuous robot control task schematic diagram described in the specific embodiment of the present application is shown in the figure;

[0066] Figure 7 The training curve comparison schematic diagram described in the specific embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0067] The term "embodiment" is mentioned herein means that the specific features, structures, or characteristics described in connection with the embodiment can be included in at least one embodiment of the present application. The term "embodiment" appearing in various places in the specification does not necessarily refer to the same embodiment, nor does it particularly limit the independence or association between other embodiments. In principle, in the present application, as long as there is no technical contradiction or conflict, each technical feature mentioned in each embodiment can be combined in any way to form a corresponding implementable technical solution.

[0068] Unless otherwise defined, the meanings of the technical terms used herein are the same as those commonly understood by those skilled in the art to which the present application belongs; the use of related terms herein is only for the purpose of describing specific embodiments, and is not intended to limit the present application.

[0069] In the description of the present application, the phrase "and / or" is a description of the logical relationship between the objects, which means that there can be three relationships, for example, A and / or B, which means that there are three cases: A exists, B exists, and A and B exist at the same time. In addition, the character " / " herein generally represents that the associated objects before and after are a "or" logical relationship.

[0070] In the present application, the terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual quantity, primary and secondary or order relationship between the entities or operations.

[0071] In the present application, without more limitation, the "includes", "contains", "has" or other similar expressions used in the sentence are intended to cover non-exclusive inclusion, and these expressions do not exclude the presence of other elements in the process, method or product including the described elements, so that the process, method or product including a series of elements can not only include those limited elements, but also include other elements not explicitly listed, or also include the elements inherent to such process, method or product.

[0072] As the same as the understanding in the "Guidelines for Examination", in the present application, the expressions such as "greater than", "less than", "exceed" are understood as not including the number; the expressions such as "above", "below", "within" are understood as including the number. In addition, in the description of the embodiments of the present application, the meaning of "multiple" is more than two (including two), and similar expressions related to "multiple" are also understood in this way, for example, "multiple groups", "multiple times", etc., unless otherwise explicitly limited.

[0073] In the description of the embodiments of the present application, the spatially relative terms used, such as "central", "longitudinal", "transverse", "length", "width", "thickness", "upper", "lower", "front", "back", "left", "right", "vertical", "horizontal", "vertical", "top", "bottom", "inner", "outer", "clockwise", "counterclockwise", "axial", "radial", "circumferential", etc. The indicated orientation or positional relationship is based on the orientation or positional relationship shown in the specific embodiment or the drawing, and is only for the convenience of describing the specific embodiment of the present application or for the reader to understand, and does not indicate or imply that the indicated device or component must have a particular position, a particular orientation, or be constructed or operated in a particular orientation, and therefore cannot be understood as a limitation on the embodiments of the present application.

[0074] Unless otherwise expressly specified or limited, the terms "mount", "connect", "connection", "fixed", "set", and the like used in the description of the embodiments of the present application should be interpreted broadly. For example, the "connection" can be fixed connection, or detachable connection, or integral setting; it can be mechanical connection, or electrical connection, or communication connection; it can be direct connection, or indirect connection through intermediate medium; it can be internal communication of two elements or interaction relationship between two elements. For those skilled in the art to which the present application belongs, the specific meaning of the above terms in the embodiments of the present application can be understood according to the specific circumstances.

[0075] In the design embodiment of the normalized flow model, the random strategy that can represent the complex distribution can be considered to be introduced, wherein the normalized flow is considered to be introduced into the deep reinforcement learning algorithm to represent the random strategy distribution as a probability model that can realize sampling, reasoning and accurate probability density calculation. In some embodiments, the classic normalized flow is introduced, the Real NVP normalized flow strategy network is introduced into the SAC and PPO algorithm, the exploration ability and the learning ability of the sparse reward scene of the algorithm are enhanced, the planar normalized flow and the radiological normalized flow strategy network are also introduced into the SAC algorithm, and the contradiction between the normalized flow expression ability and the neural network parameter complexity is balanced. Please refer to Figure 1 , the deep reinforcement learning method based on the marginal normalized flow strategy includes the following steps, S10 constructs a strategy network and a decision network, the strategy network includes a plurality of simulation coupling layers, the simulation coupling layers include s(m) and t(m) which all adopt a full connection neural network structure, the strategy network is used to receive environment parameter input and generate control output, the decision network is a full connection neural network structure, and the decision network is used to predict the maximum value of the control output of the strategy network.

[0076] In some embodiments, this method can be used to simulate the decision-making process of artificial intelligence / agents under given environmental parameters. The agent here can be a robot for environmental exploration, etc., as long as it can interact with the environment, acquire information about environmental changes, and generate action commands. A policy network receives analog inputs of the environmental state and outputs action parameters as decision values. To avoid the interaction between the policy network and the environment getting stuck in local optima, a decision network is also required. The decision network is configured to predict the output of the policy network, and its parameters are adjusted to minimize the global network loss. The decision network can decide whether the output of the policy network should be applied to the agent. By designing the decision network, the output function of the policy network can be predicted, thereby obtaining a policy that transcends local optima and determining which action parameters can be applied to the agent, achieving the technical effect of better controlling the agent's action strategy.

[0077] To better simulate the decision-making actions of intelligent agents, such as Figure 2 In the illustrated embodiment, the policy network of this application further includes several biomimetic coupling layers. Each biomimetic coupling layer includes s(m) and t(m) of a fully connected neural network structure. The policy network receives input of latent variable m and environmental input St. The latent variable m follows a Gaussian distribution. The policy network generates output of latent variable m and action output value a. The latent variable m acts on all biomimetic coupling layers. Through the specific settings of the policy network described above, by introducing latent variable m as one set of input values, the output value of the policy network can be optimized, making the output value of the policy network easier to predict, and further optimizing the action value output of the global network.

[0078] By introducing the aforementioned policy network, the probability density estimation method can also be differentiated. The transformation process of the marginal normalized flow can be described as follows:

[0079]

[0080] Where f(x) maps the distribution of sample x to the distribution of latent variable z, representing the inference process, and the inverse function f -1 (z) maps the distribution of the latent variable z to the distribution of the sample x, representing the sampling process. The following mainly introduces two aspects: affine coupling layers and probability density estimation methods. The affine coupling layer of marginal normalized flow differs from the affine coupling layer in classic autoregressive flow, such as Real NVP, because it does not divide the features into two parts and then perform local mixing.

[0081] Instead, a new latent variable *m* is introduced, which can be viewed as a common guiding feature flowing through all dimensions. Given an input [m, x], the output of this affine coupling layer is [m, y], and this transformation process f iIt can be expressed as follows:

[0082]

[0083] where denotes the element-wise multiplication symbol, s(m) and t(m) are two neural networks, and this inverse transformation process f i -1 It can be expressed as follows:

[0084]

[0085] The specific affine coupling layer network structure can be seen from Figure 3 The state condition s t ) is not considered. It can be seen that the additional introduced hidden variable m does not change in the transformation, but is mixed into all dimensions, and this structure can avoid the selection rule of the input features of the affine coupling layer s(m) and t(m) in the Real NVP algorithm. This symmetric feature mixing structure makes the expression ability of the marginal normalization flow not only depend on the number of layers of the coupling layer, but also depend on the width of each layer. The Jacobian matrix of the affine coupling layer is also particularly easy to calculate, as follows:

[0086]

[0087] where |x| denotes the determinant of the matrix x, x T denotes the transpose of x, denotes the partial derivative symbol, diag(x) denotes the diagonal matrix of the vector x, exp denotes the exponential function, D denotes the dimension of the s(m) vector.

[0088] The present application proposes a new probability density estimation method for the improved affine coupling layer structure. The probability density estimation of the marginal normalization flow is based on the variable substitution theorem. Given the fixed distribution of the hidden variable z and m, such as the Gaussian normal distribution, the probability density distribution of x is the marginal of the probability density distribution of [m, x], and is expressed as follows:

[0089]

[0090] And

[0091]

[0092] where h0=x and h K =z, denotes the expectation of f(x) for x subject to p(x). This marginal normalization flow probability density estimation process is similar to the process of calculating the probability density function of x, which needs to resample the hidden variable m through the inverse function f -1 to calculate the hidden variable z.

[0093] In some specific application examples, two value networks are initially included. and a policy network π φ θ i Let be the parameters of the value network Q, and φ be the parameters of the policy network π, where the value network... The input is the state value s t and action value a t The output is in s t Action a occurs t Expected rewards generated; policy network The input is the state value s t The latent variable m sampled at time t t and the latent variable z sampled at time t t The output value is the action value a. t Both are deep neural networks. Furthermore, this invention employs a "target network" method, including two target value networks. and a target decision network For the parameters of the target value network Q, Let be the parameters of the target policy network π, where the target value network π is the parameter of the target policy network π. With value network The network structures are exactly the same, and the target decision network With policy network π φ The network structures are completely identical. Since the policy distribution of the policy network based on the marginal normalized flow model is not a standard distribution, the maximum value of the policy distribution cannot be directly calculated for use in policy evaluation and application after training. Therefore, this invention proposes the concept of a decision network and constructs a decision network. To determine the parameters of network M, The input value is the state value s t The output value is the latent variable m corresponding to the maximum value of the policy distribution. max This can produce the maximum value of the policy distribution. For use in strategy evaluation.

[0094] In a further embodiment, such as Figure 4 As shown, the method also includes the following steps: S40, randomly initializing the Q-value network. Policy Network π φ and determine the network The parameters are i = 1, 2; θ iare parameters of the Q value network, and are parameters of the policy network, the Q value network adopts a double Q structure, adopts a fully connected neural network structure, the policy network adopts the proposed marginal normalization flow policy network, s(m) and t(m) each adopt a fully connected neural network structure; the decision network is used to predict the maximum value of the policy distribution, and also adopts a fully connected neural network structure; wherein the Q value network is used to evaluate the sum of expected rewards, or determine an initial value according to environment parameters.

[0095] S41, initializing a target value Q network and a target decision network The parameters of the Q value network and the policy network π φ and the decision network are the same as the initialization parameters. The algorithm is trained by using a target value network and a target decision network method, and the robustness of the algorithm is improved;

[0096] S42, initializing a replay buffer pool

[0097] S43, guiding the agent to continuously interact with the environment through the policy network π φ In each interaction process, the hidden variable is sampled represents a normal distribution with a mean of μ and a variance of σ, and an action value a is generated t =f φ (m t , z t ; s t ), the environment returns a reward value r(s t , a t ) and a next state s t+1 , and stores such a tuple in the replay buffer pool , that is

[0098] S44, updating the Q value network parameters by minimizing the Bellman residual error loss function

[0099] S45, updating the policy network π φ parameters by minimizing the KL-divergence expected loss function.

[0100] S46, updating the parameters of the decision network by minimizing the decision network loss function

[0101] S47, using the decision network to predict the maximum value of the corresponding action distribution a t max ​​​the hidden variable value [m t max , z t max ].

[0102] The hidden variable value [m t max , z t max ] is predicted through the above scheme, the probability density of the policy network output can be further predicted, the problem that the probability density of the policy network output cannot be obtained in the prior art is solved, and therefore a global optimization scheme can be better obtained in the design of robot environment exploration. The training of the policy network and the decision network based on the marginal normalization flow proposed by the present application is realized through the above steps, a policy network with a diversity distribution is obtained, the efficiency of policy exploration is improved, the sampling and probability estimation of the policy network are very convenient, and the maximum value of the diversity policy distribution obtained by the decision network can be sampled and used for policy evaluation.

[0103] In an embodiment of the present application, the loss function of the minimized Bellman residual error loss function is:

[0104]

[0105] wherein γ is a decay coefficient, which can be set to 0.99, , which indicates that the sampled (s t , a t ) obeys the distribution of the replay buffer pool . The loss function includes two versions of considering cross-entropy and not considering cross-entropy. When cross-entropy is not considered,

[0106]

[0107] when cross-entropy is considered,

[0108]

[0109] The algorithm without considering cross-entropy is called MNF-AC algorithm, and the algorithm considering cross-entropy is called MNF-SAC algorithm.

[0110] By applying the above loss function to update the Q value network function, the parameters of the Q value network can be more accurate and effective.

[0111] In an embodiment of the present application, the KL-divergence expected loss function is minimized as follows:

[0112] The loss function thereof is as follows:

[0113]

[0114] wherein Z(st ) represents the partition function, used to normalize the distribution of the Q-value function, where the action value a t By policy network π φ The following is generated:

[0115] a t =tanh(f φ -1 (m t , z t ;s t ))

[0116] The tanh activation function is used to restrict action boundaries, so we can further obtain:

[0117]

[0118] Indicates sampling It follows a normal distribution.

[0119] In one embodiment of this application, the network loss function is determined as follows:

[0120]

[0121] Where z t μ Indicate z t The mean of the Gaussian distribution. Because of the presence of the affine coupling layer, the mean of the Gaussian distribution of the latent variables usually does not correspond to the maximum value of the generated sample distribution. Therefore, we propose a decision network. To predict the maximum value a of the corresponding action distribution t max The hidden variable value [m t max , z t max ], here considering m t The main role of the decision distribution is to predict only m. t max By designing the loss function described above, the decision network can better predict action values ​​and make better decisions regarding the output action values ​​of the policy network.

[0122] In a further embodiment, after step S47, a step is performed to softly update the target network, including the target value Q network. and target decision network The soft update rules are as follows:

[0123]

[0124]

[0125] where τ is a soft update coefficient that can be used to control the update speed, which can be set to 0.02 in the embodiment.

[0126] In some other embodiments of the present application, some experimental examples of the present scheme are also demonstrated, from which preliminary verification can be carried out: first, in order to verify the expression ability of the marginal normalization flow, a marginal normalization flow containing only one layer of affine coupling layer is adopted, the parameters of which are randomly generated, and the hidden variables m and z are both sampled from the Gaussian distribution The two-dimensional probability density diagram of part x is shown in Figure 5 It can be seen that only one layer of marginal normalization flow has rich expression ability.

[0127] In some other embodiments, the Actor-Critic deep reinforcement learning method based on the marginal normalization flow strategy proposed by the present application is experimented on the Mujuco continuous robot control baseline task, including Walker-v2, HalfCheetah-v2, Ant-v2 and Humanoid-v2 continuous high-dimensional control tasks, and the environment interaction is carried out through the OpenAI gym interface, as shown in Figure 6 The robots of various forms need to be controlled to achieve the walking task.

[0128] In some specific embodiments, for the Q value network and the decision network, a two-layer fully connected neural network with a size of 256x256 can be adopted, and the activation function adopts the Relu activation function; for the marginal normalization flow strategy network, s(m) and t(m) both adopt a two-layer fully connected neural network with a size of 256x256, and the activation function adopts the Relu activation function; the hidden variables m and z are both sampled from the Gaussian distribution The Adam optimizer is selected as the optimizer, the neural network is trained based on Pytorch, and the training curve is compared with the existing classical SAC algorithm and the SAC algorithm NF-SAC based on the Real NVP normalization flow, as shown in Figure 7 . Figure 7 The training curve is generated by multiple random seed experiments, and is smoothed by the Savitzky-Golay filter to enhance the visibility, it can be seen that the proposed MNF-AC algorithm has the best performance on the first three experiments, and from the stability, the method based on the marginal normalization flow has flat or superior performance, and compared with the NF-SAC algorithm, the parameter amount of the marginal normalization flow is reduced to one fourth of the original, and the algorithm space utilization is higher.

[0129] A storage medium of a deep reinforcement learning based on a marginal normalization flow strategy, storing a computer program, the computer program being executed when being run, comprising the steps of,

[0130] A policy network and a decision network are constructed. The policy network includes several simulation coupling layers. The simulation coupling layers include s(m) and t(m), which are both fully connected neural network structures. The policy network is used to receive environmental parameter inputs and generate control outputs. The decision network is a fully connected neural network structure and is used to predict the maximum value of the control output of the policy network.

[0131] In one embodiment of this application, the following steps are included: randomly initializing the Q-value network. Policy Network π φ and determine the network The parameters are i = 1, 2; the Q-value network adopts a double Q structure and a fully connected neural network structure; the policy network adopts the proposed marginal normalized flow policy network; s(m) and t(m) adopt fully connected neural network structures respectively; the determination network used to predict the maximum value of the policy distribution also adopts a fully connected neural network structure.

[0132] Initialize the target value Q network and target decision network Parameters and Q-value network Policy Network π φ and determine the network The initialization parameters are the same. The algorithm is trained using a target value network and a target decision network method to improve its robustness;

[0133] Initialize the replay cache pool

[0134] Through policy network π φ The agent is guided to continuously interact with the environment by sampling latent variables. Generate action value a t =f φ (m t , z t ;s t The environment returns a reward value r(s). t a t ) and the next state s t+1 Store such a tuple in the replay cache pool. Li, that is

[0135] The Q-value network is updated by minimizing the Bellman residual error loss function. parameter,

[0136] The policy network π is updated by minimizing the KL-divergence expectation loss function. φ parameter,

[0137] The action value a t By policy network πφ is generated as follows:

[0138] a t = tanh(f φ -1 (m t , z t ; s t ))

[0139] The decision network is updated by minimizing a decision network loss function .

[0140] The decision network is used to predict the latent variable values [m t max , z t max , a t max ] corresponding to the action distribution maximum a t .

[0141] In an embodiment of the present application, the loss function for minimizing the Bellman residual error loss function is:

[0142]

[0143] Without considering the cross-entropy,

[0144]

[0145] With considering the cross-entropy,

[0146]

[0147] In an embodiment of the present application, the KL-divergence expected loss function is as follows:

[0148]

[0149] The tanh activation function is used to limit the action boundary, so further obtained:

[0150]

[0151] where Z(s t ) represents the partition function, which is used to normalize the distribution of the Q value function.

[0152] In an embodiment of the present application, the decision network loss function is as follows:

[0153]

[0154] where z t μ represents z tMean of Gaussian distribution.

[0155] It is noted that, in this document, the terms such as first and second, etc., are used merely to distinguish one entity or action from another, and do not necessarily require or imply that there is any such actual relationship or order between these entities or actions. Also, the terms "including", "containing" or any other variants thereof are intended to cover a non-exclusive inclusion, so that a process, method, article or terminal device including a list of elements does not only include those elements, but also includes other elements not explicitly listed, or further includes elements inherent in such a process, method, article or terminal device. Without more limitations, the elements defined by the statement "including" or "containing" do not exclude the presence of other elements in the process, method, article or terminal device including the elements. In addition, in this document, "greater than", "less than", "exceed", etc. are understood as not including the number itself; "above", "below", "within", etc. are understood as including the number itself.

[0156] Those skilled in the art should understand that the above-mentioned embodiments can be provided as a method, device or computer program product. These embodiments can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. All or part of the steps in the method involved in the above-mentioned embodiments can be completed by instructing relevant hardware by program, and the program can be stored in a storage medium readable by computer device for executing all or part of the steps of the above-mentioned embodiments. The computer device includes but is not limited to: personal computer, server, general-purpose computer, special-purpose computer, network device, embedded device, programmable device, smart mobile terminal, smart home device, wearable smart device, vehicle-mounted smart device, etc.; the storage medium includes but is not limited to: RAM, ROM, magnetic disk, magnetic tape, optical disk, flash memory, U disk, mobile hard disk, memory card, memory stick, network server storage, network cloud storage, etc.

[0157] The above-mentioned embodiments are described with reference to the flowcharts and / or block diagrams of the method, device (system) and computer program product according to the embodiments. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of the flows and / or blocks in the flowcharts and / or block diagrams can be realized by computer program instructions. These computer program instructions can be provided to the processor of the computer device to produce a machine, so that the instructions executed by the processor of the computer device produce a process for implementing the specified functions of one or more flows and / or blocks in one or more flows and / or blocks. Figure 1 One or more flows and / or blocks Figure 1 An apparatus that performs the functions specified in one or more flows and / or blocks.

[0158] These computer program instructions can also be stored in a computer program product, that can direct a computer, other programmable data processing apparatus, or other devices to function in a Figure 1 one or more processes and / or blocks specified in a flowchart Figure 1 one or more blocks or multiple blocks in a flowchart.

[0159] These computer program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing other processes Figure 1 one or more processes and / or blocks specified in a flowchart Figure 1 one or more blocks or multiple blocks in a flowchart.

[0160] Although the above embodiments have been described, those skilled in the art can make additional changes and modifications to the embodiments once they know the basic inventive concept, and therefore the above description is only an embodiment of the present application, and does not limit the patent protection scope of the present application. Any equivalent structure or equivalent process transformation using the content of the present application specification and drawings, or direct or indirect application in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A deep reinforcement learning method based on a marginal normalized flow strategy, applied to simulate the decision-making process of an artificial intelligence / agent under given parameter conditions, wherein the agent is a robot for environmental exploration, characterized in that... Includes the following steps, A policy network and a decision network are constructed. The policy network includes several simulation coupling layers, including s(m) and t(m). The policy network is used to receive environmental parameter inputs obtained by the agent and generate control outputs. The decision network is a fully connected neural network structure. The decision network is used to predict the maximum value of the control output of the policy network, thereby obtaining a decision that surpasses the local optimum, and to determine whether the output of the policy network should be applied to the agent. The process includes the following steps: randomly initializing the Q-value network. Policy Network and determine the network The parameters, The Q-value network adopts a double-Q structure and a fully connected neural network structure. The policy network adopts the proposed marginal normalized flow policy network, and s(m) and t(m) respectively adopt fully connected neural network structures. The network is used to predict the maximum value of the policy distribution. Initialize the target Q value network and target policy network Target Q value network Initialization parameters and Q-value network The initialization parameters are the same for the target policy network. Initialization parameters and policy network The initialization parameters are the same, and the algorithm is trained using the target value network and target policy network methods; Initialize the replay cache pool ; Through policy network The agent is guided to continuously interact with the environment by sampling latent variables. Generate action value The environment returns a reward value. and the next state Store such a tuple in the replay cache pool. Li, that is ; The Q-value network is updated by minimizing the Bellman residual error loss function. parameter, The policy network is updated by minimizing the KL-divergence expectation loss function. parameter, The action value By policy network The following is generated: ， The decision network is updated by minimizing its loss function. The parameters, Using decision networks To predict the maximum value of the corresponding action distribution latent variable values .

2. The deep reinforcement learning method based on the marginal normalized flow strategy according to claim 1, characterized in that, The minimized Bellman residual error loss function is: ； This includes two versions: one considering cross-entropy and one not. When cross-entropy is not considered... ， When considering cross-entropy, 。 3. The deep reinforcement learning method based on the marginal normalized flow strategy according to claim 1, characterized in that, The expected loss function of minimizing the KL divergence is as follows: Its loss function is as follows: ， The activation function is used to restrict action boundaries, so we further obtain: ， in This represents the partition function, used to normalize the distribution of the Q-value network function.

4. The deep reinforcement learning method based on the marginal normalized flow strategy according to claim 1, characterized in that, The loss function that determines the network is as follows: ； in express The mean of a Gaussian distribution.

5. A deep reinforcement learning storage medium based on a marginal normalized flow strategy, characterized in that, The computer program stores a computer program that, when executed, includes the following steps: Construct a policy network and a decision network, wherein the policy network includes several simulation coupling layers, and the simulation coupling layers include and The policy network is used to receive environmental parameter inputs acquired by the agent and generate control outputs. The decision network is a fully connected neural network structure. The decision network is used to predict the maximum value of the control output of the policy network, thereby obtaining a decision that surpasses the local optimum and determining whether the output of the policy network should be applied to the agent, which is a robot for environmental exploration. The computer program also executes the following when it is run: The following steps are used to randomly initialize the Q-value network. Policy Network and determine the network The parameters, The Q-value network adopts a dual-Q structure and a fully connected neural network structure, while the policy network adopts the proposed marginal normalized flow policy network. and Each employs a fully connected neural network structure; the network is then used to predict the maximum value of the policy distribution. Initialize the target Q value network and target policy network Target Q value network Initialization parameters and Q-value network The initialization parameters are the same for the target policy network. Initialization parameters and policy network The initialization parameters are the same, and the algorithm is trained using the target value network and target policy network methods; Initialize the replay cache pool ; Through policy network The agent is guided to continuously interact with the environment by sampling latent variables. Generate action value The environment returns a reward value. and the next state Store such a tuple in the replay cache pool. Li, that is ; The Q-value network is updated by minimizing the Bellman residual error loss function. parameter, The policy network is updated by minimizing the KL-divergence expectation loss function. parameter, The action value By policy network The following is generated: ， The decision network is updated by minimizing its loss function. The parameters, Using decision networks To predict the maximum value of the corresponding action distribution latent variable values .

6. The deep reinforcement learning storage medium based on the marginal normalized flow strategy according to claim 5, characterized in that, The minimized Bellman residual error loss function is: ， Without considering cross-entropy, ， When considering cross-entropy, 。 7. The deep reinforcement learning storage medium based on the marginal normalized flow strategy according to claim 5, characterized in that, The expected loss function of minimizing the KL divergence is as follows: Its loss function is as follows: ， The activation function is used to restrict action boundaries, so we further obtain: ， in This represents the partition function, used to normalize the distribution of the Q-value network function.

8. The deep reinforcement learning storage medium based on the marginal normalized flow strategy according to claim 5, characterized in that, The loss function that determines the network is as follows: ， in express The mean of a Gaussian distribution.

Images