Off-line reinforcement learning method based on behavior action generation

Abstract

The invention discloses an offline reinforcement learning method based on behavior action generation, and the method comprises the steps: collecting a first data set of an intelligent agent, and carrying out the preprocessing of the first data set, and obtaining an offline data set; constructing a first diffusion model for training to obtain a final diffusion model; for each state in the offline data set, acquiring M actions corresponding to the state from the final diffusion model to form a support action data set; constructing a strategy network, N value networks, a target strategy network and N target value networks; training the strategy network and the value network by using the offline data set and the support action data set, and updating network parameters of the target strategy network and the target value network at the same time; and making a decision by using the learned optimal target strategy network, so that the intelligent agent performs a reinforcement learning task in an offline environment. According to the method, the limitation of an existing offline learning method on strategy learning is reduced, and the completeness and execution effect of a robot or an automatic driving task are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the field of machine learning, and in particular to an offline reinforcement learning method based on behavior action generation. Background Art

[0002] Reinforcement Learning (RL) is a machine learning method that learns the optimal strategy through the interaction between the agent and the environment. It is widely used in robot control, game AI, autonomous driving and other fields. Reinforcement learning does not learn knowledge directly from data, but takes actions based on the feedback of the environment. The process is a continuous interactive process, the purpose of which is to obtain the maximum cumulative reward from the beginning of the interaction with the environment to the end of the interaction. Deep reinforcement learning combines the traditional reinforcement learning method based on Markov decision process with deep learning technology, which greatly improves the ability of reinforcement learning to solve complex problems. It has now been implemented in many real-life applications such as games and robot control and has achieved remarkable results. However, there are always certain obstacles to the implementation of reinforcement learning. For example, traditional reinforcement learning methods usually require the agent to interact with the simulation environment continuously and learn the optimal strategy through trial and error. However, this real-time interactive method may have many challenges in real tasks, such as high data collection costs, low simulation efficiency, safety risks, and the difference between simulation and real applications because the simulation cannot be completely equivalent to the real situation. These problems have caused certain challenges to the implementation of online reinforcement learning.

[0003] Offline reinforcement learning (RL), also known as batch reinforcement learning, is a popular sub-area of ​​reinforcement learning that requires the agent to be trained from a static, deterministic dataset and no additional exploration of the environment is allowed during the entire policy training phase. This is very attractive in many real-world domains, such as robotics, logistics, and operations research, where real-world exploration with untrained policies is costly or dangerous, and existing data is available. However, directly applying value-based off-policy reinforcement learning algorithms in offline settings is ineffective due to the inability to interact with the environment and collect new data. This is because policy evaluation may encounter out-of-distribution actions, resulting in incorrect policy value estimates, which in turn affect policy improvement. In the offline reinforcement learning setting, the agent cannot interact with the environment to correct errors caused by out-of-distribution actions. These errors are continuously amplified through backtracking and iterative learning, resulting in a significant degradation in algorithm performance.

[0004] In order to solve this problem, the mainstream methods of offline reinforcement learning in the past can be divided into two categories. One is a divergence-based method that prevents the distribution of learning strategies and behavioral strategies from deviating too far by adding explicit or implicit divergence constraints to offline learning strategies. The other is to suppress the Q value of the policy function to solve the problem of overestimation of actions outside the distribution. For example, one method is to directly use the generalization characteristics of the neural network itself, and use the prediction error of the Q value of the state-action pair in the offline data set using an integrated Q-value network to fit the uncertainty of the state action, so that there is no need to add constraints to the strategy, and directly punish the strategy learning based on the estimated uncertainty. The integrated offline reinforcement learning method. Policy constraints can solve the problem of overestimation of actions outside the distribution, but because it restricts the learning strategy to the behavioral strategy, it will greatly limit the quality of the behavioral strategy. Although value constraints can also solve the problem of overestimation of actions outside the distribution, since policy learning and value learning are mutually carried out, while suppressing the value function, it will also constrain the policy function, resulting in suboptimality of the learning strategy. Summary of the invention

[0005] In view of the above-mentioned defects of the prior art, the technical problem to be solved by the present invention is to provide an offline reinforcement learning method based on behavior action generation, which aims to generate behavior actions through a diffusion model to evaluate the degree of deviation of the current policy output. The degree of deviation reflects the difference between the current policy and the distribution of the offline data set. The higher the degree of deviation, the higher the suppression of the corresponding value function, which can effectively solve the problem of excessive suppression effect in existing offline reinforcement learning, resulting in overly conservative policy learning.

[0006] To achieve the above object, the present invention discloses an offline reinforcement learning method based on behavior action generation, an offline reinforcement learning method based on behavior action generation, characterized in that the method comprises:

[0007] Step S1, collecting a first data set of the agent and preprocessing it to obtain an offline data set; wherein each data in the offline data set includes the current state of the agent, the current execution action, the current reward and the state at the next moment;

[0008] Step S2: constructing a first diffusion model and initializing the first diffusion model; using the offline data set to train the first diffusion model to obtain a final diffusion model;

[0009] Step S3: for each state in the offline data set, collect M actions corresponding to the state from the final diffusion model to form a support action set for the state; combine the support action sets to form a support action data set; wherein the state includes the current state and the next moment state;

[0010] Step S4: constructing a policy network, N value networks, a target policy network consistent with the structure of the policy network, and N target value networks consistent with the structure of the N value networks based on a deep deterministic policy gradient algorithm;

[0011] Step S5: training the policy network and the value network using the offline data set and the supporting action data set, and simultaneously updating the network parameters of the target policy network and the network parameters of the target value network;

[0012] Step S6: Use the learned optimal target strategy network to make decisions so that the agent can perform reinforcement learning tasks in an offline environment.

[0013] Optionally, the step S3 specifically includes:

[0014] For each state in the offline dataset, in the final diffusion model β w (·|s) is sampled M times, and the generated action set is represented as {a1,a2,a3,...,a M |β w (·|s)}; where β w (·|s) represents the distribution of the behavior actions of the diffusion model under a given state s, w is the network parameter of the diffusion model, and the action set constitutes the support action set

[0015] Combine the supported action sets corresponding to all the states in the offline data set to form the supported action data set Where M represents the number of sampling times of each state in the offline dataset, β w Represents the diffusion model used to generate actions.

[0016] Optionally, in step S5, using the offline data set and the supporting action data set to train the value network includes:

[0017] Using the experience samples (s, a, s) in the offline dataset and the supporting action dataset ′ ) updates the N value networks, whose loss function consists of a TD loss function and a regularization loss function, and the updating method is to minimize the following loss function by a gradient descent method:

[0018]

[0019] in, represents the overall loss function of the value network, represents the TD loss function of the value network, represents the regularized loss function of the value network, represents the network parameters of the nth value network, n∈{1,...,N}, s represents the current state, a represents the currently executed action, s ′ Indicates the state at the next moment.

[0020] Optionally, the method further includes:

[0021] The continuous action is discretized using the support action set to adapt the TD loss function. The update target in the TD loss function is to evaluate each action in the support action set and select the optimal action. The update method is the following loss function minimized by the gradient descent method:

[0022]

[0023] in, represents the TD loss function of the nth value network, represents the nth value network parameter, n∈{1,...,N}, represents the output of the value network n when the input is (s, a), R(s, a) represents the current reward value of (s, a), γ represents the return discount value, Indicates the nth ′ The target value network, n ′ ∈{1,...,N}, Indicates that in state s ′ Next, from the supported action set Select the action a that maximizes the expected Q value of the N value networks. * ,

[0024] It means taking the smallest target value network value from the N target value networks.

[0025] Optionally, the regularization loss function is used to suppress the value of out-of-distribution actions and avoid affecting the data output value in the offline data set when training the value network. The regularization loss function formula is as follows:

[0026]

[0027] in, Indicates the target strategy network when the state is s The output value is a ′ , (a ′ ,s) indicates that the nth value network has an input of (s,a ′ ), represents the network parameters of the nth value network, represents the supported action set when the state is s, β w represents the diffusion model, Represents the distance function between the policy output action and the support action set.

[0028] Optionally, the distance function is used to calculate the supported action set And the policy network output action a ′ The minimum distance value between them, the distance function is as follows:

[0029]

[0030] Among them, λ is a hyperparameter, Represents the supported action when the state is s. The distance function is used to calculate the minimum distance between the supported action and the output value of the policy network when the state is s.

[0031] Optionally, in step S5, using the offline data set and the supporting action data set to train the policy network includes:

[0032] Using the experience samples (s, a, s) in the offline dataset and the supporting action dataset ′ ) updates the policy network, and its loss function evaluates the value space of the N target value networks to find the action that maximizes the output value of the N target value networks. The updating method is to minimize the following loss function by gradient descent:

[0033]

[0034] in, is the loss function of the policy network, φ is the policy network parameter, a~π φ (s) indicates that the output of the policy network is a when the state is s, represents the confidence lower bound of the value network.

[0035] Optionally, the optimal value of the action-state space in the offline data set is found by alternating optimization, while ensuring that the value network accurately estimates the data of the offline data set, while reducing the estimated value of the value network for out-of-distribution actions, thereby exploring high-reward actions.

[0036] Optionally, the intelligent agent includes a robot and an autonomous driving device.

[0037] Compared with the prior art, the beneficial effects of the present invention include: the present invention can extract high-quality strategies from offline data sets, and in view of the problem that existing offline reinforcement learning causes excessive strategy suppression when solving the problem of overestimation of out-of-distribution actions, the present invention adopts a diffusion model sampling method to generate a support data set, which can solve the problem of overestimation of out-of-distribution actions while also learning excellent strategies. In addition, compared with directly using the diffusion model for strategy generation, the present invention has a faster strategy generation speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is a flowchart of an offline reinforcement learning method based on behavior action generation provided by a specific embodiment of the present invention;

[0039] Figure 2 It is a structural diagram of a supporting action set generation module provided by a specific embodiment of the present invention. DETAILED DESCRIPTION

[0040] The present invention discloses an offline reinforcement learning method based on behavior action generation. Those skilled in the art can refer to the content of this article and appropriately improve the technical details. It should be pointed out in particular that all similar substitutions and modifications are obvious to those skilled in the art, and they are all considered to be included in the present invention. The method and application of the present invention have been described through preferred embodiments, and relevant personnel can obviously modify or appropriately change and combine the methods and applications described herein without departing from the content, spirit and scope of the present invention to implement and apply the technology of the present invention.

[0041] The applicant has found through research that existing offline reinforcement learning methods are easily affected by out-of-distribution actions, which leads to reduced reinforcement learning effects.

[0042] Therefore, the embodiment of the present invention provides an offline reinforcement learning method based on behavior action generation, such as Figure 1 As shown, the method includes:

[0043] Step S1, collect the first data set of the intelligent agent and perform preprocessing to obtain an offline data set; wherein each data in the offline data set includes the current state of the intelligent agent, the current execution action, the current reward and the state at the next moment.

[0044] In this specific embodiment, the intelligent agent includes a robot and an autonomous driving device.

[0045] It should be noted that the intelligent agent may include various intelligent devices that interact with the outside world.

[0046] Step S2: construct a first diffusion model and initialize the first diffusion model; use the offline data set to train the first diffusion model to obtain a final diffusion model.

[0047] In this specific embodiment, in step S2, the diffusion model is modeled using a stochastic differential equation (SDE), and is sampled using a DPM-Solver-based method, which converts the SDE into an ordinary differential equation (ODE) to accelerate the sampling process, and updates the diffusion model network parameters using the empirical samples (s, a) in the offline data set D, using a gradient descent method to minimize the following loss function:

[0048]

[0049] in, represents the neural network loss function of the diffusion model, w is the neural network parameter of the diffusion model, represents the expected symbol, ~ represents the sampling operation, (a0,s)~D represents the empirical sample (a0,s) sampled from the offline dataset D, ∈ represents the predefined noise, N(0,I) represents the standard Gaussian noise, t represents the diffusion model time step, U([0,T]) represents the uniform sampling from time 0 to time T, a t represents the action at diffusion time t, ∈ w (a t ,t|s) means that the neural network input is a in state s t and the output at t.

[0050] Step S3: for each state in the offline data set, M actions corresponding to the state are collected from the final diffusion model to form a supporting action set of the state; and each supporting action set is combined to form a supporting action data set.

[0051] The state includes the current state and the state at the next moment.

[0052] In this specific embodiment, step S3 specifically includes:

[0053] like Figure 2 As shown, for each state in the offline dataset, in the final diffusion model β w (·|s) is sampled M times, and the generated action set is represented as {a1,a2,a3,...,a M |β w (·|s)}; where β w (·|s) represents the distribution of the behavior actions of the diffusion model under a given state s, w is the network parameter of the diffusion model, and the action set constitutes the support action set

[0054] Combine the supporting action sets corresponding to all states in the offline data set to form a supporting action data set Where M represents the number of sampling times of each state in the offline dataset, βw Represents the diffusion model used to generate actions.

[0055] Step S4: construct a policy network, N value networks, a target policy network consistent with the policy network structure, and N target value networks consistent with the N value network structures based on a deep deterministic policy gradient algorithm.

[0056] It should be noted that the Deep Deterministic Policy Gradient (DDPG) algorithm is a reinforcement learning algorithm used to solve policy optimization problems in continuous action spaces. The DDPG algorithm combines deep neural networks and deterministic policy gradient methods to effectively handle reinforcement learning problems in high-dimensional, continuous action spaces. The goal of the algorithm is to learn a deterministic policy that outputs the optimal action under a given state. This is achieved through two neural networks: one for learning the action value function (Q-value function) and the other for learning the policy.

[0057] Step S5: Use the offline data set and the supporting action data set to train the policy network and the value network, and simultaneously update the network parameters of the target policy network and the network parameters of the target value network.

[0058] In this specific embodiment, step S5 uses the offline data set and the supporting action data set to train the value network, including:

[0059] Using experience samples (s, a, s) from offline datasets and support action datasets ′ ) updates the N value networks, whose loss function consists of the TD loss function and the regularization loss function. The update method is to minimize the following loss function through the gradient descent method:

[0060]

[0061] in, represents the overall loss function of the value network, represents the TD loss function of the value network, represents the regularized loss function of the value network, represents the network parameters of the nth value network, n∈{1,...,N}, s represents the current state, a represents the current execution action, s ′ Indicates the state at the next moment.

[0062] In this specific embodiment, the method further includes:

[0063] The continuous actions are discretized using the support action set to adapt the TD loss function. The update target in the TD loss function is to evaluate each action in the support action set and select the optimal action. The update method is the following loss function minimized by the gradient descent method:

[0064]

[0065] in, represents the TD loss function of the nth value network, represents the nth value network parameter, n∈{1,...,N}, represents the output of the value network n when the input is (s, a), R(s, a) represents the current reward value of (s, a), and γ represents the return discount value. Indicates the nth ′ target value network, n ′ ∈{1,...,N}, Indicates that in state s ′ Down, Indicates taking the smallest target value network value from N target value networks. From the supported action set Select the action a that maximizes the expected Q value of the N value networks * The formula is as follows:

[0066]

[0067] in, It means taking the average of N value networks, Represents the network parameters of the i-th value network.

[0068] In this specific embodiment, the regularization loss function is used to suppress the value of out-of-distribution actions and avoid affecting the data output value in the offline data set when training the value network. The regularization loss function formula is as follows:

[0069]

[0070] in, Indicates the target strategy network when the state is s The output value is a ′ , It means that the nth value network has an input of (s, a ′ ), represents the network parameters of the nth value network, represents the supported action set when the state is s, β w represents the diffusion model, Represents the distance function between the policy output action and the supported action set.

[0071] Furthermore, the distance function is used to calculate the set of supported actions And the policy network output action a ′ The minimum distance value between them, the distance function is as follows:

[0072]

[0073] Among them, λ is a hyperparameter, Represents the supporting action when the state is s. The distance function is used to calculate the minimum distance between the supporting action and the output value of the policy network when the state is s, and is used to evaluate the quality of the policy output.

[0074] In this specific embodiment, step S5 uses the offline data set and the supporting action data set to train the policy network, including:

[0075] Using experience samples (s, a, s) from offline datasets and support action datasets ′ ) updates the policy network. Its loss function evaluates the value space of N target value networks and seeks the action that maximizes the output value of N target value networks. The update method is to minimize the following loss function by gradient descent:

[0076]

[0077] in, is the loss function of the policy network, φ is the policy network parameter, a~π φ (s) means that the output of the policy network is a when the state is s. It represents the confidence lower bound of the value network, and its formula is as follows:

[0078]

[0079] in, represents the expected value of N target value networks, α is a hyperparameter, α=0.01, n∈{1,...,N}, Represents the standard deviation of N target value networks. Using the confidence lower bound method to calculate the value function can prevent overestimation problems and make network learning more robust.

[0080] In this embodiment, in step S5, the network parameters of the N target value networks and the strategy network are updated according to the following formula:

[0081]

[0082] in, is the nth target value network parameter, is the nth target value network parameter, n∈{1,...,N}, τ is a hyperparameter, which is 0.005, φ is the policy network parameter, The target policy network parameters are updated after each round of training of the policy network or value network. The purpose of delaying the update of the target value network is to improve the stability of training and prevent overestimation problems.

[0083] Step S6: Use the learned optimal target policy network to make decisions so that the agent can perform reinforcement learning tasks in an offline environment.

[0084] In this specific embodiment, the optimal value of the action-state space in the offline data set is found through alternating optimization. While ensuring that the value network accurately estimates the data of the offline data set, the value network's estimated value for out-of-distribution actions is reduced, thereby exploring high-reward actions.

[0085] The embodiment of the present invention can extract high-quality strategies from offline data sets. In view of the problem that existing offline reinforcement learning causes excessive strategy suppression when solving the problem of overestimation of out-of-distribution actions, a diffusion model is used to sample and generate a support data set. While solving the problem of overestimation of out-of-distribution actions, excellent strategies can also be learned. In addition, compared with directly using the diffusion model for strategy generation, the embodiment of the present invention has a faster strategy generation speed.

[0086] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0087] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0088] The above are only preferred embodiments of the present invention and are not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. An offline reinforcement learning method based on behavior action generation, characterized in that: The method comprises: Step S1, collecting a first data set of the agent and preprocessing it to obtain an offline data set; wherein each data in the offline data set includes the current state of the agent, the current execution action, the current reward and the state at the next moment; Step S2: constructing a first diffusion model and initializing the first diffusion model; training the first diffusion model using the offline data set to obtain a final diffusion model; Step S3: for each state in the offline data set, collect M actions corresponding to the state from the final diffusion model to form a support action set for the state; combine the support action sets to form a support action data set; wherein the state includes the current state and the next moment state; Step S4: constructing a policy network, N value networks, a target policy network consistent with the structure of the policy network, and N target value networks consistent with the structure of the N value networks based on a deep deterministic policy gradient algorithm; Step S5: training the policy network and the value network using the offline data set and the supporting action data set, and simultaneously updating the network parameters of the target policy network and the network parameters of the target value network; Step S6: Use the learned optimal target strategy network to make decisions so that the agent can perform reinforcement learning tasks in an offline environment.

2. The offline reinforcement learning method based on behavior action generation according to claim 1, characterized in that: The step S3 specifically includes: For each state in the offline dataset, in the final diffusion model β w (·|s) is sampled M times, and the generated action set is represented as {a1,a2,a3,...,a M |β w (·|s)}; where β w (·|s) represents the distribution of the behavior actions of the diffusion model under a given state s, w is the network parameter of the diffusion model, and the action set constitutes the support action set Combine the supported action sets corresponding to all the states in the offline data set to form the supported action data set Where M represents the number of sampling times of each state in the offline dataset, β w Represents the diffusion model used to generate actions.

3. The offline reinforcement learning method based on behavior action generation according to claim 1, characterized in that: The step S5 uses the offline data set and the supporting action data set to train the value network, including: Using the experience samples (s, a, s) in the offline dataset and the supporting action dataset ′ ) updates the N value networks, whose loss function consists of a TD loss function and a regularization loss function, and the updating method is to minimize the following loss function by a gradient descent method: in, represents the overall loss function of the value network, represents the TD loss function of the value network, represents the regularized loss function of the value network, represents the network parameters of the nth value network, n∈{1,...,N}, s represents the current state, a represents the currently executed action, s ′ Indicates the state at the next moment.

4. The offline reinforcement learning method based on behavior action generation according to claim 3, characterized in that: The method further comprises: The continuous action is discretized using the support action set to adapt the TD loss function. The update target in the TD loss function is to evaluate each action in the support action set and select the optimal action. The update method is the following loss function minimized by the gradient descent method: in, represents the TD loss function of the nth value network, represents the nth value network parameter, n∈{1,...,N}, represents the output of the value network n when the input is (s, a), R(s, a) represents the current reward value of (s, a), γ represents the return discount value, Indicates the nth ′ The target value network, n ′ ∈{1,...,N}, Indicates that in state s ′ Next, from the supported action set Select the action a that maximizes the expected Q value of the N value networks. * , It means taking the smallest target value network value from the N target value networks.

5. The offline reinforcement learning method based on behavior action generation according to claim 3, characterized in that: The regularization loss function is used to suppress the value of out-of-distribution actions and avoid affecting the data output value in the offline data set when training the value network. The regularization loss function formula is as follows: in, Indicates the target strategy network when the state is s The output value is a ′ , (a ′ ,s) indicates that the nth value network has an input of (s,a ′ ) output, represents the network parameters of the nth value network, represents the supported action set when the state is s, β w represents the diffusion model, Represents the distance function between the policy output action and the support action set.

6. The offline reinforcement learning method based on behavior action generation according to claim 5, characterized in that: The distance function is used to calculate the supported action set And the policy network output action a ′ The minimum distance value between them, the distance function is as follows: Among them, λ is a hyperparameter, Represents the supported action when the state is s. The distance function is used to calculate the minimum distance between the supported action and the output value of the policy network when the state is s.

7. The offline reinforcement learning method based on behavior action generation according to claim 1, characterized in that: The step S5 uses the offline data set and the supporting action data set to train the policy network, including: Using the experience samples (s, a, s) in the offline dataset and the supporting action dataset ′ ) updates the policy network, and its loss function evaluates the value space of the N target value networks to find the action that maximizes the output value of the N target value networks. The updating method is to minimize the following loss function by gradient descent: in, is the loss function of the policy network, φ is the policy network parameter, a~π φ (s) indicates that the output of the policy network is a when the state is s, represents the confidence lower bound of the value network.

8. The offline reinforcement learning method based on behavior action generation according to any one of claims 1 to 7, characterized in that: By finding the optimal value of the action-state space in the offline data set through alternating optimization, while ensuring that the value network accurately estimates the data of the offline data set, the estimated value of the value network for out-of-distribution actions is reduced, thereby exploring high-reward actions.

9. The offline reinforcement learning method based on behavior action generation according to claim 1, characterized in that: The intelligent agents include robots and autonomous driving devices.

Images