Model-based offline reinforcement learning training method

Abstract

The present invention provides a model-based offline reinforcement learning training method that includes policy constraints and uncertainty estimation. First, the dynamic model is updated: the number of set members, the number of model forward predictions, and the model set, policy network, and value function network are initialized; static data are randomly sampled from the static data set and the model is updated based on it, and this step is repeated until the model converges. Secondly, the policy is updated: the sampled static data is predicted multiple times through the model to obtain predicted data and uncertainty estimates and the predicted rewards are subtracted from the uncertainty to obtain dynamic data; the dynamic data is put into the experience pool and static data and dynamic data are sampled from the static data set and the experience pool; the state in the dynamic data is input into the policy network and the model to obtain the prediction of the next state; the policy network and the value function network are updated through the static data, dynamic data, and the prediction of the next state, and the above-mentioned policy update steps are repeated until the policy network converges.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a model-based offline reinforcement learning training method including strategy constraints and uncertainty estimation. Background Art

[0002] Reinforcement learning has made tremendous progress in many fields. Generally speaking, reinforcement learning requires interaction with the environment. In some cases, such as in autonomous driving and healthcare, these interactions can be costly or even prohibited. For this reason, offline reinforcement learning, which learns from static datasets without interacting with the environment, has become a promising direction. Model-based offline reinforcement learning learns a dynamic model of the environment from static datasets, which helps improve data utilization and achieve better performance.

[0003] Existing model-based offline reinforcement learning algorithms are subject to distribution shift and model bias. Offline reinforcement learning training relies entirely on static datasets and aims to learn policies that can perform beyond the dataset's support. This leads to distribution shift between the static training data and the policy trajectory in the real environment. Distribution shift causes the Bellman operator to overestimate the Q function, resulting in a tendency to select out-of-distribution actions. Model bias is an inherent problem in model-based reinforcement learning, causing the policy to be directed into unknown territory and ultimately disrupting policy learning. State mismatch between the policy and the static dataset is a common phenomenon in offline reinforcement learning. This can cause the policy to encounter out-of-distribution states in the real environment, exacerbating the adverse effects of model bias. Therefore, addressing the overestimation of the Q function and reducing state mismatch can mitigate the negative impact of distribution shift and model bias on model-based offline reinforcement learning algorithms. Summary of the Invention

[0004] To overcome the shortcomings of the existing technology, the present invention provides a model-based offline reinforcement learning training method that includes policy constraints and uncertainty estimation. It solves the overestimation problem of the Q function by subtracting uncertainty from the reward function, and reduces the state access mismatch between the policy and the static dataset by constraining the policy, thereby achieving better performance.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] A model-based offline reinforcement learning training method with policy constraints and uncertainty estimation includes the following steps:

[0007] Step 1: Update the dynamic model of the environment, including:

[0008] Step S1: Set the number of set members , the number of forward predictions of the dynamics model , conservatively estimated weights ;

[0009] Step S2: Initialize the dynamic model set of the environment , ,in Represents a dynamics model, reinforcement learning policy network , value function network , static experience pool , dynamic experience pool ;

[0010] Step S3: convert the storage format into a quad The data of the static data set is put into the static experience pool Among them Represents the current state. Represents the current moment action, Represents the current moment reward, Represents the state at the next moment;

[0011] Step S4: From the static experience pool Random sampling to obtain static data , static data Quadruple formal representation;

[0012] Step S5: Based on static data Update dynamic model ensemble ;

[0013] Repeat steps S4 and S5 until the dynamic model set convergence;

[0014] Step 2: Update the reinforcement learning policy network, including:

[0015] Step S6: From the static experience pool Sampling static data , and through the dynamic model set and strategic networks Predicting the future Step data, get predicted data and obtain uncertainty estimates ;

[0016] Step S7: Prediction data Rewards in Subtract uncertainty estimate Get dynamic data , and put it into the dynamic experience pool middle;

[0017] Step S8: From the static experience pool and dynamic experience pool Sample the same number of static data and dynamic data ;

[0018] Step S9: From the dynamic experience pool Sampling dynamic data Status in Input policy network , get the action , and thus get the state-action pair ;

[0019] Step S10: The obtained state-action pair Input dynamics model collection , get the prediction of the next state ;

[0020] Step S11: Static data obtained by sampling and dynamic data and the prediction of the next state To update the policy network ;

[0021] Step S12: Static data obtained by sampling and dynamic data Update value function network ;

[0022] Repeat steps S6-S12 until the policy network convergence.

[0023] Furthermore, the step S5 specifically includes:

[0024] Step S5a: Static data in and Represented as a state-action pair ;

[0025] Step S5b: Set the state action pair Input dynamics model collection The reward at the current moment is obtained and the next moment state The set of mean and variance of the Gaussian distribution ,in , representing the kinetic model The output result is, represent Obtained The mean of represent Obtained The variance of represent Obtained The mean of represent Obtained variance;

[0026] Step S5c: For each dynamic model in the set M , updated by maximum likelihood, as shown in formula (1), where The loss function when updating the model:

[0027] (1)

[0028] Furthermore, the step S6 specifically includes:

[0029] Step S6a: From the static dataset Random sampling to obtain static data , static data Quadruple formal representation;

[0030] Step S6b: Step prediction, the state Input policy network Get action And combine to get the state-action pair ;

[0031] Step S6c: Set the state action pair Input dynamics model collection The current moment reward is obtained by sampling : ,in represents the Gaussian distribution, is the mean of the Gaussian distribution, is the variance of the Gaussian distribution, and the next moment state : ,in And combine to get a quaternion ,in Representative Step prediction;

[0032] Step S6d: Repeat steps S6b and S6c times, get the predicted data , Quadruple Formally expressed, and status obtained Uncertainty estimate : .

[0033] Furthermore, the step S7 specifically includes:

[0034] Step S7a: Prediction data Rewards in Subtract uncertainty estimate , get new rewards , and get dynamic data , By quad composition;

[0035] Step S7b: Dynamic data Add to dynamic experience pool middle.

[0036] Furthermore, the step S10 specifically includes:

[0037] Step S10a: The obtained state-action pair Input dynamics model collection In the example, we get the reward representing the current moment and the prediction of the next state The set of mean and variance of the Gaussian distribution ,in , representing the kinetic model The output result is, represent Obtained The mean of represent Obtained The variance of represent Obtained The mean of represent Obtained variance;

[0038] Step S10b: Sampling to get the prediction of the next state ,in .

[0039] Furthermore, in step S11, the static data obtained by sampling and dynamic data and the prediction of the next state To update the policy network , as shown in formula (2), where Represents dynamic data The state in Represents static data The state in Status The corresponding next state prediction, state Status and status A mixture of is the loss function during strategy update, Represents the status Input policy network and get action : hour The expectation of the function, Representatives from and mid-sampling and hour The expectation of the function, To conservatively estimate the weights:

[0040] (2)

[0041] Furthermore, the step S12 specifically includes:

[0042] Step S12a: The static data obtained by sampling and dynamic data Mixed into data ,data You can use the quad In the form:

[0043] Step S12b: The loss function provided by formula (3) Update value function network ,in Represents the action at the next moment.

[0044] (3)

[0045] Compared with the prior art, the present invention has the following beneficial effects:

[0046] (1) The present invention solves the Q-value overestimation problem caused by distribution shift faced by model-based offline reinforcement learning through uncertainty estimation, thereby ensuring that the strategy will not tend to select state-action pairs outside the distribution, but will execute reliable state-action pairs, thereby reducing the negative impact of distribution shift on model-based offline reinforcement learning algorithms;

[0047] (2) By constraining the policy, the present invention reduces the state access mismatch between the policy and the static dataset, which can reduce the probability of the policy encountering out-of-distribution states, thereby alleviating the negative impact of model bias on the model-based offline reinforcement learning algorithm;

[0048] (3) The present invention can be combined with any model-based offline reinforcement learning algorithm and has strong universality. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 This is a flow chart of a model-based offline reinforcement learning training method including strategy constraints and uncertainty estimation in an embodiment of the present invention; DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention is further described in detail below through specific implementation and in conjunction with the accompanying drawings.

[0051] like Figure 1 As shown, an embodiment of the present invention provides a model-based offline reinforcement learning training method including policy constraints and uncertainty estimation, including the following steps:

[0052] Step 1: Update the dynamic model of the environment, including:

[0053] Step S1: Set the number of set members , the number of forward predictions of the dynamics model , conservatively estimated weights ;

[0054] Step S2: Initialize the dynamic model set of the environment , ,in Represents a dynamics model, reinforcement learning policy network , value function network , static experience pool , dynamic experience pool ;

[0055] Step S3: convert the storage format into a quad The data of the static data set is put into the static experience pool Among them Represents the current state. Represents the current action. Represents the current moment reward, Represents the state at the next moment;

[0056] Step S4: From the static experience pool Random sampling to obtain static data , static data Quadruple formal representation;

[0057] Step S5: Based on static data Update dynamic model ensemble ,include:

[0058] Step S5a: Static data in and Represented as a state-action pair ;

[0059] Step S5b: Set the state action pair Input dynamics model collection The reward at the current moment is obtained and the next moment state The set of mean and variance of the Gaussian distribution ,in , representing the kinetic model The output result is, represent Obtained The mean of represent Obtained The variance of represent Obtained The mean of represent Obtained The variance of .

[0060] Step S5c: For each dynamic model in the set M , updated by maximum likelihood, as shown in formula (1), where The loss function when updating the model:

[0061] (1)

[0062] Repeat steps S4 and S5 until the dynamic model set convergence;

[0063] Step 2: Update the reinforcement learning policy network, including:

[0064] Step S6: From the static experience pool Sampling static data , and through the dynamic model set and strategic networks Predicting the future Step data, get predicted data and obtain uncertainty estimates , specifically including:

[0065] Step S6a: From the static dataset Random sampling to obtain static data , static data Quadruple formal representation;

[0066] Step S6b: Step prediction, the state Input policy network Get action And combine to get the state-action pair ;

[0067] Step S6c: Set the state action pair Input dynamics model collection The current moment reward is obtained by sampling : ,in represents the Gaussian distribution, is the mean of the Gaussian distribution, is the variance of the Gaussian distribution, and the next moment state : ,in And combine to get a quadruple ,in Representative Step prediction;

[0068] Step S6d: Repeat steps S6b and S6c times, get the predicted data , Quadruple Formally expressed, and status obtained Uncertainty estimate : ;

[0069] Step S7: Prediction data Rewards in Subtract uncertainty estimate Get dynamic data , and put it into the dynamic experience pool Specifically, it includes:

[0070] Step S7a: Prediction data Rewards in Subtract uncertainty estimate , get new rewards , and get dynamic data , By quad composition;

[0071] Step S7b: Dynamic data Add to dynamic experience pool middle;

[0072] Step S8: From the static experience pool and dynamic experience pool Sample the same number of static data and dynamic data ;

[0073] Step S9: From the dynamic experience pool Sampling dynamic data Status in Input policy network , get the action , and thus get the state-action pair ;

[0074] Step S10: The obtained state-action pair Input dynamics model collection , get the prediction of the next state , specifically including:

[0075] Step S10a: The obtained state-action pair Input dynamics model collection In the example, we get the reward representing the current moment and the prediction of the next state The set of mean and variance of the Gaussian distribution ,in , representing the kinetic model The output result is, represent Obtained The mean of represent Obtained The variance of represent Obtained The mean of represent Obtained variance;

[0076] Step S10b: Sampling to get the prediction of the next state ,in ;

[0077] Step S11: Static data obtained by sampling and dynamic data and the prediction of the next state To update the policy network , as shown in formula (2), where Represents dynamic data The state in Represents static data The state in Status The corresponding next state prediction, state Status and status A mixture of is the loss function during strategy update, Represents the status Input policy network and get action : hour The expectation of the function, Representatives from and mid-sampling and hour The expectation of the function, To conservatively estimate the weights:

[0078] (2)

[0079] Step S12: Static data obtained by sampling and dynamic data Update value function network ,include:

[0080] Step S12a: The static data obtained by sampling and dynamic data Mixed into data ,data You can use the quad In the form:

[0081] Step S12b: The loss function provided by formula (3) Update value function network ,in Represents the action at the next moment.

[0082] (3)

[0083] Repeat steps S6-S12 until the policy network convergence.

[0084] The above embodiments are provided for the purpose of describing the present invention only and are not intended to limit the scope of the present invention. The scope of the present invention is defined by the appended claims. Various equivalent substitutions and modifications made without departing from the spirit and principles of the present invention are intended to be within the scope of the present invention.

Claims

1. A model-based offline reinforcement learning training method with policy constraints and uncertainty estimation, characterized by: The steps include: Step 1: Update the dynamic model of the environment, including: Step S1: Set the number of set members , the number of forward predictions of the dynamics model , conservatively estimated weights ; Step S2: Initialize the dynamic model set of the environment , ,in Represents a dynamics model, reinforcement learning policy network , value function network , static experience pool , dynamic experience pool ; Step S3: convert the storage format into a quad The data of the static data set is put into the static experience pool Among them Represents the current state. Represents the current moment action, Represents the current moment reward, Represents the state at the next moment; Step S4: From the static experience pool Random sampling to obtain static data , static data Quadruple formal representation; Step S5: Based on static data Update dynamic model ensemble ,include: Step S5a: Static data in and Represented as a state-action pair ; Step S5b: Set the state action pair Input dynamics model collection The reward at the current moment is obtained and the next moment state The set of mean and variance of the Gaussian distribution ,in , representing the kinetic model The output result is, represent Obtained The mean of represent Obtained The variance of represent Obtained The mean of represent Obtained variance; Step S5c: For each dynamic model in the set M , updated by maximum likelihood, as shown in formula (1), where The loss function when updating the model: （1） Repeat steps S4 and S5 until the dynamic model set convergence; Step 2: Update the reinforcement learning policy network, including: Step S6: From the static experience pool Sampling static data , and through the dynamic model set and strategic networks Predicting the future Step data, get predicted data and obtain uncertainty estimates ; Step S7: Prediction data Rewards in Subtract uncertainty estimate Get dynamic data , and put it into the dynamic experience pool middle; Step S8: From the static experience pool and dynamic experience pool Sample the same number of static data and dynamic data ; Step S9: From the dynamic experience pool Sampling dynamic data Status in Input policy network , get the action , and thus get the state-action pair ; Step S10: The obtained state-action pair Input dynamics model collection , get the prediction of the next state ; Step S11: Static data obtained by sampling and dynamic data and the prediction of the next state To update the policy network ; Step S12: Static data obtained by sampling and dynamic data Update value function network ,include: Step S12a: The static data obtained by sampling and dynamic data Mixed into data ,data Use quad In the form: Step S12b: The loss function is updated by the value function provided by formula (3) Update value function network ,in Represents the next moment action: （3） in, Represents the status Input policy network and get action : yes Expectation of a function; Repeat steps S6-S12 until the policy network convergence.

2. A model-based offline reinforcement learning training method with policy constraints and uncertainty estimation according to claim 1, characterized in that: The step S6 specifically includes: Step S6a: From the static dataset Random sampling to obtain static data , static data Quadruple formal representation; Step S6b: In the step prediction, the state Input policy network Get action And combine to get the state-action pair ; Step S6c: Set the state action pair Input dynamics model collection The current moment reward is obtained by sampling : ,in represents the Gaussian distribution, is the mean of the Gaussian distribution, is the variance of the Gaussian distribution, and the next moment state : ,in And combine to get a quaternion ,in Representative Step prediction; Step S6d: Repeat steps S6b and S6c times, get the predicted data , Quadruple Formally expressed, and status obtained Uncertainty estimate of : .

3. A model-based offline reinforcement learning training method with policy constraints and uncertainty estimation according to claim 2, characterized in that: The specific examples include: Step S7a: Prediction data Rewards in Subtract uncertainty estimate , get new rewards , and get dynamic data , By quad composition; Step S7b: Dynamic data Add to dynamic experience pool middle.

4. A model-based offline reinforcement learning training method with policy constraints and uncertainty estimation according to claim 3, characterized in that: The step S10 specifically includes: Step S10a: The obtained state-action pair Input dynamics model collection In the example, we get the reward representing the current moment and the prediction of the next state The set of mean and variance of the Gaussian distribution ,in , representing the kinetic model The output result is, represent Obtained The mean of represent Obtained The variance of represent Obtained The mean of represent Obtained variance; Step S10b: Sampling to get the prediction of the next state ,in .

5. A model-based offline reinforcement learning training method including policy constraints and uncertainty estimation according to claim 4, characterized in that: In step S11, the static data obtained by sampling and dynamic data and the prediction of the next state To update the policy network , as shown in formula (2), where Represents dynamic data The state in Represents static data The state in Status The corresponding next state prediction, state Status and status A mixture of is the loss function during strategy update, Represents the status Input policy network and get action : hour The expectation of the function, Representatives from and mid-sampling and hour The expectation of the function, To conservatively estimate the weights: （2）。

Images