Dynamic self-adaptive offline-to-online reinforcement learning method
By employing a dynamic adaptive offline-to-online reinforcement learning method, utilizing variational autoencoders to quantify the uncertainty of state-action pairs, and combining online interaction and network weighted fusion, the problem of value estimation bias and policy instability during the offline-to-online transition is solved, enabling rapid adaptation and efficient learning of the robot policy network.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-02
- Publication Date
- 2026-03-27
AI Technical Summary
Existing offline-to-online reinforcement learning methods suffer from value estimation bias and policy instability during the transition, resulting in low efficiency in utilizing online data and slow adaptation speed.
We employ a dynamic adaptive offline-to-online reinforcement learning approach. By constructing the robot's state space, action space, and reward function, we utilize an actor-critic architecture and a variational autoencoder for offline reinforcement learning to obtain offline policies and value networks. We then train the online value network through online interaction and greedy presampling. Finally, we use the uncertainty weights of state-action pairs to perform network weighted fusion and update the policy network.
It enables rapid adaptation to new environments at low cost, improves task rewards, avoids early performance crashes and oscillations, and ensures the security, stability, and efficient learning of the policy network.
Smart Images
Figure CN121745205A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of reinforcement learning, and particularly relates to a dynamic adaptive offline-to-online reinforcement learning method. BACKGROUND
[0002] Reinforcement learning has a wide range of applications in robot control and intelligent decision-making. Online reinforcement learning optimizes the policy based on the reward signal through continuous interaction with the environment, thereby achieving adaptive learning of complex tasks, but it has high cost and safety risks. Offline reinforcement learning only relies on historical data for training, but it is limited by data distribution and is prone to value overestimation and policy degradation.
[0003] Existing offline-to-online reinforcement learning methods attempt to combine offline safety and online flexibility, but often introduce pessimistic bias (algorithms are intentionally designed to systematically underestimate the value of state-action pairs, especially for samples that are not common or have never appeared in the data set) by relying too much on offline data, which suppresses exploration and leads to low efficiency of online data utilization and slow adaptation speed. Existing methods such as hybrid replay or value regularization have not effectively solved the value estimation bias and policy instability problems during the transition from offline to online. SUMMARY
[0004] In view of the above deficiencies in the prior art, the present application provides a dynamic adaptive offline-to-online reinforcement learning method.
[0005] To achieve the above-mentioned application purposes, the technical solutions adopted by the present application are as follows: A dynamic adaptive offline-to-online reinforcement learning method, comprising the following steps: Modeling a robot control problem as a Markov decision process to construct the state space, action space and reward function of the robot, and constructing an offline data set based on the state space, action space and reward function of the robot; Based on the offline data set, the actor-critic architecture and the variational autoencoder are trained offline to obtain the offline policy network, the offline value network and the trained variational autoencoder of the robot; Initialize the online value network of the robot, freeze the parameters of the offline policy network of the robot, and use the offline policy network of the robot to interact with the environment for greedy pre-sampling to obtain an online buffer data set, and train the initialized online value network of the robot based on the online buffer data set to obtain the online value network of the robot; Based on the trained variational autoencoder, uncertainty weights of state-action pairs are obtained, the offline value network and the online value network of the robot are weighted and fused based on the uncertainty weights of the state-action pairs to obtain a composite value network of the robot, and the offline policy network of the robot is updated based on the composite value network of the robot to complete a dynamic self-adaptive offline-to-online reinforcement learning method and obtain a final policy network of the robot.
[0006] Further, the actor-critic architecture is subjected to offline reinforcement learning based on the offline data set to obtain the offline policy network and the offline value network of the robot, and the specific process is as follows: An optimization target model of the policy network and an optimization target model of the value network are constructed. Based on the offline data set, the actor-critic architecture is subjected to offline reinforcement learning by using the optimization target model of the policy network and the optimization target model of the value function to obtain the offline policy network and the offline value network of the robot.
[0007] Further, the optimization target model of the policy network has an expression as follows: Wherein: is a corresponding expected value when the parameter of the policy network is , is an expected calculation on a state and an action sampled from the offline data set , is a state of the robot, is an action of the robot, is a value of taking the action in the state when the parameter of the value network is , is an action generated in the state when the parameter of the policy network is , is a balance factor of a behavior cloning regularization term, is the behavior cloning regularization term, is a reference action in the state .
[0008] Further, the optimization target model of the value network has an expression as follows: Wherein: is a corresponding expected value when the parameter of the value network is , is an expected calculation on a state and an action Perform expectation calculation. The state of the robot, For the robot's actions, For state With action The corresponding reward function value, In the state Next action The next state that is reached later. When the parameters of the value network are At that time, in the state Take action below value, As a discount factor, When the parameters of the value network are At that time, in the state Take action value, For the parameters of the target value network, When the parameters of the policy network are At that time, in the state The actions generated below.
[0009] Furthermore, offline reinforcement learning is performed on the variational autoencoder based on the offline dataset to obtain the trained variational autoencoder. The specific process is as follows: The loss function of the variational autoencoder is constructed based on the reconstruction loss and KL divergence; The state-action pairs in the discrete dataset are input into the encoder in the variational autoencoder to obtain the latent variables; The latent variables are fed into the decoder in the variational autoencoder to obtain the reconstructed state-action pairs; Based on the loss function, state-action pairs, latent variables, and reconstructed state-action pairs of the variational autoencoder, backpropagation and parameter updates are performed on the variational autoencoder to obtain the trained variational autoencoder.
[0010] Furthermore, the loss function of the variational autoencoder is expressed as follows: in: When the encoder parameters in the variational autoencoder are The parameters of the decoder are When, input The subsequent loss function value, The state of the robot, For the robot's actions, To calculate the expected value of the latent variables in the encoder output, When the encoder parameters are When, input Latent variables obtained later , This is the difference between the state-action pair and the reconstructed state-action pair output by the decoder. The difference between the latent variable distribution and the prior distribution of the encoder output. For Kullback-Leibler divergence, Let be the prior distribution of the latent variables.
[0011] Furthermore, the uncertainty weights of the state-action pairs are obtained based on the trained variational autoencoder. Specifically, the process is as follows: based on the trained variational autoencoder and the state-action pairs, the negative log-likelihood of the loss function of the trained variational autoencoder is obtained; the negative log-likelihood of the loss function of the trained variational autoencoder is determined as the negative log-likelihood of the state-action pairs; and the uncertainty weights of the state-action pairs are obtained based on the negative log-likelihood of the state-action pairs.
[0012] Furthermore, the uncertainty weights of state-action pairs are obtained based on the negative log-likelihood of the state-action pairs, and their expression is as follows: , in: State-action pairs The normalized value of the negative log-likelihood. The state of the robot, For the robot's actions, It is a natural constant. State-action pairs The negative log-likelihood, State-action pairs Uncertainty weights, The threshold value set.
[0013] Furthermore, based on the uncertainty weights of state-action pairs, the robot's offline and online value networks are weighted and fused to obtain the robot's composite value network, the expression of which is: in: For composite value networks in state Next action value, The state of the robot, For the robot's actions, For state and actions The corresponding value function value, is a discount factor, is an uncertainty weight of the state-action pair , is a value of the online value network performing the action in the state , is a value of the offline value network performing the action in the state .
[0014] The beneficial effects of the present application are: (1) The present application accurately quantifies the uncertainty of each state-action pair by using a variational autoencoder, for low uncertainty areas, trust the frozen offline evaluation network, keep safe and stable; for high uncertainty areas, trust the online evaluation network, encourage exploration, which breaks the shackles of pessimistic bias, makes the policy network dare to try and learn better strategies, thereby breaking through the upper limit of offline data set performance and obtaining higher final task reward; (2) The present application initializes the online value network of the robot, freezes the parameters of the offline policy network of the robot, and uses the offline policy network of the robot to interact with the environment for greedy pre-sampling to obtain an online buffer dataset, and trains the initialized online value network of the robot based on the online buffer dataset to obtain the online value network of the robot. This process provides a reliable initial value for the online review network at a low cost, avoiding performance collapse and oscillation caused by unstable policy network and value network in the early stage, and then in the later process, the uncertainty weight of the state-action pair is obtained according to the trained variational autoencoder, and the offline value network and the online value network of the robot are weighted and fused based on the uncertainty weight of the state-action pair to obtain a composite value network of the robot, which enables the offline policy network to quickly absorb the knowledge of the new environment, and then enables each piece of data collected online to be used for learning efficiently, rather than being overwhelmed by the huge offline data, which reduces the training time and cost. BRIEF DESCRIPTION OF DRAWINGS
[0015] Figure 1 is a dynamic adaptive offline-to-online reinforcement learning method flowchart. DETAILED DESCRIPTION
[0016] The specific embodiments of the present application are described below to facilitate understanding of the present application by those skilled in the art, but it should be clear that the present application is not limited to the scope of the specific embodiments, and for those skilled in the art, it is obvious that various changes are within the spirit and scope of the present application defined and determined by the appended claims, and all applications utilizing the concept of the present application are within the scope of protection.
[0017] like Figure 1 As shown, a dynamic adaptive offline-to-online reinforcement learning method includes steps S1-S4, as detailed below: S1. Model the robot control problem as a Markov decision process to construct the robot's state space, action space, and reward function, and build an offline dataset based on the robot's state space, action space, and reward function.
[0018] In an optional embodiment of the present invention, the robot's state space comprises the robot's posture and joint angles in a time sequence; the robot's motion space comprises the robot's execution of control commands in a time sequence, including speed commands and torque commands. The robot's reward function is used to evaluate the merits of taking a certain action in a specific state; the present invention constructs the robot's reward function by weighting three indicators: task completion, energy consumption, and time efficiency; each state-action pair in the present invention has a reward function value.
[0019] S2. Based on the offline dataset, perform offline reinforcement learning on the actor-critic architecture and variational autoencoder respectively to obtain the robot's offline policy network, offline value network, and trained variational autoencoder.
[0020] In an optional embodiment of the present invention, the present invention performs offline reinforcement learning on the actor-critic architecture based on an offline dataset to obtain the robot's offline policy network and offline value network. The specific process is as follows: Construct optimization objective models for policy networks and value networks.
[0021] The optimization objective model of the policy network is expressed as follows: in: When the parameters of the policy network are The expected value corresponding to the time. For the offline dataset Mid-sampled state and actions Perform expectation calculation. The state of the robot, For the robot's actions, When the parameters of the value network are At that time, in the state Take action below value, When the parameters of the policy network are At that time, in the state The actions generated below, a balancing factor of the behavior cloning regularization term, a behavior cloning regularization term, a reference action in a state .
[0022] an optimization target model of the value network, and an expression thereof is: wherein: is a corresponding expected value when parameters of the value network are , is an expectation calculation on sampled from an offline data set , is a state of the robot, is an action of the robot, is a reward function value corresponding to the state and the action , is a next state reached after the action is performed in the state , is a value of taking the action in the state when parameters of the value network are , is a discount factor, is a value of taking the action in the state when parameters of the value network are , is a parameter of a target value network, is an action generated in the state when parameters of the policy network are .
[0023] Based on the offline data set, the actor-critic architecture is subjected to offline reinforcement learning by using an optimization target model of the policy network and an optimization target model of the value function, so as to obtain an offline policy network and an offline value network of the robot.
[0024] The present application is based on the offline data set to perform offline reinforcement learning on the variational autoencoder, so as to obtain the trained variational autoencoder, and the specific process is: A loss function of the variational autoencoder is constructed based on a reconstruction loss and a KL divergence, and an expression thereof is: wherein: is an input when parameters of an encoder in the variational autoencoder are and parameters of a decoder are , a loss function value after the loss function value, is a state of the robot, is an action of the robot, is an expectation calculation on a latent variable output by the encoder, is a latent variable obtained after the encoder's parameters are , input a latent variable obtained after the encoder's parameters are , is a difference between the state-action pair and the reconstructed state-action pair output by the decoder, is a difference between the latent variable distribution output by the encoder and the prior distribution, is a Kullback-Leibler divergence, is a prior distribution of the latent variable, and the prior distribution of the latent variable is a simple and fixed distribution specified in advance, and in the present application, the prior distribution of the latent variable is specified in advance as a standard normal distribution.
[0025] input the state-action pair in the discrete data set to the encoder in the variational autoencoder to obtain the latent variable.
[0026] input the latent variable to the decoder in the variational autoencoder to obtain the reconstructed state-action pair.
[0027] based on the loss function of the variational autoencoder, the state-action pair, the latent variable and the reconstructed state-action pair, the variational autoencoder is backpropagated and the parameters are updated to obtain the trained variational autoencoder.
[0028] S3, initialize the online value network of the robot, freeze the parameters of the offline policy network of the robot, and perform greedy pre-sampling by interacting with the environment using the offline policy network of the robot to obtain an online buffer data set, and train the initialized online value network of the robot based on the online buffer data set to obtain the online value network of the robot.
[0029] In an optional embodiment of the present application, the present application initializes the online value function of the robot, and the initialized online value function of the robot has the same structure as the offline value function of the robot.
[0030] The present application uses the offline policy network of the robot to interact with the environment to perform greedy pre-sampling to obtain an online buffer data set, and the specific process is as follows: selecting an action according to the offline policy network of the robot, with a probability performing limited random / noise exploration with a probability to obtain an online buffer data set, The probability parameters are set according to actual tasks. The online buffered data set obtained in the application not only maintains consistency with the offline data induced distribution, but also increases diversity within a controllable range.
[0031] The online value function of the initialized robot is trained based on the online buffered data set to obtain the online value function of the robot, and the specific process is as follows: the online value function of the initialized robot is trained based on the online buffered data set and the optimization target model of the value function to obtain the online value function of the robot. The application collects a consistent online buffered data set with the offline distribution by using a conservative behavior strategy first, and only trains the online value network of the robot, which avoids the error amplification caused by the strategy network of the robot being pushed out of the offline data set due to unreliable online value function estimation of the robot in the early stage, reduces the cold start risk, and provides a more reliable online value network of the robot for subsequent updating of the strategy network of the robot.
[0032] S4, the uncertainty weight of the state-action pair is obtained based on the trained variational autoencoder, and the offline value network and the online value network of the robot are weighted and fused based on the uncertainty weight of the state-action pair to obtain the composite value network of the robot, and the offline strategy network of the robot is updated based on the composite value network of the robot to complete the dynamic self-adaptive offline-to-online reinforcement learning method and obtain the final strategy network of the robot.
[0033] In an optional embodiment of the application, the uncertainty weight of the state-action pair is obtained based on the trained variational autoencoder, and the specific process is as follows: the negative log likelihood of the loss function of the trained variational autoencoder is obtained based on the trained variational autoencoder and the state-action pair; the negative log likelihood of the loss function of the trained variational autoencoder is determined as the negative log likelihood of the state-action pair; and the uncertainty weight of the state-action pair is obtained based on the negative log likelihood of the state-action pair, and its expression is as follows: , wherein: is the negative log likelihood of the state-action pair , is the state of the robot, is the action of the robot, is a natural constant, is the negative log likelihood of the state-action pair , is the uncertainty weight of the state-action pair , is a set threshold value, The value of the state-action pair is set according to actual tasks.
[0034] The offline value network and the online value network of the robot are weighted and fused based on the uncertainty weight of the state-action pair to obtain a composite value network of the robot, and the expression is: wherein: is the value of the composite value network in the state executing the action , is the state of the robot, is the action of the robot, is the value function value corresponding to the state and the action , is a discount factor, is the uncertainty weight of the state-action pair , is the value of the online value network in the state executing the action , is the value of the offline value network in the state executing the action .
[0035] The offline policy network of the robot is updated based on the composite value network of the robot to complete a dynamic adaptive offline-to-online reinforcement learning method, and finally obtain a policy network of the robot, and the specific process is: calculating the action output by the offline policy network of the robot; based on the action output by the offline policy network of the robot, updating the parameters of the offline policy network of the robot by maximizing the output of the composite value network of the robot, to complete a dynamic adaptive offline-to-online reinforcement learning method, and finally obtain a policy network of the robot. Through this dynamic adaptive method, continuous optimization is realized, and finally a final policy network of the robot is obtained, which is safe and reliable in a real environment and can efficiently complete complex tasks.
[0036] Those skilled in the art will appreciate that the embodiments described herein are presented for the purpose of helping the reader understand the principles of the present application, and should be understood as not limiting the scope of protection of the present application to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations according to the technical inspiration disclosed in the present application without departing from the essence of the present application, and these modifications and combinations are still within the scope of protection of the present application.
Claims
1. A dynamically adaptive offline-to-online reinforcement learning method, characterized in that, Includes the following steps: The robot control problem is modeled as a Markov decision process to construct the robot's state space, action space, and reward function, and an offline dataset is built based on the robot's state space, action space, and reward function. Offline reinforcement learning was performed on the actor-critic architecture and variational autoencoder based on the offline dataset to obtain the robot's offline policy network, offline value network, and trained variational autoencoder. Initialize the robot's online value network, freeze the parameters of the robot's offline policy network, and use the robot's offline policy network to perform greedy presampling with the environment to obtain an online buffer dataset. Then, train the initialized robot's online value network based on the online buffer dataset to obtain the robot's online value network. The uncertainty weights of state-action pairs are obtained based on the trained variational autoencoder. The robot's offline value network and online value network are then weighted and fused based on these uncertainty weights to obtain the robot's composite value network. The robot's offline policy network is then updated based on the composite value network to complete the dynamic adaptive offline-to-online reinforcement learning method and obtain the final robot policy network.
2. The dynamically adaptive offline-to-online reinforcement learning method according to claim 1, characterized in that, Offline reinforcement learning is performed on the actor-critic architecture based on an offline dataset to obtain the robot's offline policy network and offline value network. The specific process is as follows: Construct optimization objective models for policy networks and value networks; Based on an offline dataset, offline reinforcement learning is performed on the actor-critic architecture using the optimization objective model of the policy network and the optimization objective model of the value function to obtain the robot's offline policy network and offline value network.
3. The dynamically adaptive offline-to-online reinforcement learning method according to claim 2, characterized in that, The optimization objective model of the policy network is expressed as follows: in: When the parameters of the policy network are The expected value corresponding to the time. For the offline dataset Mid-sampled state and actions Perform expectation calculation. The state of the robot, For the robot's actions, When the parameters of the value network are At that time, in the state Take action below value, When the parameters of the policy network are At that time, in the state The actions generated below, The balance factor for the behavioral clonal regularization term. For behavior cloning regularization, In the state The following are reference actions.
4. The dynamically adaptive offline-to-online reinforcement learning method according to claim 2, characterized in that, The optimization objective model of the value network is expressed as follows: in: When the parameters of the value network are The expected value corresponding to the time. For the offline dataset Mid-sampling Perform expectation calculation. The state of the robot, For the robot's actions, For state With action The corresponding reward function value, In the state Next action The next state that is reached later. When the parameters of the value network are At that time, in the state Take action below value, As a discount factor, When the parameters of the value network are At that time, in the state Take action value, For the parameters of the target value network, When the parameters of the policy network are At that time, in the state The actions generated below.
5. The dynamically adaptive offline-to-online reinforcement learning method according to claim 1, characterized in that, Offline reinforcement learning is performed on the variational autoencoder based on an offline dataset to obtain the trained variational autoencoder. The specific process is as follows: The loss function of the variational autoencoder is constructed based on the reconstruction loss and KL divergence; The state-action pairs in the discrete dataset are input into the encoder in the variational autoencoder to obtain the latent variables; The latent variables are fed into the decoder in the variational autoencoder to obtain the reconstructed state-action pairs; Based on the loss function, state-action pairs, latent variables, and reconstructed state-action pairs of the variational autoencoder, backpropagation and parameter updates are performed on the variational autoencoder to obtain the trained variational autoencoder.
6. The dynamically adaptive offline-to-online reinforcement learning method according to claim 5, characterized in that, The loss function of the variational autoencoder is expressed as follows: in: When the encoder parameters in the variational autoencoder are The parameters of the decoder are When, input The subsequent loss function value, The state of the robot, For the robot's actions, To calculate the expected value of the latent variables in the encoder output, When the encoder parameters are When, input Latent variables obtained later , This is the difference between the state-action pair and the reconstructed state-action pair output by the decoder. The difference between the latent variable distribution and the prior distribution of the encoder output. For Kullback-Leibler divergence, Let be the prior distribution of the latent variables.
7. The dynamically adaptive offline-to-online reinforcement learning method according to claim 1, characterized in that, The uncertainty weights of state-action pairs are obtained based on the trained variational autoencoder. Specifically, the negative log-likelihood of the loss function of the trained variational autoencoder is obtained based on the trained variational autoencoder and the state-action pairs. The negative log-likelihood of the loss function of the trained variational autoencoder is determined as the negative log-likelihood of the state-action pairs. The uncertainty weights of the state-action pairs are obtained based on the negative log-likelihood of the state-action pairs.
8. The dynamically adaptive offline-to-online reinforcement learning method according to claim 7, characterized in that, The uncertainty weights of state-action pairs are obtained based on the negative log-likelihood of the state-action pairs, and their expression is as follows: ， in: State-action pairs The normalized value of the negative log-likelihood. The state of the robot, For the robot's actions, It is a natural constant. State-action pairs The negative log-likelihood, State-action pairs Uncertainty weights, The threshold value set.
9. The dynamically adaptive offline-to-online reinforcement learning method according to claim 1, characterized in that, The robot's offline and online value networks are weighted and fused based on the uncertainty weights of state-action pairs to obtain the robot's composite value network, which is expressed as follows: in: For composite value networks in state Next action value, The state of the robot, For the robot's actions, For state and actions The corresponding value function value, As a discount factor, State-action pairs Uncertainty weights, For online value networks in state Next action value, For offline value networks in state Next action The value of.