Model-based offset sensing agent offline reinforcement learning method and device

Abstract

The invention provides a model-based offset sensing agent offline reinforcement learning method and device. The method comprises the steps of obtaining an agent offline data set; training an integrated model according to the offline data set; replacing the environment where the intelligent agent is located according to the integrated model, and generating synthetic data; extracting samples from the offline data set and the synthetic data, and determining an offset perception reward according to a state transition classifier and a strategy classifier; wherein the state transition classifier is used for compensating difference between prediction of the integrated model and a real environment, and the strategy classifier is used for compensating distribution offset caused by strategy updating; and according to a sample corresponding to the offset perception reward, updating an intelligent agent operation strategy by adopting an SAC algorithm. According to the method, the problem of data distribution mismatching in a real environment and a model is solved by uniformly processing the offline data set and the synthetic data from a probabilistic reasoning framework.

Description

Technical Field

[0001] This disclosure relates to the field of offline reinforcement learning technology, and in particular to a model-based offset-aware agent offline reinforcement learning method and apparatus. Background Technology

[0002] Offline reinforcement learning learns policies from offline datasets generated from behavioral policies, thus avoiding online interaction with the environment. This approach shows great potential in data-driven decision-making scenarios, suitable for exploring costly or risky environments, and is an important technique demonstrating practical significance, widely used in fields such as robot control and autonomous driving. Model-based frameworks are particularly effective for offline reinforcement learning. It learns environmental models from offline datasets and uses these models to generate synthetic data for policy training. However, directly using offline datasets and synthetic data to learn policies introduces the challenge of distribution shift. This shift causes the training objective to deviate from the true objective of reinforcement learning, resulting in poor test performance.

[0003] Traditional model-based offline reinforcement learning methods focus on directly exploiting policy shifts in online reinforcement learning and heuristically leveraging model uncertainty to mitigate the effects of assignment bias. However, these methods result in a biased reinforcement learning training objective that addresses only the distributional shifts caused by model inaccuracies, without addressing the broader problem of distribution bias. Summary of the Invention

[0004] This disclosure provides a model-based offset-aware agent offline reinforcement learning method and apparatus to solve the distribution offset problem in existing model-based agent offline reinforcement learning processes.

[0005] In view of the above problems, firstly, this disclosure provides a model-based offset-aware agent offline reinforcement learning method, including: Obtain the offline dataset of the intelligent agent; Train the ensemble model based on the offline dataset; Based on the integrated model, synthetic data is generated to represent the environment in which the agent exists; Samples are extracted from the offline dataset and synthetic data, and offset-aware rewards are determined based on a state transition classifier and a policy classifier; wherein the state transition classifier is used to compensate for the difference between the prediction of the ensemble model and the real environment, and the policy classifier is used to compensate for the distribution offset caused by policy updates. Based on the samples corresponding to the offset-aware rewards, the SAC algorithm is used to update the agent's operating strategy.

[0006] Secondly, a model offset-aware offline reinforcement learning device for intelligent agents is provided, comprising: The offline dataset acquisition module is used to acquire offline datasets for the intelligent agent. An ensemble model training module is used to train an ensemble model based on the offline dataset; A synthetic data generation module is used to generate synthetic data based on the integrated model to represent the environment in which the agent is located; The offset-aware reward determination module is used to extract samples from the offline dataset and synthetic data, and determine the offset-aware reward based on the state transition classifier and the policy classifier; wherein, the state transition classifier is used to compensate for the difference between the prediction of the ensemble model and the real environment, and the policy classifier is used to compensate for the distribution offset caused by the policy update. The policy update module is used to update the agent's policy based on the sample corresponding to the offset-aware reward using the SAC algorithm.

[0007] The beneficial effects of the embodiments disclosed herein include: This disclosure provides a model-based offset-aware agent offline reinforcement learning method and apparatus, comprising: acquiring an offline dataset of the agent; training an ensemble model based on the offline dataset; generating synthetic data by using the ensemble model to represent the agent's environment; extracting samples from the offline dataset and synthetic data, and determining offset-aware rewards based on a state transition classifier and a policy classifier; wherein the state transition classifier is used to compensate for the difference between the ensemble model's prediction and the real environment, and the policy classifier is used to compensate for the distribution offset caused by policy updates; and updating the agent's policy using the SAC algorithm based on the samples corresponding to the offset-aware rewards. The model-based offset-aware agent offline reinforcement learning method provided in this disclosure addresses the problem of data distribution mismatch between the real environment and the model by uniformly processing offline datasets and synthetic data from a probabilistic inference framework. It more broadly solves the distribution offset problem, thereby ensuring that the policy trained using model-generated data can be effectively deployed in the real environment. It effectively solves the problem of policy performance degradation caused by distribution offset in offline reinforcement learning, and has significant value for applications in real-world scenarios with historical data where the agent cannot interact with the environment. Attached Figure Description

[0008] Figure 1 A flowchart illustrating a model-based offset-aware agent offline reinforcement learning method provided in this embodiment of the disclosure; Figure 2 This is a structural diagram of a model-based offset-aware agent offline reinforcement learning device provided in an embodiment of this disclosure. Detailed Implementation

[0009] This disclosure provides a model-based offset-aware agent offline reinforcement learning method and apparatus. Preferred embodiments of this disclosure are described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustrative and explanatory purposes only and are not intended to limit this disclosure. Furthermore, the embodiments and features described in this application can be combined with each other unless otherwise specified.

[0010] This disclosure provides a model-based offset-aware agent offline reinforcement learning method, such as... Figure 1 As shown, it includes: S101. Obtain the offline dataset of the intelligent agent; S102. Train the ensemble model based on the offline dataset; S103. Generate synthetic data by substituting the environment of the agent with the integrated model; S104. Samples are drawn from offline datasets and synthetic data, and offset-aware rewards are determined based on state transition classifiers and policy classifiers; wherein, the state transition classifier is used to compensate for the difference between the prediction of the ensemble model and the real environment, and the policy classifier is used to compensate for the distribution offset caused by policy updates. S105. Based on the samples corresponding to the offset-aware reward, update the agent's operating strategy using the SAC algorithm.

[0011] In this disclosure, within the fields of artificial intelligence, reinforcement learning, and intelligent control, an agent can refer to an entity that perceives its environment and makes decisions to achieve a specific goal. Examples include robots, non-player characters in video games, and virtual assistants. Reinforcement learning, as one of the three core paradigms of machine learning, has achieved remarkable success in many complex tasks, such as video games and Go. Furthermore, reinforcement learning has shown great potential in practical applications requiring decision-making, such as robot control, autonomous driving, healthcare, and news recommendation. Offline reinforcement learning refers to learning policies from offline datasets generated from behavioral policies, thereby avoiding online interaction with the environment. Offline reinforcement learning shows great potential in data-driven decision-making scenarios, is suitable for exploring costly or risky environments, and is an important technology demonstrating practical significance, widely used in robot control, autonomous driving, and other fields. Model-based frameworks are particularly effective for agent-based offline reinforcement learning. Model-based agent-based offline reinforcement learning learns environmental models from offline datasets and uses these environmental models to generate synthetic data for policy training. However, directly using offline datasets and synthetic data to learn policies introduces the challenge of distribution shift. Distribution shift refers to the problem where the distribution of states and actions accessed in the real environment is biased due to differences between the offline and synthetic datasets used for training and the actual environment in which the agent operates. This shift causes the training objective to deviate from the true objective of reinforcement learning, resulting in poor test performance. Traditional model-based offline reinforcement learning methods focus on directly utilizing policy deviations from online reinforcement learning and heuristically leveraging model uncertainty to mitigate the effects of distribution shift. However, these methods lead to a biased reinforcement learning training objective that only addresses distribution variations caused by model inaccuracies, without addressing the broader distribution shift problem.

[0012] A Markov Decision Process (MDP) defines a six-tuple. ,in For state space, For the action space, [0,1] is the reward function. For environmental transfer dynamics, among which To define in state space probability space on As a discount factor, Let the initial state distribution be given. Given an MDP, for any policy... A state value function can be defined. and action value function ( function) ,in Representation strategy, Indicates state, Indicates an action, Indicates a time step. This expresses expectation. The training objective of reinforcement learning is to find a strategy. To maximize cumulative returns The optimal strategy is defined as the strategy that maximizes cumulative profit, denoted as: , Satisfies the Bellman equation: ,in This represents the optimal action value function. Indicates the next state. This indicates the next action. However, offline reinforcement learning agents can only access offline datasets. Unable to access environment transfer dynamics Therefore, due to the inaccuracy of neural network fitting, the agent is trained using the Bellman equation. Functions are often overestimated The value of is difficult to accurately optimize the strategy, which in turn leads to performance crashes during actual deployment.

[0013] In this embodiment, the offline dataset refers to a batch of complete experience samples pre-collected and stored under the premise that the agent no longer interacts with the environment, which can be collected in the environment through behavioral strategies. The algorithms in this disclosure can be implemented using Datasets for Deep Data-Driven Reinforcement Learning (D4RL). For example, taking a robot as the agent, to achieve continuous control of the robot, offline data collected from the Multi-Joint Dynamics with Contact (MuJoCo) simulation environment of the OpenAI reinforcement learning environment library (Gym) is used to optimize the strategy, including high-quality to low-quality offline datasets. The state space consists of physical information related to the specific task, such as the position, velocity, and angular velocity of each part of the robot, as well as the angle and angular velocity of each joint. Action space... For continuous motion space, represent the physical variables that can be used to control the task, such as the applied torque of each hinge joint. Reward function. An ensemble model is used to measure the reward an agent receives for taking a certain action in a specific state. Its design depends on the specific task and varies across different environments. For example, in tasks such as walking and jumping for robots, the reward function typically consists of positive rewards that encourage a healthy state and forward movement, and negative rewards that penalize excessive movements or high friction. The ensemble model can be trained using an offline dataset and may include... Each of the three probabilistic dynamic transition models is used to fit the environmental transition dynamics, thereby providing an uncertainty measure through error averaging and covariance, and improving prediction accuracy. The ensemble model can learn the environmental transition dynamics from offline datasets, making the prediction results as close as possible to the next state and reward in the real environment. The ensemble model is used to replace the agent's environment to generate synthetic data. For example, it can be used to launch in parallel according to the current policy. b 1 track, length of each track h The ensemble model is used to recursively deduce the next state and reward. New "state, action, reward, next state" quadruples are generated, which are then considered synthetic data. Synthetic data can supplement the deficiencies of offline data and reduce reliance on real-world data collection.

[0014] Furthermore, starting from an unbiased offline reinforcement learning training objective, the distribution shift problem in model-based offline reinforcement learning is decomposed using a probabilistic inference framework. It is shown that the distribution shift problem stems not only from the inconsistency between the model and the environment's predictions of subsequent states, but also from policy changes during the learning process. Mitigating distribution shift requires addressing both issues simultaneously. Based on this, a shift-aware reward is designed to replace the original reinforcement learning reward, and it is proven that the offline reinforcement learning alternative objective constituted by the shift-aware reward is a lower bound of the true optimization objective of reinforcement learning, thus realizing a model-based shift-aware agent offline reinforcement learning method. A state transition classifier and a policy classifier are set up. The state transition classifier compensates for the difference between the ensemble model's predictions and the real environment, while the policy classifier compensates for the distribution shift caused by policy updates. The state transition classifier represents the probability that a transition sample is generated by the environment, and the policy classifier represents the probability that a given state and action pair is generated by the current policy. Samples are drawn from the offline dataset and synthetic data, and the shift-aware reward is determined based on the state transition classifier and the policy classifier. By compensating for the shift, the reliability of the reward is ensured, avoiding degradation in policy training. The SAC algorithm can refer to an offline algorithm based on maximum entropy reinforcement learning. Its core principle is to maximize the cumulative reward while maintaining the randomness of the policy by training the Q-function and policy. Samples with offset-aware rewards are input into the SAC algorithm, and the agent's policy is iteratively updated by minimizing the Q-value loss and policy loss.

[0015] This application's embodiments address the data distribution mismatch between real-world environments and models by uniformly processing offline datasets and synthetic data within a probabilistic inference framework. It more broadly resolves the distribution shift problem, ensuring that policies trained using model-generated data can be effectively deployed in real-world environments. This effectively solves the performance degradation issue caused by distribution shift in offline reinforcement learning, and is of significant value for applications in real-world scenarios with historical data where agents cannot interact with the environment.

[0016] In another embodiment of this disclosure, step S102 above, training the ensemble model based on the offline dataset, includes: Step 1: Extract state-action pairs from the offline dataset and input them into a Gaussian model. The model outputs the probability distribution of the next state and reward. The Gaussian model is trained using maximum likelihood estimation, as shown in the formula: ; in, Represents the parameters of the Gaussian model. Indicates the time step, ( () represents the time step extracted from the offline dataset. The state and actions are correct. Indicates the next time step state, Indicates a reward. Indicates at time step State and action pair ( Under the condition of ), the next time step state and rewards The probability distribution, Indicates a Gaussian distribution. This represents the mean of a Gaussian distribution. Represents the covariance matrix of the Gaussian distribution; Step 2: The ensemble model determined by the Gaussian model is expressed as follows: ,in, This represents a probability dynamic transition model. This indicates the number of probabilistic dynamic transition models in the ensemble model. Indicates the first The formula for any probability dynamic transition model is as follows: ; in, Represents the parameters of the Gaussian model, ( () represents any state and action pair. Indicates the next state. Indicates a reward. This indicates that in any state and action pair ( Under the condition of ), the next state and rewards The probability distribution is used to fit the dynamics of environmental transitions. Indicates a Gaussian distribution. This represents the mean of a Gaussian distribution. Let represent the covariance matrix of the Gaussian distribution.

[0017] In this embodiment, a Gaussian model is trained using maximum likelihood estimation to construct an ensemble model that accurately fits the dynamics of environmental transitions, achieving a comprehensive fit to the environmental dynamics. Regarding step 1 above, data is obtained from an offline dataset... Train a Gaussian model To fit the dynamics of environmental transfer The Gaussian model input at time step State and action Output the next time step. state and rewards The Gaussian distribution effectively characterizes the uncertainty of state transitions in a continuous state space. The mean and covariance of the Gaussian distribution can quantify the predicted central tendency and uncertainty, facilitating the control of sample diversity when generating synthetic data, thus better meeting the reinforcement learning requirement for rich and reliable synthetic data. The Gaussian model is trained using Maximum Likelihood Estimation (MLE). Maximum Likelihood allows for adjustments to the Gaussian model parameters. To best fit the observations in the offline dataset, that is, for the samples in the offline dataset, to make the Gaussian model predict... exist( Maximizing probability under certain conditions. Through maximum likelihood training, the Gaussian model learns the state transition patterns in the offline dataset, providing high-quality basic prediction units for subsequent ensemble models. For step 2 above, a probabilistic dynamic transition model is obtained based on the Gaussian model. The trained Gaussian model can be used as the probabilistic dynamic transition model. The ensemble model includes... An ensemble model is a probabilistic dynamic transition model. In this model, the input state and action pair can be any pair, no longer limited to offline datasets, and can cover a wider range of scenarios. The ensemble model can fully fit the dynamics of the environment and simultaneously predict the next state. and rewards It covers both the "state feedback" and "reward feedback" of the environment to the agent's actions, more closely resembling the interaction logic of the real environment. Furthermore, through multi-model integration, it can offset the local errors of a single probability dynamic transfer model. For example, it can offset the overfitting of a certain probability dynamic transfer model to a specific scene, making the prediction results more stable and providing a reliable virtual environment for the subsequent generation of synthetic data.

[0018] In another embodiment of this disclosure, it further includes: initializing the agent's policy, policy exit length, and policy exit batch size; In step S103 above, synthetic data is generated by replacing the environment of the agent with the integrated model, including: Step 1: Determine the batch size based on the strategy, and sample multiple states from the offline dataset as initial states; Step 2: For each initial state, based on the policy's derivation length, sample multiple actions from the current policy; Step 3: For each action, randomly select a probability dynamic transition model from the ensemble model, input the state and action pair corresponding to each action, and output the next state and reward. Step 4: Use the state and action pair corresponding to each action, as well as the next state and reward corresponding to each action, as samples to determine the synthetic data.

[0019] In this embodiment, starting from the actual initial state in the offline dataset, and combining it with the current policy to generate actions, synthetic data that meets the policy requirements is generated in batches by simulating environmental responses through integrated model diversity, thus effectively supplementing the offline data. The agent's policy can refer to the agent's current behavioral model, and the policy can be represented as... π This determines the basis for generating actions in the synthetic data, ensuring that the actions conform to the behavioral patterns of the intelligent system. The policy delivery length is represented as... h It can refer to the number of consecutive action steps generated in a single initial state (e.g., T (Step), that is, starting from the initial state, generating including T A complete trajectory segment of "state, action, reward, next state" is used to ensure the temporal continuity of the synthetic data. The policy exit batch size is represented as... b This can refer to the number of initial states generated at one time, i.e., the number of states generated simultaneously. b Independent trajectories are used to achieve batch production of synthetic data, improving generation efficiency. For step 1 above, random samples are taken from the offline dataset. b Each state is used as the initial state. Uniform sampling covers a wider range of scenarios, avoiding overly extreme initial states. For step 2 above, for each initial state, the length is derived according to the strategy. h Continuous sampling from the current strategy hEach action involves several steps. For step 3 above, an ensemble model is used to simulate feedback from a real-world environment. For each action, a probabilistic dynamic transition model is randomly selected from the ensemble model. The input is the state and action pair corresponding to each action, and the output is the next state and reward. For step 4 above, the simulated interaction process is organized into standardized samples. The state and action pair corresponding to each action, as well as the next state and reward corresponding to each action, are used as samples to obtain synthetic data. Initial states are sampled from the offline dataset to anchor the starting point of the synthetic data to the actual interaction scenarios of the agent, avoiding the generation of invalid data detached from the real environment and ensuring the relevance of the synthetic data to the actual task. A probabilistic dynamic transition model is randomly selected from the ensemble model to generate feedback. The differences between multiple models increase the diversity of the synthetic data while offsetting the prediction bias of a single probabilistic dynamic transition model, improving the reliability of the synthetic data. By controlling the batch size and length of the policy exit, the number of samples and the integrity of the trajectory are controlled, allowing for the batch generation of multi-step interactive synthetic data. This quickly compensates for insufficient offline datasets or incomplete scenario coverage, reducing reliance on real-world interactions.

[0020] In another embodiment of this disclosure, the synthetic data includes: a model dataset, which is used to characterize the synthetic data generated through the ensemble model; The state transition classifier is trained as follows: Based on the model dataset and the offline dataset, a state transition classifier is trained by constructing a function that minimizes the cross-entropy loss function, expressed as follows: ;

[0021] in, Represents the cross-entropy loss function. The parameters represent the state transition classifier. Represents a state transition classifier. Indicates state, Indicates an action, Indicates the next state. Indicates the state ,action Next state In this case, the probability that the sample is generated by the environment. Represents an offline dataset. This represents the average of the samples in the offline dataset. Represents the model dataset, This represents the average of the samples in the model dataset. This indicates an update operation. This represents an optimization algorithm used to minimize the cross-entropy loss function. The cross-entropy loss function is expressed with respect to the parameters. The gradient.

[0022] In this embodiment, using an offline dataset and a model dataset, the state transition classifier learns to distinguish sample sources by minimizing the cross-entropy loss function, ultimately obtaining a state transition classifier that can quantify the difference between the ensemble model's predictions and the real environment. The synthetic data includes a model dataset, which characterizes the synthetic data generated by the ensemble model. The synthetic data can be input into a model dataset buffer to obtain the model dataset. The simulated dataset and the synthetic data can be the same. The loss representing real-world data can be used to measure the prediction accuracy of the state transition classifier for real-world data. For real-world data, we hope... The predicted value is close to 1, indicating that these samples are considered to be generated by the environment. Therefore, this part of the loss encourages... It provides a high probability for real-world environmental data. This represents the loss on the model-generated data, used to measure the prediction accuracy of the state transition classifier for that data. For the model-generated data, we hope... The predicted value is close to 0, meaning these samples are considered not to have been generated by the environment. Therefore, this part of the loss encourages... The ensemble model generates synthetic data with low probabilities. This is done to minimize the cross-entropy loss function. ,use Optimize the algorithm to update the parameters of the state transition classifier . The optimization algorithm is an adaptive learning rate optimization algorithm that combines the characteristics of momentum and adaptive learning rate, and can effectively handle problems with non-stationary objectives and sparse gradients. In each iteration, The optimization algorithm is based on the gradient of the loss function. To update parameters To minimize the loss function. State transition classifier It can learn to distinguish between real-world data and model-generated data, thus providing an accurate estimate for calculating offset-aware rewards. This helps the algorithm better handle distribution offset problems in offline reinforcement learning for agents.

[0023] In another embodiment of this disclosure, the synthetic data includes: a policy dataset, which is used to characterize the data collected by the current policy in the ensemble model; The policy classifier is trained in the following manner: Based on the policy dataset and the offline dataset, a loss function is constructed to train the policy classifier, as expressed by the formula: =

[0024]

[0025]

[0026]

[0027] in, Represents the policy classifier, The parameters represent the policy classifier. Indicates state, Indicates an action, Indicates the state ,action In this case, the probability that a sample is generated by the current policy, and the policy classifier includes a first policy classifier and a second policy classifier. This represents the first policy classifier. This represents the second policy classifier. This represents the parameters of the first policy classifier. This represents the parameters of the second policy classifier. This represents the loss function of the first policy classifier. This represents the loss function of the second policy classifier. Represents the policy dataset, This represents the average of the samples in the policy dataset. Represents an offline dataset. This represents the average of the samples in the offline dataset. This indicates an update operation. This represents an optimization algorithm used to minimize the loss function. The loss function of the first policy classifier with respect to the parameters is... gradient, The loss function of the second policy classifier with respect to the parameters The gradient.

[0028] In this embodiment of the disclosure, a cross-entropy loss function is constructed using a first policy classifier and a second policy classifier, utilizing the policy dataset and the offline dataset. The optimization algorithm jointly optimizes the parameters of the first and second policy classifiers, ultimately enabling the policy classifiers to accurately determine whether a sample was generated by the current policy and quantify the distribution shift caused by policy updates. The policy dataset is used to represent the data collected in the ensemble model by the current policy. The policy dataset can be stored in a policy dataset buffer. The first policy classifier can output the probability that state and action pairs were generated by the policy. The second policy classifier can output only the probability that a state was generated by the policy. The difference between the log-odds of the first and second policy classifiers is then used to obtain the corrected probability ratio. This allows for both state and action pairs, and state-level analysis, capturing finer-grained distribution shifts. A loss function is constructed to minimize the policy classifier during training. Cross-entropy loss ensures that the policy classifier converges to the optimal probability estimate. Optimization algorithm update It features an adaptive learning rate, ensuring stable and fast training. Once training is complete, it can be used with any... It provides real-time quantified values ​​of policy shifts, offering crucial input for subsequent reward adjustments. Through a dual-classifier design, it effectively identifies whether samples originate from the current policy or an offline dataset, providing a core tool for quantifying distribution shifts caused by policy updates.

[0029] In another embodiment of this disclosure, the synthetic data includes: a model dataset, which is used to characterize the synthetic data generated through the ensemble model; In step S104 above, samples are extracted from the offline dataset and synthetic data, and the offset-aware reward is determined based on the state transition classifier and policy classifier, expressed by the formula: ; in, Indicates state, Indicates an action, Indicates the next state. This indicates offset-perceived reward. This indicates the reward corresponding to the state and action. Represents a state transition classifier. Represents the policy classifier, and Indicates a sample, Represents the model dataset, Represents an offline dataset. and This represents the hyperparameter that controls the level of reward.

[0030] In this embodiment of the disclosure, a state transition classifier is used for samples from different sources (offline datasets and synthetic data). and strategy classifier The output of the original reward Targeted adjustments were made to ultimately obtain a offset-aware reward that could compensate for both "model-environment offset" and "policy update offset". When the samples come from the model dataset, the formula is used. ; State transition classifier Used to compensate for the difference between the predictions of the ensemble model and the real environment.

[0031] When the samples come from an offline dataset, the formula is used. ; A policy classifier is used to compensate for the distribution shift caused by policy updates. The adjusted offset-aware reward more closely reflects the joint distribution of the real environment and the current policy, providing a more effective guidance signal for subsequent policy updates in the SAC algorithm, thereby improving training stability and final policy performance.

[0032] In another embodiment of this disclosure, in step S104 above, samples are extracted from the offline dataset and synthetic data, and the offset-aware reward is determined based on the state transition classifier and the policy classifier, expressed by the formula: ; in, Indicates state, Indicates an action, Indicates the next state. This indicates offset-perceived reward. This indicates the reward corresponding to the state and action. Represents a state transition classifier. Represents the policy classifier, and This refers to samples drawn from offline datasets or synthetic data. and This represents the hyperparameter that controls the level of reward.

[0033] In this embodiment, for samples from synthetic and offline data, the outputs of the state transition classifier and the policy classifier are simultaneously fused using a general formula to jointly adjust the original reward, ultimately obtaining a offset-aware reward that can simultaneously compensate for both "model-environment offset" and "policy update offset". Offset perception reward It can be estimated using the formula: ; in, and These are two hyperparameters used to control the degree of reward adjustment. Represents the environment transfer function, This represents the learned transfer function model. This is the current strategy. This is used for training data generation strategies. In actual training, the environment transfer function... and data collection strategies It is usually unknown and cannot be calculated precisely. Therefore, a state transition classifier and a policy classifier are used to estimate it. and . Represents a state transition classifier. Indicates the state ,action Next state In this case, the probability that the sample is generated by the environment. Represents the policy classifier, Indicates the state ,action In this case, the probability that a sample is generated by the current policy. This leads to the following formula: ; This formula incorporates adjustment terms for both the state transition classifier and the policy classifier, compensating for the discrepancy between the ensemble model's predictions and the real environment, as well as the distribution shift caused by policy updates, in a single calculation. This avoids the complexity of handling different scenarios, improves the comprehensiveness of reward adjustment, and reduces computational complexity. It utilizes unbiased offline reinforcement learning objective theory and a shift-aware reward design. A probabilistic inference framework demonstrates that mitigating distribution shift requires addressing not only the bias between the model and the environment, but also... The differences also require handling the variations between training data and policies, i.e. Based on this, a offset-aware reward is derived, and the alternative optimization objective constructed by it serves as an effective lower bound for the true reinforcement learning optimization objective. The offset-aware reward more accurately reflects the reward signal under the influence of distribution offset, effectively mitigating training bias caused by model bias and policy bias, and promoting the effective online deployment of offline training strategies.

[0034] In another embodiment of this disclosure, the method further includes: Step 1: Initialize the maximum number of iterations; Step 2: Based on the maximum number of iterations, iteratively update the agent's operating strategy, including: generating synthetic data by replacing the agent's environment with the ensemble model; extracting samples from the offline dataset and synthetic data, and determining the offset-aware reward based on the state transition classifier and policy classifier; updating the agent's operating strategy using the SAC algorithm based on the samples corresponding to the offset-aware reward; returning to the step of generating synthetic data by replacing the agent's environment with the ensemble model, until the maximum number of iterations is reached, and determining the final operating strategy of the agent.

[0035] In this embodiment, by setting a maximum number of iterations, the process of "generating synthetic data, calculating offset-aware rewards, and updating the policy using the SAC algorithm" is repeated in each iteration until the maximum number of iterations is reached, ultimately yielding the optimized agent policy. For step 1, setting a maximum number of iterations for policy training provides a clear termination condition for the iteration process, avoiding infinite iterations or premature termination. For step 2, steps S103, S104, and S105 are repeated in each iteration until the maximum number of iterations is reached, at which point the final policy is output. Through multiple iterations, the policy can be gradually optimized based on continuously updated synthetic data and offset-aware rewards, avoiding the limitations of single-step training and ultimately converging to a better policy. During the iteration process, the generation of synthetic data depends on the ensemble model, reward adjustment depends on the classifier, policy optimization drives the improvement of synthetic data quality, and high-quality synthetic data, in turn, feeds back into policy optimization, improving the overall training effect.

[0036] Based on the same disclosed concept, this disclosure also provides a model-based offset-aware agent offline reinforcement learning device. Since the principle of the problem solved by these devices is similar to that of the aforementioned model-based offset-aware agent offline reinforcement learning method, the implementation of this device can refer to the implementation of the aforementioned method, and the repeated parts will not be described again.

[0037] This disclosure provides an offline reinforcement learning device for an agent with model offset awareness, such as... Figure 2 As shown, it includes: Offline dataset acquisition module 201 is used to acquire offline datasets for the intelligent agent; The ensemble model training module 202 is used to train the ensemble model based on the offline dataset; The synthetic data generation module 203 is used to generate synthetic data based on the integrated model to represent the environment in which the agent is located; The offset-aware reward determination module 204 is used to extract samples from the offline dataset and synthetic data, and determine the offset-aware reward based on the state transition classifier and the policy classifier; wherein the state transition classifier is used to compensate for the difference between the prediction of the ensemble model and the real environment, and the policy classifier is used to compensate for the distribution offset caused by the policy update. The policy update module 205 is used to update the policy of the agent based on the sample corresponding to the offset perception reward using the SAC algorithm.

[0038] In another embodiment of this disclosure, the integrated model training module 202 is used to extract state-action pairs from the offline dataset and input them into a Gaussian model, outputting the probability distribution of the next state and reward, and training the Gaussian model using maximum likelihood estimation, as expressed by the formula: ; in, Represents the parameters of the Gaussian model. Indicates the time step, ( () represents the time step extracted from the offline dataset. The state and actions are correct. Indicates the next time step state, Indicates a reward. Indicates at time step State and action pair ( Under the condition of ), the next time step state and rewards The probability distribution, Indicates a Gaussian distribution. This represents the mean of a Gaussian distribution. Represents the covariance matrix of the Gaussian distribution; The ensemble model determined based on the Gaussian model is expressed as follows: ,in, This represents a probability dynamic transition model. This indicates the number of probabilistic dynamic transition models in the ensemble model. Indicates the first The formula for any probability dynamic transition model is as follows: ; in, Represents the parameters of the Gaussian model, ( () represents any state and action pair. Indicates the next state. Indicates a reward. This indicates that in any state and action pair ( Under the condition of ), the next state and rewards The probability distribution is used to fit the dynamics of environmental transitions. Indicates a Gaussian distribution. This represents the mean of a Gaussian distribution. Let represent the covariance matrix of the Gaussian distribution.

[0039] In another embodiment of this disclosure, the synthetic data generation module 203 is further used to initialize the agent's policy, policy exit length, and policy exit batch size; The batch size is determined according to the strategy, and multiple states are sampled from the offline dataset as initial states; For each initial state, the length is derived according to the policy, and multiple actions are sampled from the current policy; For each action, a probability dynamic transition model is randomly selected from the integrated model. The state and action pair corresponding to each action are input, and the next state and reward are output. The state and action pair corresponding to each action, as well as the next state and reward corresponding to each action, are used as samples to determine the synthetic data.

[0040] In another embodiment of this disclosure, the synthetic data includes: a model dataset, the model dataset being used to characterize the synthetic data generated through an ensemble model; The offset-aware reward determination module 204 is used to train the state transition classifier in the following manner: Based on the model dataset and the offline dataset, the state transition classifier is trained by constructing a function that minimizes the cross-entropy loss function, as expressed by the formula: ;

[0041] in, Represents the cross-entropy loss function. The parameters represent the state transition classifier. Represents a state transition classifier. Indicates state, Indicates an action, Indicates the next state. Indicates the state ,action Next state In this case, the probability that the sample is generated by the environment. Represents an offline dataset. This represents the average of the samples in the offline dataset. Represents the model dataset, This represents the average of the samples in the model dataset. This indicates an update operation. This represents an optimization algorithm used to minimize the cross-entropy loss function. The cross-entropy loss function is expressed with respect to the parameters. The gradient.

[0042] In another embodiment of this disclosure, the synthetic data includes: a policy dataset, which is used to characterize the data collected by the current policy in the ensemble model; The offset-aware reward determination module 204 is used to train the policy classifier in the following manner: Based on the policy dataset and the offline dataset, a loss function is constructed to train the policy classifier, as expressed by the formula: , =

[0043]

[0044]

[0045]

[0046] in, Represents the policy classifier, The parameters represent the policy classifier. Indicates state, Indicates an action, Indicates the state ,action In the case of a sample being generated by the current policy, the policy classifier includes a first policy classifier and a second policy classifier. This represents the first policy classifier. This represents the second policy classifier. This represents the parameters of the first policy classifier. This represents the parameters of the second policy classifier. This represents the loss function of the first policy classifier. This represents the loss function of the second policy classifier. Represents the policy dataset, This represents the average of the samples in the policy dataset. Represents an offline dataset. This represents the average of the samples in the offline dataset. This indicates an update operation. This represents an optimization algorithm used to minimize the loss function. The loss function of the first policy classifier with respect to the parameters is... gradient, The loss function of the second policy classifier with respect to the parameters The gradient.

[0047] In another embodiment of this disclosure, the synthetic data includes: a model dataset, the model dataset being used to characterize the synthetic data generated through an ensemble model; The offset-aware reward determination module 204 is used to extract samples from the offline dataset and synthetic data, and determine the offset-aware reward based on the state transition classifier and policy classifier, as expressed by the formula: ; in, Indicates state, Indicates an action, Indicates the next state. This indicates offset-perceived reward. This indicates the reward corresponding to the state and action. Represents a state transition classifier. Represents the policy classifier, and Indicates a sample, Represents the model dataset, Represents an offline dataset. and This represents the hyperparameter that controls the level of reward.

[0048] In another embodiment of this disclosure, the offset-aware reward determination module 204 is used to extract samples from the offline dataset and synthetic data, and determine the offset-aware reward based on the state transition classifier and the policy classifier, as expressed by the formula: ; in, Indicates state, Indicates an action, Indicates the next state. This indicates offset-perceived reward. This indicates the reward corresponding to the state and action. Represents a state transition classifier. Represents the policy classifier, and This refers to samples drawn from offline datasets or synthetic data. and This represents the hyperparameter that controls the level of reward.

[0049] In another embodiment of this disclosure, the strategy update module 205 is further configured to initialize the maximum number of iterations; Based on the maximum number of iterations, the strategy for iteratively updating the agent's operation includes: Based on the ensemble model, synthetic data is generated to represent the agent's environment. Samples are extracted from the offline dataset and synthetic data, and offset-aware rewards are determined based on the state transition classifier and policy classifier. Based on the samples corresponding to the offset-aware rewards, the SAC algorithm is used to update the agent's operating policy. The process of generating synthetic data based on the ensemble model is repeated until the maximum number of iterations is reached, at which point the final operating policy of the agent is determined.

[0050] Through the above description of the embodiments, those skilled in the art can clearly understand that the embodiments of this disclosure can be implemented in hardware or by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions of the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, mobile hard drive, etc.) and includes several instructions to cause a computer device (such as a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of this disclosure.

[0051] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of a preferred embodiment, and the modules or processes in the drawings are not necessarily essential for implementing this disclosure.

[0052] Those skilled in the art will understand that the modules in the apparatus of the embodiments can be distributed in the apparatus of the embodiments as described in the embodiments, or they can be located in one or more devices different from this embodiment with corresponding changes. The modules of the above embodiments can be combined into one module, or they can be further divided into multiple sub-modules.

[0053] The sequence numbers of the embodiments disclosed above are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0054] Obviously, those skilled in the art can make various modifications and variations to this disclosure without departing from its spirit and scope. Therefore, if such modifications and variations fall within the scope of the claims of this disclosure and their equivalents, this disclosure is also intended to include such modifications and variations.

Claims

1. A model-based off-line reinforcement learning method for a model-based off-line reinforcement learning method for an offset-aware agent, the method comprising: The method comprises the following steps: obtaining an offline dataset of an agent; training an integrated model according to the offline dataset; generating synthetic data according to the integrated model instead of an environment in which the agent is located; extracting samples from the offline dataset and the synthetic data, and determining an offset-aware reward according to a state transition classifier and a policy classifier, wherein the state transition classifier is used to compensate for the difference between the integrated model prediction and the real environment, and the policy classifier is used to compensate for the distribution offset caused by policy updating; updating the policy of the agent by using a SAC algorithm according to the sample corresponding to the offset-aware reward.

2. The method of claim 1, wherein, The step of training the integrated model according to the offline dataset comprises the following steps: extracting a state and an action pair from the offline dataset as an input of a Gaussian model, outputting a probability distribution of a next state and a reward, and training the Gaussian model by using maximum likelihood, which is expressed by a formula as follows: ； in, Represents the parameters of the Gaussian model. Indicates the time step, ( () represents the time step extracted from the offline dataset. The state and actions are correct. Indicates the next time step state, Indicates a reward. Indicates at time step State and action pair ( Under the condition of ), the next time step state and rewards The probability distribution, Indicates a Gaussian distribution. This represents the mean of a Gaussian distribution. Represents the covariance matrix of the Gaussian distribution; The integrated model determined according to the Gaussian model is represented as: wherein, represents a probabilistic dynamic transition model, represents the number of probabilistic dynamic transition models in the integrated model, represents the i-th probabilistic dynamic transition model, and the formula for any probabilistic dynamic transition model is represented as: ​ ； where, denotes the Gaussian model parameters, denotes any state and action pair, denotes the next state, denotes the reward, denotes the probability distribution of the next state and reward given any state and action pair used to fit the environment transition dynamics, denotes a Gaussian distribution, denotes the mean of the Gaussian distribution, denotes the covariance matrix of the Gaussian distribution.

3. The method of claim 2, wherein, The method further comprises the following steps: initializing the policy of the agent, the policy rollout length and the policy rollout batch size; The step of generating the synthetic data according to the integrated model instead of the environment in which the agent is located comprises the following steps: sampling a plurality of states from the offline dataset as initial states according to the policy rollout batch size; sampling a plurality of actions from the current policy according to the policy rollout length for each initial state; selecting a probability dynamic transition model from the integrated model at random for each action, inputting a state and an action pair corresponding to each action, and outputting a next state and a reward; determining the synthetic data by taking the state and the action pair corresponding to each action, and the next state and the reward corresponding to each action as samples.

4. The method of claim 1, wherein, The synthetic data comprises a model dataset, wherein the model dataset is used to represent the synthetic data generated by the integrated model. The state transition classifier is trained in the following manner: training the state transition classifier by constructing a minimum cross-entropy loss function according to the model dataset and the offline dataset, which is expressed by a formula as follows: ； wherein, denotes a cross-entropy loss function, denotes parameters of a state transition classifier, denotes a state transition classifier, denotes a state, denotes an action, denotes a next state, denotes a probability that a sample is generated by the environment in a state , an action , a next state denotes an offline dataset, denotes a mean of samples in the offline dataset, denotes a model dataset, denotes a mean of samples in the model dataset, denotes an update operation, denotes an optimization algorithm for minimizing the cross-entropy loss function, denotes a gradient of the cross-entropy loss function with respect to the parameters .​ 5. The method of claim 1, wherein, The synthetic data comprises a policy dataset, wherein the policy dataset is used to represent the data collected by the current policy in the integrated model. The policy classifier is trained in the following manner: training the policy classifier by constructing a minimum loss function according to the policy dataset and the offline dataset, which is expressed by a formula as follows: ， = wherein, denotes a policy classifier, denotes a parameter of the policy classifier, denotes a state, denotes an action, denotes a probability that a sample is generated by a current policy in a state , an action , the policy classifier comprising a first policy classifier and a second policy classifier, denotes the first policy classifier, denotes the second policy classifier, denotes a parameter of the first policy classifier, denotes a parameter of the second policy classifier, denotes a loss function of the first policy classifier, denotes a loss function of the second policy classifier, denotes a policy dataset, denotes an average over samples in the policy dataset, denotes an offline dataset, denotes an average over samples in the offline dataset, denotes an update operation, denotes an optimization algorithm for minimizing a loss function, denotes a gradient of the loss function of the first policy classifier with respect to a parameter , denotes a gradient of the loss function of the second policy classifier with respect to a parameter .

6. The method of claim 1, wherein, The synthetic data comprises a model dataset, wherein the model dataset is used to represent the synthetic data generated by the integrated model. The step of extracting samples from the offline dataset and the synthetic data, and determining the offset-aware reward according to the state transition classifier and the policy classifier is expressed by a formula as follows: ； wherein, represents a state, represents an action, represents a next state, represents an offset-aware reward, represents a state and action pair corresponding reward, represents a state transition classifier, represents a policy classifier, and represents a sample, represents a model dataset, represents an offline dataset, and represents a hyperparameter controlling the extent of the reward.

7. The method of claim 1, wherein, The step of extracting samples from the offline dataset and the synthetic data, and determining the offset-aware reward according to the state transition classifier and the policy classifier is expressed by a formula as follows: ； wherein, represents a state, represents an action, represents a next state, represents an offset-aware reward, represents a state and action pair corresponding reward, represents a state transition classifier, represents a policy classifier, and represents a sample drawn from an offline dataset or synthetic data, and represents a hyperparameter controlling the extent of the reward.

8. The method of claim 1, wherein, The method further comprises the following steps: initializing a maximum number of iterations; updating the policy of the agent by iteration according to the maximum number of iterations, which comprises the following steps: According to the integrated model instead of the environment where the agent is located, synthetic data is generated; samples are extracted from the offline data set and the synthetic data, and offset-aware rewards are determined according to the state transition classifier and the policy classifier; the SAC algorithm is used to update the policy of the agent according to the sample corresponding to the offset-aware reward, and the step of generating synthetic data according to the integrated model instead of the environment where the agent is located is returned until the maximum iteration number is reached, and the final policy of the agent is determined.

9. A model shift aware agent off-line reinforcement learning apparatus, comprising: Comprise: An offline data set acquisition module for acquiring an offline data set of an agent; An integrated model training module for training an integrated model according to the offline data set; A synthetic data generation module for generating synthetic data according to the integrated model instead of the environment where the agent is located; An offset-aware reward determination module for extracting samples from the offline data set and the synthetic data, and determining offset-aware rewards according to a state transition classifier and a policy classifier; wherein the state transition classifier is used to compensate for the difference between the integrated model prediction and the real environment, and the policy classifier is used to compensate for the distribution offset caused by policy updating; A policy updating module for updating the policy of the agent using the SAC algorithm according to the sample corresponding to the offset-aware reward.

10. The apparatus of claim 9, wherein, The integrated model training module is used to extract state and action pairs from the offline data set and input them into a Gaussian model to output the probability distribution of the next state, and the Gaussian model is trained using maximum likelihood, which is represented by the formula: ； wherein, denotes a Gaussian model parameter, denotes a time step, denotes a state and action pair at time step extracted from the offline dataset, denotes the next time step state, denotes a reward, denotes a probability distribution over the next time step state and action pair given the state and action pair at time step and reward, denotes a Gaussian distribution, denotes a mean of a Gaussian distribution, denotes a covariance matrix of a Gaussian distribution;​​ An integrated model determined according to the Gaussian models is represented as: wherein, represents a probabilistic dynamic transition model, represents a number of probabilistic dynamic transition models in the integrated model, represents the i-th probabilistic dynamic transition model, and a formula for any probabilistic dynamic transition model is represented as: ​ ； where, denotes the Gaussian model parameters, denotes any state and action pair, denotes the next state, denotes the reward, denotes the probability distribution of the next state and reward given any state and action pair used to fit the environment transition dynamics, denotes a Gaussian distribution, denotes the mean of the Gaussian distribution, denotes the covariance matrix of the Gaussian distribution.

Images