Offline reinforcement learning method based on score model

Abstract

The invention discloses an offline reinforcement learning method based on a score model, and the method comprises the steps: building a score generation model, learning a gradient field of an offline data set through employing a denoising score matching technology, estimating a score value, introducing a score as a penalty term, and preventing an intelligent agent from exploring a region where the data density is rapidly reduced. In the training process, real data and synthetic data are combined, a soft actor commentator framework is used for optimizing a commentator network, a penalty term is introduced to limit the influence of the synthetic data on a strategy target, meanwhile, a Q value is maximized, an intelligent agent strategy is optimized in combination with an entropy term, the synthetic data quality is improved, and the robustness and generalization ability of the model are enhanced. In scenes such as robot control, the method can ensure safe operation of the robot in a complex environment, effectively utilizes the synthetic data to improve the strategy performance, and reduces the risk of online data acquisition. Particularly, when tasks such as robot navigation are processed, key information near a target point can be better handled, strategy failure caused by complexity of data distribution is avoided, and excellent practicability and adaptability are shown.

Description

Technical Field

[0001] The present invention relates to offline reinforcement learning technology and its applications, and particularly to an offline reinforcement learning method based on a score model and its application in robot control. Background Art

[0002] Offline Reinforcement Learning (Offline RL), also known as batch reinforcement learning, allows an agent to learn from a pre-collected static dataset, avoiding expensive and potentially dangerous online exploration. This feature is particularly important in the field of robot tasks (such as robotic arm operation, autonomous driving, and drone navigation), as these fields cannot accept online data collection through trial and error. By using a static dataset, offline reinforcement learning provides a controlled training environment while reducing the risk of online data collection.

[0003] However, offline reinforcement learning faces significant challenges, among which distribution shift is one of the main problems. Distribution shift refers to the situation where the state-action pairs encountered by the agent are significantly different from the training data, which may lead to inaccurate or overly extreme value estimates, thereby affecting policy evaluation and performance. Solving the distribution shift problem is crucial for improving the robustness and generalization ability of offline reinforcement learning algorithms, enabling them to work effectively in dynamic real-world environments. Current solutions are mainly divided into model-free and model-based methods. Model-free methods usually improve the stability and robustness of learning by introducing conservative penalties to the value function or restricting the policy to be close to the behavior policy.

[0004] Different from model-free methods, model-based offline reinforcement learning improves data efficiency and generalization ability by using the learned dynamics model to generate synthetic data. This synthetic data can expand the state space, thereby enhancing policy performance. However, model-based methods introduce new technical challenges, such as model error, which makes it crucial to introduce conservative measures. Although current research has proposed using metrics such as the distance between the learned model and the real model for penalty, accurately calculating these metrics is still difficult. Therefore, alternative strategies have been proposed, such as indirectly dealing with model uncertainty by regularizing the value function or making adversarial modifications to the transition dynamics.

[0005] Despite some progress, model-based offline reinforcement learning still faces technical bottlenecks. For example, the "reachable boundary" problem reveals that in the synthetic data generated by the dynamics model, value overestimation may occur for states close to the end of the trajectory, indicating that the performance of model-based offline reinforcement learning is limited by some difficult-to-detect complex problems.

[0006] It should be noted that the information disclosed in the above background art section is only for understanding the background of the present application, and thus may include information that does not constitute the prior art known to those of ordinary skill in the art. Summary of the Invention

[0007] The main object of the present invention is to overcome the defects existing in the above background art, and to provide an offline reinforcement learning method based on a score model and its application in robot control.

[0008] To achieve the above object, the present invention adopts the following technical solutions:

[0009] An offline reinforcement learning method based on a score model, comprising:

[0010] Constructing a score generation model: using an offline data set, training through a denoising score matching model, adding noise and gradually adjusting, learning the gradient field of the offline data set to estimate the score value;

[0011] Introducing a penalty mechanism: taking the score as a penalty term, based on the relationship between the data set density and the state-action distribution, defining the upper bound of the estimation error and the maximum approximation error, analyzing the return gap, generating a penalty value through the score gradient, and avoiding the agent from exploring regions where the density drops rapidly;

[0012] Optimizing the training process: generating synthetic data based on a dynamic model, training by combining real data and synthetic data, optimizing the critic network using a soft actor-critic framework, introducing a penalty term to limit the impact of synthetic data on the policy objective, and at the same time optimizing the agent's policy by maximizing the Q value and combining the entropy term, improving the quality of synthetic data to enhance the robustness and generalization ability of the model.

[0013] As an application, an offline reinforcement learning method based on a score model for robot control, comprising:

[0014] Constructing a score generation model: using an offline data set in a robot control task, training through a denoising score matching model, adding noise and gradually adjusting, learning the gradient field of the robot state-action space to estimate the score value;

[0015] Introducing a penalty mechanism: taking the score as a penalty term, based on the relationship between the data set density and the state-action distribution in the robot control task, defining the upper bound of the estimation error and the maximum approximation error, analyzing the return gap, generating a penalty value through the score gradient, avoiding the robot from exploring regions where the density drops rapidly during the control process, so as to ensure that the robot operates within a safe range;

[0016] Training process optimization: Synthetic data is generated based on a dynamic model in robot control tasks, and training is carried out by combining real data and synthetic data. The critic network is optimized using the Soft Actor-Critic framework. A penalty term is introduced to limit the impact of synthetic data on the robot control strategy target. At the same time, the robot control strategy is optimized by maximizing the Q value and combining the entropy term to improve the quality of synthetic data, so as to enhance the robustness and generalization ability of the robot control model.

[0017] The present invention has the following beneficial effects:

[0018] The present invention proposes an offline reinforcement learning method based on a score model (referred to as ScorePen-MORL). By constructing a score generation model, introducing a penalty mechanism, and optimizing the training process, it effectively alleviates the distribution shift problem in offline reinforcement learning, especially for the challenges brought by the data density mutation region ("cliff" region). This method uses a denoising score matching model to learn the gradient field of the offline dataset and generates score values as penalty signals to guide the policy to avoid regions with a rapid decline in data density, thereby improving the stability and generalization ability of policy optimization.

[0019] In robot control tasks, the method of the present invention demonstrates significant advantages. Experimental results on standard robot datasets such as MuJoCo and NeoRL show that this method can effectively improve the performance of robots in high-dimensional continuous control tasks, especially in scenarios involving complex dynamics and high-precision control. Robots usually need to perform fine operations in unknown environments and meet strict safety constraints. By constructing an accurate penalty mechanism, this method enables the robot to avoid entering high-risk regions with sparse data during the learning process, ensuring safety and stability. For example, in the manipulator operation task, this method can more accurately control the trajectory of the actuator, reducing collisions and failure rates; in the autonomous driving task, ScorePen-MORL can optimize the decision-making process and improve the stability of the vehicle in boundary situations (such as overtaking and obstacle avoidance); in the UAV navigation task, this method can more sensitively perceive the data distribution characteristics near the target point, avoid policy failure caused by sparse data, and improve the accuracy and safety of navigation.

[0020] In addition, the method of the present invention can enhance the training process by combining synthetic data, reducing the dependence on online data collection, thereby improving the training efficiency and reducing the deployment risk. Experiments show that this method exceeds the current optimal model benchmarks in both D4RL and NeoRL tasks, further verifying its excellent practicality and adaptability in robot control.

[0021] Other beneficial effects in the embodiments of the present invention will be further described below. Description of the Drawings

[0022] Figure 1 This is the overall flowchart of the offline reinforcement learning method based on the score model according to the embodiments of the present invention.

[0023] Figure 2 This is the algorithm architecture diagram of the offline reinforcement learning method based on the score model according to the embodiments of the present invention. Specific embodiments

[0024] The following provides a detailed description of the embodiments of the present invention. It should be emphasized that the following description is merely exemplary and not intended to limit the scope and application of the present invention.

[0025] The present invention focuses on a key problem in model-based offline reinforcement learning, namely the "mutation region" problem. This problem exposes the limitations of existing penalty mechanisms, especially in regions where the density of the dataset undergoes mutations. The inventors have observed that the penalties generated by these methods in mutation regions fail to accurately reflect uncertainty, resulting in the ineffective utilization of knowledge in these regions. This problem is particularly prominent when the key information required for optimizing the policy is located in or near these regions, such as the target point in a navigation task. In addition, "mutation regions" are difficult to locate in the dataset because the density function of the original data distribution is unknown, and these regions may arise due to various reasons, such as physical constraints, etc.

[0026] The present invention aims to solve the "mutation region" problem through gradient field reconstruction technology based on diffusion models, using the data gradients generated by diffusion models as the basis for penalties to improve the robustness of policy optimization and cope with diverse task requirements. The present invention not only demonstrates theoretical effectiveness but also verifies the practicality of this method through a large number of experiments, especially performing well in "mutation region" scenarios where existing methods cannot be fully solved.

[0027] Refer to Figure 1 and Figure 2 According to the embodiments of the present invention, an offline reinforcement learning method based on a score model is provided, including:

[0028] Construct a score generation model: Utilize the offline dataset, train through a denoising score matching model, add noise and gradually adjust to learn the gradient field of the offline dataset to estimate the score value;

[0029] Introduce a penalty mechanism: Use the score as a penalty term, based on the relationship between the dataset density and the state-action distribution, define the upper bound of the estimation error and the maximum approximation error, analyze the return gap, and generate penalty values through the score gradient to prevent the agent from exploring regions where the density rapidly decreases;

[0030] Training process optimization: Generate synthetic data based on a dynamic model, combine real data and synthetic data for training, optimize the critic network using the Soft Actor-Critic framework, introduce a penalty term to limit the impact of synthetic data on the policy objective, and at the same time optimize the agent's policy by maximizing the Q-value and combining an entropy term to improve the quality of synthetic data and enhance the robustness and generalization ability of the model.

[0031] In some embodiments, in the step of constructing the score generation model, training is completed by minimizing a specific loss function, and an improved form based on the score diffusion model is introduced to reformulate the loss function into a more easily optimized expression to improve the training efficiency and stability of the model.

[0032] In some embodiments, in the step of introducing the penalty mechanism, Langevin dynamics is used to sample new points from the state-action space to improve the sample coverage rate in high-density regions, and by controlling the density threshold of the new points, avoid entering regions where the density drops rapidly, thereby enhancing the model's ability to handle sparse state-action pairs.

[0033] In some embodiments, in the step of training process optimization, multiple dynamic models are integrated through maximum likelihood training, and each dynamic model outputs a Gaussian distribution of the next state and reward to generate synthetic data, and training is combined with real data.

[0034] In some embodiments, in the step of training process optimization, the critic network is updated by minimizing the temporal difference (TD) error, and when calculating the target value of synthetic data, an additional penalty term is introduced to limit the excessive impact of synthetic data on the policy objective, and the penalty term measures the difference between synthetic data and real data through the quadratic norm to balance synthetic data and real data.

[0035] In some embodiments, in the step of training process optimization, the agent's policy is optimized by maximizing the Q-value and combining an entropy term, and the entropy term guides the agent to explore more during training to balance the exploitation and exploration of the policy.

[0036] In some embodiments, the method further includes identifying and processing "mutation regions" in the dataset, detecting mutations in data density through changes in score gradients, identifying "mutation regions", and specifically adjusting the penalty mechanism to improve the robustness of policy optimization.

[0037] In some embodiments, the method further includes one or more of the following processing strategies:

[0038] An applicability extension strategy for different dataset qualities or task scenarios, by adjusting the parameters of the penalty mechanism or introducing additional regularization terms to adapt to different data distributions and task requirements;

[0039] Optimize the computing resources during the training process by simplifying the gradient field calculation process or using sparse representation methods, thereby reducing the computational cost and improving the running efficiency of the model.

[0040] In some embodiments, the method further includes one or more of the following optimization strategies:

[0041] Only correct the gradient field for specific "mutation regions" (cliff regions), rather than globally, thereby reducing the computational cost;

[0042] Indirectly handle the gradient changes in the "mutation region" by introducing a separate density estimation module, replacing the direct gradient correction in the original scheme;

[0043] Embed the gradient field calculation process into the model training stage and utilize shared features to reduce computational redundancy;

[0044] Use a multi-stage training strategy. In the initial stage, pre-train using traditional methods, and then fine-tune the final model through the gradient field correction strategy.

[0045] As an application, an embodiment of the present invention also provides an offline reinforcement learning method based on a score model for robot control, including:

[0046] Construct a score generation model: Utilize the offline dataset in the robot control task, train through a denoising score matching model, add noise and gradually adjust to learn the gradient field of the robot state-action space to estimate the score value;

[0047] Introduce a penalty mechanism: Use the score as a penalty term, based on the relationship between the dataset density and the state-action distribution in the robot control task, define the upper bound of the estimation error and the maximum approximation error, analyze the return gap, generate a penalty value through the score gradient, and avoid the robot exploring regions where the density drops rapidly during the control process, thereby ensuring that the robot operates within a safe range;

[0048] Optimize the training process: Generate synthetic data in the robot control task based on a dynamic model, train by combining real data and synthetic data, use the soft actor-critic framework to optimize the critic network, introduce a penalty term to limit the impact of synthetic data on the robot control strategy target, and at the same time optimize the robot control strategy by maximizing the Q value and combining the entropy term to improve the quality of synthetic data and enhance the robustness and generalization ability of the robot control model.

[0049] The offline reinforcement learning method based on the score model proposed by the present invention constructs a score generation model, uses the denoising score matching technique to learn the gradient field of the offline dataset, estimates the score value, and introduces the score as a penalty term to prevent the agent from exploring regions where the data density drops rapidly. During the training process, real data and synthetic data are combined, and the soft actor-critic framework is used to optimize the critic network. A penalty term is introduced to limit the impact of synthetic data on the policy objective. At the same time, the agent's policy is optimized by maximizing the Q value and combining the entropy term, improving the quality of synthetic data, enhancing the robustness and generalization ability of the model. In scenarios such as robot control, this method can ensure the safe operation of the robot in complex environments, effectively utilize synthetic data to improve policy performance, and reduce the risk of online data collection. Especially when dealing with tasks such as robot navigation, it can better handle the key information near the target point, avoid policy failure caused by the complexity of data distribution, and demonstrate excellent practicality and adaptability.

[0050] The specific embodiments of the present invention, their algorithm examples, and experimental verifications are further described below.

[0051] The present invention proposes a model-based offline reinforcement learning method called ScorePen-MORL. The core of the method includes the following three aspects: First, it details how to construct a score generation model from the offline dataset to estimate the score value; then, it elaborates on the theoretical motivation for using the score as a penalty term and analyzes its role and significance in the model-based offline reinforcement learning problem; finally, it explains how to introduce this penalty mechanism during the training process and comprehensively presents the architecture of ScorePen-MORL, including the specific training process and pseudocode.

[0052] Construction of the score generation model

[0053] Before delving into the core content of the method, a method based on the score diffusion model is first introduced to approximate the gradient field of the offline dataset (i.e., the change trend of the state-action distribution). To learn this gradient field, a denoising score matching model is adopted and trained by minimizing a specific loss function. During this process, noise is added to the states and actions in the data. The noise is generated according to the rules of the diffusion model and changes gradually according to a schedule. Directly optimizing this loss function may pose practical challenges, so an improved form based on the score diffusion model is introduced to reformulate the loss function into a more easily optimizable expression. When the model training converges, the gradient field can be extracted by solving a mathematical equation, thereby better capturing the characteristics of the state-action distribution and laying a foundation for subsequent research.

[0054] Theoretical motivation and analysis

[0055] In the field of model-based offline reinforcement learning, existing research (such as Count-MORL) has shown that state-action pairs frequently occurring in the offline dataset have smaller estimation errors and faster convergence rates. This method draws on the theory of Count-MORL to establish a connection between the gradient field of the dataset and the state-action estimation error. Specifically, gradient field information is captured by a score, and the density function of the dataset is used to approximate the state-action distribution, and then the upper bound of the estimation error and the maximum approximation error based on the density of the real dataset are defined. These definitions help analyze the return gap between the estimated Markov decision process (MDP) and the real MDP, which is jointly affected by data density and estimation error.

[0056] To enhance the model's ability to handle sparse state-action pairs, Langevin dynamics is used to sample new points from the state-action space, thereby improving the sample coverage rate in the high-density region. It is further proved that when the density of the new points is higher than a certain threshold, this sampling method can effectively control the data distribution and avoid entering the region where the density drops rapidly. This theory provides a theoretical basis for using the score as a penalty mechanism - when the state-action point deviates from the data distribution, its score gradient increases, generating a higher penalty value, effectively preventing the agent from exploring the transition region where the density drops rapidly.

[0057] Penalty mechanism during training

[0058] ScorePen-MORL is extended based on the MOPO method. Similar to MOPO, this method trains a dynamic model ensemble through maximum likelihood, where each dynamic model outputs a Gaussian distribution of the next state and reward. During training, real data and synthetic data are used in combination. The specific steps include:

[0059] 1) The dynamic model generates initial states sampled from the real dataset, generates synthetic trajectories, and adds the synthetic data to the synthetic dataset.

[0060] 2) The critic updates the critic network by using the soft actor-critic (SAC) framework and updates the Q network by minimizing the temporal difference (TD) error. When calculating the target value of the synthetic data, a penalty term is additionally introduced to limit the excessive influence of the synthetic data on the policy target, specifically by measuring the difference between the synthetic data and the real data through the quadratic norm.

[0061] 3) The policy optimizes the agent's policy by maximizing the Q value and combining an entropy term. The entropy term guides the agent to explore more during training, rather than just exploiting the current knowledge.

[0062] Detailed process of the algorithm example:

[0063] 1) Train the dynamic model set using the maximum likelihood method To generate virtual data.

[0064] 2) Optimize the diffusion model h according to the score-based diffusion model loss ξ (s,a), where (s,a) is a state-action pair and ξ is the neural network parameter. The score-based diffusion model loss is:

[0065]

[0066] where is the forward propagation process of the diffusion model, denotes the distribution probability function of samples in the original dataset, σ is a hyperparameter, and z is Gaussian noise with mean 0 and variance 1.

[0067] 3) Propose a new way to calculate the objective:

[0068]

[0069] where r is the original / predicted reward, γ is the discount factor, is the Q-network of the baseline algorithm, and π φ (a∣s) is the policy network of the baseline algorithm, α,β are hyperparameters, (s,a,t,s') is a sample obtained from the dataset and a' is the output of the baseline algorithm π φ (a'∣s').

[0070] 4) Uniformly sample from the dataset and the virtual dataset, and update the Q-network and policy network of the baseline algorithm according to the proposed objective calculation method.

[0071] Pseudocode of the algorithm

[0072]

[0073] The pseudocode describes the training process of ScorePen-MORL, including steps such as generating synthetic data from the dynamic model, updating the critic, and optimizing the agent's policy. During the training process, the method improves the quality of synthetic data by introducing a penalty mechanism, ensuring the balance between real data and synthetic data, thereby enhancing the robustness and generalization ability of the model.

[0074] ScorePen-MORL has important potential in robot applications, especially in scenarios involving high-dimensional continuous control tasks, such as robotic arm operation, autonomous driving, and drone navigation. Since robot systems usually face complex dynamics and uncertain environments, constructing appropriate penalty terms to avoid entering low-data-density regions (i.e., "cliff" regions) is crucial for improving the reliability and safety of the model.

[0075] In robotic tasks, the application of ScorePen-MORL can be reflected in the following aspects:

[0076] 1. Manipulator operation: In robotic manipulation tasks, the state space of the manipulator is usually high-dimensional, and some operations (such as grasping, assembly, and fine manipulation) require precise control of the end effector's trajectory. Traditional density or uncertainty penalty methods may be difficult to accurately identify dangerous areas, while ScorePen-MORL can dynamically adjust the penalty based on the data density gradient, enabling the manipulator to more robustly adapt to OOD states during training and reducing the risk of collisions or task failures.

[0077] 2. Autonomous driving: In autonomous driving tasks, the decision-making space of the vehicle involves various environmental factors (such as road structure, pedestrians, traffic flow, etc.), and different driving strategies may lead to drastic changes in the data density distribution. For example, strategies such as sudden braking, lane changing, and overtaking may involve rare but critical decision points, forming "cliff" regions. ScorePen-MORL enables the vehicle to more safely explore boundary cases, improve generalization ability, and reduce unstable responses to abnormal situations through a more refined penalty mechanism.

[0078] 3. UAV navigation: In UAV path planning and obstacle avoidance tasks, the complexity of the environment (such as obstacle distribution, wind field influence) may lead to sparse data in some areas, thus forming the "cliff" phenomenon. Traditional density penalties may be difficult to accurately characterize these areas, while ScorePen-MORL can dynamically sense changes in the data distribution, adjust the strategy, making the UAV more cautious in high-risk areas and improving the stability and safety of navigation.

[0079] The excellent performance of ScorePen-MORL on the MuJoCo and NeoRL datasets further validates its applicability and advantages in robotic tasks. Both the MuJoCo and NeoRL datasets cover robotic tasks with high-dimensional continuous control, so they can be used as standard evaluation environments for robotic applications.

[0080] 1. MuJoCo dataset: MuJoCo (Multi-Joint dynamics with Contact) is a widely used physical simulation environment for robotic control, including tasks such as semi-humanoid robots (such as Hopper, Walker, Humanoid), manipulators (such as Reacher), and quadruped robots. The state spaces of these tasks are complex, the control signals are high-dimensional, and some actions (such as running at high speed, jumping, balancing) may involve drastic changes in the data distribution, forming "cliffs".

[0081] Region. The good performance of ScorePen-MORL on MuJoCo tasks indicates that it can effectively identify and handle these critical regions, making the model more robust in high-risk actions (such as high acceleration, contact force changes, etc.) and improving the overall performance.

[0082] 2. NeoRL Dataset: NeoRL is an emerging offline reinforcement learning benchmark that includes a richer set of robot control tasks, covering different dynamic characteristics in real-world scenarios. For example, some of the tasks involve more complex robot operations, such as high-precision trajectory tracking, interaction control, etc. Experiments on the NeoRL dataset show that ScorePen-MORL also demonstrates excellent performance in these tasks, indicating that its method is not only applicable to standard MuJoCo tasks but also can improve the reliability of policies in more challenging robot environments.

[0083] In summary, both the MuJoCo and NeoRL datasets are essentially high-dimensional continuous control datasets similar to robots. The significant performance improvement of ScorePen-MORL on these datasets indicates that this method has great potential in the application of robot tasks, can help robots learn safe and robust policies more efficiently, and is applicable to various real-world robot control tasks.

[0084] Experiments and effects

[0085] The performance of ScorePen-MORL was evaluated on the standard offline reinforcement learning benchmark D4RL. It was compared with a variety of offline reinforcement learning algorithms, which are divided into the following two categories: 1) Model-free methods: BC, CQL, TD3+BC, and EDAC; 2) Model-based methods: MOPO, COMBO, TT, RAMBO, Count-MORL, and MOBILE.

[0086] The evaluation covered 12 datasets in the Gym domain, involving three robot environments: hopper, walker2d, and halfcheetah. Each environment contains four types of datasets: random, medium, medium-replay, and medium-expert. The random dataset consists of transitions generated by a random policy, the medium dataset comes from an early-stopped SAC policy, the medium-replay dataset contains the replay buffer generated during the training of the medium policy, and the medium-expert dataset is a mixture of sub-optimal data and expert data. All experimental datasets are in the "v2" version.

[0087]

[0088] The NeoRL benchmark aims to simulate real-world data collection scenarios using more conservative strategies, generating a dataset that is narrower and more restricted, which poses a greater challenge to offline reinforcement learning algorithms. This invention focuses on 9 datasets, covering three environments - HalfCheetah-v3, Hopper-v3, and Walker2d-v3. These datasets are divided into three categories (L, M, H) according to quality, representing low, medium, and high-quality data respectively. Each task provides 100, 1000, and 10000 training trajectories, and this invention uniformly selects 1000 trajectories for experiments.

[0089]

[0090] Alternative Embodiment

[0091] To improve algorithm efficiency while reducing complexity, this invention designs the following alternative embodiment. Although the performance is slightly inferior to the best solution, it is significantly better than the prior art. For example:

[0092] ο Optimization point: Only perform gradient field correction on specific cliff regions instead of global correction, thereby reducing the computational cost.

[0093] ο Advantage: Compared with existing methods, this solution can significantly improve the policy performance under limited computing resources and is suitable for deployment scenarios with hardware constraints.

[0094] Other Variation Schemes:

[0095] ο Variation 1: Indirectly handle the gradient change in the cliff region by introducing a separate density estimation module, replacing the direct gradient correction in the original scheme.

[0096] ο Variation 2: Embed the calculation process of the gradient field into the model training stage and use shared features to reduce computational redundancy.

[0097] ο Variation 3: Use a multi-stage training strategy. In the initial stage, use traditional methods for pre-training, and then fine-tune the final model through the gradient field correction strategy.

[0098] The key technical contributions of this invention include but are not limited to the following:

[0099] 1. Modeling and Application of Gradient Field

[0100] ο Calculation method of the dataset gradient field, including the detection of data point density changes and the identification method of cliff regions.

[0101] ο Conservative policy design based on the gradient field for policy optimization in reinforcement learning.

[0102] 2. Definition and Processing Mechanism of Cliff Regions

[0103] ο Definition of the cliff area and its representation in the state-action space.

[0104] ο Revised strategies proposed for the cliff area and their applicability conditions.

[0105] 3. Combined optimization schemes with other algorithms

[0106] ο Combination methods of ScorePen-MORL with other baseline algorithms (such as Count-MORL or MOBILE) and their performance improvement mechanisms.

[0107] ο Provide a general "plug-and-play" modular design to ensure compatibility with other offline reinforcement learning algorithms.

[0108] 4. Efficient implementation methods

[0109] ο Resource optimization for the gradient field calculation process, including methods to simplify the calculation process or use sparse representations.

[0110] ο Efficient implementation schemes adapted to various hardware architectures to reduce deployment difficulties and improve operation efficiency.

[0111] 5. Performance verification and applicability expansion

[0112] ο Performance verification processes applicable to different benchmark tests (such as D4RL and NeoRL) and their result analysis.

[0113] ο Applicability expansion strategies for different dataset qualities or task scenarios.

[0114] In summary, the present invention proposes an offline reinforcement learning method based on a score model. By constructing a score generation model, introducing a penalty mechanism, and optimizing the training process, it effectively solves the distribution shift problem in offline reinforcement learning, especially the challenges in the regions of sudden changes in dataset density. This method uses a denoising score matching model to learn the gradient field of the offline dataset and generates score values as the basis for penalties, preventing the agent from entering the regions with a rapid decline in data density during exploration, thereby significantly improving the robustness and generalization ability of policy optimization. In the robot control scenario, the advantages of the method of the present invention are particularly prominent. Robot control tasks usually face complex environments and strict safety requirements. The present invention ensures that the robot can operate within a safe range through an accurate penalty mechanism and an efficient training strategy, while effectively using synthetic data to improve policy performance and reduce the risk of online data collection. In addition, when dealing with tasks such as robot navigation, the method can better handle the key information near the target point and avoid policy failure caused by the complexity of data distribution, demonstrating excellent practicality and adaptability.

[0115] An embodiment of the present invention further provides a storage medium for storing a computer program, which when executed, at least executes the method described above.

[0116] An embodiment of the present invention further provides a control device, including a processor and a storage medium for storing a computer program; wherein, the processor is configured to at least execute the method described above when executing the computer program.

[0117] An embodiment of the present invention further provides a processor, which executes a computer program and at least executes the method described above.

[0118] The storage medium can be implemented by any type of non-volatile storage device, or a combination thereof. Among them, the non-volatile memory can be a read-only memory (ROM, Read Only Memory), a programmable read-only memory (PROM, Programmable Read-Only Memory), an erasable programmable read-only memory (EPROM, Erasable Programmable Read-Only Memory), an electrically erasable programmable read-only memory (EEPROM, Electrically Erasable Programmable Read-Only Memory), a ferromagnetic random access memory (FRAM, Ferromagnetic Random Access Memory), a flash memory (Flash Memory), a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM, Compact Disc Read-Only Memory); the magnetic surface memory can be a disk memory or a tape memory. The storage medium described in the embodiments of the present invention is intended to include, but is not limited to, these and any other suitable types of memories.

[0119] In several embodiments provided by the present invention, it should be understood that the disclosed systems and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of the units is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined, or can be integrated into another system, or some features can be ignored, or not executed. In addition, the couplings, direct couplings, or communication connections between the various components shown or discussed can be through some interfaces, and the indirect couplings or communication connections of devices or units can be electrical, mechanical, or other forms.

[0120] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed over multiple network units; some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0121] In addition, in each embodiment of the present invention, all the functional units may be integrated in one processing unit, or each unit may be separately used as one unit, or two or more units may be integrated in one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of a combination of hardware and software functional units.

[0122] Those of ordinary skill in the art can understand that all or part of the steps of implementing the above method embodiments can be completed by hardware related to program instructions. The foregoing program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps including the above method embodiments; and the foregoing storage medium includes: removable storage devices, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), magnetic disks or optical disks and other various media that can store program codes.

[0123] Alternatively, if the above-mentioned integrated units of the present invention are implemented in the form of software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiments of the present invention essentially or the part that contributes to the prior art can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in the various embodiments of the present invention. And the foregoing storage medium includes: removable storage devices, ROM, RAM, magnetic disks or optical disks and other various media that can store program codes.

[0124] The methods disclosed in the several method embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments.

[0125] The features disclosed in the several product embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new product embodiments.

[0126] The features disclosed in the several method or device embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0127] The above content is a further detailed description of the present invention in combination with specific preferred embodiments. It cannot be determined that the specific implementation of the present invention is only limited to these descriptions. For those skilled in the technical field to which the present invention pertains, without departing from the concept of the present invention, several equivalent substitutions or obvious modifications can be made, and as long as the performance or use is the same, they should all be regarded as belonging to the protection scope of the present invention.

Claims

1. An offline reinforcement learning method based on a fractional model, characterized in that: include: Build a score generation model: Use the offline dataset to train the denoising score matching model, add noise and gradually adjust it, learn the gradient field of the offline dataset to estimate the score value; Introduce a penalty mechanism: use the score as a penalty term, define the upper bound of the estimation error and the maximum approximation error based on the relationship between the dataset density and the state-action distribution, analyze the reward gap, generate a penalty value through the score gradient, and prevent the agent from exploring areas where the density drops rapidly; Optimization of the training process: Generate synthetic data based on the dynamic model, combine real data with synthetic data for training, use the soft actor-critic framework to optimize the critic network, introduce penalty terms to limit the impact of synthetic data on policy goals, and optimize the agent strategy by maximizing the Q value and combining the entropy term to improve the quality of synthetic data and enhance the robustness and generalization ability of the model.

2. The offline reinforcement learning method based on the fractional model according to claim 1, characterized in that: In the step of constructing the score generation model, training is completed by minimizing a specific loss function, and an improved form based on the score diffusion model is introduced to reformulate the loss function into an expression that is easier to optimize, so as to improve the training efficiency and stability of the model.

3. The offline reinforcement learning method based on the fractional model according to claim 1 or 2, characterized in that: In the step of introducing the penalty mechanism, Langevin dynamics is used to sample new points from the state-action space to improve the sample coverage rate in high-density areas, and by controlling the density threshold of new points, entering the area where the density drops rapidly is avoided, thereby enhancing the model's ability to handle sparse state-action pairs.

4. The offline reinforcement learning method based on the fractional model according to any one of claims 1 to 3, characterized in that: In the step of optimizing the training process, multiple dynamic models are integrated through maximum likelihood training, and each dynamic model outputs a Gaussian distribution of the next state and reward to generate synthetic data, which is then combined with real data for training.

5. The offline reinforcement learning method based on the fractional model according to any one of claims 1 to 4, characterized in that: In the step of optimizing the training process, the critic network is updated by minimizing the temporal difference (TD) error, and when calculating the target value of synthetic data, an additional penalty term is introduced to limit the excessive impact of synthetic data on the policy target. The penalty term measures the difference between synthetic data and real data through the quadratic norm to balance synthetic data and real data.

6. The offline reinforcement learning method based on the fractional model according to any one of claims 1 to 5, characterized in that: In the step of optimizing the training process, the agent strategy is optimized by maximizing the Q value and combining the entropy term, which guides the agent to perform more exploration during training to balance the utilization and exploration of the strategy.

7. The offline reinforcement learning method based on the fractional model according to any one of claims 1 to 6, characterized in that: It also includes identifying and processing "mutation regions" in the data set, detecting mutations in data density through changes in score gradients, identifying "mutation regions", and adjusting the penalty mechanism in a targeted manner to improve the robustness of strategy optimization.

8. The offline reinforcement learning method based on the fractional model according to any one of claims 1 to 7, characterized in that: It also includes one or more of the following processing strategies: The applicability expansion strategy for different dataset qualities or task scenarios is to adapt to different data distributions and task requirements by adjusting the parameters of the penalty mechanism or introducing additional regularization terms; Optimize the computing resources during the training process, reduce the computing cost and improve the operating efficiency of the model by simplifying the gradient field calculation process or using sparse representation methods.

9. The offline reinforcement learning method based on the fractional model according to any one of claims 1 to 8, characterized in that: It also includes one or more of the following optimization strategies: Only perform gradient field correction on specific "mutation areas" (cliff areas) instead of global correction, thus reducing computational costs; By introducing a separate density estimation module, the gradient change in the "mutation area" is indirectly processed, replacing the direct gradient correction in the original solution; The calculation process of the gradient field is embedded into the model training phase, and shared features are used to reduce computational redundancy. A multi-stage training strategy is used, with traditional methods used for pre-training in the early stages, and then the final model is fine-tuned through a gradient field correction strategy.

10. An offline reinforcement learning method based on a fractional model for robot control, characterized in that: include: Build a score generation model: Use offline datasets from robot control tasks, train a denoised score matching model, add noise and gradually adjust it, learn the gradient field of the robot state-action space to estimate the score value; Introducing a penalty mechanism: Using the score as a penalty term, based on the relationship between the dataset density and the state-action distribution in the robot control task, defining the upper bound of the estimation error and the maximum approximation error, analyzing the reward gap, and generating a penalty value through the score gradient to prevent the robot from exploring areas with rapidly decreasing density during the control process, thereby ensuring that the robot operates within a safe range; Optimization of the training process: Generate synthetic data in robot control tasks based on dynamic models, combine real data with synthetic data for training, use the soft actor-critic framework to optimize the critic network, introduce penalty terms to limit the impact of synthetic data on the robot control strategy objectives, and optimize the robot control strategy by maximizing the Q value and combining the entropy term to improve the quality of synthetic data and enhance the robustness and generalization ability of the robot control model.

Images