Off-line reinforcement learning strategy optimization method and system based on experience quantile sorting

Abstract

The invention discloses an off-line reinforcement learning strategy optimization method and system based on empirical quantile sorting, and aims to solve the problems that in a traditional method, strategy optimization greatly depends on an absolute value of a Q value, and training is unstable. According to the method, a ranking function is introduced to replace a Q function to serve as a strategy supervision signal, target Q values are ranked through an empirical cumulative distribution function (ECDF), normalized quantile labels are generated, and relative advantages of actions are described. By constructing a sorting network and minimizing the mean square error between the output of the sorting network and a sorting label, the sorting supervised learning of the action quality is realized. In the strategy updating stage, the strategy is dynamically optimized by combining the sorting score and the behavior cloning loss, and the strategy offset problem is relieved. The method focuses on the relative ranking relation of the Q values rather than absolute values, improves the training robustness and generalization ability, is excellent in performance in multiple offline tasks, and has good engineering application prospects.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, specifically relating to an offline reinforcement learning method and system based on experience quantile ranking. It uses the Rank function to replace the Q function for policy network training, and is suitable for agent policy optimization under conditions lacking online interaction. Background Technology

[0002] In real-world applications, reinforcement learning (RL) often faces challenges such as high interaction costs, security risks, or unreproducible environments, limiting its online sampling capabilities. To address these limitations, offline reinforcement learning (OfflineRL) has emerged, aiming to learn optimal policies while relying entirely on a fixed dataset. These methods no longer interact with the environment in real time but instead extract effective supervisory signals from historical trajectories, thus showing great promise in fields such as robot control, autonomous driving, industrial manufacturing, and financial trading. However, due to the inconsistent distribution between the learned policy and the behavioral policy, offline reinforcement learning is prone to policy shift and overestimation problems, leading to unstable training and degraded policy performance.

[0003] To address the policy shift problem in offline training, several algorithms have introduced behavior cloning constraints on top of traditional reinforcement learning, enabling policies to be constrained to the vicinity of the data distribution. Among them, TD3+BC is an improved method combining TD3 (Twin Delayed Deep Deterministic Policy Gradient) and Behavior Cloning (BC). By adding a behavior cloning loss term to the Actor update process of TD3, it guides the policy to more closely resemble the original data distribution, effectively mitigating the problem of policy shift. This method performs well on multiple standard datasets, combining simplicity of implementation, stable training, and practicality. However, the policy optimization process of TD3+BC still heavily relies on the numerical accuracy of the Q-function, with its policy gradient directly provided by the Q-value as a supervision signal. When the Q-function exhibits systematic bias or high variance, policy learning is easily misled, and performance may even collapse. This problem is particularly severe in scenarios with sparse training data or scarce rewards.

[0004] To enhance the robustness of policy learning and reduce reliance on the absolute value of Q-values, an empirical ranking-based policy optimization method is proposed. The core idea of ​​this method is to introduce a Rank function to characterize the relative order of superiority among different actions, thus transforming the policy optimization objective from "maximizing Q-values" to "maximizing ranking scores." Compared to directly using Q-values, this method relies solely on the Q-function to provide relative ranking information between actions, making it more tolerant of Q-function biases and significantly improving the stability and generalization ability of the policy optimization process under noisy data or estimation error conditions. This method can be seamlessly integrated into the TD3+BC framework while retaining its structural advantages, providing a more robust optimization path for offline reinforcement learning. Summary of the Invention

[0005] This invention proposes an offline reinforcement learning policy optimization method and system based on empirical quantile ranking, aiming to address the problems of estimation bias, instability, and poor generalization ability of traditional Q-value supervision methods in offline reinforcement learning. This method uses "relative advantage ranking" instead of "absolute Q-value" as the policy learning objective, thereby guiding the agent to learn more discriminative behavioral policies without relying on environmental interaction. It is widely applicable to intelligent control tasks with limited data and high risk sensitivity.

[0006] The core algorithm flow is as follows:

[0007] Unlike traditional Q-value-based reinforcement learning methods, this invention innovatively introduces a ranking function network to characterize the relative Q-value ranking of policy behavior in historical experience. By constructing supervisory signals through empirical quantiles and training the dominance function, it avoids problems such as large Q-value scaling, incomparability across tasks, and instability in the early stages of training. The entire algorithm framework comprises five steps: Critic training, ranking supervision construction, ranking network training, Actor training, and target network soft update, forming an efficient, stable, and robust policy learning process.

[0008] In the process of constructing the ranking supervision, this invention uses the Empirical Cumulative Distribution Function (ECDF) to rank the target Q value, mapping the ranking position of each sample to an unbiased estimate in the interval [0,1], which serves as the supervision signal for the dominance function network. Compared with traditional numerical regression, this ranking supervision signal has the following advantages: (1) automatic normalization without additional scale adjustment; (2) relatively stable ranking and strong anti-interference ability; (3) more direct optimization guidance for the policy network.

[0009] In terms of engineering implementation, this invention offers advantages such as high modularity and ease of deployment. The ranking network shares the same structural framework and input format as the Critic network, differing only in the supervision objective, making it easy to integrate into existing reinforcement learning systems. Furthermore, the ranking supervision construction process does not rely on external environment interaction and can be directly applied to offline static datasets, significantly improving the algorithm's real-world adaptability.

[0010] The specific construction process of this invention is as follows: First, batch data is sampled from the offline data buffer, including the current state, the action taken, the next state, the immediate reward, and the discount factor. When calculating the target Q-value, a delayed target policy network (Target Actor) is used to generate the next action, and truncated Gaussian noise is added to enhance the policy's robustness. Then, the target Q-function (TargetCritic) is used to score the next state and policy action, and the smaller value of Q1 and Q2 is selected to alleviate the Q-value overestimation problem. Finally, this value is substituted into the Bellman update formula to obtain the target Q-value used for advantage training:

[0011] Subsequently, based on the target Q-value of this batch of samples, the torch.sort operation is used to sort them in ascending order, obtaining ordered values ​​and their corresponding original indices. The empirical quantile of the i-th sample in the sorted sequence is then calculated. , where n represents the number of samples, to obtain the sorting labels corresponding to the target Q value.

[0012] Finally, the sorting results are restored according to the original index to form a sorting supervision signal corresponding to the sampled data.

[0013] During the training phase of the ranking network, the state and action are taken as input, and the relative ranking score of the current action is output. The output of the ranking function is normalized to the [0,1] interval using the sigmoid function, and the mean squared error loss (MSE Loss) is used to minimize the difference between the ranking supervision signal and the target signal, thereby guiding the dominant network to learn the ranking information for each action.

[0014] The aforementioned mechanism for constructing the ranking supervision signal is equivalent to replacing the traditional Q-value function with an empirical quantile ranking function as the training objective for behavior evaluation. This mechanism emphasizes the relative advantage of an action in the current batch of samples, effectively mitigating the problem of misleading gradient signals caused by inconsistent Q-value scales or training instability. Furthermore, due to the inherent normalizing properties of quantiles, it also improves generalization and robustness across different tasks and data distributions.

[0015] In the method of this invention, the ranking function not only constructs the dominance function supervision signal, but also plays a crucial role in the update process of the policy network. The ranking function, through empirical quantiles, provides an unbiased and normalized characterization of the relative merits of the current action, thereby guiding the optimization direction of the policy and making the policy more inclined to generate high-quality actions that are superior in ranking. Specifically, in each policy network update phase, actions generated using the current policy are... The ranking score relative to historical actions is evaluated through a ranking network. Since the advantage network has learned to rank action quality through ranking supervision signals, this output can be seen as the relative ranking score of the current policy output action. We further construct a policy loss function with ranking scores as weighting coefficients, in the following form:

[0016] The first term encourages the strategy to generate actions that rank highly in terms of advantage, while the second term is an imitation learning term used to maintain the proximity between the strategy and the behavioral strategy. This represents the dynamic weight between the two.

[0017] Through the above design, the ranking function gives the policy network a "relative advantage-guided" optimization objective, which makes policy updates not only consider the absolute Q value or reward of the action, but also its relative advantages and disadvantages in historical data, thereby improving the generalization performance and stability of the policy on offline data. Attached Figure Description

[0018] Figure 1 This invention provides an overall architecture diagram of an offline strategy optimization method based on empirical quantile ranking.

[0019] Figure 2 This is the neural architecture of the Critic network in the example of this invention.

[0020] Figure 3 This is the neural architecture of the sorting network in the example of the present invention.

[0021] Figure 4 This is the neural architecture of the Actor network in the example of this invention.

[0022] Figure 5 This is a comparison chart showing the normalized scores of the original TD3+BC using the offline strategy optimization method based on empirical quantile ranking of the present invention and the halfcheetah environment medium-random-v2 dataset in D4RL. Detailed Implementation

[0023] To make the objectives, technical solutions, and beneficial effects of this invention clearer and more explicit, the invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described are for illustrative purposes only and do not constitute a limitation on the scope of protection of this invention. Without departing from the spirit and substance of this invention, those skilled in the art can make various modifications or equivalent substitutions to the embodiments based on this specification, all of which should be covered within the scope of protection of this invention.

[0024] In a preferred embodiment of the present invention, the entire training process mainly includes key steps such as neural network initialization, construction of sorted supervision signals, and policy optimization and updating. The implementation process of each stage will be described in detail below in sequence.

[0025] Step 1: Constructing a Neural Network In the initialization phase, three key sub-networks need to be constructed, each undertaking a different responsibility and working together to complete policy learning and value evaluation in the offline reinforcement learning process. This step aims to provide functional support for subsequent ranked supervision generation and policy optimization, and its execution process specifically includes: Define the dimensional information of states and actions, clarify the structure of the observation space and action space, and ensure that the network structure adapts to the task requirements; Initialize the Actor network, whose input is a state vector and output is a continuous action vector. The Tanh function is used to ensure that the range of output actions is limited to a predefined legal interval. Initialize the Critic network, whose input is a concatenated vector of state and action, and whose output is a Q-value estimate of the current state-action pair, which serves as the basis for policy evaluation and target generation; The initialization sorting network is consistent with the Critic network, but the output is compressed to the [0,1] interval by the Sigmoid function, which represents the normalized sorting score of the relative sorting, and is used to build quantile labels later; Independent optimizers are used to manage parameter updates for the three types of networks separately, so as to flexibly adjust the learning rate and optimization strategy of each part; The Actor network, Critic network, and ranking network are copied as corresponding target networks to generate training objectives, thereby reducing the risk of non-stationarity and distribution shift during training.

[0026] After completing the above initialization steps, you can begin constructing experience ranking labels and enter the supervised training phase of the reinforcement learning strategy.

[0027] • Actor Network: This network is used to map policy decisions from the state space to the action space. It adopts a three-layer feedforward fully connected structure, with ReLU activation in the hidden layer and Tanh mapping in the output layer to satisfy the action range constraints. • Critic Network: Consists of two parallel structures (Q1 and Q2) used to estimate the reward value of each state-action pair, mitigating the Q-value overestimation problem through a minimum value technique; • Ranking Network: Its structure is consistent with the Critic subnetwork. It is used to learn the ranking advantage relationship of state-action pairs. Its output is normalized by the Sigmoid function to represent relative ranking information.

[0028] Step 2: Training the Critic Network To construct a reliable value evaluation mechanism, the Critic network needs to learn the true reward estimate of the state-action pair by minimizing the TD error. The specific training process is as follows: For each sample Use the target Actor to generate the next action. ,in To truncate Gaussian noise; The corresponding double Q value estimate is calculated using the target Critic network. and And take the minimum value as the target return. ; The current Critic network predictions The TD error loss function is formed by combining the target Q value and the target Q value:

[0029] Use this loss function to backpropagate and update the Critic parameters; Repeat the above process to make the Critic gradually approach the true return function.

[0030] Step 3: Generate sorting labels (ECDF calculation) After initializing the neural network, a supervisory signal needs to be provided to the ranking network using a quantile ranking method. The core of this step lies in constructing the empirical cumulative distribution function (ECDF) labels to guide the network in learning the relative value ranking of state-action pairs. To improve training stability, the ranking process does not directly use the original Q-values, but instead ranks them based on the TD target values ​​calculated by the Critic target network. The specific execution process is as follows: Calculate the target Q value: For each sample in a training batch, calculate the target Q value using the TD3 standard; Ascending sort: Sort the Q values ​​of all samples in ascending order to construct a sorted sequence; Calculate empirical quantile labels: Assign the corresponding quantile label to the i-th sample after sorting. The calculation method is as follows: , where n is the total number of samples in the batch, ensuring that all quantile values ​​fall within the interval [0,1]; Label back-mapping: To maintain a one-to-one correspondence between the labels and the original input samples, the above quantile results are back-mapping back to the original sample order; As a training supervision signal: the quantile labels of the samples in this batch are used as the regression target of the ranking network for supervised learning.

[0031] Through the above process, the ranking network can learn to output a normalized ranking score that reflects relative value without relying on an exact Q-value fit, thereby enhancing the stability of training and the ranking expression ability.

[0032] Step 4: Train the sorting network After obtaining the empirical quantile labels (ECDF labels) calculated based on the target Q-value, a ranking network (hereinafter referred to as the ranking network) needs to be trained so that its output can accurately reflect the relative superiority or inferiority of the state-action pair under the current policy. This network does not directly regress the absolute value of the Q-value, but learns a ranking mapping function in the interval [0,1], and the output represents the ranking probability or normalized score.

[0033] The training process for the ranking network is as follows: Using state-action pairs (s, a) as input, the output probability scores are obtained by forward propagation through a sorting network. Align the network output with the quantile labels obtained in step two as a monitoring signal; A loss function is constructed, using mean squared error to measure the deviation between the predicted ranking output and the target label, as specifically expressed below:

[0034] Where n is the number of samples in the training batch. This is the predicted output for the I-th sample. For the corresponding empirical quantile labels The aforementioned loss is backpropagated to update the sorting network parameters; Repeat the above process for batch training, so that the ranking network gradually fits the relative order between Q values, thereby achieving fine-grained modeling of the quality of the policy output actions.

[0035] Compared to regression functions based on direct Q-values, the quantile supervision signals used in ranking networks have stronger stability and generalization ability, which can effectively alleviate the training instability caused by Q-function bias and help improve the performance of the strategy in out-of-distribution states.

[0036] Step 5: Train the policy network The policy network is used to generate optimal actions given a state. Its training objective is to maximize the normalized score of the corresponding action in the ranking network for the current state, thereby improving policy quality. Simultaneously, to prevent the policy from deviating excessively from the original empirical distribution, a behavior cloning loss is introduced to regularize the policy. The specific training process is as follows: Input the state sample s from the current batch into the policy network to generate predicted actions. ; The generated policy action is concatenated with the state and then input into the ranking network to obtain its normalized ranking score, which is used as the advantage estimate of the current policy action. The policy network is updated with the goal of maximizing the expected ranking score in the offline reinforcement learning process. Simultaneously, the behavioral cloning loss between the policy action and the corresponding action in the offline data is calculated, in the form of:

[0037] The ranking score loss and behavior cloning loss are weighted and combined to construct the total loss function as follows:

[0038] in, These are adjustable hyperparameters used to control the weight balance between ranking-driven learning and behavioral cloning constraints; Backpropagation is performed on the loss function to optimize the policy network parameters. The ranking network has its parameters fixed during the policy training phase and is only used to provide ranking scores.

[0039] This step guides the strategy towards better actions in the empirical data by maximizing the ranking score, while using behavioral cloning to suppress excessive deviation, thus achieving a balance between performance and stability.

[0040] Step Six: Synchronize the Policy Update Mechanism with the Target Network To improve training stability and prevent policy divergence, a separate update frequency and soft synchronization mechanism are used to coordinate the updates of the policy network, evaluation network, and ranking network. The specific process is as follows: Updating the Critic and Rank networks: In each training iteration, a forward propagation and parameter update are performed on the Critic and Rank networks to enable them to capture the value assessment and relative ranking of state-action pairs in a timely manner. Delayed updates to the policy network: To prevent the policy from overfitting the instantaneous estimates of the evaluation network, the policy network is updated only once every k steps (e.g., k=2), ensuring that the policy optimization process responds more stably and robustly to the advantage signal; The soft update mechanism for the target network: To further suppress training oscillations, a soft update mechanism is used to smoothly update the parameters of both the policy target network and the evaluation target network after each main network update. The update formula is as follows:

[0041] in, These are the target network parameters. Indicates the current network parameters. This is the soft update coefficient, which is usually set to 0.005 to achieve stable and gradual parameter replacement.

[0042] This mechanism enables the co-evolution of the policy network and the value estimation module, effectively mitigating the problem of accumulated valuation bias during policy training and improving the stability and convergence of the overall training process.

[0043] Step 7: Selection and Use of Training Dataset The training data used in this invention comes from the standard open reinforcement learning dataset D4RL (Datasets for Deep Data-Driven Reinforcement Learning), which was released by the Berkeley AI Research Lab. This dataset is widely used for the evaluation and development of offline reinforcement learning algorithms and has the advantages of being open and transparent, having broad coverage, and having controllable quality. It can meet the training needs of ranking supervision networks and policy optimization networks.

[0044] The specific usage process is as follows: Dataset Selection and Loading: Based on the experimental task objectives, select corresponding environments from the D4RL dataset (e.g., halfcheetah, hopper, walker2d, etc. in the MuJoCo control tasks), and load the data using the environment registration interface to obtain an offline experience data pool containing a large number of state-action-reward transition sequences. .

[0045] Data structure composition: Each trajectory includes the current state, the action to be performed, the next state, the immediate reward, and a termination marker. It also includes environment-defined state and action dimensions for feature processing and normalization operations in subsequent value function iteration, ranking supervision, and policy training.

[0046] Data Processing and Batch Sampling: During training, fixed-size mini-batch data is randomly sampled from the D4RL dataset in batches for target Q-value estimation, empirical quantile ranking label generation, ranking network training, and policy updates. To improve offline sample utilization and diversity, data pre-buffering and random rearrangement mechanisms are employed.

[0047] Sample distribution adaptation and task generalization: In response to the data distribution offset and trajectory quality differences (such as different subsets such as medium, expert, and replay) in the D4RL dataset, this invention converts action quality into relative ranking labels through an empirical ranking mechanism, so that the training process does not need to rely on the reward information of absolute scale, thereby improving the generalization ability and robustness under diverse data quality.

[0048] Legality statement: The D4RL dataset is a publicly released academic research dataset, used only for non-commercial technical research and method verification, which is consistent with the invention purpose stated in the current patent.

[0049] By using the D4RL dataset as the offline data source for this invention, not only is the consistency and repeatability of the experiment guaranteed, but the ranking function training process also has good universality and practical applicability, providing a high-quality and controllable empirical foundation for policy optimization networks.

[0050] Step 8: System Deployment and Application Interface Design To ensure the efficient application and flexible integration of this invention in practical engineering, this section details a system deployment scheme and key interface design for offline reinforcement learning tasks. The overall system structure covers core functional modules such as data processing, model training, and policy inference, supporting rapid deployment and invocation in simulation or real-world environments.

[0051] The system features a modular deployment architecture, providing a unified data loading and standardized interface for reading required data from standard offline datasets (such as D4RL) and performing a series of preprocessing operations, including state normalization, action constraints, and reward pruning. This module provides the interface `load_dataset(env_id)` to load data environments for different tasks, and also supports efficient sampling of training samples from the data buffer pool via `sample_batch(batch_size)`, exhibiting good environment versatility and scalability. The training and optimization module integrates the training processes of the ranking network, Critic network, and Actor network into a unified optimization workflow, supporting single-machine or multi-GPU parallel training. This module provides a flexible hyperparameter management mechanism, allowing users to set parameters such as learning rate, target update frequency, policy delay steps, and behavior clone weights via configuration files. Upon completion of training, this module outputs the policy network model file (e.g., .pt format) and complete training logs for use in subsequent deployment or evaluation phases.

[0052] The strategy reasoning and application interface module provides a strategy service interface to the upper-layer system, facilitating the provision of action prediction functionality after deployment. The `predict_action(state)` interface receives the environment state and outputs the optimal action generated by the current strategy; the `evaluate_rank(state, action)` interface evaluates the ranking score of any state-action pair, assisting the upper-layer system in understanding strategy preferences; and the `get_policy_stats()` interface returns statistical information during strategy execution, such as strategy entropy and behavioral bias, for monitoring and performance evaluation.

[0053] System overall deployment capabilities: It supports compatibility with the native evaluation interface of the D4RL framework, facilitating the output of standardized performance metrics (such as normalized scores). Furthermore, in real-world equipment control scenarios with state observation capabilities, the policy network output can be directly used as control input, applied to complex engineering decision-making tasks such as path planning, operation scheduling, and task switching. In addition, the system supports compatibility with heterogeneous state input types, including vector states and image states, and supports processing discrete or continuous action spaces, adapting to industrial-grade reinforcement learning optimization problems.

[0054] Code implementation: Model components are developed using the PyTorch framework, and training visualization and performance tracking can be performed using TensorBoard or WandB.

Claims

1. An offline reinforcement learning policy optimization method and system based on empirical quantile ranking, characterized in that, Includes the following steps: Three types of neural network modules are constructed: policy network, value evaluation network, and ranking network, which are used for action generation, Q-value evaluation, and action ranking score prediction, respectively. In each training round, a state-action-reward-end marker-next state quintuple is sampled from a pre-stored offline experience dataset, and the target Q-value is calculated to estimate the future reward of the action. Based on the target Q value, empirical quantile labels are constructed, and the Q values ​​of each sample are sorted and mapped to the [0,1] interval to form a sorting supervision signal; Train the ranking network so that its output ranking score approximates the empirical quantile labels mentioned above, and minimize it using mean squared error loss; Train the policy network, maximize the score of the current policy action evaluated by the ranking network, and introduce behavior cloning loss to regularize it; A soft update strategy is adopted to periodically synchronize the target network, thereby improving training stability and convergence.

2. The method as described in claim 1, characterized in that, The sorting labels are constructed using the empirical cumulative distribution function (ECDF), defined as follows: ； in Let be the empirical quantile label of the i-th sample, n be the total number of samples in the batch, and rank represent the position in the Q-value ranking.

3. The method as described in claim 1, characterized in that, The sorting network structure is the same as the Critic network, both being three-layer feedforward neural networks that use the ReLU activation function. However, its output is compressed to the [0,1] interval by the Sigmoid function, representing the relative sorting probability.

4. The method as described in claim 1, characterized in that, The loss function of the policy network consists of a ranking score guide term and a behavior cloning term, in the form of: ； in For balance coefficient, The policy network outputs actions. For the action of sampling from the dataset, The sorting network outputs action scores.

5. The method as described in claim 4, characterized in that, The weighting coefficients It can be manually adjusted according to the training environment and dataset to better achieve policy convergence training in different environments and datasets.

6. The method as described in claim 1, characterized in that, The target Q-value is calculated using the TD3 algorithm, which uses the minimum value output by two independent Critic networks to alleviate the Q-value overestimation problem. Its expression is as follows: 。 7. The method as described in claim 1, characterized in that, The policy network is updated every k steps to reduce the policy's sensitivity to the current valuation error, where k is a positive integer and is set to 2 by default.

8. The method as described in claim 1, characterized in that, The soft update mechanism is implemented using the following formula: ； in For the current network parameters, For the target network parameters, This is the soft update factor, with a default value of 0.

005.

9. The method as described in claim 1, characterized in that, The sorting supervision signal does not depend on the absolute value of the original Q value, but only focuses on the relative superiority and inferiority of the actions, thereby improving the algorithm's robustness to data bias and noise.

10. The method as described in claim 1, characterized in that, The empirical data used for training comes from the publicly available offline reinforcement learning dataset D4RL, which covers multiple control tasks in the MuJoCo environment, and is standardized to improve sample consistency and feature transferability.

Images