A preference reinforcement learning method based on dynamic boundary division

Abstract

This invention relates to the field of reinforcement learning and preference modeling technology, specifically to a preference reinforcement learning method based on dynamic boundary partitioning. Through an adaptive boundary trajectory construction mechanism and a binary search-assisted sample allocation strategy, it achieves efficient human preference querying and reward model training. Specifically, it reduces the number of human comparisons by dynamically maintaining multi-level sample boundaries, generates high-quality preference pairs using intra-group smoothing and inter-group comparison, and gradually improves the discriminative ability of the reward model by combining a course learning strategy. This effectively solves the problems of high query cost, low sample utilization efficiency, and reward overfitting in traditional preference learning, significantly improving the sample efficiency and policy performance of reinforcement learning systems based on human feedback.

Description

Technical Field

[0001] This invention relates to the fields of reinforcement learning and preference modeling, specifically to a preference reinforcement learning method based on dynamic boundary partitioning. Background Technology

[0002] In the fields of reinforcement learning and artificial intelligence, human feedback-based reinforcement learning (RLHF) plays a crucial role in complex decision-making tasks such as robot manipulation, game AI, dialogue systems, and autonomous driving. By transforming human preferences into reward signals, RLHF can guide agents to learn policy behaviors that align with human values. However, in practical applications, obtaining high-quality human preference annotations faces significant challenges: the time cost of human experts is high, the preference query process is time-consuming and laborious, and the contribution of samples of different quality levels to the training of reward models varies greatly, resulting in serious bottlenecks in traditional preference learning methods in terms of sample efficiency, query cost, and model generalization ability.

[0003] Traditional human preference learning methods typically employ random or uncertain sampling strategies to collect trajectory samples, then generate preference data through pairwise comparisons to train reward models. However, in complex task environments, this approach suffers from three core problems: First, query efficiency is low, as random sampling cannot effectively identify the most informative samples, resulting in a significant waste of human comparisons on trajectories of similar quality; second, sample utilization is insufficient, as existing methods often ignore the hierarchical relationship between sample quality, failing to fully utilize information from intra-group smoothing and inter-group comparisons; and finally, there is the risk of reward overfitting, as reward models are prone to overfitting noisy labels when preference data is of inconsistent quality or insufficient in quantity, leading to policy learning deviating from true human preferences.

[0004] To overcome these limitations, researchers have begun exploring more efficient preference query and reward learning frameworks. However, existing methods still face key challenges in query strategy optimization, sample organization, and training stability: While active learning-based uncertainty sampling methods (such as Disagreement-based Sampling) can identify samples with high information content, they are prone to getting trapped in local optima in complex tasks, leading to query bias and insufficient sample diversity; ranking-based methods (such as Rank-based Preference Learning) attempt to utilize the relative quality relationships between samples, but lack dynamic adaptation mechanisms, making it difficult to cope with changes in sample distribution during strategy evolution; while progressive training strategies based on curriculum learning can improve training stability, they typically rely on predefined difficulty metrics and cannot dynamically adjust the learning progress based on real-time feedback. Furthermore, when dealing with large-scale trajectory pools, the computational complexity of existing methods increases linearly with the number of samples, making it difficult to meet the needs of real-time interactive scenarios.

[0005] To address these issues, researchers have attempted various improvements: including using ensemble learning methods (such as Reward Ensemble) to identify uncertain samples by analyzing the differences in predictions from multiple reward models. However, the computational cost of ensemble models increases significantly (inference latency increases by 3-5 times), and query blind spots still exist in sparse sample regions. Active query strategies based on Bayesian optimization (such as Bayesian Preference Elicitation) attempt to model the probability distribution of human preferences, but the computational complexity of posterior inference in high-dimensional trajectory spaces is extremely high (time complexity...). The methods are difficult to scale to large-scale tasks. While adversarial training methods (such as Adversarial Reward Learning) enhance the robustness of reward models, the generation of adversarial examples introduces additional training instability and can easily lead to pattern collapse in the reward function. Graph-based neural network-based sample relationship modeling methods (such as Graph-based Preference Learning) capture complex dependencies between trajectories, but the construction and updating of graph structures require significant computational resources, making them difficult to deploy in real-time scenarios. Multi-stage training frameworks (such as Two-stage Reward Learning) first perform coarse-grained screening and then fine-grained comparison, but the accumulation of errors between stages leads to a decrease in the performance of the final reward model (a 10-15% reduction in rewards in standard benchmark tests). These methods still have fundamental limitations: the trade-off between query efficiency and sample diversity remains unresolved; static sample organization methods are difficult to adapt to dynamically evolving policy distributions; the balance between training stability and computational efficiency is difficult to achieve; and a gap exists between general-purpose methods and task-specific requirements. Summary of the Invention

[0006] The purpose of this invention is to provide a preference reinforcement learning method based on dynamic boundary partitioning, which aims to solve the problems of low sample efficiency and high human feedback cost caused by existing human feedback-based reinforcement learning methods, such as low query efficiency, insufficient sample utilization, and reward overfitting. The invention achieves a unified optimization of efficient preference pair generation, stable reward model training, and policy performance improvement under a limited human feedback budget.

[0007] To achieve the above objectives, this invention provides a preference reinforcement learning method based on dynamic boundary partitioning, comprising the following steps:

[0008] Step 1: Initialization A boundary trajectory is defined and a quality-ordered set of boundaries is maintained. The boundary trajectories are used to classify sample quality levels. Boundary trajectories are selected from the initial trajectory pool using a uniform sampling strategy to ensure that the boundaries cover different quality levels.

[0009] Step 2: For newly acquired sample trajectories, a binary search strategy is used to compare them with boundary trajectories to determine their quality group. Each comparison requires only one human preference judgment, with a time complexity of O(n log n). ;

[0010] Step 3: Generate within-group preference pairs and between-group preference pairs based on the group to which the sample belongs. Within-group preference pairs are used to smooth out quality differences within groups, while between-group preference pairs are used to strengthen quality differentiation between groups. Within-group preference pairs are formed by randomly selecting two samples from the same group, while between-group preference pairs are formed by selecting one sample from each of different quality groups.

[0011] Step 4: Sort the preference pairs according to the course learning strategy, prioritize training the less difficult between-group preference pairs, and gradually transition to the more difficult within-group preference pairs. The difficulty coefficient is defined as the absolute value of the quality difference between the groups to which the two samples belong.

[0012] Step 5: Train the reward model using the ranked preference pairs, calculate the preference probability using the Bradley-Terry model, and optimize the reward function. The formula for calculating the preference probability is:

[0013]

[0014] in For the sigmoid function, For trajectory Reward points;

[0015] Step 6: Periodically update the boundary trajectory set, promote high-quality samples to new boundaries, eliminate low-quality boundaries, maintain the dynamic adaptability of the boundaries, and maintain two buffer boundaries above the highest quality group to capture potential high-quality samples.

[0016] Optionally, the initialization of the boundary trajectory in step 1 is selected from the initial trajectory pool using a uniform sampling strategy. Compared with the uncertain sampling strategy, the uniform sampling strategy can achieve higher reward model training efficiency with the same number of human queries.

[0017] Optionally, the time complexity of the binary search strategy described in step 2 is... ,in To determine the number of boundary trajectories, samples are assigned to them through recursive comparison. One of the quality groups significantly reduced the number of human comparisons.

[0018] Optionally, in step 3, the generation of intra-group preference pairs follows the principle of uniform distribution, randomly selecting two samples from the same group to form a preference pair, which is marked as a tie; the generation of inter-group preference pairs follows the principle of cross-group, selecting one sample from each of different quality groups to form a preference pair, and the high-quality group sample is marked as a preference item.

[0019] Optionally, the course learning strategies described in step 4 are ranked according to the difficulty coefficient of the preference pairs. The difficulty coefficient is defined as the absolute value of the quality difference between the groups to which the two samples belong, i.e. ,in and They represent the first Group and No. The preference probabilities of the Bradley-Terry model described in step 5 are optimized by maximizing the log-likelihood function of the preference pairs to improve the reward model parameters.

[0020] Optionally, the boundary update strategy in step 6 includes: for newly collected trajectory segments First, compare it with the current highest quality boundary. Compare; if If the boundary update is not performed, then no boundary update will be performed; if Then it is further compared with the two buffer boundaries, and according to Three update scenarios are performed relative to the position of the buffer boundary: when At that time, identify the active boundary with the lowest sampling frequency in the previous training cycle and remove it, then shift all subsequent boundaries one position to the left. Set as the new highest boundary; when When, use the current one first. Execute standard updates, and then Replace with new ;when At that time, use the current Implement standard updates, Move to Location, finally Set as new Through the aforementioned buffer boundary-assisted update mechanism, the active boundary is ensured to evolve gradually with the policy, maintaining a balanced and information-rich grouping structure.

[0021] Optionally, the method also includes a buffer boundary mechanism: maintaining two buffer boundaries above the highest quality group to capture potentially high-quality samples, and when the number of samples within the buffer boundaries reaches a threshold, elevating them to formal boundaries and triggering an overall adjustment of the boundary set.

[0022] This invention provides a preference reinforcement learning method based on dynamic boundary partitioning. Through a dynamic boundary trajectory construction mechanism and a binary search-assisted sample allocation strategy, it achieves efficient human preference query and reward model training. Specifically, it reduces the number of human comparisons by dynamically maintaining multi-level quality boundaries, generates high-quality preference pairs by utilizing intra-group smoothing and inter-group comparison, and gradually improves the discrimination ability of the reward model by combining a course learning strategy. This effectively solves the problems of high query cost, low sample utilization efficiency, and reward overfitting in traditional preference learning, and significantly improves the sample efficiency and policy performance of reinforcement learning systems based on human feedback. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a schematic diagram illustrating the principle and structure of a preference reinforcement learning method based on dynamic boundary partitioning according to the present invention.

[0025] Figure 2 This is a schematic diagram of the experimental results of a specific embodiment of the present invention in a robot motion control simulation scenario.

[0026] Figure 3 This is a schematic diagram of the ablation experiment results in a specific embodiment of the present invention. Detailed Implementation

[0027] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0028] This invention provides a preference reinforcement learning method based on dynamic boundary partitioning, comprising the following steps:

[0029] Step 1: Initialization The system establishes boundary trajectories and maintains a quality-ordered set of boundaries, which are used to classify sample quality levels.

[0030] Step 2: For newly collected sample trajectories, compare them with boundary trajectories using a binary search strategy to determine their quality group. Each comparison requires only one human preference judgment.

[0031] Step 3: Generate within-group preference pairs and between-group preference pairs based on the group to which the sample belongs. Within-group preference pairs are used to smooth out quality differences within groups, while between-group preference pairs are used to enhance quality differentiation between groups.

[0032] Step 4: Sort the preference pairs according to the course learning strategy, prioritize training the less difficult between-group preference pairs, and gradually transition to the more difficult within-group preference pairs;

[0033] Step 5: Train the reward model using the sorted preference pairs, calculate the preference probabilities using the Bradley-Terry model, and optimize the reward function;

[0034] Step 6: Regularly update the boundary trajectory set, promote high-quality samples to new boundaries, eliminate low-quality boundaries, and maintain the dynamic adaptability of the boundaries.

[0035] Please see Figure 1 , Figure 1 This is a block diagram illustrating the principle and structure of the method of the present invention. The following description, in conjunction with specific implementation steps, provides further explanation:

[0036] In step 1, the implementation process is as follows:

[0037] 1) Boundary trajectory initialization: Initially, samples are taken from the environment. Boundary trajectory Each boundary trajectory contains 50 time steps. Yes. These boundary trajectories are sorted according to preference strength, satisfying... ,in This represents the least preferred sample. These represent the most preferred samples. These boundaries divide the sample space into... Preference groups ,in Including preference strength between and between samples ( (As a virtual boundary in edge cases). Therefore, if Anything belonging to trajectory segment All are superior to any belonging to trajectory segment .

[0038] 2) Boundary Set Maintenance: To maintain the dynamic adaptability of the boundaries, this method employs a buffer boundary mechanism. Two auxiliary buffer boundaries are maintained outside the main boundary set. and This is used to absorb occasional, abnormally high-quality samples and prevent boundary blocking (i.e., abnormally strong samples prematurely occupying the top boundary and disrupting the gradual lifting of the boundary). In experiments on tasks such as Cheetah-run, Sweep-into, and Drawer-open, the number of boundaries was set. This parameter exhibits good balance across tasks of varying complexity.

[0039] In step 2, the implementation process is as follows:

[0040] 1) Binary search strategy: After establishing dynamic boundaries (including buffer boundaries), each newly acquired trajectory segment... The preferences are assigned to groups using a binary search. The search process begins with the complete boundary interval and... With the middle boundary Compare. If Superior If the search continues in the right half, then continue in the left half; otherwise, continue in the right half. This process repeats until... Placed into the correct group The key is that, by leveraging the orderliness of the boundaries, this strategy reduces the comparison overhead for each trajectory segment from... Reduce to This significantly improves the efficiency of preference generation. For example, when In this case, each assignment only requires two human comparisons to determine the group.

[0041] 2) Group assignment example: Assume the boundary set is (in and (as a buffer boundary), new trajectory segment After comparing with each boundary, the preference relationship is determined as follows: Therefore, Assigned to group In DMControl and Meta-World benchmark tests, this binary search strategy significantly reduced the number of human comparisons while maintaining the quality of the generated results based on preferences.

[0042] In step 3, the implementation process is as follows:

[0043] 1) Generation of inter-group preference pairs: Once the trajectory segment When assigned to a group, it can generate multiple preference pairs by leveraging the group's hierarchical structure. Specifically, for those from higher-level groups... Each trajectory segment Preference for Marked as ,express Superior This ensures that each trajectory segment contributes multiple cross-group preference pairs. For example, if , Then, preference pairs are generated. And marked as .

[0044] 2) Within-group preference pair generation: Trajectory segments within the same group are considered to be in a tie. The presence of within-group pairs indicates that their quality is comparable. These within-group pairs serve as regularizers for the reward model: they encourage smooth reward predictions for samples of similar quality and prevent overfitting only to coarse-grained between-group differences. Ablation experiments verify that omitting within-group pairs significantly reduces the performance of the reward model, confirming the necessity of within-group pairs as regularizers.

[0045] 3) Number of preference pairs generated: Assume that all groups have a total of A sample, a certain group has If there are samples, then the number of between-group preference pairs in that group is . The number of in-group preference pairs is This mechanism allows for the generation of multiple preference pairs from a small number of human comparisons, significantly amplifying data efficiency.

[0046] The specific implementation process of step 4 includes:

[0047] 1) Course Learning Strategy Design: To optimize the training efficiency of the reward model, a course learning strategy is applied to rank preference pairs based on inter-group difficulty. Inter-group difficulty is defined. The smaller one Indicate more difficult preference comparisons (e.g., Compare It's even more difficult because and They are closer in terms of preference intensity.

[0048] 2) Difficulty ranking and training scheduling: In practice, training is prioritized. Larger preference pairs (e.g., exist Previously), as training progressed, more difficult pairs (smaller ones) were gradually added. This "easy-to-difficult" curriculum ensures that the reward model gradually learns robust preference distinctions, improving convergence speed and sample efficiency.

[0049] 3) Sorting implementation: Sort all preference pairs according to... Sort in descending order and store in a sorted preference pair buffer. During training, samples are taken sequentially in this order, and the sorting is updated after each training round.

[0050] The execution process of step 5 is as follows:

[0051] 1) Bradley-Terry Model Implementation: The reward model calculates preference probabilities based on the Bradley-Terry model. For two trajectory segments... and The preference probability is defined as:

[0052]

[0053] in express Compare This better aligns with the expectations of human annotators. The reward model is optimized by minimizing the cross-entropy loss between the preference predictor and human preferences:

[0054]

[0055] in Indicates preference for the dataset. The distribution of preference labels provided for humans , respectively represent Superior , Superior There is no difference between the two.

[0056] 2) Reward Model Architecture: The reward model is implemented using an ensemble network consisting of three MLPs. Each MLP contains three layers with 256 hidden units per layer. The output layer uses the tanh activation function to constrain the range of reward values. In the DMControl task, this architecture significantly reduces reward prediction error and exhibits better generalization ability compared to a single-network architecture.

[0057] 3) Training Process: The Adam optimizer was used, with a learning rate set to 0.0005 (Cheetah, Walker tasks), 0.0001 (Quadruped task), or 0.0003 (Meta-World task). The batch size was 512, and the reward model parameters were updated in each training iteration. In the Cheetah-run task, after 100 training rounds, the reward model achieved a prediction accuracy of 92%, significantly outperforming the baseline method (approximately 78%).

[0058] Step 6 is implemented as follows:

[0059] 1) Boundary Update Mechanism: To maintain synchronization between boundary and strategy improvements, an adaptive boundary update mechanism is adopted. For newly acquired trajectory segments... First, compare it with the current top boundary. Comparison. If Instead of performing boundary updates, it groups the data according to the binary search procedure. Assign them to the appropriate groups. If Continue comparing it with the two buffer boundaries and update the active boundary set accordingly.

[0060] 2) Three scenarios for updating rules:

[0061] Case 1: If Execution standard update: Identify the activity boundary with the lowest sampling frequency in the previous period. (That is, using the fewest boundaries in preference comparisons), shift all subsequent activity boundaries one position to the left, and set... ,Right now .

[0062] Scenario 2: If First use the current replace Perform the same standard update, then use replace .

[0063] Scenario 3: If Use the current one again Perform a standard update, and then Move to Finally set .

[0064] 3) The role of buffered boundaries: This buffered update mechanism enables DBbRL to smoothly respond to continuous policy improvements while avoiding sudden boundary shifts caused by isolated high-quality outliers. In the visualization experiment of the Sweep-into task, the version without buffered boundaries exhibited boundary jumps (a single trajectory with a reward of 500 caused the boundary to jump directly from 100 to 500), while the version with buffered boundaries maintained a gradual increase in the boundary, ensuring the stability of the inter-group preference signal.

[0065] 4) Boundary update frequency: Boundary updates are performed once every 20,000 iterations (Cheetah, Walker tasks), 30,000 iterations (Quadruped task), or 5,000 iterations (Meta-World task). This update frequency allows the boundary to adapt to policy improvements in a timely manner, while avoiding training instability caused by excessively frequent updates.

[0066] Optionally, in step 1, the boundary trajectories are initialized using a uniform sampling strategy selected from the initial trajectory pool. Compared to uncertain sampling strategies (such as sampling based on the variance of ensemble model predictions), the uniform sampling strategy achieves higher reward model training efficiency with the same number of human queries. In comparative experiments on Cheetah-run, Sweep-into, and Drawer-open tasks, the uniform sampling strategy consistently outperforms the uncertain sampling strategy. We believe this is because the DBbRL method coarsely groups trajectories through dynamic boundaries and utilizes intra-group smoothing for stable learning, thus benefiting more from the diversity and representative coverage of the state-action space, rather than focusing on regions with high reward uncertainty.

[0067] Optionally, in step 2, the time complexity of the binary search strategy is O(n). ,in The number of boundary trajectories. When At that time, each trajectory segment allocation only requires two human comparisons; when Only three comparisons are needed. This strategy significantly reduces the number of human comparisons while maintaining the impact of preferences on the quality of the generated data.

[0068] Optionally, in step 3, the generation of preference pairs within a group follows the principle of uniform distribution, randomly selecting two samples from the same group to form a preference pair, which is then labeled as a tie. The generation of inter-group preference pairs follows the cross-group principle, selecting one sample from each of the different quality groups to form a preference pair, with the high-quality group sample being labeled as the preference item. .exist Under the boundary settings, the ratio of intra-group pairs to inter-group pairs is approximately 1:2, and this ratio remains relatively stable across different tasks.

[0069] Optionally, in step 4, the course learning strategies are ranked according to the difficulty coefficient of the preference pairs. The difficulty coefficient is defined as the absolute value of the quality difference between the groups to which the two samples belong, i.e. ,in and They represent the first Group and No. Groups. In training scheduling, priority is given to training... Larger pairs (e.g.) ), gradually transitioning to Smaller pairs (e.g.) ).

[0070] Optionally, in step 5, the preference probabilities of the Bradley-Terry model are optimized for reward model parameters by maximizing the log-likelihood function of the preference pairs. The reward model employs an ensemble learning strategy, consisting of three independent MLPs, with the outputs of each MLP being averaged and aggregated to obtain the final reward prediction. In DMControl's Quadruped-run task, the ensemble model reduces the prediction variance by approximately 40% compared to the single model, improving the stability of reward estimation.

[0071] Optionally, in step 6, the boundary update strategy includes: when the number of samples in a group exceeds a preset threshold (e.g., 20% of the number of samples in the group), selecting the highest quality sample from that group as the new boundary to insert into the boundary set; simultaneously, if the size of the boundary set exceeds... (Including buffered boundaries), the lowest quality boundaries are discarded to maintain the compactness and representativeness of the boundary set. In multiple tasks of DMControl and Meta-World, this strategy makes the boundary update process smoother and avoids training instability caused by boundary jumps.

[0072] Optionally, the method also includes a buffer boundary mechanism: maintaining two buffer boundaries above the highest quality group to capture potentially high-quality samples. When the number of samples within the buffer boundaries reaches a threshold, they are promoted to formal boundaries, triggering a global adjustment of the boundary set. In visualization experiments on the Cheetah-run, Drawer-open, and Sweep-into tasks, the buffer boundaries effectively prevented boundary blocking, enabling the boundaries to be progressively raised and maintaining the stability of the inter-group preference signal.

[0073] Furthermore, the present invention also conducts experimental simulations and comparative analyses through specific embodiments:

[0074] This paper compares the preference reinforcement learning method based on dynamic boundary partitioning with existing benchmark methods. The benchmark methods compared are as follows:

[0075] (1) Entropy-based preference sampling method: PEBBLE, which is a widely adopted preference reinforcement learning framework. It selects the trajectory segment with the highest uncertainty for human preference query through the entropy maximization strategy, and improves sample efficiency by utilizing offline learning and unsupervised pre-training.

[0076] (2) Semi-supervised reward learning method: SURF, which expands the number of preference pairs by generating data augmentation and pseudo-labels, and improves the training efficiency of reward model by using unlabeled data, and performs well in scenarios with limited feedback.

[0077] (3) Exploration method based on reward uncertainty: RUNE identifies the trajectory segment with the most information by integrating the prediction variance of the reward model, dynamically adjusts the intrinsic reward weight, and enhances the ability to explore areas with high uncertainty.

[0078] (4) Meta-reward network method: MRN, which adopts a two-layer optimization framework and learns the reward function through implicit differential, so that the reward model can adapt to the policy distribution changes in different stages and improve the consistency of rewards across stages.

[0079] (5) Robust preference learning method: RIME, which filters noisy preference labels through a confidence weighting mechanism and dynamically adjusts sample weights using the confidence interval of reward prediction, thus maintaining stable performance in noisy feedback environments.

[0080] (6) Temporal alignment preference reinforcement learning method: STAIR, which solves the stage misalignment problem by using temporal distance metric and enhances temporal consistency by using contrastive learning, and performs well in long sequence tasks.

[0081] Dataset settings

[0082] Datasets: This invention uses two benchmark datasets, DeepMind Control Suite and Meta-World, to evaluate the performance of the method.

[0083] The DeepMind Control Suite (DMControl) dataset is a standard benchmark dataset for evaluating reinforcement learning algorithms. It contains a series of continuous control tasks to test the motion control capabilities of agents in physically simulated environments. This invention selects three representative tasks for evaluation: Cheetah-run, Walker-walk, and Quadruped-run. These tasks are characterized by continuous state and action spaces and have varying degrees of complexity and challenge. The Cheetah-run task requires the agent to control a 2D cheetah model to run rapidly on a horizontal surface, with a state dimension of 17 and an action dimension of 6. The Walker-walk task requires the agent to control a 2D walker model to maintain balance and move forward, with a state dimension of 24 and an action dimension of 6. The Quadruped-run task requires the agent to control a 3D quadruped robot model to run on complex terrain, with a state dimension of 78 and an action dimension of 12. During the training process of each task, the agent interacts with the environment to generate trajectories. Each trajectory contains 50 time-step (state, action) pairs, which are used for subsequent preference learning and reward model training.

[0084] The Meta-World dataset is a benchmark dataset for evaluating meta-reinforcement learning and multi-task learning capabilities. It contains 50 different robot manipulation tasks, covering various maneuvering skills such as grasping, pushing, opening doors, and drawer manipulation. This invention selects six representative tasks for evaluation: Sweep-into, Window-open, Door-open, Hammer, Drawer-open, and Drawer-close. These tasks are characterized by requiring refined maneuvering skills and long-term planning abilities, and there are significant differences between the tasks. The Sweep-into task requires the agent to use a robotic arm to sweep an object into a designated area, with 12 state dimensions and 4 action dimensions; the Window-open task requires the agent to operate a robotic arm to open a window, with 12 state dimensions and 4 action dimensions; the Door-open task requires the agent to open a door, with 12 state dimensions and 4 action dimensions; the Hammer task requires the agent to use a hammer to hit a nail, with 12 state dimensions and 4 action dimensions; the Drawer-open and Drawer-close tasks require the agent to open and close a drawer, respectively, with 12 state dimensions and 4 action dimensions. During the training of each task, the agent interacts with the environment to generate trajectories. Each trajectory contains 50 time-step (state, action) pairs, which are used for subsequent preference learning and reward model training.

[0085] In the experimental setup, each task had a training cycle of 1 million time steps, and the human feedback budget was set to perform a preference query every 10,000 steps, generating 10 preference pairs per query. All experiments were repeated 5 times, and the mean performance and standard deviation were reported. The evaluation metrics used were the average episode return for the DMControl task and the success rate for the Meta-World task, where the success rate was defined as the percentage of times the task was successfully completed in 100 evaluation rounds.

[0086] Experimental Results Analysis

[0087] The system utilizes DeepMind Control Suite and the Meta-World dataset. The objective is to evaluate the sample efficiency and policy performance of the method under a limited human feedback budget. The model interacts with the environment under an initial policy and learns preferences during continuous policy improvement. The environment cycles through different task difficulties in a fixed order (e.g., easy → medium → hard → repetitive) to verify the method's resistance to forgetting and its ability to continuously improve during long-term training.

[0088] Experimental results show that the proposed method performs exceptionally well on the DMControl dataset, with an average per-round return improvement of approximately 19.3% compared to the best baseline method. Compared to other methods, such as SURF, which does not consider dynamic boundary partitioning and thus has lower preference pair generation efficiency when faced with samples of different quality levels, and RUNE, which focuses on exploring reward uncertainty, is prone to instability in preference signals in continuous learning scenarios, the proposed method effectively solves the query efficiency problem in preference learning through dynamic boundary trajectory construction and a binary search-assisted sample allocation strategy, demonstrating significant sample efficiency and resistance to forgetting.

[0089] It also achieved excellent results on the Meta-World dataset, with an average success rate that is about 68.0% higher than the best baseline method. When facing complex operation tasks such as Hammer, existing methods have encountered performance bottlenecks and limited success rate improvements; however, the method of this invention achieves continuous performance improvements, which shows that the preference reinforcement learning method based on dynamic boundary partitioning can effectively address the policy improvement needs in complex operation tasks and enhance the agent's operational capabilities in diverse task environments.

[0090] Figure 2 This paper demonstrates the practical application of the present invention in robot motion control tasks. By visualizing the motion trajectories of the agent in tasks such as Cheetah-run, Sweep-into, and Hammer, it can be intuitively seen that the agent trained by the method of the present invention can complete motion control tasks more stably and efficiently. Compared with baseline methods, the agent trained by the present invention has significantly improved adaptability in complex dynamic environments, and its motion trajectory is smoother and more natural.

[0091] Figure 3 It shows the number of boundaries The ablation experiment results were compared with different boundary number settings ( The impact on method performance. Experimental results show that when At that time, the method achieved state-of-the-art performance on both the DMControl and Meta-World datasets. The number of boundaries was too small ( This can lead to insufficiently refined quality hierarchy division, resulting in reduced generation efficiency due to bias; an excessive number of boundaries ( This would increase the number of comparisons performed by humans, reducing query efficiency. This result verifies the rationality of the boundary number setting in this invention. It achieves the best balance between learning efficiency and query cost.

[0092] In summary, compared with existing methods, the present invention has the following advantages:

[0093] (1) Step 1 establishes a multi-level quality hierarchy through dynamic boundary trajectory initialization and maintenance, providing a clear grouping basis for subsequent preference pair generation. This eliminates the need for additional prior knowledge or assumptions, and divides the pairs solely based on the intrinsic quality differences of the trajectories, reducing the interference of noisy samples on preference learning;

[0094] (2) Step 2 uses a binary search strategy to efficiently assign new trajectories to the corresponding quality groups. Each assignment only requires... Secondary human comparison significantly reduces query costs. By leveraging the ordered nature of boundaries, the group affiliation of trajectories is dynamically adjusted, narrowing the cognitive differences between different quality levels and enhancing preference learning efficiency.

[0095] (3) Step 3 generates multiple preference pairs by jointly generating intra-group preference pairs and inter-group preference pairs, making full use of a single human comparison to generate multiple preference pairs and avoiding the high cost of repeated queries. Intra-group pairs act as regularizers to smooth the quality differences within groups, while inter-group pairs strengthen the quality distinction between groups, thus achieving high efficiency and diversity in preference pair generation;

[0096] (4) Step 4 uses a course learning strategy to rank the preference pairs by difficulty, prioritizing the training of lower-difficulty between-group pairs and gradually transitioning to higher-difficulty within-group pairs, thereby enhancing the learning stability of the reward model. The ranked preference pairs serve as a unified training sequence, reducing the convergence difficulty during training and improving cross-task consistency;

[0097] (5) Step 5 calculates the preference probability and optimizes the reward function using the Bradley-Terry model, uses the ensemble learning strategy to improve the stability of reward prediction, replaces the traditional hard label with a soft probability distribution, reduces the interference of human preference noise on the training of the reward model, and enhances the robustness of the model.

[0098] (6) Step 6 is based on an adaptive boundary update mechanism to dynamically adjust the boundary trajectory set, promote high-quality samples to new boundaries, and eliminate low-quality boundaries to ensure the continuity of preference learning and the stability of the model. During the update, a buffer boundary mechanism is combined to prevent boundary blocking, realize the gradual growth of the boundary, and ensure the real-time performance and long-term adaptability of preference learning.

[0099] The above description discloses only one or more preferred embodiments of the present invention, and should not be construed as limiting the scope of the present invention. Those skilled in the art will understand that all or part of the processes of the above embodiments can be implemented, and equivalent changes made in accordance with the claims of the present invention are still within the scope of the invention.

Claims

1. A preference reinforcement learning method based on dynamic boundary partitioning, characterized in that, Includes the following steps: Step 1: Initialization The system establishes boundary trajectories and maintains a quality-ordered set of boundaries, which are used to classify sample quality levels. Step 2: For newly collected sample trajectories, compare them with boundary trajectories using a binary search strategy to determine their quality group. Each comparison requires only one human preference judgment. Step 3: Generate within-group preference pairs and between-group preference pairs based on the group to which the sample belongs. Within-group preference pairs are used to smooth out differences in quality within groups, while between-group preference pairs are used to strengthen the differentiation of preference strength between groups. Step 4: Sort the preference pairs according to the course learning strategy, prioritize training the less difficult between-group preference pairs, and gradually transition to the more difficult within-group preference pairs; Step 5: Train the reward model using the sorted preference pairs, calculate the preference probabilities using the Bradley-Terry model, and optimize the reward function; Step 6: Regularly update the boundary trajectory set, promote high-quality samples to new boundaries, eliminate low-quality boundaries, and maintain the dynamic adaptability of the boundaries.

2. The preference reinforcement learning method based on dynamic boundary partitioning as described in claim 1, characterized in that, The initialization of the boundary trajectory in step 1 is achieved by selecting from the initial trajectory pool through a uniform sampling strategy, ensuring that the boundary covers different quality levels and avoiding boundary deviations caused by uncertain sampling.

3. The preference reinforcement learning method based on dynamic boundary partitioning as described in claim 1, characterized in that, The time complexity of the binary search strategy described in step 2 is: ,in To determine the number of boundary trajectories, samples are assigned to them through recursive comparison. One of the quality groups.

4. The preference reinforcement learning method based on dynamic boundary partitioning as described in claim 1, characterized in that, In step 3, the generation of intra-group preference pairs follows the principle of uniform distribution, randomly selecting two samples from the same group to form a preference pair; the generation of inter-group preference pairs follows the principle of cross-group, selecting one sample from each of different quality groups to form a preference pair, and the high-quality group sample is marked as the preference item.

5. The preference reinforcement learning method based on dynamic boundary partitioning as described in claim 1, characterized in that, The course learning strategies described in step 4 are ranked according to the difficulty coefficient of the preference pairs. The difficulty coefficient is defined as the absolute value of the quality difference between the groups to which the two samples belong, i.e. ,in and They represent the first Group and No. Group; The formula for calculating the preference probability of the Bradley-Terry model mentioned in step 5 is as follows: in For the sigmoid function, For trajectory The reward score is obtained by optimizing the reward model parameters by maximizing the log-likelihood function of the preference pair.

6. The preference reinforcement learning method based on dynamic boundary partitioning as described in claim 1, characterized in that, The boundary update strategy described in step 6 includes: when the quality of a new sample exceeds that of the current highest-quality sample, selecting and eliminating the sample with the lowest update frequency from the boundary set; simultaneously, if the size of the boundary set exceeds... (Including buffer boundaries), the lowest quality boundaries are eliminated to maintain the compactness and representativeness of the boundary set.

7. The preference reinforcement learning method based on dynamic boundary partitioning as described in claim 1, characterized in that, The method also includes a buffer boundary mechanism: two buffer boundaries are maintained above the highest quality group to capture potentially high-quality samples. When the number of samples within the buffer boundary reaches a threshold, it is promoted to a formal boundary and triggers an overall adjustment of the boundary set.

Images