Multi-path rollback confidence-aware imitation learning method for robot skill training
By employing a multi-path rollback belief-aware imitation learning method, the problems of parameter sensitivity and training instability in the pre-training stage of traditional imitation learning algorithms are solved, enabling rapid, stable convergence and efficient execution of robot skill training.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- NAT UNIV OF DEFENSE TECH
- Filing Date
- 2026-05-11
- Publication Date
- 2026-06-05
AI Technical Summary
Traditional imitation learning algorithms are sensitive to parameter settings during the pre-training stage in practical applications. The training process is slow and unstable, especially when using imperfect demonstration data, which can easily lead to training divergence due to gradient direction conflicts.
We employ a multi-path rollback confidence-aware imitation learning method, which reduces dependence on a single path through parallel multi-path pseudo-updates and weighted averaging mechanisms. Furthermore, we optimize the policy model through gradient consistency screening and dynamic weighted ensemble to improve training stability and efficiency.
It accelerates the convergence process of robot skill training, improves training stability and execution efficiency, and ensures higher stability and accuracy in complex tasks.
Smart Images
Figure CN122143072A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of robot imitation learning technology, and in particular to a multi-path rollback belief-aware imitation learning method for robot skill training. Background Technology
[0002] Traditional imitation learning algorithms typically assume that demonstration data originates from high-quality experts. However, in practical engineering applications, collecting high-quality expert data faces challenges such as high costs and difficulty. Therefore, current research generally relaxes the quality requirements for demonstration data, encompassing multiple quality levels. Currently, mainstream research on imperfect demonstration data focuses on using ranking algorithms or confidence modeling methods to address quality differences. These methods largely rely on a large amount of manually labeled ranking tags or true confidence information. Confidence-aware imitation learning (CAIL), by jointly optimizing strategies and confidence, effectively utilizes imperfect demonstration data with only a small amount of trajectory ranking labels, significantly improving algorithm performance. However, this method still has limitations in practical applications: the training process is sensitive to parameter settings in the pre-training phase, and its two-layer optimization structure is prone to gradient direction conflicts in the objective functions of a single inner and outer layer, while the confidence is still updated, leading to a slow and unstable training process. Summary of the Invention
[0003] Therefore, it is necessary to provide a multi-path rollback belief-aware imitation learning method for robot skill training that can improve the control stability and efficiency of robot complex skill training, in order to address the above-mentioned technical problems.
[0004] A multi-path rollback belief-aware imitation learning method for robot skill training, the method comprising: Obtain a demonstration dataset consisting of demonstration trajectories of a hybrid mass robot. The demonstration dataset contains the state sequences and action sequences of the robot performing the task.
[0005] Construct a two-layer optimized policy model that includes a policy network, a discriminator network, and a confidence evaluation model.
[0006] The demonstration dataset is input into the two-layer optimization strategy model. The discriminator network and the confidence evaluation model are pre-trained using a parallel multi-path pseudo-update and weighted average algorithm, and the initial parameters are output.
[0007] The following sub-steps are executed iteratively to jointly optimize the two-level optimization strategy model: multiple sets of update candidates are generated in parallel based on the initial parameters.
[0008] Calculate the loss gradients for the inner and outer layers of each update candidate, and evaluate the consistency of direction to filter out effective update parameter results.
[0009] The updated parameter results are used as candidates to assign dynamic weights. After weighted averaging, the discriminator network and confidence evaluation model are updated to obtain the optimized two-layer optimization strategy model. The robot's skills are trained and controlled based on the optimized two-layer optimization strategy model.
[0010] The aforementioned multi-path rollback confidence-aware imitation learning method for robot skill training first introduces a multi-path parallel pseudo-update and weighted averaging mechanism. This mechanism generates multiple independent exploration paths by replicating model parameters. Each path independently attempts parameter updates during pre-training, and the results of each path are weighted and averaged to determine the initial parameters. This significantly reduces the dependence on a single initial path or specific pre-training parameters, avoiding the risk of getting trapped in local optima or causing subsequent training divergence due to poor initialization. Second, addressing the gradient direction contradictions and parameter update oscillations caused by the inherent conflict between the dual objectives of imitation learning and confidence assessment during training, a gradient consistency-based direction selection and dynamic weighted integration are used. In each main iteration, for each group of parallel-generated parameter update candidates, the gradients of their inner imitation learning loss and outer ranking consistency loss are calculated. Through a key gradient consistency judgment operation, only update components with synergistic gradient directions (i.e., simultaneously beneficial to both optimization objectives) are retained, while conflicting update signals are masked. Finally, an adaptive dynamic weighting strategy is used to integrate the selected candidate updates. This strategy does not treat all candidates equally. Instead, it adaptively assigns different fusion weights based on the strength of gradient consistency for each candidate path. Update directions with stronger consistency and clearer contributions are given higher weights, thus dominating the ensemble update. This intelligent weighting mechanism allows the optimization process to continuously focus on the most robust and effective update directions, smoothing the training trajectory and accelerating model convergence. Ultimately, this enables robot skill training to converge to high-performance policies faster and more stably, resulting in superior stability, accuracy, and execution efficiency in complex real-world control tasks. Attached Figure Description
[0011] Figure 1 This is a diagram illustrating an application scenario of a multipath rollback belief-aware imitation learning method for robot skill training in one embodiment. Figure 2 This is a flowchart illustrating a multi-path rollback belief-aware imitation learning method for robot skill training in one embodiment. Detailed Implementation
[0012] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.
[0013] The multi-path rollback belief-aware imitation learning method for robot skill training provided by this invention can be applied to, for example... Figure 1 The MRCAIL (Multipath Rollback Confidence-Aware Imitation Learning) example shown is a more stable and robust framework built on top of the CAIL framework. The demo data, shown in the solid box on the left, is derived from the demo strategy. ,Include Trajectory Each trajectory consists of a state-action sequence. The system learns the confidence level of each trajectory through a confidence modeling module. This is used to reweight the distribution of the demonstration data. Inner optimization is performed in the solid box on the right: Generator. Interacting with the Mujuco simulation environment (Reacher, Hopper, Walker2d, HalfCheetah, and Ant), the system samples trajectories and stores them in the playback buffer. The pseudo-discriminator uses these samples, along with demonstration data, to update and generate its parameters. The outer layer optimization is based on the updated pseudo-discriminator by sampling evaluation data (such as partial ranking). ), calculate external loss And pseudo-update confidence level Finally, the discriminator is updated by weighting the results of multi-path exploration through a gradient consistency mechanism that is jointly optimized from inner and outer layers. With confidence level .
[0014] In one embodiment, such as Figure 2 As shown, a multi-path rollback belief-aware imitation learning method for robot skill training is provided, which is then applied to... Figure 1 Taking the framework in [the document] as an example, the steps include: Step 202: Obtain a demonstration dataset consisting of demonstration trajectories of hybrid mass robots.
[0015] The demonstration dataset contains state sequences and action sequences of the robot performing tasks.
[0016] Specifically, denoted as .in: and Represent the state space and action space respectively; It is the state transition probability of the environment; The reward function is unknown. This is the discount factor. The agent uses a policy... Interact with the environment to generate a trajectory Its performance is measured by expected cumulative return: .
[0017] In imitation learning, given a model dataset ,in The total number of demonstration tracks, each track Each corresponds to a state-action interaction sequence in the MDP. Without loss of generality, the trajectory... Represented as length is ordered sequence: ,in , respectively corresponding The state of time and Actions at specific moments. The core objective of imitation learning tasks is to learn the optimal policy based on a model dataset. , making Generated trajectory distribution Compared with the demonstration dataset The trajectory distribution represented Achieve optimal matching; ultimately, ensure optimal matching through distributed matching. exist The maximum expected return criterion is met.
[0018] In standard imitation learning scenarios, it is often assumed that a model dataset is used. All originate from expert strategies With the optimal strategy Equivalent to maximizing expected returns. In the field of imperfect demonstration studies, demonstration data encompasses expert strategies. Non-expert strategies The generated demonstration samples, in form Each trajectory in From unknown hybrid strategies Generate. (Can be done) Consider it as a convex combination of multiple sub-policies: ,satisfy , .
[0019] Furthermore, in practice, a small amount of trajectory sorting is used to construct evaluation data, namely: In CAIL, a general imitation learning framework for imperfect demonstration data, it receives demonstration datasets containing different levels of optimality. Limited assessment data And it evaluates the loss function, thereby learning a high-performing strategy. This allows CAIL to neither require the existence of an optimal model in the demonstration dataset, nor to learn from mixed demonstration datasets. It accurately extracts effective information and successfully avoids interference caused by suboptimal demonstrations.
[0020] Step 204: Construct a two-layer optimized policy model that includes a policy network, a discriminator network, and a confidence evaluation model.
[0021] Specifically, a learnable confidence function is introduced into the CAIL framework. This function analyzes the state-action pairs in the expert demonstration data. Weighted processing is applied to guide the discriminator's attention, making it more focused on high-quality samples. Furthermore, within the external optimization framework, the CAIL algorithm compares the real trajectory with the policy-generated trajectory to optimize the confidence function. Dynamic updates and optimizations.
[0022] Step 206: Input the demonstration dataset into the two-layer optimization strategy model, and use the algorithm of parallel multi-path pseudo-update and weighted average to pre-train the discriminator network and confidence evaluation model, and output the initial parameters.
[0023] Specifically, the inner imitation loss is designed as follows: the inner goal is to achieve the following at the current confidence level. The goal is to minimize the difference between the generation strategy and the reweighted demonstration distribution. Its loss function consists of two parts: The generator (i.e., policy) loss remains unchanged, and optimization is still performed by maximizing the implicit reward of the discriminator output: ; The loss of the complete discriminator is: ; Furthermore, an outer evaluation loss is designed: to learn the confidence function. CAIL requires an external signal to evaluate the quality of the current strategy. This is especially relevant when a limited number of trajectory sorting labels are available. , ,in Determined by the label ratio. Indicates the first The true quality score of each trajectory. Reward signal induced by the discriminator. It can calculate the model prediction return for each trajectory: ; In the CAIL framework, a modified marginal ranking loss is used as the outer loss: ; in Defined as: ; in A parameter is used to ensure the Lipschitz continuity of the loss function. This loss penalty model predicts a ranking inconsistent with the true ranking and exhibits directionality: when... , otherwise ,when At that time, the loss increases, thus driving Update in the direction of sorting accuracy.
[0024] Step 208, iteratively execute the following sub-steps to jointly optimize the two-layer optimization strategy model: generate multiple sets of update candidates in parallel based on the initial parameters.
[0025] Specifically, before each iteration, the current discriminator parameters are... , copy A separate copy Then, for each copy Each time a subset of the strategy trajectory is sampled from the replay buffer, an inner pseudo-update is performed: ; in The discriminator's pseudo-update learning rate can take different values. For the first The inner discriminator loss calculated by sampling each copy: ; Then, the labels were sorted using the actual trajectory. The pseudo-updated discriminator Predicting returns using models Constructing the outer layer loss , perform confidence Pseudo-update; ; Finish After each parallel pseudo-update, MRCAIL performs a weighted average to obtain the integrated discriminator parameters. , ; , ; This mechanism employs a multi-path pseudo-update strategy to explore the discriminator parameter space in multiple directions and optimizes it through weighted averaging, effectively avoiding the local traps that are easily encountered in single-path updates. During the discriminator pre-training phase, this method significantly enhances the model's sensitivity to the number of pre-training steps. Subsequent experimental results further demonstrate that the multi-path exploration strategy can drastically reduce the number of steps required for pre-training.
[0026] Step 210: Calculate the loss gradients corresponding to the inner and outer layers of each update candidate, and evaluate the consistency of direction to filter the effective update parameter results.
[0027] Step 212: Using the filtered updated parameter results as candidates, assign dynamic weights, and update the discriminator network and confidence evaluation model after weighted averaging to obtain the optimized two-layer optimization strategy model. The robot's skills are then trained and controlled based on the optimized two-layer optimization strategy model.
[0028] Specifically, after obtaining the initial discriminator during the pre-training phase, MRCAIL enters a two-layer optimization main loop. In each iteration, the system first executes... A separate pseudo-update path is obtained and Subsequently, for each pseudo-updated discriminant... and Introducing a multi-gradient guidance mechanism to achieve more reliable parameter updates: 1. Externally guided gradient calculation: using real trajectory sorting labels Discriminator Calculate the outer gradient This is used to subsequently determine whether the update direction of the pseudo-discriminator is appropriate.
[0029] 2. Internal guided gradient calculation: at confidence level Below, discriminator Resample the policy trajectory and calculate the inner optimization gradient. .
[0030] 3. To ensure that the update direction benefits both imitation learning and ranking consistency, MRCAIL performs a dot product on the two sets of gradients and only retains the pseudo-update discriminator parameters of the one with the positive contribution: ; in This represents element-wise multiplication. This operation effectively suppresses the update direction conflict between inner and outer targets. Finally, based on the filtered gradients, softmax is used to adaptively weight each path: ; in Temperature coefficient, final discriminator and Update using weighted average: .
[0031] In the aforementioned multi-path rollback confidence-aware imitation learning method for robot skill training, firstly, a multi-path parallel pseudo-update and weighted averaging mechanism is introduced. This mechanism generates multiple independent exploration paths by replicating model parameters. Each path independently attempts to update parameters during pre-training, and finally, the results of each path are weighted and averaged to determine the initial parameters. This significantly reduces the dependence on a single initial path or specific pre-training parameters, avoiding the risk of getting trapped in local optima or causing subsequent training divergence due to poor initialization. Secondly, to address the gradient direction contradictions and parameter update oscillations caused by the inherent conflict between the dual objectives of imitation learning and confidence assessment during training, a gradient consistency direction screening and dynamic weighted integration are used. In each main iteration, for each group of parallel-generated parameter update candidates, the gradients of their inner imitation learning loss and outer ranking consistency loss are calculated. Through a key gradient consistency judgment operation, only those update components with coordinated gradient directions (i.e., simultaneously beneficial to both optimization objectives) are retained, while update signals with conflicting directions are masked. Then, an adaptive dynamic weighting strategy is used to integrate the selected candidate updates. This strategy does not treat all candidates equally. Instead, it adaptively assigns different fusion weights based on the strength of gradient consistency for each candidate path. Update directions with stronger consistency and clearer contributions are given higher weights, thus dominating the ensemble update. This intelligent weighting mechanism allows the optimization process to continuously focus on the most robust and effective update directions, smoothing the training trajectory and accelerating model convergence. Ultimately, this enables robot skill training to converge to high-performance policies faster and more stably, resulting in superior stability, accuracy, and execution efficiency in complex real-world control tasks.
[0032] In one embodiment, the parameters of the discriminator network and the confidence evaluation model are copied to form k independent parameter copies. For each parameter copy, a pseudo-update based on the inner layer imitation learning objective and a pseudo-update based on the outer layer ranking consistency evaluation objective are performed. The discriminator network parameter copies obtained after the K updates are arithmetically averaged to obtain the initial parameters of the pre-trained discriminator network. Similarly, the confidence evaluation model parameter copies obtained after the K updates are arithmetically averaged to obtain the initial parameters of the pre-trained confidence evaluation model.
[0033] In one embodiment, the inner imitation learning loss function is constructed by training a demonstration data distribution weighted by a confidence assessment model and the current data distribution generated by a policy network. This inner imitation learning loss function includes a discriminator loss function and a policy network loss function. ; ; in, Let the discriminator loss function be... Let the policy network loss function be... The expected reward corresponding to the trajectory distribution generated by the expert strategy. For confidence function, The expected reward for the trajectory generated by the current strategy. The expected reward corresponding to the trajectory distribution generated by the current strategy. For the state-action pairs in the demonstration data, the loss gradient corresponding to each update candidate in the inner layer is calculated based on the inner layer imitation learning loss function. An outer layer ranking consistency loss function is constructed based on the robot demonstration trajectory with manually labeled quality ranking. ; in, Let the outer sorting consistency loss function be... A function to determine whether the predicted sorting is inconsistent with the actual sorting. and These represent the predicted cumulative returns for different iteration rounds calculated based on the reward function induced by the current discriminator. For indicator functions, and The first Article and Section The quality score of manually labeled trajectories. The loss gradient corresponding to the outer layer is calculated using the outer layer ranking consistency loss function.
[0034] In one embodiment, the inner layer imitation loss and the outer layer ranking loss are calculated separately through parallel independent sampling based on the initial parameters, generating multiple sets of update candidates.
[0035] In one embodiment, the element-wise product of the inner layer loss gradient and the outer layer loss gradient corresponding to each group of update candidates is calculated, and the non-positive elements in the product result are set to zero to obtain a non-negative consistency metric vector. ; in, For consistency measure vectors, For discriminator, The inner layer loss gradient, The outer layer loss gradient, This is an element-wise multiplication. Valid update parameter results are selected based on the consistency metric vector.
[0036] In one embodiment, the filtered update parameter results are used as candidates to assign dynamic weights, and the L1 norm of the consistency metric vector corresponding to each group of update candidates is calculated. Based on the L1 norm of all update candidates, the weight coefficients of the update candidates are obtained by normalization using the softmax function. ; in, These are the weighting coefficients. For temperature coefficient, For the first The group consistency metric vector, where K is the total number of updated parameter results. The discriminator network and confidence evaluation model are updated separately after a weighted average of the updated candidate weight coefficients.
[0037] It should be understood that, although Figure 2 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 2 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.
[0038] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), Synchlink, DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.
[0039] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0040] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the appended claims.
Claims
1. A multi-path rollback belief-aware imitation learning method for robot skill training, characterized in that, The method includes: Obtain a demonstration dataset consisting of demonstration trajectories of a hybrid mass robot; the demonstration dataset includes the state sequence and action sequence of the robot performing the task; Construct a two-layer optimized policy model that includes a policy network, a discriminator network, and a confidence evaluation model; The demonstration dataset is input into the two-layer optimization strategy model, and the discriminator network and the confidence evaluation model are pre-trained using a parallel multi-path pseudo-update and weighted average algorithm, and the initial parameters are output. The following sub-steps are executed iteratively to jointly optimize the two-layer optimization strategy model: multiple sets of update candidates are generated in parallel based on the initial parameters; Calculate the loss gradients corresponding to the inner and outer layers of each update candidate, and evaluate the consistency of direction to filter the effective update parameter results; The updated parameter results selected are used as candidates to assign dynamic weights. After weighted averaging, the discriminator network and the confidence evaluation model are updated to obtain the optimized two-layer optimization strategy model. The robot's skills are trained and controlled according to the optimized two-layer optimization strategy model.
2. The method according to claim 1, characterized in that, The demonstration dataset is input into the two-layer optimization strategy model, and the discriminator network and the confidence evaluation model are pre-trained using a parallel multi-path pseudo-update and weighted average algorithm, outputting initial parameters, including: The parameters of the discriminator network and the confidence evaluation model are copied to form k independent parameter copies; For each of the parameter copies, perform one pseudo-update based on the inner imitation learning objective and one pseudo-update based on the outer ranking consistency evaluation objective; The initial parameters of the pre-trained discriminator network are obtained by arithmetically averaging the discriminator network parameter copies obtained after K updates. And the initial parameters of the pre-trained confidence evaluation model are obtained by arithmetically averaging the copies of the confidence evaluation model parameters obtained after K updates.
3. The method according to claim 1, characterized in that, Calculate the loss gradients for the inner and outer layers of each update candidate, and evaluate the consistency of direction to filter valid update parameter results, including: The inner imitation learning loss function is constructed by training the demonstration data distribution weighted by the confidence evaluation model and the current data distribution generated by the policy network, wherein the inner imitation learning loss function includes a discriminator loss function and a policy network loss function: in, Let the discriminator loss function be... Let the policy network loss function be... The expected reward corresponding to the trajectory distribution generated by the expert strategy. For confidence function, The expected reward for the trajectory generated by the current strategy. The expected reward corresponding to the trajectory distribution generated by the current strategy. This is a sample of state-action pairs in the data; Calculate the loss gradient corresponding to the inner layer of each update candidate based on the inner layer imitation learning loss function; Construct an outer sorting consistency loss function based on the robot demonstration trajectory with manually labeled quality ranking: in, Let the outer sorting consistency loss function be... A function to determine whether the predicted sorting is inconsistent with the actual sorting. and These represent the predicted cumulative returns for different iteration rounds calculated based on the reward function induced by the current discriminator. For indicator functions, and The first Article and Section The quality score of manually labeled trajectories; The loss gradient corresponding to the outer layer is calculated using the outer layer order consistency loss function.
4. The method according to any one of claims 1 to 3, characterized in that, Multiple update candidates are generated in parallel based on the initial parameters, including: Based on the initial parameters, the inner imitation loss and the outer sorting loss are calculated separately through parallel independent sampling to generate multiple sets of update candidates.
5. The method according to claim 4, characterized in that, The consistency of the evaluation direction is used to filter valid updated parameter results, including: Calculate the element-wise product of the inner layer loss gradient and the outer layer loss gradient for each group of update candidates, and set the non-positive elements in the product result to zero to obtain a non-negative consistency metric vector: in, For consistency measure vectors, For discriminator, The inner layer loss gradient, The outer layer loss gradient, This is element-wise multiplication; Valid update parameter results are selected based on the consistency metric vector.
6. The method according to claim 5, characterized in that, The discriminator network and the confidence evaluation model are updated using the selected updated parameter results as candidates, dynamically weighted, and then after weighted averaging, including: The updated parameter results selected are used as candidates to assign dynamic weights, and the L1 norm of the consistency metric vector corresponding to each group of updated candidates is calculated. Based on the L1 norm of all the update candidates, the weight coefficients of the update candidates are obtained by normalization using the softmax function: in, These are the weighting coefficients. For temperature coefficient, For the first Group consistency metric vector, where K is the total number of the updated parameter results; The discriminator network and the confidence evaluation model are updated respectively by weighting the weight coefficients of the updated candidates.