A robust self-adaptive multi-skill learning method for quadruped robots based on key frame guidance

Abstract

The application discloses a kind of based on key frame guide's robust adaptive multi-skill learning method of four-legged robot, comprising: determining the target key frame set of multi-skill learning.In flat terrain environment, according to the probability sampling of key frame generates skill instruction according to skill task value, and controls four-legged robot to execute action, and collects training trajectory and is respectively stored into policy learning and self-imitation learning experience playback pool.Through policy learning cycle, policy network and value function network are trained;Through self-imitation learning cycle, self-imitation learning discriminator network is trained to extract high-quality trajectory.After completing flat terrain training, enter complex terrain environment, and carry out adaptive training under the guidance of trained policy network and self-imitation learning discriminator network, realize the migration of skill primitive to complex environment.The method can effectively consider multi-skill learning and terrain adaptability, improve the robustness, flexibility and multi-task execution performance of four-legged robot.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of robot control and intelligent learning, and particularly relates to a robust adaptive multi-skill learning method for a quadruped robot based on key frame guidance. BACKGROUND

[0002] With the rapid development of deep reinforcement learning technology, quadruped robots have made significant progress in the field of motion control and environmental adaptation. Existing research shows that quadruped robots can achieve autonomous motion control in unstructured terrain environments to a certain extent by relying only on proprioceptive information (such as joint angles, body posture, speed, etc.). For example, stable passing through stairs, obstacles, and uneven ground. This kind of deep reinforcement learning-based method plays an important role in improving the terrain adaptability of quadruped robots and lays the foundation for the application of robots in real environments.

[0003] However, most existing technologies focus on learning a single motion pattern or a single task target, and usually only train corresponding motion strategies for specific terrain scenarios, making it difficult to simultaneously consider the behavior differences under multiple task requirements. In practical applications, quadruped robots often need to flexibly switch between multiple skill behaviors according to different task targets, such as adopting a low posture to crawl to improve concealment in wildlife photography tasks, or standing or lifting the posture to expand the field of view in device inspection or environmental perception tasks. This kind of demand for coexistence of "skill diversity" and "terrain adaptability" is difficult to effectively meet by existing single-skill or weak multi-skill control strategies, thereby limiting the application value of quadruped robots in complex multi-task scenarios.

[0004] For multi-skill learning problems, the existing technology mainly adopts two technical routes: one is the reinforcement learning method based on reward engineering, which guides the robot to learn multiple skill behaviors under the same strategy framework by artificially designing a fine reward function. For example, CN119849301A discloses a four-legged robot motion control method and system based on reinforcement learning. By constructing a four-legged robot simulation model, the motion behavior of the robot can be simulated and analyzed in a virtual environment, which significantly reduces the development cost and cycle. The balance control problem is converted into a Markov decision process, and the state, action and reward function are defined, which helps to realize a more intelligent and adaptive balance strategy. The deterministic policy gradient algorithm is used to interact with the simulation model for training, and an efficient balance control strategy can be obtained. The robot behavior is adjusted in real time to maintain balance. Using the hierarchical control idea, the walking task is divided into two sub-tasks: foot point selection and gait generation, which are realized by reinforcement learning respectively, improving the flexibility and stability of the robot walking on irregular terrain. The other is a data-driven method based on imitation learning, such as Adversarial Motion Prior (AMP) technology, which realizes the imitation and switching of multi-skill actions by introducing an expert demonstration dataset. The above methods can realize stable execution of multiple skills in flat terrain or controlled environment, and to some extent reduce the training difficulty of multi-skill switching.

[0005] However, when the above multi-skill learning methods are extended to complex terrain environments, they all have obvious limitations. For the reward engineering-based method, both "skill learning" and "terrain adaptation" need to be considered in unstructured complex terrain, which often requires the design of highly complex and coupled reward functions, increasing the difficulty of engineering implementation and easily introducing reward conflicts, leading to unstable training process or even strategy degradation. For the imitation learning-based method, although it can reduce the reward design cost to some extent, this method relies heavily on expert dataset. Most of the existing public or available datasets are collected in flat terrain, lacking effective coverage of complex rough terrain motion patterns. If the expert data containing only flat terrain information is directly applied to complex terrain training, it is easy to cause overfitting due to insufficient modeling of terrain-specific dynamic characteristics, and further lead to unstable or even failure in complex terrain.

[0006] In recent years, some research has begun to try to introduce skill conditioning control or skill scheduling mechanism, exploring technical solutions that combine multi-skill control with terrain traversal ability. Although the above research has improved the flexibility of the model to some extent, it still has a high dependence on complex reward functions or fixed skill representations, and its model structure has limited scalability and generalization ability, making it difficult to efficiently adapt to more diverse skill types and more complex terrain environments, and has not fundamentally solved the contradiction between multi-skill learning and terrain adaptation.

[0007] In summary, how to realize the "terrain adaptability" and "skill diversity" of robots in complex challenging environments through motion imitation technology is still a key problem to be solved in the field of quadruped robot motion control. SUMMARY

[0008] The purpose of the present application is to overcome the defects of the prior art and provide a quadruped robot robust adaptive multi-skill learning method based on key frame guidance.

[0009] The purpose of the present application can be achieved by the following technical solutions:

[0010] The present application provides a quadruped robot robust adaptive multi-skill learning method based on key frame guidance, comprising the following steps:

[0011] Determine the target key frame set of multi-skill learning, and initialize the network parameters, policy learning experience replay pool and self-imitation learning experience replay pool;

[0012] Under the flat terrain environment, based on the target key frame set, according to the skill task value, probability sampling is performed to generate skill instructions;

[0013] Based on the generated skill instructions, control the quadruped robot to interact with the flat terrain environment, collect the training trajectory, and store it in the policy learning experience replay pool and the self-imitation learning experience replay pool, respectively;

[0014] Based on the policy learning experience replay pool, perform a policy learning cycle to train the policy network and the value function network of the quadruped robot, and learn various skills in a single policy; the value function network includes a task evaluator network and an imitation evaluator network;

[0015] Based on the self-imitation learning experience replay pool, perform a self-imitation learning discriminator learning cycle to train the self-imitation learning discriminator network;

[0016] After completing the preset training in the flat terrain environment, enter the training stage in the complex terrain environment, and based on the trained policy network and self-imitation learning discriminator network, adaptively train the quadruped robot in the complex terrain environment.

[0017] Further, the target key frame set of multi-skill learning includes key frames corresponding to each skill, each key frame representing a decisive pose or action state of a specific skill, and the skill includes crawling, biped standing, quadruped walking, stilts, base forward walking, and base backward walking.

[0018] The key frame includes the joint angle, pose information and motion duration of the quadruped robot, and is expressed as:

[0019]

[0020] in, Indicates the first i One keyframe; Indicates the first i Robot joint angles at each keyframe; Indicates the first i The attitude information at each keyframe includes the position, rotation angle, linear velocity, angular velocity, and base height of the robot base. Indicates the first i The duration of the action represented by each keyframe.

[0021] Furthermore, the set of target keyframes is selected according to the following criteria:

[0022] Each keyframe for a skill can fully represent the core action features of that skill.

[0023] For each skill, select a keyframe to represent the start, execution, and end of the skill's process.

[0024] Furthermore, the initialization of network parameters, the policy learning experience replay pool, and the self-imitation learning experience replay pool specifically includes:

[0025] Initialize network parameters, including: randomly initializing the parameters of the policy network and value function network of the quadruped robot;

[0026] Replay pool of strategy learning experiences With self-imitation learning experience replay pool Set to empty to complete initialization.

[0027] Furthermore, the step of generating skill instructions based on the target keyframe set and probability sampling according to the skill task value specifically includes:

[0028] At the beginning of each training round, a reference state is associated with each skill, and the reference state is set as the keyframe corresponding to the skill.

[0029] Based on the reference state, the task evaluator is used to evaluate the task value of each skill's corresponding reference state to obtain the task value of each skill.

[0030] Using a task evaluator network based on a value function network, the task value of each skill's corresponding reference state is evaluated, and the sampling probability of each skill is calculated.

[0031]

[0032] in, Indicates the total number of skills; Indicates the index of the skill being sampled; This indicates that the first training round is selected. i The sampling probability of each skill; Indicates the first i Reference status corresponding to each skill; This represents the value function of the task evaluator network;

[0033] Based on the calculated probabilities of each skill, the skills are randomly sampled, and the sampled skills are encoded into skill instructions to control the behavior of the quadruped robot in the current training round.

[0034] Furthermore, the step of controlling the quadruped robot to interact with the flat terrain environment based on the generated skill instructions, collecting training trajectories, and storing them in the policy learning experience replay pool and the self-imitation learning experience replay pool respectively, specifically includes:

[0035] Based on the generated skill instructions, the quadruped robot is controlled to perform corresponding skill actions in a flat terrain environment. In each training round, a reinforcement learning basic tuple consisting of state, action, and task reward is collected to form a policy trajectory. ;in, This describes the state of a quadruped robot in a flat terrain environment; This represents the action output by the policy network in a given state; This indicates the task reward related to the current skill task;

[0036] During trajectory acquisition, a coverage check is performed on the skill sampling results. If a skill is detected as not being sampled within a preset time window, the sampling opportunity of the skill with the most sampling times is replaced. Based on the acquired strategy trajectories, the strategy trajectories are... Store in the strategy learning experience replay pool This is used for subsequent updates of the policy network and value function network;

[0037] At the end of each training round, the quality of the policy trajectory is evaluated. The quality evaluation is based on a combination of task reward and trajectory similarity, wherein the trajectory similarity is calculated by using a dynamic time warping method to determine the distance between the policy trajectory and the reference trajectory.

[0038] Based on the similarity between task rewards and trajectories, the comprehensive evaluation value of the strategy trajectory is calculated using the following formula:

[0039]

[0040] in, Represents the strategy trajectory The overall evaluation value; This represents a combined reward, used to measure the overall task completion performance of a quadruped robot during the current skill execution process; This represents the policy trajectory generated by the current policy; This represents a reference trajectory corresponding to the skill, where the reference trajectory is the expected motion trajectory constructed from the keyframes corresponding to the skill. Represents the trajectory representation function that performs feature mapping on the policy trajectory; This represents the distance between trajectories calculated using the dynamic time warping method; This represents the expectation operation;

[0041] For each skill, a corresponding threshold is maintained in the self-imitation learning experience replay pool. When the comprehensive evaluation value of the skill corresponding to the strategy trajectory is greater than the threshold corresponding to the skill, the strategy trajectory is stored in the self-imitation learning experience replay pool. Otherwise, the strategy trajectory is discarded; the threshold corresponding to the skill is updated based on the highest comprehensive evaluation value of the corresponding skill in the self-imitation learning experience replay pool.

[0042] Furthermore, the policy learning loop, based on the policy learning experience replay pool, is used to train the policy network and value function network of the quadruped robot, learning various skills in a single policy, specifically including:

[0043] In each policy learning cycle, from the policy learning experience replay pool Medium-sampled trajectory dataset ;

[0044] Trajectory dataset obtained by sampling Policy network for quadruped robots Update the formula as follows:

[0045]

[0046] in, The policy network is represented in the first... The parameters are updated after the next policy learning cycle; Indicates the first The number of policy trajectories used in each policy learning loop; Indicates the time length of each trajectory; Indicates the state Below, policy network Output Action The probability of; For the current strategy The advantage function is used to measure the action. Relative to state The advantages and disadvantages; The policy network is represented in the first... Output action during the secondary strategy learning loop The probability of; Defined as:

[0047]

[0048] in, The pruning factor for strategy updates;

[0049] Using the same trajectory dataset For value function networks Update the formula as follows:

[0050]

[0051] in, The value function network represents the first... The parameters are updated after the next policy learning cycle; The parameters represent the value function network; The value function network represents the state. Value estimation of the output; Represents the policy trajectory from time step The initial cumulative reward, the cumulative reward formula is:

[0052]

[0053] in, The cumulative reward for the quadruped robot at the current time step; For self-imitation learning rewards, it is represented as:

[0054]

[0055] in, This represents the discriminative output of the self-imitation learning discriminator network for input feature pairs; This represents a function that performs feature mapping on the state; These represent the robot observations at the previous time step and the current time step, respectively. To terminate the punishment, it is indicated as follows:

[0056]

[0057] in, A positive penalty constant is given when the robot meets the failure termination condition, and the failure termination condition includes: The regularized reward is represented as:

[0058]

[0059] in, , Preset weighting coefficients; The policy network represents time. t The output action; This indicates the change between the current action and the previous action; The residual reward is represented as:

[0060]

[0061] in, , These are the time steps for the skills learning phase and the terrain fine-tuning phase, respectively. Reference actions generated under the same observation conditions during the skill learning phase;

[0062] After each policy learning cycle, the update steps for the policy network and value function network are repeated until the preset number of training rounds or policy convergence conditions are met.

[0063] Furthermore, the step of executing a self-imitation learning discriminator learning loop based on the self-imitation learning experience replay pool to train the self-imitation learning discriminator network specifically includes:

[0064] In each self-imitation learning discriminator learning loop, the self-imitation learning experience replay pool is used. High-quality strategy trajectory sampling;

[0065] Using the high-quality policy trajectories obtained from sampling and the state transition data generated by the current policy, the self-imitation learning discriminator network is improved. Update the formula as follows:

[0066]

[0067]

[0068] in, Represents the parameters of the self-imitation learning discriminator network; This represents a function that performs feature mapping on the state; These represent the robot observations at the previous time step and the current time step, respectively. Indicates the current strategy Induced state transition distribution; This represents the discriminative output of the self-imitation learning discriminator network on the input feature pairs, used to distinguish between high-quality trajectories and policy-generated trajectories; This is the expected operation.

[0069] Furthermore, the adaptive training of the quadruped robot in complex terrain environments, based on a trained policy network and a self-imitation learning discriminator network, specifically includes:

[0070] Perform adaptive training in complex terrain environments, including slopes, crossbars, discrete support points, and stairs;

[0071] During adaptive training, the parameters of the self-imitation learning discriminator network are frozen. The trained self-imitation learning discriminator network is used to discriminate the policy trajectory without updating its parameters, and the policy network is adaptively trained.

[0072] At the beginning of each adaptive training round, a complex terrain environment is initialized and the terrain type is randomly selected. Skill instructions for the current training round are generated by probability sampling according to the skill task value, and the joint positions of the quadruped robot are initialized according to the keyframes corresponding to the skills.

[0073] Based on the current skill instructions and policy network control, the quadruped robot performs actions in complex terrain, and during the execution, it collects reinforcement learning tuples consisting of state, action and reward to form the policy trajectory of the current training round.

[0074] During trajectory acquisition, the task reward of the quadruped robot at each time step is calculated. The task reward includes termination penalty, regularization reward and residual reward.

[0075] Based on the task reward, a value function network is constructed to estimate and calculate the advantage function; the advantage function is used to adaptively update the policy network, and the policy network parameters are optimized through the policy gradient method to complete the adaptive training of the quadruped robot.

[0076] Furthermore, the advantage function is constructed using a multi-evaluator architecture, and is expressed as follows:

[0077]

[0078] in, The dominant function; i Indicates the evaluator index; Indicates the first i The advantage functions corresponding to each evaluator, with the first evaluator being the task evaluator network. The first evaluator is used to estimate the cumulative reward of the task; the second evaluator is a mimicry evaluator network. Used to estimate rewards for self-imitation learning Cumulative returns; , They represent the first i The mean and standard deviation of the advantage function calculated by each evaluator in the current policy learning batch; For the first i The combined weights corresponding to each evaluator;

[0079] The combined weights Step by step with training time t Dynamic adjustment, the formula is:

[0080]

[0081]

[0082] in, These are preset hyperparameters; , These are the time steps for the skills learning phase and the terrain fine-tuning phase, respectively.

[0083] Compared with the prior art, the present invention has the following advantages:

[0084] (1) Compared with existing quadruped robot control methods that train only for a single terrain or a single motion pattern, existing technologies are difficult to meet the requirements of terrain adaptability and skill diversity in complex application scenarios, resulting in frequent retraining or strategy failure of the robot under different task objectives. This invention constructs a multi-skill learning framework based on keyframe guidance, using the keyframes corresponding to each skill as the core representation of the skill, and conditionally controlling the policy network through skill instructions during the training process, so that the same policy network can stably learn and execute multiple skill behaviors, thereby realizing flexible skill switching in complex task scenarios and effectively improving the applicability and practical value of quadruped robots under multi-task conditions.

[0085] (2) To address the problems of existing multi-skill reinforcement learning methods relying on highly complex reward engineering in complex terrains, resulting in difficult reward design and potential training instability, this invention introduces a keyframe-guided self-imitation learning mechanism. By automatically selecting high-quality policy trajectories in flat terrain environments and constructing a self-imitation learning experience replay pool, it provides stable skill primitive constraints for policy learning, significantly reducing the dependence on complex artificial reward functions. This technical solution can guide the policy network to form a clearly structured and reusable skill movement pattern without increasing the complexity of reward design, thereby improving the stability and convergence efficiency of the multi-skill learning process.

[0086] (3) To address the problem that existing imitation learning-based methods heavily rely on expert datasets, and that most existing expert data originates from flat terrain and is difficult to generalize to complex terrain environments, this invention combines self-imitation learning with online trajectory selection. During training, high-quality policy trajectories are dynamically generated and selected as imitation objects, thus avoiding dependence on external expert datasets. Simultaneously, by constructing reference trajectories using skill keyframes and introducing a trajectory similarity evaluation mechanism, the imitation learning process can focus on the essential motion characteristics of the skill, thereby effectively improving the model's generalization ability and robustness under complex terrain conditions.

[0087] (4) To address the problem of inconsistent learning progress among different skills and the tendency for some skills to be overlooked during multi-skill training, leading to learning imbalance, this invention proposes a probabilistic sampling and coverage constraint mechanism based on the value of skill tasks. This mechanism uses a task evaluator to assess the value of each skill's reference state and performs probabilistic sampling of skills based on the evaluation results. Simultaneously, a coverage check is introduced to prevent skills from remaining unsampled for extended periods. This technical feature enables adaptive reinforcement sampling of skills that have not yet been mastered, while maintaining the stability of already acquired skills, thereby achieving dynamic balance in the multi-skill learning process and improving overall training efficiency.

[0088] (5) To address the problem that a single value function is prone to large estimation variance and unstable policy gradient when facing heterogeneous reward signals, this invention adopts a multi-evaluator (MuC) value function architecture. Task rewards and self-imitation learning rewards are estimated by independent evaluator networks, and the advantage functions of each evaluator are standardized and then weighted and fused during the policy update phase. This architecture effectively reduces mutual interference between different reward signals, improves the accuracy of value estimation, enhances the stability of the policy update process, and provides a reliable guarantee for collaborative training of multi-skill learning and terrain adaptation.

[0089] (6) To address the issue that self-imitation reward residuals may interfere with terrain adaptation learning during the adaptive training phase in complex terrain, this invention freezes the self-imitation learning discriminator network and introduces a dynamic weighting mechanism for the advantage function combination that adjusts with time steps. This allows the training focus to gradually shift from skill imitation to terrain adaptation as the training progresses, thereby enhancing the robot's adaptability to the dynamic characteristics of complex terrain while maintaining the existing skill structure. This technical solution effectively avoids skill degradation and policy drift, improving the robot's motion stability and robustness in complex terrain.

[0090] (7) This invention introduces a residual reward term during the complex terrain training phase to constrain the consistency between the policy output and the actions during the skill learning phase. This prevents the policy from generating unnecessary and drastic deviations without inhibiting terrain adaptability, thereby improving the deployability and safety of the policy in real robot systems. This design helps achieve a smooth transition from flat to complex terrain and improves the stable performance of the training policy on actual hardware platforms. Attached Figure Description

[0091] Figure 1 This is a flowchart of a robust adaptive multi-skill learning method for quadruped robots according to an embodiment of the present invention.

[0092] Figure 2 This is a schematic diagram of the optimal trajectory estimator method according to an embodiment of the present invention. Detailed Implementation

[0093] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0094] Example 1:

[0095] This invention addresses the core issue of the scarcity of high-quality datasets in complex terrains. The industry urgently needs a compact and universal method for representing skills—one that describes core skill features with minimal resources, thereby reducing the strong constraints of complete expert datasets on motion patterns. In existing technologies, the concept of "keyframes" in character animation offers a solution: in animation production, animators only need to define a few key pose frames, and the system can automatically interpolate and generate inter-frame transitions, eliminating the need to manually create each frame, significantly reducing the complexity and resource consumption of animation production. This approach can be further extended to the field of robotics: when humans learn new skills, they also do not need to imitate complete action sequences, but rather extract a few decisive action frames from demonstrations (such as videos, text instructions, or instructor demonstrations), repeatedly imitate and optimize successful experiences, and ultimately generalize to new scenarios.

[0096] This invention aims to provide a robust adaptive multi-skill learning method for quadruped robots based on keyframes, addressing the problems of existing technologies that rely on large expert datasets and struggle to balance skill and terrain adaptability. This method enables quadruped robots to acquire and smoothly switch between multiple skills in complex terrains. The objective of this invention can be achieved through the following technical solutions:

[0097] A robust adaptive multi-skill learning method for quadruped robots based on keyframe guidance, such as Figure 1 As shown, it includes the following steps:

[0098] Step S1: Determine the target keyframe set for multi-skill learning, and initialize the network parameters, policy learning experience replay pool, and self-imitation learning experience replay pool; wherein, the target keyframe set for multi-skill learning includes keyframes corresponding to each skill, each keyframe representing a decisive posture or action state of a specific skill, skills including crawling, bipedal standing, quadrupedal walking, stilt walking, forward-leaning walking, and backward-leaning walking;

[0099] Keyframes include the quadruped robot's joint angles, pose information, motion velocity, acceleration, and motion duration, represented as: ,in, Indicates the first i One keyframe; Indicates the firsti Robot joint angles at each keyframe; Indicates the first i The pose information for each keyframe includes the position, rotation angle, linear velocity, angular velocity, and height of the robot base. Indicates the first i Each keyframe represents the duration of an action. The target keyframe set is selected based on the following criteria:

[0100] Each keyframe for a skill can fully represent the core action features of that skill.

[0101] For each skill, select a keyframe to represent the start, execution, and end of the skill's process.

[0102] Initialization parameters are: policy parameters Parameters of the self-imitation discriminator Experience replay pool Self-imitation learning experience replay pool The maximum judgment value is 0.

[0103] Step S2: In a flat terrain environment, based on the target keyframe set, probability sampling is performed according to the skill task value to generate skill instructions, specifically including:

[0104] Sampling skill commands (based on skill initialization technology, probabilistic sampling calculated according to skill task value, encoded as one-hot vectors) (Integrated observation)

[0105] One of the core challenges of multi-skill learning is exploration bias, where policies tend to overfit simpler sub-skills while neglecting those more difficult to learn. To counteract this bias, this invention introduces a skill initialization technique that probabilistically prioritizes undertrained skills. This technique utilizes a multi-evaluator (MuC) architecture, employing a task evaluator to assess the performance of each skill and dynamically adjusting its sampling probability based on the skill's task value. This achieves balanced training across all skills within the policy network, mitigating catastrophic forgetting and promoting comprehensive skill development.

[0106] Specifically, at the start of each training round, for each skill i Associate a reference state The task value corresponding to the reference state is calculated using a task evaluator. In practical applications, the reference state will be... Set as the keyframe corresponding to this skill keyframe This corresponds to the state associated with this skill. The skill is selected at the beginning of the turn. i The probability formula is as follows:

[0107]

[0108] in, Indicates the total number of skills; Indicates the index of the skill being sampled; This indicates that the first training round is selected. i The sampling probability of each skill; Indicates the first i Reference status corresponding to each skill; This represents the value function of the task evaluator network;

[0109] Based on the calculated probabilities of each skill, skills are randomly sampled, and these sampled skills are encoded as skill instructions to control the quadruped robot's behavior in the current training round. To ensure that all skills are represented during training, a coverage check is performed: if a skill has not been sampled recently, the sampling opportunity of the most frequently sampled skill is replaced to maintain a balanced skill distribution. Thanks to this selector-based mechanism, the policy can naturally learn smooth transitions between different skills.

[0110] Step S3: Based on the generated skill instructions, control the quadruped robot to interact with the flat terrain environment, collect training trajectories, and store them in the policy learning experience replay pool and the self-imitation learning experience replay pool respectively, such as... Figure 2 As shown, it specifically includes:

[0111] Identifying which past trajectories are of high quality is crucial for effective learning. Intuitively, this translates to obtaining higher task rewards. The trajectory is more likely to align with the desired action. However, relying solely on carefully tuned rewards may not always generate the most ideal trajectory, as the robot may exhibit strange behavior in an attempt to maximize the reward and get trapped in local optima. Therefore, utilizing... The norm's Dynamic Time Warping (DTW) value is used to compare the policy trajectory and target pose, matching them temporally and calculating the distance between trajectories. The expected value is then used. As an evaluation indicator, among them This represents the trajectory sampled from the policy distribution. This represents the robot's reference trajectory.

[0112] Finally, combining task characteristics and trajectory similarity, the trajectory is calculated. Evaluation value :

[0113]

[0114] This is used as a standard for evaluating high-quality trajectories. Among them, combined rewards... Used to determine whether the robot achieves the expected reward in each attempt, including termination penalties. To stabilize regularized rewards for real-world training And a new type of residual reward . , , These are the weights for the three types of rewards. At the end of each round, this value is calculated, and for each skill, a corresponding threshold is maintained in the self-imitation learning experience replay pool. When the comprehensive evaluation value of the skill corresponding to the strategy trajectory is greater than the threshold corresponding to the skill, the strategy trajectory is stored in the self-imitation learning experience replay pool. Otherwise, discard the strategy trajectory; the threshold corresponding to the skill is updated based on the highest comprehensive evaluation value of the corresponding skill in the self-imitation learning experience replay pool.

[0115] Step S4: Execute a policy learning loop based on the policy learning experience replay pool to train the policy network and value function network of the quadruped robot, learning various skills in a single policy; the value function network includes a task evaluator network and an imitation evaluator network, specifically including:

[0116] In each policy learning cycle, from the policy learning experience replay pool Medium-sampled trajectory dataset ;

[0117] Trajectory dataset obtained by sampling Policy network for quadruped robots Update the formula as follows:

[0118]

[0119] in, The policy network is represented in the first... The parameters are updated after the next policy learning cycle; Indicates the first The number of policy trajectories used in each policy learning loop; Indicates the time length of each trajectory; Indicates the state Below, policy network Output Action The probability of; For the current strategy The advantage function is used to measure the action. Relative to state The advantages and disadvantages; The policy network is represented in the first... Output action during the secondary strategy learning loop The probability of; Defined as:

[0120]

[0121] in, The pruning factor for strategy updates;

[0122] Using the same trajectory dataset For value function networks Update the formula as follows:

[0123]

[0124] in, The value function network represents the first... The parameters are updated after the next policy learning cycle; The parameters represent the value function network; The value function network represents the state. Value estimation of the output; Represents the policy trajectory from time step The initial cumulative reward total;

[0125] After each policy learning cycle, the update steps for the policy network and value function network are repeated until the preset number of training rounds or policy convergence conditions are met.

[0126] Step S5: Execute the self-imitation learning discriminator learning loop based on the self-imitation learning experience replay pool to train the self-imitation learning discriminator network, specifically including:

[0127] In each self-imitation learning discriminator learning loop, the self-imitation learning experience replay pool is used. High-quality strategy trajectory sampling;

[0128] Using the high-quality policy trajectories obtained from sampling and the state transition data generated by the current policy, the self-imitation learning discriminator network is improved. Update the formula as follows:

[0129]

[0130]

[0131] in, Represents the parameters of the self-imitation learning discriminator network; This represents a function that performs feature mapping on the state; These represent the robot observations at the previous time step and the current time step, respectively. Indicates the current strategy Induced state transition distribution; This represents the discriminative output of the self-imitation learning discriminator network on the input feature pairs, used to distinguish between high-quality trajectories and policy-generated trajectories; This is the expected operation.

[0132] Step S6: After completing the preset training in a flat terrain environment, the training phase proceeds to a complex terrain environment. In the complex terrain environment, the quadruped robot undergoes adaptive training based on the trained policy network and self-imitation learning discriminator network. This specifically includes:

[0133] In complex terrain (including slopes, crossbars, discrete support points, and stairs, with each terrain having 10 difficulty levels and superimposed amplitudes), In a fractal noise environment:

[0134] Disable the self-imitation module and repeat the operation until the preset number of iterations is completed. :

[0135] Initialize the environment and skills (randomly select the terrain, and initialize the robot joint positions based on the keyframes according to the sampled skill commands in step S2).

[0136] The task reward for collecting trajectories and calculating the self-imitation reward is expressed as follows:

[0137]

[0138] in, The task reward for the quadruped robot at the current time step; For self-imitation learning rewards, it is represented as:

[0139]

[0140] in, This represents the discriminative output of the self-imitation learning discriminator network for input feature pairs; This represents a function that performs feature mapping on the state; These represent the robot observations at the previous time step and the current time step, respectively. To terminate the punishment, it is indicated as follows:

[0141]

[0142] in, A positive penalty constant is given when the robot meets the failure termination condition, and a negative reward is given when the failure termination condition is met. The failure termination condition includes: The regularized reward is represented as:

[0143]

[0144] in, , Preset weighting coefficients; The policy network represents time. t The output action; This indicates the change between the current action and the previous action; The residual reward is represented as:

[0145]

[0146] in, , These are the time steps for the skills learning phase and the terrain fine-tuning phase, respectively. Reference actions generated under the same observation conditions during the skill learning phase;

[0147] This residual penalty ensures that actions during the terrain fine-tuning phase align with those during the skill learning phase, preventing adverse deviations. In the terrain fine-tuning phase, in addition to self-imitation learning rewards... In addition, all other reward items remain active to guide the robot to achieve robust terrain adaptation. The training environment context prediction module employs a conditional variational autoencoder, taking proprioceptive observations and skill commands as inputs, and outputting predicted velocities and latent vectors, optimized through reconstruction loss and KL divergence.

[0148] The policy network is updated using the MuC architecture: relying solely on a single evaluator to estimate the composite value function is prone to excessive load due to heterogeneous reward signals, leading to problems such as large variance in value estimation and unstable policy gradients; furthermore, self-imitation learning of rewards... The resulting residual gradients may persist, interfering with the terrain learning process. To address this issue, this invention employs a multi-evaluator (MuC) architecture, where each evaluator... It is responsible for estimating the value of a specific subset of rewards. Specifically, the task evaluator... Used to estimate combined reward Cumulative rewards, mimicking the evaluator This is used to estimate the imitation reward. The cumulative reward; each evaluator is trained independently to focus on processing its corresponding reward signal. At each policy update, the standardized advantage function (including mean and standard deviation) for each evaluator is first calculated, then calculated according to the formula:

[0149]

[0150] Combine them. Among them, The value function corresponding to each evaluator network. , These are the mean and variance of each network, respectively. For combined weights.

[0151] Unlike existing technologies, this invention is based on hyperparameters, which change stepwise with training time. t The portfolio weights are dynamically adjusted using the following formula:

[0152]

[0153]

[0154] in This is a hyperparameter. The linear weighting scheduling mechanism can gradually shift the learning focus from skill imitation to terrain adaptation as training progresses, ensuring target adaptability at different training stages.

[0155] Skill switching and deployment: The trained policy network is exported in ONNX format (size < 500kB) and deployed on the robot's onboard computer (operating frequency 50Hz). Skill commands are input via joystick to achieve smooth switching between different skills and complete multi-skill traversal tasks in complex indoor and outdoor environments.

[0156] Example 2:

[0157] This invention aims to evaluate skill acquisition capabilities by extracting a trained policy network and deploying it on real Solo-8 and Unitree Go1 robot hardware platforms. The robot's executed skill actions are compared with corresponding training keyframes. Simultaneously, the robot's base height and pitch angle data are recorded in real time during skill execution and skill switching. Experimental results show that the robot's actual motion trajectory matches the designed keyframe height, confirming that the method can accurately reproduce the target motion pattern. Furthermore, the recorded base height and pitch angle data are completely consistent with the skill parameter table, verifying the accuracy of the robot's behavior under keyframe constraints. During different skill switching processes, the robot's motion trajectory remains smooth and continuous, without significant oscillations or delays, demonstrating excellent stability and controllability in dynamic switching scenarios. In summary, the embodiments of this invention demonstrate that, guided solely by keyframes, the method of this invention can efficiently achieve multi-skill learning and smooth switching between skills. Furthermore, its skill reproduction similarity is comparable to methods that rely on large-scale expert datasets, highlighting the advantages of this invention as a compact keyframe-driven platform. It can generate multi-skill demonstration samples for quadruped robots and support the efficient acquisition of complex motion task data, providing a practical foundation for subsequent technology promotion and functional expansion.

[0158] Example 3:

[0159] This invention embodiment describes the process of the Solo-8 robot performing multi-skill movements, specifically including: (a) using walking skills to traverse wedge-shaped obstacles and soft ground; (b) using pitch and head-up skills to climb a 10 cm high step; (c) using crawling skills to traverse a 15 cm high stone-filled passage; (d) using crawling skills to traverse uneven grass and bicycle obstacles, while simultaneously using walking skills to descend steps and traverse smooth surfaces. All of the above tasks require the robot to lift its legs to cross protruding obstacles. Although this leg-lifting action is not explicitly coded in the keyframes, it is naturally formed during the terrain fine-tuning stage. It should be noted that the Solo-8 robot is compact (weighing only 2 kg) and has limited joint torque (only 1 Nm, approximately 1 unitree). The Go1 robot is only 1 / 30th the size of the Go1 robot and lacks a hip joint structure. These hardware characteristics make it significantly challenging to move on rough terrain. However, the method in this embodiment of the invention can still support the robot in completing the aforementioned difficult tasks. At the same time, because the Solo-8 has no joint range of motion limitation, the robot in this embodiment of the invention can achieve extremely low-profile movement, thus successfully passing through narrow gaps. In addition, its lightweight design, combined with the optimized control of the center of mass by the method of this invention, also improves the stability of movement on smooth surfaces. These results fully demonstrate the robustness of the training strategy of the method of this invention, which can support the robot to use a variety of skills to traverse complex terrain.

[0160] Example 4:

[0161] To further evaluate generalization ability, this invention proposes a new bipedal standing skill that can be represented by a single keyframe. This skill requires the robot to stand upright on its hind limbs, which is more challenging due to reduced stability and the need for continuous gait adjustments to maintain balance. In experiments, this invention incorporates the keyframe of this new skill into the existing strategy that has already trained five skills. After 10,000 steps of skill learning and 30,000 steps of terrain fine-tuning training, the new skill is gradually integrated into the original framework. When the robot switches from walking to bipedal posture, it first lifts its forebody and transfers its center of gravity to its hind limbs, then uses its forelimbs to push off the ground to gain momentum. The upward momentum brings the base close to vertical. During this stage, the hind limbs continuously adjust to maintain balance, demonstrating that the method of the present invention can effectively learn more challenging skills while maintaining stable control. In addition, the embodiments of the present invention also evaluated the bipedal skill on various unstructured terrains (including gentle slopes, discrete stepping stones, sponges, stairs, sloping grass, and areas with potholes and gravel). Among them, stairs and discrete terrains are the most difficult. Relying on the robustness of the training strategy of the present invention, the robot can still safely traverse stairs with a height of up to 10 cm. The above results further confirm that the method of the present invention can achieve stable movement of the robot in complex environments.

[0162] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0163] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A robust adaptive multi-skill learning method for quadruped robots based on keyframe guidance, characterized in that, Includes the following steps: Determine the target keyframe set for multi-skill learning, and initialize network parameters, policy learning experience replay pool, and self-imitation learning experience replay pool; In flat terrain environments, skill instructions are generated by probabilistic sampling based on the target keyframe set and the skill task value. Based on the generated skill instructions, the quadruped robot is controlled to interact with the flat terrain environment, training trajectories are collected, and stored in the strategy learning experience replay pool and the self-imitation learning experience replay pool respectively. The policy learning loop is executed based on the policy learning experience replay pool to train the policy network and value function network of the quadruped robot, and various skills are learned in a single policy; the value function network includes a task evaluator network and an imitation evaluator network. The self-imitation learning discriminator learning loop is executed based on the self-imitation learning experience replay pool to train the self-imitation learning discriminator network; After completing the preset training in a flat terrain environment, the training phase begins in a complex terrain environment. In the complex terrain environment, the quadruped robot undergoes adaptive training based on the trained policy network and the self-imitation learning discriminator network.

2. The robust adaptive multi-skill learning method for quadruped robots based on keyframe guidance according to claim 1, characterized in that, The target keyframe set for multi-skill learning includes keyframes corresponding to each skill. Each keyframe represents a decisive posture or action state for a specific skill. The skills include crawling, bipedal standing, quadrupedal walking, stilt walking, walking on a base with a forward lean, and walking on a base with a backward lean. The keyframes include the quadruped robot's joint angles, posture information, and motion duration, represented as: in, Indicates the first i One keyframe; Indicates the first i Robot joint angles at each keyframe; Indicates the first i The attitude information at each keyframe includes the position, rotation angle, linear velocity, angular velocity, and base height of the robot base. Indicates the first i The duration of the action represented by each keyframe.

3. The robust adaptive multi-skill learning method for quadruped robots based on keyframe guidance according to claim 2, characterized in that, The target keyframe set is selected based on the following criteria: Each keyframe for a skill can fully represent the core action features of that skill. For each skill, select a keyframe to represent the start, execution, and end of the skill's process.

4. The robust adaptive multi-skill learning method for quadruped robots based on keyframe guidance according to claim 1, characterized in that, The initialization of network parameters, the policy learning experience replay pool, and the self-imitation learning experience replay pool specifically include: Initialize network parameters, including: randomly initializing the parameters of the policy network and value function network of the quadruped robot; Replay pool of strategy learning experiences With self-imitation learning experience replay pool Set to empty to complete initialization.

5. A robust adaptive multi-skill learning method for quadruped robots based on keyframe guidance according to claim 1, characterized in that, The process of generating skill instructions based on the target keyframe set and probability sampling according to the skill task value specifically includes: At the beginning of each training round, a reference state is associated with each skill, and the reference state is set as the keyframe corresponding to the skill. Based on the reference state, the task evaluator is used to evaluate the task value of each skill's corresponding reference state to obtain the task value of each skill. Using a task evaluator network based on a value function network, the task value of each skill's corresponding reference state is evaluated, and the sampling probability of each skill is calculated. in, Indicates the total number of skills; Indicates the index of the skill being sampled; This indicates that the first training round is selected. i The sampling probability of each skill; Indicates the first i Reference status corresponding to each skill; This represents the value function of the task evaluator network; Based on the calculated probabilities of each skill, the skills are randomly sampled, and the sampled skills are encoded into skill instructions to control the behavior of the quadruped robot in the current training round.

6. A robust adaptive multi-skill learning method for quadruped robots based on keyframe guidance according to claim 1, characterized in that, The process of controlling the quadruped robot to interact with flat terrain environments based on generated skill instructions, collecting training trajectories, and storing them in the policy learning experience replay pool and the self-imitation learning experience replay pool respectively includes: Based on the generated skill instructions, the quadruped robot is controlled to perform corresponding skill actions in a flat terrain environment. In each training round, a reinforcement learning basic tuple consisting of state, action, and task reward is collected to form a policy trajectory. ;in, This describes the state of a quadruped robot in a flat terrain environment; This represents the action output by the policy network in a given state; This indicates the task reward related to the current skill task; During trajectory acquisition, a coverage check is performed on the skill sampling results. If a skill is detected as not being sampled within a preset time window, the sampling opportunity of the skill with the most sampling times is replaced. Based on the acquired strategy trajectories, the strategy trajectories are... Store in the strategy learning experience replay pool This is used for subsequent updates of the policy network and value function network; At the end of each training round, the quality of the policy trajectory is evaluated. The quality evaluation is based on a combination of task reward and trajectory similarity, wherein the trajectory similarity is calculated by using a dynamic time warping method to determine the distance between the policy trajectory and the reference trajectory. Based on the similarity between task rewards and trajectories, the comprehensive evaluation value of the strategy trajectory is calculated using the following formula: in, Represents the strategy trajectory The overall evaluation value; This represents a combined reward, used to measure the overall task completion performance of a quadruped robot during the current skill execution process; This represents the policy trajectory generated by the current policy; This represents a reference trajectory corresponding to the skill, where the reference trajectory is the expected motion trajectory constructed from the keyframes corresponding to the skill. Represents the trajectory representation function that performs feature mapping on the policy trajectory; This represents the distance between trajectories calculated using the dynamic time warping method; This represents the expectation operation; For each skill, a corresponding threshold is maintained in the self-imitation learning experience replay pool. When the comprehensive evaluation value of the skill corresponding to the strategy trajectory is greater than the threshold corresponding to the skill, the strategy trajectory is stored in the self-imitation learning experience replay pool. Otherwise, the strategy trajectory is discarded; the threshold corresponding to the skill is updated based on the highest comprehensive evaluation value of the corresponding skill in the self-imitation learning experience replay pool.

7. A robust adaptive multi-skill learning method for quadruped robots based on keyframe guidance according to claim 1, characterized in that, The policy learning loop, based on the policy learning experience replay pool, is used to train the policy network and value function network of the quadruped robot, learning various skills in a single policy, specifically including: In each policy learning cycle, from the policy learning experience replay pool Medium-sampled trajectory dataset ; Trajectory dataset obtained by sampling Policy network for quadruped robots Update the formula as follows: in, The policy network is represented in the first... The parameters are updated after the next policy learning cycle; Indicates the first The number of policy trajectories used in each policy learning loop; Indicates the time length of each trajectory; Indicates the state Below, policy network Output Action The probability of; For the current strategy The advantage function is used to measure the action. Relative to state The advantages and disadvantages; The policy network is represented in the first... Output action during the secondary strategy learning loop The probability of; Defined as: in, The pruning factor for strategy updates; Using the same trajectory dataset For value function networks Update the formula as follows: in, The value function network represents the first... The parameters are updated after the next policy learning cycle; The parameters represent the value function network; The value function network represents the state. Value estimation of the output; Represents the policy trajectory from time step The initial cumulative reward, the cumulative reward formula is: in, The cumulative reward for the quadruped robot at the current time step; For self-imitation learning rewards, it is represented as: in, This represents the discriminative output of the self-imitation learning discriminator network for input feature pairs; This represents a function that performs feature mapping on the state; These represent the robot observations at the previous time step and the current time step, respectively. To terminate the punishment, it is indicated as follows: in, A positive penalty constant is given when the robot meets the failure termination condition, and the failure termination condition includes: The regularized reward is represented as: in, , Preset weighting coefficients; The policy network represents time. t The output action; This indicates the change between the current action and the previous action; The residual reward is represented as: in, , These are the time steps for the skills learning phase and the terrain fine-tuning phase, respectively. Reference actions generated under the same observation conditions during the skill learning phase; After each policy learning cycle, the update steps for the policy network and value function network are repeated until the preset number of training rounds or policy convergence conditions are met.

8. A robust adaptive multi-skill learning method for quadruped robots based on keyframe guidance according to claim 1, characterized in that, The step of executing a self-imitation learning discriminator learning loop based on a self-imitation learning experience replay pool to train the self-imitation learning discriminator network specifically includes: In each self-imitation learning discriminator learning loop, the self-imitation learning experience replay pool is used. High-quality strategy trajectory sampling; Using the high-quality policy trajectories obtained from sampling and the state transition data generated by the current policy, the self-imitation learning discriminator network is improved. Update the formula as follows: in, Represents the parameters of the self-imitation learning discriminator network; This represents a function that performs feature mapping on the state; These represent the robot observations at the previous time step and the current time step, respectively. Indicates the current strategy Induced state transition distribution; This represents the discriminative output of the self-imitation learning discriminator network on the input feature pairs, used to distinguish between high-quality trajectories and policy-generated trajectories; This is the expected operation.

9. A robust adaptive multi-skill learning method for quadruped robots based on keyframe guidance according to claim 1, characterized in that, The adaptive training of the quadruped robot in complex terrain environments, based on a trained policy network and a self-imitation learning discriminator network, specifically includes: Perform adaptive training in complex terrain environments, including slopes, crossbars, discrete support points, and stairs; During adaptive training, the parameters of the self-imitation learning discriminator network are frozen. The trained self-imitation learning discriminator network is used to discriminate the policy trajectory without updating its parameters, and the policy network is adaptively trained. At the beginning of each adaptive training round, a complex terrain environment is initialized and the terrain type is randomly selected. Skill instructions for the current training round are generated by probability sampling according to the skill task value, and the joint positions of the quadruped robot are initialized according to the keyframes corresponding to the skills. Based on the current skill instructions and policy network control, the quadruped robot performs actions in complex terrain, and during the execution, it collects reinforcement learning tuples consisting of state, action and reward to form the policy trajectory of the current training round. During trajectory acquisition, the task reward of the quadruped robot at each time step is calculated. The task reward includes termination penalty, regularization reward and residual reward. Based on the task reward, a value function network is constructed to estimate and calculate the advantage function; the advantage function is used to adaptively update the policy network, and the policy network parameters are optimized through the policy gradient method to complete the adaptive training of the quadruped robot.

10. A robust adaptive multi-skill learning method for quadruped robots based on keyframe guidance according to claim 9, characterized in that, The advantage function is constructed using a multi-evaluator architecture and is expressed as follows: in, The dominant function; i Indicates the evaluator index; Indicates the first i The advantage functions corresponding to each evaluator, with the first evaluator being the task evaluator network. The first evaluator is used to estimate the cumulative reward of the task; the second evaluator is a mimicry evaluator network. Used to estimate rewards for self-imitation learning Cumulative returns; , They represent the first i The mean and standard deviation of the advantage function calculated by each evaluator in the current policy learning batch; For the first i The combined weights corresponding to each evaluator; The combined weights Step by step with training time t Dynamic adjustment, the formula is: in, These are preset hyperparameters; , These are the time steps for the skill learning phase and the terrain fine-tuning phase, respectively.

Images