Quadruped robot robust adaptive multi-skill learning method based on key frame guidance

Abstract

The invention discloses a robust self-adaptive multi-skill learning method for a quadruped robot based on key frame guidance. The robust self-adaptive multi-skill learning method comprises the following steps: determining a target key frame set of multi-skill learning; in a flat terrain environment, probability sampling is carried out on the key frames according to skill task values to generate skill instructions, the quadruped robot is controlled to execute actions, and training tracks are collected and stored in a strategy learning experience playback pool and a self-simulation learning experience playback pool respectively. Training the strategy network and the value function network through strategy learning circulation; a self-mimicking learning discriminator network is trained through a self-mimicking learning loop to extract a high quality trajectory. And after the flat terrain training is completed, entering a complex terrain environment, and carrying out adaptive training under the guidance of the trained strategy network and the self-simulation learning discriminator network so as to realize the migration of the skill primitives to the complex environment. According to the method, multi-skill learning and terrain adaptive capacity can be effectively considered, and the robustness, flexibility and multi-task execution performance of the quadruped robot are improved.

Description

Technical Field

[0001] This invention belongs to the field of robot control and intelligent learning technology, specifically relating to a robust adaptive multi-skill learning method for quadruped robots based on keyframe guidance. Background Technology

[0002] With the rapid development of deep reinforcement learning technology, quadruped robots have made significant progress in motion control and environmental adaptation. Existing research shows that quadruped robots can achieve autonomous motion control in unstructured terrain environments to a certain extent, relying only on proprioceptive information (such as joint angles, body posture, and velocity), for example, stably traversing stairs, obstacles, and uneven ground. These deep reinforcement learning-based methods have played a crucial role in improving the terrain adaptability of quadruped robots, laying the foundation for their application in real-world environments.

[0003] However, most existing technologies focus on learning a single movement pattern or a single task objective, typically training corresponding movement strategies only for specific terrain scenarios, making it difficult to simultaneously accommodate behavioral differences under various task requirements. In practical applications, quadruped robots often need to flexibly switch between multiple skill behaviors depending on the task objective. For example, in wildlife photography tasks, a low-profile crawling posture is needed to improve concealment, while in equipment inspection or environmental perception tasks, a standing or raised posture is needed to expand the field of vision. This requirement for both "skill diversity" and "terrain adaptability" is difficult for existing single-skill or weak multi-skill control strategies to effectively meet, thus limiting the application value of quadruped robots in complex multi-task scenarios.

[0004] To address the multi-skill learning problem, existing technologies mainly employ two approaches: one is reinforcement learning based on reward engineering. This approach involves manually designing sophisticated reward functions to guide robots in learning multiple skill behaviors within the same policy framework. For example, CN119849301A discloses a quadruped robot motion control method and system based on reinforcement learning. By constructing a quadruped robot simulation model, its motion behavior can be simulated and analyzed in a virtual environment, significantly reducing R&D costs and time. It transforms the balance control problem into a Markov decision process, defining states, actions, and reward functions, which helps to achieve a more intelligent and adaptive balance strategy. By using a deterministic policy gradient algorithm to interact with the simulation model for training, an efficient balance control strategy can be obtained, adjusting the robot's behavior in real time to maintain balance. Utilizing a hierarchical control approach, the walking task is decomposed into two sub-tasks: foot selection and gait generation, which are implemented separately through reinforcement learning, improving the robot's walking flexibility and stability on irregular terrain. The other approach is a data-driven method based on imitation learning, such as Adversarial Motion Prior (AMP). This approach introduces expert demonstration datasets to achieve the imitation and switching of multiple skill actions. The above methods can achieve stable execution of multiple skills in flat terrain or controlled environments, which reduces the training difficulty of switching between multiple skills to a certain extent.

[0005] However, when these multi-skill learning methods are extended to complex terrain environments, they all reveal significant limitations. For reward engineering-based methods, simultaneously addressing both "skill acquisition" and "terrain adaptation" in unstructured complex terrain often requires designing highly complex and coupled reward functions. This not only increases the difficulty of engineering implementation but also easily introduces reward conflicts, leading to training instability or even policy degradation. While imitation learning-based methods can reduce reward design costs to some extent, they are highly dependent on expert datasets. Most existing publicly available datasets are collected under flat terrain conditions, lacking effective coverage of motion patterns in complex and rough terrain. Directly applying expert data containing only flat terrain information to training in complex terrain can easily lead to overfitting due to insufficient modeling of terrain-specific dynamic features, resulting in unstable motion or even failure in complex terrain.

[0006] In recent years, some studies have begun to explore technical solutions that combine multi-skill control with terrain traversal capabilities by introducing skill-conditional control or skill scheduling mechanisms. Although these studies have improved the flexibility of the models to some extent, they still generally rely heavily on complex reward functions or fixed skill representations. Their model structure has limited scalability and generalization ability, making it difficult to efficiently adapt to richer skill types and more complex terrain environments. They have not yet fundamentally resolved the contradiction between multi-skill learning and terrain adaptation.

[0007] In summary, how to achieve both "terrain adaptability" and "skill diversity" of robots in complex and challenging environments through motion imitation technology remains a key issue that urgently needs to be addressed in the field of quadruped robot motion control. Summary of the Invention

[0008] The purpose of this invention is to overcome the shortcomings of the existing technology and provide a robust adaptive multi-skill learning method for quadruped robots based on keyframe guidance.

[0009] The objective of this invention can be achieved through the following technical solutions: This invention provides a robust adaptive multi-skill learning method for quadruped robots based on keyframe guidance, comprising the following steps: Determine the target keyframe set for multi-skill learning, and initialize network parameters, policy learning experience replay pool, and self-imitation learning experience replay pool; In flat terrain environments, skill instructions are generated by probabilistic sampling based on the target keyframe set and the skill task value. Based on the generated skill instructions, the quadruped robot is controlled to interact with the flat terrain environment, training trajectories are collected, and stored in the strategy learning experience replay pool and the self-imitation learning experience replay pool respectively. The policy learning loop is executed based on the policy learning experience replay pool to train the policy network and value function network of the quadruped robot, and various skills are learned in a single policy; the value function network includes a task evaluator network and an imitation evaluator network. The self-imitation learning discriminator learning loop is executed based on the self-imitation learning experience replay pool to train the self-imitation learning discriminator network; After completing the preset training in a flat terrain environment, the training phase begins in a complex terrain environment. In the complex terrain environment, the quadruped robot undergoes adaptive training based on the trained policy network and the self-imitation learning discriminator network.

[0010] Furthermore, the target keyframe set for multi-skill learning includes keyframes corresponding to each skill, with each keyframe representing a decisive posture or action state for a specific skill. The skills include crawling, bipedal standing, quadrupedal walking, stilt walking, walking on a base with a forward lean, and walking on a base with a backward lean. The keyframes include the quadruped robot's joint angles, posture information, and motion duration, represented as: in, Indicates the first i One keyframe; Indicates the first iRobot joint angles at each keyframe; Indicates the first i The attitude information at each keyframe includes the position, rotation angle, linear velocity, angular velocity, and base height of the robot base. Indicates the first i The duration of the action represented by each keyframe.

[0011] Furthermore, the set of target keyframes is selected according to the following criteria: Each keyframe for a skill can fully represent the core action features of that skill. For each skill, select a keyframe to represent the start, execution, and end of the skill's process.

[0012] Furthermore, the initialization of network parameters, the policy learning experience replay pool, and the self-imitation learning experience replay pool specifically includes: Initialize network parameters, including: randomly initializing the parameters of the policy network and value function network of the quadruped robot; Replay pool of strategy learning experiences With self-imitation learning experience replay pool Set to empty to complete initialization.

[0013] Furthermore, the step of generating skill instructions based on the target keyframe set and probability sampling according to the skill task value specifically includes: At the beginning of each training round, a reference state is associated with each skill, and the reference state is set as the keyframe corresponding to the skill. Based on the reference state, the task evaluator is used to evaluate the task value of each skill's corresponding reference state to obtain the task value of each skill. Using a task evaluator network based on a value function network, the task value of each skill's corresponding reference state is evaluated, and the sampling probability of each skill is calculated. in, Indicates the total number of skills; Indicates the index of the skill being sampled; This indicates that the first training round is selected. i The sampling probability of each skill; Indicates the first i Reference status corresponding to each skill; This represents the value function of the task evaluator network; Based on the calculated probabilities of each skill, the skills are randomly sampled, and the sampled skills are encoded into skill instructions to control the behavior of the quadruped robot in the current training round.

[0014] Furthermore, the step of controlling the quadruped robot to interact with the flat terrain environment based on the generated skill instructions, collecting training trajectories, and storing them in the policy learning experience replay pool and the self-imitation learning experience replay pool respectively, specifically includes: Based on the generated skill instructions, the quadruped robot is controlled to perform corresponding skill actions in a flat terrain environment. In each training round, a reinforcement learning basic tuple consisting of state, action, and task reward is collected to form a policy trajectory. ;in, This describes the state of a quadruped robot in a flat terrain environment; This represents the action output by the policy network in a given state; This indicates the task reward related to the current skill task; During trajectory acquisition, a coverage check is performed on the skill sampling results. If a skill is detected as not being sampled within a preset time window, the sampling opportunity of the skill with the most sampling times is replaced. Based on the acquired strategy trajectories, the strategy trajectories are... Store in the strategy learning experience replay pool This is used for subsequent updates of the policy network and value function network; At the end of each training round, the quality of the policy trajectory is evaluated. The quality evaluation is based on a combination of task reward and trajectory similarity, wherein the trajectory similarity is calculated by using a dynamic time warping method to determine the distance between the policy trajectory and the reference trajectory. Based on the similarity between task rewards and trajectories, the comprehensive evaluation value of the strategy trajectory is calculated using the following formula: in, Represents the strategy trajectory The overall evaluation value; This represents a combined reward, used to measure the overall task completion performance of a quadruped robot during the current skill execution process; This represents the policy trajectory generated by the current policy; This represents a reference trajectory corresponding to the skill, where the reference trajectory is the expected motion trajectory constructed from the keyframes corresponding to the skill. Represents the trajectory representation function that performs feature mapping on the policy trajectory; This represents the distance between trajectories calculated based on the dynamic time warping method; This represents the expectation operation; For each skill, a corresponding threshold is maintained in the self-imitation learning experience replay pool. When the comprehensive evaluation value of the skill corresponding to the strategy trajectory is greater than the threshold corresponding to the skill, the strategy trajectory is stored in the self-imitation learning experience replay pool. Otherwise, the strategy trajectory is discarded; the threshold corresponding to the skill is updated based on the highest comprehensive evaluation value of the corresponding skill in the self-imitation learning experience replay pool.

[0015] Furthermore, the policy learning loop, based on the policy learning experience replay pool, is used to train the policy network and value function network of the quadruped robot, learning various skills in a single policy, specifically including: In each policy learning cycle, from the policy learning experience replay pool Medium-sampled trajectory dataset ; Trajectory dataset obtained by sampling Policy network for quadruped robots Update the formula as follows: in, The policy network is represented in the first... The parameters are updated after the next policy learning cycle; Indicates the first The number of policy trajectories used in each policy learning loop; Indicates the time length of each trajectory; Indicates the state Below, policy network Output Action The probability of; For the current strategy The advantage function is used to measure the action. Relative to state The advantages and disadvantages; The policy network is represented in the first... Output action during the secondary strategy learning loop The probability of; Defined as: in, The pruning factor for strategy updates; Using the same trajectory dataset For value function networks Update the formula as follows: in, The value function network represents the first... The parameters are updated after the next policy learning cycle; The parameters represent the value function network; The value function network represents the state. Value estimation of the output; Represents the policy trajectory from time step The initial cumulative reward, the cumulative reward formula is: in, The cumulative reward for the quadruped robot at the current time step; For self-imitation learning rewards, it is represented as: in, This represents the discriminative output of the self-imitation learning discriminator network for input feature pairs; Represents a function that performs feature mapping on a state; These represent the robot observations at the previous time step and the current time step, respectively. To terminate the punishment, it is indicated as follows: in, A positive penalty constant is given when the robot meets the failure termination condition, and the failure termination condition includes: The regularized reward is represented as: in, , Preset weighting coefficients; The policy network represents time. t The output action; This indicates the change between the current action and the previous action; The residual reward is represented as: in, , These are the time steps for the skills learning phase and the terrain fine-tuning phase, respectively. Reference actions generated under the same observation conditions during the skill learning phase; After each policy learning cycle, the update steps for the policy network and value function network are repeated until the preset number of training rounds or policy convergence conditions are met.

[0016] Furthermore, the step of executing a self-imitation learning discriminator learning loop based on the self-imitation learning experience replay pool to train the self-imitation learning discriminator network specifically includes: In each self-imitation learning discriminator learning loop, the self-imitation learning experience replay pool is used. High-quality strategy trajectory sampling; Using the high-quality policy trajectories obtained from sampling and the state transition data generated by the current policy, the self-imitation learning discriminator network is improved. Update the formula as follows: in, Represents the parameters of the self-imitation learning discriminator network; Represents a function that performs feature mapping on a state; These represent the robot observations at the previous time step and the current time step, respectively. Indicates the current strategy Induced state transition distribution; This represents the discriminative output of the self-imitation learning discriminator network on the input feature pairs, used to distinguish between high-quality trajectories and policy-generated trajectories; This is the expected operation.

[0017] Furthermore, the adaptive training of the quadruped robot in complex terrain environments, based on a trained policy network and a self-imitation learning discriminator network, specifically includes: Perform adaptive training in complex terrain environments, including slopes, crossbars, discrete support points, and stairs; During adaptive training, the parameters of the self-imitation learning discriminator network are frozen. The trained self-imitation learning discriminator network is used to discriminate the policy trajectory without updating its parameters, and the policy network is adaptively trained. At the beginning of each adaptive training round, a complex terrain environment is initialized and the terrain type is randomly selected. Skill instructions for the current training round are generated by probability sampling according to the skill task value, and the joint positions of the quadruped robot are initialized according to the keyframes corresponding to the skills. Based on the current skill instructions and policy network control, the quadruped robot performs actions in complex terrain, and during the execution, it collects reinforcement learning tuples consisting of state, action and reward to form the policy trajectory of the current training round. During trajectory acquisition, the task reward of the quadruped robot at each time step is calculated. The task reward includes termination penalty, regularization reward and residual reward. Based on the task reward, a value function network is constructed to estimate and calculate the advantage function; the advantage function is used to adaptively update the policy network, and the policy network parameters are optimized through the policy gradient method to complete the adaptive training of the quadruped robot.

[0018] Furthermore, the advantage function is constructed using a multi-evaluator architecture, and is expressed as follows: in, The dominant function; i Indicates the evaluator index; Indicates the first iThe advantage functions corresponding to each evaluator, with the first evaluator being the task evaluator network. The first evaluator is used to estimate the cumulative reward of the task; the second evaluator is a mimicry evaluator network. Used to estimate rewards for self-imitation learning Cumulative returns; , They represent the first i The mean and standard deviation of the advantage function calculated by each evaluator in the current policy learning batch; For the first i The combined weights corresponding to each evaluator; The combined weights Step by step with training time t Dynamic adjustment, the formula is: in, These are preset hyperparameters; , These are the time steps for the skills learning phase and the terrain fine-tuning phase, respectively.

[0019] Compared with the prior art, the present invention has the following advantages: (1) Compared with existing quadruped robot control methods that train only for a single terrain or a single motion pattern, existing technologies are difficult to meet the requirements of terrain adaptability and skill diversity in complex application scenarios, resulting in frequent retraining or strategy failure of the robot under different task objectives. This invention constructs a multi-skill learning framework based on keyframe guidance, using the keyframes corresponding to each skill as the core representation of the skill, and conditionally controlling the policy network through skill instructions during the training process, so that the same policy network can stably learn and execute multiple skill behaviors, thereby realizing flexible skill switching in complex task scenarios and effectively improving the applicability and practical value of quadruped robots under multi-task conditions.

[0020] (2) To address the problems of existing multi-skill reinforcement learning methods relying on highly complex reward engineering in complex terrains, resulting in difficult reward design and potential training instability, this invention introduces a keyframe-guided self-imitation learning mechanism. By automatically selecting high-quality policy trajectories in flat terrain environments and constructing a self-imitation learning experience replay pool, it provides stable skill primitive constraints for policy learning, significantly reducing the dependence on complex artificial reward functions. This technical solution can guide the policy network to form a clearly structured and reusable skill movement pattern without increasing the complexity of reward design, thereby improving the stability and convergence efficiency of the multi-skill learning process.

[0021] (3) To address the problem that existing imitation learning-based methods heavily rely on expert datasets, and that most existing expert data originates from flat terrain and is difficult to generalize to complex terrain environments, this invention combines self-imitation learning with online trajectory selection. During training, high-quality policy trajectories are dynamically generated and selected as imitation objects, thus avoiding dependence on external expert datasets. Simultaneously, by constructing reference trajectories using skill keyframes and introducing a trajectory similarity evaluation mechanism, the imitation learning process can focus on the essential motion characteristics of the skill, thereby effectively improving the model's generalization ability and robustness under complex terrain conditions.

[0022] (4) To address the problem of inconsistent learning progress among different skills and the tendency for some skills to be overlooked during multi-skill training, leading to learning imbalance, this invention proposes a probabilistic sampling and coverage constraint mechanism based on the value of skill tasks. This mechanism uses a task evaluator to assess the value of each skill's reference state and performs probabilistic sampling of skills based on the evaluation results. Simultaneously, a coverage check is introduced to prevent skills from remaining unsampled for extended periods. This technical feature enables adaptive reinforcement sampling of skills that have not yet been mastered, while maintaining the stability of already acquired skills, thereby achieving dynamic balance in the multi-skill learning process and improving overall training efficiency.

[0023] (5) To address the problem that a single value function is prone to large estimation variance and unstable policy gradient when facing heterogeneous reward signals, this invention adopts a multi-evaluator (MuC) value function architecture. Task rewards and self-imitation learning rewards are estimated by independent evaluator networks, and the advantage functions of each evaluator are standardized and then weighted and fused during the policy update phase. This architecture effectively reduces mutual interference between different reward signals, improves the accuracy of value estimation, enhances the stability of the policy update process, and provides a reliable guarantee for collaborative training of multi-skill learning and terrain adaptation.

[0024] (6) To address the issue that self-imitation reward residuals may interfere with terrain adaptation learning during the adaptive training phase in complex terrain, this invention freezes the self-imitation learning discriminator network and introduces a dynamic weighting mechanism for the advantage function combination that adjusts with time steps. This allows the training focus to gradually shift from skill imitation to terrain adaptation as the training progresses, thereby enhancing the robot's adaptability to the dynamic characteristics of complex terrain while maintaining the existing skill structure. This technical solution effectively avoids skill degradation and policy drift, improving the robot's motion stability and robustness in complex terrain.

[0025] (7) This invention introduces a residual reward term during the complex terrain training phase to constrain the consistency between the policy output and the actions during the skill learning phase. This prevents the policy from generating unnecessary and drastic deviations without inhibiting terrain adaptability, thereby improving the deployability and safety of the policy in real robot systems. This design helps achieve a smooth transition from flat to complex terrain and improves the stable performance of the training policy on actual hardware platforms. Attached Figure Description

[0026] Figure 1 This is a flowchart of a robust adaptive multi-skill learning method for quadruped robots according to an embodiment of the present invention. Figure 2 This is a schematic diagram of the optimal trajectory estimator method according to an embodiment of the present invention. Detailed Implementation

[0027] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0028] Example 1: This invention addresses the core issue of the scarcity of high-quality datasets in complex terrains. The industry urgently needs a compact and universal method for representing skills—one that describes core skill features with minimal resources, thereby reducing the strong constraints of complete expert datasets on motion patterns. In existing technologies, the concept of "keyframes" in character animation offers a solution: in animation production, animators only need to define a few key pose frames, and the system can automatically interpolate and generate inter-frame transitions, eliminating the need to manually create each frame, significantly reducing the complexity and resource consumption of animation production. This approach can be further extended to the field of robotics: when humans learn new skills, they also do not need to imitate complete action sequences, but rather extract a few decisive action frames from demonstrations (such as videos, text instructions, or instructor demonstrations), repeatedly imitate and optimize successful experiences, and ultimately generalize to new scenarios.

[0029] This invention aims to provide a robust adaptive multi-skill learning method for quadruped robots based on keyframes, addressing the problems of existing technologies that rely on large expert datasets and struggle to balance skill and terrain adaptability. This method enables quadruped robots to acquire and smoothly switch between multiple skills in complex terrains. The objective of this invention can be achieved through the following technical solutions: A robust adaptive multi-skill learning method for quadruped robots based on keyframe guidance, such as Figure 1 As shown, it includes the following steps: Step S1: Determine the target keyframe set for multi-skill learning, and initialize the network parameters, policy learning experience replay pool, and self-imitation learning experience replay pool; wherein, the target keyframe set for multi-skill learning includes keyframes corresponding to each skill, each keyframe representing a decisive posture or action state of a specific skill, skills including crawling, bipedal standing, quadrupedal walking, stilt walking, forward-leaning walking, and backward-leaning walking; Keyframes include the quadruped robot's joint angles, pose information, motion velocity, acceleration, and motion duration, represented as: ,in, Indicates the first i One keyframe; Indicates the first i Robot joint angles at each keyframe; Indicates the first i The pose information for each keyframe includes the position, rotation angle, linear velocity, angular velocity, and height of the robot base. Indicates the first i Each keyframe represents the duration of an action. The target keyframe set is selected based on the following criteria: Each keyframe for a skill can fully represent the core action features of that skill. For each skill, select a keyframe to represent the start, execution, and end of the skill's process.

[0030] Initialization parameters are: policy parameters Parameters of the self-imitation discriminator Experience replay pool Self-imitation learning experience replay pool The maximum judgment value is 0.

[0031] Step S2: In a flat terrain environment, based on the target keyframe set, probability sampling is performed according to the skill task value to generate skill instructions, specifically including: Sampling skill commands (based on skill initialization technology, probabilistic sampling calculated according to skill task value, encoded as one-hot vectors) (Integrated observation) One of the core challenges of multi-skill learning is exploration bias, where policies tend to overfit simpler sub-skills while neglecting those more difficult to learn. To counteract this bias, this invention introduces a skill initialization technique that probabilistically prioritizes undertrained skills. This technique utilizes a multi-evaluator (MuC) architecture, employing a task evaluator to assess the performance of each skill and dynamically adjusting its sampling probability based on the skill's task value. This achieves balanced training across all skills within the policy network, mitigating catastrophic forgetting and promoting comprehensive skill development.

[0032] Specifically, at the beginning of each training round, for each skill i Associate a reference state The task value corresponding to the reference state is calculated using a task evaluator. In practical applications, the reference state will be... Set as the keyframe corresponding to this skill keyframe This corresponds to the state associated with this skill. The skill is selected at the beginning of the turn. i The probability formula is as follows: in, Indicates the total number of skills; Indicates the index of the skill being sampled; This indicates that the first training round is selected. i The sampling probability of each skill; Indicates the first i Reference status corresponding to each skill; This represents the value function of the task evaluator network; Based on the calculated probabilities of each skill, skills are randomly sampled, and these sampled skills are encoded as skill instructions to control the quadruped robot's behavior in the current training round. To ensure that all skills are represented during training, a coverage check is performed: if a skill has not been sampled recently, the sampling opportunity of the most frequently sampled skill is replaced to maintain a balanced skill distribution. Thanks to this selector-based mechanism, the policy can naturally learn smooth transitions between different skills.

[0033] Step S3: Based on the generated skill instructions, control the quadruped robot to interact with the flat terrain environment, collect training trajectories, and store them in the policy learning experience replay pool and the self-imitation learning experience replay pool respectively, such as... Figure 2 As shown, it specifically includes: Identifying which past trajectories are of high quality is crucial for effective learning. Intuitively, this translates to obtaining higher task rewards. The trajectory is more likely to align with the desired action. However, relying solely on carefully tuned rewards may not always generate the most ideal trajectory, as the robot may exhibit strange behavior in an attempt to maximize the reward and get trapped in local optima. Therefore, utilizing... The norm's Dynamic Time Warping (DTW) value is used to compare the policy trajectory and target pose, matching them temporally and calculating the distance between trajectories. The expected value is then used. As an evaluation indicator, among them This represents the trajectory sampled from the policy distribution. This represents the robot's reference trajectory.

[0034] Finally, combining task characteristics and trajectory similarity, the trajectory is calculated. Evaluation value : This is used as a standard for evaluating high-quality trajectories. Among them, combined rewards... Used to determine whether the robot achieves the expected reward in each attempt, including termination penalties. To stabilize regularized rewards for real-world training And a new type of residual reward . , , These are the weights for the three types of rewards. At the end of each round, this value is calculated, and for each skill, a corresponding threshold is maintained in the self-imitation learning experience replay pool. When the comprehensive evaluation value of the skill corresponding to the strategy trajectory is greater than the threshold corresponding to the skill, the strategy trajectory is stored in the self-imitation learning experience replay pool. Otherwise, discard the strategy trajectory; the threshold corresponding to the skill is updated based on the highest comprehensive evaluation value of the corresponding skill in the self-imitation learning experience replay pool.

[0035] Step S4: Execute a policy learning loop based on the policy learning experience replay pool to train the policy network and value function network of the quadruped robot, learning various skills in a single policy; the value function network includes a task evaluator network and an imitation evaluator network, specifically including: In each policy learning cycle, from the policy learning experience replay pool Medium-sampled trajectory dataset ; Trajectory dataset obtained by sampling Policy network for quadruped robots Update the formula as follows: in, The policy network is represented in the first... The parameters are updated after the next policy learning cycle; Indicates the first The number of policy trajectories used in each policy learning loop; Indicates the time length of each trajectory; Indicates the state Below, policy network Output Action The probability of; For the current strategy The advantage function is used to measure the action. Relative to state The advantages and disadvantages; The policy network is represented in the first... Output action during the secondary strategy learning loop The probability of; Defined as: in, The pruning factor for strategy updates; Using the same trajectory dataset For value function networks Update the formula as follows: in, The value function network represents the first... The parameters are updated after the next policy learning cycle; The parameters represent the value function network; The value function network represents the state. Value estimation of the output; Represents the policy trajectory from time step The initial cumulative reward total; After each policy learning cycle, the update steps for the policy network and value function network are repeated until the preset number of training rounds or policy convergence conditions are met.

[0036] Step S5: Execute the self-imitation learning discriminator learning loop based on the self-imitation learning experience replay pool to train the self-imitation learning discriminator network, specifically including: In each self-imitation learning discriminator learning loop, the self-imitation learning experience replay pool is used. High-quality strategy trajectory sampling; Using the high-quality policy trajectories obtained from sampling and the state transition data generated by the current policy, the self-imitation learning discriminator network is improved. Update the formula as follows: in, Represents the parameters of the self-imitation learning discriminator network; Represents a function that performs feature mapping on a state; These represent the robot observations at the previous time step and the current time step, respectively. Indicates the current strategy Induced state transition distribution; This represents the discriminative output of the self-imitation learning discriminator network on the input feature pairs, used to distinguish between high-quality trajectories and policy-generated trajectories; This is the expected operation.

[0037] Step S6: After completing the preset training in a flat terrain environment, the training phase proceeds to a complex terrain environment. In the complex terrain environment, the quadruped robot undergoes adaptive training based on the trained policy network and self-imitation learning discriminator network. This specifically includes: In complex terrain (including slopes, crossbars, discrete support points, and stairs, with each terrain having 10 difficulty levels and superimposed amplitudes), In a fractal noise environment: Disable the self-imitation module and repeat the operation until the preset number of iterations is completed. : Initialize the environment and skills (randomly select the terrain, and initialize the robot joint positions based on the keyframes according to the sampled skill commands in step S2). The task reward for collecting trajectories and calculating the self-imitation reward is expressed as follows: in, The task reward for the quadruped robot at the current time step; For self-imitation learning rewards, it is represented as: in, This represents the discriminative output of the self-imitation learning discriminator network for input feature pairs; Represents a function that performs feature mapping on a state; These represent the robot observations at the previous time step and the current time step, respectively. To terminate the punishment, it is indicated as follows: in, A positive penalty constant is given when the robot meets the failure termination condition, and a negative reward is given when the failure termination condition is met. The failure termination condition includes: The regularized reward is represented as: in, , Preset weighting coefficients; The policy network represents time. t The output action; This indicates the change between the current action and the previous action; The residual reward is represented as: in, , These are the time steps for the skills learning phase and the terrain fine-tuning phase, respectively. Reference actions generated under the same observation conditions during the skill learning phase; This residual penalty ensures that actions during the terrain fine-tuning phase align with those during the skill learning phase, preventing adverse deviations. In the terrain fine-tuning phase, in addition to self-imitation learning rewards... In addition, all other reward items remain active to guide the robot to achieve robust terrain adaptation. The training environment context prediction module employs a conditional variational autoencoder, taking proprioceptive observations and skill commands as inputs, and outputting predicted velocities and latent vectors, optimized through reconstruction loss and KL divergence. The policy network is updated using the MuC architecture: relying solely on a single evaluator to estimate the composite value function is prone to excessive load due to heterogeneous reward signals, leading to problems such as large variance in value estimation and unstable policy gradients; furthermore, self-imitation learning of rewards... The resulting residual gradients may persist, interfering with the terrain learning process. To address this issue, this invention employs a multi-evaluator (MuC) architecture, where each evaluator... It is responsible for estimating the value of a specific subset of rewards. Specifically, the task evaluator... Used to estimate combined reward Cumulative rewards, mimicking the evaluator This is used to estimate the imitation reward. The cumulative reward; each evaluator is trained independently to focus on processing its corresponding reward signal. At each policy update, the standardized advantage function (including mean and standard deviation) for each evaluator is first calculated, then calculated according to the formula: Combine them. Among them, The value function corresponding to each evaluator network. , These are the mean and variance of each network, respectively. For combined weights.

[0038] Unlike existing technologies, this invention is based on hyperparameters, which change stepwise with training time. t The portfolio weights are dynamically adjusted using the following formula: in This is a hyperparameter. The linear weighting scheduling mechanism can gradually shift the learning focus from skill imitation to terrain adaptation as training progresses, ensuring target adaptability at different training stages.

[0039] Skill switching and deployment: The trained policy network is exported in ONNX format (size < 500kB) and deployed on the robot's onboard computer (operating frequency 50Hz). Skill commands are input via joystick to achieve smooth switching between different skills and complete multi-skill traversal tasks in complex indoor and outdoor environments.

[0040] Example 2: This invention aims to evaluate skill acquisition capabilities by extracting a trained policy network and deploying it on real Solo-8 and Unitree Go1 robot hardware platforms. The robot's executed skill actions are compared with corresponding training keyframes. Simultaneously, the robot's base height and pitch angle data are recorded in real time during skill execution and skill switching. Experimental results show that the robot's actual motion trajectory matches the designed keyframe height, confirming that the method can accurately reproduce the target motion pattern. Furthermore, the recorded base height and pitch angle data are completely consistent with the skill parameter table, verifying the accuracy of the robot's behavior under keyframe constraints. During different skill switching processes, the robot's motion trajectory remains smooth and continuous, without significant oscillations or delays, demonstrating excellent stability and controllability in dynamic switching scenarios. In summary, the embodiments of this invention demonstrate that, guided solely by keyframes, the method of this invention can efficiently achieve multi-skill learning and smooth switching between skills. Furthermore, its skill reproduction similarity is comparable to methods that rely on large-scale expert datasets, highlighting the advantages of this invention as a compact keyframe-driven platform. It can generate multi-skill demonstration samples for quadruped robots and support the efficient acquisition of complex motion task data, providing a practical foundation for subsequent technology promotion and functional expansion.

[0041] Example 3: This invention embodiment describes the process of the Solo-8 robot performing multi-skill movements, specifically including: (a) using walking skills to traverse wedge-shaped obstacles and soft ground; (b) using pitch and head-up skills to climb a 10 cm high step; (c) using crawling skills to traverse a 15 cm high stone-filled passage; (d) using crawling skills to traverse uneven grass and bicycle obstacles, while simultaneously using walking skills to descend steps and traverse smooth surfaces. All of the above tasks require the robot to lift its legs to cross protruding obstacles. Although this leg-lifting action is not explicitly coded in the keyframes, it is naturally formed during the terrain fine-tuning stage. It should be noted that the Solo-8 robot is compact (weighing only 2 kg) and has limited joint torque (only 1 Nm, approximately 1 unitree). The Go1 robot is only 1 / 30th the size of the Go1 robot and lacks a hip joint structure. These hardware characteristics make it significantly challenging to move on rough terrain. However, the method in this embodiment of the invention can still support the robot in completing the aforementioned difficult tasks. At the same time, because the Solo-8 has no joint range of motion limitation, the robot in this embodiment of the invention can achieve extremely low-profile movement, thus successfully passing through narrow gaps. In addition, its lightweight design, combined with the optimized control of the center of mass by the method of this invention, also improves the stability of movement on smooth surfaces. These results fully demonstrate the robustness of the training strategy of the method of this invention, which can support the robot to use a variety of skills to traverse complex terrain.

[0042] Example 4: To further evaluate generalization ability, this invention proposes a new bipedal standing skill that can be represented by a single keyframe. This skill requires the robot to stand upright on its hind limbs, which is more challenging due to reduced stability and the need for continuous gait adjustments to maintain balance. In experiments, this invention incorporates the keyframe of this new skill into the existing strategy that has already trained five skills. After 10,000 steps of skill learning and 30,000 steps of terrain fine-tuning training, the new skill is gradually integrated into the original framework. When the robot switches from walking to bipedal posture, it first lifts its forebody and transfers its center of gravity to its hind limbs, then uses its forelimbs to push off the ground to gain momentum. The upward momentum brings the base close to vertical. During this stage, the hind limbs continuously adjust to maintain balance, demonstrating that the method of the present invention can effectively learn more challenging skills while maintaining stable control. In addition, the embodiments of the present invention also evaluated the bipedal skill on various unstructured terrains (including gentle slopes, discrete stepping stones, sponges, stairs, sloping grass, and areas with potholes and gravel). Among them, stairs and discrete terrains are the most difficult. Relying on the robustness of the training strategy of the present invention, the robot can still safely traverse stairs with a height of up to 10 cm. The above results further confirm that the method of the present invention can achieve stable movement of the robot in complex environments.

[0043] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0044] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A robust adaptive multi-skill learning method for quadruped robots based on keyframe guidance, characterized in that, Includes the following steps: Determine the target keyframe set for multi-skill learning, and initialize network parameters, policy learning experience replay pool, and self-imitation learning experience replay pool; In flat terrain environments, skill instructions are generated by probabilistic sampling based on the set of target keyframes and the value of the skill task. Based on the generated skill instructions, the quadruped robot is controlled to interact with the flat terrain environment, training trajectories are collected, and stored in the strategy learning experience replay pool and the self-imitation learning experience replay pool respectively. The policy learning loop is executed based on the policy learning experience replay pool to train the policy network and value function network of the quadruped robot, and various skills are learned in a single policy; the value function network includes a task evaluator network and an imitation evaluator network. The self-imitation learning discriminator learning loop is executed based on the self-imitation learning experience replay pool to train the self-imitation learning discriminator network; After completing the preset training in a flat terrain environment, the robot enters the training phase in a complex terrain environment. In the complex terrain environment, the quadruped robot undergoes adaptive training based on the trained policy network and the self-imitation learning discriminator network.

2. The robust adaptive multi-skill learning method for quadruped robots based on keyframe guidance according to claim 1, characterized in that, The target keyframe set for multi-skill learning includes keyframes corresponding to each skill. Each keyframe represents a decisive posture or action state for a specific skill. The skills include crawling, bipedal standing, quadrupedal walking, stilt walking, walking on a base with a forward lean, and walking on a base with a backward lean. The keyframes include the joint angles, posture information, and motion duration of the quadruped robot, represented as: in, Indicates the first i One keyframe; Indicates the first i Robot joint angles at each keyframe; Indicates the first i The attitude information at each keyframe includes the position, rotation angle, linear velocity, angular velocity, and base height of the robot base. Indicates the first i The duration of the action represented by each keyframe.

3. The robust adaptive multi-skill learning method for quadruped robots based on keyframe guidance according to claim 2, characterized in that, The target keyframe set is selected based on the following criteria: Each keyframe for a skill can fully represent the core action features of that skill. For each skill, select a keyframe to represent the start, execution, and end of the skill's process.

4. The robust adaptive multi-skill learning method for quadruped robots based on keyframe guidance according to claim 1, characterized in that, The initialization of network parameters, the policy learning experience replay pool, and the self-imitation learning experience replay pool specifically include: Initialize network parameters, including: randomly initializing the parameters of the policy network and value function network of the quadruped robot; Replay pool of strategy learning experiences With self-imitation learning experience replay pool Set to empty to complete initialization.

5. A robust adaptive multi-skill learning method for quadruped robots based on keyframe guidance according to claim 1, characterized in that, The process of generating skill instructions based on the target keyframe set and probability sampling according to the skill task value specifically includes: At the beginning of each training round, a reference state is associated with each skill, and the reference state is set as the keyframe corresponding to the skill. Based on the reference state, the task evaluator is used to evaluate the task value of each skill's corresponding reference state to obtain the task value of each skill. Using a task evaluator network based on a value function network, the task value of each skill's corresponding reference state is evaluated, and the sampling probability of each skill is calculated. in, Indicates the total number of skills; Indicates the index of the skill being sampled; This indicates that the first training round is selected. i The sampling probability of each skill; Indicates the first i Reference status corresponding to each skill; This represents the value function of the task evaluator network; Based on the calculated probabilities of each skill, the skills are randomly sampled, and the sampled skills are encoded into skill instructions to control the behavior of the quadruped robot in the current training round.

6. A robust adaptive multi-skill learning method for quadruped robots based on keyframe guidance according to claim 1, characterized in that, The process of controlling the quadruped robot to interact with flat terrain environments based on generated skill instructions, collecting training trajectories, and storing them in the policy learning experience replay pool and the self-imitation learning experience replay pool respectively includes: Based on the generated skill instructions, the quadruped robot is controlled to perform corresponding skill actions in a flat terrain environment. In each training round, a reinforcement learning basic tuple consisting of state, action, and task reward is collected to form a policy trajectory. ;in, This describes the state of a quadruped robot in a flat terrain environment; This represents the action output by the policy network in a given state; This indicates the task reward related to the current skill task; During trajectory acquisition, a coverage check is performed on the skill sampling results. If a skill is detected as not being sampled within a preset time window, the sampling opportunity of the skill with the most sampling times is replaced. Based on the acquired strategy trajectories, the strategy trajectories are... Store in the strategy learning experience replay pool This is used for subsequent updates of the policy network and value function network; At the end of each training round, the quality of the policy trajectory is evaluated. The quality evaluation is based on a combination of task reward and trajectory similarity, wherein the trajectory similarity is calculated by using a dynamic time warping method to calculate the distance between the policy trajectory and the reference trajectory. Based on the similarity between task rewards and trajectories, the comprehensive evaluation value of the strategy trajectory is calculated using the following formula: in, Represents the strategy trajectory The overall evaluation value; This represents a combined reward, used to measure the overall task completion performance of a quadruped robot during the current skill execution process; This represents the policy trajectory generated by the current policy; This represents a reference trajectory corresponding to the skill, where the reference trajectory is the expected motion trajectory constructed from the keyframes corresponding to the skill. Represents the trajectory representation function that performs feature mapping on the policy trajectory; This represents the distance between trajectories calculated based on the dynamic time warping method; This represents the expectation operation; For each skill, a corresponding threshold is maintained in the self-imitation learning experience replay pool. When the comprehensive evaluation value of the skill corresponding to the strategy trajectory is greater than the threshold corresponding to the skill, the strategy trajectory is stored in the self-imitation learning experience replay pool. Otherwise, the strategy trajectory is discarded; the threshold corresponding to the skill is updated based on the highest comprehensive evaluation value of the corresponding skill in the self-imitation learning experience replay pool.

7. A robust adaptive multi-skill learning method for quadruped robots based on keyframe guidance according to claim 1, characterized in that, The policy learning loop, based on the policy learning experience replay pool, is used to train the policy network and value function network of the quadruped robot, learning various skills in a single policy, specifically including: In each policy learning cycle, from the policy learning experience replay pool Medium-sampled trajectory dataset ; Trajectory dataset obtained by sampling Policy network for quadruped robots Update the formula as follows: in, The policy network is represented in the first... The parameters are updated after the next policy learning cycle; Indicates the first The number of policy trajectories used in each policy learning loop; This indicates the time length of each trajectory; Indicates the state Below, policy network Output Action The probability of; For the current strategy The advantage function is used to measure the action. Relative to state The advantages and disadvantages; The policy network is represented in the first... Output action during the secondary strategy learning loop The probability of; Defined as: in, The pruning factor for strategy updates; Using the same trajectory dataset For value function networks Update the formula as follows: in, The value function network represents the first... The parameters are updated after the next policy learning cycle; The parameters represent the value function network; The value function network represents the state. Value estimation of the output; Represents the policy trajectory from time step The initial cumulative reward, the cumulative reward formula is: in, The cumulative reward for the quadruped robot at the current time step; For self-imitation learning rewards, it is represented as: in, This represents the discriminative output of the self-imitation learning discriminator network for input feature pairs; Represents a function that performs feature mapping on a state; These represent the robot observations at the previous time step and the current time step, respectively. To terminate the punishment, it is indicated as follows: in, A positive penalty constant is given when the robot meets the failure termination condition, and the failure termination condition includes: The regularized reward is represented as: in, , Preset weighting coefficients; The policy network represents time. t The output action; This indicates the change between the current action and the previous action; The residual reward is represented as: in, , These are the time steps for the skills learning phase and the terrain fine-tuning phase, respectively. Reference actions generated under the same observation conditions during the skill learning phase; After each policy learning cycle, the update steps for the policy network and value function network are repeated until the preset number of training rounds or policy convergence conditions are met.

8. A robust adaptive multi-skill learning method for quadruped robots based on keyframe guidance according to claim 1, characterized in that, The step of executing a self-imitation learning discriminator learning loop based on a self-imitation learning experience replay pool to train the self-imitation learning discriminator network specifically includes: In each self-imitation learning discriminator learning loop, the self-imitation learning experience replay pool is used. High-quality strategy trajectory sampling; Using the high-quality policy trajectories obtained from sampling and the state transition data generated by the current policy, the self-imitation learning discriminator network is improved. Update the formula as follows: in, Represents the parameters of the self-imitation learning discriminator network; Represents a function that performs feature mapping on a state; These represent the robot observations at the previous time step and the current time step, respectively. Indicates the current strategy Induced state transition distribution; This represents the discriminative output of the self-imitation learning discriminator network on the input feature pairs, used to distinguish between high-quality trajectories and policy-generated trajectories; This is the expected operation.

9. A robust adaptive multi-skill learning method for quadruped robots based on keyframe guidance according to claim 1, characterized in that, The adaptive training of the quadruped robot in complex terrain environments, based on a trained policy network and a self-imitation learning discriminator network, specifically includes: Perform adaptive training in complex terrain environments, including slopes, crossbars, discrete support points, and stairs; During adaptive training, the parameters of the self-imitation learning discriminator network are frozen. The trained self-imitation learning discriminator network is used to discriminate the policy trajectory without updating its parameters, and the policy network is adaptively trained. At the beginning of each adaptive training round, a complex terrain environment is initialized and the terrain type is randomly selected. Skill instructions for the current training round are generated by probability sampling according to the skill task value, and the joint positions of the quadruped robot are initialized according to the keyframes corresponding to the skills. Based on the current skill instructions and policy network control, the quadruped robot performs actions in complex terrain, and during the execution, it collects reinforcement learning tuples consisting of state, action and reward to form the policy trajectory of the current training round. During trajectory acquisition, the task reward of the quadruped robot at each time step is calculated. The task reward includes termination penalty, regularization reward and residual reward. Based on the task reward, a value function network is constructed to estimate and calculate the advantage function; the advantage function is used to adaptively update the policy network, and the policy network parameters are optimized through the policy gradient method to complete the adaptive training of the quadruped robot.

10. A robust adaptive multi-skill learning method for quadruped robots based on keyframe guidance according to claim 9, characterized in that, The advantage function is constructed using a multi-evaluator architecture and is expressed as follows: in, The dominant function; i Indicates the evaluator index; Indicates the first i The advantage functions corresponding to each evaluator, with the first evaluator being the task evaluator network. The first evaluator is used to estimate the cumulative reward of the task; the second evaluator is a mimicry evaluator network. Used to estimate rewards for self-imitation learning Cumulative returns; , They represent the first i The mean and standard deviation of the advantage function calculated by each evaluator in the current policy learning batch; For the first i The combined weights corresponding to each evaluator; The combined weights Step by step with training time t Dynamic adjustment, the formula is: in, These are preset hyperparameters; , These are the time steps for the skills learning phase and the terrain fine-tuning phase, respectively.

Images