A quadruped robot reinforcement learning training method based on stage adaptive reward weight and TD error priority sampling
Patent Information
- Application Number
- CN202610852278.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-06-12
- Publication Date
- 2026-09-15
Smart Images
Figure CN122755422A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a reinforcement learning training method for quadruped robots based on stage-adaptive reward weights and TD error-priority sampling, belonging to the field of quadruped robot reinforcement learning training technology. Background Technology
[0002] Current motion control methods for quadruped robots (robot dogs) are mainly divided into two categories: model-based control methods and reinforcement learning-based control methods. However, both of these methods still have certain problems.
[0003] Model-based control methods employ kinematic / dynamic modeling, combined with algorithms such as model predictive control (MPC) and virtual model control, to generate joint trajectories by solving optimization problems. These methods perform well on flat terrain, but require accurate robot dynamics models, have poor adaptability to terrain changes, and struggle to achieve complex dynamic movements (such as jumping and running).
[0004] Reinforcement learning-based control methods are widely used in the motion control of quadruped robots. A typical framework is as follows: Training environment: Training is performed using simulation environments (such as Isaac Gym, Isaac Sim, and MuJoCo), with parallel sampling accelerating learning; Algorithm foundation: The mainstream algorithm is Proximal Policy Optimization (PPO), employing an Actor-Critic architecture where the policy network outputs actions and the value network estimates state values; Reward function design: Typically composed of multiple linearly weighted sub-reward items, including velocity tracking rewards (encouraging the robot to reach the target speed), posture stability rewards (penalizing body tilting), energy consumption penalties (reducing joint torque), and joint safety penalties (limiting the joint angle range); Experience replay: Interaction samples (state, action, reward, next state) are stored in an experience pool, and randomly and uniformly sampled from it during training to update network parameters; Training process: Gradient descent is used to optimize the policy network and value network to maximize cumulative rewards. Existing reinforcement learning methods have implemented basic movements such as walking and running on robots like the Unitree Go2 and H1, and have made some progress in simulation-to-real-machine transfer (Sim2Real). Although reinforcement learning-based control methods have achieved good results, they still have the following technical limitations:
[0005] 1. The reward function has fixed weights and cannot adapt to changes during the training phase.
[0006] Existing methods use fixed weight coefficients to sum the weights of each sub-reward item, and the weights remain unchanged throughout the training process. The problem is that in the early stage of training, the robot needs to learn basic movements (such as walking forward) first. At this time, the weights of posture stability and energy consumption penalty are too high, which will hinder exploration. In the later stage of training, the robot can move and needs to finely optimize posture and energy consumption, but fixed weights cannot achieve this kind of phased goal switching.
[0007] This leads to slow training convergence, and the final strategy may get stuck in a local optimum (such as stiff movements or excessive energy consumption).
[0008] 2. The sample collection method is uniform, but the utilization rate of high-value samples is low.
[0009] Existing methods uniformly sample from an experience pool, with each experience having an equal probability of being selected. The problem is that not all samples are equally important for model updates. For example, keyframes such as the moment the robot falls, the moment it crosses an obstacle, and keyframes with sudden reward mutations contain richer information and are more valuable for improving the policy; while ordinary frames of smooth walking contain less information. Because uniform sampling causes the model to spend a lot of computational effort on mundane samples, it suffers from insufficient learning of key events, slow convergence, and poor generalization ability for rare but important situations (such as fall recovery).
[0010] 3. There is mutual interference between reward items.
[0011] Multiple reward items may conflict with each other. For example, "velocity tracking" encourages rapid movement, while "posture stability" encourages smooth movement, and it is difficult to balance the two with fixed weights. The problem is that when the weights are not set properly, extreme situations may occur, such as "staying still for stability" or "striking violently for speed." Therefore, manual parameter tuning is required repeatedly, resulting in low training efficiency and difficulty in finding the optimal weight combination.
[0012] 4. Lack of refined control over the training phase
[0013] Existing methods treat the entire training process as a single phase, failing to consider the inherent phased characteristics of reinforcement learning (exploration → exploitation → fine-tuning). The problem is that excessive pursuit of stability in the early stages inhibits exploration, while excessive pursuit of speed in the later stages disrupts the learned stable postures. This leads to oscillations in the training process, making it difficult to consistently converge to a high-performance policy. Summary of the Invention
[0014] This invention provides a reinforcement learning training method for quadruped robots based on stage-adaptive reward weights and TD error-priority sampling, aiming to solve the following problems existing in the prior art:
[0015] 1. Fixed reward function weights: The importance of each sub-reward item cannot be dynamically adjusted according to the training phase, resulting in insufficient exploration in the early stage of training and difficulty in optimization in the later stage.
[0016] 2. Low sample utilization: The uniform sampling strategy ignores the value differences between samples, the key events are not fully learned, and the convergence speed is slow.
[0017] 3. Interference between reward items: Multiple reward objectives are difficult to balance under fixed weights, requiring repeated manual parameter adjustments.
[0018] 4. Lack of phased training: Differentiated learning strategies were not designed for the exploration state, stable state, and fine-tuning state, resulting in low training efficiency.
[0019] 5. The introduction of adaptive reward weights can cause the Markov Decision Process (MDP) of the environment to become non-stationary. If the traditional TD error-first sampling is directly applied, the old samples in the experience pool will generate severely distorted TD errors because "old weights" are used. This will cause the model to overfit to outdated, high-error "wrong experiences" and trigger policy collapse.
[0020] The objective of this invention is achieved through the following technical solution:
[0021] A reinforcement learning training method for quadruped robots based on stage-adaptive reward weights and TD error-priority sampling includes the following steps:
[0022] Step 1: Construct a reinforcement learning training model based on the Actor-Critic architecture, including a policy network and a value network; wherein, the policy network is used to output actions based on the current state information of the quadruped robot; the value network is used to estimate the current state value or state-action value;
[0023] Step 2: The training process of the reinforcement learning training model in Step 1 is divided into multiple stages using a joint decision mechanism; each stage corresponds to a different reward weight configuration through a stage adaptive reward weight calculation method.
[0024] Step 3: Collect state information samples of the quadruped robot in real time, perform key event detection on the samples and attach event labels to construct training samples;
[0025] Step 4: Calculate the initial composite importance index based on the training samples constructed in Step 3 and the current stage reward weights obtained in Step 2, and then write the samples and their corresponding priorities into the experience replay pool; monitor the rate of change of the reward weight vector after each training round or after each preset number of training steps.
[0026] Step 5: Employ a phase-aware directional resampling method to construct a basic sampling probability distribution based on the importance index of frequently sampled samples; extract a batch of samples from the experience replay pool in Step 4 according to the sampling probability; calculate the loss functions of the policy network and value network based on the reinforcement learning training model in Step 1; update the parameters of the policy network and value network using gradient descent; recalculate the TD error and composite importance index of the sampled samples after the update; and update the corresponding nodes in the priority tree.
[0027] Step Six: Based on the high-priority sample distribution obtained in Step Five, establish a two-way feedback loop between reward weights and priority sampling. When the training reaches the preset convergence condition or the upper limit of the training rounds, end the training. Export the trained policy network into a deployment format, first deploy it to the simulation platform for verification, and then deploy it to a real quadruped robot for real-time motion control. Perform online performance monitoring and make incremental fine-tuning as needed.
[0028] Preferably, the joint determination mechanism in step two includes the following steps:
[0029] In each training round Internally, the monitoring metrics include: average return growth rate. TD error variance Average tracking error Number of consecutive falls and frequency of falling The training process is divided into three stages: exploratory, stable, and fine-tuning, based on monitoring metrics.
[0030] (1) When the most recent consecutive The training epochs switch from the exploratory state to the stable state when all of the following conditions are met:
[0031]
[0032] in, To explore the lower bound of the average return growth rate monitoring indicator. To explore the upper limit of the TD error variance monitoring index. To explore the upper limit of the average tracking error monitoring index, To explore the lower limit of the continuous fall-free monitoring index, To explore the upper limit of the fall frequency monitoring index;
[0033] (2) When the most recent consecutive The system switches from the steady state to the fine-tuning state when all of the following conditions are met in a training epoch:
[0034]
[0035] in, This indicates that the growth in returns is slowing down. This represents the upper limit of the steady-state threshold for the average return growth rate monitoring indicator. , , , ; This represents the lower bound of the steady-state condition for the TD error variance monitoring index. This represents the lower bound of the steady-state performance of the average tracking error monitoring index. This represents the upper limit of the steady-state value of the monitoring indicator for consecutive falls-free periods. This represents the lower limit of the steady-state condition for the fall frequency monitoring index;
[0036] (3) To avoid frequent stage jittering during switching, a minimum dwell requirement is set, that is, each stage must maintain at least Switching is only allowed after one training round;
[0037]
[0038] in, This indicates the number of consecutive training rounds required before the current phase switch;
[0039] A phase switch is executed only if the current stage dwell time meets the minimum dwell requirement and the switching conditions are met consecutively a preset number of times.
[0040] After entering a stable state or fine-tuning state, if the most recent consecutive Any of the following situations occur within a single statistics window:
[0041]
[0042] in, These are the abnormal thresholds used during fall frequency, TD error variance, and average tracking error phase backoff or delayed handover, respectively. ;
[0043] This will trigger a phase rollback or delay subsequent switching.
[0044] Preferably, the specific steps of the stage adaptive reward weight calculation method described in step two are as follows:
[0045] Regarding the first Each reward item, during training time The weights are defined as follows:
[0046]
[0047] in: For the first Each reward item at time The weights; Basic weights; To adjust the amplitude; The moment to begin adjustment; For smooth transition coefficient;
[0048] The rewards include at least: velocity tracking reward, attitude stability reward, energy consumption penalty, joint safety penalty, and smooth motion reward; in the exploration state, the weight of velocity tracking reward is increased, while the weight of attitude stability reward and energy consumption penalty is decreased; in the stable state, the weight of attitude stability reward and energy consumption penalty is gradually increased; in the fine-tuning state, the weight of joint safety penalty and smooth motion reward is increased.
[0049] Preferably, the state information in step three includes at least: body posture information, joint angles, joint angular velocities, foot contact state, body linear velocity, body angular velocity, and IMU measurement data; the policy network adjusts the current state accordingly. Output Action The quadruped robot interacts with the environment based on the aforementioned actions to obtain the next state. ;
[0050] The key event detection includes: foot slippage, support phase switching, body instability recovery, obstacle crossing transition, and sudden changes in landing error;
[0051] The training samples are constructed as follows:
[0052]
[0053] in: This is the current state; For the current action; As a comprehensive reward; The next state; This is a termination marker; For event tags; This is a stage label.
[0054] Preferably, the specific steps for calculating the initial composite importance index in step four are as follows:
[0055] The comprehensive reward formula is defined as follows:
[0056]
[0057] in, For the total number of reward items, For sub-reward items; the aforementioned Sub-reward items can be defined as:
[0058] (1) Speed tracking reward
[0059]
[0060] in: The current linear velocity of the machine body. For the target linear velocity, The current angular velocity of the machine body, For the target angular velocity, These are the normalization coefficients;
[0061] (2) Attitude stability reward
[0062]
[0063] in: For roll angle and pitch angle, The current height of the aircraft. For reference height, These are the weighting coefficients;
[0064] (3) Energy consumption penalty
[0065]
[0066] in: For the first Joint torque, For the first Joint angular velocity, For the number of joints, To control the cycle;
[0067] (4) Joint safety penalty
[0068]
[0069] in, It is the first The current angle of each joint. It is the first Safety angle threshold for each joint It is the first Safety angle threshold for each joint It is the first The safe angular velocity threshold for each joint;
[0070] (5) Smooth motion reward
[0071]
[0072] in, It is a joint exist The target location at any given moment, and the drastic change in the penalty action command;
[0073] Calculate the composite importance index for the samples. The specific steps are as follows:
[0074] First, calculate the TD error:
[0075]
[0076] in: Discount factor; For value networks; This is a termination marker;
[0077] Redefining the magnitude of reward mutation:
[0078]
[0079] Define the advantage function estimate as: ,
[0080] The composite importance index is defined as follows:
[0081]
[0082] in: Preset weighting coefficients; For stage coefficients; For event coefficients; This is the rare transfer coefficient.
[0083] Preferably, the specific steps for monitoring the rate of change of the reward weight vector in step four are as follows:
[0084] Let the current reward weight vector be:
[0085]
[0086] Its relative rate of change is defined as:
[0087]
[0088] in, It is a very small positive number.
[0089] when When this occurs, a dynamic recalibration of the experience pool is triggered. At this time, samples whose current priority ranking is within a preset proportion are extracted from the experience replay pool, and their comprehensive reward is recalculated using the new reward weight. TD error and composite importance index And update the corresponding node in the priority tree.
[0090] Preferably, the specific steps of using a stage-aware directional resampling method in step five to construct the basic sampling probability distribution based on the importance index of frequently sampled samples are as follows:
[0091] The sampling index is defined as Then the first The sampling probability of each sample is:
[0092]
[0093] in: For the first The probability that a sample will be sampled; For the first The composite importance index of each sample; the sampling index is defined as... This is used to control the degree of bias in priority sampling; This is the importance normalization term for all candidate samples in the experience replay pool;
[0094] Specifically: In the exploration state, increase the sampling weight of boundary exploration samples and failure recovery samples in the current stage; in the steady state, increase the sampling weight of posture correction samples and energy consumption optimization samples in the current stage; in the fine-tuning state, increase the sampling weight of motion smoothing samples and joint safety samples in the current stage.
[0095] Preferably, the value network loss function in step five is defined as:
[0096]
[0097] in, The sample batch size is used for each parameter update by drawing samples from the experience replay pool. For the first The squared term of the TD error corresponding to each sample;
[0098] The policy network loss function is defined as:
[0099]
[0100] in, For parameters The policy function represented by the policy network; For a moment The actions recorded in the output of the next policy network or in the samples; For a moment The state;
[0101] The gradient descent method is used to update the policy network parameters. and value network parameters .
[0102] Preferably, the specific steps for establishing a two-way feedback loop between the reward weight and the priority sampling in step six are as follows:
[0103] Feedback A: Correction of Sample Distribution and Phase Switching Timing
[0104] The phase switching is first determined based on monitoring indicators; after the basic conditions for phase switching are met, the label distribution of high-priority samples in the most recent training statistical windows is statistically analyzed to assist in correcting the timing of phase switching.
[0105] Let the first The percentage of event-type tags in high-priority samples is as follows:
[0106]
[0107] in, For the first Number of class-labeled samples;
[0108] Feedback B: Changes in reward weights and recalibration of the experience pool
[0109] When the reward weight vector When the change exceeds the threshold, the high-priority old samples in the experience pool are recalculated to ensure that the sample priority is consistent with the current stage goal.
[0110] Preferably, the online performance monitoring in step six includes: speed tracking error, posture stability, energy consumption level, joint safety, and motion smoothness;
[0111] The incremental fine-tuning involves updating the policy network with a low learning rate and simultaneously updating the value network in the background for subsequent offline retraining or online evaluation.
[0112] Compared with the prior art, the beneficial effects of the present invention are as follows:
[0113] 1. Improve training efficiency: By using stage-adaptive reward weights, the robot can focus on different learning objectives at different training stages, accelerating convergence. It quickly learns basic movements in the early stages and then refines them in later stages, reducing ineffective exploration.
[0114] 2. Improve sample utilization: The resampling mechanism based on TD error priority sampling, namely the stage-aware directional resampling method, enables high-value samples to be used more frequently, accelerates the model's learning of key events, and avoids the inefficiency caused by uniform sampling.
[0115] 3. Enhance motion stability: Strengthen joint safety penalties and smooth rewards in fine-tuning state, reduce over-excitation, protect robot hardware, and improve motion smoothness and safety after deployment.
[0116] 4. Reduced manual parameter tuning: The adaptive weight mechanism, namely the stage adaptive reward weight calculation method, can automatically adjust the priority of reward targets, reducing the trial and error cost of manually adjusting reward weights.
[0117] 5. High versatility: This method does not depend on a specific robot model or simulation environment and can be widely applied to reinforcement learning training of various quadruped robots.
[0118] This invention employs a stage-adaptive reward weight adjustment mechanism. The training process is divided into multiple stages (exploration state, stable state, and fine-tuning state) based on the number of training rounds or steps. The weight coefficients of each sub-reward item are dynamically adjusted in different stages, and the gradual change in weight is achieved through a stage-adaptive reward weight calculation formula, avoiding abrupt changes that could lead to training instability. Exploration state: Increase the weight of speed tracking rewards and decrease the weights of posture stability and energy consumption penalties to encourage the robot to actively attempt movement. Stable state: Gradually increase the weights of posture stability and energy consumption penalties to make movement smoother and more energy-efficient. Fine-tuning state: Increase the weight of joint safety penalties and add smooth movement rewards to prevent over-excitation and protect the hardware.
[0119] This invention employs a sample selection mechanism based on Time Difference Error (TD) prioritization sampling. During experience replay, instead of uniform sampling, it calculates the importance of each sample based on the absolute value of its temporal difference error (TD error), constructing a weighted sampling probability distribution. A larger TD error indicates a less accurate prediction by the model for that sample, higher learning value, and a greater probability of it being sampled. Factors such as reward variation and dominance function value can be incorporated into the overall importance index to further improve sampling quality. In this invention, the stage-adaptive reward weight adjustment mechanism and the TD error-based priority sampling sample selection mechanism work synergistically. Adaptive weights allow the robot to focus on appropriate learning objectives at each stage, while priority sampling makes the model pay more attention to key events, jointly improving training efficiency, convergence speed, and the quality of the final policy.
[0120] This invention introduces a "dynamic recalibration" mechanism within the experience pool, which addresses the problem of "old experience TD errors becoming invalid when weights change." When training skips stages (e.g., from the exploration state to the stable state), the reward weights are adjusted accordingly. When significant changes occur, the old overall rewards stored in the experience pool This approach is no longer applicable. At this point, a "dynamic recalibration" step is introduced. By setting a threshold, the top 20% of high-value samples in the priority ranking are extracted. The weighted comprehensive reward R and TD error are recalculated using the latest reward weights, and then their positions in the priority tree are updated. This completely eliminates the "expiration of experience" problem caused by adaptive weights, ensuring that the experience sampled in priority is absolutely beneficial to the current training phase.
[0121] This invention employs a bidirectional smooth buffer design to prevent sudden weight mutations from causing a sudden explosion in TD errors and resulting in drastic fluctuations in network parameters. The formula for calculating the stage-adaptive reward weights... The function's role is not only to change the reward; its derivative is strictly limited to a certain range. The time constant of the function It is strongly tied to the sampling update frequency of the experience pool. That is, it ensures that before a significant 1% change in weights occurs, the value network has already performed at least N gradient descent updates on the samples with the new weights through priority sampling. This allows the value network time to "digest" the new rules, making the change in TD error continuous and avoiding catastrophic forgetting.
[0122] This invention utilizes the "complementary effect" of a composite importance index. Since a single TD error is highly vulnerable to dynamic changes in the reward function, a more robust evaluation index is needed. The composite importance index can mitigate inaccuracies in local TD error calculations caused by stage switching or weight changes. (The magnitude of the change in reward before and after the action) and The (strategic advantage function) still plays a role. For example, at the moment a robot accidentally falls, regardless of whether the current focus is on speed or stability, the state value before and after this action will change drastically, leading to... The impact is significant; therefore, by employing composite dimensions, even when the TD error creates a "blind spot" due to weight switching, it can still reliably select high-value keyframes (such as falls and obstacle crossings) for priority learning.
[0123] This invention employs a stage-aware "directional resampling" scheduling method, ensuring that priority sampling considers not only error but also the current training stage. Based on the calculated sampling probabilities, a stage mask matrix is introduced, representing the stage-aware directional resampling method. In the exploration state (high velocity weight): samples with "high velocity variance" are prioritized in the sampling pool. In the fine-tuning state (high safety and smoothness weight): the algorithm automatically amplifies the priority of samples with "large joint torque change rates (i.e., abrupt action changes)," forcing the robot to review erroneous experiences that caused its disjointed movements. This achieves true "coordination"—the reward weights indicate the general direction, i.e., the current stage's learning content, while the priority sampling mechanism provides precise learning materials. Attached Figure Description
[0124] Figure 1 This is a flowchart of a reinforcement learning training method for a quadruped robot based on stage adaptive reward weights and TD error priority sampling, according to the present invention. Detailed Implementation
[0125] The present invention will be further described in detail below with reference to the accompanying drawings: This embodiment is implemented under the premise of the technical solution of the present invention, and detailed implementation methods are given, but the protection scope of the present invention is not limited to the following embodiments.
[0126] like Figure 1 As shown in the figure, the reinforcement learning training method for a quadruped robot based on stage adaptive reward weights and TD error priority sampling involved in this embodiment includes the following steps:
[0127] Step 1: Construct a reinforcement learning training model
[0128] Construct a reinforcement learning training model based on the Actor-Critic architecture, including a policy network and a value network.
[0129] The policy network is used to output actions based on the current state information of the quadruped robot; the value network is used to estimate the current state value or state-action value for subsequent calculation of TD error, advantage function and network update.
[0130] Step 2: Dynamic Stage Identification and Triggering
[0131] The training process of the reinforcement learning training model constructed in step 1 is divided into an exploration state, a stable state, and a fine-tuning state.
[0132] The stage identification adopts a joint judgment mechanism, rather than static switching based solely on fixed training rounds or fixed training steps.
[0133] In each training round Within this period, at least the following monitoring metrics should be calculated: 1. Average return growth rate 2. TD error variance 3. Average tracking error 4. Number of consecutive times without falling 5. Frequency of falls ,
[0134] The above indicators are joint judgment indicators and do not represent a simple sum of values.
[0135] Preferably, the following determination logic can be adopted:
[0136] (1) Exploratory state → Stable state
[0137] When the most recent consecutive The training epochs switch from the exploratory state to the stable state when all of the following conditions are met:
[0138]
[0139] in, To explore the lower bound of the average return growth rate monitoring indicator. To explore the upper limit of the TD error variance monitoring index. To explore the upper limit of the average tracking error monitoring index, To explore the lower limit of the continuous fall-free monitoring index, To explore the upper limit of the fall frequency monitoring index;
[0140] (2) Stable state → Fine-tuning state
[0141] When the most recent consecutive The system switches from the steady state to the fine-tuning state when all of the following conditions are met in a training epoch:
[0142]
[0143] in, This indicates that the growth in returns is slowing down. Average return growth rate The smooth threshold corresponding to the transition from the steady state to the fine-tuning state. This represents the upper limit of the steady-state threshold for the average return growth rate monitoring indicator. , , , ; This represents the lower bound of the steady-state condition for the TD error variance monitoring index. This represents the lower bound of the steady-state performance of the average tracking error monitoring index. This represents the upper limit of the steady-state value of the monitoring indicator for consecutive falls-free periods. This represents the lower limit of the steady-state condition for the fall frequency monitoring index;
[0144] In addition, the upper and lower limits of the threshold of the monitoring indicator (i.e. , , , (etc.) are determined through pre-experiment or historical training data statistics, and can be normalized and adjusted according to different quadruped robot platforms and task scenarios.
[0145] (3) Minimum dwell time and rollback mechanism
[0146] To avoid frequent stage jittering during switching, a minimum dwell time requirement is set, meaning each stage must maintain at least [a certain duration]. Only after a certain number of training rounds is it allowed to switch again.
[0147] Right now:
[0148]
[0149] in, This indicates the number of consecutive training rounds required before the current phase switch;
[0150] A phase switch is executed only if the current stage dwell time meets the minimum dwell requirement and the switching conditions are met consecutively a preset number of times. In this implementation, the unit of measurement for the minimum dwell time is the training round.
[0151] After entering a stable state or fine-tuning state, if the most recent consecutive Any of the following situations occur within a single statistics window:
[0152]
[0153] in, These are the abnormal thresholds used during fall frequency, TD error variance, and average tracking error phase backoff or delayed handover, respectively. ;
[0154] This triggers a phase rollback or delays subsequent transitions. That is, if persistent instability, amplified value fluctuations, or degraded tracking performance occur after a transition, the phase state machine allows a rollback from the current phase to the previous phase, or maintenance of the current phase without entering the next phase. In the aforementioned phases... , , This represents the entry threshold used when transitioning from the exploratory state to the stable state, while These are the abnormal thresholds used for phase rollback or delayed handover. The former determines whether the condition is good enough to enter the next phase, while the latter determines whether the condition has become unstable enough to warrant a rollback after entering the next phase. Entering a stable state requires that the fall rate, TD error variance, and average tracking error have all decreased to a relatively good level. However, during rollback, the condition cannot be reversed immediately after exceeding the threshold, otherwise it will cause phase jitter. Therefore, the rollback threshold should generally be more "lenient" than the entry threshold, allowing entry first and triggering a rollback only when the condition deteriorates to a worse level.
[0155] Furthermore, the upper and lower limits of the threshold are determined through pre-experiment or historical training data statistics, and can be normalized and adjusted according to different quadruped robot platforms and task scenarios.
[0156] Step 3: Calculation of Stage Adaptive Reward Weights
[0157] Regarding the first Each reward item, during training time The weights are defined as follows:
[0158]
[0159] in: For the first Each reward item at time The weights; Basic weights; To adjust the amplitude; The moment to begin adjustment; This is the smooth transition coefficient.
[0160] The above function enables continuous changes in reward weights, avoiding training oscillations caused by sudden weight changes during phase switching.
[0161] In this embodiment, the reward items include at least: 1. speed tracking reward, 2. posture stability reward, 3. energy consumption penalty, 4. joint safety penalty, and 5. smooth motion reward.
[0162] Specifically: In the exploration state, increase the weight of speed tracking reward and decrease the weight of attitude stability reward and energy consumption penalty; in the stable state, gradually increase the weight of attitude stability reward and energy consumption penalty; in the fine-tuning state, increase the weight of joint safety penalty and smooth motion reward.
[0163] Step 4: Environmental Interaction and Status Acquisition
[0164] In a simulation environment or a real quadruped robot platform, the state information of the quadruped robot is collected in real time. The state information includes at least: 1. body posture information, 2. joint angles, 3. joint angular velocities, 4. foot contact state, 5. body linear velocity, 6. body angular velocity, and 7. IMU measurement data;
[0165] The policy network depends on the current state Output Action The quadruped robot interacts with the environment based on the aforementioned actions to obtain the next state. .
[0166] Step 5: Critical Event Detection and Tag Attachment
[0167] Obtaining state transition Subsequently, key event detection was performed on the samples, and event labels were attached. .
[0168] Critical events refer to specific event labels that are detected and attached to training samples after a state transition is obtained through environmental interaction. Essentially, they are labels for discrete, detectable events occurring at a specific moment, used to identify rare but high-value critical state transitions during the quadruped robot training process in the sample construction phase. The role of critical events is to label specific high-value events, solving the problem of "which samples have event-related value."
[0169] The key events include: 1. foot slippage, 2. support phase switching, 3. recovery from instability, 4. obstacle crossing transition, and 5. sudden change in landing error.
[0170] Preferably: foot slippage can be determined based on "the foot being in contact and the tangential velocity of the foot exceeding a set threshold"; support phase switching can be determined based on changes in the contact foot set; body instability recovery can be determined based on the recovery to a stable range within a preset window after the roll angle or pitch angle exceeds the limit; obstacle crossing transition can be determined based on the combined change in the forward displacement of the body and the lifting height of the foot; sudden change in landing error can be determined based on the increase in the deviation between the actual foot landing point and the expected landing point exceeding a threshold.
[0171] By attaching event labels, subsequent priority sampling can focus more on rare but high-value key samples in quadruped robot scenarios.
[0172] Step 6: Construct samples and calculate the overall reward
[0173] Training samples are constructed based on the current state, action, sub-reward items, and next state. The samples are represented as follows:
[0174]
[0175] in: This is the current state; For the current action; As a comprehensive reward; The next state; This is a termination marker; For event tags; This is a stage label.
[0176] The comprehensive reward is defined as:
[0177]
[0178] in, For the total number of reward items, The stage adaptive reward weights are calculated in step 3.
[0179] Preferably, Sub-reward items can be defined as:
[0180] (1) Speed tracking reward
[0181]
[0182] in: The current linear velocity of the machine body. For the target linear velocity, The current angular velocity of the machine body, For the target angular velocity, These are the normalization coefficients;
[0183] (2) Attitude stability reward
[0184]
[0185] in: For roll angle and pitch angle, The current height of the aircraft. For reference height, These are the weighting coefficients;
[0186] (3) Energy consumption penalty
[0187]
[0188] in: For the first Joint torque, For the first Joint angular velocity, For the number of joints, To control the cycle;
[0189] (4) Joint safety penalty
[0190]
[0191] in, It is the first The current angle of each joint. It is the first Safety angle threshold for each joint It is the first Safety angle threshold for each joint It is the first The safe angular velocity threshold for each joint.
[0192] (5) Smooth motion reward
[0193]
[0194] in, It is a joint exist The target location at any given moment, and the drastic change in the penalty action command.
[0195] Calculation of composite importance index
[0196] Calculate the composite importance index for the samples. .
[0197] First, calculate the TD error:
[0198]
[0199] in: Discount factor; For value networks; This is a termination marker.
[0200] Redefining the magnitude of reward mutation:
[0201]
[0202] Define the advantage function estimate as: .
[0203] Therefore, the composite importance index is defined as:
[0204]
[0205] in: Preset weighting coefficients; For stage coefficients; For event coefficients; This is the rare transfer coefficient.
[0206] The rare transfer coefficient is preferably defined as follows:
[0207]
[0208] in, The category of the transfer or event to which the sample belongs. This represents the cumulative number of times this type of sample appears in the experience pool.
[0209] This can increase the priority of low-frequency, high-value samples such as slippage, obstacle crossing, and instability recovery.
[0210] Rare transition coefficients are used to increase the sampling weight of low-frequency but high-value state transitions, including samples such as foot slippage, support phase abrupt change, instability recovery, obstacle crossing transition, and landing error abrupt change.
[0211] Step 7A: Priority Experience Storage and Priority Tree Update
[0212] Sample and their corresponding importance indicators Write the experience replay pool and update the priority tree structure synchronously.
[0213] The priority tree structure is preferably a summation tree structure, which is used to support efficient sampling and node updates according to priority.
[0214] Changes in reward weights are linked to changes in sample priority: Changes in reward weights will affect the overall reward. TD error and importance indicators This, in turn, affects the sorting of samples in the priority tree.
[0215] Step 7B: Dynamic recalibration trigger monitoring and experience pool recalibration
[0216] After each training round or at a preset number of training steps, monitor the rate of change of the reward weight vector. Let the current reward weight vector be:
[0217]
[0218] Its relative rate of change is defined as:
[0219]
[0220] in, It is a very small positive number.
[0221] when At that time, dynamic recalibration of the experience pool is triggered.
[0222] At this point, samples whose current priority ranking falls within a preset range are extracted from the experience replay pool, preferably the top 20% of samples, and their comprehensive rewards are recalculated using the new reward weights. TD error and composite importance index And update the corresponding node in the priority tree.
[0223] This monitors changes in the reward weight vector, not changes in network parameters.
[0224] Step 8: Stage-aware directional resampling
[0225] The basic sampling probability distribution is constructed based on the importance index of the frequently sampled samples during the phase-aware targeted resampling process.
[0226] The sampling index is defined as Then the first The sampling probability of each sample is:
[0227]
[0228] in: For the first The probability that a sample will be sampled; For the first The composite importance index of each sample; the sampling index is defined as... This is used to control the degree of bias in priority sampling; This is the importance normalization term for all candidate samples in the experience replay pool.
[0229] Specifically: In the exploration state, increase the sampling weight of boundary exploration samples and failure recovery samples in the current stage; in the steady state, increase the sampling weight of posture correction samples and energy consumption optimization samples in the current stage; in the fine-tuning state, increase the sampling weight of motion smoothing samples and joint safety samples in the current stage, and increase the sampling priority or sampling probability weight of these samples in stage-aware directional resampling, so as to increase the probability of them being extracted from the experience replay pool, thereby increasing their extraction probability.
[0230] This makes the priority sampling depend not only on the TD error, but also on the types of samples that truly need to be learned in the current training phase.
[0231] The boundary exploration, failure recovery, posture correction, energy consumption optimization, motion smoothing, and joint safety samples are categorized based on key event labels, stage labels, sample state features, and the current training objective. Essentially, this categorization falls under the "sample category" or "sample grouping" used in stage-aware directional resampling and sample distribution-assisted judgment. Failure recovery samples strongly correspond to key events such as organism instability recovery and can be understood as sample categories derived from relevant key event labels. Boundary exploration, posture correction, energy consumption optimization, motion smoothing, and joint safety samples primarily represent sample types that require focused learning at different training stages and are not part of the key event ontology. The purpose of these samples is that after key event detection and event labeling, the training system needs to further determine, based on the current training stage, "which type of sample should be prioritized for learning in the current stage." Therefore, in the exploration state, the focus is on boundary exploration samples and failure recovery samples to enhance basic exploration and recovery capabilities; in the steady state, the focus is on attitude correction samples and energy consumption optimization samples to improve motion stability and energy efficiency; and in the fine-tuning state, the focus is on motion smoothing samples and joint safety samples to improve control smoothness and protect hardware safety. These samples serve to support stage-aware directional resampling and stage-switching auxiliary correction, addressing the question of "which type of sample should be prioritized for learning in the current stage."
[0232] Step 9: Network parameter update and priority recalculation
[0233] A batch of samples is drawn from the experience pool according to the sampling probability, the policy network loss function and the value network loss function are calculated, and the policy network parameters are updated using gradient descent. and value network parameters .
[0234] Preferably, the value network loss can be defined as:
[0235]
[0236] in, The sample batch size is used for each parameter update by drawing samples from the experience replay pool. For the first The squared term of the TD error corresponding to each sample;
[0237] The policy network loss can be defined as:
[0238]
[0239] in, For parameters The policy function represented by the policy network; For a moment The actions recorded in the output of the next policy network or in the samples; For a moment The state;
[0240] Parameters in the value network loss function These correspond to the states that have already appeared, the next state, the comprehensive reward, the termination flag, the discount factor, and the value network output, with the aim of making the value network output approach the TD target.
[0241] Parameters in the policy network loss function These correspond to the output of the aforementioned policy network, the estimated value of the advantage function, and the parameters required for policy updates. The purpose is to optimize the action output policy based on the sampled samples.
[0242] After the update is completed, the TD error and composite importance index are recalculated for the samples sampled in this round, and the corresponding nodes in the priority tree are updated for use in the next round of training.
[0243] Step 10: Two-way feedback closed-loop mechanism and convergence judgment
[0244] This invention further establishes a two-way feedback closed loop between reward weights and priority sampling.
[0245] Feedback A: Sample distribution → Phase switching timing correction
[0246] The phase switching is first determined based on the monitoring indicators in step 2. After the basic conditions for phase switching are met, the label distribution of high-priority samples in the most recent training statistical windows is statistically analyzed to assist in correcting the timing of phase switching.
[0247] Let the first The percentage of event-type tags in high-priority samples is as follows:
[0248]
[0249] in, For the first Number of class-labeled samples.
[0250] Preferably:
[0251] If the percentage of labels such as failure recovery, foot slippage, boundary exploration, and body instability recovery is still higher than the preset threshold, the process will be postponed to the next stage.
[0252] When the proportions of labels such as posture correction, energy consumption optimization, motion smoothing, and joint safety continue to increase and reach the preset threshold, it is confirmed to enter the next stage.
[0253] If, after entering a new phase, the proportion of unstable or failed recovery labels rises again for several consecutive statistical windows, then the system will revert to the previous phase or delay the subsequent switch.
[0254] The phrase "determine the stage based on the sample distribution" here is an auxiliary correction to the result of "dividing the stage based on the monitoring indicators" in step 2.
[0255] Feedback B: Reward weight change → Experience pool recalibration
[0256] When the change in the reward weight vector exceeds the threshold, the high-priority old samples in the experience pool are recalculated to ensure that the sample priority is consistent with the current stage goal, thus avoiding the expiration of experience and policy oscillation caused by the use of old weights for historical samples.
[0257] Through the two feedback chains mentioned above, the following closed loop is formed:
[0258] Improving sample quality → Prioritizing the use of high-value samples → Continuous strategy optimization → Adaptive adjustment at each stage
[0259] Training ends when the training reaches the preset convergence condition or the maximum number of training rounds.
[0260] The preset convergence condition refers to the following: the monitoring index has met the conditions for entering the fine-tuning state and has remained stable for several consecutive training rounds, and the sample distribution auxiliary judgment has not triggered the stage rollback.
[0261] Step 11: Model Deployment and Online Optimization
[0262] The trained policy network is exported to a deployment format, preferably ONNX or TorchScript, and first deployed to a simulation platform for verification, and then deployed to a real quadruped robot for real-time motion control.
[0263] During the deployment phase, continuously monitor: 1. speed tracking error, 2. attitude stability, 3. energy consumption level, 4. joint safety, and 5. motion smoothness.
[0264] Incremental fine-tuning is triggered when online performance deteriorates, new terrain disturbances occur, or key anomalous samples accumulate.
[0265] The incremental fine-tuning is preferably performed with a low learning rate update for the policy network, and the value network can be updated synchronously in the background for subsequent offline retraining or online evaluation.
[0266] Example 1
[0267] During training, a dynamic stage identification mechanism is first employed to divide the training process into three stages: exploration, stable, and fine-tuning. In each training round, the average reward growth rate, TD error variance, average tracking error, number of consecutive fall-free periods, and fall frequency are calculated and used as the joint criteria for stage switching. When, over multiple consecutive training rounds, the average reward growth rate exceeds a first threshold, the TD error variance falls below a first threshold, the average tracking error falls below a first threshold, the number of consecutive fall-free periods reaches a first set value, and the fall frequency falls below a first set value, the system switches from the exploration stage to the stable stage. When, over multiple consecutive training rounds, the reward growth slows down, the TD error variance further decreases, the average tracking error further decreases, the number of consecutive fall-free periods reaches a second set value, and the fall frequency falls below a second set value, the system switches from the stable stage to the fine-tuning stage. To prevent frequent stage jittering, a minimum dwell time is set for each stage. If significant instability occurs again over multiple consecutive training rounds after entering the stable or fine-tuning stage, the stage state machine is allowed to revert to the previous stage or delay entering the next stage.
[0268] For different training stages, this embodiment employs a stage-adaptive reward weight adjustment mechanism for each sub-reward item. For the first... Each reward item, at any time The weights consist of a base weight, adjustment range, initial adjustment point, and smooth transition coefficient, and are continuously varied through a hyperbolic tangent function. In the exploration state, the speed tracking reward weight is increased to encourage the robot to learn basic movements first. In the steady state, the posture stability reward and energy consumption penalty weights are gradually increased to make the robot's movements smoother and more energy-efficient. In the fine-tuning state, the joint safety penalty and smooth movement reward weights are increased to reduce over-excitation and protect the hardware.
[0269] During environmental interaction, the quadruped robot's body posture, joint angles, joint angular velocities, foot contact states, body linear velocity, body angular velocity, and IMU data are collected in real time. The policy network outputs actions based on the current state and interacts with the environment to obtain the next state. Subsequently, events such as foot slippage, support phase switching, body instability recovery, obstacle crossing transition, and sudden changes in landing error are detected, and event labels are attached to the corresponding samples. This allows subsequent priority sampling to focus more effectively on rare but high-value key samples in the quadruped robot scenario.
[0270] Next, training samples are constructed based on the current state, action, sub-reward vector, comprehensive reward, next state, termination flag, event label, and stage label. The comprehensive reward is obtained by multiplying each sub-reward item by its corresponding weight and then summing the results. Then, the TD error is calculated using the value network, and a composite importance index is constructed by combining the reward mutation magnitude, advantage function, stage coefficient, event coefficient, and rare transition coefficient. The rare transition coefficient is used to increase the sampling priority of low-frequency but high-value state transitions, especially applicable to uncommon but crucial events for policy optimization, such as slippage, obstacle crossing, and instability recovery.
[0271] Subsequently, the samples and their composite importance indices are written into the experience replay pool, and the priority tree structure is updated simultaneously. At the same time, the rate of change of the reward weight vector is monitored; when the change of the reward weight exceeds a preset threshold, high-priority samples are extracted from the experience pool, and the comprehensive reward, TD error, and composite importance indices are recalculated using the new reward weights. The corresponding nodes in the priority tree are then updated, thereby achieving dynamic recalibration of the experience pool and preventing historical samples under the old weights from continuing to participate in training with distorted priorities.
[0272] During the sampling phase, a basic sampling probability is first constructed based on the composite importance index of the samples. Then, a stage mask function is introduced in conjunction with the current training stage to achieve stage-aware targeted resampling. In the exploration state, the focus is on amplifying boundary exploration and failure recovery samples; in the steady state, the focus is on amplifying posture correction and energy consumption optimization samples; and in the fine-tuning state, the focus is on amplifying motion smoothing and joint safety samples. In this way, priority sampling not only follows the TD error but also the optimization objective of the current training stage.
[0273] Using the sample batches obtained from resampling, the policy network loss and value network loss are calculated, and the policy network parameters and value network parameters are updated using gradient descent. After the parameters are updated, the TD error and composite importance index of the samples sampled in this round are recalculated, and the corresponding nodes in the priority tree are updated for use in the next round of training. Then, the main decision for stage switching is made again based on the monitoring indicators, and the timing of stage switching is further corrected by combining the distribution of high-priority sample labels. When there is still a high proportion of labels for failure recovery, foot slippage, boundary exploration, or body instability recovery in high-priority samples, it indicates that the robot is still in the basic capability remediation stage and should postpone entering the next stage; when high-priority samples gradually shift to labels for posture correction, energy consumption optimization, motion smoothing, and joint safety, it indicates that the training focus has shifted, and it can be confirmed to switch to the next stage; if a large proportion of instability labels reappear in high-priority samples after the switch, it can revert to the previous stage or postpone the subsequent switch. This constitutes a two-layer stage switching mechanism of "main decision based on monitoring indicators + auxiliary correction based on sample label distribution".
[0274] Once the model converges, the trained policy network is exported in deployment format and validated on a simulation platform. After successful validation, it is deployed to a real quadruped robot. During deployment, speed tracking error, posture stability, energy consumption, joint safety, and motion smoothness are continuously monitored. When online performance degrades, the terrain environment changes, or abnormal samples accumulate, incremental fine-tuning of the policy network is triggered, and the value network can be updated synchronously in the background to improve subsequent online adaptability and offline retraining effectiveness.
[0275] The above description is merely a preferred embodiment of the present invention. These specific embodiments are different implementations based on the overall concept of the present invention, and the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.
Claims
1. A reinforcement learning training method for a quadruped robot based on stage-adaptive reward weights and TD error-priority sampling, characterized in that, Includes the following steps: Step 1: Construct a reinforcement learning training model based on the Actor-Critic architecture, including a policy network and a value network; wherein, the policy network is used to output actions based on the current state information of the quadruped robot; the value network is used to estimate the current state value or state-action value; Step 2: The training process of the reinforcement learning training model in Step 1 is divided into multiple stages using a joint decision mechanism; each stage corresponds to a different reward weight configuration through a stage adaptive reward weight calculation method. Step 3: Collect state information samples of the quadruped robot in real time, perform key event detection on the samples and attach event labels to construct training samples; Step 4: Calculate the initial composite importance index based on the training samples constructed in Step 3 and the current stage reward weights obtained in Step 2, and then write the samples and their corresponding priorities into the experience replay pool; monitor the rate of change of the reward weight vector after each training round or after each preset number of training steps. Step 5: Employ a phase-aware directional resampling method to construct a basic sampling probability distribution based on the importance index of frequently sampled samples; extract a batch of samples from the experience replay pool in Step 4 according to the sampling probability; calculate the loss functions of the policy network and value network based on the reinforcement learning training model in Step 1; update the parameters of the policy network and value network using gradient descent; recalculate the TD error and composite importance index of the sampled samples after the update; and update the corresponding nodes in the priority tree. Step Six: Based on the high-priority sample distribution obtained in Step Five, establish a two-way feedback loop between reward weights and priority sampling. When the training reaches the preset convergence condition or the upper limit of the training rounds, end the training. Export the trained policy network into a deployment format, first deploy it to the simulation platform for verification, and then deploy it to a real quadruped robot for real-time motion control. Perform online performance monitoring and make incremental fine-tuning as needed.
2. The quadruped robot reinforcement learning training method based on stage adaptive reward weights and TD error-priority sampling according to claim 1, characterized in that, The joint determination mechanism described in step two includes the following steps: In each training round Internally, the monitoring metrics include: average return growth rate. TD error variance Average tracking error Number of consecutive falls and frequency of falling The training process is divided into three stages: exploratory, stable, and fine-tuning, based on monitoring metrics. (1) When the most recent consecutive The exploratory state transitions to the stable state when all of the following conditions are met in a training epoch: in, To explore the lower bound of the average return growth rate monitoring indicator. To explore the upper limit of the TD error variance monitoring index. To explore the upper limit of the average tracking error monitoring index, To explore the lower limit of the continuous fall-free monitoring index, To explore the upper limit of the state for the fall frequency monitoring index; (2) When the most recent consecutive The system switches from the steady state to the fine-tuning state when all of the following conditions are met in a training epoch: in, This indicates that the growth in returns is slowing down. This represents the upper limit of the steady-state threshold for the average return growth rate monitoring indicator. , , , ; This represents the lower bound of the steady-state condition for the TD error variance monitoring index. This represents the lower bound of the steady-state performance of the average tracking error monitoring index. This represents the upper limit of the steady-state value of the monitoring indicator for consecutive falls-free periods. This represents the lower limit of the steady-state condition for the fall frequency monitoring index; (3) To avoid frequent stage jittering during switching, a minimum dwell requirement is set, that is, each stage must maintain at least Switching is only allowed after a certain number of training rounds; in, This indicates the number of consecutive training rounds required before the current phase switch; A phase switch is executed only if the current stage dwell time meets the minimum dwell requirement and the switching conditions are met consecutively a preset number of times. After entering a stable state or fine-tuning state, if the most recent consecutive Any of the following situations occur within a single statistics window: in, These are the abnormal thresholds used during fall frequency, TD error variance, and average tracking error phase backoff or delayed handover, respectively. ; This will trigger a phase rollback or delay subsequent switching.
3. The quadruped robot reinforcement learning training method based on stage adaptive reward weights and TD error-priority sampling according to claim 2, characterized in that, The specific steps of the stage adaptive reward weight calculation method described in step two are as follows: Regarding the first Each reward item during training. The weights are defined as follows: in: For the first Each reward item at time The weights; Basic weights; To adjust the amplitude; The moment to begin adjustment; For smooth transition coefficient; The rewards include at least: velocity tracking reward, attitude stability reward, energy consumption penalty, joint safety penalty, and smooth motion reward; in the exploration state, the weight of velocity tracking reward is increased, while the weight of attitude stability reward and energy consumption penalty is decreased; in the stable state, the weight of attitude stability reward and energy consumption penalty is gradually increased; in the fine-tuning state, the weight of joint safety penalty and smooth motion reward is increased.
4. The quadruped robot reinforcement learning training method based on stage adaptive reward weights and TD error-priority sampling according to claim 3, characterized in that, The state information mentioned in step three includes at least: body posture information, joint angles, joint angular velocities, foot contact state, body linear velocity, body angular velocity, and IMU measurement data; the policy network then adjusts the current state accordingly. Output Action The quadruped robot interacts with the environment based on the aforementioned actions to obtain the next state. ; The key event detection includes: foot slippage, support phase switching, body instability recovery, obstacle crossing transition, and sudden changes in landing error; The training samples are constructed as follows: in: This is the current state; For the current action; As a comprehensive reward; The next state; This is the end marker; For event tags; For stage labels.
5. The quadruped robot reinforcement learning training method based on stage adaptive reward weights and TD error-priority sampling according to claim 4, characterized in that, The specific steps for calculating the initial composite importance index in step four are as follows: The comprehensive reward formula is defined as follows: in, For the total number of reward items, For sub-reward items; the aforementioned Sub-reward items can be defined as: (1) Speed tracking reward in: The current linear velocity of the machine body. For the target linear velocity, The current angular velocity of the machine body. For the target angular velocity, These are the normalization coefficients; (2) Attitude stability reward in: For roll angle and pitch angle, The current height of the aircraft. For reference height, These are the weighting coefficients; (3) Energy consumption penalty in: For the first Joint torque, For the first Joint angular velocity, For the number of joints, To control the cycle; (4) Joint safety penalty in, It is the first The current angle of each joint. It is the first Safety angle thresholds for each joint It is the first Safety angle thresholds for each joint It is the first The safe angular velocity threshold for each joint; (5) Smooth motion reward in, It is a joint exist The target location at any given moment, and the drastic change in the penalty action command; Calculate the composite importance index for the samples. The specific steps are as follows: First, calculate the TD error: in: Discount factor; For value networks; This is the end marker; Redefining the magnitude of reward mutation: Define the advantage function estimate as , The composite importance index is defined as follows: in: Preset weighting coefficients; For stage coefficients; For event coefficients; This is the rare transfer coefficient.
6. The quadruped robot reinforcement learning training method based on stage adaptive reward weights and TD error-priority sampling according to claim 5, characterized in that, The specific steps for monitoring the rate of change of the reward weight vector in step four are as follows: Let the current reward weight vector be: Its relative rate of change is defined as: in, It is a very small positive number. when When this occurs, a dynamic recalibration of the experience pool is triggered. At this time, samples whose current priority ranking is within a preset proportion are extracted from the experience replay pool, and their comprehensive reward is recalculated using the new reward weight. TD error and composite importance index And update the corresponding node in the priority tree.
7. The quadruped robot reinforcement learning training method based on stage adaptive reward weights and TD error-priority sampling according to claim 6, characterized in that, The specific steps of constructing the basic sampling probability distribution using the importance index of the frequently sampled samples in step five are as follows: The sampling index is defined as Then the first The sampling probability of each sample is: in: For the first The probability that a sample will be sampled; For the first The composite importance index of each sample; the sampling index is defined as... This is used to control the degree of bias in priority sampling; This is the importance normalization term for all candidate samples in the experience replay pool; Specifically: In the exploration state, increase the sampling weight of boundary exploration samples and failure recovery samples in the current stage; in the steady state, increase the sampling weight of posture correction samples and energy consumption optimization samples in the current stage; in the fine-tuning state, increase the sampling weight of motion smoothing samples and joint safety samples in the current stage.
8. The quadruped robot reinforcement learning training method based on stage adaptive reward weights and TD error-priority sampling according to claim 7, characterized in that, The value network loss function in step five is defined as follows: in, The sample batch size is used for each parameter update by drawing samples from the experience replay pool. For the first The squared term of the TD error corresponding to each sample; The policy network loss function is defined as: in, For parameters The policy function represented by the policy network; For a moment The actions recorded in the output of the next policy network or in the samples; For a moment The state; The gradient descent method is used to update the policy network parameters. and value network parameters .
9. A reinforcement learning training method for a quadruped robot based on stage-adaptive reward weights and TD error-priority sampling according to claim 8, characterized in that, The specific steps for establishing a two-way feedback loop between reward weights and priority sampling, as described in step six, are as follows: Feedback A: Correction of Sample Distribution and Phase Switching Timing The phase switching is first determined based on monitoring indicators; after the basic conditions for phase switching are met, the label distribution of high-priority samples in the most recent training statistical windows is statistically analyzed to assist in correcting the timing of phase switching. Let the first The percentage of event-type tags in high-priority samples is as follows: in, For the first Number of class-labeled samples; Feedback B: Changes in reward weights and recalibration of the experience pool When the reward weight vector When the change exceeds the threshold, the high-priority old samples in the experience pool are recalculated to ensure that the sample priority is consistent with the current stage goal.
10. The quadruped robot reinforcement learning training method based on stage adaptive reward weights and TD error-priority sampling according to claim 1, characterized in that, The online performance monitoring mentioned in step six includes: speed tracking error, posture stability, energy consumption level, joint safety, and motion smoothness; The incremental fine-tuning involves updating the policy network with a low learning rate and simultaneously updating the value network in the background for subsequent offline retraining or online evaluation.