A hierarchical control method for a quadruped robot based on motion syntax

Abstract

The application relates to a four-legged robot hierarchical control method based on motion syntax, and belongs to the technical field of foot-type robot control. The method comprises the following steps: acquiring a state vector of the four-legged robot at a current time; inputting the state vector at the current time into a trained robot hierarchical controller to obtain torque instructions of each joint motor at the current time; wherein the robot hierarchical controller comprises the following modules: a high-level reinforcement learning strategy module, which is used for obtaining an action at the current time based on the state vector; a syntax interpretation module, which is used for mapping the action at the current time into a motion safety parameter vector; a central pattern generator compilation module, which is used for generating expected motion trajectories of each leg based on the motion safety parameter vector; a bottom-layer control module, which is used for obtaining torque instructions of each joint corresponding to each leg based on the expected motion trajectories of each leg; and torque instructions of each joint motor are input into each joint motor driver to drive the four-legged robot to perform corresponding actions.

Description

Technical Field

[0001] This invention relates to the field of legged robot control technology, and in particular to a hierarchical control method for quadruped robots based on motion grammar. Background Technology

[0002] Quadruped robots have broad application prospects in fields such as exploration and rescue, and material transportation due to their superior terrain adaptability and mobility. However, achieving stable motion control of quadruped robots in unstructured terrain environments, including steep slopes, gravelly sandy areas, and obstacle-filled paths, remains a technical challenge in this field.

[0003] Traditional model-based motion control methods for quadruped robots place high demands on the accuracy of complex dynamic models. In practical applications, due to the complexity and uncertainty of unstructured terrain, it is difficult to obtain accurate dynamic models, which often results in model-based robot motion control methods failing to achieve stable robot walking when facing unknown terrain.

[0004] In recent years, intelligent control methods, represented by reinforcement learning, have been gradually applied to the motion control of quadruped robots in unstructured terrain environments. However, reinforcement learning strategies typically use neural networks to directly output continuous control parameters, and their decision-making logic is a "black box." Engineers find it difficult to understand the reasons why the neural network outputs specific control parameters under specific conditions, making it difficult to troubleshoot faults. At the same time, existing methods usually rely on the penalty term in the reward function to constrain the robot's dangerous behavior. This "soft constraint" approach cannot mathematically guarantee safety, and the robot may still output dangerous parameter combinations in unknown terrain, leading to falls. Summary of the Invention

[0005] Based on the above analysis, the embodiments of the present invention aim to provide a hierarchical control method for quadruped robots based on motion grammar, in order to solve the problems of uncontrollable decision logic and lack of security in existing quadruped robot control methods.

[0006] The objective of this invention is mainly achieved through the following technical solutions: On the one hand, this invention provides a hierarchical control method for quadruped robots based on motion grammar, comprising the following steps: Obtain the state vector of the quadruped robot at the current moment; The current state vector is input into the trained robot hierarchical controller to obtain the torque commands for each joint motor at the current moment. The robot hierarchical controller includes: a high-level reinforcement learning strategy module for obtaining the current action of the quadruped robot based on the current state vector; a syntax interpretation module for mapping the current action of the quadruped robot into a motion safety parameter vector; a central pattern generator compilation module for generating the desired motion trajectory of each leg based on the motion safety parameter vector; and a low-level control module for obtaining the torque commands for each joint corresponding to each leg based on the desired motion trajectory of each leg. The torque commands of each joint motor at the current moment are input to the motor drivers of each joint of the quadruped robot to drive the quadruped robot to perform the corresponding actions.

[0007] Furthermore, the syntax interpretation module performs the following steps to map the current action of the quadruped robot into a motion safety parameter vector: Based on the current state vector and the preset constraints corresponding to each action, determine whether the action at the current moment is safe; When the action is safe, the action at the current moment is mapped to the original motion parameter vector based on the preset action execution parameter table; When the original motion parameter vector is within the preset safety subspace range corresponding to the action, the original motion parameter vector is used as the motion safety parameter vector; When the original motion parameter vector is not within the safety subspace corresponding to the action, the original motion parameter vector is projected to the nearest point in the safety subspace to obtain the motion safety parameter vector. When the action is unsafe, the preset motion parameter vector is used as the motion safety parameter vector.

[0008] Furthermore, the state vector includes the quadruped robot's torso roll angle, torso pitch angle, torso roll angular velocity, torso pitch angular velocity, contact state of each foot end, and rotational speed of each foot end wheel assembly; The step of determining whether the action at the current moment is safe based on the current state vector and the preset constraints corresponding to each action includes: Determine whether the maximum absolute value of the torso roll angle and the torso pitch angle is greater than a preset fuselage tilt angle threshold. Determine whether the number of ground-contacting legs of the quadruped robot is greater than 1; Determine whether the maximum value of the rotational speed of each foot wheel set is greater than a preset wheel set rotational speed threshold; Determine whether the maximum value of the angular velocities of each joint of the quadruped robot is greater than a preset joint angular velocity threshold; Determine whether the remaining battery power or terrain slope of the quadruped robot meets the preset action execution conditions at the current moment; If any of the above conditions are not met, the action is deemed unsafe.

[0009] Furthermore, the motion safety parameter vector includes convergence coefficient, angular frequency, diagonal coupling weight, same-side coupling weight, target amplitude, and phase offset; The safety subspace range is the range of values ​​for each motion safety parameter pre-calibrated based on the physical limits of the quadruped robot.

[0010] Furthermore, the central mode generator compilation module uses the Hopf oscillator model, taking each parameter in the motion safety parameter vector as the coefficient of the Hopf oscillator differential equation system, and obtaining the expected motion trajectory of each joint by solving the differential equation system.

[0011] Furthermore, the high-level reinforcement learning strategy module includes a multilayer perceptron; The multilayer perceptron is trained through reinforcement learning, including: In the simulation environment, the current state vector of the quadruped robot, the action output by the multilayer perceptron at the current moment, the reward from environmental feedback, and the state vector at the next moment are collected to construct experience data, which is then stored in the experience replay pool. When the experience data in the experience replay pool reaches a preset amount, several pieces of experience data are randomly selected as training data. Based on the training data, with the goal of maximizing the cumulative reward, the parameters of the multilayer perceptron are updated through backpropagation until the model converges or reaches a preset number of training rounds, thus obtaining a trained multilayer perceptron.

[0012] Furthermore, the cumulative reward is obtained by weighted summation of task rewards and safety violation penalties; wherein the weight of the safety violation penalty is dynamically adjusted during the training process. The task reward is obtained by weighted summation of speed tracking reward, fuselage altitude reward, attitude stability reward, energy efficiency reward, and motion smoothness reward; The safety violation penalty includes a first penalty item and a second penalty item; wherein, the first penalty item is a preset penalty value set when the number of the quadruped robot's ground-touching legs is less than 2 at any time; the second penalty item is calculated based on the current state vector, action, and preset constraints of the training data.

[0013] Furthermore, the weights of the safety violation penalties are adjusted using the dual ascent method; wherein, when the proportion of safety violation samples in a batch of training data exceeds a preset maximum safety violation proportion constraint value, the weights are increased based on the proportion of safety violation samples and the maximum safety violation proportion constraint value; when the proportion of safety violation samples is lower than the maximum safety violation proportion constraint value, the weights are decreased based on the proportion of safety violation samples and the maximum safety violation proportion constraint value.

[0014] Furthermore, when updating the parameters of the multilayer perceptron, the following policy gradient calculation formula is used: ; in, Represents the gradient operator; The parameters of the multilayer sensor are represented; Represents a policy network for reinforcement learning; The action at time t is indicated; The state vector at time t represents the state vector at time t; Represents the dominance function at time t; Let represent the gradient contribution term of the safety violation penalty at time t; This indicates the weight of the penalty for safety violations. Furthermore, the speed tracking reward is used to reward the quadruped robot for its actual movement speed keeping up with the command speed; The body height reward is used to reward the quadruped robot for maintaining its body height at a preset height; The attitude stability reward is used to reward the quadruped robot for having roll and pitch angles close to zero. The energy efficiency reward is used to reward the low power consumption of each joint of the quadruped robot; The action smoothing reward is used to reward actions that are consistent at adjacent time points.

[0015] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects: 1. The robot hierarchical controller constructed in this invention uses a four-layer architecture with decoupled design, which makes the rhythm of the bottom central pattern generator unaffected by the fluctuation of the high-level strategy. Even if the high-level neural network outputs erroneous symbols due to sensor noise, the safety filtering mechanism of the syntax interpreter can intercept dangerous instructions and ensure that the robot returns to a safe gait, which greatly enhances the fault tolerance of the system and improves the robustness of the robot.

[0016] 2. This invention sets up a dual safety mechanism in the robot hierarchical controller. By judging whether the action is feasible before the action is executed, blocking the impossible actions, and correcting the original parameters to the predefined safety subspace through safety projection, the invention prevents the generation of actions with dangerous parameter combinations. Furthermore, the invention transforms the robot stability constraints into linear temporal logic formulas and embeds them into reinforcement learning training through the Lagrange relaxation method, so that the trained robot hierarchical controller has dual safety constraints.

[0017] 3. In the training module of high-level reinforcement learning strategy, this invention uses the dual ascent method to dynamically adjust the weight of safety violation penalties in the cumulative reward based on the comparison result of the average value of safety violation penalties and the preset maximum safety violation constraint value. This enables the training process to maximize task rewards while satisfying safety constraints, thus solving the problem of balancing safety and efficiency in traditional methods with fixed weights.

[0018] 4. This invention employs a Hopf oscillator model in the central pattern generator compilation module and uses each parameter in the motion safety parameter vector as the coefficient of the Hopf oscillator differential equation system. The desired motion trajectory of each joint is generated by numerical integration, thus achieving a smooth mapping from action to continuous trajectory generation and ensuring the stability of robot motion.

[0019] In this invention, the above-described technical solutions can be combined with each other to achieve more preferred combinations. Other features and advantages of this invention will be set forth in the following description, and some advantages may become apparent from the description or be learned by practicing the invention. The objects and other advantages of this invention can be realized and obtained from what is particularly pointed out in the description and drawings. Attached Figure Description

[0020] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts.

[0021] Figure 1 This is a flowchart illustrating a hierarchical control method for a quadruped robot based on motion grammar, as described in an embodiment of the present invention. Detailed Implementation

[0022] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, which form part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not intended to limit the scope of the present invention.

[0023] A specific embodiment of the present invention discloses a hierarchical control method for a quadruped robot based on motion grammar, such as... Figure 1 As shown, it includes the following steps S1-S3: Step S1: Obtain the state vector of the quadruped robot at the current moment.

[0024] Specifically, the state vector of the quadruped robot at the current moment is obtained through the sensors mounted on the robot. This state vector represents the instantaneous state of the quadruped robot, including its torso roll angle, torso pitch angle, torso roll angular velocity, torso pitch angular velocity, contact state of each foot end, and rotational speed of each foot wheel assembly. The torso roll angle and the torso pitch angle are used to represent the body posture of the quadruped robot, and are acquired in real time by an inertial measurement unit (IMU) installed at the center of the robot's torso.

[0025] The torso roll angular velocity and the torso pitch angular velocity are used to represent the rotational motion state of the quadruped robot's torso, and are acquired in real time by the gyroscope in the inertial measurement unit (IMU).

[0026] The foot contact state of each foot is used to indicate the foot contact state of each foot. The contact between the sole of the foot and the ground is sensed by a pressure sensor integrated at the end of the leg kinetic chain to obtain the foot pressure value of each foot. The foot contact state is obtained by comparing it with a preset pressure threshold. When the foot contact state of each foot is 0, it means that there is no contact with the ground. When the foot contact state of each foot is 1, it means that there is contact with the ground.

[0027] The rotational speed of each foot wheel assembly is used to indicate the wheel slippage or speed tracking error when the quadruped robot has a hybrid wheel-leg configuration and is in wheel mode. The rotational speed of the motor is acquired by the motor encoder integrated in the wheel drive unit corresponding to each foot, and the linear velocity of the wheel assembly is calculated through the transmission ratio as the rotational speed of each foot wheel assembly. It should be noted that when the quadruped robot is a purely legged robot without wheels, the state vector is a vector with the rotational speeds of each foot wheel assembly removed.

[0028] Step S2: Input the current state vector into the trained robot hierarchical controller to obtain the torque commands of each joint motor at the current moment; wherein, the robot hierarchical controller includes: a high-level reinforcement learning strategy module, used to obtain the current action of the quadruped robot based on the current state vector; a syntax interpretation module, used to map the current action of the quadruped robot into a motion safety parameter vector; a central pattern generator compilation module, used to generate the desired motion trajectory of each leg based on the motion safety parameter vector; and a low-level control module, used to obtain the torque commands of each joint corresponding to each leg based on the desired motion trajectory of each leg.

[0029] Specifically, the robot hierarchical controller includes a high-level reinforcement learning strategy module, a syntax interpretation module, a central pattern generator compilation module, and a low-level control module connected in sequence. This is used to achieve physical isolation between high-level intelligent decision-making and low-level deterministic execution. Through the hierarchical architecture, the high-dimensional continuous control problem is decomposed into a low-dimensional discrete decision problem and a parameterized trajectory generation problem, ensuring the real-time performance and stability of the system. At the same time, each module can be independently optimized and replaced. The data flow of the four modules is passed from top to bottom, and the output of each layer is directly used as the input of the next layer, forming a tight logical chain.

[0030] Furthermore, the high-level reinforcement learning policy module is a policy network using a multilayer perceptron. It forward-propagates the input state vector at the current moment, calculates the logits score of each action in the action space, normalizes it using the Softmax function to obtain the probability distribution of the actions, and then uses a greedy policy to select the action with the highest probability as the action of the quadruped robot at the current moment. The action space includes standing, swinging, leg lifting, pushing off the ground, transition, climbing, jumping, and wheeled modes.

[0031] Furthermore, the multilayer perceptron is trained through reinforcement learning, including: In the simulation environment, the current state vector of the quadruped robot, the action output by the multilayer perceptron at the current moment, the reward from environmental feedback, and the state vector at the next moment are collected to construct experience data, which is then stored in the experience replay pool. When the experience data in the experience replay pool reaches a preset amount, a number of experience data are randomly selected as training data.

[0032] Based on the training data, with the goal of maximizing the cumulative reward, the parameters of the multilayer perceptron are updated through backpropagation until the model converges or reaches a preset number of training rounds, thus obtaining a trained multilayer perceptron.

[0033] Specifically, a virtual model of the quadruped robot is constructed on simulation platforms such as NVIDIA Isaac Lab, and various unstructured terrains are set as training scenarios; at each time step t, empirical data quadruples are constructed. ,in, This represents the state vector at time t; Let t represent the output action of the quadruped robot at time t. In this embodiment, in order to improve the exploration ability of the policy network and avoid getting trapped in local optima during the training process, a sampling strategy is adopted when selecting the output action, that is, the action is randomly selected according to the probability distribution of the Softmax output, so as to ensure that the quadruped robot fully explores various action combinations, especially actions that are currently considered to have a low probability but may bring higher long-term rewards. Representation based on state vector and actions The calculated reward; This represents the state vector of the quadruped robot at the next moment after it performs an action.

[0034] The empirical data quadruples are stored in the empirical replay pool. Once the empirical replay pool has accumulated a certain amount of empirical data, training is started to ensure stable data distribution. At the same time, a small batch of data is randomly selected from the pool for each training session to break the temporal correlation of the data and avoid the occurrence of local optima. For example, the small batch of data can be 50 empirical data points.

[0035] More specifically, during training, the reinforcement learning policy network takes the current state as input and, by learning from historical experience, determines the action to be taken in a given state in order to maximize the expected reward. The output of the policy network corresponds to the action space, and its optimization goal is to make the generated action more likely to reach a high-reward state in the current state.

[0036] Furthermore, the cumulative reward is obtained by weighted summation of task rewards and safety violation penalties; wherein the weight of the safety violation penalty is dynamically adjusted during the training process.

[0037] Specifically, this embodiment uses a constrained Markov decision process framework, and the training objective is to find the optimal policy. This maximizes the expected cumulative reward while satisfying safety constraints; the formula for calculating the expected cumulative reward is as follows: ;in, Represents the expectation operator; This represents the discount factor, with a value range of 100%. ; This represents the task reward at time t; This indicates the weight of the security violation penalty; This indicates the penalty for the aforementioned security violation; This represents the state vector at time t; This represents the action at time t.

[0038] Furthermore, the task reward is obtained by weighted summation of speed tracking reward, fuselage altitude reward, attitude stability reward, energy efficiency reward, and motion smoothness reward.

[0039] Specifically, the task reward is calculated using the following formula:

[0040] in, Indicates speed tracking reward; Indicates the weight of the speed tracking reward; Indicates a height bonus for the aircraft. Indicates the weight of the fuselage height bonus; Indicates a stable attitude reward; Indicates the weight of the attitude stability reward; Indicates an energy efficiency reward; Indicates the weight of the energy efficiency reward; Indicates a smooth reward for the action; This indicates the weight of the smoothness reward for the action.

[0041] The speed tracking reward is used to reward the quadruped robot for keeping up with the commanded speed in terms of actual movement speed. The smaller the error, the higher the reward. The formula is as follows: ; in, Indicates the speed of the instruction; The actual speed is calculated by fusing the inertial measurement unit (IMU) and the wheel encoder. The IMU measures the fuselage acceleration and integrates it over time to obtain the IMU speed. The wheel encoder measures the rotational speed of each wheel group and converts it into the fuselage speed. This speed is then fused with the IMU speed using complementary filtering or Kalman filtering to obtain an estimate of the actual speed.

[0042] The body height reward is used to reward the quadruped robot for maintaining its body height at a preset height, preventing it from being too high or too low. The formula is as follows: ; in, Indicates the preset height of the fuselage; The height of the quadruped robot is indicated by the angles of each joint and the robot's kinematic model. The position of the foot relative to the torso is calculated, and the height of the torso relative to the ground is obtained by combining the foot's contact with the ground state.

[0043] The attitude stability reward, which rewards the quadruped robot for having roll and pitch angles close to zero to maintain a horizontal position, is calculated using the following formula: ; in, This represents the roll angle in the state vector; This represents the pitch angle in the state vector.

[0044] The energy efficiency reward is used to reward the low power consumption of each joint of the quadruped robot, avoiding unnecessary torque output and high-speed movement, thereby reducing energy consumption. Its formula is as follows: ; in, This indicates the torque command for the i-th joint; This represents the angular velocity of the i-th joint.

[0045] The action smoothing reward, used to reward actions that are consistent at adjacent time points, is formulated as follows: .

[0046] in, Indicates the action at time t; This represents the action at time t-1; This indicates an indicator function that takes the value 1 if the condition is true and 0 otherwise.

[0047] Furthermore, the safety violation penalty includes a first penalty item and a second penalty item; wherein, the first penalty item is a preset penalty value set when the number of ground-touching legs of the quadruped robot is less than 2 at any time; the second penalty item is calculated based on the current state vector, action, and preset constraints of the training data.

[0048] Specifically, the safety violation penalty is used to constrain the movement behavior of the quadruped robot and prevent planned actions from causing it to enter a dangerous state.

[0049] The first penalty term represents a stability penalty, which mathematically guarantees the stability and safety of the quadruped robot by strictly prohibiting the number of its legs in contact with the ground from being less than two at any time, i.e., a single-leg support or suspended state. More specifically, the first penalty term is based on the definition of the Linear Temporal Logic (LTL) safety specification.

[0050] The Linear Temporal Logic (LTL) safety specification is a logic language used to describe how a system changes over time. It can express temporal relationships such as "always", "eventually", and "until". In this embodiment, the "always" operator in the linear temporal logic is used to formally describe the safety attributes that the quadruped robot must permanently satisfy.

[0051] Understandably, based on the linear sequential logic, the calculation method for the first penalty term is as follows: at each time step t, the number of legs touching the ground is calculated based on the foot contact state of each foot in the state vector at the current time. When the number of legs touching the ground is less than 2, it is determined that the safety regulations are violated, and the first penalty term is set to a preset penalty value, which can be set to -100 for example.

[0052] Furthermore, the second penalty term is used to constrain other unsafe behaviors of the quadruped robot, including body posture safety, joint speed safety, wheel speed safety, and safety conditions corresponding to each action; wherein, The body posture safety is used to prevent the quadruped robot from becoming unstable and falling due to excessive body tilt. When the maximum value of the absolute values ​​of the torso roll angle and the torso pitch angle of the quadruped robot is greater than a preset body tilt angle threshold, it is judged as unsafe, and the body posture safety is set to a preset penalty value, which can be set to -10 for example.

[0053] The joint speed safety is used to protect the actuators of each joint of the quadruped robot from malfunction due to excessive rotational speed. When the absolute value of the angular velocity of any joint of the quadruped robot exceeds the preset maximum permissible angular velocity, it is determined to be unsafe, and the joint speed safety is set to a preset penalty value, which can be set to -10 for example.

[0054] The wheel set speed safety is used to protect the wheel drive module of the quadruped robot and prevent the wheel set from going out of control or being damaged due to excessive speed. When the absolute value of the speed of any wheel set at the end of the quadruped robot exceeds the preset maximum allowable speed, it is determined to be unsafe, and the wheel set speed safety is set to a preset penalty value, which can be set to -10 for example.

[0055] The safety conditions corresponding to each action are used to ensure that the quadruped robot has sufficient resources and environmental conditions when executing the output actions of the high-level reinforcement learning strategy module, so as to prevent action execution failure or safety accidents due to insufficient conditions.

[0056] As mentioned above, the actions include standing, swinging, leg lifting, pushing off the ground, transitioning, climbing, jumping, and wheeled modes. When the action is standing, since standing is the most basic safety-assured action and does not involve complex dynamic movements or high power output, no constraints are set, and standing is allowed in any state to ensure that the robot always has a safe state that can be retreated. When the action is swinging, the remaining battery power is required to be no less than a preset first battery threshold, for example, 10%, to avoid the robot being unable to maintain balance due to energy consumption of the swinging action in a low battery state. When the action is leg lifting or pushing off the ground, the remaining battery power is required to be no less than a preset second battery threshold, for example, 20%, to ensure that the leg lifting action has enough energy to complete the foot leaving the ground and crossing, and that the pushing off the ground action can generate enough propulsion. When the action is transitioning, it serves as an intermediate buffer state during different states or mode switching. Constraints are set to allow execution in any state to ensure smooth gait switching. When the action is climbing, the remaining battery power must not be lower than a preset third battery threshold, for example, 30%, and the slope of the terrain where the quadruped robot is currently located must not be greater than a preset maximum climbing slope, for example, 45°, to ensure that the climbing action has sufficient energy and the terrain conditions are suitable. When the action is jumping, the remaining battery power must not be lower than a preset fourth battery threshold, for example, 50%, and the body pitch angle must not be greater than a preset maximum jump pitch angle, for example, 10°, to ensure that the jumping action has sufficient energy and the body posture is suitable for takeoff. When the action is wheeled mode, the slope of the terrain where the quadruped robot is currently located must not be greater than a preset maximum wheeled mode slope, for example, 20°, to prevent slippage or loss of control when switching to wheeled mode on terrain with excessively steep slopes.

[0057] Understandably, when the state of the quadruped robot or the terrain slope does not meet the safety conditions corresponding to the current action, the action safety is set to a preset penalty value, for example, -10. It should be noted that the terrain slope used for determining the condition in this embodiment can be estimated through body perception information, including: when the quadruped robot's foot touches the ground, calculating the position of the foot relative to the body using forward kinematics based on the body posture angle and the angles of each leg joint, and combining this with the body posture at the moment of contact with the ground to infer the current terrain slope.

[0058] Furthermore, the weight of the security violation penalties The weights are used to balance the proportion of task rewards and safety constraints in the cumulative reward. These weights are dynamically adjusted during training so that the policy network can maximize task rewards while satisfying safety constraints, thus achieving an adaptive balance between safety and efficiency.

[0059] Specifically, the weights of the safety violation penalties are adjusted using the dual-ascending method; wherein, when the proportion of safety violation samples in a batch of training data exceeds a preset maximum safety violation proportion constraint value, the weights are increased based on the proportion of safety violation samples and the maximum safety violation proportion constraint value; when the proportion of safety violation samples is lower than the maximum safety violation proportion constraint value, the weights are decreased based on the proportion of safety violation samples and the maximum safety violation proportion constraint value.

[0060] More specifically, the weights of the security violation penalties are adjusted using the following formula: ; in, Indicates the weight of the current batch; Indicates the updated weights; This indicates the preset learning rate; This represents an empirical estimate of the proportion of safety violations based on the current training batch data. In this embodiment, the proportion of safety violation samples among all samples in the batch is used as the expected estimate. Here, a safety violation sample is an action at the current moment that violates any of the safety violation penalties. This represents the preset maximum safety violation ratio constraint value, which can be set to 0.01 for example. This indicates a non-negative projection operation, meaning that when the value inside the parentheses is less than 0, it is set to 0, ensuring that the multiplier is always non-negative, thereby maximizing the task reward while ensuring safety.

[0061] Furthermore, in this embodiment, the multilayer perceptron is trained using the policy gradient method. When updating the parameters of the multilayer perceptron, the following policy gradient calculation formula is used: ; in, Represents the gradient operator; The parameters of the multilayer sensor are represented; Represents a policy network for reinforcement learning; The action at time t is indicated; The state vector at time t represents the state vector at time t; Represents the dominance function at time t; The gradient contribution term for the security violation penalty at time t is used to penalize the policy network for selecting actions that may lead to security violations. This indicates the weight of the penalty for safety violations. Specifically, in traditional policy gradient methods, the policy gradient calculation formula relies solely on the advantage function to update parameters during policy function training. In this embodiment, a safety violation penalty term is introduced into the policy gradient calculation formula, enabling the policy network to simultaneously learn to efficiently complete tasks and avoid dangerous behaviors during training, ultimately obtaining the optimal policy that maximizes task rewards while satisfying safety constraints.

[0062] Furthermore, the syntax interpretation module performs the following steps to map the current action of the quadruped robot into a motion safety parameter vector: Based on the current state vector and the preset constraints corresponding to each action, determine whether the action at the current moment is safe; When the action is safe, the action at the current moment is mapped to the original motion parameter vector based on the preset action execution parameter table; When the original motion parameter vector is within the preset safety subspace range corresponding to the action, the original motion parameter vector is used as the motion safety parameter vector; When the original motion parameter vector is not within the safety subspace corresponding to the action, the original motion parameter vector is projected to the nearest point in the safety subspace to obtain the motion safety parameter vector. When the action is unsafe, the preset motion parameter vector is used as the motion safety parameter vector.

[0063] Specifically, to ensure the safety of the quadruped robot's movement, this embodiment first determines the safety of the current action output by the high-level reinforcement learning strategy module. Based on the current state vector and the preset constraints corresponding to each action, determine whether the action at the current moment is safe, including: Determine whether the maximum absolute value of the torso roll angle and the torso pitch angle is greater than a preset fuselage tilt angle threshold. Determine whether the number of ground-contacting legs of the quadruped robot is greater than 1; Determine whether the maximum value of the rotational speed of each foot wheel set is greater than a preset wheel set rotational speed threshold; Determine whether the maximum value of the angular velocities of each joint of the quadruped robot is greater than a preset joint angular velocity threshold; Determine whether the remaining battery power or terrain slope of the quadruped robot meets the preset action execution conditions at the current moment; If any of the above conditions are not met, the action is deemed unsafe.

[0064] Specifically, the method for determining the safety of an action at the current moment is the same as the method for determining each penalty item in the safety violation penalty used to calculate the cumulative reward in the aforementioned training high-level reinforcement learning strategy module, and will not be repeated here; when any item of the safety judgment is not satisfied, the action is determined to be unsafe, and when all items of the safety judgment are satisfied, the action is determined to be safe.

[0065] More specifically, the motion safety parameter vector includes six parameters used to drive the central pattern generator (CPG) to generate the desired motion trajectory for each joint, including convergence coefficient, angular frequency, diagonal coupling weight, ipsilateral coupling weight, target amplitude, and phase shift; among which, The convergence coefficient The convergence coefficient is used to control the speed at which the central pattern generator converges to the limit cycle. The larger the convergence coefficient, the faster the central pattern generator responds and the faster it can track the target amplitude and frequency, but it may cause sudden changes in torque command. The smaller the convergence coefficient, the smoother the transition, but there is a response delay.

[0066] The angular frequency The angular frequency is used to control the oscillation period of the central pattern generator, which determines the speed of the robot's movement. The larger the angular frequency, the higher the movement frequency, and the faster the quadruped robot walks. When the angular frequency is 0, the central pattern generator outputs a constant value, which corresponds to static actions such as standing.

[0067] The diagonal coupling weight It is used to control the movement synchronization of the diagonal legs of the quadruped robot, namely, the left front-right rear and the right front-left rear; increasing the diagonal coupling weight can enhance the coordination of the diagonal gait.

[0068] The same-side coupling weight It is used to control the motion synchronization between the same-side legs of the quadruped robot, namely, left front-left rear and right front-right rear; increasing the same-side coupling weight can enhance the coordination of the same-side gait.

[0069] The target amplitude It is used to control the steady-state amplitude output of the central pattern generator, and its value determines the range of motion of the joint and the swing height of the foot; the larger the target amplitude, the larger the stride and the stronger the obstacle crossing ability, but it may exceed the mechanical limit.

[0070] The phase shift This is used to control the initial phase of the output of the central pattern generator to stagger the movement phases of each leg and achieve different gaits; for example, in the diagonal trot gait, the phase difference between the diagonal legs is set to π.

[0071] Furthermore, the action execution parameter table is a predefined key-value pair mapping table. Each action corresponds to a set of motion parameter vectors, which are determined in advance through offline experiments and debugging before system deployment. Based on the action at the current moment, the action is mapped to the original motion parameter vector through a lookup operation. However, the original motion parameter vector obtained by looking up the table may have some parameters that exceed the physical limits of the robot. Therefore, this embodiment further presets a safety subspace, which is a continuous value range of each motion safety parameter pre-calibrated based on the physical limits of the quadruped robot, used for safety verification and correction of the original motion parameter vector.

[0072] For example, the value range of each parameter in the security subspace is set as follows: Set the range of values ​​for the convergence coefficient to be: Its lower limit ensures that the central mode generator can establish a stable limit cycle, and its upper limit is used to prevent excessively fast convergence from causing sudden changes in joint torque.

[0073] The range of angular frequency values ​​is set as follows: Exceeding the frequency limit may cause the motor to overheat or mechanically resonate.

[0074] Set the range of values ​​for the diagonal coupling weights to be: The coupling weights are usually normalized, and exceeding the upper limit may lead to oscillation and divergence.

[0075] Set the range of values ​​for the same-side coupling weights as follows: This is to prevent excessive coupling between the legs from causing motion interference.

[0076] Set the range of values ​​for the target amplitude as follows: Exceeding the upper limit may cause the foot to go out of the working space or cause mechanical interference.

[0077] Set the phase offset value range to Since the phase is usually in Within the period, values ​​outside the range have no physical meaning.

[0078] It should be noted that the above safety boundary values ​​are calculated based on the physical parameters of the quadruped robot used in the experiment of this embodiment. For robots of different sizes or configurations, those skilled in the art can make adaptive adjustments to the above safety boundary values ​​according to their physical limits, and these adjustments are still within the protection scope of this invention.

[0079] Furthermore, when the original motion parameter vector is not within the safe subspace corresponding to the action, the original motion parameter vector is projected to the nearest point within the safe subspace to obtain the motion safety parameter vector; wherein, the projection is to find a point within the safe subspace such that the Euclidean distance between that point and the original motion parameter vector is minimized.

[0080] In this embodiment, the safe subspace is the Cartesian product of independent continuous intervals of each parameter. Therefore, the above projection operation can be simplified to constraining each parameter within its corresponding safe interval. That is, for the convergence coefficient, the projection formula is: ;in, and Let represent the minimum and maximum values ​​of the convergence coefficients, respectively; for angular frequency, the projection formula is... ;in, and Let represent the minimum and maximum angular frequencies, respectively; for diagonally coupled weights, the projection formula is: ;in, and Let represent the minimum and maximum values ​​of the diagonal coupling weights, respectively; for the same-side coupling weights, the projection formula is: ;in, and Let represent the minimum and maximum values ​​of the same-side coupling weights, respectively; for the target amplitude, the projection formula is: ;in, and Let represent the minimum and maximum values ​​of the target amplitude, respectively; for phase shift, the projection formula is: ;in, and These represent the minimum and maximum values ​​of the phase offset, respectively.

[0081] Understandably, the essence of projection is to pull physically unexecutable parameters back into a safe area while preserving the kinematic characteristics of the original motion as much as possible. For example, when the target amplitude required by the original motion is too large, the maximum allowable amplitude should still be maintained after projection, rather than directly discarding the motion or switching to other motions, so as to preserve the original motion intention to the greatest extent while ensuring safety.

[0082] More specifically, when the action violates any of the safety judgment requirements, in order to ensure the movement safety of the quadruped robot, a preset safety action, such as a standing action, and the corresponding movement safety parameter vector are output, so that the quadruped robot is set to a stable standing state, the dangerous action command is blocked, and the quadruped robot is always in a safe state.

[0083] Furthermore, the central mode generator compilation module uses the Hopf oscillator model, taking each parameter in the motion safety parameter vector as the coefficient of the Hopf oscillator differential equation system, and obtaining the expected motion trajectory of each joint by solving the differential equation system.

[0084] Specifically, the Hopf oscillator is a nonlinear oscillator with stable limit cycle characteristics, capable of generating smooth periodic motion trajectories. By adjusting the oscillator parameters, the frequency and amplitude of the output trajectory can be independently controlled, making it suitable for motion control of quadruped robots.

[0085] More specifically, a Hopf oscillator is established for each leg of the quadruped robot, and the oscillators are coordinated between the legs through coupling terms. The state variable differential equations of the Hopf oscillator corresponding to each leg are as follows: ; in, Let represent the time derivative of the master state variable of the oscillator of the i-th leg; Let represent the derivative of the auxiliary state variable of the oscillator of the i-th leg with respect to time; Indicates the convergence coefficient; Let i represent the main state variable of the oscillator of the i-th leg; Let represent the auxiliary state variable of the oscillator of the i-th leg, which is orthogonal to the main state variable; Let represent the expected joint angle trajectory of the i-th leg; This represents the shape factor, used to control the shape of the oscillator waveform; Indicates angular frequency; Indicates the target amplitude; The coupling weight between the i-th leg and the j-th leg is represented by the weights constructed based on the diagonal coupling weights and the same-side coupling weights; when i=j, That is, the legs themselves are not coupled; when i ≠ j, if the i-th leg and the j-th leg are diagonally related, then If the i-th leg and the j-th leg are on the same side, then .

[0086] In each control cycle (e.g., 10ms), the central mode generator compilation module obtains the motion safety parameter vector at the current moment, solves the above differential equation system using a numerical integration method, and obtains the state variable value at the next moment. For example, the fourth-order Runge-Kutta method can be used for numerical integration. Since the fourth-order Runge-Kutta method has high accuracy and stability, it can maintain numerical stability under a long step size.

[0087] Understandably, by using the parameters in the motion safety parameter vector as coefficients of the Hopf oscillator differential equation system, effective decoupling between high-level discrete decision-making and low-level continuous execution is achieved. The action decision layer only needs to output the motion intention, i.e. the action, and the central pattern generator is responsible for generating a continuous rhythmic trajectory that satisfies the intention. This reduces the learning difficulty of the policy network and ensures the smoothness and coordination of the motion.

[0088] Furthermore, the underlying control module receives the expected motion trajectory of each leg output by the central pattern generator compilation module, converts the foot task space trajectory into the expected angle of each joint corresponding to each leg through inverse kinematics calculation, and finally calculates the torque command of each joint corresponding to each leg using impedance control law.

[0089] Step S3: Input the torque command of each joint motor at the current moment to the driver of each joint motor of the quadruped robot to drive the quadruped robot to perform the corresponding action.

[0090] Specifically, the torque command of each joint motor at the current moment is input to the driver of each joint motor of the quadruped robot, driving each joint motor to output the corresponding torque, so that the quadruped robot can perform the desired action at the current moment and complete the current control cycle.

[0091] Furthermore, when it is necessary to add new actions to the quadruped robot, the action to be added is added to the action space. At the same time, according to the kinematic characteristics of the quadruped robot, the parameter mapping relationship corresponding to the action to be added is configured to the action execution parameter table, and the corresponding constraints are set. Finally, the training data corresponding to the action to be added is obtained in the simulation environment, and all parameters except the last fully connected layer of the multilayer perceptron of the high-level reinforcement learning policy module are frozen. Based on the training data corresponding to the action to be added, the multilayer perceptron is fine-tuned and trained. Thus, the addition of actions can be completed with a small number of samples without destroying the original motor skills.

[0092] In summary, the hierarchical control method for quadruped robots based on motion grammar, as described in this embodiment of the invention, has the following beneficial effects: 1. The robot hierarchical controller constructed in this embodiment of the invention uses a four-layer architecture with decoupled design, which makes the rhythm of the bottom central pattern generator unaffected by the fluctuation of the high-level strategy. Even if the high-level neural network outputs erroneous symbols due to sensor noise, the safety filtering mechanism of the syntax interpreter can intercept dangerous instructions and ensure that the robot returns to a safe gait, which greatly enhances the fault tolerance of the system and improves the robustness of the robot.

[0093] 2. In this embodiment of the invention, a dual safety mechanism is set in the robot hierarchical controller. By determining whether the action is feasible before the action is executed, the infeasible actions are shielded, and the original parameters are corrected to a predefined safety subspace through safety projection, the actions of dangerous parameter combinations are prevented. The robot stability constraints are transformed into linear temporal logic formulas and embedded into reinforcement learning training through the Lagrange relaxation method, so that the trained robot hierarchical controller has dual safety constraints.

[0094] 3. In the high-level reinforcement learning strategy training module, this invention uses the dual ascent method to dynamically adjust the weight of the safety violation penalty in the cumulative reward based on the comparison between the average value of the safety violation penalty and the preset maximum safety violation constraint value. This allows the training process to maximize the task reward while satisfying the safety constraints, solving the problem of balancing safety and efficiency in traditional methods with fixed weights.

[0095] 4. In this embodiment of the invention, the Hopf oscillator model is used in the central pattern generator compilation module, and the parameters in the motion safety parameter vector are used as coefficients of the Hopf oscillator differential equation system. The expected motion trajectory of each joint is generated by numerical integration, which realizes a smooth mapping from action to continuous trajectory generation and ensures the stability of robot motion.

[0096] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware, and the program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.

[0097] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A hierarchical control method for a quadruped robot based on motion grammar, characterized in that, Includes the following steps: Obtain the state vector of the quadruped robot at the current moment; The current state vector is input into the trained robot hierarchical controller to obtain the torque commands for each joint motor at the current moment. The robot hierarchical controller includes: a high-level reinforcement learning strategy module for obtaining the current action of the quadruped robot based on the current state vector; a syntax interpretation module for mapping the current action of the quadruped robot into a motion safety parameter vector; a central pattern generator compilation module for generating the desired motion trajectory of each leg based on the motion safety parameter vector; and a low-level control module for obtaining the torque commands for each joint corresponding to each leg based on the desired motion trajectory of each leg. The torque commands of each joint motor at the current moment are input to the motor drivers of each joint of the quadruped robot to drive the quadruped robot to perform the corresponding actions.

2. The method according to claim 1, characterized in that, The syntax interpretation module performs the following steps to map the current action of the quadruped robot into a motion safety parameter vector: Based on the current state vector and the preset constraints corresponding to each action, determine whether the action at the current moment is safe; When the action is safe, the action at the current moment is mapped to the original motion parameter vector based on the preset action execution parameter table; When the original motion parameter vector is within the preset safety subspace range corresponding to the action, the original motion parameter vector is used as the motion safety parameter vector; When the original motion parameter vector is not within the safety subspace corresponding to the action, the original motion parameter vector is projected to the nearest point in the safety subspace to obtain the motion safety parameter vector. When the action is unsafe, the preset motion parameter vector is used as the motion safety parameter vector.

3. The method according to claim 2, characterized in that, The state vector includes the quadruped robot's torso roll angle, torso pitch angle, torso roll angular velocity, torso pitch angular velocity, contact state of each foot end, and rotational speed of each foot end wheel assembly; The step of determining whether the action at the current moment is safe based on the current state vector and the preset constraints corresponding to each action includes: Determine whether the maximum absolute value of the torso roll angle and the torso pitch angle is greater than a preset fuselage tilt angle threshold. Determine whether the number of ground-contacting legs of the quadruped robot is greater than 1; Determine whether the maximum value of the rotational speed of each foot wheel set is greater than a preset wheel set rotational speed threshold; Determine whether the maximum value of the angular velocities of each joint of the quadruped robot is greater than a preset joint angular velocity threshold; Determine whether the remaining battery power or terrain slope of the quadruped robot meets the preset action execution conditions at the current moment; If any of the above conditions are not met, the action is deemed unsafe.

4. The method according to claim 2, characterized in that, The motion safety parameter vector includes convergence coefficient, angular frequency, diagonal coupling weight, same-side coupling weight, target amplitude, and phase offset; The safety subspace range is the range of values ​​for each motion safety parameter pre-calibrated based on the physical limits of the quadruped robot.

5. The method according to any one of claims 1-4, characterized in that, The central mode generator compilation module uses the Hopf oscillator model, taking each parameter in the motion safety parameter vector as the coefficient of the Hopf oscillator differential equation system, and obtaining the expected motion trajectory of each joint by solving the differential equation system.

6. The method according to claim 1, characterized in that, The high-level reinforcement learning strategy module includes a multilayer perceptron; The multilayer perceptron is trained through reinforcement learning, including: In the simulation environment, the current state vector of the quadruped robot, the action output by the multilayer perceptron at the current moment, the reward from environmental feedback, and the state vector at the next moment are collected to construct experience data, which is then stored in the experience replay pool. When the experience data in the experience replay pool reaches a preset amount, several pieces of experience data are randomly selected as training data. Based on the training data, with the goal of maximizing the cumulative reward, the parameters of the multilayer perceptron are updated through backpropagation until the model converges or reaches a preset number of training rounds, thus obtaining a trained multilayer perceptron.

7. The method according to claim 6, characterized in that, The cumulative reward is obtained by weighted summation of task rewards and safety violation penalties; wherein the weight of the safety violation penalty is dynamically adjusted during training. The task reward is obtained by weighted summation of speed tracking reward, fuselage altitude reward, attitude stability reward, energy efficiency reward, and motion smoothness reward; The safety violation penalty includes a first penalty item and a second penalty item; wherein, the first penalty item is a preset penalty value set when the number of the quadruped robot's ground-touching legs is less than 2 at any time; the second penalty item is calculated based on the current state vector, action, and preset constraints of the training data.

8. The method according to claim 7, characterized in that, The weights of the safety violation penalties are adjusted using a dual-increase method. Specifically, when the proportion of safety violation samples in a batch of training data exceeds a preset maximum safety violation proportion constraint value, the weights are increased based on the proportion of safety violation samples and the maximum safety violation proportion constraint value; when the proportion of safety violation samples is lower than the maximum safety violation proportion constraint value, the weights are decreased based on the proportion of safety violation samples and the maximum safety violation proportion constraint value.

9. The method according to claim 8, characterized in that, When updating the parameters of the multilayer perceptron, the following policy gradient calculation formula is used: ； in, Represents the gradient operator; The parameters of the multilayer sensor are represented; Represents a policy network for reinforcement learning; The action at time t is indicated; The state vector at time t represents the state vector at time t; The dominant function at time t; Let represent the gradient contribution term of the safety violation penalty at time t; This indicates the weight of the penalty for safety violations.

10. The method according to any one of claims 7-9, characterized in that, The speed tracking reward is used to reward the quadruped robot for its actual movement speed keeping up with the command speed. The body height reward is used to reward the quadruped robot for maintaining its body height at a preset height; The attitude stability reward is used to reward the quadruped robot for having roll and pitch angles close to zero. The energy efficiency reward is used to reward the low power consumption of each joint of the quadruped robot; The action smoothing reward is used to reward actions that are consistent at adjacent time points.

Images