Four-legged robot motion control method and device based on contrast learning and storage medium

Abstract

The present application relates to a quadruped robot motion control method and device based on contrastive learning and a storage medium, wherein in a self-guided latent representation learning network, historical observation data is used to output a predicted representation and a target representation by using an online network and a target network respectively, a first loss function based on negative cosine similarity is used to optimize the representation consistency of the predicted representation and the target representation, and latent features and speed prediction values are extracted; a near-end policy optimization network includes a critic network with external information as input; in the near-end policy optimization network, real-time observation data after removing linear velocity, latent features and speed prediction values are output as joint control instructions through a three-layer multilayer perceptron. The present application does not need to rely on external sensors such as vision or complete environment privilege information, and can realize stable control only by using proprioceptive input (IMU, encoder data), has lower hardware cost and stronger platform adaptability, and can be directly deployed on a robot platform without external sensors.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of motion control, in particular to a quadruped robot motion control method based on contrast learning, a device and a storage medium. BACKGROUND

[0002] In recent years, quadruped robots have made significant technological breakthroughs in mobility and payload capacity, enabling agile movement in complex terrains. This has enabled quadruped robots to exhibit broad application potential in areas such as material transportation, field search and rescue, and special operations. Currently, the research on quadruped robot control is mainly divided into two categories: traditional model-based methods and emerging data-driven methods.

[0003] Model-based control methods include model predictive control (MPC), whole-body control (WBC), and contact-implicit motion control. Model-based control methods have been theoretically analyzed and experimentally verified in terms of improving motion agility and stability, but their performance is heavily dependent on the accuracy of system modeling, and require complete prior knowledge of the environment, making them less adaptable in complex and variable terrain scenarios.

[0004] Data-driven learning-based control methods use large-scale simulation datasets for training, demonstrating superior motion potential to traditional methods in real-world environments. They not only achieve basic motion control, but also complete complex skills such as object manipulation and high-difficulty parkour, with better environmental adaptability and task generalization ability. However, such methods have the following shortcomings:

[0005] Perception bottleneck: In order to achieve high agility and complex terrain adaptation, quadruped robots must rely on accurate environmental state perception; however, existing sensing systems have limitations. Body sensing sensors (such as inertial measurement units IMU and joint encoders) can only provide limited information such as base angular velocity, joint angle / speed; external environment sensors are easily disturbed by noise and incomplete observation, severely weakening the robustness of the motion control system.

[0006] Additional sensor dependence and training efficiency issues: Some solutions rely on vision systems and other external sensors to obtain environmental information, not only increasing the difficulty of adapting different robot platforms, but also significantly reducing training efficiency due to high-dimensional visual data processing, making it difficult to achieve universal deployment. SUMMARY

[0007] The present application provides a quadruped robot motion control method based on contrast learning, a device and a storage medium, aiming to at least solve one of the technical problems existing in the prior art.

[0008] The technical solution of the present application is a quadruped robot motion control method based on contrast learning, comprising the following steps:

[0009] S100, collect the proprioceptive data of the robot, form real-time observation data, splice the real-time observation data into historical observation data, and input the historical observation data into a self-guided latent representation learning network;

[0010] S200, in the self-guided latent representation learning network, the historical observation data is used to output a prediction representation and a target representation respectively by using an online network and a target network, a representation consistency of the prediction representation and the target representation is optimized by using a first loss function based on a negative cosine similarity, and a latent feature and a speed prediction value are extracted;

[0011] S300, input the output parameters of the self-guided latent representation learning network into a proximal policy optimization network; wherein the proximal policy optimization network comprises a critic network with external information as input;

[0012] S400, in the proximal policy optimization network, the real-time observation data after removing the linear velocity, the latent feature and the speed prediction value are outputted as joint control instructions by a three-layer multilayer perceptron to realize joint control of four legs of the quadruped robot.

[0013] According to some embodiments of the present application, the step S200 comprises:

[0014] S201, in the self-guided latent representation learning network, data augmentation is performed on the historical observation data;

[0015] S202, for the data-augmented historical observation data, the prediction representation is outputted by using the online network through a first encoder, an online projection head and a prediction head, and the target representation is outputted by using the target network through a second encoder and a target projection head;

[0016] S203, the cosine similarity is set as the first loss function of the self-guided latent representation learning network, the representation consistency of the prediction representation and the target representation is optimized by using the first loss function, and the latent feature is outputted by using a first latent layer and the speed prediction value is outputted by using a speed estimation layer;

[0017] S204, the latent feature and the speed prediction value are inputted into the proximal policy optimization network.

[0018] According to some embodiments of the present application, in the step S203, the cosine similarity is set as the first loss function of the self-guided latent representation learning network, and the representation consistency of the prediction representation and the target representation is optimized by using the first loss function, comprising:

[0019] The prediction representation and the target representation are normalized, and the normalization is represented as follows:

[0020] ,

[0021] ,

[0022] wherein, is a normalized predicted representation, is the predicted representation, denotes a Euclidean norm of for feature normalization, is a normalized target representation, is the target representation, denotes a Euclidean norm of for feature normalization;

[0023] The cosine similarity is set as the first loss function of the self-guided latent representation learning network, and the representation consistency of the normalized predicted representation and the normalized target representation is optimized using the first loss function.

[0024] According to some embodiments of the present application, the first loss function of the self-guided latent representation learning network is represented as follows:

[0025] ,

[0026] wherein, is the first loss function of the self-guided latent representation learning network, is the normalized predicted representation, is the normalized target representation.

[0027] According to some embodiments of the present application, the total loss function when the proximal policy optimization network and the self-guided latent representation learning network are trained is represented as follows:

[0028] ,

[0029] wherein, is the total loss function when the proximal policy optimization network and the self-guided latent representation learning network are trained, is the first loss function of the self-guided latent representation learning network, is a second loss function of a proximal policy optimization network, is a third loss function, is a fourth loss function, is a fifth loss function.

[0030] According to some embodiments of the present application, the second loss function of the proximal policy optimization network is represented as follows:

[0031] ,

[0032] wherein, is a second loss function of the proximal policy optimization network, denotes expectation, is a current policy a probability of outputting an action in a current state, is an old policy a probability of outputting an action in a current state, is a clipping function, , denotes a state, an action at a current time, , denotes a reward-related advantage function.

[0033] According to some embodiments of the present application, the fourth loss function is represented as follows:

[0034] ,

[0035] wherein, is the fourth loss function, denotes expectation, denotes a reward-related state value function, denotes a last reward function, denotes a state at a current time.

[0036] According to some embodiments of the present application, the fifth loss function is represented as follows:

[0037] ,

[0038] wherein, is the fifth loss function, denotes expectation, denotes a cost-related state value function, denotes a kth cost function, denotes a state at a current time.

[0039] The technical solution of the present application also relates to a computer device comprising a memory and a processor, wherein the processor implements the method as described above when executing a computer program stored in the memory.

[0040] The technical solution of the present application also relates to a computer readable storage medium, which stores program instructions, wherein the program instructions are executed by a processor to implement the method as described above.

[0041] The beneficial effects of the present application include: collecting the proprioceptive data of the robot, forming real-time observation data, splicing the real-time observation data into historical observation data, inputting into the self-guided latent representation learning network, in the self-guided latent representation learning network, using the online network and the target network to output the prediction representation and the target representation respectively for the historical observation data, using the first loss function based on the negative cosine similarity to optimize the representation consistency of the prediction representation and the target representation, and extracting the latent feature and the speed prediction value, inputting the output parameters of the self-guided latent representation learning network into the proximal policy optimization network; wherein the proximal policy optimization network includes a critic network with external information as input, in the proximal policy optimization network, the real-time observation data after removing the linear velocity, the latent feature and the speed prediction value are output through a three-layer multilayer perceptron to realize the joint control instruction of the four legs of the quadruped robot.

[0042] The present application does not need to rely on external sensors such as vision or complete environment privilege information, and can realize stable control only by proprioceptive input (IMU, encoder data), has lower hardware cost and stronger platform adaptability, and can be directly deployed on a robot platform without external sensors. The self-guided latent representation learning algorithm extracts a hybrid latent embedding that fuses motion speed and stability from a historical observation sequence, constructs a robust latent representation space that fuses motion speed and stability, reduces the interference of noise on state estimation, and improves the motion stability and speed tracking accuracy of the robot in complex terrain.

[0043] In addition, additional aspects and advantages of the present application will be given in part in the following description, part will become apparent from the following description, or will be understood by practicing the present application. BRIEF DESCRIPTION OF DRAWINGS

[0044] Figure 1 is an optional flowchart of the quadruped robot motion control method of contrast learning in the embodiment of the present application.

[0045] Figure 2 is a schematic diagram of the proximal policy optimization network and the self-guided latent representation learning network in the embodiment of the present application.

[0046] Figure 3 is a schematic diagram of the self-guided latent representation learning network in the embodiment of the present application.

[0047] Figure 4 is a schematic diagram of four groups of contrast experiment data of the ablation experiment in the embodiment of the present application.

[0048] Figure 5 is a comparison tracking error diagram of linear velocity and angular velocity under two instruction conditions in the embodiment of the present application.

[0049] Figure 6This is a comparison chart of terrain level curves in an embodiment of the present invention.

[0050] Figure 7 This is a survival rate comparison analysis chart in an embodiment of the present invention. Detailed Implementation

[0051] The following will provide a clear and complete description of the concept, specific structure, and technical effects of the present invention in conjunction with the embodiments and accompanying drawings, so as to fully understand the purpose, solution, and effects of the present invention. It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other.

[0052] It should be noted that, unless otherwise specified, when a feature is referred to as "fixed" or "connected" to another feature, it can be directly fixed or connected to the other feature, or indirectly fixed or connected to the other feature. Furthermore, the descriptions of "upper," "lower," "left," "right," "top," and "bottom" used in this invention are only relative to the relative positional relationships of the various components of the invention in the accompanying drawings.

[0053] Furthermore, unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in this specification is for the purpose of describing particular embodiments only and not for limiting the invention. The term "and / or" as used herein includes any combination of one or more of the associated listed items.

[0054] It should be understood that although the terms first, second, third, etc., may be used to describe various elements in this invention, these elements should not be limited to these terms. These terms are only used to distinguish elements of the same type from each other. For example, a first element may also be referred to as a second element without departing from the scope of this invention, and similarly, a second element may also be referred to as a first element.

[0055] Reference Figures 1 to 3 In some embodiments, the comparative learning method for quadruped robot motion control of the present invention includes at least the following steps:

[0056] S100: Collect the robot's proprioceptive data to form real-time observation data, stitch the real-time observation data into historical observation data, and input it into the self-guided latent representation learning network;

[0057] S200. In the self-guided latent representation learning network, for historical observation data, the online network and the target network output the predicted representation and the target representation respectively. The first loss function based on negative cosine similarity is used to optimize the representation consistency between the predicted representation and the target representation, and latent features and velocity prediction values ​​are extracted.

[0058] S300, Input the output parameters of the self-guided latent representation learning network into the proximal policy optimization network; wherein, the proximal policy optimization network includes a critic network that takes external information as input;

[0059] In the S400, the real-time observation data after removing linear velocity, latent features, and velocity prediction values ​​are used to output joint control commands through a three-layer multilayer perceptron to achieve joint control of the four legs of the quadruped robot.

[0060] As can be seen, this invention does not rely on external sensors such as vision or complete privileged environmental information; stable control can be achieved with only proprioceptive input (IMU, encoder data). It has lower hardware costs, stronger platform adaptability, and can be directly deployed on robot platforms without external sensors. By extracting a hybrid latent embedding that integrates motion velocity and stability from historical observation sequences using a self-guided latent representation learning algorithm, a robust latent representation space integrating motion velocity and stability is constructed. This reduces noise interference in state estimation and improves the robot's motion stability and velocity tracking accuracy in complex terrain.

[0061] Understandably, this invention relies solely on proprioceptive input for "blind walking" motion control (without requiring additional external sensors or privileged environmental information), reducing dependence on hardware sensors and improving platform versatility. It constructs a robust latent representation space that integrates motion speed and stability, reducing noise interference with state estimation and improving the robot's motion stability and velocity tracking accuracy in complex terrain. By combining constrained reinforcement learning with domain randomization and curriculum learning strategies, it accelerates training convergence, narrows the gap between simulation and reality, ensures the robot achieves agile movement within safe physical constraints, and possesses excellent generalization capabilities, allowing direct deployment in complex real-world scenarios where it has not been trained.

[0062] The specific implementation method of step S100 is as follows:

[0063] The method in this embodiment of the invention collects the robot's proprioceptive data to form real-time observation data, splices the real-time observation data into historical observation data, and inputs it into a self-guided latent representation learning network.

[0064] Specifically, the proprioception data acquisition module collects the robot's proprioception data, including base angular velocity (3D), projected gravity vector (3D), velocity command (3D), joint position (12D), joint velocity (12D), and previous action (12D), forming 45-dimensional real-time observation data. This data is then stitched together in a 5-frame sequence to form 225-dimensional historical observation data, which is then input into the BYOL (Bootstrap Your Own Latent) network.

[0065] Specific implementation method of step S200:

[0066] In the self-guided latent representation learning network, the method of this invention outputs a predicted representation and a target representation using an online network and a target network, respectively, for historical observation data. The consistency between the predicted representation and the target representation is optimized using a first loss function based on negative cosine similarity, and latent features and velocity prediction values ​​are extracted.

[0067] In some embodiments, step S200 includes:

[0068] S201. In a self-guided latent representation learning network, data augmentation is performed on historical observation data;

[0069] S202. For the augmented historical observation data, use an online network to output a predicted representation through a first encoder, an online projection head, and a prediction head. For the augmented historical observation data, use a target network to output a target representation through a second encoder and a target projection head.

[0070] S203. Set cosine similarity as the first loss function of the self-guided latent representation learning network, use the first loss function to optimize the representation consistency between the predicted representation and the target representation, and use the first latent layer to output latent features and use the velocity estimation layer to output velocity prediction values.

[0071] S204. Input the latent features and velocity predictions into the proximal policy optimization network.

[0072] In some embodiments, in step S203, cosine similarity is set as the first loss function of the self-guided latent representation learning network, and the representation consistency between the predicted representation and the target representation is optimized using the first loss function, including:

[0073] The prediction and target representations are normalized as follows:

[0074] ,

[0075] ,

[0076] In the formula, This is the normalized prediction representation. The prediction is represented as follows. express The Euclidean norm is used for feature normalization. This is the normalized target representation. Let the target be represented. express The Euclidean norm is used for feature normalization;

[0077] Cosine similarity is set as the first loss function of the self-guided latent representation learning network, and the first loss function is used to optimize the representation consistency between the normalized predicted representation and the normalized target representation.

[0078] In some embodiments, the first loss function of a self-guided latent representation learning network is expressed as follows:

[0079] ,

[0080] In the formula, Let be the first loss function of the self-guided latent representation learning network. This is the normalized prediction representation. This is the normalized target representation.

[0081] In a specific embodiment, the update process of the self-guided latent representation learning network includes:

[0082] The first weight of the online network is updated using gradients, and the second weight of the target network is updated using the updated first weight, as shown below:

[0083] ,

[0084] ,

[0085] In the formula, As the first weight in the online network, This refers to the Adam optimizer. Let be the first loss function of the self-guided latent representation learning network. This represents the gradient of the first loss function. The learning rate for the network optimizer. This represents the moving average update hyperparameter of the target network. It is the second weight of the target network.

[0086] It should be noted that the first encoder is composed of an online encoder and a first latent layer, and the second encoder is composed of a target encoder and a second latent layer.

[0087] Specifically, see Figure 2 The policy network processes some observations, as well as the outputs of the latent and velocity layers, and performs comparative optimization to match the robot's subsequent states. The entire... Figure 2 The framework uses BYOL and PPO for joint training.

[0088] Specifically, the BYOL contrastive learning module employs a dual-network architecture (online network + target network) to augment historical observation data (selecting a combination from Gaussian noise injection and data truncation). The online network outputs a predicted representation through a first encoder, an online projector, and a prediction head, while the target network outputs a target representation through a second encoder and a target projector. The consistency between the two representations is optimized using negative cosine similarity as the loss function, and 16-dimensional latent features (latest layer output) and 3-dimensional velocity features (velocity estimation layer output) are extracted.

[0089] The Bootstrap Your Own Latent (BYOL) method proposed in this invention effectively improves the flexibility and robustness of robot training strategies while ensuring stable optimization of the learning-based framework. Furthermore, this invention introduces a constrained reward learning framework for training the motion controller of a quadruped robot, significantly reducing the reliance on manually designed reward functions.

[0090] The objective of this invention is to develop a blind-walking motion controller for legged robots, enabling robust movement in challenging unstructured terrain. This controller relies solely on proprioceptive sensor data (such as joint position and body posture) to adjust base movement and foot placement. The controller receives velocity commands specifying desired forward speed, lateral speed, and turning angular velocity, and must be able to track these commands even when the terrain undergoes unexpected changes.

[0091] This invention employs an asymmetric actor-critic architecture. The policy network (actor) uses only historical ontological observation data as input, while the critic network (critic) incorporates additional external information such as terrain geometry and other environmental data to enhance the policy's adaptability.

[0092] See Figure 3 The self-guided latent representation learning network employs a dual-network architecture, consisting of an online network and a target network. The online network is assigned the first weight. The first encoder, which contains a multi-layer perceptron (MLP) structure, is defined as a parameter. The first encoder can output the latent representation of the input sample. In this invention, This represents the combination of an inline encoder and the first latent layer. Inline projection head. and prediction head This is an MLP with the same output dimension. The target network has the same architecture as the online network, but does not include a prediction head. Furthermore, the second weight is frozen through a gradient-free operation. .

[0093] The policy network processes some observations, as well as the outputs of the latent and velocity layers, and performs comparative optimization to match the robot's subsequent states. The entire framework is jointly trained using BYOL and PPO.

[0094] Input x is the collected historical information of proprioception. BYOL uses data augmentation techniques to transform historical observational changes into... and The online network starts from the first processed sample The output is represented as The target network starts from the second processed sample. The output is .

[0095] BYOL utilizes and To calculate the network training loss. This requires... and conduct Normalization, therefore, is defined and ,in See Figure 3 The result is a prediction from an online network. yes The Euclidean norm (L2 norm) is used for feature normalization. It is the projection output of the target network. yes The Euclidean norm (L2 norm) is used for feature normalization. and This is the result after L2 normalization. Then, the loss function of the BYOL network is defined as the normalized predicted value. and The negative cosine similarity between them is shown in the following formula:

[0096] ,

[0097] The losses s and s' need to be input into the online network and the target network respectively for computation. The BYOL update process can be summarized in two steps: the first step is to update the first weight using gradients. The second step is to use the updated version. To update the second weight This update method can be described by the following two formulas:

[0098] ,

[0099] ,

[0100] in, The learning rate for the network optimizer. This represents the moving average update hyperparameter of the target network, where the learning rate is the same as the learning rate of PPO.

[0101] BYOL network relies on minimization This reduces the distance between the representation distributions of inputs s and s', thereby learning the representation of the input training data samples.

[0102] In another possible implementation, BYOL utilizes and To calculate the network training loss. This requires... and conduct Normalization, therefore, is defined and Then the loss function of the BYOL network is defined as the normalized predicted value. and The negative cosine similarity between them is shown in the following formula:

[0103] ,

[0104] in, Indicates the target projection head.

[0105] Specific implementation method of step S300:

[0106] The method of this invention inputs the output parameters of a self-guided latent representation learning network into a proximal policy optimization network; wherein, the proximal policy optimization network includes a critic network that takes external information as input.

[0107] In another possible implementation, real-time observation data is input into a near-end policy optimization network, where the linear velocity of the real-time observation data is removed to obtain real-time observation data after removing the linear velocity.

[0108] Specific implementation method of step S400:

[0109] In the method of this invention embodiment, the real-time observation data after removing linear velocity, potential features and velocity prediction values ​​are processed through a three-layer multilayer perceptron to output joint control commands, thereby realizing joint control of the four legs of a quadruped robot.

[0110] In some embodiments, the total loss function during training of the proximal policy optimization network and the self-guided latent representation learning network is expressed as follows:

[0111] ,

[0112] In the formula, The total loss function during training of the proximal policy optimization network and the self-guided latent representation learning network. Let be the first loss function of the self-guided latent representation learning network. To optimize the second loss function of the network for near-end policies, For the third loss function, This is the fourth loss function. This is the fifth loss function.

[0113] In some embodiments, the second loss function of the near-end policy optimization network is expressed as follows:

[0114] ,

[0115] In the formula, The second loss function of the network is optimized for the near-end policy. Expressing expectations, For the current strategy The probability of outputting an action in the current state. For the old strategy The probability of outputting an action in the current state. For the clipping function, , Indicates the current state or action. , This represents the advantage function related to rewards.

[0116] In some embodiments, the fourth loss function is expressed as follows:

[0117] According to some embodiments of the present invention, the fourth loss function is expressed as follows:

[0118] ,

[0119] In the formula, For the fourth loss function, Expressing expectations, This represents the state-value function related to rewards. This represents the final reward function. It indicates the current state.

[0120] In some embodiments, the fifth loss function is expressed as follows:

[0121] ,

[0122] In the formula, For the fifth loss function, Expressing expectations, This represents a cost-related state-value function. Let k be the cost function. It indicates the current state.

[0123] In a specific embodiment, the third loss function is expressed as follows:

[0124] ,

[0125] In the formula, For the third loss function, The sixth loss function, The seventh loss function is ReLU, which represents the activation function.

[0126] In a specific embodiment, the sixth loss function is expressed as follows:

[0127] ,

[0128] In the formula, The sixth loss function, Expressing expectations, For the current strategy The probability of outputting an action in the current state. For the old strategy The probability of outputting an action in the current state. For cost-related advantage functions, For the clipping function, , Indicates the current state or action. .

[0129] In a specific embodiment, the seventh loss function is expressed as follows:

[0130] ,

[0131] In the formula, The seventh loss function, Let T represent expectation, and T represent time. Let d represent the discount factor raised to the power of t, and d represent the threshold of the k-th constraint. A general term, Let t be the instantaneous cost at step t.

[0132] In a specific embodiment, the quadruped robot motion control method based on comparative learning further includes the following steps:

[0133] 282-dimensional privileged observation data (including extended body state, terrain features, and private variables) are collected and input into a proximal policy optimization (PPO) network. In the PPO network, the privileged observation data is output as state value through a 3-layer multi-layer perceptron (MLP). The state value is used to calculate the advantage function and value loss of the PPO network to assist in policy network optimization.

[0134] Specifically, the PPO policy network module takes "real-time observation data (excluding linear velocity) + latent features output by BYOL + velocity features" as input (64 dimensions in total), and outputs 12-dimensional joint control commands (corresponding to the joint control of the four legs of the quadruped robot) through 3 layers of MLP (hidden layer dimensions 512, 256, 128), realizing the mapping from sensor input to control commands.

[0135] PPO Value Network Module: Taking 282-dimensional privileged observation data (including extended body state, terrain features, and private variables) as input, it outputs state value through a 3-layer MLP to calculate the advantage function and value loss of PPO, and assists in policy network optimization.

[0136] Specifically, module connections and data flow:

[0137] Proprioception data acquisition module → BYOL contrastive learning module: Input 5 frames of historical proprioception data (225 dimensions) into the BYOL contrastive learning module for data augmentation and extraction of latent features and velocity features.

[0138] BYOL Contrastive Learning Module → PPO Policy Network Module: The 16-dimensional latent features and 3-dimensional velocity features, combined with real-time proprioceptive data after removing linear velocity, are input into the PPO Policy Network Module as the basis for generating control commands.

[0139] PPO Strategy Network Module → Robot Actuator: Outputs 12-dimensional joint control commands to drive robot movement.

[0140] Environment and Robot State → PPO Value Network Module: Collects privileged observation data and inputs it into the value network, calculates the value loss, feeds it back to the PPO optimizer, and co-optimizes the policy network parameters with the BYOL loss.

[0141] Using "PPO loss (including policy loss and value loss) + BYOL loss" as the total loss function, the parameters of the policy network and the online BYOL network are optimized through gradient descent, while the parameters of the target BYOL network are updated using exponential moving average (EMA), thus achieving end-to-end training of the policy.

[0142] Understandably, this invention proposes an asymmetric actor-critic architecture based on self-guided latent (BYOL) contrastive learning. The policy network relies solely on proprioceptive input (such as joint position, body posture, etc.), while the critic network combines external state information (such as terrain geometry parameters) to achieve end-to-end mapping from raw sensor input to motion control commands, without the need for complex hierarchical design.

[0143] By using the BYOL algorithm to extract hybrid latent embeddings that integrate motion velocity and stability from historical observation sequences, a highly robust latent representation space is constructed to capture key motion features.

[0144] The Constrained RL (Constrained Reinforcement Learning) framework is introduced, which combines multiple loss functions such as PPO (Proximal Policy Optimization) loss, constraint violation loss, and value loss to form a total loss function. This ensures that the robot's motion performance is guaranteed while its operation conforms to physical constraints such as joint angles, torque, and speed.

[0145] In the BYOL training framework, a combination of two of the four data augmentation strategies—Gaussian noise injection, data truncation, geometric scaling, and downsampling—is used to process the input historical observation data, thereby improving the model's robustness and representation learning ability.

[0146] This invention can be widely applied to scenarios where robots need to achieve stable and agile movement in complex, unstructured terrain (such as slopes, stairs, gravel, etc.), such as field search and rescue, and material transportation.

[0147] Specifically, constraint reinforcement learning:

[0148] In the training process of Reinforcement Learning (RL), a reward function is designed to maximize the sum of expected rewards through trial and error by the agent. However, in real-world robot applications, complete freedom is impossible. Therefore, constraints are set to guide the robot's learning and training towards an ideal direction. The traditional RL training process can be described as a Partially Observed Markov Decision Process (POMDP), using six tuples... To present. Among them, It is the set of all states of the environment. It is a subset of the state space, containing only environmental information observable by the agent. A complete set of robot-performable actions Assign a scalar reward to each state-action pair. The probability distribution of state transitions after an action is performed. The ultimate goal is to maximize cumulative reward to optimize the strategy. It is a discount factor. In the current strategy The expectations below It is the goal of reinforcement learning, that is, the expectation of the reward function. It is the discount factor raised to the power of t. , , This represents the current state, action, and next state, where T represents time.

[0149] ,

[0150] Constrained reinforcement learning can be described as a constrained Markov decision process (CMDP), thus defining the constrained RL problem. A CMDP is defined as follows: , ,yes The cost function, This refers to costs ranging from 1 to K. The set of real numbers, Let k be the cost function. If the initial state distribution is given, then the expected discounted return of the cost function is:

[0151] ,

[0152] Therefore, the problem of overall constraints can be expressed as:

[0153] ,

[0154] It is to maximize the expected value. It is the threshold of the k-th constraint.

[0155] Therefore, for constrained reinforcement learning, the advantage function is calculated as follows:

[0156] ,

[0157] in, For cost-related advantage functions, For cost-related action value functions, For cost-related state value functions,

[0158] ,

[0159] in, Let t be the instantaneous cost at step t. Let be the cost-related state value function for the state at time t+1.

[0160] The reward function for constrained RL can be divided into two parts. The first part is the loss function of PPO, as follows:

[0161] ,

[0162] in, The loss function for the cost-advantage-based PPO strategy is... This represents the probability of the current policy outputting an action in the current state. This represents the probability of the old strategy outputting an action in the current state. For the clipping function, , For cost-related advantage functions, It expresses expectation.

[0163] The second item is the penalty for breach of the constraint, as follows:

[0164] ,

[0165] d is the one above. The collective term for these two items, the combined loss function is:

[0166] .

[0167] Specifically, data augmentation:

[0168] One of the core aspects of BYOL's training process is data augmentation. Figure 3 As can be seen, data augmentation is required for the input historical observations. In the experimental design, two data augmentation modes need to be selected. A total of four data augmentation modes were designed: the first is to add Gaussian noise, the second is to truncate the data, the third is to geometrically scale the data, and the last is to downsample the data.

[0169] Specifically, training:

[0170] During the training of Unitree go2, contrastive learning and constraint reinforcement learning were combined to enable it to walk on challenging terrains. Since constraints do not replace all reward functions, the first term of the loss term is the PPO loss term:

[0171] ,

[0172] It also includes the loss term for the value and the loss term for the constraint value, as follows:

[0173] ,

[0174] ,

[0175] in, This represents the reward state value network. Let the final reward function be the sum of the loss functions throughout the entire training process:

[0176] ,

[0177] The entire flowchart is as follows Figure 3 As shown, the entire process is divided into two parts: PPO and BYOL. PPO consists of a critic, a cost value, and a policy. The policy input includes three parts: the current output after removing linear velocity observations, the output of the latent layer, and the output of the vel layer. BYOL consists of an online network and a target network. The loss of BYOL is calculated... When doing this, it is necessary to first perform data augmentation on historical observations and then select two data augmentation methods to calculate the loss. See [link to relevant documentation]. Figure 2 The critic network is the output of the critic network, the cost value is the output of the constraint network, external information is input into the critic network and the constraint network, the policy is the output of the policy network, and the ontological perception observation data is input into BYOL, processed and then input into the policy network.

[0178] Specifically, states and actions:

[0179] For unitree Go2 (an educational quadruped robot), the dimension of movement is four legs, that is... For observations, in a policy, the only observation is proprioception. This includes 3D angular velocity, 3D projected gravity, 3D command, 12D joint position, 12D joint velocity, and the action from the previous time step. The input to the Byol network is observations from five time steps, including the current time step and the previous four. For the inputs to critic and cost, they are privileged observations. This includes 48-dimensional observations with linear velocity, 187-dimensional terrain information, and 47-dimensional private variables, including 4-dimensional foot contact, 1-dimensional delay, 4-dimensional mass parameters, 1-dimensional coefficient of friction, 1-dimensional coefficient of restitution, 12-dimensional motor strength, 12-dimensional kp, and 12-dimensional kd.

[0180] Specifically, the Policy structure and parameters are:

[0181] All modules used in this invention are MLPs, and detailed parameter settings are shown in Table 1.

[0182] Table 1 Policy Parameters:

[0183]

[0184] Specifically, the reward function:

[0185] In the experimental setup, a complete reward mechanism was designed to ensure the Go2 robot could stably traverse complex terrain. This mechanism comprises three core components: a tracking reward module to ensure the robot accurately tracks commanded velocities (linear and angular velocities); a stability penalty module to optimize the robot's posture balance, altitude maintenance, and energy efficiency; and a state reward module specifically handling special states such as foot contact with the ground to enhance the overall robustness of the system. Detailed reward function settings are shown in Table 2.

[0186] Table 2 Reward Function:

[0187]

[0188] Desired linear velocity vector The robot's actual linear velocity vector, Desired yaw rate, The robot's actual yaw rate, z-linear velocity, The robot's actual roll angle, Gravity projection Robot joint acceleration vectors Robot joint torque vector, Robot joint velocity vectors The desired height of the robot's base (body). The actual height of the robot base Desired height of the toe (ball of the foot) The actual height of the foot The robot motion vector at time step t. The robot motion vector at time step t-1. The robot motion vector at time step t-2. Filter value for foot-to-ground contact state. Collision count counter.

[0189] Specifically, constraints:

[0190] In addition to setting the above reward function, constraints were also applied to constrain the joint angle, torque, and speed, as detailed in Table 3.

[0191] Table 3 Constraint Settings:

[0192]

[0193] Indicates the lowest angle of joint rotation. Indicates the maximum rotation angle of the joint. Indicates joint torque limitation. For joint speed limitation. It is the joint angle of the current i-th joint. It is the torque of the current i-th joint. It is the torque of the current i-th joint, where i represents the joint and N represents the number of joints = 12.

[0194] Specifically, domain randomization:

[0195] To enhance the robustness of the control strategy and achieve efficient simulation-to-physical transfer, a comprehensive dynamic domain randomization strategy was implemented during training. This strategy includes: random variations in the robot's mass and center of mass position; dynamic adjustment of the ground friction coefficient (with the coefficient of restitution remaining constant); random disturbances to the drive system motor gain and joint PD control parameters; the introduction of control system time delay; and the application of periodic external force interference and random disturbances. This multi-dimensional parameter randomization method significantly enhances the strategy's adaptability to various physical environments. Specific domain randomization configurations are detailed in Table 4.

[0196] Table 4: Domain Randomization

[0197]

[0198] It should be understood that sensor dependence is reduced: existing technologies rely on external sensors such as vision or complete privileged environmental information, while this invention only requires proprioceptive input (IMU, encoder data) to achieve stable control, resulting in lower hardware costs, stronger platform adaptability, and direct deployment on robot platforms without external sensors.

[0199] Enhanced Simulation-to-Reality Transfer Capability: Existing "teacher-student" methods suffer from performance loss. This invention combines BYOL contrastive learning with domain randomization to achieve "zero-sample" simulation-to-reality transfer (without additional real-world data fine-tuning), enabling stable movement even in untrained real-world terrains (such as grass, gravel, and stairs).

[0200] Training efficiency and generalization ability optimization: Compared with HIM (slow iteration speed) and SLR (failed to converge after 10,000 iterations), this invention converges completely within 10,000 iterations, and its speed in increasing terrain difficulty to level 6 (before 1,000 iterations) far exceeds that of existing methods. At the same time, its survival rate in complex terrains such as stairs and slopes (e.g., 100% survival rate when the difficulty of climbing stairs is level 9) is higher than that of HIM and SLR.

[0201] Co-optimization of motion stability and velocity: Existing methods mostly optimize velocity or stability separately. This invention extracts and integrates the potential features of velocity and stability through BYOL. The linear velocity tracking error (when vx=2m / s) is reduced by 15%-20% compared with MoB and SLR, and the angular velocity tracking error is reduced by 10%-15%. At the same time, it can quickly recover balance under external disturbances (such as external force impact) (recovery time <0.5s).

[0202] Specifically, the experiment:

[0203] The performance of the proposed method was evaluated in both simulated and real-world environments through ablation and comparative experiments.

[0204] The environment was set up using the IsaacGym simulator accelerated by an NVIDIA RTX 4080 Super GPU to perform multi-terrain parallel training on 4096 Go2 quadruped robots. As training progressed, the policy performance showed a steady improvement trend. Each training round was limited to 20 seconds (corresponding to 1000 control steps at a control frequency of 50Hz), and the round terminated when the time limit was reached or the robot fell. The joint PD controller parameters were configured with a proportional gain of 40, a derivative gain of 1.0, and an observation history length of 5 steps. Other training hyperparameters are shown in Table 5.

[0205] Table 5 Training parameters:

[0206]

[0207] First, ablation experiment:

[0208] A systematic ablation study was conducted through simulation experiments. The evaluation metrics used were the Normalized Linear Velocity Tracking Score (NLTS) and the Normalized Angular Velocity Tracking Score (NATS). The experiments covered five terrain types: smooth-slope, rough-slope, stairs-up, stairs-down, and discrete terrain, with distribution ratios of [0.1, 0.1, 0.35, 0.25, 0.2]. The formulas for calculating NLTS and NATS are as follows:

[0209] ,

[0210] in The target value for linear velocity. This represents the target value for angular velocity.

[0211] Ablation experiments include the following four configurations:

[0212] Baseline model: It does not use the BYOL framework, but retains the encoder structure and optimizes it using the PPO algorithm.

[0213] The model without a velocity layer (w / o-vel-layer) sets the velocity signal input to the policy network to zero.

[0214] No-latent-layer model: Sets the latent features input to the policy network to zero.

[0215] All ablation experiment results in this invention are based on data collected from 3000 training iterations, such as... Figure 4 As shown. From Figure 4 The four sets of comparative experimental data yield the following key conclusions: First, the proposed BYOL method exhibits the best performance in all comparative experiments; second, both the latent layer and the velocity layer in the BYOL framework significantly contribute to policy performance, and removing either component leads to performance degradation. Notably, when the velocity layer input is removed, the policy exhibits significant instability, further validating the crucial role of the velocity layer in maintaining policy stability.

[0216] Figure 4 In the study, the learning curves are (a) standardized linear velocity tracking score, (b) standardized angular velocity tracking score, (c) terrain level, and (d) average reward.

[0217] Second, comparative experiments:

[0218] This invention compares the BYOL method with recent state-of-the-art methods in the literature, evaluating seven representative approaches, including MoB, HIM, the Baseline model (where both the policy and critic networks use privileged information), SLR, and the Oracle method (where Oracle represents a policy trained using the complete observation history). The comparative analysis focuses on tracking errors in linear and angular velocities, with quantitative results as follows: Figure 5 As shown.

[0219] Figure 5 This is a comparison of linear velocity and angular velocity tracking error under two different command conditions: (ac) (a) Error, (b) Error and (c) Error; (df) , respectively display (d) Error, (e) Error and (f) error.

[0220] In the comparative experiments, the baseline model demonstrated a significant advantage across all performance metrics because its policy network can acquire complete privileged information. In contrast, the Oracle variant of BYOL (which feeds all historical observations as additional input to the policy network) should theoretically outperform the proposed basic BYOL framework. HIM and SLR, similar to this invention, explore training paradigms that do not rely on privileged information and do not employ teacher-student frameworks; while MoB is a multi-gait training policy, therefore this invention only compares its velocity tracking performance in trot gait. Experimental results are as follows... Figure 5 As shown. The vertical axis values ​​in the graph indicate that when When the training range is exceeded, the tracking error increases. The proposed method shows lower tracking errors than MoB, SLR, and HIM in both linear velocity (x and y directions) and angular velocity (yaw direction). Notably, since the proposed method achieves good convergence within 10,000 training iterations, all comparative experiments were conducted with the same number of training steps to ensure fairness. In contrast, SLR still fails to converge completely after 10,000 iterations, mainly because it completely discards privileged information; while HIM has lower training efficiency, requiring 4-5 seconds per iteration on an NVIDIA 4090 GPU, resulting in a longer training time. From a policy design perspective, both SLR and HIM focus on policy training that does not rely on privileged information, while the proposed method exhibits superior tracking accuracy, validating the effectiveness of the BYOL framework. Furthermore, MoB, as a multi-gait training policy, exhibits relatively high error in the trot gait tracking task. For reference, using a baseline model and Oracle method with complete input information (privileged data or complete historical observation data) should theoretically achieve optimal performance. However, as... Figure 5 As shown in (a), (c), and (f), the proposed BYOL method performs comparably to these two methods. Figure 5 In (c), it even surpassed the baseline model. Figure 5 The proposed method surpasses the Oracle method in (c) and (e), further highlighting its competitiveness.

[0221] By comparing the terrain-level training performance of these methods, Figure 6 It is clearly shown that the proposed BYOL method successfully improved the terrain level to level 6 before the number of iterations reached 1000, outperforming other comparative methods and demonstrating a faster training convergence speed.

[0222] Furthermore, this invention compares and analyzes the survival rates of BYOL, HIM, and SLR methods under different terrains and difficulty levels. All experimental results are evaluated based on a model trained for 10,000 iterations.

[0223] See Figure 7 On slopes and rough terrain, all three methods (BYOL, HIM, and SLR) successfully navigated even at the highest difficulty level. However, on staircase terrain, the HIM method failed to complete upward tasks above level 9 due to the robot's low center of gravity; while the SLR method performed poorly during downward descent, with the robot prone to falling as the difficulty increased. Overall, BYOL achieved the highest survival rate across all terrain types. Due to the high randomness of discrete obstacle terrain—even at the same difficulty level, differences in terrain generation can lead to highly inconsistent experimental results, making stable comparisons difficult—it was not included in the comparative analysis.

[0224] Fourth, deployment in a real-world environment:

[0225] The proposed method was deployed on the Unitree Go2 robot and extensively evaluated in various terrains within a university town. The robot demonstrated stable and agile movement performance during its journey.

[0226] This quadruped robot is able to navigate various terrains, including narrow paths, stair climbing, bush crossings, and forest roads. In all scenarios, the robot successfully completed its terrain traversal tasks without relying on any external privileged information.

[0227] Furthermore, the robot must traverse more diverse terrains along the designated route, including open grasslands, uneven brick paths, long staircases descending, gravel surfaces, steep inclines, and lawn-paved pavers. The robot is able to overcome all these challenges.

[0228] This invention also provides a computer device including a memory and a processor, wherein the processor performs the above-described method when executing a computer program stored in the memory.

[0229] This invention also provides a computer-readable storage medium storing program instructions thereon, which, when executed by a processor, implement the method described above.

[0230] It should be understood that the method steps in the embodiments of the present invention can be implemented or carried out by computer hardware, a combination of hardware and software, or by computer instructions stored in a non-transitory computer-readable storage medium. The method can use standard programming techniques. Each program can be implemented in a high-level procedural or object-oriented programming language to communicate with the computer system. However, if necessary, the program can be implemented in assembly or machine language. In any case, the language can be a compiled or interpreted language. Furthermore, for this purpose, the program can run on a programmed application-specific integrated circuit (ASIC).

[0231] Furthermore, the procedures described herein may be performed in any suitable order unless otherwise indicated herein or otherwise clearly contradicted by the context. The procedures described herein (or variations and / or combinations thereof) may be executed under the control of one or more computer systems configured with executable instructions, and may be implemented by hardware or a combination thereof as code (e.g., executable instructions, one or more computer programs, or one or more applications) that commonly executes on one or more processors. The computer program comprises a plurality of instructions executable by one or more processors.

[0232] Furthermore, the method can be implemented in any suitable type of computing platform, including but not limited to personal computers, minicomputers, mainframes, workstations, networked or distributed computing environments, standalone or integrated computer platforms, or in communication with charged particle tools or other imaging devices, etc. Aspects of the invention can be implemented as machine-readable code stored on a non-transitory storage medium or device, whether removable or integrated into a computing platform, such as a hard disk, optical read and / or write storage medium, RAM, ROM, etc., such that it is readable by a programmable computer, and when the storage medium or device is read by the computer, it can be used to configure and operate the computer to perform the processes described herein. Furthermore, the machine-readable code, or portions thereof, can be transmitted via wired or wireless networks. The invention described herein includes these and other different types of non-transitory computer-readable storage media when such media comprises instructions or programs that implement the steps described above in conjunction with a microprocessor or other data processor. When programmed according to the methods and techniques described in the invention, the invention may also include the computer itself.

[0233] A computer program can be applied to input data to perform the functions described herein, thereby transforming the input data to generate output data stored in non-volatile memory. The output information can also be applied to one or more output devices, such as a display. In a preferred embodiment of the invention, the transformed data represents physical and tangible objects, including specific visual depictions of physical and tangible objects generated on the display.

[0234] The above description is merely a preferred embodiment of the present invention. The present invention is not limited to the above-described embodiments. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention, as long as they achieve the technical effects of the present invention by the same means, should be included within the scope of protection of the present invention. Within the scope of protection of the present invention, the technical solutions and / or implementation methods can have various modifications and variations.

Claims

1. A quadruped robot motion control method of contrast learning, characterized by, The method comprises the following steps: S100, collecting body sensation data of the robot to form real-time observation data, splicing the real-time observation data into historical observation data, and inputting the historical observation data into a self-guided latent representation learning network; the body sensation data comprises base angular velocity, projected gravity vector, speed instruction, joint position, joint speed and previous time action data; S200, in the self-guided latent representation learning network, the historical observation data is used to output a prediction representation and a target representation respectively by using an online network and a target network, a first loss function based on negative cosine similarity is used to optimize the representation consistency of the prediction representation and the target representation, and latent features and speed prediction values are extracted; The step S200 comprises: S201, in the self-guided latent representation learning network, data augmentation is performed on the historical observation data; S202, for the data-augmented historical observation data, the online network is used to output the prediction representation through a first encoder, an online projection head and a prediction head, and the target network is used to output the target representation through a second encoder and a target projection head; S203, the cosine similarity is set as the first loss function of the self-guided latent representation learning network, the representation consistency of the prediction representation and the target representation is optimized by using the first loss function, the latent features are output by using a first latent layer, and the speed prediction values are output by using a speed estimation layer; S204, the latent features and the speed prediction values are input into a proximal policy optimization network; S300, the output parameters of the self-guided latent representation learning network are input into the proximal policy optimization network; wherein the proximal policy optimization network comprises a critic network with external information as input; the external information comprises terrain geometric information and environment data; S400, in the proximal policy optimization network, the real-time observation data after removing linear speed, the latent features and the speed prediction values are output as joint control instructions by a three-layer multilayer perceptron to realize joint control of four legs of the quadruped robot.

2. The contrastive learning quadruped robot motion control method of claim 1, wherein, In the step S203, the cosine similarity is set as the first loss function of the self-guided latent representation learning network, and the representation consistency of the normalized prediction representation and the normalized target representation is optimized by using the first loss function. The first loss function of the self-guided latent representation learning network is represented as follows: ， ， wherein is the normalized predicted representation, is the predicted representation, denotes the Euclidean norm of for feature normalization, is the normalized target representation, is the target representation, denotes the Euclidean norm of for feature normalization; The total loss function when the proximal policy optimization network and the self-guided latent representation learning network are trained is represented as follows:

3. The contrastive learning quadruped robot motion control method according to claim 2, characterized in that, The second loss function of the proximal policy optimization network is represented as follows: ， wherein is the first loss function for the self-guided latent representation learning network, is the normalized predicted representation, is the normalized target representation.

4. The contrastive learning quadruped robot motion control method of claim 1, wherein, The fourth loss function is represented as follows: ， wherein the total loss function for the proximal policy optimization network and the self- guided latent representation learning network, the first loss function for the self-guided latent representation learning network, a second loss function for a proximal policy optimization network, a third loss function, a fourth loss function, a fifth loss function; The fifth loss function is represented as follows: ， wherein is a second loss function for the on-policy optimization network, denotes the expectation, is the current policy is the probability of outputting an action in the current state, is the old policy is the probability of outputting an action in the current state, is a clipping function, , denotes the state, action at the current time step, = 0.2, denotes the reward-dependent advantage function; The third loss function is represented as follows: ， wherein is the fourth loss function, denotes the expectation, denotes the reward-dependent state value function, denotes the last reward function, denotes the state at the current time instant; The sixth loss function is represented as follows: ， wherein is the fifth loss function, denotes the expectation, denotes the cost-dependent state value function, denotes the kth cost function, denotes the current state; ​ ， wherein is the third loss function, is the sixth loss function, is the seventh loss function, and ReLU denotes an activation function. ​ ， wherein is the sixth loss function, denotes the expectation, is the current policy is the probability of outputting an action in the current state, is the old policy is the probability of outputting an action in the current state, is the cost-dependent advantage function, is the clipping function, , denotes the state, action at the current time, = 0.2; The seventh loss function is represented as follows: ， wherein is the seventh loss function, denotes expectation, T denotes time, denotes t-th power of discount factor, d denotes threshold value of k-th constraint collectively, is the immediate cost of t-th step.

5. A computer apparatus comprising a memory and a processor, characterised in that, The processor implements the method of any one of claims 1 to 4 when executing the computer program stored in the memory.

6. A computer-readable storage medium having stored thereon program instructions, wherein, The program instructions implement the method of any one of claims 1 to 4 when executed by a processor.

Images