A method and system for motion control of a legged robot

Abstract

The application belongs to the technical field of robot motion control, and proposes a kind of foot formula robot motion control method and system, to long short-term hybrid historical observation matrix, robot simulation environment in privileged observation information is respectively trained to student encoder, teacher encoder;Set weight coefficient, the encoding vector output by student encoder and teacher encoder is weighted and fused, the vector after weighted and fused is used as the input of strategy network with long short-term hybrid historical observation matrix, and the strategy network is trained using proximal policy optimization algorithm;Weight coefficient gradually decays to zero with the increase of robot speed and terrain difficulty faced in training process, to realize that the output of teacher encoder is dominant in early training, and the output of student encoder is dominant in later training.The strategy network trained using privileged information in simulation has reasoning ability, which can be more effectively transferred to the scene in actuality relying only on the body sensor.

Description

Technical Field

[0001] This invention belongs to the field of robot motion control technology, and in particular relates to a method and system for motion control of a legged robot. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] Legged robots, with their biomimetic locomotion mechanisms, have demonstrated excellent adaptability in unstructured and complex terrains and have been widely used in scenarios such as inspection and rescue. Traditional model-based methods can achieve stable and efficient control of their motion, but these methods rely on accurate dynamic models, require tuning a large number of parameters, and lack generalization ability for unknown environments.

[0004] In recent years, deep reinforcement learning has provided new insights into the control of legged robots. Among these, proximal policy optimization algorithms, which use ontological perception information as observations for training, have become mainstream methods. However, these methods typically assume perfectly observable states or rely on privileged information from simulations (such as terrain parameters) for training. In real-world deployments, the lack of privileged information and interference from real-world environmental noise make it difficult for the robot to accurately assess the environmental situation, thus weakening the stability and robustness of its motion.

[0005] To address this, researchers introduced representation learning methods such as a teacher-student learning framework. This framework adds a parallel teacher-student encoder before the policy network, trains the teacher encoder using simulated privileged information, and uses feature distillation to enable the student encoder to mimic its behavior, thus compensating for the lack of privileged information in real-world deployments. However, due to the differences in inputs between the teacher and student encoders, this method faces challenges such as output feature distribution shifts and the accumulation of imitation biases. Summary of the Invention

[0006] To overcome the shortcomings of the prior art, the present invention provides a method and system for controlling the motion of a legged robot, enabling the reasoning ability of a policy network trained with privileged information in simulation to be more effectively transferred to real-world scenarios that rely solely on the body's sensors.

[0007] To achieve the above objectives, the present invention adopts the following technical solution: In a first aspect, the present invention provides a method for motion control of a legged robot, comprising: The ontological observation data generated during the robot's interaction with the environment at various time periods are fused together with the short-term and long-term historical observations to obtain a hybrid short-term and long-term historical observation matrix. The student encoder and the teacher encoder are trained using a mixed long-term and short-term historical observation matrix and privileged observation information in a robot simulation environment, respectively. During the training process, the output of the student encoder is made to continuously approach the output of the teacher encoder. Weight coefficients are set, and the encoded vectors output by the student encoder and the teacher encoder are weighted and fused. The weighted and fused vector and the long-short mixed historical observation matrix are used as the input of the policy network, and the policy network is trained by the near-end policy optimization algorithm. Based on the robot's current observation information, the robot's future actions are obtained using the trained policy network.

[0008] In a second aspect, the present invention provides a legged robot motion control system, comprising: The observation module is configured to: merge the short-term and long-term historical observations into a hybrid short-term and long-term historical observation matrix based on the ontological observation data generated during the robot's interaction with the environment at various time periods. The teacher-student encoder module is configured to train the student encoder and the teacher encoder using a mixed long-term and short-term historical observation matrix and privileged observation information in the robot simulation environment, respectively. During the training process, the output of the student encoder continuously approaches the output of the teacher encoder. The strategy module is configured to: set weight coefficients, perform weighted fusion of the encoded vectors output by the student encoder and the teacher encoder, use the weighted fused vector and the long-short mixed historical observation matrix as input to the strategy network, and train the strategy network using a proximal policy optimization algorithm; wherein, the weight coefficients gradually decay to zero as the robot speed and the difficulty of the terrain increase during the training process, so that the fusion result in the early stage of training is mainly derived from the output of the teacher encoder, and the fusion result in the later stage of training is mainly derived from the output of the student encoder; The control module is configured to: obtain the robot's future actions based on the robot's current observation information and the trained policy network.

[0009] Thirdly, the present invention provides an electronic device including a memory and a processor, and computer instructions stored in the memory and running on the processor, wherein the computer instructions, when executed by the processor, perform the method described in the first aspect.

[0010] Fourthly, the present invention provides a computer-readable storage medium for storing computer instructions, which, when executed by a processor, perform the method described in the first aspect.

[0011] Fifthly, the present invention provides a computer program product, including a computer program that, when executed by a processor, implements the method described in the first aspect.

[0012] The above one or more technical solutions have the following beneficial effects: In this invention, by dividing short-term and long-term historical observations, the fused short- and long-term hybrid historical observation matrix simultaneously retains the robot's relatively accurate recent state and approximate long-term motion context information, improving the control strategy's utilization of temporal information. Through the setting of weight coefficients, in the early stages of training, the policy network performance is poor; therefore, the teacher encoder, which uses privileged information for encoding, takes the lead, enabling the policy network to quickly adapt to environmental changes and perform basic motion control. As the robot's speed and the difficulty of the terrain increase, representing an improvement in the policy network's performance, the input focus shifts towards the student encoder, encouraging the policy network to utilize more ontological observation information for exploration. This alleviates the output offset problem easily caused by the large input differences between the teacher and student encoders, allowing the reasoning ability of the policy network trained with privileged information in simulation to be more effectively transferred to real-world scenarios relying solely on ontological sensors.

[0013] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0014] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0015] Figure 1 This is a flowchart of a transition control method for assisting a quadruped robot in switching control strategies, as described in an embodiment of the present invention. Figure 2 This is an overall framework diagram of the system in an embodiment of the present invention; Figure 3 This is a diagram illustrating the effect of the quadruped robot moving through various terrains when the control strategy is deployed in a simulation according to an embodiment of the present invention. Figure 4 This is a diagram showing the effect of the quadruped robot going up and down stairs when the control strategy is deployed in the simulation according to the embodiment of the present invention. Detailed Implementation

[0016] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0017] It should be noted that the terminology used herein is for the purpose of describing particular implementations only and is not intended to limit the exemplary implementations of the present invention.

[0018] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0019] Example 1 like Figure 1 As shown, this embodiment discloses a motion control method for a legged robot, including: The ontological observation data generated during the robot's interaction with the environment at various time periods are fused together with the short-term and long-term historical observations to obtain a hybrid short-term and long-term historical observation matrix. The student encoder and the teacher encoder are trained using a mixed long-term and short-term historical observation matrix and privileged observation information in a robot simulation environment, respectively. During the training process, the output of the student encoder is made to continuously approach the output of the teacher encoder. Weight coefficients are set, and the encoded vectors output by the student encoder and the teacher encoder are weighted and fused. The weighted and fused vector and the long-short mixed historical observation matrix are used as the input of the policy network, and the policy network is trained by the near-end policy optimization algorithm. Based on the robot's current observation information, the robot's future actions are obtained using the trained policy network.

[0020] This embodiment, by dividing short-term and long-term historical observations, allows the fused hybrid short- and long-term historical observation matrix to simultaneously retain relatively accurate recent states of the robot and approximate long-term motion context information, thereby improving the control strategy's utilization of temporal information. Through the setting of weight coefficients, in the early stages of training, when the policy network performance is poor, the teacher encoder, which uses privileged information for encoding, takes the lead, enabling the policy network to quickly adapt to environmental changes and perform basic motion control. As the robot's speed and the difficulty of the terrain increase, representing an improvement in the policy network's performance, the input focus shifts towards the student encoder, encouraging the policy network to utilize more ontological observation information for exploration. This alleviates the output offset problem that easily arises from the large input differences between the teacher and student encoders, allowing the reasoning ability of the policy network trained using privileged information in simulation to be more effectively transferred to real-world scenarios that rely solely on ontological sensors.

[0021] The following is combined with Figures 1-2 This embodiment provides a detailed description of a legged robot motion control method, which specifically includes the following steps: Step 1: For the ontological observation data generated during the robot's interaction with the environment at various time periods, the short-term historical observations and long-term historical observations are fused together to obtain a mixed short-term and long-term historical observation matrix.

[0022] In this embodiment, short-term and long-term historical observation windows are divided, and the long-term observations are feature-compressed and spliced ​​together to form a mixed input matrix of short-term and long-term historical observations.

[0023] The training process begins with the robot's initial state. For the ontological observation data generated during the robot's interaction with the environment at various time points, short-term and long-term windows are defined. Short-term historical observations are used directly, while long-term historical observations are compressed and aligned using a CNN network, and finally, they are combined into an observation matrix that integrates both short- and long-term historical information.

[0024] The overall training framework is now being built, with this step primarily focusing on the input portion of the control strategy. During training, the robot continuously interacts with its environment, generating observational data. This observational information can be categorized into privileged observational information and proprioceptive observational information, with privileged observational information equivalent to the robot's state. It contains information such as speed and terrain elevation that robots cannot obtain in reality, while the ontological observation information This refers to the ontological data that robots can directly obtain in the real world.

[0025] Ontology observation data The requirements for a student encoder include: 3D linear velocity instructions. 3D robot angular velocity 3D robot gravity vector The position of each joint of a 12-dimensional robot 12-dimensional joint velocity Action in the 12th dimension at the previous moment The dataset comprises 45 dimensions. During training, observations generated at multiple time steps can form an observation sequence, which serves as historical information for the policy network to make decisions. To fully utilize this information, two observation window lengths are defined: a short-term historical information window and a long-term historical information window.

[0026] For a short-term historical information window, it includes observations from the current moment and the nine immediate past time steps, i.e. Short-term history reflects the robot's immediate, recent state and is designed to enable control strategies to respond quickly to the current state.

[0027] For long-term historical information windows, they include historical observations from earlier time steps, specifically the observations from the last 30 time steps immediately following the short-term historical information window. Long-term history reflects the robot's long-term state, aiming to enable control strategies to determine the current trend of motion changes. However, long-term historical observations have a high dimensionality and may contain redundant information. Directly combining them with short-term history can easily allow them to take precedence, resulting in a lack of real-time performance in the control strategy.

[0028] Therefore, this embodiment uses a lightweight convolutional network to compress its features, allowing them to be naturally integrated into the overall observation matrix, that is:

[0029] in, These are the parameters of the convolutional network. The extracted long-term historical feature vector matrix has the same dimensions as the short-term historical observation matrix. same.

[0030] Finally, the long-term historical feature matrix and the short-term historical feature matrix are concatenated to form the final long- and short-term hybrid historical observation matrix, denoted as . This mixed long- and short-term historical observation matrix The input is the student encoder described below, a mixed long- and short-term historical observation matrix. At the same time, it retains relatively accurate recent state information and general long-term motion context information of the robot, improving the degree to which the control strategy utilizes temporal information.

[0031] Step 2: Use the mixed long-short-term historical observation matrix as the input to the student encoder and the privileged observation information in the robot simulation environment as the input to the teacher encoder. During the training process, make the output of the student encoder continuously approach the output of the teacher encoder. Set weight coefficients and perform weighted fusion of the encoding vectors output by the student encoder and the teacher encoder. The weighted fused vector and the mixed long-short-term historical observation matrix are used as the input to the policy network.

[0032] A teacher encoder and a student encoder are established. The teacher encoder is input with privileged observation information from the simulation, while the student encoder is input with the mixed long-short-term historical observation matrix constructed in step 1. During training, the outputs of the two encoders are weighted and fused, with the weights adjusted according to the task difficulty. Ultimately, the output of the student encoder becomes dominant. The resulting fused output vector is concatenated with the current mixed long-short-term historical observation matrix and input into the policy network. This ensures that the policy network adapts synchronously to both the teacher and student encoders throughout the process, and knowledge distillation is used to make the output of the student encoder fit the teacher encoder.

[0033] This step primarily transitions the model from reasoning using privileged environment information from simulation to making judgments using only ontological observation information. First, a teacher encoder is constructed. With student encoder Teacher encoder The input is privileged observation information provided by the simulation environment; the student encoder The input is long-term and short-term historical ontological observation information, namely the long-term and short-term hybrid historical observation matrix constructed in step 1. .

[0034] Both the teacher encoder and the student encoder are constructed using MLP networks, and their outputs are both 19-dimensional feature vectors. This achieves compression of high-dimensional privileged and ontological historical observation information, ensuring that the information input to the control policy network is a lightweight feature vector.

[0035] The mapping of the teacher encoder is denoted as , Indicates privileged observation information, Let represent the learnable parameters in the teacher encoder; the mapping of the student encoder is denoted as . , This represents the mixed long- and short-term historical observation matrix constructed in step 1. This represents the learnable parameters in the teacher encoder.

[0036] Traditional teacher-student encoder architectures optimize student and teacher encoders separately, meaning that only the mapping output of the teacher encoder is used during training. The input control policy network is optimized by the teacher encoder along with the policy network based on the policy gradient generated by reinforcement learning. The student encoder's mapped output... The policy network is only input during the inference process after training. Its optimization direction is to mimic the output of the teacher encoder, and it is not optimized along with the policy network. The loss function in this stage is defined as the mean square error of both:

[0037] in, This represents the total number of elements involved in the calculation, which is the product of the total dimension of the feature vector and the number of samples.

[0038] Therefore, while traditional methods achieve knowledge transfer, they suffer from three problems: 1) The input information of the teacher encoder and the student encoder differs significantly in terms of data dimension, physical meaning, and data distribution. Consequently, the student encoder cannot accurately follow the output of the teacher encoder, and its feature distribution is prone to drift. 2) The student encoder does not optimize with the policy gradient generated during reinforcement learning; it only performs imitation learning. When dealing with data not encountered during training, it is prone to producing meaningless outputs. 3) The control policy network is not optimized along with the student encoder during training, and directly connecting to the student encoder during inference can easily compromise robustness.

[0039] Therefore, this embodiment adopts a coherent teacher-student encoder architecture, that is, constructing a fused feature vector. The outputs of the teacher encoder and student encoder are fed into the policy network together during training. Feature vectors are then fused. for:

[0040] in, It is a dynamically adjusted weighting coefficient, specifically defined as:

[0041] in, This indicates the average terrain difficulty level the robot is currently in during training. This indicates the maximum terrain difficulty level. This represents the current average speed of all robots. This indicates the maximum given speed.

[0042] Initially set to 1, the value gradually decreases to 0 as the robot's speed and terrain difficulty increase during training. This represents that in the early stages of training, the teacher encoder output dominates, while in the later stages, the student encoder output becomes dominant. The significance of this is that in the early stages of training, the policy network performance is relatively poor, so the teacher encoder, which uses privileged information for encoding, takes the lead, enabling the policy network to quickly adapt to environmental changes and perform basic motion control. As the robot's speed and terrain difficulty increase, representing improved policy network performance, the input focus shifts towards the student encoder, encouraging the policy to utilize more ontological observation information for exploration.

[0043] Throughout the process, the student encoder mimics the output of the teacher encoder, and the loss function is the same as in traditional methods. This embodiment enables the policy network, teacher encoder, and student encoder to be jointly optimized. The policy network naturally adapts to the hybrid characteristics of different encoders, achieving smooth knowledge transfer.

[0044] Step 3: Construct a history encoder and a future state encoder. Use the mixed long-short-term historical observation matrix as the input to the history encoder, and use the future observations corresponding to the mixed long-short-term historical observation matrix as the input to the future state encoder.

[0045] This embodiment establishes a historical encoder and a future state encoder, which narrows the similarity of historical and future encoded vector features under the same robot motion trajectory, thereby improving the robot's temporal perception capability.

[0046] Currently, to better utilize temporal information to assist control strategies in decision-making, historical observation sequences are often used as input to policy networks. These networks can then predict the robot's next state in the environment, enabling them to learn more discriminative state representations from high-dimensional observations and thus giving the strategy a certain predictive ability. However, relying on regression to accurately predict the next state can lead to overfitting. Real-world environments are complex, and when faced with complex combinations of historical information unseen by the strategy, it may cause the strategy to predict and adjust in a completely wrong direction, leading to catastrophic consequences.

[0047] Therefore, this embodiment extends the student encoder constructed in step 2 with an output head to output the encoded vector of historical observation information. In addition, an extra encoder is constructed to process the corresponding future observation information. This reduces the similarity between historical and future feature vectors within the same trajectory while increasing the similarity between different trajectories. Simultaneously, the last three dimensions of the historical observation information encoded vector are used to fit privileged velocity information.

[0048] This section aims to teach the encoder to distinguish the robot's response patterns under different environments, thereby assisting the control strategy in making better decisions. First, based on the student encoder built in step 2, an additional output head is added, denoted as the historical encoder. It shares a hidden layer with the student encoder, and its output is denoted as... There are a total of 19 dimensions. Among them, Represents a 16-dimensional history encoding vector. This represents the robot's estimated 3D velocity at time t. and Together they form the output. This represents the mixed long- and short-term historical observation matrix constructed in step 2. This indicates the parameters that this encoder can learn.

[0049] Next, an additional independent future state encoder is constructed. Future State Encoder It is also composed of a multi-layer MLP network. Future state encoder The input is the actual ontology observation of the next time step. The output is It has 16 dimensions, consistent with the historical encoding vectors mentioned above. This represents the parameters that the future state encoder can learn. This encoder does not share parameters with other networks and is specifically designed to extract features from future observations.

[0050] During training, data is collected along the robot's interaction trajectory with the environment. For each time step... Its previous long- and short-term mixed historical observation matrix Corresponding future observations If both originate from the same trajectory, then their encoded vectors constitute a positive sample pair. By combining sample pairs from multiple time steps across multiple trajectories, a batch of data can be formed.

[0051] Let the batch size be That is, the batch contains The historical encoding vector and the future encoding vector are denoted as follows: as well as Both are A cross-correlation matrix is ​​calculated along the batch dimension between historical and future encoded vectors. Each element , representing the similarity between the i-th dimension of the historical encoding vector and the j-th dimension of the future encoding vector, is calculated as follows:

[0052] Where the indices i and j range from 1 to 16, therefore, the resulting cross-correlation matrix... It is A square formation.

[0053] Based on the cross-correlation matrix, the optimized loss function for similarity matching is defined as:

[0054] The first term of the loss function is the alignment term; the optimization process forces the diagonal elements of the cross-correlation matrix to... The term is close to 1, making the historical and future encoded vectors from the same type of trajectory highly correlated in their corresponding feature dimensions, thus enabling the historical and future state encoders to learn temporally related common representations. The second term is a redundancy reduction term; the optimization process forces the off-diagonal elements of the cross-correlation matrix to... Approaching 0 reduces redundant dependencies between different feature dimensions, enabling each dimension of the encoding vector to carry independent information representation.

[0055] Furthermore, in order to make full use of historical encoding information, this embodiment uses the last 3D output of the historical encoder as the estimated speed. And to make it fit the true speed, the loss function is defined as the mean square error between the two, expressed as:

[0056] in, This represents the total number of elements involved in the calculation, which is the product of the feature vector dimension and the number of samples. This indicates the robot's estimated velocity at the current moment; This represents the robot's actual speed obtained from the simulation environment at the current moment.

[0057] The historical encoder and the future state encoder work together to optimize and minimize both loss functions. Thus, the hidden layer shared by the historical encoder and the student encoder implicitly encodes the correlation between environment type and robot response, and explicitly estimates the robot's actual motion speed, thereby improving the control strategy's adaptability to the environment.

[0058] This embodiment encodes the historical observation sequence and the actual next observation state using a history encoder and a future state encoder, respectively. The history sequence encoder is another output head of the student encoder, and the hidden layers are shared by both. During training, historical-future state pairs within the same trajectory are used as positive samples, while historical sequences and future state pairs from different trajectories are used as negative samples. This maximizes the similarity of positive sample pairs in the feature space and minimizes the similarity of negative sample pairs, giving the features extracted by the history encoder temporal discriminative power. The history encoder can implicitly establish a correlation between historical and future data at the feature level and reduce redundant relationships between feature dimensions. This allows the policy to maintain reliable decision-making even when facing unseen complex state combinations, effectively mitigating the performance degradation that may occur due to overfitting during training when the control policy faces unfamiliar environments.

[0059] Step 4: Based on the robot's current observation information, obtain the robot's future actions based on the trained policy network.

[0060] This embodiment employs an architecture comprising one policy network and one evaluation network. The policy network outputs the actions the robot must perform, while the evaluation network assesses the reward. The reward function mainly includes multi-directional velocity following, continuity maintenance, and torque constraint. Based on the constructed network, input, and objective function, the PPO algorithm is used to optimize the transition strategy. After multiple iterations, the control strategy can fully utilize environmental and temporal information, learning to control the robot to move stably in unknown and complex environments.

[0061] A reinforcement learning network architecture is constructed, and a reward function is defined. Based on the overall network, observation, input structure, and reward function, the PPO algorithm is trained using a proximal strategy optimization.

[0062] This embodiment employs an architecture consisting of one policy network and one evaluation network. Current ontology observations. The fused feature vector constructed in steps 2 and 3 Estimate speed Together, these form the overall observation matrix, which the policy network uses as input and outputs the actions the robot needs to perform. The evaluation network takes privileged observations as input and outputs estimated cumulative rewards.

[0063] The core reward function mainly includes speed-following reward. Torque Reward Balanced rewards and z-axis speed limit As shown below: Speed ​​Follow Rewards To encourage the robot to track the desired linear and angular velocities, an exponential form is used to smooth them, which can be expressed as:

[0064] in, , , This represents the given command for the axial velocity along the x and y axes and the angular velocity along the z axis. , , This indicates the robot's current actual speed.

[0065] Torque bonus The purpose is to limit the output torque of the robot joints, thereby protecting the hardware and reducing energy consumption, which can be expressed as:

[0066] in, This represents the torque value output by each joint at the current time step.

[0067] Balanced rewards The purpose is to maintain the balance of the quadruped robot during movement and ensure that the robot's base is as horizontal as possible with respect to the ground. This is defined as follows:

[0068] in, , This represents the projection components of the normalized gravity vector onto the horizontal plane of the robot base in the x and y directions.

[0069] z-axis speed limit The purpose is to limit the linear velocity along the z-axis of the body coordinate system of a quadruped robot during movement, in order to ensure smooth robot motion and reduce bumps. It is defined as follows:

[0070] in, This represents the velocity along the z-axis of the quadruped robot's body coordinate system.

[0071] Based on this, during training, in each iteration, the robot continuously interacts with the environment from its initial state, collecting a state trajectory containing information such as observations, actions, reward values, and constraint values. Based on this trajectory, the evaluation network is calculated. The value of each state is estimated, and the reward advantage function is derived using the generalized advantage estimation method. Therefore, the policy network optimization loss function can be obtained as follows:

[0072] in, Indicates the current strategy. This indicates the strategy before the update. This represents the pruning hyperparameter, used to limit the policy update step size. This indicates the operation of calculating the mean.

[0073] Subsequently, the reward evaluation network was updated. The loss function is defined as follows: The objective is to minimize the mean squared error between the estimated value and the actual reward.

[0074] in, This represents the estimated cumulative reward. This represents the cumulative true value of the reward / reward discount.

[0075] During training, the parameters of the policy network, evaluation network, teacher-student encoder, and future state encoder are updated together, and the total loss function is as follows:

[0076] After multiple rounds of iterative optimization, the strategy can fully integrate mixed long- and short-term observations, implicit temporal guidance provided by the historical-future state alignment module, and distillation privilege information provided by the coherent teacher-student network, enabling the robot to achieve stable movement in unknown and complex terrains relying solely on its own proprioceptive information. Exemplary illustrations of the robot's movement in diverse terrains are shown below. Figure 3 As shown in the example diagram, the steps are designed to be ascending and descending. Figure 4 As shown.

[0077] In summary, the hybrid control method described above, which combines long-term and short-term historical observations as input, a coherent teacher-student encoder, and alignment with historical-future state features, significantly enhances the quadruped robot's temporal and environmental perception capabilities in complex unstructured terrain, thereby improving its motion robustness.

[0078] Example 2 The purpose of this embodiment is to provide a motion control system for a legged robot, including: The observation module is configured to: merge the short-term and long-term historical observations into a hybrid short-term and long-term historical observation matrix based on the ontological observation data generated during the robot's interaction with the environment at various time periods. The teacher-student encoder module is configured to train the student encoder and the teacher encoder using a mixed long-term and short-term historical observation matrix and privileged observation information in the robot simulation environment, respectively. During the training process, the output of the student encoder continuously approaches the output of the teacher encoder. The strategy module is configured to: set weight coefficients, perform weighted fusion of the encoded vectors output by the student encoder and the teacher encoder, use the weighted fused vector and the long-short mixed historical observation matrix as input to the strategy network, and train the strategy network using a proximal policy optimization algorithm; wherein, the weight coefficients gradually decay to zero as the robot speed and the difficulty of the terrain increase during the training process, so that the fusion result in the early stage of training is mainly derived from the output of the teacher encoder, and the fusion result in the later stage of training is mainly derived from the output of the student encoder; The control module is configured to: obtain the robot's future actions based on the robot's current observation information and the trained policy network.

[0079] In further embodiments, the following is also provided: An electronic device includes a memory and a processor, as well as computer instructions stored in the memory and running on the processor. When executed by the processor, the computer instructions perform the method described in Embodiment 1. For brevity, further details are omitted here.

[0080] It should be understood that in this embodiment, the processor can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0081] Memory may include read-only memory and random access memory, and provides instructions and data to the processor. A portion of memory may also include non-volatile random access memory. For example, memory may also store information about the device type.

[0082] A computer-readable storage medium for storing computer instructions, which, when executed by a processor, perform the method described in Embodiment 1.

[0083] The method in Embodiment 1 can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor. The software modules can reside in readily available storage media in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory; the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, a detailed description is not provided here.

[0084] A computer program product includes a computer program that, when executed by a processor, implements the method described in Embodiment 1.

[0085] The present invention also provides at least one computer program product tangibly stored on a non-transitory computer-readable storage medium. The computer program product includes computer-executable instructions, such as instructions included in program modules, which execute in a device on a target real or virtual processor to perform the processes / methods described above. Typically, program modules include routines, programs, libraries, objects, classes, components, data structures, etc., that perform specific tasks or implement specific abstract data types. In various embodiments, the functionality of program modules can be combined or divided among program modules as needed. The machine-executable instructions for the program modules can execute within a local or distributed device. In a distributed device, the program modules can reside in both local and remote storage media.

[0086] The computer program code used to implement the methods of the present invention may be written in one or more programming languages. This computer program code may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the computer or other programmable data processing device, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a computer, partially on a computer, as a stand-alone software package, partially on a computer and partially on a remote computer, or entirely on a remote computer or server.

[0087] In the context of this invention, computer program code or related data may be carried by any suitable carrier to enable a device, apparatus, or processor to perform the various processes and operations described above. Examples of carriers include signals, computer-readable media, and the like. Examples of signals may include electrical, optical, radio, sound, or other forms of propagation signals, such as carrier waves, infrared signals, etc.

[0088] Those skilled in the art will recognize that the units and algorithm steps described in conjunction with the embodiments herein can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0089] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A method for motion control of a legged robot, characterized in that, include: The ontological observation data generated during the robot's interaction with the environment at various time periods are fused together with the short-term and long-term historical observations to obtain a hybrid short-term and long-term historical observation matrix. The student encoder and the teacher encoder are trained using a mixed long-term and short-term historical observation matrix and privileged observation information in a robot simulation environment, respectively. During the training process, the output of the student encoder is made to continuously approach the output of the teacher encoder. Weight coefficients are set, and the encoded vectors output by the student encoder and the teacher encoder are weighted and fused. The weighted fused vector and the long-short mixed historical observation matrix are used as input to the policy network. The policy network is trained using a proximal policy optimization algorithm. The weight coefficients gradually decay to zero as the robot speed and the difficulty of the terrain increase during the training process, so that the fusion result in the early stage of training is mainly derived from the output of the teacher encoder, and the fusion result in the later stage of training is mainly derived from the output of the student encoder. Based on the robot's current observation information, the robot's future actions are obtained using the trained policy network.

2. The motion control method for a legged robot as described in claim 1, characterized in that, The ontological observation data generated during the robot's interaction with the environment at various time intervals are fused together with the short-term and long-term historical observations to obtain a hybrid short- and long-term historical observation matrix, as follows: A lightweight convolutional network is used to compress the features of long-term historical observation information, so that the compressed long-term historical observation matrix has the same dimension as the short-term historical observation matrix. By concatenating the long-term historical observation matrix with the short-term historical observation matrix, a hybrid long-term and short-term historical observation matrix is ​​obtained.

3. The motion control method for a legged robot as described in claim 1, characterized in that, It also includes constructing a history encoder and a future state encoder, using a mixed long-short-term historical observation matrix as the input to the history encoder, and using the future observations corresponding to the mixed long-short-term historical observation matrix as the input to the future state encoder; During the training of the history encoder and the future state encoder, the history-future state pairs in the same trajectory are used as positive samples, and the history-future state pairs in different trajectories are used as negative samples. By maximizing the similarity of positive sample pairs in the feature space and minimizing the similarity of negative sample pairs in the feature space, the features extracted by the future state encoder have temporal discriminative power; wherein, the future state encoder and the student encoder share the hidden layer.

4. The motion control method for a legged robot as described in claim 1, characterized in that, In training the policy network using the proximal policy optimization algorithm, the reward function includes velocity following reward, torque reward, balance reward, and z-axis velocity constraint. The speed following reward is used to encourage the robot to track the desired linear and angular velocities; the torque reward is used to limit the output torque of the robot joints; the balance reward is used to maintain the robot's balance during movement; and the z-axis speed limit is used to limit the linear velocity of the body coordinate system along the z-axis during robot movement.

5. The motion control method for a legged robot as described in claim 1, characterized in that, The loss function for the policy network optimization is: in, Indicates the current strategy; Indicates the strategy before the update; This represents the pruning hyperparameter, used to limit the policy update step size; This indicates the operation of calculating the mean; Indicates the actions performed by the robot; For ontological observation; This is a reward advantage function.

6. The motion control method for a legged robot as described in claim 1, characterized in that, The weighting coefficients are specifically: in, This indicates the average terrain difficulty level the robot is currently in during training. Indicates the maximum terrain difficulty level; This represents the current average speed of all robots. This indicates the maximum given speed.

7. A motion control system for a legged robot, characterized in that, include: The observation module is configured to: merge the short-term and long-term historical observations into a hybrid short-term and long-term historical observation matrix based on the ontological observation data generated during the robot's interaction with the environment at various time periods. The teacher-student encoder module is configured to train the student encoder and the teacher encoder using a mixed long-term and short-term historical observation matrix and privileged observation information in the robot simulation environment, respectively. During the training process, the output of the student encoder continuously approaches the output of the teacher encoder. The strategy module is configured to: set weight coefficients, perform weighted fusion of the encoded vectors output by the student encoder and the teacher encoder, use the weighted fused vector and the long-short mixed historical observation matrix as input to the strategy network, and train the strategy network using a proximal policy optimization algorithm; wherein, the weight coefficients gradually decay to zero as the robot speed and the difficulty of the terrain increase during the training process, so that the fusion result in the early stage of training is mainly derived from the output of the teacher encoder, and the fusion result in the later stage of training is mainly derived from the output of the student encoder; The control module is configured to: obtain the robot's future actions based on the robot's current observation information and the trained policy network.

8. An electronic device, characterized in that, It includes a memory and a processor, as well as computer instructions stored in the memory and running on the processor, which, when executed by the processor, perform the method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, Used to store computer instructions, which, when executed by a processor, perform the method described in any one of claims 1-6.

10. A computer program product, characterized in that, Includes a computer program, which, when executed by a processor, implements the method described in any one of claims 1-6.

Images