Robot motion control strategy network training method and device based on state representation learning

Abstract

The present disclosure provides a robot motion control strategy network training method and device based on state representation learning, relating to the technical field of robots. The method comprises: obtaining privileged states and ontology perception states in the training stage, and performing zero mask processing on the privileged states to construct masked states. The privileged states and their masked states are respectively input into a policy encoder sharing weights, and further generate a prediction vector via a predictor. On this basis, a contrast loss function is constructed based on the prediction vector and the opposite end representation vector, and the parameters of the policy encoder are updated according to the contrast loss function. The present disclosure can promote the ontology perception state to form a distribution in the embedding space that is more discriminative and task-related, improve the expression ability of key dynamic changes, and thus improve the accuracy and stability of the action strategy generated by the control strategy network.

Description

Technical Field

[0001] This disclosure relates to the field of robotics technology, and to a method and apparatus for training a robot motion control strategy network based on state representation learning. Background Technology

[0002] In whole-body control (WBC) of humanoid robots, the control policy network needs to generate complex joint movements based on sensor observations to perform tasks such as standing, walking, and imitation. The current mainstream approach is to use a reinforcement learning (RL) framework to train the control policy network, leveraging the robot's interaction experience with its environment to improve behavioral performance. However, this approach faces two key technical bottlenecks:

[0003] Due to the highly complex dynamics and strong coupling of multiple degrees of freedom in humanoid robot systems, reinforcement learning algorithms still have room for improvement in training sample efficiency under partially observable states. Specifically, when relying solely on the body's perceived state, the control policy network may struggle to accurately model the dynamic features required for the current task in some situations, leading to slow convergence or even getting stuck in local optima during training.

[0004] To alleviate the aforementioned problems, some technical solutions introduce State Representation Learning (SRL) methods to extract more task-relevant low-dimensional feature representations from raw observation data. However, these techniques often rely on constructing sample pairs using a single perceptual modality, making it difficult to effectively utilize privileged states available during training but unavailable in real-world deployments. This limits the expressive power of the state embedding space, thereby affecting the accuracy and stability of the action policies generated by the control policy network.

[0005] Therefore, the current training methods for robot motion control strategy networks still need further optimization.

[0006] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0007] This disclosure provides a method and apparatus for training robot motion control policy networks based on state representation learning, in order to solve the problem that the state embedding space representation capability is limited in related technologies, thereby affecting the accuracy and stability of the action policies generated by the control policy network.

[0008] Additional aspects and advantages of this disclosure will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this disclosure.

[0009] According to a first aspect of this disclosure, a method for training a robot motion control policy network based on state representation learning is provided. The state representation learning model framework includes a policy encoder and a predictor. The method includes:

[0010] Obtain the robot's ontological perception state and privileged state at the current time step;

[0011] The privileged state is zero-masked to obtain a mask state corresponding to the privileged state. The zero-masking process includes setting at least the part of the privileged state that does not belong to the ontology perception state to zero.

[0012] The privileged state and the mask state are respectively input into the policy encoder with shared weights to generate the corresponding first representation vector and second representation vector;

[0013] The first representation vector and the second representation vector are respectively input into the predictor to obtain the corresponding first prediction vector and the second prediction vector;

[0014] The contrastive loss function is calculated based on the first prediction vector and the second representation vector, and the parameters of the policy encoder are updated based on the contrastive loss function.

[0015] In one exemplary embodiment of this disclosure, the privileged state is subjected to zero-mask processing to obtain a mask state corresponding to the privileged state, including:

[0016] Determine the index position in the privileged state corresponding to the non-ontology-aware state dimension;

[0017] The values ​​belonging to the ontological perception state dimension in the privileged state are retained unchanged;

[0018] Force the values ​​at at least some index positions to be zero to generate a mask state.

[0019] In one exemplary embodiment of this disclosure, zero-mask processing includes:

[0020] Set all portions of the privileged state that do not belong to the ontology perception state to zero.

[0021] In one exemplary embodiment of this disclosure, a contrastive loss function is calculated based on a first prediction vector and a second representation vector, and a second prediction vector and a first representation vector, including:

[0022] Perform a stopping gradient operation on the second representation vector to obtain the first target vector;

[0023] Perform a stopping gradient operation on the first representation vector to obtain the second target vector;

[0024] Calculate the negative first similarity value between the first predicted vector and the first target vector;

[0025] Calculate the second similarity negative value between the second predicted vector and the second target vector;

[0026] The negative values ​​of the first and second similarities are added together to obtain the contrast loss function.

[0027] In one exemplary embodiment of this disclosure, calculating a first similarity negative value between a first predicted vector and a first target vector includes:

[0028] The first prediction vector is normalized using the L2 norm to obtain the normalized prediction vector;

[0029] The first target vector is normalized using the L2 norm to obtain the normalized target vector;

[0030] Calculate the dot product of the normalized prediction vector and the normalized target vector;

[0031] Taking the negative value of the dot product result yields the negative first similarity value.

[0032] In one exemplary embodiment of this disclosure, the state representation learning model framework further includes a policy head network; the method further includes:

[0033] The second representation vector is input into the policy head network to obtain the action distribution at the current time step;

[0034] The target joint position residuals are obtained by sampling from the motion distribution;

[0035] The proximal policy optimization loss is calculated based on the reward signal obtained from the interaction between the target joint position residual and the environment.

[0036] Obtain the first gradient value corresponding to the contrastive loss function, and obtain the second gradient value corresponding to the near-end policy optimization loss;

[0037] The parameters of the policy encoder and policy head network are updated based on the first and second gradient values.

[0038] In one exemplary embodiment of this disclosure, updating the parameters of the policy encoder and the policy head network based on a first gradient value and a second gradient value includes:

[0039] Obtain the preset balance coefficient;

[0040] Multiply the contrast loss function by the balance coefficient to obtain the weighted contrast loss;

[0041] The total loss function is obtained by adding the near-end policy optimization loss and the weighted comparison loss.

[0042] Perform backpropagation on the total loss function to update the parameters of the policy encoder and policy head network.

[0043] In one exemplary embodiment of this disclosure, the near-end policy optimization loss is added to the weighted contrastive loss to obtain the total loss function, including:

[0044] Get the current training iteration count;

[0045] Determine whether the training iteration count meets the preset update interval condition;

[0046] If the update interval condition is met, the near-end policy optimization loss and the weighted contrast loss are added together to obtain the total loss function.

[0047] In one exemplary embodiment of this disclosure, the method further includes:

[0048] If the update interval condition is not met, only the near-end policy optimization loss is retained as the total loss function.

[0049] In one exemplary embodiment of this disclosure, the privileged state and the mask state are respectively input into a policy encoder with shared weights, including:

[0050] Construct a policy encoder that includes a multi-layer fully connected neural network;

[0051] The privileged state is used as the first input to a fully connected neural network, which outputs a first representation vector.

[0052] The mask state is used as the second input to batch input into the same fully connected neural network, and the output is the second representation vector;

[0053] In this process, the first input batch and the second input batch share the weight parameters of each layer of the fully connected neural network.

[0054] In one exemplary embodiment of this disclosure, the first representation vector and the second representation vector are respectively input to the predictor, including:

[0055] Construct a multilayer perceptron containing a bottleneck layer as the predictor, where the input dimension and output dimension of the predictor are the same.

[0056] The first representation vector is input into the multilayer perceptron, and the first prediction vector is output after nonlinear transformation.

[0057] The second representation vector is input into the multilayer perceptron, and the second prediction vector is output after nonlinear transformation.

[0058] In one exemplary embodiment of this disclosure, the state representation learning model framework further includes a value network; the method further includes:

[0059] The privileged state is input into the value encoder of the value network to obtain the value feature vector;

[0060] Input the value feature vector into the value head network of the value network, and output the estimated value of the current state;

[0061] Calculate the value function loss based on the difference between the estimated value and the actual return;

[0062] The parameters of the value encoder and value head network are updated using the value function loss.

[0063] In one exemplary embodiment of this disclosure, obtaining the robot's ontological perception state at the current time step includes:

[0064] Acquire joint position and velocity data collected by the robot's joint sensors;

[0065] Acquire the angular velocity data and gravity vector data of the robot base collected by the robot base inertial measurement unit;

[0066] The joint position data, joint velocity data, base angular velocity data, and gravity vector data are stitched together to form the body perception state.

[0067] In one exemplary embodiment of this disclosure, obtaining the privileged state of the robot at the current time step includes:

[0068] Read the robot's base linear velocity data and base position data in the simulation environment;

[0069] Read the height map scan data of the terrain where the robot is located in the simulation environment;

[0070] The privileged state is constructed by combining the body perception state with the base linear velocity data, base position data, and height map scan data.

[0071] In one exemplary embodiment of this disclosure, before obtaining the robot's ontology perception state and privileged state at the current time step, the method further includes:

[0072] Load the robot model and environment model into the physics simulator;

[0073] Control the robot model to execute action sequences within the environment model;

[0074] The sensor observation data during the execution of the action sequence is recorded as the body's perception state;

[0075] Record the internal state data of the physics engine during the execution of the action sequence as a privileged state.

[0076] In one exemplary embodiment of this disclosure, recording the internal state data of the physics engine during the execution of the action sequence as a privileged state includes:

[0077] Extract the global Cartesian position and global Cartesian velocity of each link in the robot from the physics engine;

[0078] Extract the contact force data between the robot's feet and the ground from the physics engine;

[0079] Extract the physical friction coefficient data of the ground from the physics engine;

[0080] The global Cartesian position, global Cartesian velocity, contact force data, and physical friction coefficient data are merged into the privileged state.

[0081] In one exemplary embodiment of this disclosure, the method further includes:

[0082] Establish gridded sampling points on a horizontal plane with the robot base as the center;

[0083] The height value of each sampling point relative to the robot's foot is obtained through X-ray detection;

[0084] Flatten the height values ​​of all sampling points into a one-dimensional vector;

[0085] Embed a one-dimensional vector into the tail of the privileged state.

[0086] In one exemplary embodiment of this disclosure, updating the parameters of the policy encoder based on the contrastive loss function includes:

[0087] Calculate the gradient of the contrastive loss function with respect to the parameters of the predictor;

[0088] Calculate the gradient of the contrastive loss function with respect to the parameters of the policy encoder;

[0089] The weights of the predictor and policy encoder are updated synchronously based on the gradients of their respective parameters using a stochastic gradient descent optimizer.

[0090] The parameter updates of the policy encoder are subject to dual supervision from the policy gradient and the contrastive loss gradient.

[0091] According to a second aspect of this disclosure, a robot motion control method is provided, comprising:

[0092] Obtain the robot's real-time ontological perception state and construct an all-zero vector consistent with the privileged state dimension;

[0093] The real-time ontology perception state is filled into the corresponding dimension of the all-zero vector to generate the inference input state;

[0094] The inference input state is input into a pre-trained policy encoder to obtain the target representation vector;

[0095] The target representation vector is input into a pre-trained policy head network, and the action policy for controlling the robot's motion is output based on the policy head network.

[0096] The policy encoder and policy head network are obtained according to the robot motion control policy network training method based on state representation learning in the first aspect of this disclosure.

[0097] According to a third aspect of this disclosure, a robot motion control policy network training device based on state representation learning is provided. The state representation learning model framework includes a policy encoder and a predictor; the device includes:

[0098] The state acquisition module is used to acquire the robot's ontological perception state and privileged state at the current time step;

[0099] The mask processing module is used to perform zero mask processing on the privileged state to obtain the mask state corresponding to the privileged state. The zero mask processing includes setting at least the part of the privileged state that does not belong to the ontology perception state to zero.

[0100] The representation vector generation module is used to input the privileged state and the mask state into the shared-weight policy encoder to generate the corresponding first representation vector and second representation vector;

[0101] The prediction vector generation module is used to input the first representation vector and the second representation vector into the predictor to obtain the corresponding first prediction vector and second prediction vector, respectively.

[0102] The parameter update module is used to calculate the contrastive loss function based on the first prediction vector and the second representation vector, and the second prediction vector and the first representation vector, and update the parameters of the policy encoder based on the contrastive loss function.

[0103] According to a fourth aspect of this disclosure, a robot motion control device is provided, comprising:

[0104] The vector construction module is used to obtain the robot's real-time ontology perception state and construct an all-zero vector consistent with the privileged state dimension.

[0105] The state generation module is used to fill the real-time ontology perception state into the corresponding dimension of the all-zero vector to generate the inference input state;

[0106] The vector generation module is used to input the inference input state into the pre-trained policy encoder to obtain the target representation vector;

[0107] The strategy output module is used to input the target representation vector into the pre-trained strategy head network, and output the action strategy to control the robot's movement based on the strategy head network;

[0108] The policy encoder and policy head network are obtained according to the robot motion control policy network training method based on state representation learning in the first aspect of this disclosure.

[0109] According to a fifth aspect of this disclosure, an electronic device is provided, comprising:

[0110] Processor; and

[0111] A memory that stores computer-readable instructions, which, when executed by a processor, implement the method as described in the above embodiments.

[0112] According to a sixth aspect of this disclosure, a robot is provided, comprising:

[0113] Processor; and

[0114] A memory that stores computer-readable instructions, which, when executed by a processor, implement the method as described in the above embodiments.

[0115] In one exemplary embodiment of this disclosure, the robot includes any one of legged robots, quadruped robots, bipedal robots, wheeled robots, wheel-legged robots, quadrupedal robots, humanoid robots, and mobile robots.

[0116] According to a seventh aspect of this disclosure, a computer-readable storage medium is provided that stores computer program code instructions, which, when invoked by a robot's processor, cause the robot to perform the method as described in the above embodiments.

[0117] As can be seen from the above technical solution, this disclosure possesses at least one of the following advantages and positive effects:

[0118] This disclosure acquires privileged states and ontology-aware states during the training phase and performs zero-masking on the privileged states to construct masked states, thereby forming a set of structurally complementary input samples. The privileged states and their masked states are respectively input into a policy encoder with shared weights, and further, a predictor generates prediction vectors. Based on this, an unsupervised contrastive loss function is constructed using the prediction vectors and the counterpart representation vectors, and the parameters of the policy encoder are updated according to the contrastive loss function. Through this structured training path, the policy encoder, while maintaining consistency with the input dimension during deployment, can effectively receive structural supervision implicit in the privileged information during the training phase. This promotes a more discriminative and task-relevant distribution of the ontology-aware states in the embedding space, improving the ability to express key dynamic changes, and ultimately enhancing the accuracy and stability of the action policies generated by the control policy network. Attached Figure Description

[0119] To more clearly illustrate the technical solutions in the embodiments of this disclosure, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0120] Figure 1 A system architecture diagram is shown that can be applied to the robot motion control policy network training method and robot motion control method based on state representation learning in the embodiments of this disclosure.

[0121] Figure 2 A flowchart illustrating a robot motion control policy network training method based on state representation learning is shown in an embodiment of this disclosure.

[0122] Figure 3 A schematic diagram of a process for obtaining the body perception state of a robot at the current time step is shown in an embodiment of this disclosure.

[0123] Figure 4 A schematic diagram of a process for obtaining the privileged state of a robot at the current time step is shown in an embodiment of this disclosure.

[0124] Figure 5 A schematic diagram of a process for zero-masking privileged states in an embodiment of this disclosure is shown.

[0125] Figure 6 A schematic diagram of a process for generating representation vectors according to an embodiment of this disclosure is shown.

[0126] Figure 7 A schematic diagram of a process for generating prediction vectors according to an embodiment of this disclosure is shown.

[0127] Figure 8 A flowchart illustrating the calculation of a contrastive loss function is shown in an embodiment of this disclosure.

[0128] Figure 9 This illustration shows a schematic diagram of the principle of training a robot motion control policy network based on state representation learning in an embodiment of this disclosure.

[0129] Figure 10 A schematic diagram of a process for updating policy encoder parameters is shown in an embodiment of this disclosure.

[0130] Figure 11 This illustration shows a schematic diagram of the principle of another robot motion control policy network training based on state representation learning in an embodiment of this disclosure.

[0131] Figure 12 A schematic diagram of a process for updating parameters of a policy encoder and a policy head network according to an embodiment of this disclosure is shown.

[0132] Figure 13 A schematic diagram of a process for updating value network parameters is shown in an embodiment of this disclosure.

[0133] Figure 14 A schematic diagram of a state representation learning framework for training a motion control strategy network for human robots is shown in an embodiment of this disclosure.

[0134] Figure 15 A flowchart illustrating a robot motion control method according to an embodiment of this disclosure is shown.

[0135] Figure 16 This diagram illustrates the execution effect of the robot motion control method in a motion imitation task according to an embodiment of the present disclosure.

[0136] Figure 17 This diagram illustrates the execution effect of a robot motion control method in a speed tracking task according to an embodiment of the present disclosure.

[0137] Figure 18 This illustration shows a performance change trend of different training methods throughout the training process in an embodiment of this disclosure.

[0138] Figure 19 A block diagram of a robot motion control policy network training device based on state representation learning is shown in an embodiment of this disclosure.

[0139] Figure 20 A block diagram of a robot motion control device according to an embodiment of the present disclosure is shown.

[0140] Figure 21 A schematic diagram of a robot according to an embodiment of this disclosure is shown.

[0141] Figure 22 A schematic diagram of the structure of a computer system suitable for implementing the embodiments of the present disclosure is shown.

[0142] Figure 23 A schematic diagram of a computer-readable storage medium according to an embodiment of the present disclosure is shown. Detailed Implementation

[0143] In this disclosure, the terms "first" and "second" are used for description only and do not indicate relative importance or imply the number of technical features. Therefore, the features referred to as "first" or "second" may explicitly or implicitly include at least one of those features. "A plurality of" means at least two, unless otherwise expressly defined.

[0144] First, the relevant terms used in the exemplary embodiments of this disclosure will be explained:

[0145] State representation learning is the process of automatically extracting key features from raw observation information using a model, and transforming high-dimensional observation data containing redundancy or noise into compact, low-dimensional representation vectors that reflect key task information. By more effectively representing state features, subsequent policy networks can generate stable and accurate action policies based on this representation.

[0146] A state representation learning model framework refers to the overall network structure used to perform state representation learning and subsequent policy optimization. The state representation learning model framework in this disclosure includes, but is not limited to, modules such as a policy encoder, a predictor, a policy head network, and a value network. The policy encoder receives the ontology-aware state or the privileged state after zero-masking processing to generate a low-dimensional representation vector. The predictor further maps the representation vector output by the policy encoder to generate a prediction vector, which is used to construct a contrastive loss, thereby improving the stability and discriminative ability of the representation vector. During the reinforcement learning phase, the policy head network maps the representation vector generated by the policy encoder to an action distribution, allowing sampling to obtain the joint control commands or action residuals for the current time step. The value network estimates the value information of the current state based on the representation vector, such as the state value or advantage function, providing value guidance for policy optimization. The state representation learning model framework in this disclosure can simultaneously complete representation learning and policy learning during the training phase, enabling the model to generate stable and reliable robot motion control policies during the deployment phase with input conditions relying solely on the ontology-aware state.

[0147] Propriocity perception state: refers to the observational information that the robot can directly acquire through its own sensors at the current time step, including but not limited to joint angles, joint velocities, posture information output by the inertial measurement unit, and foot contact information. Propriocity perception state can be obtained during both the training and deployment phases and is the actual input source for the motion control strategy network.

[0148] Privileged states refer to extended observational information that can be read from within the physical simulator during the training phase. However, this type of information is usually not obtainable through the robot's own perception in a real deployment environment, or even if the state information can be obtained, it is not available on a real robot. Privileged states can include physical quantities such as terrain height, local environmental structure, dynamic parameters, and contact force data, which are used to provide richer supervision signals during training to enhance the expressiveness and stability of the representation vectors generated by the policy encoder.

[0149] Masked state: refers to the state vector formed by zero-masking the privileged state. Optionally, by setting the parts of the privileged state that are not accessible during the deployment phase to zero, the masked state maintains the same dimension as the privileged state, but retains valid values ​​only in the ontology perception dimension. The masked state and the full privileged state constitute a pair of inputs with clear structural differences, used to build complementary samples in contrastive learning and prompting the policy encoder to learn more task-relevant representations.

[0150] Contrastive loss function: This refers to a loss function constructed based on the similarity between two vectors generated from the two input branches, used to constrain the policy encoder to maintain consistent representational ability under different input conditions. This disclosure adopts a contrastive loss in the form of negative cosine similarity, which improves the consistency between the privileged state and its mask state in the representation space, providing unsupervised structural supervision signals for state representation learning.

[0151] Stop gradient operation: This refers to blocking gradient propagation on a certain computation path during backpropagation, so that the vector output by that path is used only as a fixed target for forward computation, without affecting the update of related parameters.

[0152] Negative similarity: This refers to taking the negative of the cosine similarity between two normalized vectors and using it as the loss to form the optimization objective. The higher the cosine similarity and the smaller the negative value, the more consistent and structurally clear the representation vectors generated by the policy encoder and predictor during training.

[0153] L2 norm (Euclidean norm) normalization: This refers to standardizing the representation vector or prediction vector using the L2 norm, normalizing its length to the unit sphere, which is used to calculate stable cosine similarity and avoid the impact of vector scale differences on training stability.

[0154] Joint position residuals refer to the joint position increments obtained by sampling the motion distribution output by the strategy head network. They represent the adjustment amount of the robot's target joint position relative to its current position at the current time step. In robot control, joint position residuals are used to generate the target joint command for the next action.

[0155] Figure 1 A system architecture diagram is shown that can be applied to the robot motion control policy network training method and robot motion control method based on state representation learning in the embodiments of this disclosure. Figure 1 As shown, the system architecture 100 may include a terminal device 101, a robot 102, a network 103, and a server 104.

[0156] The terminal device 101 includes, but is not limited to, desktop computers, laptops, smartphones, and tablets. The terminal device 101 is equipped with a graphical user interface (GUI) to provide visual interaction during training. This interface displays in real-time information such as the robot 102's ontological perception state, privileged state, and representation vectors and prediction vectors generated by the policy encoder and predictor during the training phase, facilitating user monitoring of the training process and model convergence status. Furthermore, the terminal device 101 provides parameter configuration functions, allowing users to set relevant parameters in zero-mask processing, the network structure of the policy encoder and predictor, the weight coefficients of the contrastive loss, and training hyperparameters such as the optimizer's learning rate and momentum. Through collaborative work with the server 104, the terminal device 101 can monitor the entire training process, record data, manage parameters, and provide visual analysis of training results.

[0157] Robot 102 is used to acquire proprioceptive state during the training data generation phase. It is equipped with at least inertial measurement units (IMUs) and joint angle encoders, enabling it to acquire its own posture, velocity, and joint state data in real time. After training is completed and the motion control strategy network is deployed, robot 102 can generate corresponding joint control actions based on the motion control strategy network. It is important to note that robot 102 includes at least a main controller and joint actuators. During the deployment phase, the main controller receives proprioceptive state data from real sensors and performs forward inference from the motion control strategy network to generate a motion strategy for controlling the joint actuators. The joint actuators are then responsible for executing the actual joint movements according to this motion strategy.

[0158] Server 104 is equipped with a training module for performing state representation learning and policy training, and can execute the training process in a physical simulation environment. Specifically, server 104 can acquire the robot's ontology perception state and privileged state at the current time step in the simulation environment, perform zero-masking processing on the privileged state to generate a masked state, and input the privileged state and masked state into a policy encoder with shared weights to generate corresponding first representation vectors and second representation vectors. Further, server 104 inputs the first and second representation vectors into a predictor to obtain a first and a second prediction vector, and calculates a contrastive loss function based on the first and second prediction vectors and the second and first prediction vectors. Server 104 can update the parameters of the policy encoder based on the contrastive loss function, thereby learning to generate stable, task-relevant state representations, providing a feature basis for subsequent policy optimization. After training, server 104 can deploy the trained motion control policy network containing the policy encoder to the main controller of robot 102 to support real-time motion generation of the robot during actual operation.

[0159] Network 103 serves as the medium for providing a communication link between terminal device 101, robot 102, and server 104. Network 103 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc. It should be understood that... Figure 1 The number and type of terminal devices, robots, networks, and servers shown are merely illustrative. Depending on implementation needs, there can be any number and type of terminal devices, robots, networks, and servers; for example, the servers could be cloud servers.

[0160] Through the coordinated operation of various components in System Architecture 100, a complete process can be achieved, from the acquisition of the ontology-aware state and privileged state, zero-mask processing of the privileged state, generation of representation vectors by the policy encoder and predictor, construction and calculation of contrastive loss, iterative updating of model parameters, to deployment of the policy network after training. This effectively enhances the policy encoder's ability to extract robot state features, enabling the policy network to obtain a more stable and structurally clear input representation during the training phase, thereby improving the model's ability to recognize different state changes. Consequently, the deployed motion control policy network can generate more accurate and stable control actions in the actual operating environment, improving the robot's control accuracy and stability under complex task conditions.

[0161] This disclosure provides an exemplary implementation of a robot motion control policy network training method based on state representation learning. The state representation learning model framework employed in this method includes a policy encoder and a predictor, used to extract task-relevant low-dimensional state representations during the training phase through unsupervised contrastive learning, providing more structurally stable and expressive feature inputs for subsequent policy learning. Accordingly, reference... Figure 2 As shown, the method may include the following steps S201 to S205:

[0162] Step S201: Obtain the robot's ontological perception state and privileged state at the current time step;

[0163] Step S202: Perform zero-mask processing on the privileged state to obtain a mask state corresponding to the privileged state. The zero-mask processing includes setting at least the part of the privileged state that does not belong to the ontology perception state to zero.

[0164] Step S203: Input the privileged state and mask state into the shared-weight policy encoder to generate the corresponding first representation vector and second representation vector;

[0165] Step S204: Input the first representation vector and the second representation vector into the predictor to obtain the corresponding first prediction vector and second prediction vector;

[0166] Step S205: Calculate the contrastive loss function based on the first prediction vector and the second representation vector, and the second prediction vector and the first representation vector, and update the parameters of the policy encoder based on the contrastive loss function.

[0167] The robot motion control policy network training method based on state representation learning provided in this exemplary embodiment acquires privileged states and ontology-aware states during the training phase, and performs zero-masking processing on the privileged states to form complementary input samples. The privileged states and their masked states are respectively input into a policy encoder with shared weights, and a predictor generates prediction vectors. An unsupervised contrastive loss function is constructed based on the prediction vectors and the counterpart representation vectors to update the policy encoder parameters. With this structured training path, the policy encoder can still utilize privileged information to form more discriminative and task-relevant state representations while maintaining the same input format as in the deployment phase, thereby improving the accuracy and stability of the control policy network's action generation.

[0168] The following will provide a detailed description of the robot motion control policy network training method based on state representation learning in this example embodiment.

[0169] In step S201, the robot's ontological perception state and privileged state at the current time step are obtained.

[0170] The ontology-aware state describes the robot's motion and posture information that it can directly obtain from its own sensors at the current moment. This can include joint angles, joint velocities, posture information output by the inertial measurement unit, foot contact information, and other observations that can be collected in real time by the robot's internal sensors during the deployment phase. During the training phase, the ontology-aware state is generated by the simulation environment through the outputs of various sensors of the simulated robot, ensuring that the data structure input during training is consistent with that during the deployment phase, thereby ensuring that the policy network can operate stably on a real robot.

[0171] For example, refer to Figure 3 As shown, the robot's proprioception state at the current time step can be obtained according to steps S301 to S303:

[0172] Step S301: Obtain joint position data and joint velocity data collected by the robot's joint sensors.

[0173] Among them, joint position data is used to describe the angle information of each joint of the robot at the current time step, and joint velocity data is used to reflect the angular velocity of each joint.

[0174] These two types of data can be collected in real time by sensors such as joint encoders and joint motor feedback devices. They are the basic observations for describing the robot's motion state and are used to reflect the robot's instantaneous posture changes in each joint dimension.

[0175] Step S302: Obtain the angular velocity data and gravity vector data of the robot base collected by the robot base inertial measurement unit.

[0176] The base angular velocity data reflects the overall rotational speed of the robot in space, indicating the dynamic changes in rotation of the robot at the current time step. The gravity vector data represents the direction of gravity, from which the attitude orientation of the robot's base can be inferred.

[0177] These two types of data can be output in real time by the inertial measurement unit installed at the robot's base. They are important observations describing the robot's overall posture and motion dynamics and can be used to construct a basic posture reference frame for the robot in space.

[0178] Step S303: The joint position data, joint velocity data, base angular velocity data, and gravity vector data are spliced ​​together to form the body perception state.

[0179] Optionally, the acquired joint position data, joint velocity data, base angular velocity data, and gravity vector data can be arranged in a preset order and then concatenated to form the proprioceptive state corresponding to the current time step. The preset order can be set based on the position of each observation in the state vector. For example, the joint position data can be arranged first, followed by the joint velocity data, and then the base angular velocity data and gravity vector data can be added.

[0180] By adopting a consistent splicing order, it can be ensured that the input dimensions and data structures remain consistent between the training and deployment phases, thereby enabling the policy encoder to stably receive consistent state inputs in different operating environments.

[0181] Understandably, privileged states consist of extended state information that can be directly read from the physical simulation environment during the training phase. This type of information is typically unavailable through sensors during the real robot deployment phase. Privileged states provide a more comprehensive physical quantity description for training, enabling the formation of stable and structured supervisory signals in state representation learning, thereby enhancing the expressive power of the representation vectors generated by the policy encoder.

[0182] For example, refer to Figure 4 As shown, the robot's privileged state at the current time step can be obtained according to steps S401 to S403:

[0183] Step S401: Read the robot's base linear velocity data and base position data in the simulation environment.

[0184] Among them, the linear velocity data of the base reflects the overall translational motion characteristics of the robot, while the position data of the base is used to describe the spatial position of the robot in the global coordinate system.

[0185] These two types of data can be directly generated by the simulation engine during the dynamics calculation process. They have higher accuracy than the data obtained by actual sensors and can provide reliable motion state information for training.

[0186] Step S402: Read the height map scan data of the terrain where the robot is located in the simulation environment.

[0187] Height map scan data describes the height distribution of the terrain around the robot on a horizontal plane, reflecting the geometric characteristics of the ground the robot is currently on. For example, in scenarios with uneven terrain, slope variations, or obstacles, height map scan data can provide local environmental structure information that is difficult to obtain directly during the deployment phase.

[0188] Step S403: Combine the body perception state with the base linear velocity data, base position data and height map scan data to construct the privileged state.

[0189] Optionally, the acquired ontology-aware state, along with the base linear velocity data, base position data, and height map scan data, are combined in a preset manner to construct the privileged state corresponding to the current time step. This preset method may include, but is not limited to: concatenating various types of state data into a one-dimensional vector in a fixed order; normalizing different data types separately before concatenation; performing feature encoding on the ontology-aware state, base dynamics information, and height map data separately before combination; performing dimensionality reduction on the height map data before combining it with other state data to form a feature vector; or aligning multiple types of data in the feature space through embedding to form a unified state vector. Using one or more of these methods, a structurally stable and information-complete privileged state can be constructed, providing high-quality input for subsequent state representation learning.

[0190] In this example, by fusing multiple types of simulation environment information with the robot's own observable states in the privileged state, a state vector with higher dimensions and more comprehensive physical information can be formed. This state vector can provide rich and stable supervision signals for the policy encoder, enabling it to learn more detailed dynamic characteristics and environmental structure representations during the training phase, laying the foundation for improving the overall performance of the policy network.

[0191] In a simulation environment, the robot model can be controlled to perform actions at the current time step, while simultaneously reading sensor observations and internal dynamic quantities output by the simulator, thereby generating both the ontological perception state and the privileged state. By acquiring these two types of state information, consistent and structurally matched data inputs are provided for subsequent zero-mask processing, representation vector generation, and contrastive loss calculation. This allows the state representation learning model to fully utilize privileged information during the training phase, enhancing its ability to represent environmental changes and robot dynamics, thus improving the overall representation learning effect.

[0192] It should be noted that in order to obtain the robot's ontological perception state and privileged state at the current time step, a training environment based on physical simulation needs to be pre-built during the training phase.

[0193] For example, a robot model and an environment model are loaded into a physics simulator. The robot model is used to simulate the dynamic characteristics of the robot's joints, links, and related sensors, while the environment model is used to provide a simulation description of the terrain, colliders, and interactive objects.

[0194] Subsequently, the robot model is controlled to execute action sequences within the environment model. These action sequences can be generated by stochastic policies, heuristic controllers, or pre-trained policies to cover diverse poses and motion states, thereby constructing a representative training sample set.

[0195] During the execution of the action sequence by the robot model, sensor observation data is recorded as the ontology perception state. Specifically, during the execution of the action sequence by the robot model, observation data output by various sensor simulation modules, such as joint angles, joint velocities, inertial measurement unit data, and contact information, are recorded in real time through the simulator interface. The ontology perception state is consistent in data type and structure with the data collected by real sensors during the deployment phase, and can serve as a common input for the policy network during both the training and deployment phases.

[0196] Simultaneously, the internal state data of the physics engine during the execution of the action sequence is recorded as privileged states. This internal state data includes, but is not limited to, ground contact depth, external force information, object pose, future trajectory, and precise dynamic quantities. This privileged state data can provide additional physical information and structural supervision for state representation learning during the training phase, enhancing the policy encoder's ability to represent environmental dynamics and robot motion laws.

[0197] In some example implementations, the internal state data of the physics engine during the execution of action sequences can be recorded during the training phase and used as a source for constructing privileged states.

[0198] Specifically, the global Cartesian position and global Cartesian velocity of each link in the robot are extracted from the physics engine. The global Cartesian position describes the spatial relationship of each link in the world coordinate system, while the global Cartesian velocity reflects the actual motion trend of the link at the current moment. These two types of data provide more complete dynamic information than the body's perceived state, which helps to form a more refined state representation during the training phase.

[0199] Extracting contact force data between the robot's feet and the ground from the physics engine, including normal contact force and tangential friction force, can reflect the mechanical characteristics of the robot's feet interacting with the environment and help improve the accuracy of state representation during the training phase.

[0200] Extracting the physical friction coefficient data of the ground in the physics engine, which reflects the material properties of the ground on which the robot is located, has an important impact on the robot's gait control and motion generation, and is used to provide additional environmental dynamics supervision for the state representation learning model.

[0201] Finally, the global Cartesian position, global Cartesian velocity, contact force data, and physical friction coefficient data are merged into the privileged state. For example, the global Cartesian position, global Cartesian velocity, foot contact force data, and ground friction coefficient data are merged into a unified state vector after dimensional alignment and formatting, forming the privileged state corresponding to the current time step.

[0202] In this example, the privileged state serves as an additional input during the training phase, providing richer structural supervision for state representation learning. This enables the policy encoder to learn more fully the environmental dynamics and body motion patterns during the training phase, thereby enhancing its expressive power in subsequent action policy generation.

[0203] In some example implementations, in order to provide more comprehensive environmental geometry information to the state representation learning model during the training phase and improve the adaptability of the motion control policy network under complex terrain conditions, the composition of privileged states can be further improved.

[0204] Specifically, a gridded sampling point is established on a horizontal plane centered on the robot base to construct a gridded terrain sampling area around the robot base. For example, in the horizontal plane where the robot base is projected, a regular two-dimensional grid can be divided along the front-back and left-right directions, with the base projection point as the origin, according to a preset grid resolution and sampling radius. A sampling point is set at the center of each grid, which can form a gridded area covering continuous terrain in the robot's foot area and its vicinity, used to characterize the undulation features of the local terrain.

[0205] Subsequently, ray detection is used to obtain the height value of each sampling point relative to the robot's foot. For example, in a physical simulation environment, starting from the coordinates of each sampling point on the horizontal plane, a ray is emitted vertically, and the intersection point of the ray with the terrain surface is detected. Based on the height difference between the height coordinates of the intersection point and the reference plane of the robot's foot, the terrain height of the corresponding sampling point is calculated. When the ray does not intersect with the terrain, a default height value can be assigned to the sampling point or interpolation can be used to ensure the numerical continuity of the height values ​​throughout the sampling area.

[0206] Next, the height values ​​of all sampling points are flattened into a one-dimensional vector. For example, the two-dimensional grid can be traversed according to a preset row-first or column-first order, and the height values ​​corresponding to each sampling point can be written into the same data structure in sequence to form a one-dimensional height vector of fixed length, which is used to express the local terrain information around the robot in a compact form.

[0207] Finally, the obtained one-dimensional vector is embedded into the tail of the privileged state. Specifically, when constructing the privileged state, we can first organize the dynamic information related to the global position, velocity, contact force, and friction coefficient of the robot links, and then add the height vector as an additional field to the end of the privileged state, so that the terrain height information participates in the state representation learning as part of the privileged state.

[0208] By introducing local terrain height encoding, the policy encoder can not only receive the robot's own dynamic information during the training phase, but also perceive the terrain changes in the foot area at the same time. This helps to learn a state representation that is more sensitive to complex terrain and more adaptable, which is beneficial to improving the stability and generalization ability of subsequent motion control strategies.

[0209] In step S202, the privileged state is zero-masked to obtain a mask state corresponding to the privileged state. The zero-masking process includes setting at least a portion of the privileged state that does not belong to the ontology perception state to zero.

[0210] To ensure that the state representation learning model can utilize both privileged information and ontology-aware information during the training phase, while maintaining an input format consistent with actual available observations during the deployment phase, the privileged state needs to be zero-masked to generate a masked state with the same structure as the privileged state but with limited information.

[0211] It should be noted that zero-masking can be performed on all or part of the non-entity-aware dimensions. For example, in the case of partial zeroing, only some fields in the non-entity-aware state dimensions can be zeroed. For the fields corresponding to the non-entity-aware states that are not zeroed, their values ​​can remain unchanged, or they can be filled with preset constants, mean values, etc., depending on the specific implementation. They can also be replaced with noise values ​​generated by preset probability distributions (such as Gaussian or uniform distributions) to form mask states with different levels of information constraint. In the case of complete zeroing, all fields in the non-entity-aware dimensions are uniformly set to zero, so that the mask state only retains the valid information corresponding to the entity-aware state.

[0212] Specifically, privileged states consist of two parts: one part is the ontologically perceived state that can be obtained during both the training and deployment phases, such as joint angles, joint velocities, and inertial measurement unit (IMU) observations; the other part is supplementary information provided only by the simulation environment during the training phase, such as link global position information, contact force information, terrain height information, and ground friction coefficient. To construct complementary state pairs that can serve as inputs for contrastive learning, it is necessary to distinguish between these two types of information from the privileged states.

[0213] In some example implementations, references Figure 5 As shown, the privileged state can be zero-masked according to steps 501 to S503 to obtain the mask state corresponding to the privileged state:

[0214] Step S501: Determine the index position in the privileged state corresponding to the dimension of the non-ontology-aware state.

[0215] Step S502: Keep the values ​​of the dimensions belonging to the ontology perception state in the privileged state unchanged.

[0216] Step S503: Force the values ​​at at least some index positions to be zero to generate a mask state.

[0217] In this example, the data dimensions belonging to the ontology-aware state and those belonging to the non-aware state can be identified first based on the predefined field structure of the privileged state. An index set for the non-aware state dimensions can then be established based on the identification results. The predefined field structure specifies the order of various data types in the privileged state. For example, data fields corresponding to the ontology-aware state are placed in the first few dimensions of the corresponding state vector, while extended information such as link global position, link velocity, contact force, and terrain height are arranged sequentially in the subsequent dimensions according to a preset order. Once this field structure is determined, the dimensional range of various data types in the state vector is fixed, thus allowing the determination of the index positions corresponding to the non-aware state.

[0218] Next, for the dimensions belonging to the ontology-aware state within the privileged state, their original values ​​remain unchanged to ensure that the data in the mask state for these dimensions is consistent with the actual observations during deployment. Subsequently, the zeroing method can be selected according to actual needs, such as performing zeroing processing only on some dimensions, or uniformly setting the values ​​of all dimensions belonging to the non-ontology-aware state to zero, that is, clearing the privileged information of these dimensions and forming the mask state by filling in zero values.

[0219] In the exemplary implementation of this disclosure, the training process is primarily based on a mask format where all non-ontology-aware state dimensions are set to zero. Zero-filling ensures that the input format remains numerically identical to that of the privileged state, while also explicitly indicating that these dimensions are unavailable or unobservable during deployment. However, it should be understood that this disclosure is not limited to setting all dimensions to zero; partial zeroing can also be an optional implementation method depending on actual needs.

[0220] By applying zero-masking to the privileged state, the privileged state and the masked state are kept consistent in the input dimension but complementary in information content. This allows them to be used as positive and negative samples in contrastive learning to be input into the policy encoder with shared weights. This enables a representation learning mechanism that relies on structural differences rather than label supervision, which helps improve the policy encoder's ability to extract key dynamic features.

[0221] In step S203, the privileged state and the mask state are respectively input into the policy encoder with shared weights to generate the corresponding first representation vector and second representation vector.

[0222] In the exemplary implementation of this disclosure, the policy encoder, as an important component of the state representation learning model, is used to map high-dimensional input states to a low-dimensional, task-relevant representation space. During training, to maintain consistency in model structure between the training and deployment phases, a single policy encoder structure can be employed, with both input branches sharing the same network weights.

[0223] First, the privileged state, without masking, is input into the policy encoder. Since the privileged state contains ontology-aware information as well as additional dynamic information available only during the training phase, a first representation vector containing richer environmental and bodily dynamic features is obtained through layer-by-layer feature extraction by the policy encoder. This first representation vector is used during training to provide more adequate supervision, prompting the policy encoder to learn more discriminative and structured state embeddings.

[0224] Subsequently, the masked state, after zero-masking, is input into the same policy encoder. The masked state retains only the ontology-aware information available during both training and deployment phases, with all other dimensions set to zero. Therefore, compared to the privileged state, its input content is closer to the actual observable conditions during deployment. After mapping through a policy encoder with shared weights, a second representation vector is obtained. Since the two inputs share the same policy encoder weights, the second representation vector can be compared with the first representation vector in the same representation space.

[0225] Through a dual-branch, shared-weight structure, the first representation vector reflects the state features containing complete and usable information, while the second representation vector corresponds to the state features formed by the actual observable information during the deployment phase. Subsequently, by combining the predictor output and the comparative loss calculation, the policy encoder can be guided to fully utilize the structural advantages of the privileged information from the training phase while maintaining the input constraints of the deployment phase, thereby improving the state representation learning effect and subsequent policy learning performance.

[0226] For example, refer to Figure 6 As shown, step S203 may further include steps 601 to S603:

[0227] Step S601: Construct a policy encoder containing a multi-layer fully connected neural network.

[0228] Each layer of the fully connected neural network can include a linear mapping unit, a non-linear activation function, and an optional normalization operation. Through multi-layered, step-by-step feature transformation, the policy encoder can continuously extract effective information from the input state, compressing the original high-dimensional data into a lower-dimensional, more compact representation space, thereby forming embedded features that are beneficial for subsequent policy learning.

[0229] Step S602: The privileged state is used as the first input batch to input the fully connected neural network, and the first representation vector is output.

[0230] The privileged state contains complete dynamic information and extended environmental information that can be obtained from the simulation environment during the training phase. After forward propagation by the policy encoder, a first representation vector is obtained. The first representation vector can reflect relatively complete state features and provides sufficient structural supervision for the policy encoder during contrastive learning training, which helps the policy encoder learn highly discriminative embedded representations.

[0231] Step S603: Input the mask state as the second input batch into the same fully connected neural network, and output the second representation vector.

[0232] In this design, the masked state retains only the ontology-aware data that can be directly observed during both training and deployment phases, with the remaining dimensions filled with zero values. Since the first and second input batches share the weight parameters of each layer of the fully connected neural network during computation, the masked state and the privileged state are encoded within the same network structure. Therefore, the output second representation vector resides in the same representation space as the first representation vector. The final output second representation vector reflects the state representation formed by observable information under actual deployment conditions, and is used to train the policy encoder to maintain stable expressive power even under input constraints.

[0233] In this example, both the privileged state and the masked state are feature extracted using the same policy encoder structure, ensuring consistency in the representation space. This dual-branch, weighted encoding method not only enhances the model's feature extraction capabilities during training by leveraging privileged information, but also ensures that the model can still generate reliable state representations during deployment by relying solely on ontology-aware information.

[0234] In step S204, the first representation vector and the second representation vector are respectively input into the predictor to obtain the corresponding first prediction vector and the second prediction vector.

[0235] In the exemplary implementation of this disclosure, the predictor is used to further feature map the representation vectors generated by the policy encoder, making the mapped prediction vectors more suitable for constructing contrastive losses. The predictor may consist of several fully connected network layers and activation functions. By applying additional nonlinear transformations to the representation vectors, it enables the representations generated by different input branches to establish a stable and comparable mapping relationship in the same feature space.

[0236] The first representation vector obtained based on the privileged state can be input into the predictor. Since the first representation vector contains structural features of the privileged information, a first predicted vector can be obtained through multi-layer mapping of the predictor. The first predicted vector is used to measure similarity with the second representation vector generated from the mask state when constructing the contrastive loss.

[0237] The second representation vector obtained based on the mask state can be input into the same predictor. The second predicted vector output by the predictor will be used for comparison calculation with the first representation vector. The predictor can be configured to share the same set of network parameters for the two input branches, or it can adopt an independent predictor structure. Regardless of the form adopted, it can achieve the mapping processing of the two types of input representation vectors, and this disclosure does not limit it.

[0238] The dual-branch prediction mapping structure enables the first and second representation vectors to form corresponding prediction vectors under the prediction generator's mapping. Subsequently, a contrastive loss function can be constructed based on the prediction vectors and the counterpart representation vectors, thereby enhancing the policy encoder's ability to learn the structural correlations between input states, improving the consistency and discriminativeness of state representations, and providing a more reliable embedding foundation for subsequent policy learning.

[0239] For example, refer to Figure 7 As shown, step S204 may further include steps 701 to S703:

[0240] Step S701: Construct a multilayer perceptron containing a bottleneck layer as a predictor, with the input dimension and output dimension of the predictor being the same.

[0241] The predictor can employ a multilayer perceptron structure with a bottleneck layer, such as consisting of several fully connected layers, nonlinear activation functions, and optional normalization layers. The bottleneck layer is used to compress the representation vector in the intermediate feature space, enabling the predictor to learn a more structured mapping relationship while keeping the output features compactly represented with the input features.

[0242] To facilitate dimensional alignment with the policy encoder output, the predictor's input and output dimensions can be set to be the same, ensuring that the predicted vectors can be directly used for comparison calculations.

[0243] Step S702: Input the first representation vector into the multilayer perceptron and output the first prediction vector after nonlinear transformation.

[0244] The first representation vector obtained from the privileged state is input into the multilayer perceptron as described in step S702. After undergoing several layers of linear transformation and nonlinear activation, the first representation vector is mapped to another feature space, outputting a first prediction vector. The first prediction vector carries the structural features extracted from the privileged information and is an important component in constructing the contrastive loss.

[0245] Step S703: Input the second representation vector into the multilayer perceptron and output the second prediction vector after nonlinear transformation.

[0246] The second representation vector obtained from the masked state is input into the same multilayer perceptron. The predictor can also share all weight parameters in the two input branches. The second representation vector undergoes a nonlinear transformation under the same network structure to generate the second prediction vector. By maintaining the sharing of predictor weights, it can be ensured that the two input paths form a consistent mapping relationship in the same feature space, making the two prediction vectors comparable.

[0247] In this example, the representation vectors of the privileged branch and the masked branch are processed by the predictor to obtain the corresponding prediction vectors, which provide a feature basis for subsequent calculation of the similarity between the prediction vector and the counterpart representation vector. This ensures that the predictor maintains a consistent mapping rule under different input conditions, thereby improving the stability and effectiveness of the contrastive learning process.

[0248] In step S205, a contrastive loss function is calculated based on the first prediction vector and the second representation vector, and the second prediction vector and the first representation vector, and the parameters of the policy encoder are updated based on the contrastive loss function.

[0249] In the exemplary implementation of this disclosure, contrastive loss is used to measure the feature consistency between two input branches after encoding and prediction mapping. By maximizing their similarity, the policy encoder is guided to learn a low-dimensional state representation that remains discriminative during the deployment phase.

[0250] For example, using the first predicted vector and the second representation vector as a pair of inputs, their cosine similarity is calculated. Cosine similarity reflects the proximity of two vectors in direction and is unaffected by changes in vector magnitude. A higher similarity value indicates a more consistent feature structure between the first predicted vector and the second representation vector, which is beneficial for establishing alignment between privileged branches and mask branches during training. Similarly, the same similarity calculation is performed using the second predicted vector and the first representation vector as another pair of inputs. This bidirectional structure further constrains the mapping direction of the predictor, ensuring consistency between the two input paths during training, thereby improving training stability.

[0251] After obtaining the similarity between two pairs of vectors, the cosine similarity can be negative and averaged to construct the contrastive loss function for the current training step. This contrastive loss function does not rely on additional label information but directly utilizes the difference in observability between the privileged state and the masked state, enabling the model to learn a more reliable state representation under unsupervised conditions.

[0252] After calculating the contrastive loss, backpropagation is performed on the policy encoder and predictor to update their parameters. The policy encoder obtains structural supervision from privileged information based on this contrastive loss, improving state representation capabilities while maintaining the input format during deployment. The predictor learns a stable feature mapping method constrained by the contrastive loss, ensuring consistency between the two input branches in the same feature space and avoiding training problems such as representation collapse. The model can gradually improve the robustness and usability of state representations, providing a more effective input foundation for subsequent action policy learning.

[0253] In some example implementations, references Figure 8 As shown, the contrastive loss function can be calculated according to steps 801 to S805:

[0254] Step S801: Perform a stop gradient operation on the second representation vector to obtain the first target vector.

[0255] Specifically, by performing a gradient-stopping operation on the second representation vector, the contrastive loss stops propagating gradients backward on this branch, thus avoiding the privileged branch and mask branch simultaneously putting update pressure on the policy encoder parameters. At this point, the contrastive loss will primarily act on the policy encoder through another branch, while simultaneously imposing constraints on the predictor's forward path to reduce training instability caused by competing gradients, making the overall optimization process smoother.

[0256] Step S802: Perform a stop gradient operation on the first representation vector to obtain the second target vector.

[0257] Similarly, by performing a gradient-stopping operation on the first representation vector, the gradient of this branch is completely blocked during the backpropagation phase, serving only as a stable reference in the comparison calculation process and not participating in the parameter updates of the policy encoder or predictor. The second target vector is subsequently used to calculate the similarity with the second prediction vector, and structurally corresponds to the first target vector in step S801.

[0258] By providing stable targets with stopping gradient processing for each of the two input branches, the contrastive loss has a fixed alignment benchmark in the bidirectional mapping relationship, which helps to reduce gradient competition between the two branches and improve the stability and convergence performance of the overall contrastive learning process.

[0259] Step S803: Calculate the first similarity negative value between the first predicted vector and the first target vector.

[0260] Specifically, the similarity can be measured based on the directional consistency of the two vectors. This is achieved by normalizing the first predicted vector and the first target vector, and then calculating their dot product to represent the closeness of the two vectors in the feature space. Subsequently, the negative value of this dot product is used to obtain the negative first similarity value.

[0261] For example, let the policy encoder be... The predictor is Then we have:

[0262] ,

[0263] ,

[0264] in, In a privileged state, In mask state, Let be the first representation vector. This is the second representation vector. The first prediction vector, This is the second prediction vector.

[0265] Furthermore, the first predicted vector is normalized using the L2 norm to obtain a normalized predicted vector corresponding to the first predicted vector, and the first target vector is also normalized using the L2 norm to obtain a normalized target vector corresponding to the first target vector. Next, the dot product of the normalized predicted vector and the normalized target vector is calculated, and the result of the dot product is negative to obtain the first negative similarity value, i.e.:

[0266]

[0267] in, The first similarity score is negative. This is the normalized prediction vector corresponding to the first prediction vector. This is the normalized target vector corresponding to the first target vector. The first prediction vector L2 norm, The first target vector L2 norm, This indicates the second representation vector. Perform a stop gradient operation.

[0268] Understandably, the first negative similarity value is used to characterize the degree of difference between the two vectors and serves as one of the components of the subsequent contrast loss, reflecting the feature alignment ability of the first predicted vector with the first target vector.

[0269] Step S804: Calculate the second similarity negative value between the second predicted vector and the second target vector.

[0270] Similarly, using the second predicted vector and the second target vector as a set of inputs, the same processing procedure as step S803 is repeated to obtain the second negative similarity value. The second negative similarity value is also used to characterize the degree of difference between the two vectors and serves as one of the components of the subsequent contrast loss, reflecting the feature alignment ability of the second predicted vector with the second target vector.

[0271] For example, the second predicted vector is normalized using the L2 norm to obtain a normalized predicted vector corresponding to the second predicted vector, and the second target vector is also normalized using the L2 norm to obtain a normalized target vector corresponding to the second target vector. Next, the dot product of the normalized predicted vector and the normalized target vector is calculated, and the result of the dot product is negative to obtain the second similarity negative value, i.e.:

[0272]

[0273] in, The second similarity is negative. This is the normalized prediction vector corresponding to the second prediction vector. This is the normalized target vector corresponding to the second target vector. The second prediction vector L2 norm, The second target vector L2 norm, This indicates the first representation vector. Perform a stop gradient operation.

[0274] It can be seen that the structure of negative bidirectional similarity can constrain two input branches simultaneously during training, enabling the predictor to form consistent feature mappings under different input paths, thereby enhancing the stability of the comparison relationship and improving the overall training effect.

[0275] Step S805: Add the negative values ​​of the first similarity and the second similarity to obtain the contrast loss function.

[0276] After obtaining the negative values ​​of the first and second similarities, they can be added together to obtain the contrastive loss function for the current training step. For example:

[0277]

[0278] in, To compare loss functions, The first similarity score is negative. The second similarity is negative.

[0279] The contrastive loss function obtained in this step is used as the training optimization objective. The parameters of the policy encoder and predictor are updated through backpropagation, so that the model gradually learns a low-dimensional state representation that still has stability and discriminative power under the deployment input conditions.

[0280] refer to Figure 9 The diagram illustrates the principle of training a robot motion control policy network based on state representation learning. The robot acquires a proprioceptive state 901 related to its current action. Simultaneously, it acquires a privileged state 902 containing high-dimensional environmental information within the simulation environment. The proprioceptive state 901 and the privileged state 902 are input to a policy encoder 903, which performs feature extraction and representation compression on the input states. It is important to note that the parameters of the policy encoder 903 are shared in both input branches, ensuring that both types of inputs generate comparable feature representations under the same mapping structure.

[0281] After processing by the policy encoder 903, a first representation vector 904 generated from the ontology-aware state and a second representation vector 905 generated from the privileged state are obtained. The first representation vector 904 and the second representation vector 905 are used to describe the structural features of the two types of states in the low-dimensional representation space, respectively.

[0282] During the training phase, a contrastive learning operation can be performed based on the first representation vector 904 and the second representation vector 905, enabling the policy encoder to simultaneously capture the structural correlation between the ontology-aware state and the privileged state. By minimizing the contrastive loss between the two types of representation vectors, the policy encoder can progressively align the two state distributions during the representation learning process, allowing the representation generated solely based on the ontology-aware state to implicitly contain high-dimensional information from the privileged state.

[0283] Through such Figure 9 The training process shown demonstrates that the policy encoder can obtain a representation space with stronger generalization and discrimination capabilities for the input state, providing a more reliable representation basis for the subsequent motion control policy network to make action decisions under conditions without privileged information.

[0284] It should be noted that, Figure 9 The schematic diagram shown is only for illustrating the main training idea of ​​the embodiments disclosed in this specification, namely, aligning the representation spaces of the ontology-aware state and the privileged state through a contrastive learning mechanism. Figure 9The processing path associated with the predictor is not shown. The predictor is an auxiliary network used in contrastive learning to perform nonlinear mapping on the output of the policy encoder, and it does not participate in the actual inference process of the motion control policy network after training. In a specific implementation, the predictor can optionally be set after the policy encoder to project the generated representation vector, thereby enhancing the optimization effect of the contrastive loss.

[0285] Further, refer to Figure 10 As shown, the process of updating the policy encoder parameters in step S205 can further include steps 1001 to S1003:

[0286] Step 1001: Calculate the gradient of the contrastive loss function with respect to the parameters of the predictor.

[0287] In this step, the contrastive loss function is used as the optimization objective, and its partial derivatives with respect to the trainable parameters within the predictor are calculated using automatic differentiation. Since both the first and second predicted vectors in the contrastive loss are outputs of the predictor, the backpropagation of the loss will pass through the entire forward path of the predictor, allowing the weights of each layer within the predictor to obtain corresponding gradients. These gradients are then used to adjust the feature mapping, resulting in greater directional consistency between the predicted vectors and the target vectors.

[0288] Step 1002: Calculate the gradient of the contrastive loss function with respect to the parameters of the policy encoder.

[0289] Subsequently, the gradient of the contrastive loss function with respect to the policy encoder parameters is calculated in the same manner. The policy encoder is responsible for extracting representation vectors from the privileged state and the masked state. Therefore, the part of the loss function involving the first representation vector and the second representation vector will be synchronously applied to the feature extraction layer of the policy encoder through backpropagation, so that the policy encoder adjusts the structure of the representation space according to the contrastive constraints.

[0290] Step 1003: Use the stochastic gradient descent optimizer to synchronously update the weights of the predictor and policy encoder based on the gradients of their respective parameters.

[0291] After obtaining the parameter gradients of the policy encoder and predictor, stochastic gradient descent optimizers such as SGD (Stochastic Gradient Descent), Adam (Adaptive Moment Estimation), or RMSProp (Root Mean Square Propagation) can be used to synchronously update the parameters of the policy encoder and predictor. Among them, Adam and RMSProp are adaptive optimization algorithms based on stochastic gradients, which adaptively adjust the learning rate by introducing first-order and second-order momentum estimates to adapt to the gradient scale of different parameters.

[0292] The stochastic gradient descent optimizer iteratively adjusts the current weights of the policy encoder and predictor based on the gradient and its update rule, continuously reducing the contrastive loss. By synchronizing the updates of the two networks, they can maintain a consistent convergence direction during training, thus ensuring a stable feature correspondence between the prediction branch and the representation branch.

[0293] It should be noted that in the above update process, the predictor's parameter update is only supervised by the contrastive loss gradient. In addition to receiving the gradient from the contrastive loss, the policy encoder will also receive the backpropagation signal of the policy gradient or the value function gradient during the reinforcement learning stage, thus forming dual supervision. This allows the policy encoder to simultaneously take into account the structural learning objective of the state representation and the policy optimization objective, thereby improving the overall performance and stability of the final motion control policy network.

[0294] refer to Figure 11 The diagram illustrates another principle for training a robot motion control policy network based on state representation learning. During the training phase, the simulation environment provides privileged states, allowing the real robot to directly acquire its proprioceptive state. It's important to note that privileged states include both proprioceptive and non-proprioceptive states.

[0295] To enable the policy encoder to process two types of inputs simultaneously, alignment processing is performed on the two types of states. In the privileged branch, the privileged states are directly input into the policy encoder via an identity mapping, allowing the policy encoder to utilize all the structural information contained in the privileged states. In the ontology-aware branch, zero-masking processing is performed on the privileged states, setting the state dimensions corresponding to the non-ontology-aware parts to zero, in order to construct a weak-information branch for contrastive learning.

[0296] The two types of states after alignment are input into the policy encoder 903. Figure 11 The diagram shows two policy encoders 903, representing the forward calls of the policy encoder 903 on two data branches. In actual training, the two share the same set of network parameters to ensure that the two classes of states generate comparable representation vectors in the same representation space.

[0297] The representation vector output by the policy encoder 903 in the privileged branch is further input into the predictor 1101 to project and map the representation vector generated by the policy encoder 903, so that the vector has a feature form that is more suitable for constructing similarity measurement, thereby improving the training effect of contrastive learning.

[0298] In the ontology perception branch, the representation vector output by the policy encoder 903 is used as the comparison target, but a gradient pausing operation is performed on this branch to prevent this path from participating in gradient backpropagation. This is to ensure that the optimization direction mainly comes from the supervision signal provided by the privileged state, and at the same time to avoid the two branches from collapsing synchronously in the early stage of training.

[0299] After obtaining the two types of representation vectors, a similarity loss can be constructed so that the representation vector generated by the privileged state can approximate the representation vector generated by the ontology-aware state after the predictor mapping. By minimizing the similarity loss, the policy encoder can gradually learn the structural information in the privileged state, enabling it to generate high-quality representations that are more relevant to the task and can be used for policy learning even when only the ontology-aware state is provided.

[0300] Through such Figure 11 The training process shown demonstrates that the policy encoder can significantly improve the representational ability of the network while keeping the network structure unchanged, thus enabling the reinforcement learning policy to maintain stable and reliable control performance in actual deployments that do not require privileged states.

[0301] In some example implementations, the state representation learning model framework also includes a policy head network. The policy head network, together with the policy encoder, forms the core structure of the policy learning phase, generating distributed action outputs based on the state representations extracted by the policy encoder.

[0302] refer to Figure 12 As shown, the policy encoder and policy head network can be trained according to steps 1201 to S1205:

[0303] Step S1201: Input the second representation vector into the policy head network to map and obtain the action distribution of the current time step.

[0304] The second representation vector, obtained from the masked state via the policy encoder, is used as the input to the policy head network. For example, the policy head network can consist of multiple fully connected layers, capable of mapping the representation vector to a probability distribution corresponding to the action space. This action distribution describes the probability of the robot choosing an action in the current state and forms the basis for subsequent action sampling and policy optimization.

[0305] Step S1202: Sample the target joint position residual from the motion distribution.

[0306] After obtaining the motion distribution, the target joint position residual can be obtained by sampling from the motion distribution according to a preset sampling strategy. The target joint position residual represents the target position offset of each robot joint relative to its current position at the current time step, and is determined by the motion quantity output by the strategy head network. The preset sampling strategy can be random sampling, selection of the highest probability motion, etc., and this disclosure does not limit the specific method of the sampling strategy.

[0307] The target joint position residual serves as the input for the executed action, controlling the target displacement of each joint of the robot and interacting with the environment to obtain corresponding feedback information.

[0308] Step S1203: Calculate the proximal policy optimization loss based on the reward signal obtained from the interaction between the target joint position residual and the environment.

[0309] The sampled target joint position residuals are input into the simulation environment to enable the robot to perform corresponding actions, and a reward signal is obtained based on the interaction between the robot and the environment. Subsequently, based on the reward signal, action distribution, and value estimation information, a proximal policy optimization loss is constructed and calculated. This disclosure does not limit the specific form of the proximal policy optimization loss.

[0310] The proximal policy optimization loss is used to measure the improvement effect of the current policy relative to the old policy. It is an important optimization metric in the policy learning phase and is used to ensure that policy updates improve performance while maintaining the stability of the training process.

[0311] Step S1204: Obtain the first gradient value corresponding to the contrast loss function, and obtain the second gradient value corresponding to the near-end policy optimization loss.

[0312] After obtaining the contrastive loss and the proximal policy optimization loss, backpropagation can be performed on these two losses to extract their respective gradient information. The first gradient value generated by the contrastive loss primarily reflects the consistency of the state representation between the two input branches, guiding the policy encoder to adjust its feature extraction method. The second gradient value generated by the proximal policy optimization loss reflects the direction and magnitude of the policy update, optimizing the policy head network and its input representation, enabling the action policy to gradually improve through interaction with the environment.

[0313] Step S1205: Update the parameters of the policy encoder and policy head network based on the first gradient value and the second gradient value.

[0314] In this step, the model parameters are updated using the first and second gradient values ​​obtained in step S1204. For the policy encoder, its parameters simultaneously receive structural supervision from the contrastive loss and policy supervision from the proximal policy optimization, thereby improving policy learning performance while maintaining the discriminative power of the state representation. For the policy head network, its parameter updates are mainly based on the second gradient value, allowing the action distribution to be progressively optimized with policy iteration. By comprehensively utilizing these two types of gradients, the policy encoder and policy head network can maintain a consistent optimization direction during training, thereby improving the overall stability of the training process and policy performance.

[0315] For example, in order to take into account both the contrastive loss and the proximal policy optimization loss during the policy learning process, the two types of losses can be weighted and combined to form a unified optimization objective.

[0316] First, a preset balancing coefficient can be obtained to adjust the contribution of contrastive loss to the total loss. This balancing coefficient can be set to a fixed value according to the needs of the training phase, or dynamically adjusted according to a preset strategy in multiple training rounds to take into account the emphasis on representation learning and policy updating at different training phases.

[0317] Subsequently, the contrastive loss function is multiplied by the balance coefficient to obtain the weighted contrastive loss. By weighting the contrastive loss, the influence of state representation learning on the overall policy update can be effectively controlled, ensuring that state representation learning and policy learning maintain a reasonable weight relationship during training.

[0318] Next, the near-end policy optimization loss and the weighted contrastive loss are added together to obtain the total loss function. This total loss function integrates the dual optimization directions of the action policy improvement objective and the state representation structural constraints, and can serve as a common optimization objective for both the policy encoder and the policy head network.

[0319] In some example implementations, in order to further control the frequency of the contrastive loss's participation in the overall policy update process and improve training stability, the timing of the contrastive loss's participation can be controlled according to the training iteration rounds when constructing the loss function.

[0320] Specifically, the current training iteration count can be obtained first to identify which training batch or training round the model is currently in, which is the basis for determining whether to introduce contrastive loss.

[0321] Subsequently, it is determined whether the number of training iterations meets the preset update interval condition. The update interval condition can be a fixed step size interval, such as participating in the contrastive loss once every certain number of training iterations; or it can be an interval strategy that changes dynamically based on the training phase, such as increasing the participation frequency of the contrastive loss in the early stage of training and decreasing its participation in the later stage of training to balance the role of representation learning and policy optimization. This disclosure does not limit the specific content of the update interval condition.

[0322] It should be noted that if the training iteration count meets the preset update interval condition, the near-end policy optimization loss and weighted contrast loss are added together in the current training step to obtain the total loss function as the optimization target of this round.

[0323] For example:

[0324]

[0325] in, For the total loss function, To reinforce learning loss, Let the state represent the learning loss. The scheduling function is related to the training time T. This is a preset balancing coefficient used to control the weight of the state representation learning loss in the total loss. In this example, the reinforcement learning loss... The loss is optimized using a proximal strategy, and the loss is learned by representing the state. Using comparative loss, at this time, That is .

[0326] At this point, the contrastive loss and policy loss participate in the parameter update simultaneously, which helps to enhance the representational power of the policy encoder.

[0327] Conversely, if the number of training iterations does not meet the preset update interval condition, only the proximal policy optimization loss is retained as the total loss function in the current training step, and the weighted contrastive loss is not introduced. By pausing the participation of contrastive loss in some training iterations, excessive continuous gradient sources can be avoided, which would subject the policy encoder to excessive structural constraints, thereby reducing gradient oscillations and improving the stability of the training process.

[0328] In this example, by using an interval-based approach based on training rounds, a more flexible balance can be established between policy optimization and state representation learning, enabling the model to adopt matching loss combination strategies at different training stages, thereby improving the overall training convergence efficiency and stability.

[0329] Furthermore, after obtaining the total loss function, backpropagation is performed on the total loss function to update the parameters of the policy encoder and the policy head network. For example, the gradient of the total loss function with respect to the parameters of the policy encoder and the policy head network is calculated, and the corresponding network parameters are updated based on this gradient.

[0330] By employing the weighted summation and joint optimization method in this example, the policy encoder can meet the requirements of policy optimization for action generation while maintaining structural representation capabilities. The policy head network can also continuously improve policy performance based on the shared representation space, thereby enhancing the overall effectiveness and stability of the robot motion control policy.

[0331] In some example implementations, the state representation learning model framework also includes a value network for evaluating the value of the current state, further improving training stability and sample utilization efficiency during policy learning. The value network can estimate the robot's long-term cumulative reward in the current state, and its output can serve as an important basis for policy optimization, and is updated collaboratively with the policy head network.

[0332] refer to Figure 13 As shown, the parameters of the value network can be updated according to steps S1301 to S1304:

[0333] Step 1301: Input the privileged state into the value encoder of the value network to obtain the value feature vector.

[0334] The value encoder can be composed of a multi-layer fully connected network to extract deep features related to value judgment from high-dimensional states and generate value feature vectors through feature compression and nonlinear mapping.

[0335] Because privileged states contain complete dynamic data and additional environmental information from the simulation environment, the value encoder can acquire more complete supervision information during the training phase than during the deployment phase, thereby improving the expressive power and accuracy of value assessment.

[0336] Step 1302: Input the value feature vector into the value head network of the value network and output the estimated value of the current state.

[0337] For example, a value head network can consist of several fully connected layers to further map the value feature vector, ultimately outputting a scalar-form estimated value. This estimated value measures the robot's expected cumulative reward in the current state and is a crucial component of policy optimization, such as proximal policy optimization.

[0338] Step 1303: Calculate the value function loss based on the difference between the estimated value and the actual return.

[0339] After obtaining the estimated value from the output of the value head network, the estimated value is compared with the actual return calculated based on the reward sequence to calculate the value function loss.

[0340] The true reward can be obtained by accumulating discounts on rewards over several future time steps, and is used to measure the actual benefit of the current state over a long time scale. The value function loss can take the form of mean squared error, Huber loss (a robust loss function that combines the characteristics of mean squared error and absolute error), etc., to measure the magnitude of the deviation between the estimated value and the true reward.

[0341] In this step, the value function loss reflects the accuracy of the value network assessment and serves as a basis for guiding the value network to continuously adjust its parameters to improve its value prediction capabilities.

[0342] Step 1304: Update the parameters of the value encoder and value head network using the value function loss.

[0343] For example, the gradient of the value function loss with respect to the parameters of the value encoder and value head network is calculated through backpropagation, and the parameters of the value encoder and value head network are updated using a stochastic gradient descent-type optimizer. As training iterates, the value network can gradually improve the prediction accuracy of the long-term value of the state, thus providing a more stable and reliable value guidance for the policy head network during the policy iteration process of reinforcement learning.

[0344] In this example, the value network continuously improves the accuracy of state value assessment during training and complements the policy head network. The value network provides a stable estimate of future returns for each state, while the policy head network generates action policies. Together, they stabilize the overall policy learning process and improve the algorithm's control performance in complex task scenarios.

[0345] refer to Figure 14 The diagram illustrates a state representation learning framework for training robot motion control policy networks. This framework achieves structural decoupling and joint optimization of the state representation learning process and the reinforcement learning process by encoding features of the ontology-perceived state and privileged state respectively, and applying a state representation learning loss to the output of the policy encoder or value encoder according to a preset configuration.

[0346] exist Figure 14In this process, two types of input states can be obtained: ontology-aware state and privileged state. First, the ontology-aware state is input into the policy encoder 903 to extract a first feature representation 1401 that can be used for action decision-making. The policy encoder 903 can be a multi-layer neural network structure, generating a low-dimensional representation through feature extraction of the input states. The privileged state is input into the value encoder 1403 to extract a second feature representation 1404 for value estimation. During the training phase, the value encoder 1403 can utilize privileged information to learn a more structured environmental representation, thereby providing a more accurate value estimate for policy optimization.

[0347] After obtaining the two types of feature representations, the first feature representation 1401 generated by the policy encoder can be input into the policy head network 1402 to output the action policy. The second feature representation 1404 generated by the value encoder can be input into the value head network 1405 to output the state value estimate.

[0348] To enhance the representation space's support for policy learning, the state representation learning framework can construct SRL losses based on two types of feature representations. For example, an SRL loss can be applied to the output of the policy encoder 903 or the output of the value encoder 1403 to avoid policy instability caused by simultaneously constraining the two sub-networks. The SRL loss can be a contrastive loss, a reconstruction loss, etc.

[0349] In the policy optimization part, the Proximal Policy Optimization (PPO) loss can be constructed using the proximal policy optimization method, and backpropagation can be performed to update the loss based on the action policy output by the policy head network 1402 and the value estimate output by the value head network 1405.

[0350] During parameter updates, the SRL loss and PPO loss can work together on the network structure to achieve policy optimization training based on state representation learning. For example, the parameters of the policy encoder 903, value encoder 1403, policy head network 1402, and value head network 1405 can be jointly updated so that the total loss is a weighted sum of the PPO loss and the SRL loss.

[0351] In this framework, the state representation learning process and the reinforcement learning process are independent in terms of network structure, but are coupled in the parameter update stage, thereby improving training efficiency and enhancing the generalization ability of the policy in complex tasks.

[0352] After training, the trained policy encoder and policy head network can be deployed to an actual robot system to generate robot motion commands in a real environment. Accordingly, this disclosure also provides a robot motion control method, referencing... Figure 15As shown, the method may include the following steps S1501 to S1504:

[0353] Step S1501: Obtain the robot's real-time ontological perception state and construct an all-zero vector consistent with the privileged state dimension.

[0354] During the deployment phase, the robot can acquire its own perception status in real time through its onboard sensors, such as joint position, joint velocity, base angular velocity, and gravity vector, which are used to describe the robot's current posture and motion information.

[0355] Since privileged states are only available from the simulation environment during the training phase and cannot be directly obtained during the deployment phase, it is necessary to construct an all-zero vector with the same dimensions as the privileged states. This all-zero vector serves as a padding template in subsequent steps to align the input structure of the inference phase with that of the training phase. This ensures that the representation vector processed by the policy encoder maintains the same dimensions and data layout during deployment as during training, thereby ensuring that the policy network performs inference in a consistent feature space and avoiding performance degradation caused by inconsistent input formats.

[0356] Step S1502: Fill the real-time ontology perception state into the corresponding dimension of the all-zero vector to generate the inference input state.

[0357] In this step, the real-time ontology-aware state is filled into the corresponding positions of the all-zero vector according to the dimension mapping rules used in the training phase, forming the inference input state. It can be understood that the inference input state only contains valid data in the dimension corresponding to the ontology-aware state; the other dimensions remain zero to indicate that this information is unavailable during the deployment phase.

[0358] By maintaining the same dimensional layout of the input vector as during the training phase, the policy network can be prevented from being biased in the inference process due to changes in the input structure. At the same time, it can be ensured that privileged information from the training phase is not mistakenly introduced into the deployment environment.

[0359] Step S1503: Input the inference input state into the pre-trained policy encoder to obtain the target representation vector.

[0360] The constructed inference input state is fed into the trained policy encoder. Based on its learned feature extraction capabilities, the policy encoder extracts the main features of the current state from the inference input state and generates the corresponding target representation vector. This target representation vector serves as the direct input to the policy head network, supporting the generation of subsequent action policies.

[0361] Step S1504: Input the target representation vector into the pre-trained policy head network, and output the action policy for controlling the robot's motion based on the policy head network.

[0362] Finally, the target representation vector is input into the trained policy head network. The policy head network infers action policies, such as joint position increments or joint control commands, based on the representation of the current state. The generated action policies can be sent to the robot's joint actuators to drive the robot to complete the corresponding actions.

[0363] In this example, during the deployment phase, the robot can construct inference inputs based solely on its own observable ontological perception state and achieve real-time motion control based on a trained motion control policy network (including a policy encoder and a policy head network). This allows it to maintain its motion generation capabilities and control stability during training even without privileged information.

[0364] It is understood that the policy encoder and policy head network used in the inference process are trained according to the robot motion control policy network training method based on state representation learning described in detail in other embodiments of this disclosure, and will not be repeated here.

[0365] Furthermore, the robot motion control method of this disclosure is applicable to various types of control tasks. (See reference...) Figure 16 The diagram illustrates the performance of this method in a motion imitation task. After deploying a pre-trained motion control policy network, the robot can perform complex motion imitations without privileged information, achieving stable reproduction of predefined human motion sequences. Throughout the motion execution, the robot maintains coherent joint trajectories, stable posture control, and high motion consistency, demonstrating that this method still possesses reliable motion generation capabilities even in the absence of privileged information.

[0366] refer to Figure 17 The diagram illustrates the performance of this method in a speed tracking task. After receiving progressively changing target speed commands, the robot can generate a control strategy based on real-time acquired proprioceptive perception data, enabling its own speed to quickly approach the target speed curve. This method can complete continuous speed adjustment tasks relying solely on proprioceptive perception data and possesses excellent speed following capability and command response characteristics.

[0367] The robot motion control method disclosed herein exhibits good generalization ability and control stability in various task environments. This method does not rely on privileged states used during the training phase and can achieve efficient control of complex humanoid movements in actual deployment scenarios where only ontology perception data is provided.

[0368] refer to Figure 18The diagram illustrates the performance trends of different training methods throughout the training process. The horizontal axis represents training progress as a percentage, reflecting the relative timeline from start to finish of policy training; 0% indicates training has not yet begun, and 100% indicates the entire training process is complete. The vertical axis represents the normalized score, reflecting the overall performance of the control policy at the current training stage. A higher normalized score indicates better overall performance of the trained action policy in task execution.

[0369] exist Figure 18 The document presents five different training methods: PPO (baseline reinforcement learning method), PPO+VAE (state representation learning method based on variational autoencoder), PPO+SPR (state prediction method based on temporal consistency), PPO+SimSiam (state representation method based on negative sample contrastive learning), and PPO+PvP (state representation method based on ontology-privileged contrastive learning) provided in the embodiments of this disclosure.

[0370] Depend on Figure 18 It can be seen that, compared with the other four training methods, the PPO+PvP method adopted in this disclosure can achieve a higher normalized score in the early stage of training. Moreover, in the subsequent training process, the score curve of this method remains at a high level and shows a stable upward trend. This indicates that by introducing comparative state representation learning based on ontology perception information and privileged information in the policy learning process, better feature representations can be obtained in the training stage, improving the accuracy and stability of the action policies generated by the control policy network, thereby achieving a more reliable and coherent robot motion control effect.

[0371] In an exemplary embodiment of this disclosure, a robot motion control policy network training device based on state representation learning is also provided, wherein the state representation learning model framework employed includes a policy encoder and a predictor. (Reference) Figure 19 As shown, the robot motion control policy network training device 1900 based on state representation learning includes a state acquisition module 1901, a mask processing module 1902, a representation vector generation module 1903, a prediction vector generation module 1904, and a policy network training module 1905, wherein:

[0372] The state acquisition module 1901 is used to acquire the robot's ontological perception state and privileged state at the current time step;

[0373] The mask processing module 1902 is used to perform zero mask processing on the privileged state to obtain a mask state corresponding to the privileged state. The zero mask processing includes setting at least a portion of the privileged state that does not belong to the ontology perception state to zero.

[0374] The representation vector generation module 1903 is used to input the privileged state and the mask state into the shared weight policy encoder to generate the corresponding first representation vector and second representation vector;

[0375] The prediction vector generation module 1904 is used to input the first representation vector and the second representation vector into the predictor to obtain the corresponding first prediction vector and the second prediction vector;

[0376] The parameter update module 1905 is used to calculate the contrastive loss function based on the first prediction vector and the second representation vector, and the second prediction vector and the first representation vector, and update the parameters of the policy encoder based on the contrastive loss function.

[0377] The specific details of each module in the above-mentioned robot motion control policy network training device based on state representation learning have been described in detail in the corresponding robot motion control policy network training method based on state representation learning, so they will not be repeated here.

[0378] In an exemplary embodiment of this disclosure, a robot motion control device is also provided. (See reference...) Figure 20 As shown, the robot motion control device 2000 includes a vector construction module 2001, a state generation module 2002, a vector generation module 2003, and a strategy output module 2004, wherein:

[0379] Vector construction module 2001 is used to obtain the robot's real-time ontology perception state and construct an all-zero vector consistent with the privileged state dimension;

[0380] The state generation module 2002 is used to fill the real-time ontology perception state into the corresponding dimension of the all-zero vector to generate the inference input state;

[0381] The vector generation module 2003 is used to input the inference input state into the pre-trained policy encoder to obtain the target representation vector;

[0382] The strategy output module 2004 is used to input the target representation vector into the pre-trained strategy head network and output the action strategy for controlling the robot's motion based on the strategy head network.

[0383] The policy encoder and policy head network are configured according to embodiments of this disclosure, as shown below. Figure 2 The robot motion control policy network training method based on state representation learning is shown.

[0384] The specific details of each module in the above-mentioned robot motion control device have been described in detail in the corresponding robot motion control method, so they will not be repeated here.

[0385] In an exemplary embodiment of this disclosure, a robot is also provided. The robot includes a processor and a memory, the memory storing computer-readable instructions that, when executed by the processor, implement the described method. The robot includes either a humanoid robot or a dual-arm robot. (Reference) Figure 21 The diagram shown is a schematic of a humanoid robot.

[0386] refer to Figure 22 As shown, an electronic device capable of implementing the above method is also provided. The electronic device 2200 includes a processor 2201 and a memory 2202. The memory 2202 stores computer-readable instructions, which, when executed by the processor 2201, implement the method of this disclosure.

[0387] In an exemplary embodiment of this disclosure, a computer-readable storage medium is also provided, having stored thereon computer program code instructions that, when invoked by a robot's processor, cause the robot to perform the method described in the embodiments.

[0388] refer to Figure 23 As shown, a program product 2300 for implementing the above-described method according to an embodiment of the present disclosure is described. This product may employ a portable compact disc read-only memory (CD-ROM) and include program code, and may run on a terminal device, such as a personal computer. However, the program product of the present disclosure is not limited thereto. In this document, a readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0389] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, touch terminal, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0390] Finally, the above preferred embodiments are only used to illustrate the technical solutions of this application and are not restrictive. Although this application has been described in detail, those skilled in the art should understand that changes in form and detail can be made without departing from the scope defined by the claims of this application. The dimensions in the drawings are not related to the specific physical object, and the physical object dimensions can be arbitrarily changed.

Claims

1. A method for training a robot motion control policy network based on state representation learning, characterized in that, The state representation learning model framework includes a policy encoder and a predictor; the method includes: Obtain the robot's ontological perception state and privileged state at the current time step; The privileged state is subjected to zero-masking processing to obtain a mask state corresponding to the privileged state. The zero-masking processing includes setting at least a portion of the privileged state that does not belong to the ontology perception state to zero. The privileged state and the mask state are respectively input into the policy encoder with shared weights to generate corresponding first representation vector and second representation vector; The first representation vector and the second representation vector are respectively input into the predictor to obtain the corresponding first prediction vector and the second prediction vector; A contrastive loss function is calculated based on the first prediction vector and the second representation vector, and the second prediction vector and the first representation vector, and the parameters of the policy encoder are updated based on the contrastive loss function.

2. The robot motion control policy network training method based on state representation learning according to claim 1, characterized in that, The step of performing zero-masking processing on the privileged state to obtain a masked state corresponding to the privileged state includes: Determine the index position in the privileged state corresponding to the non-ontology-aware state dimension; The values ​​belonging to the ontology perception state dimension in the privileged state are retained unchanged; The mask state is generated by forcibly assigning zero to at least some of the values ​​at the index positions.

3. The robot motion control policy network training method based on state representation learning according to claim 1, characterized in that, The zero-mask processing includes: Set all portions of the privileged state that do not belong to the ontology perception state to zero.

4. The robot motion control policy network training method based on state representation learning according to claim 1, characterized in that, The step of calculating the contrastive loss function based on the first predicted vector and the second representation vector, and the second predicted vector and the first representation vector, includes: Perform a gradient stopping operation on the second representation vector to obtain the first target vector; Perform a gradient stabilization operation on the first representation vector to obtain the second target vector; Calculate the negative first similarity value between the first predicted vector and the first target vector; Calculate the second similarity negative value between the second predicted vector and the second target vector; The first negative similarity value is added to the second negative similarity value to obtain the contrast loss function.

5. The robot motion control policy network training method based on state representation learning according to claim 4, characterized in that, The calculation of the negative first similarity value between the first predicted vector and the first target vector includes: The first prediction vector is normalized using the L2 norm to obtain a normalized prediction vector; The first target vector is normalized using the L2 norm to obtain a normalized target vector; Calculate the dot product between the normalized prediction vector and the normalized target vector; The negative value of the dot product is obtained to obtain the first negative similarity value.

6. The method for training a robot motion control policy network based on state representation learning according to claim 1, characterized in that, The state representation learning model framework further includes a policy head network; the method further includes: The second representation vector is input into the policy head network to map the action distribution at the current time step. The target joint position residual is obtained by sampling from the motion distribution; Based on the reward signal obtained from the interaction between the target joint position residual and the environment, the proximal strategy optimization loss is calculated. Obtain the first gradient value corresponding to the contrast loss function, and obtain the second gradient value corresponding to the near-end policy optimization loss; The parameters of the policy encoder and the policy head network are updated based on the first gradient value and the second gradient value.

7. The robot motion control policy network training method based on state representation learning according to claim 6, characterized in that, The step of updating the parameters of the policy encoder and the policy head network based on the first gradient value and the second gradient value includes: Obtain the preset balance coefficient; Multiplying the contrast loss function by the balance coefficient yields the weighted contrast loss; The total loss function is obtained by adding the near-end policy optimization loss to the weighted comparison loss; Backpropagation is performed on the total loss function to update the parameters of the policy encoder and the policy head network.

8. The method for training a robot motion control policy network based on state representation learning according to claim 7, characterized in that, The step of adding the near-end policy optimization loss to the weighted contrastive loss to obtain the total loss function includes: Get the current training iteration count; Determine whether the training iteration round count meets the preset update interval condition; If the update interval condition is met, the near-end policy optimization loss and the weighted comparison loss are added together to obtain the total loss function.

9. The method for training a robot motion control policy network based on state representation learning according to claim 8, characterized in that, The method further includes: If the update interval condition is not met, then only the near-end policy optimization loss is retained as the total loss function.

10. The method for training a robot motion control policy network based on state representation learning according to claim 1, characterized in that, The policy encoder that inputs the privileged state and the mask state into the shared weights includes: Construct a policy encoder that includes a multi-layer fully connected neural network; The privileged state is used as the first input batch to input the fully connected neural network, and the first representation vector is output. The mask state is used as a second input batch to be input into the same fully connected neural network, and the second representation vector is output. In this process, the first input batch and the second input batch share the weight parameters of each layer of the fully connected neural network.

11. The method for training a robot motion control policy network based on state representation learning according to claim 1, characterized in that, The step of inputting the first representation vector and the second representation vector into the predictor includes: A multilayer perceptron containing a bottleneck layer is constructed as the predictor, wherein the input dimension and output dimension of the predictor are the same. The first representation vector is input into the multilayer perceptron, and the first prediction vector is output after nonlinear transformation. The second representation vector is input into the multilayer perceptron, and the second prediction vector is output after nonlinear transformation.

12. The robot motion control policy network training method based on state representation learning according to claim 1, characterized in that, The state representation learning model framework further includes a value network; the method further includes: The privileged state is input into the value encoder of the value network to obtain the value feature vector; The value feature vector is input into the value head network of the value network, and the estimated value of the current state is output. Calculate the value function loss based on the difference between the estimated value and the actual return; The parameters of the value encoder and the value head network are updated using the value function loss.

13. The method for training a robot motion control policy network based on state representation learning according to claim 1, characterized in that, Obtaining the robot's ontological perception state at the current time step includes: Acquire joint position and velocity data collected by the robot's joint sensors; Acquire the angular velocity data and gravity vector data of the robot base collected by the robot base inertial measurement unit; The joint position data, joint velocity data, base angular velocity data, and gravity vector data are spliced ​​together to form the body perception state.

14. The robot motion control policy network training method based on state representation learning according to claim 1, characterized in that, Obtain the robot's privileged state at the current time step, including: Read the robot's base linear velocity data and base position data in the simulation environment; Read the height map scan data of the terrain where the robot is located in the simulation environment; The privileged state is constructed by combining the body perception state with the base linear velocity data, the base position data, and the height map scan data.

15. The method for training a robot motion control policy network based on state representation learning according to claim 1, characterized in that, Before obtaining the robot's ontological perception state and privileged state at the current time step, the following steps are also included: Load the robot model and environment model into the physics simulator; Control the robot model to execute a sequence of actions in the environment model; The sensor observation data during the execution of the action sequence is recorded as the body's perception state; The internal state data of the physics engine during the execution of the action sequence is recorded as a privileged state.

16. The method for training a robot motion control policy network based on state representation learning according to claim 15, characterized in that, The recorded internal state data of the physics engine during the execution of the action sequence is used as a privileged state, including: Extract the global Cartesian position and global Cartesian velocity of each link in the robot from the physics engine; Extract the contact force data between the robot's feet and the ground from the physics engine; Extract the physical friction coefficient data of the ground from the physics engine; The global Cartesian position, the global Cartesian velocity, the contact force data, and the physical friction coefficient data are merged into the privileged state.

17. The method for training a robot motion control policy network based on state representation learning according to claim 16, characterized in that, The method further includes: Establish gridded sampling points on a horizontal plane with the robot base as the center; The height value of each sampling point relative to the robot's foot is obtained through X-ray detection; Flatten the height values ​​of all sampling points into a one-dimensional vector; The one-dimensional vector is embedded into the tail of the privileged state.

18. The method for training a robot motion control policy network based on state representation learning according to claim 1, characterized in that, The updating of the policy encoder parameters based on the contrastive loss function includes: Calculate the gradient of the contrastive loss function with respect to the parameters of the predictor; Calculate the gradient of the contrastive loss function with respect to the parameters of the policy encoder; The weights of the predictor and the policy encoder are updated synchronously based on the gradients of their respective parameters using a stochastic gradient descent optimizer. The parameter updates of the policy encoder are subject to dual supervision from the policy gradient and the contrastive loss gradient.

19. A robot motion control method, characterized in that, include: Obtain the robot's real-time ontological perception state and construct an all-zero vector consistent with the privileged state dimension; The real-time ontology perception state is filled into the corresponding dimension of the all-zero vector to generate the inference input state; The inference input state is input into a pre-trained policy encoder to obtain the target representation vector; The target representation vector is input into a pre-trained policy head network, and an action policy for controlling the robot's movement is output based on the policy head network. The policy encoder and the policy head network are obtained by the robot motion control policy network training method based on state representation learning according to any one of claims 1 to 18.

20. A robot motion control policy network training device based on state representation learning, characterized in that, The state representation learning model framework includes a policy encoder and a predictor; the device includes: The state acquisition module is used to acquire the robot's ontological perception state and privileged state at the current time step; A masking module is used to perform zero-mask processing on the privileged state to obtain a masked state corresponding to the privileged state. The zero-mask processing includes setting at least a portion of the privileged state that does not belong to the ontology perception state to zero. The representation vector generation module is used to input the privileged state and the mask state into the policy encoder with shared weights to generate corresponding first representation vectors and second representation vectors; A prediction vector generation module is used to input the first representation vector and the second representation vector into the predictor to obtain the corresponding first prediction vector and second prediction vector; The parameter update module is used to calculate a contrastive loss function based on the first prediction vector and the second representation vector, and the second prediction vector and the first representation vector, and update the parameters of the policy encoder based on the contrastive loss function.

21. A robot motion control device, characterized in that, include: The vector construction module is used to obtain the robot's real-time ontology perception state and construct an all-zero vector consistent with the privileged state dimension. The state generation module is used to fill the real-time ontology perception state into the corresponding dimension of the all-zero vector to generate the inference input state; The vector generation module is used to input the inference input state into a pre-trained policy encoder to obtain the target representation vector; The strategy output module is used to input the target representation vector into a pre-trained strategy head network and output an action strategy for controlling the robot's movement based on the strategy head network. The policy encoder and the policy head network are obtained by the robot motion control policy network training method based on state representation learning according to any one of claims 1 to 18.

22. An electronic device, characterized in that, include: processor; as well as A memory storing computer-readable instructions that, when executed by the processor, implement the method as described in any one of claims 1 to 19.

23. A robot, characterized in that, include: processor; as well as A memory storing computer-readable instructions that, when executed by the processor, implement the method as described in claim 19.

24. The robot according to claim 23, characterized in that, The robots include mobile robots.

25. The robot according to claim 23, characterized in that, The robots include humanoid robots.

26. The robot according to claim 23, characterized in that, The robot includes either a legged robot or a wheeled robot.

27. The robot according to claim 23, characterized in that, The robot includes a wheeled robot.

28. The robot according to claim 23, characterized in that, The robot includes any one of quadruped robots, bipedal robots, and four-wheeled robots.

29. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program code instructions that, when invoked by the robot's processor, cause the robot to perform the method as described in any one of claims 1 to 19.

Images